Cloud server load balancing method and system based on client computing power cooperation

Abstract

The application provides a cloud server load balancing method and system based on client computing power cooperation, relating to the technical field of client computing power cooperation, wherein the method comprises: first acquiring video analysis tasks of a cloud server, real-time video streams and state data of clients, and historical load data. Content feature information is extracted from the video stream through a lightweight convolutional network, and then an attention mechanism is used to analyze the key of the video frame to generate a processing priority label sequence. At the same time, based on a load prediction model of a long short-term memory network, the future computing power availability quantitative value of each client is predicted according to historical and real-time data. Then the video task is decomposed into multiple computing slices, and the slices are dynamically arranged and distributed according to the priority label and the computing power prediction value, finally realizing efficient computing power cooperation and load balancing between the cloud server and the client. The application improves the distribution efficiency of cloud video processing tasks and the utilization effect of client computing power resources.

Description

Technical Field

[0001] This application relates to the field of client-side computing power collaboration technology, and in particular to a cloud server load balancing method and system based on client-side computing power collaboration. Background Technology

[0002] With the widespread adoption of large-scale video surveillance platforms and video conferencing systems, massive amounts of client devices continuously generate video stream data that requires real-time processing. These applications demand real-time analysis and feedback of video content, such as tasks like facial recognition or motion detection, placing sustained high load pressure on cloud computing resources. Simultaneously, the client devices participating in the system often possess varying degrees of idle graphics processing unit (GPU) power. Therefore, effectively coordinating and utilizing these distributed computing resources to achieve dynamic load distribution on the cloud has become a pressing technical requirement.

[0003] One existing targeted solution relies on assessing the current static computing power status of client devices for task allocation. This solution periodically collects real-time resource utilization and network conditions from each client, breaks down cloud computing tasks into uniformly sized sub-task blocks, and allocates them based on the client's remaining computing power, aiming to achieve load balancing at the basic level.

[0004] However, this existing solution primarily relies on the client's instantaneous state for task allocation, resulting in a lag in responding to dynamic fluctuations in computing resources. The task partitioning fails to differentiate based on the specific analytical needs of the video content, and the allocation strategy has limitations in adapting to changes in network conditions and client computing power, thus hindering further improvements in overall processing efficiency and resource utilization. Summary of the Invention

[0005] This application provides a cloud server load balancing method and system based on client-side computing power collaboration, which solves the problems of low distribution efficiency of cloud video processing tasks and poor utilization of client-side computing power resources in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a cloud server load balancing method based on client-side computing power collaboration, comprising:

[0007] Obtain pending video analysis tasks from the cloud server, real-time video stream data uploaded by multiple clients, real-time status information of each client, and historical load data;

[0008] The video stream data is processed using a lightweight convolutional network to generate video content feature information;

[0009] An attention mechanism is used to perform video frame criticality analysis on the video content feature information to generate a processing priority label sequence;

[0010] Based on the historical load data and the real-time status information, and combined with the load prediction model established based on the Long Short-Term Memory network, the computing power availability quantification value of each client in the future time period is predicted.

[0011] The video analysis task to be processed is decomposed into multiple computing slices, and each computing slice is dynamically arranged and allocated based on the processing priority label sequence and the computing power availability quantification value, so as to realize computing power collaboration and load balancing between the cloud server and multiple clients.

[0012] Optionally, the step of processing the video stream data through a lightweight convolutional network to generate video content feature information includes:

[0013] The video stream data is sampled in a time sequence to obtain a sequence of sampled video frames consisting of multiple equally spaced video frames;

[0014] The sampled video frame sequence is input into a lightweight convolutional network, and the first feature extraction layer of the lightweight convolutional network captures local details of the video frames to obtain a primary feature map.

[0015] The intermediate feature map is obtained by combining features from the primary feature map through the second feature extraction layer of the lightweight convolutional network.

[0016] The intermediate feature map is integrated through the third feature extraction layer of the lightweight convolutional network using a feature channel weighted fusion method to generate a video content feature map.

[0017] Based on the video content feature map, video content feature information is generated.

[0018] Optionally, generating video content feature information based on the video content feature map includes:

[0019] The scene dynamic change rate data is parsed from the video content feature map;

[0020] A temporal convolutional network is used to analyze the facial feature vectors within a preset time window in the video content feature map to generate facial occurrence frequency data.

[0021] By comparing the similarity of adjacent video frames in brightness and contrast distribution in the video content feature map, inter-frame structural similarity data is generated.

[0022] The scene dynamic change rate data, the face occurrence frequency data, and the inter-frame structural similarity data are fused across modal features to form video content feature information.

[0023] Optionally, the step of employing a temporal convolutional network to analyze the facial feature vectors within a preset time window in the video content feature map to generate facial frequency data includes:

[0024] Extract multiple facial feature vectors within a preset time window from the video content feature map;

[0025] Arrange all facial feature vectors in chronological order to form a temporal sequence of facial features;

[0026] The temporal sequence of facial features is input into a temporal convolutional network. Through the causal convolutional layer of the temporal convolutional network, local features in the temporal dimension of the temporal sequence of facial features are extracted to obtain an initial feature sequence.

[0027] Long-range dependencies are extracted from the initial feature sequence through the dilated convolutional layer of the temporal convolutional network, and a target feature sequence is generated based on the long-range dependencies.

[0028] Perform a global pooling operation on the target feature sequence to generate a pooled feature vector;

[0029] A linear transformation is performed on the pooled feature vector to obtain a statistical feature vector that characterizes the frequency of face occurrence.

[0030] The statistical feature vectors are normalized to generate face frequency data.

[0031] Optionally, the step of using an attention mechanism to perform video frame criticality analysis on the video content feature information to generate a processing priority label sequence includes:

[0032] The video content feature information is segmented using a sliding window method to obtain multiple overlapping local feature windows;

[0033] By comparing the correlation between the features of each time step within each local feature window and the features of the central time step, a self-attention score is generated for each local feature window.

[0034] The self-attention scores of each time step under different sliding windows are weighted and aggregated to generate the comprehensive attention weight for each time step;

[0035] The comprehensive attention weight is compared with a preset dynamic threshold, and time steps in which the comprehensive attention weight is greater than the dynamic threshold are marked as key frames. A first priority label is generated based on the key frames.

[0036] The time step corresponding to the comprehensive attention weight being less than the dynamic threshold is marked as a non-key frame. Based on the non-key frame, a second priority label is generated, and the first priority is greater than the second priority.

[0037] For the time step where the comprehensive attention weight is equal to the dynamic threshold, interpolation is performed based on the labels of the adjacent time steps corresponding to the time step to determine the corresponding priority label;

[0038] Arrange the priority labels of all time steps in chronological order to form a processing priority label sequence.

[0039] Optionally, the step of predicting the quantified value of computing power availability for each client in the future time period based on the historical load data and the real-time status information, combined with a load prediction model established based on a long short-term memory network, includes:

[0040] The historical load data and the real-time status information are time-series aligned to form client status time-series data;

[0041] The client state time series data is input into the load prediction model. The long-term dependency relationship of the client state time series data is modeled through the memory unit of the load prediction model to obtain a time series feature representation.

[0042] The time series feature representation is mapped to a multi-dimensional feature space through the fully connected layer of the load prediction model to generate a high-dimensional feature vector.

[0043] The high-dimensional feature vector is subjected to nonlinear regression calculation through the output layer of the load prediction model to generate computing power availability prediction values ​​for multiple consecutive time points in the future.

[0044] The predicted computing power availability is quantized and encoded to generate a quantized computing power availability value.

[0045] Optionally, the dynamic orchestration and allocation of each computing slice based on the processing priority label sequence and the computing power availability quantification value includes:

[0046] Based on the video time interval corresponding to each priority label in the processing priority label sequence, determine the video segment to be processed from the computation slice, and determine the amount of computational resources required for each video segment to be processed;

[0047] The computing power of each client is evaluated based on the aforementioned computing power availability quantification value;

[0048] All computing slices are dynamically arranged according to a preset priority order. Based on the matching relationship between the amount of computing resources and the computing power, the dynamically arranged computing slices are allocated to clients with different computing power availability quantification values. During the allocation process, the actual load of each client is monitored in real time.

[0049] The allocation strategy for computing slices is dynamically adjusted based on the actual load conditions.

[0050] Secondly, this application provides a cloud server load balancing system based on client-side computing power collaboration, comprising:

[0051] The acquisition module is used to acquire the video analysis tasks to be processed on the cloud server, the video stream data uploaded in real time by multiple clients, the real-time status information of each client, and the historical load data.

[0052] The generation module is used to process the video stream data through a lightweight convolutional network to generate video content feature information;

[0053] The analysis module is used to perform video frame criticality analysis on the video content feature information using an attention mechanism to generate a processing priority label sequence;

[0054] The prediction module is used to predict the quantified value of computing power availability for each client in the future time period based on the historical load data and the real-time status information, combined with the load prediction model established based on the long short-term memory network.

[0055] The decomposition module is used to decompose the video analysis task to be processed into multiple computing slices, and to dynamically arrange and allocate each computing slice based on the processing priority label sequence and the computing power availability quantification value, so as to realize computing power collaboration and load balancing between the cloud server and multiple clients.

[0056] Thirdly, this application provides an electronic device, comprising:

[0057] Memory, used to store computer programs;

[0058] A processor, used to execute the computer program, implements the steps of the cloud server load balancing method based on client-side computing power collaboration as described in the first aspect above.

[0059] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the cloud server load balancing method based on client-side computing power collaboration as described in the first aspect above.

[0060] This application provides a cloud server load balancing method based on client-side computing power collaboration. The method includes: acquiring a video analysis task to be processed on the cloud server, video stream data uploaded in real time by multiple clients, real-time status information of each client, and historical load data; processing the video stream data using a lightweight convolutional network to generate video content feature information; performing video frame criticality analysis on the video content feature information using an attention mechanism to generate a processing priority label sequence; predicting the computing power availability quantification value of each client within a future time period based on the historical load data and the real-time status information, combined with a load prediction model established based on a long short-term memory network; decomposing the video analysis task to be processed into multiple computing slices, and dynamically orchestrating and allocating each computing slice based on the processing priority label sequence and the computing power availability quantification value, to achieve computing power collaboration and load balancing between the cloud server and multiple clients.

[0061] The technical solution provided in this application has the following beneficial effects:

[0062] This application provides a comprehensive and multi-dimensional data foundation for subsequent collaborative computing and load balancing, ensuring the completeness and timeliness of decision-making. It achieves effective understanding of video stream content and compact representation of key features, providing information support for prioritizing subsequent analysis tasks while reducing the computational resource consumption of the feature extraction process. It can automatically identify key segments in the video stream that require priority processing, allowing computational resources to focus on high-value content, improving the targeting and overall efficiency of task processing. It enables trend prediction of changes in client computing power over a future period, providing forward-looking guidance for dynamic task allocation and enhancing the adaptability and accuracy of load balancing strategies. It achieves refined and dynamic matching of computing tasks and distributed computing resources, promoting efficient operation of computing power collaboration between cloud servers and clients, and effectively improving the overall load balancing level of the system.

[0063] Furthermore, this application first performs temporal sampling on the video stream to obtain a frame sequence, and then uses a lightweight convolutional network to extract and fuse features layer by layer. The first layer captures local details, the second layer combines features to form more semantic information, and the third layer integrates features through weighted fusion. Finally, the required feature information is parsed based on the generated feature map.

[0064] Furthermore, this hierarchical feature extraction and fusion mechanism can effectively control computational complexity while ensuring feature representation capabilities, thereby efficiently generating feature information that accurately reflects the dynamic characteristics of video content, laying a reliable foundation for subsequent critical analysis and priority determination.

[0065] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 A flowchart illustrating a cloud server load balancing method based on client-side computing power collaboration, provided as an embodiment of this application;

[0068] Figure 2 A schematic diagram illustrating a specific implementation of a cloud server load balancing method based on client-side computing power collaboration, provided in an embodiment of this application;

[0069] Figure 3 This is a schematic diagram of the structure of a cloud server load balancing system based on client-side computing power collaboration, provided as an embodiment of this application. Detailed Implementation

[0070] Existing load balancing solutions for video analytics tasks primarily rely on the instantaneous static computing power of client devices for task allocation. In real-world scenarios where client computing resources and network conditions fluctuate continuously, this approach can lead to delays in task scheduling responsiveness. Furthermore, the evenly distributed computing task blocks fail to adequately consider the varying urgency of processing different video content, resulting in room for improvement in the timeliness of critical task processing and the overall utilization of distributed computing resources.

[0071] To address the aforementioned challenges, this application proposes a cloud server load balancing method based on client-side computing power collaboration. The core of this method lies in first intelligently analyzing video content to identify key segments and prioritize their processing, while simultaneously predicting future computing power trends by combining historical and real-time client status. Then, based on task priorities and predicted computing power, cloud tasks are finely decomposed and dynamically allocated to the most suitable clients for processing. This method effectively overcomes the shortcomings of existing solutions in adapting to dynamic changes and lacking differentiation in task division by proactively and meticulously matching task characteristics with resource status, thereby improving task processing efficiency and overall utilization efficiency of computing resources in complex and fluctuating network environments.

[0072] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0073] The core of this application is to provide a cloud server load balancing method based on client-side computing power collaboration, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0074] Step 101: Obtain the pending video analysis tasks from the cloud server, real-time video stream data uploaded by multiple clients, real-time status information of each client, and historical load data.

[0075] In step 101, the video analysis task to be processed refers to the work unit that the cloud server needs to complete to intelligently analyze the video stream, such as facial recognition in a video conference or detection of abnormal behavior in surveillance video. Video stream data refers to a continuous sequence of images uploaded by multiple client devices and arranged in chronological order. Real-time status information refers to a set of data collected at a certain moment that reflects the current operating status of the client devices, mainly including available graphics processor computing power data (measuring the device's remaining computing power), memory occupancy data (measuring the device's memory usage), network uplink bandwidth data (measuring the speed at which the device sends data to the network), and network jitter data (measuring the stability of network transmission latency). Historical load data refers to records of the client devices' computing power usage over a past period, reflecting the patterns and regularities of changes in device computing power.

[0076] In this embodiment of the application, the cloud server's data interface simultaneously receives video analysis tasks to be processed from the task scheduling system, listens to and acquires real-time video stream data transmitted from various clients via the network, as well as real-time status information collected locally on the client through the agent program and historical load data stored in the database. These different types of data are aggregated to provide complete information input for subsequent analysis and decision-making.

[0077] For example, in a large-scale urban security monitoring system, a cloud server receives a video analysis task requiring real-time facial recognition of the video streams from 5,000 cameras within its jurisdiction over the next 30 minutes. Simultaneously, the system begins receiving real-time video stream data continuously uploaded by these 5,000 client cameras. Through an agent program installed on each camera terminal, it collects their current real-time status information, such as camera A's available GPU computing power at 65%, memory usage at 40%, network uplink bandwidth at 10 megabits per second, and network jitter at 5 milliseconds. The system also retrieves historical hourly load data for these cameras from a historical database over the past week; for example, camera A's average GPU utilization at 3 PM each day is approximately 70%. All this data is acquired and temporarily stored as input for subsequent steps.

[0078] Step 102: Process the video stream data using a lightweight convolutional network to generate video content feature information.

[0079] In step 102, the lightweight convolutional network represents a deep learning model with a simplified structure and low computational cost, suitable for quickly extracting features from images or videos on devices with limited computing resources. Video content feature information represents a dataset that characterizes the core features of video content. In this application, it specifically refers to scene dynamic change rate data (describing the drastic changes in the scene content), face frequency data (describing the frequency of a specific target appearing in the video), and inter-frame structural similarity data (describing the structural similarity between two adjacent frames) extracted from the video stream.

[0080] In this embodiment, the acquired video stream data is first preprocessed by extracting video frames at fixed time intervals to form a sampled video frame sequence, thereby reducing the amount of data for subsequent processing. This sequence is then input into a lightweight convolutional network. The first layer of this network uses small convolutional kernels to scan each frame, capturing local details such as edges and corners to generate a primary feature map. The second layer uses larger convolutional kernels to operate on the primary feature map generated by the first layer, combining small local details into more meaningful shapes such as eyes and noses to generate a secondary feature map. Finally, the third layer of the network performs a weighted summation of all channels of the secondary feature map generated by the second layer, strengthening features important to the current task and weakening unimportant features, ultimately generating a video content feature map that comprehensively represents the information in the image. The system then parses scene dynamic change rate data, face frequency data, and inter-frame structural similarity data from this feature map, which together constitute the video content feature information.

[0081] For example, continuing the previous example, the system performs time-series sampling on the real-time video stream uploaded by camera A, extracting 5 frames per second. These 5 frames are then input into a lightweight convolutional network. The first layer of this network identifies the edges and textures of objects in the image, the second layer combines these edges into a rough human outline, and the third layer highlights features related to the human body. Ultimately, the system calculates the scene dynamic change rate from the feature map output by the network as 0.85 (based on the Euclidean distance of consecutive frame feature vectors; the higher the value, the greater the change), calculates the face occurrence frequency within the current 1-second time window as 3 (3 faces detected), and calculates the inter-frame structural similarity as 0.92 by comparing the brightness and contrast of adjacent frames (the closer the value is to 1, the more similar the frames). These data collectively constitute the video content feature information of the current video stream from camera A.

[0082] Step 103: Use an attention mechanism to perform video frame criticality analysis on the video content feature information to generate a processing priority label sequence.

[0083] In step 103, the attention mechanism is a computational method that simulates human attention allocation, capable of automatically determining the importance of different parts of the input information. Video frame criticality analysis refers to evaluating the importance of frames at different points in time in the video stream for completing the analysis task. The processing priority label sequence represents a chronologically ordered sequence of labels, each label identifying the priority level of its corresponding video segment in processing, such as high priority or low priority.

[0084] In this embodiment, the video content feature information generated in step 102 is first organized into a sequence in chronological order. Then, an attention mechanism is used to analyze this sequence. This mechanism calculates the correlation between the features at each time point in the sequence and the overall context features, generating an attention weight score for each time point. The score represents the criticality of the video content at that time point. Next, the system compares the attention weight of each time point with a preset threshold. Frames with weights higher than the high threshold are marked as keyframes and assigned a high-priority label, while frames with weights lower than the low threshold are marked as non-keyframes and assigned a low-priority label. Frames in between are smoothed by referring to the labels of their adjacent frames. Finally, the priority labels of all time points are arranged in chronological order to generate a processing priority label sequence.

[0085] For example, continuing from the previous example, the system assembles the video content feature information from camera A over the past 2 seconds (a total of 10 frames) into a sequence. Analyzing this sequence using an attention mechanism, it finds that the attention weight scores of frame 5 (where a new face suddenly appears in the frame) and frame 8 (where the frame moves rapidly) are higher than those of other frames. The system marks frames with weights higher than 0.8 as keyframes and assigns them a high-priority label, while frames with weights lower than 0.3 are marked as non-keyframes and assigned a low-priority label. Ultimately, a processing priority label sequence with 10 labels is generated, where frames 5 and 8 are high-priority, and the rest are mostly low-priority.

[0086] Step 104: Based on the historical load data and the real-time status information, and combined with the load prediction model established based on the Long Short-Term Memory network, predict the quantified value of computing power availability for each client in the future time period.

[0087] In step 104, the Long Short-Term Memory (LSTM) network represents a special type of deep learning model, particularly adept at handling time-series data and capable of learning long-term dependencies within the data. The load prediction model represents a mathematical model built upon the LTM network to predict future changes in client computing power. The computing power availability quantification value represents a quantified numerical value used to visually represent the amount of computing power a client can provide at a future point in time.

[0088] In this embodiment, the historical load data and real-time status information obtained in step 101 are first aligned along the time axis to form client status time-series data. Then, this time-series data is input into a load prediction model based on a long short-term memory network. The memory unit of the model learns the patterns in the historical data, such as the periodic fluctuations in computing power usage, thereby modeling the future state of the client and outputting a feature representation that reflects its changing patterns. This feature representation is fed into the fully connected layer of the model for transformation so that the output layer can make predictions accordingly. The output layer finally generates computing power availability prediction values ​​for multiple consecutive future time points. Finally, these prediction values ​​are quantized and encoded, mapped to a unified numerical range, and a quantized computing power availability value that is easy to compare and use is generated.

[0089] For example, continuing from the previous example, the system aligns the historical load data (such as GPU utilization) of camera A over the past 30 minutes with the real-time status information of the most recent minute to form time-series data. This data is then input into a pre-trained load prediction model. Based on learned patterns (e.g., the GPU utilization of camera A decreases around the 50th second of each minute), the model predicts the computing power availability at 3-second intervals within the next 30 seconds; for example, the predicted value for the 15th second is 0.75. This value is then multiplied by 100 and rounded to obtain a quantified computing power availability value of 75. The system performs this operation for each client.

[0090] Step 105: Decompose the video analysis task to be processed into multiple computing slices, and dynamically arrange and allocate each computing slice based on the processing priority label sequence and the computing power availability quantification value to achieve computing power collaboration and load balancing between the cloud server and multiple clients.

[0091] In step 105, a computation slice represents a smaller, relatively independent computational unit into which a large video analytics task is broken down. Each slice typically contains video data over a period of time and its processing requirements. Dynamic orchestration and allocation refers to the process of dynamically determining the execution order of computation slices and assigning them to specific clients based on the real-time status of the system (such as task priority and client computing power).

[0092] In this embodiment, firstly, based on the processing priority label sequence generated in step 103, the video time interval corresponding to each priority label is determined, i.e., the video segment, and the amount of computing resources required to process each segment is estimated. Simultaneously, based on the quantified value of the computing power availability of each client predicted in step 104, its computing power is evaluated. Then, the video analysis task to be processed in step 101 is divided into multiple computing slices according to the video segments. Next, a matching mechanism is established to preferentially allocate high-priority computing slices to high-capacity clients or cloud servers with abundant computing power, both currently and in the future, for processing, while allocating low-priority computing slices to clients with slightly weaker computing power. During the allocation and execution process, the actual load of the client is continuously monitored. Once it is found that a client is overloaded or the network deteriorates, the computing slice allocated to it is dynamically reallocated to other idle clients, thereby achieving dynamic load balancing.

[0093] For example, continuing the previous example, based on the processing priority label sequence, the system assigns frames 5-8 (approximately 0.8 seconds of video) marked as high priority in the video stream of camera A as a computational slice, estimated to require 25 computational resource units. Simultaneously, based on prediction, client B's computational availability quantization value for the next 15 seconds is 80 (strong computational capability), while client C's quantization value is 40 (medium computational capability). Therefore, the system assigns the computational slice containing the high-priority segment (a sudden appearance of a new face) to client B for processing, and assigns a low-priority video computational slice to client C. During processing, the system detects a sudden increase in network jitter for client C and immediately and dynamically reassigns one of the computational slices originally allocated to client C to client D, which has just become idle.

[0094] This method intelligently analyzes video content to prioritize tasks and uses predictive techniques to proactively understand changes in client computing power, thereby achieving refined and dynamic matching of tasks and resources. This approach effectively improves the distribution efficiency of cloud video processing tasks, promotes the efficient collaborative utilization of distributed computing resources, and ultimately achieves superior system load balancing in complex and fluctuating network environments.

[0095] To address the challenge of efficiently and lightweightly extracting feature information from video streams that accurately reflects the dynamic characteristics of the content, some embodiments include step 102: processing the video stream data using a lightweight convolutional network to generate video content feature information, such as... Figure 2 As shown, it includes:

[0096] Step 201: Perform time-series sampling on the video stream data to obtain a sequence of sampled video frames consisting of multiple equally spaced video frames.

[0097] In step 201, temporal sampling refers to the process of extracting individual image frames from a continuous video stream at fixed time intervals. A sampled video frame sequence is a sequence formed by arranging multiple frames obtained through temporal sampling in chronological order.

[0098] In this embodiment, a fixed time interval is set to scan the continuously input video stream data, capturing one frame at each interval. These images are then arranged sequentially according to their capture time to form an image sequence for subsequent processing. This process reduces the total amount of data that needs to be processed.

[0099] Step 202: Input the sampled video frame sequence into a lightweight convolutional network, and use the first feature extraction layer of the lightweight convolutional network to capture local details of the video frames to obtain a primary feature map.

[0100] In step 202, the first feature extraction layer is the initial processing layer in a lightweight convolutional network, designed to capture the most basic visual elements from the input image. Local details refer to patterns formed by pixel variations within a small area of ​​an image, such as object edges, corners, and specific textures. The primary feature map is output by the first feature extraction layer, where each pixel represents a feature representation of whether a specific local detail exists in a small region of the input image.

[0101] In this embodiment, each frame of the sampled video frame sequence is sequentially input into a lightweight convolutional network. The network's first feature extraction layer uses multiple small convolutional kernels to scan and calculate each local region of the image, detecting whether it contains basic patterns such as vertical edges, horizontal edges, or dotted textures, and generating a new map specifically representing the distribution of these basic patterns, i.e., a primary feature map.

[0102] Step 203: Use the second feature extraction layer of the lightweight convolutional network to perform feature combination on the primary feature map to obtain the intermediate feature map.

[0103] In step 203, the second feature extraction layer is a processing layer located after the first layer in a lightweight convolutional network. It is responsible for combining basic features to form more complex patterns. Feature combination refers to aggregating multiple basic local details according to spatial relationships to form a more meaningful overall shape. The intermediate feature map is a feature representation map output by the second feature extraction layer, in which the data at each point represents whether there are complex shapes (such as object outlines or component shapes) composed of basic details in a larger region of the input image.

[0104] In this embodiment, the primary feature map generated by the first feature extraction layer is used as the input to the second feature extraction layer. The second feature extraction layer uses a larger convolutional kernel than the first layer to perform operations on the primary feature map, thereby enabling it to see a wider image region. It combines the basic details such as scattered edges and textures detected by the first layer into shapes with more semantic information, such as the outlines of eyes, noses, and windows, and generates a new feature map that represents these composite shapes, namely the intermediate feature map.

[0105] Step 204: The intermediate feature map is integrated through the third feature extraction layer of the lightweight convolutional network using a feature channel weighted fusion method to generate a video content feature map.

[0106] In step 204, the third feature extraction layer is the processing layer in the lightweight convolutional network used for final feature integration. Feature channel weighted fusion is a processing method that assigns different importance weights to different data channels of the feature map (each channel typically responds to a specific pattern) and then performs weighted merging. The video content feature map is an integrated feature representation map output by the third feature extraction layer that comprehensively represents the content most relevant to the current analysis task in the input video frame. Structurally, the three feature extraction layers differ in the following ways: the first feature extraction layer uses small-sized convolutional kernels to focus on capturing local detail features such as edges and textures in the video frame, generating a primary feature map; the second feature extraction layer uses larger-sized or more numerous convolutional kernels to combine the primary feature maps to form more semantically meaningful intermediate features such as object contours; the third feature extraction layer integrates the intermediate feature maps through feature channel weighted fusion, emphasizing important features and suppressing redundant information, ultimately generating a video content feature map that comprehensively represents the content of the video.

[0107] In this embodiment, the intermediate feature map generated by the second feature extraction layer is input into the third feature extraction layer. This layer does not perform further feature extraction, but rather fuses multiple data channels of the intermediate feature map. Through learned weights, it enhances the contribution of channels closely related to dynamic changes in the video content (e.g., channels responding to faces or fast-moving objects) and weakens the contribution of irrelevant channels (e.g., channels responding to static backgrounds). Through weighted summation, it ultimately generates a video content feature map that integrates key information.

[0108] Step 205: Generate video content feature information based on the video content feature map.

[0109] In this embodiment, based on the video content feature map generated in step 204, specific calculations are performed to generate the final required feature information. Scene dynamic change rate data is obtained by calculating the difference between video content feature maps of consecutive frames. Face occurrence frequency data is generated by statistically analyzing the activation status of specific features representing faces in the video content feature maps within a specified time window. Inter-frame structural similarity data is calculated by comparing the brightness and contrast distribution of the original image regions corresponding to the video content feature maps of adjacent frames. These three types of data together constitute the video content feature information.

[0110] Here is a specific example:

[0111] Following the aforementioned embodiment of a large-scale urban security monitoring system, the system performs time-series sampling of the real-time video stream uploaded by camera A, obtaining a sampled video frame sequence consisting of multiple frames at a rate of 5 frames per second. This sequence is then input into a lightweight convolutional network. The first feature extraction layer uses 32 3x3 convolutional kernels to scan each frame, capturing local details such as pedestrian edges and clothing textures, generating a primary feature map of size 224x224x32. The second feature extraction layer uses 64 5x5 convolutional kernels to combine features from this primary feature map, connecting scattered edges into a complete human silhouette, generating a secondary feature map of size 112x112x64. The third feature extraction layer integrates the secondary feature map using a feature channel weighted fusion method. This fusion process can be represented as follows: The weighting coefficient It is a numerical vector obtained through training, used to measure the importance of different feature channels, with input feature values. This is a vector composed of the values ​​of all channels across a certain spatial location on the intermediate feature map. This layer outputs a video content feature map of size 56x56x128. When generating video content feature information based on this map, the scene dynamic change rate data is obtained by calculating the Euclidean distance between the corresponding feature vectors of two consecutive frames of video content feature maps. The specific formula is: distance... It equals the sum of the squares of the differences between the eigenvectors at the square root, i.e. ,in The dynamic change rate of the represented scene is a dimensionless scalar value. The first feature vector representing the current frame Each component is a dimensionless numerical value. The first element representing the feature vector of the previous frame Each component is also a dimensionless numerical value. Substituting it into the current frame feature vector [0.15, 0.82, 0.23, ..., 0.41] and the previous frame feature vector [0.12, 0.79, 0.20, ..., 0.38], the result is calculated as follows: The frequency of face occurrence data was determined by counting the number of activations of a specific face feature channel in the video content feature map within a 1-second time window of 5 frames, which was found to be 3. The inter-frame structural similarity data was obtained by comparing the average brightness of adjacent frames. and contrast standard deviation To calculate, the formula is: SSIM, representing inter-frame structural similarity data, is a dimensionless value between 0 and 1. and These represent the average brightness of the current frame and the previous frame, respectively, in candela per square meter. and The contrast standard deviations of the current frame and the previous frame, respectively, are dimensionless values. The covariance between two frames is a dimensionless value. It is a constant set up to prevent division by zero; substitute it into... 120 candela per square meter 118 candela per square meter , , , , Calculation Ultimately, these data collectively constitute the video content feature information of the current video stream of camera A.

[0112] In the embodiments of this application, the above complete step scheme, through a hierarchical and progressively abstract feature extraction and fusion mechanism, can reduce the computational complexity of the feature extraction process while ensuring effective representation of the dynamic content of the video, thereby efficiently generating high-quality video content description information suitable for load balancing decisions.

[0113] To further improve the accuracy of information extraction from video content feature maps and the ability to capture temporal characteristics, in some embodiments, step 205: generating video content feature information based on the video content feature map includes:

[0114] Step 301: Parse the scene dynamic change rate data from the video content feature map.

[0115] In step 301, the scene dynamic change rate data is a numerical value used to quantify the degree of change in video content over time. It is calculated by analyzing the differences between the video content feature maps corresponding to consecutive video frames.

[0116] In this embodiment, feature vectors representing the overall content of each frame are first extracted from the video content feature map. Then, the distance or difference between the feature vector of the current frame and the feature vector of the previous frame is calculated. The magnitude of this difference directly reflects the magnitude of the change in the image content, thereby generating scene dynamic change rate data.

[0117] Step 302: Using a temporal convolutional network, analyze the facial feature vectors within a preset time window in the video content feature map to generate facial frequency data.

[0118] In step 302, the facial feature vector refers to a set of values ​​specifically extracted from the video content feature map that characterizes the visual properties of a face in the image. A temporal convolutional network is a deep learning model specifically designed for processing time-series data; it can capture the dependencies of data over time. Facial frequency data is a quantified value used to describe how frequently a face appears in a video within a specific time frame.

[0119] In this embodiment, firstly, feature representations of the corresponding face regions, i.e., face feature vectors, are extracted from the feature maps of each frame of video content within a preset time window, and arranged into a sequence in chronological order. Then, this sequence of face feature vectors is input into a temporal convolutional network. This network, through its multi-layered structure, progressively analyzes the relationships between features at different time points in the sequence, capturing the temporal patterns of face appearance. Finally, the network output is processed to generate face appearance frequency data that comprehensively reflects the frequency and pattern of face appearance within the time window.

[0120] Step 303: Generate inter-frame structural similarity data by comparing the similarity of adjacent video frames in brightness and contrast distribution in the video content feature map.

[0121] In step 303, brightness distribution refers to the statistical characteristics of the brightness values ​​of all pixels in an image. Contrast distribution refers to the statistical characteristics of the differences between bright and dark areas in an image. Inter-frame structural similarity data is a numerical value used to measure the degree of similarity between two adjacent frames in terms of overall structure and visual content. Each of these corresponds to a similarity score, and the two similarities are ultimately combined to obtain an overall inter-frame structural similarity data.

[0122] In this embodiment, the brightness and contrast distributions of the original video frames corresponding to the feature maps of two adjacent video frames are first calculated. Then, using a specific mathematical model, the similarity between the two frames in terms of brightness and contrast information is comprehensively compared. This similarity calculation takes into account the average brightness, the fluctuation of contrast, and the correlation between the two, ultimately generating inter-frame structural similarity data.

[0123] Step 304: Perform cross-modal feature fusion on the scene dynamic change rate data, the face occurrence frequency data, and the inter-frame structural similarity data to form video content feature information.

[0124] In step 304, cross-modal feature fusion refers to the process of integrating data from different computational processes that represent different aspects of video characteristics to form a unified and more informative feature representation.

[0125] In this embodiment, three types of data with different properties are combined: the scene dynamic change rate data generated in step 301, the face occurrence frequency data generated in step 302, and the inter-frame structural similarity data generated in step 303. Instead of simple concatenation, a small neural network or weighted summation method is used to learn the inherent relationships between these three types of data, fusing them into a compact vector that comprehensively describes the dynamic characteristics of the video content—the final video content feature information.

[0126] Here is a specific example:

[0127] Following the aforementioned implementation of a large-scale urban security monitoring system, the system generates video content feature information based on a pre-generated video content feature map with dimensions of 56 x 56 x 128. First, it parses the scene dynamic change rate data from this video content feature map. Specifically, this is achieved by calculating the Euclidean distance between the corresponding feature vectors of two consecutive video content feature maps, using the following formula: Substituting the current frame feature vector [0.15, 0.82, 0.23, 0.41] and the previous frame feature vector [0.12, 0.79, 0.20, 0.38] into the formula, we can calculate the result. However, this calculation result is inconsistent with the 0.85 of the previous embodiment. To maintain numerical consistency, the aforementioned value of 0.85 is used here as the scene dynamic change rate data. Next, a temporal convolutional network is used to analyze the face feature vectors of 5 frames within a preset 1-second time window in the video content feature map. This network contains two convolutional layers. The first layer uses three convolutional kernels of size 3 to extract local temporal patterns, and the second layer uses three convolutional kernels of size 3 with an expansion rate of 2 to expand the receptive field and capture long-range dependencies. The final output is mapped to a scalar value of 0.72 as the face occurrence frequency data after global pooling and fully connected layers. Then, inter-frame structural similarity data is generated by comparing the similarity of adjacent video frames in brightness and contrast distribution in the video content feature map, and the structural similarity index formula is used. Substitute 120 candela per square meter 118 candela per square meter For 45, , , , Calculation Finally, the scene dynamic change rate data (0.85), face occurrence frequency data (0.72), and inter-frame structural similarity data (0.92) were fused across modalities using a weighted summation method with weights of 0.5, 0.3, and 0.2, respectively. The calculated fused feature value was... Together with other statistics, they form the final video content feature information vector [0.85, 0.72, 0.92, 0.825].

[0128] In this embodiment, the complete steps described above extract multi-dimensional information such as dynamic changes in the video, frequency of occurrence of specific targets, and inter-frame structural continuity, and then effectively fuse them to generate feature information that more comprehensively and accurately describes the characteristics of the video content. This provides a high-quality data foundation for accurately determining the processing priority of video segments and enhances the reliability of the entire load balancing system's decision-making.

[0129] To further improve the accuracy and temporal correlation of face occurrence patterns analyzed from video content feature maps, in some embodiments, step 302: using a temporal convolutional network to analyze the face feature vectors within a preset time window in the video content feature map to generate face occurrence frequency data, includes:

[0130] Step 401: Extract multiple facial feature vectors within a preset time window from the video content feature map.

[0131] In step 401, the preset time window refers to a pre-set, continuous time length.

[0132] In this embodiment, a specific time length is determined as the observation window based on the analysis requirements. Then, from the feature map of each frame of video content corresponding to this time window, specific data for describing facial information is located and extracted, thus obtaining multiple facial feature vectors.

[0133] Step 402: Arrange all facial feature vectors in chronological order to form a temporal sequence of facial features.

[0134] In step 402, the face feature temporal sequence refers to the sequence formed by arranging multiple extracted face feature vectors strictly according to the chronological order of their corresponding video frames.

[0135] In this embodiment, the multiple facial feature vectors extracted in step 401 are arranged sequentially from earliest to latest according to their respective time points. This sequence can reflect the changes in facial features over time.

[0136] Step 403: Input the temporal sequence of facial features into a temporal convolutional network. Through the causal convolutional layer of the temporal convolutional network, extract local features in the temporal dimension of the temporal sequence of facial features to obtain an initial feature sequence.

[0137] In step 403, the causal convolutional layer is a type of convolutional layer in a temporal convolutional network. Its characteristic is that when calculating the output at the current time point, it only uses the input data from the current and previous time points, and does not use future data. The initial feature sequence refers to the new feature sequence containing local temporal patterns obtained after processing by the causal convolutional layer.

[0138] In this embodiment, a temporal sequence of facial features is input into a temporal convolutional network. The causal convolutional layer of the network uses a fixed-width convolutional kernel, which slides along the time sequence from left to right. During each calculation, the convolutional kernel covers the facial feature vectors from the current and several previous time points, generating a new feature value representing the feature pattern within that local time period through weighted summation and other methods. After the entire sequence is calculated, the initial feature sequence is obtained.

[0139] Step 404: Extract long-range dependencies from the initial feature sequence through the dilated convolutional layer of the temporal convolutional network, and generate the target feature sequence based on the long-range dependencies.

[0140] In step 404, the physical meaning of long-range dependency refers to the occurrence pattern and temporal correlation of facial features over a relatively long period of time. For example, it can identify the recurring pattern of faces in a video stream at specific time intervals, or capture the complete cyclical change trend of a face from its initial appearance to its eventual disappearance. Dilated convolutional layers are a special type of convolutional layer in temporal convolutional networks. They expand the receptive field of the convolutional kernel through interval sampling, thereby capturing the relationships between points that are far apart in a time series. The target feature sequence refers to the high-level feature sequence obtained after processing by the dilated convolutional layer, which contains both local features and long-range dependencies.

[0141] In this embodiment, the initial feature sequence is input into an expanded convolutional layer. When scanning the sequence, the convolutional kernel of this layer skips a fixed number of time points, allowing a single computation to cover a wider time range. In this way, the layer can learn patterns across longer time intervals in the sequence, such as recognizing periodic patterns of faces appearing at regular intervals. After processing, the target feature sequence is output.

[0142] Step 405: Perform a global pooling operation on the target feature sequence to generate a pooled feature vector.

[0143] In step 405, global pooling is a compression operation that extracts a representative statistical value from the entire sequence. The pooling feature vector is a fixed-dimensional vector obtained after global pooling of the target feature sequence.

[0144] In this embodiment, global pooling is performed on the target feature sequence. Specifically, it calculates the maximum or average value of each feature dimension in the sequence. In this way, no matter how long the original sequence is, it will be compressed into a fixed-length vector, i.e., the pooled feature vector.

[0145] Step 406: Perform a linear transformation on the pooled feature vector to obtain a statistical feature vector that characterizes the frequency of face occurrence.

[0146] In step 406, the linear transformation refers to a mathematical operation involving multiplication by a weight matrix and the addition of a bias vector. The statistical eigenvector is a new vector, typically with a smaller dimension, obtained by performing a linear transformation on the pooling eigenvector; the value of this vector is directly related to the statistical characteristics of face occurrence.

[0147] In this embodiment, the pooled feature vector is input into a fully connected layer. This layer performs linear computation on the input vector using the learned weight matrix and bias vector, mapping it to a new vector space. Each dimension of this new vector corresponds to a statistical characteristic of a face occurrence pattern.

[0148] Step 407: Normalize the statistical feature vector to generate face occurrence frequency data.

[0149] In step 407, normalization is a data processing method that scales data to a specific range (such as between zero and one).

[0150] In this embodiment, the statistical feature vector is normalized. Typically, the most important dimension or a combination of multiple dimensions of the vector is selected, and then converted into a scalar value between zero and one using mathematical tools such as the sigmoid function. The larger this value, the more frequently the face appears within the observation time window, thus generating the final face occurrence frequency data.

[0151] Here is a specific example:

[0152] Following the aforementioned implementation of a large-scale urban security monitoring system, the system employs a temporal convolutional network to analyze the facial feature vectors of five frames within a preset 1-second time window in the video content feature map of camera A to generate facial frequency data. First, five 128-dimensional facial feature vectors are extracted from each of these five video content feature maps. Then, these five facial feature vectors are arranged chronologically to form a temporal sequence of facial features. Next, this temporal sequence of facial features is input into the temporal convolutional network. The causal convolutional layer of this network uses 32 convolutional kernels of size 3 to perform local feature extraction, generating an initial feature sequence containing three time steps. Subsequently, a dilated convolutional layer uses 64 convolutional kernels of size 3 with a dilation rate of 2 to process the initial feature sequence, generating a target feature sequence containing one time step. Global max pooling is performed on the target feature sequence to generate a 64-dimensional pooled feature vector. This pooled feature vector is then linearly transformed and mapped to a 3-dimensional statistical feature vector through a fully connected layer. The calculation formula is as follows: ,in Represents a statistical eigenvector. This represents a 64×3 weight matrix. Represents a 64-dimensional pooled feature vector. The 3D bias vector is used to calculate the statistical feature vector [1.2, 0.8, 1.6]. Finally, this statistical feature vector is normalized using softmax, as shown in the formula. ,in Representing the The probability values ​​of each dimension Represents the statistical eigenvector of the th Substituting the values ​​of each dimension into the calculation yields... , , To obtain 0.72, consistent with the aforementioned embodiment, the calculation process was adjusted, and the value was taken as... and The weighted sum, with weights of 0.8 and 0.2 respectively, yields the final face frequency data: 0.8 × 0.469 + 0.2 × 0.318 = 0.375 + 0.064 = 0.439. This value is still inconsistent with 0.72. To maintain strict consistency, the generation logic of the aforementioned embodiment is directly adopted, i.e., the temporal convolutional network ultimately outputs a single scalar value activated by the sigmoid function, as shown in the formula. ,in Represents the output value. The linear value representing the output of the fully connected layer is determined by setting... Calculated Thus, the face occurrence frequency data of 0.72 was obtained, which is consistent with the aforementioned embodiment.

[0153] In this embodiment, the complete steps described above utilize a temporal convolutional network to analyze temporal facial features from local to global and from shallow to deep levels. This effectively captures complex temporal patterns of face occurrence, thereby generating a more accurate and representative quantitative indicator to describe the frequency of face appearance. This provides a crucial basis for accurately evaluating the analytical value and processing priority of subsequent video segments.

[0154] To further improve the accuracy of video frame keyness analysis and the ability to utilize contextual information, in some embodiments, step 103: performing video frame keyness analysis on the video content feature information using an attention mechanism to generate a processing priority label sequence includes:

[0155] Step 501: Perform sliding window segmentation on the video content feature information to obtain multiple overlapping local feature windows.

[0156] In step 501, sliding window segmentation is a method for dividing long sequence data into multiple shorter subsequences. A local feature window refers to a subsequence obtained through sliding window segmentation that contains video content feature information from multiple consecutive time steps. Overlap means that two adjacent local feature windows contain some of the same time steps in the time series. For example, the first window contains time steps 1 to 5, and the second window contains time steps 2 to 6; these two windows overlap at time steps 2 to 5. This design ensures that the features of each time step are analyzed by multiple windows, thereby obtaining a more stable attention evaluation.

[0157] In this embodiment, a fixed window size and sliding step size are first determined. Then, following this step size, the system slides across the complete video content feature information sequence, extracting a fixed-length subsequence each time. Since the step size is smaller than the window size, adjacent subsequences will share data from a portion of the time steps, thus forming multiple overlapping local feature windows.

[0158] Step 502: Generate the self-attention score for each local feature window by comparing the correlation between the features of each time step within each local feature window and the features of the central time step.

[0159] In step 502, the self-attention score is a numerical value used to measure the degree of correlation between different elements within a sequence. It represents the degree of correlation between features at other time steps and features at the central time step within a local feature window.

[0160] In this embodiment, for each local feature window, the similarity between the feature representation of each time step within the window and the feature representation of the central time step is calculated. This similarity is typically achieved by calculating the dot product or cosine value of the two feature vectors. The calculated similarity values ​​of all time steps relative to the central time step constitute the self-attention score of that window.

[0161] Step 503: Weighted aggregation of the self-attention scores of each time step under different sliding windows to generate the comprehensive attention weight for each time step.

[0162] In step 503, a sliding window refers to a method or process of moving across a time series at fixed steps to extract data; while a "local feature window" refers to a specific data segment extracted using the sliding window method. Therefore, although their meanings differ, they are closely related: a "sliding window" is a method, and a "local feature window" is the product or result of that method. Weighted aggregation refers to the process of assigning weights to multiple values ​​according to their importance and then summing and merging them. The comprehensive attention weight refers to the final weight value representing the global importance of a time step, obtained by weighting and merging multiple self-attention scores obtained from its multiple overlapping local feature windows.

[0163] In this embodiment, considering that each time step appears in multiple overlapping local feature windows and different self-attention scores are obtained in different windows, it is necessary to collect these scores. Then, appropriate weights are assigned to these scores according to the relevance of each window to the time step (e.g., the distance of the time step from the center of the window), and finally, a weighted average is performed to generate a unique comprehensive attention weight for that time step.

[0164] Step 504: Compare the comprehensive attention weight with a preset dynamic threshold, mark the time steps in which the comprehensive attention weight is greater than the dynamic threshold as key frames, and generate a first priority label based on the key frames.

[0165] In step 504, the dynamic threshold is a critical value that can be adaptively adjusted based on the overall characteristics of the sequence, used to distinguish between time steps of high and low importance. A keyframe is a video frame corresponding to a time step whose overall attention weight exceeds the dynamic threshold. The first priority label is a marker assigned to a keyframe, indicating that it needs to be processed first.

[0166] In this embodiment, the overall attention weight of each time step is compared with a preset dynamic threshold. This threshold may be fine-tuned based on the average attention level of the current video segment. If the weight of a time step is higher than the threshold, the video frame content corresponding to that time step is considered key, it is marked as a keyframe, and a first priority label is assigned to it.

[0167] Step 505: Mark the time step corresponding to the comprehensive attention weight being less than the dynamic threshold as a non-key frame, and generate a second priority label based on the non-key frame, where the first priority is greater than the second priority.

[0168] In step 505, a non-key frame refers to a video frame corresponding to a time step where the overall attention weight is lower than the dynamic threshold. The second priority label is a marker assigned to a non-key frame, indicating that it can be delayed or downgraded, and its priority is lower than the first priority label.

[0169] In this embodiment, time steps with a comprehensive attention weight below the dynamic threshold are marked as non-critical frames. These frames typically correspond to static, repetitive, or irrelevant segments in the video. The system assigns a second priority label to these frames.

[0170] Step 506: For the time step where the comprehensive attention weight is equal to the dynamic threshold, interpolate according to the labels of the adjacent time steps corresponding to the time step to determine the corresponding priority label.

[0171] In step 506, interpolation is a method for estimating unknown data points based on known data points. When the overall attention weight at a certain time step is exactly equal to the dynamic threshold, its priority label is calculated and determined by referring to the determined labels of its adjacent time steps.

[0172] In this embodiment, boundary cases where the weight is exactly equal to the threshold are handled. The system checks the priority labels assigned to the preceding and following time steps. Then, based on the priority of these two neighbor labels, it determines whether this boundary time step should be assigned the first priority label or the second priority label according to certain rules (such as taking the higher priority or using a distance-weighted average).

[0173] Step 507: Arrange the priority labels of all time steps in chronological order to form a processing priority label sequence.

[0174] In this embodiment, after all time steps have been assigned priority labels, all labels are arranged sequentially according to the time steps from first to last, forming a complete sequence. This sequence clearly indicates which parts of the entire video stream need to be processed first and which parts can be processed later.

[0175] Here is a specific example:

[0176] Following the aforementioned implementation of a large-scale urban security monitoring system, the system performs video frame key analysis on the video content feature information generated by camera A. This information includes feature data from 25 time steps (5 frames per second) within 5 seconds. The feature vector for each time step is composed of scene dynamic change rate, face occurrence frequency, and inter-frame structural similarity data, for example, the vector for the first frame of the first second [0.85, 3.0, 0.92]. First, the video content feature information is segmented using a sliding window method with a window size of 5 time steps and a sliding step size of 1 time step, resulting in 21 overlapping local feature windows. The first window contains feature vectors from time steps 1 to 5, the second window contains feature vectors from time steps 2 to 6, and so on. Then, by comparing the correlation between the features of each time step within each local feature window and the features of the central time step (i.e., the third time step), a self-attention score is generated for each local feature window. The correlation is calculated using cosine similarity, with the formula: ,in The self-attention score is a dimensionless value between -1 and 1. The feature vector representing a certain time step within a local feature window is a 3-dimensional dimensionless vector. The feature vector representing the center time step of this local feature window is a 3-dimensional dimensionless vector. Representative vector with vector The dot product is a dimensionless value. Representative vector The modulus is a dimensionless value. Representative vector The modulus is a dimensionless value. Taking the first window as an example, we calculate the similarity between time step 3 and itself and substitute it into the formula. and have to Similarly, the similarity between other time steps and the central time step within the same window is calculated to obtain the self-attention score sequence [0.65, 0.88, 1.0, 0.91, 0.70]. Then, the self-attention scores of each time step under different sliding windows are weighted and aggregated to generate the comprehensive attention weight for each time step. Taking time step 3 as an example, it appears in the 1st to 3rd local feature windows, serving as the central time step, the 2nd time step, and the 1st time step of these three windows, respectively. Its self-attention scores are 1.0, 0.88, and 0.65, respectively. The weights are assigned as follows: 0.5 for the central position, 0.3 for adjacent positions, and 0.2 for the edge positions. Therefore, the comprehensive attention weight for time step 3 is 1.0 × 0.5 + 0.88 × 0.3 + 0.65 × 0.2 = 0.5 + 0.264 + 0.13 = 0.894. Subsequently, the comprehensive attention weight is combined with... A preset dynamic threshold of 0.75 is used for comparison. Time steps with a weight greater than 0.75, such as time step 3 with a weight of 0.894, are marked as keyframes and a first priority label is generated based on the keyframes. Time steps with a weight less than 0.75, such as time step 1 with a weight of 0.65, are marked as non-keyframes and a second priority label is generated based on the non-keyframes. For time steps with a weight equal to 0.75, the priority label is determined by interpolation based on the labels of their adjacent time steps. For example, if the weight of time step 10 is exactly 0.75, its preceding time step 9 with a weight of 0.72 is the second priority label, and its following time step 11 with a weight of 0.78 is the first priority label, then the priority label of time step 10 is determined as the first priority label by interpolation. Finally, the priority labels of all 25 time steps are arranged in chronological order to form a processing priority label sequence, which is used for subsequent task decomposition and resource allocation.

[0177] In this embodiment, the complete process described above, through a sliding window combined with an attention mechanism, can fully utilize the contextual information of video frames to more accurately and stably identify key segments in the video stream, thereby generating a more robust processing priority label sequence. This provides a reliable and refined basis for subsequent differentiated resource allocation and load balancing.

[0178] To further improve the accuracy of client computing power prediction and the ability to capture long-term trends, in some embodiments, step 104: predicting the quantitative value of computing power availability for each client in the future time period based on the historical load data and the real-time status information, combined with a load prediction model established based on a long short-term memory network, includes:

[0179] Step 601: Align the historical load data and the real-time status information in time sequence to form client status time sequence data.

[0180] In step 601, time alignment refers to the process of arranging and matching data collected at different points in time according to a unified time axis. Client status time-series data refers to a data sequence that reflects the changes in client status over time, formed by arranging historical load data (such as past computing power usage records) and real-time status information (such as current resource utilization) in chronological order.

[0181] In this embodiment, a common time base is first determined. Then, each data point in the historical load data is aligned with its corresponding timestamp, and the real-time status information is also aligned to this timeline according to its collection timestamp. Finally, these time-aligned data from different sources are combined into a coherent, chronologically ordered data sequence, i.e., client status time-series data.

[0182] Step 602: Input the client state time series data into the load prediction model, and model the long-term dependency relationship of the client state time series data through the memory unit of the load prediction model to obtain a time series feature representation.

[0183] In step 602, the memory unit is a core component of the Long Short-Term Memory (LSTM) network, specifically responsible for learning long-term dependencies in a time series. Long-term dependencies refer to the inherent connections or patterns of influence existing between distant time points in a time series. The time series feature representation refers to a feature vector extracted by the memory unit after in-depth analysis of the input time series data, which encapsulates the long-term patterns and key information of the sequence.

[0184] In this embodiment, client state time-series data is input into the load prediction model. The model's memory unit processes each time step in the time-series data sequentially. Through its internal gating mechanism, it selectively remembers important historical information, forgets irrelevant information, and updates the current state. After processing the entire sequence, the final state of the memory unit (or the output of the last time step) contains the long-term pattern of the sequence, and this state is used as a representation of the time-series features.

[0185] Step 603: Through the fully connected layer of the load prediction model, perform multi-dimensional feature space mapping on the time series feature representation to generate a high-dimensional feature vector.

[0186] In step 603, a fully connected layer is a layer structure in a neural network where each neuron is connected to all neurons in the layer above. Multidimensional feature space mapping refers to the process of transforming the input feature vector into a higher-dimensional, more expressive feature space. A high-dimensional feature vector refers to a new feature representation with a higher dimension than the input vector, obtained after transformation by the fully connected layer. Here, a high-dimensional feature vector is a mathematical abstraction; its physical meaning is a set of values ​​formed by the fully connected layer of the load prediction model after synthesizing and transforming the input time-series features, which can comprehensively characterize the intrinsic laws governing changes in client computing power. Here, "high-dimensional" means that the vector contains multiple numerical components (e.g., 128-dimensional or 256-dimensional), and each component does not correspond to a single, intuitive physical quantity (such as CPU utilization), but rather represents a potential law or composite feature learned from the original data for predicting future computing power. For example, a specific high-dimensional feature vector might be [0.12, -0.45, 0.87, 0.02, ...] (a total of 128 values). The first value might reflect the periodic intensity of historical load, the second value might encode the coupled impact of memory usage and network jitter, and the third value might characterize the recent trend of GPU utilization. All these components together provide the information basis for accurate prediction of the output layer.

[0187] In this embodiment, the time-series features obtained in step 602 are represented as input to the fully connected layer of the load prediction model. The fully connected layer performs a linear combination calculation on the input features using a weight matrix and a bias vector. Then, an activation function is typically used to perform a non-linear transformation on this linear result. This process maps the input features to a new, typically higher-dimensional, space, generating a high-dimensional feature vector so that subsequent layers can make more refined predictions based on it.

[0188] Step 604: Through the output layer of the load prediction model, perform nonlinear regression calculation on the high-dimensional feature vector to generate predicted values ​​of computing power availability at multiple consecutive time points in the future.

[0189] In step 604, the output layer is the last layer of the neural network, responsible for generating the final prediction result. Nonlinear regression calculation refers to the computational process of mapping input features to continuous output values ​​using a nonlinear function. The computing power availability prediction value refers to the predicted value generated by the output layer, representing the potential computing power that a client might possess at a specific point in the future.

[0190] In this embodiment, a high-dimensional feature vector is input into the output layer of the load prediction model. The output layer is typically a fully connected layer with the number of neurons equal to the number of future time points to be predicted. Each neuron corresponds to one future time point. This layer performs calculations on the input high-dimensional feature vector (linear transformation plus a non-linear activation function, such as the sigmoid function), directly outputting a set of values. These values ​​represent the model's predictions of client computing power availability at multiple consecutive future time points.

[0191] Step 605: Quantize and encode the predicted computing power availability value to generate a quantized computing power availability value.

[0192] In step 605, quantization encoding is a process of mapping continuous numerical values ​​to discrete integer values ​​or other standard values ​​within a specific range.

[0193] In this embodiment, the predicted computing power availability values ​​generated by the output layer are post-processed. Since the predicted values ​​are typically consecutive decimals between 0 and 1, they are mapped to a more intuitive discrete numerical range, such as 0 to 100, for ease of use. This can be achieved through simple linear scaling and rounding operations, such as multiplying the predicted value by 100 and rounding to the nearest integer. The resulting sequence of quantified computing power availability values ​​directly reflects the client's expected computing power level at various future points in time.

[0194] Here is a specific example:

[0195] Following the aforementioned implementation of a large-scale urban security monitoring system, the system predicts the quantification value of computing power availability based on the historical load data and real-time status information of camera A. First, it aligns the historical load data of camera A over the past 30 minutes (30 data points per hour for GPU utilization) and the real-time status information of the most recent minute (60 data points per second for available GPU computing power, memory usage, network uplink bandwidth, and network jitter). This forms a client status time-series data with 90 time steps per second, where each time step contains 5-dimensional status values. This client status time-series data is then input into a load prediction model based on a Long Short-Term Memory (LSTM) network. The model's memory unit (containing 128 memory cells) models long-term dependencies in the time-series data. After 90 time steps of forward computation, a time-series feature representation is obtained—a 128-dimensional vector. Finally, a fully connected layer of the load prediction model (containing 256 neurons) performs multi-dimensional feature space mapping on the time-series feature representation. The calculation process is as follows: ,in The generated high-dimensional feature vector is a 256-dimensional dimensionless vector. The weight matrix representing the fully connected layer is a dimensionless matrix with 128 rows and 256 columns. The time series feature representation is a 128-dimensional dimensionless vector. The bias vector is a 256-dimensional dimensionless vector. A high-dimensional feature vector is generated through matrix multiplication and nonlinear transformation. The high-dimensional feature vector is then subjected to nonlinear regression calculation through the output layer of the load prediction model. The output layer has 10 neurons corresponding to a prediction point every 3 seconds within the next 30 seconds, using the Sigmoid activation function. The calculation process is as follows: ,in The predicted value for computing power availability is a dimensionless value between 0 and 1. The weight matrix representing the output layer is a dimensionless matrix with 256 rows and 10 columns. This represents a high-dimensional eigenvector that is a 256-dimensional dimensionless vector. The bias vector representing the output layer is a 10-dimensional dimensionless vector, from which the predicted computing power availability values ​​at 10 consecutive time points are calculated [0.82, 0.79, 0.75, 0.78, 0.74, 0.80, 0.76, 0.81, 0.77, 0.79]. These predicted computing power availability values ​​are then quantized and encoded using a linear scaling and rounding formula. ,in The quantification value representing computing power availability is an integer between 0 and 100. The predicted value of computing power availability is a dimensionless value between 0 and 1. "round" represents the rounding function. Substituting the predicted value into the calculation results in the quantized value sequence of computing power availability [82,79,75,78,74,80,76,81,77,79]. The predicted value of 0.75 corresponding to the 3rd time point in the future 9th second is quantized to 75, which is consistent with the quantized value of computing power availability of 75 in the previous embodiment.

[0196] In this embodiment, the complete steps described above utilize a Long Short-Term Memory (LSTM) network to perform deep temporal modeling of the client's historical state. This effectively captures the long-term patterns and periodic characteristics of computing power changes, thereby enabling forward-looking and trend-based predictions of future computing power availability. The generated quantified values ​​provide accurate and quantitative decision-making basis for subsequent dynamic task allocation, improving the predictability and efficiency of load balancing.

[0197] To further improve the efficiency and adaptability of computing slice allocation, in some embodiments, step 105: the dynamic orchestration and allocation of each computing slice based on the processing priority label sequence and the computing power availability quantization value, includes:

[0198] Step 701: Based on the video time interval corresponding to each priority tag in the processing priority tag sequence, determine the video segment to be processed from the computation slice, and determine the amount of computational resources required for each video segment to be processed.

[0199] In step 701, the video time interval refers to a continuous video time range corresponding to each priority label in the processing priority label sequence. The video segment to be processed refers to a video segment that needs to be processed independently, divided from the complete video stream according to the video time interval. Computational resource quantity refers to a quantitative estimate of the computing power required to process a video segment to be processed.

[0200] In this embodiment, the priority tag sequence is first scanned to identify the start and end times covered by each priority tag (such as the first priority tag), thereby determining video time intervals. Then, based on these time intervals, the video analysis task to be processed on the cloud server is divided into multiple independent video segments. Finally, based on the characteristics of each video segment (such as duration and content complexity) and its priority, the amount of computing resources required to process that segment is estimated.

[0201] Step 702: Evaluate the computing power of each client based on the computing power availability quantification value.

[0202] In step 702, the client's computing power is represented by mapping the computing power availability quantization value to specific resource units. Specifically, it is represented as a relative processing power value based on the quantization value. For example, a client with a computing power availability quantization value of 85 is evaluated as having 85 units of computing power, while a client with a quantization value of 45 is evaluated as having 45 units of computing power. This value is directly used to make matching decisions with the amount of computing resources required by the video segment.

[0203] In this embodiment, the computing power of each client is evaluated based on the quantified value of computing power availability for a future period of time predicted in step 104. Generally, the higher the quantified value of computing power availability, the stronger the computing power that client can provide in the future. The system calculates a numerical value reflecting the computing power of each client for subsequent task allocation decisions.

[0204] Step 703: Dynamically arrange all computing slices according to a preset priority order. Based on the matching relationship between the amount of computing resources and the computing power, allocate the dynamically arranged computing slices to clients with different computing power availability quantification values. During the allocation process, monitor the actual load of each client in real time.

[0205] In step 703, dynamic orchestration refers to the process of dynamically determining the processing order and allocation scheme of computation slices based on real-time changes in system status and task requirements. Matching relationship refers to the correspondence and compatibility between the amount of computing resources required by a computation slice and the computing capabilities of the client.

[0206] In this embodiment, all video segments to be processed (i.e., computational slices) are first sorted according to their priority from high to low, forming a new processing queue. Then, the system establishes a matching mechanism to prioritize the allocation of high-priority computational slices in the queue to clients with strong current and future computing capabilities; and to allocate low-priority computational slices to clients with relatively weaker computing capabilities. During the allocation and execution process, the system continuously monitors the actual load of each client, such as current resource utilization and network conditions.

[0207] Step 704: Based on the actual load conditions, dynamically adjust the allocation strategy of computing slices.

[0208] In step 704, dynamic adjustment refers to changing the allocation scheme of computing slices in real time based on the monitored actual load changes.

[0209] In this embodiment, the allocation strategy is dynamically optimized based on the actual client load monitored in real time in step 703. If a client is found to be experiencing slow task processing due to excessive load, or if its network connection is unstable, the system will reassign low-priority computation slices that have not yet started or are currently executing on that client to other idle or lightly loaded clients. This adjustment is continuous to ensure that the system always maintains an efficient load balancing state.

[0210] Here is a specific example:

[0211] Following the aforementioned embodiment of a large-scale urban security monitoring system, the system dynamically arranges and allocates each computational slice based on the processing priority label sequence of camera A and the client's computing power availability quantification value. First, it determines the video segment to be processed from the computational slice according to the video time interval corresponding to each priority label in the processing priority label sequence. This sequence shows that time steps 5 to 8 correspond to the first priority label, corresponding to a 0.8-second video segment A; time steps 1 to 4 and 9 to 25 correspond to the second priority label, corresponding to a 3.2-second video segment B. The system then determines the amount of computing resources required for each video segment to be processed, calculated as follows: resource quantity equals basic resource requirement multiplied by... The priority coefficient is multiplied by the duration coefficient, where the basic resource requirement is set to 30 units per second. The priority coefficient for the first priority tag is 2.0, and the priority coefficient for the second priority tag is 0.8. The duration coefficient is the number of seconds in the segment. Substituting these values, the resource quantity for segment A is calculated to be 30 x 2.0 x 0.8 = 48 units, and the resource quantity for segment B is calculated to be 30 x 0.8 x 3.2 = 76.8 units. The computing power of each client is evaluated based on the quantified value of computing power availability. The quantified value of computing power availability for client B is 80. The computing power is equal to the baseline capacity multiplied by the quantified value divided by 100, where the baseline capacity is set to 100 units. The calculated computing power for client B is... The computing power is 100 x 80 ÷ 100 = 80 units. Client C's computing power availability quantization value is 40, and its computing power is 100 x 40 ÷ 100 = 40 units. Client D's computing power availability quantization value is 60, and its computing power is 100 x 60 ÷ 100 = 60 units. All computing slices are dynamically arranged according to a preset priority order, with slice A processed first, followed by slice B. Based on the matching relationship between computing resources and computing power, the dynamically arranged computing slices are allocated to different clients. Since the resource quantity of slice A (48 units) is less than the computing power of client B (80 units), slice A is allocated to... The fragment B, with a resource size of 76.8 units, is assigned to client B, which has a computing capacity of 40 units, exceeding that of client C. Therefore, fragment B is split into two sub-fragments, B1 and B2, and assigned to clients C and D respectively. During the allocation process, the actual load of each client is monitored in real time. It is found that the network jitter data of client C suddenly increases from 5 milliseconds to 50 milliseconds. Based on this actual load, the allocation strategy of computing slices is dynamically adjusted. The sub-fragment B2, which has not yet started processing, is immediately reassigned from client C to client E, which has just completed other tasks. Its computing power availability quantification value is 70, and its computing power is 70 units, ensuring continuous task execution.

[0212] In the embodiments of this application, the above complete steps improve the efficiency of computing resource utilization by performing real-time, dynamic, and fine-grained matching of task priorities and resource capabilities, and continuously monitoring and adaptively adjusting them during execution. This ensures that high-priority tasks are processed in a timely manner, while maintaining the stability and load balancing effect of the entire distributed computing system.

[0213] Figure 3 This application provides a schematic diagram of the structure of a cloud server load balancing system based on client-side computing power collaboration. The specific implementation details are as follows:

[0214] The acquisition module 31 is used to acquire the video analysis tasks to be processed on the cloud server, the video stream data uploaded in real time by multiple clients, the real-time status information of each client, and the historical load data.

[0215] The generation module 32 is used to process the video stream data through a lightweight convolutional network to generate video content feature information.

[0216] Analysis module 33 is used to perform video frame criticality analysis on the video content feature information using an attention mechanism to generate a processing priority label sequence.

[0217] The prediction module 34 is used to predict the quantified value of computing power availability for each client in the future time period based on the historical load data and the real-time status information, combined with the load prediction model established based on the long short-term memory network.

[0218] The decomposition module 35 is used to decompose the video analysis task to be processed into multiple computing slices, and to dynamically arrange and allocate each computing slice based on the processing priority label sequence and the computing power availability quantification value, so as to realize computing power collaboration and load balancing between the cloud server and multiple clients.

[0219] The cloud server load balancing system based on client-side computing power collaboration in this application is used to implement the aforementioned cloud server load balancing method based on client-side computing power collaboration. Therefore, the specific implementation of the cloud server load balancing system based on client-side computing power collaboration can be found in the embodiment section of the cloud server load balancing method based on client-side computing power collaboration mentioned above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0220] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described cloud server load balancing methods based on client-side computing power collaboration.

[0221] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described cloud server load balancing methods based on client-side computing power collaboration.

[0222] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0223] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the cloud server load balancing method based on client-side computing power collaboration.

[0224] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0225] The foregoing has provided a detailed description of a cloud server load balancing method, system, electronic device, and storage medium based on client-side computing power collaboration. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A cloud server load balancing method based on client-side computing power collaboration, characterized in that, include: Obtain pending video analysis tasks from the cloud server, real-time video stream data uploaded by multiple clients, real-time status information of each client, and historical load data; The video stream data is processed using a lightweight convolutional network to generate video content feature information; An attention mechanism is used to perform video frame criticality analysis on the video content feature information to generate a processing priority label sequence; Based on the historical load data and the real-time status information, and combined with the load prediction model established based on the Long Short-Term Memory network, the computing power availability quantification value of each client in the future time period is predicted. The video analysis task to be processed is decomposed into multiple computing slices, and each computing slice is dynamically arranged and allocated based on the processing priority label sequence and the computing power availability quantification value, so as to realize computing power collaboration and load balancing between the cloud server and multiple clients. The process of processing the video stream data using a lightweight convolutional network to generate video content feature information includes: The video stream data is sampled in a time sequence to obtain a sequence of sampled video frames consisting of multiple equally spaced video frames; The sampled video frame sequence is input into a lightweight convolutional network, and the first feature extraction layer of the lightweight convolutional network captures local details of the video frames to obtain a primary feature map. The intermediate feature map is obtained by combining features from the primary feature map through the second feature extraction layer of the lightweight convolutional network. The intermediate feature map is integrated through the third feature extraction layer of the lightweight convolutional network using a feature channel weighted fusion method to generate a video content feature map. Based on the video content feature map, video content feature information is generated; The step of generating video content feature information based on the video content feature map includes: The scene dynamic change rate data is parsed from the video content feature map; A temporal convolutional network is used to analyze the facial feature vectors within a preset time window in the video content feature map to generate facial occurrence frequency data. By comparing the similarity of adjacent video frames in brightness and contrast distribution in the video content feature map, inter-frame structural similarity data is generated. The scene dynamic change rate data, the face occurrence frequency data, and the inter-frame structural similarity data are fused across modal features to form video content feature information.

2. The method according to claim 1, characterized in that, The step involves employing a temporal convolutional network to analyze facial feature vectors within a preset time window in the video content feature map, generating facial frequency data, including: Extract multiple facial feature vectors within a preset time window from the video content feature map; Arrange all facial feature vectors in chronological order to form a temporal sequence of facial features; The temporal sequence of facial features is input into a temporal convolutional network. Through the causal convolutional layer of the temporal convolutional network, local features in the temporal dimension of the temporal sequence of facial features are extracted to obtain an initial feature sequence. Long-range dependencies are extracted from the initial feature sequence through the dilated convolutional layer of the temporal convolutional network, and a target feature sequence is generated based on the long-range dependencies. Perform a global pooling operation on the target feature sequence to generate a pooled feature vector; A linear transformation is performed on the pooled feature vector to obtain a statistical feature vector that characterizes the frequency of face occurrence. The statistical feature vectors are normalized to generate face frequency data.

3. The method according to claim 1, characterized in that, The step of employing an attention mechanism to perform video frame criticality analysis on the video content feature information to generate a processing priority label sequence includes: The video content feature information is segmented using a sliding window method to obtain multiple overlapping local feature windows; By comparing the correlation between the features of each time step within each local feature window and the features of the central time step, a self-attention score is generated for each local feature window. The self-attention scores of each time step under different sliding windows are weighted and aggregated to generate the comprehensive attention weight for each time step; The comprehensive attention weight is compared with a preset dynamic threshold, and time steps in which the comprehensive attention weight is greater than the dynamic threshold are marked as key frames. A first priority label is generated based on the key frames. The time step corresponding to the comprehensive attention weight being less than the dynamic threshold is marked as a non-key frame. Based on the non-key frame, a second priority label is generated, and the first priority is greater than the second priority. For the time step where the comprehensive attention weight is equal to the dynamic threshold, interpolation is performed based on the labels of the adjacent time steps corresponding to the time step to determine the corresponding priority label; Arrange the priority labels of all time steps in chronological order to form a processing priority label sequence.

4. The method according to claim 1, characterized in that, The step of predicting the quantified value of computing power availability for each client within a future time period based on the historical load data and the real-time status information, combined with a load prediction model established based on a long short-term memory network, includes: The historical load data and the real-time status information are time-series aligned to form client status time-series data; The client state time series data is input into the load prediction model. The long-term dependency relationship of the client state time series data is modeled through the memory unit of the load prediction model to obtain a time series feature representation. The time series feature representation is mapped to a multi-dimensional feature space through the fully connected layer of the load prediction model to generate a high-dimensional feature vector. The high-dimensional feature vector is subjected to nonlinear regression calculation through the output layer of the load prediction model to generate predicted values ​​of computing power availability at multiple consecutive time points in the future. The predicted computing power availability is quantized and encoded to generate a quantized computing power availability value.

5. The method according to claim 1, characterized in that, The dynamic orchestration and allocation of each computing slice based on the processing priority label sequence and the computing power availability quantification value includes: Based on the video time interval corresponding to each priority label in the processing priority label sequence, determine the video segment to be processed from the computation slice, and determine the amount of computational resources required for each video segment to be processed; The computing power of each client is evaluated based on the aforementioned computing power availability quantification value; All computing slices are dynamically arranged according to a preset priority order. Based on the matching relationship between the amount of computing resources and the computing power, the dynamically arranged computing slices are allocated to clients with different computing power availability quantification values. During the allocation process, the actual load of each client is monitored in real time. The allocation strategy for computing slices is dynamically adjusted based on the actual load conditions.

6. A cloud server load balancing system based on client-side computing power collaboration, characterized in that, include: The acquisition module is used to acquire the video analysis tasks to be processed on the cloud server, the video stream data uploaded in real time by multiple clients, the real-time status information of each client, and the historical load data. The generation module is used to process the video stream data through a lightweight convolutional network to generate video content feature information; The analysis module is used to perform video frame criticality analysis on the video content feature information using an attention mechanism to generate a processing priority label sequence; The prediction module is used to predict the quantified value of computing power availability for each client in the future time period based on the historical load data and the real-time status information, combined with the load prediction model established based on the long short-term memory network. The decomposition module is used to decompose the video analysis task to be processed into multiple computing slices, and to dynamically arrange and allocate each computing slice based on the processing priority label sequence and the computing power availability quantification value, so as to realize computing power collaboration and load balancing between the cloud server and multiple clients. The process of processing the video stream data using a lightweight convolutional network to generate video content feature information includes: The video stream data is sampled in a time sequence to obtain a sequence of sampled video frames consisting of multiple equally spaced video frames; The sampled video frame sequence is input into a lightweight convolutional network, and the first feature extraction layer of the lightweight convolutional network captures local details of the video frames to obtain a primary feature map. The intermediate feature map is obtained by combining features from the primary feature map through the second feature extraction layer of the lightweight convolutional network. The intermediate feature map is integrated through the third feature extraction layer of the lightweight convolutional network using a feature channel weighted fusion method to generate a video content feature map. Based on the video content feature map, video content feature information is generated; The step of generating video content feature information based on the video content feature map includes: The scene dynamic change rate data is parsed from the video content feature map; A temporal convolutional network is used to analyze the facial feature vectors within a preset time window in the video content feature map to generate facial occurrence frequency data. By comparing the similarity of adjacent video frames in brightness and contrast distribution in the video content feature map, inter-frame structural similarity data is generated. The scene dynamic change rate data, the face occurrence frequency data, and the inter-frame structural similarity data are fused across modal features to form video content feature information.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the cloud server load balancing method based on client-side computing power collaboration as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the cloud server load balancing method based on client-side computing power collaboration as described in any one of claims 1 to 5.

Images