Model computing resource scheduling method and device based on frequency domain correlation

Abstract

Embodiments of the present disclosure disclose a large model computing resource scheduling method and device based on frequency domain correlation. A specific implementation of the method includes: preprocessing an original video stream to generate a set of image frames to be processed; performing frequency domain transformation on each image frame in the set of image frames to generate a corresponding set of frequency domain coefficient matrices; performing low and medium frequency component extraction and semantic enhancement operations on each frequency domain coefficient matrix in the set of frequency domain coefficient matrices to generate a corresponding set of frequency domain semantic feature vectors; determining the similarity of the frequency domain semantic feature vectors corresponding to each two adjacent frames in the set of image frames; generating semantic change probability information corresponding to each image frame to obtain a set of semantic change probability information; performing a dynamic frame extraction operation on the set of image frames to generate a set of key frames; and dynamically scheduling the computing resources of the large model. The implementation can reduce the computational redundancy and computing resource consumption of the large model in processing the video stream, and reduce the response time.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and specifically to a method and apparatus for scheduling large-scale computational resources based on frequency domain correlation. Background Technology

[0002] Currently, existing large models typically use a frame-by-frame spatial domain feature extraction approach to schedule computational resources when processing video streams. This involves performing complete large model inference on each frame of the video. However, when scheduling computational resources for large models in this way, the following technical problems usually arise: repeated computation on adjacent frames with highly similar content generates a large amount of unnecessary computational overhead, wastes computational resources, and increases inference response time. Summary of the Invention

[0003] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] Some embodiments of this disclosure propose a method, apparatus, electronic device, and computer-readable medium for scheduling large-scale computational resources based on frequency domain correlation to address one or more of the technical problems mentioned in the background section above.

[0005] In a first aspect, some embodiments of this disclosure provide a method for scheduling large model computing resources based on frequency domain correlation, comprising: responding to a request information received by the large model for an inference task of an original video stream, preprocessing the original video stream to generate a set of image frames to be processed; performing frequency domain transformation on each image frame in the set of image frames to generate a corresponding set of frequency domain coefficient matrices; performing low- and mid-frequency component extraction and semantic enhancement operations on each frequency domain coefficient matrix in the set of frequency domain coefficient matrices to generate a corresponding set of frequency domain semantic feature vectors; determining the similarity of frequency domain semantic feature vectors corresponding to every two adjacent frames in the set of image frames based on the set of frequency domain semantic feature vectors to generate an inter-frame correlation measurement sequence; generating semantic change probability information corresponding to each image frame using a pre-trained semantic change prediction model according to the inter-frame correlation measurement sequence to obtain a set of semantic change probability information; performing dynamic frame extraction on the set of image frames based on the set of semantic change probability information to generate a set of key frames; and dynamically scheduling the computing resources of the large model based on the set of key frames and the semantic change probability information corresponding to the set of key frames to execute differentiated inference tasks.

[0006] Secondly, some embodiments of this disclosure provide a large model computational resource scheduling apparatus based on frequency domain correlation, comprising: a preprocessing unit configured to preprocess the original video stream in response to the large model receiving a request information for an inference task of the original video stream to generate a set of image frames to be processed; a frequency domain transformation unit configured to perform frequency domain transformation on each image frame in the set of image frames to generate a corresponding set of frequency domain coefficient matrices; an extraction and enhancement unit configured to perform low- and mid-frequency component extraction and semantic enhancement operations on each frequency domain coefficient matrix in the set of frequency domain coefficient matrices to generate a corresponding set of frequency domain semantic feature vectors; and a determination unit configured to determine based on the frequency domain semantic... The feature vector set determines the similarity of frequency domain semantic feature vectors corresponding to every two adjacent frames in the image frame set to generate an inter-frame correlation measurement sequence. The generation unit is configured to generate semantic change probability information corresponding to each image frame based on the inter-frame correlation measurement sequence and a pre-trained semantic change prediction model to obtain a semantic change probability information set. The frame extraction unit is configured to perform dynamic frame extraction operations on the image frame set based on the semantic change probability information set to generate a key frame set. The dynamic scheduling unit is configured to dynamically schedule the computing resources of the large model based on the key frame set and the semantic change probability information corresponding to the key frame set to perform differentiated inference tasks.

[0007] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0008] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0009] The above embodiments of this disclosure have the following beneficial effects: By using the large model computational resource scheduling method based on frequency domain correlation in some embodiments of this disclosure, the computational redundancy and computational resource consumption of large models processing video streams are reduced, and the response time is shortened. Specifically, the reason for excessive computational resource consumption and response time is that existing methods perform undifferentiated complete model inference on adjacent frames with highly similar content in video sequences, failing to effectively utilize the temporal redundancy characteristics between frames, resulting in a large amount of computational power being wasted on repetitive or invalid feature extraction. Based on this, the large model computational resource scheduling method based on frequency domain correlation in some embodiments of this disclosure firstly, in response to the large model receiving a request information for the original video stream inference task, preprocesses the original video stream to generate a set of image frames to be processed. The original video stream is decoded and standardized into a unified format image frame sequence, eliminating interference from video encoding differences and resolution inconsistencies, providing regular input data for subsequent frequency domain analysis, and ensuring the stability and repeatability of the processing flow. Then, frequency domain transformation is performed on each image frame in the above image frame set to generate a corresponding set of frequency domain coefficient matrices. The image frames are transformed from the spatial domain to the frequency domain, and pixel information is reconstructed into coefficients of different frequency components. High-frequency noise and low-frequency semantic features are removed, providing a data foundation for subsequent semantic layering. Then, low- and mid-frequency components are extracted and semantic enhancement is performed on each frequency coefficient matrix in the aforementioned frequency domain coefficient matrix set to generate a corresponding frequency domain semantic feature vector set. High-frequency noise components are suppressed, and the structural and contour features of the mid- and low-frequency bands are strengthened to improve the anti-interference ability of semantic features, making the inter-frame similarity calculation more focused on stable semantic content rather than instantaneous interference. Next, based on the aforementioned frequency domain semantic feature vector set, the similarity of the frequency domain semantic feature vectors corresponding to every two adjacent frames in the aforementioned image frame set is determined to generate an inter-frame correlation measurement sequence. By calculating the cosine similarity of the frequency domain features of adjacent frames, the continuity and degree of change of the sequence content are quantified, establishing an objective indicator of inter-frame semantic association and providing data support for change detection. Finally, based on the aforementioned inter-frame correlation measurement sequence, a pre-trained semantic change prediction model is used to generate semantic change probability information corresponding to each image frame to obtain a semantic change probability information set. A pre-trained semantic change prediction model identifies semantic change patterns from inter-frame correlation measurement sequences, proactively predicting the probability of key events and achieving intelligent semantic change detection, providing a reliable basis for frame extraction decisions. Secondly, based on the aforementioned semantic change probability information set, dynamic frame extraction is performed on the aforementioned image frame set to generate a keyframe set. The frame extraction density is adaptively adjusted according to the semantic change probability, reducing redundant frames in static scenes and retaining keyframes in dynamic scenes, minimizing the number of frames processed while ensuring semantic integrity. Finally, based on the aforementioned keyframe set and the corresponding semantic change probability information, the computational resources of the large model are dynamically scheduled to execute differentiated inference tasks.By allocating computing resources of different scales according to the semantic importance of keyframes, fine-grained scheduling of computing load is achieved. While ensuring in-depth analysis of keyframes, the overall efficiency is greatly improved through lightweight inference and feature reuse, reducing the waste of computing resources and thus reducing inference response time. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0011] Figure 1 This is a flowchart of some embodiments of the resource scheduling method based on a large model of frequency domain correlation according to this disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of the large-scale computational resource scheduling device based on frequency domain correlation according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0017] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] refer to Figure 1 The flowchart 100 illustrates some embodiments of a large model computational resource scheduling method based on frequency domain correlation according to this disclosure. This large model computational resource scheduling method based on frequency domain correlation includes the following steps: Step 101: In response to the large model receiving a request for an inference task on the original video stream, preprocess the original video stream to generate a set of image frames to be processed.

[0019] In some embodiments, the execution entity (e.g., an electronic device) of the above-described large-scale model computing resource scheduling method based on frequency domain correlation can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed in the hardware devices listed above. It can be implemented as multiple software programs or software modules to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0020] In other embodiments, the execution entity may preprocess the original video stream in response to the large model receiving a request for an inference task on the original video stream to generate a set of image frames to be processed. The large model may be a deep learning model for video content understanding. For example, the large model may be GPT-4V or a Swing Transformer. The original video stream may be unprocessed, typically encoded and compressed, continuous video data. The inference task may refer to a specific objective for the large model to predict or analyze the input data. For example, the inference task may be behavior recognition, anomaly detection, or scene understanding of the video. The request information may be an instruction or data packet that triggers the large model to begin processing the video stream, including task parameters (e.g., video source address, inference task type). The set of image frames may be a preprocessed, uniformly formatted collection of images.

[0021] In some optional implementations of certain embodiments, the execution entity may, in response to the large model receiving a request for an inference task on the original video stream, preprocess the original video stream to generate a set of image frames to be processed, which may include the following steps: The first step involves decoding and sampling the original video stream at a fixed frame rate to generate an initial image frame sequence. This initial image frame sequence can be an ordered set of images obtained after decoding and initial sampling of the original video stream; the format and size are not yet standardized. In practice, firstly, the original video stream is decompressed into consecutive original image frames using a suitable decoder (e.g., the FFmpeg library). Then, images are uniformly extracted from the decoded consecutive frame sequence according to a preset fixed sampling rate (e.g., 1 frame per second). Finally, the extracted images are arranged in their original chronological order to form the initial image frame sequence.

[0022] The second step involves normalizing the size and color space of each image frame in the initial image frame sequence to generate an image frame set. In practice, this is done by first iterating through each frame in the initial image frame sequence. Then, size normalization (e.g., scaling all frames to a uniform 224x224 pixels using bilinear interpolation) and color space normalization (e.g., converting OpenCV's default BGR format to the model's common RGB format) are applied to each image frame. Finally, the processed images are combined into an image frame set.

[0023] Step 102: Perform frequency domain transformation on each image frame in the image frame set to generate the corresponding frequency domain coefficient matrix set.

[0024] In some embodiments, the execution entity may perform a frequency domain transformation on each image frame in the image frame set to generate a corresponding set of frequency domain coefficient matrices. The frequency domain transformation may be a mathematical operation that converts an image from the spatial domain (pixel intensity distribution) to the frequency domain (energy distribution of different frequency components). The set of frequency domain coefficient matrices may be a collection of coefficient matrices obtained after the frequency domain transformation of each image frame in the image frame set. The frequency domain coefficient matrix may be a matrix representing the frequency domain information of the entire image frame, formed by reassembling the processed sub-matrices of the image frame according to their original spatial positions. For example, the frequency domain coefficient matrix may be a complete 224×224 frequency matrix formed by reassembling 32 8×8 processed sub-matrices.

[0025] In some optional implementations of certain embodiments, the execution entity may perform a frequency domain transformation on each image frame in the image frame set to generate a corresponding frequency domain coefficient matrix set, which may include the following steps: The first step is to perform the following steps for each image frame in the above image frame set: Sub-step one involves performing a block segmentation operation on the aforementioned image frame to generate multiple regular image blocks. These multiple regular image blocks can be achieved by dividing the image frame into multiple regular regions of the same size, typically square blocks. In practice, first, the block size is determined (e.g., 8×8 pixels). Then, the image is evenly divided from left to right and from top to bottom. Finally, a set of regular image blocks of the same size is generated as multiple regular image blocks.

[0026] Sub-step two involves performing a discrete cosine transform (DCT) on each of the multiple regular image blocks to obtain a frequency domain coefficient submatrix for each block. This frequency domain coefficient submatrix can be the frequency coefficient matrix obtained after frequency domain transformation of a single image block. In practice, first, the DCT formula is applied to each image block to obtain the frequency coefficients of each block. Then, a frequency domain coefficient submatrix is ​​generated for each block. For example, applying 2D-DCT to each 8×8 image block yields an 8×8 submatrix composed of 64 frequency coefficients.

[0027] Step three involves normalizing each frequency domain coefficient submatrix to generate a processed submatrix for each. This processed submatrix can be the result of normalizing the frequency domain coefficient submatrix. In practice, first, the statistical characteristics of each submatrix (e.g., mean, variance) are determined; then, the coefficient values ​​are adjusted using a normalization formula. Finally, processed submatrices with consistent numerical ranges are generated.

[0028] Sub-step four involves arranging and combining the processed sub-matrices corresponding to the aforementioned image frames according to their spatial positions to generate the frequency domain coefficient matrix corresponding to the image frames. In practice, firstly, all processed sub-matrices are arranged according to their spatial positions in the original image. Then, the processed sub-matrices are concatenated. Finally, a frequency domain coefficient matrix corresponding to the size of the original image is formed.

[0029] The second step is to define the frequency domain coefficient matrices as a set. In practice, the frequency domain coefficient matrices corresponding to each image frame can be organized in frame order to form a complete set of frequency domain coefficient matrices.

[0030] Step 103: Perform low- and mid-frequency component extraction and semantic enhancement operations on each frequency domain coefficient matrix in the frequency domain coefficient matrix set to generate the corresponding frequency domain semantic feature vector set.

[0031] In some embodiments, the execution entity may perform low- and mid-frequency component extraction and semantic enhancement operations on each frequency domain coefficient matrix in the aforementioned frequency domain coefficient matrix set to generate a corresponding frequency domain semantic feature vector set. The aforementioned low- and mid-frequency components may be frequency components in the frequency domain that are energy-concentrated and characterize the main structure and stable semantics of the image, typically a set of low frequencies and some mid frequencies. For example, the low-frequency coefficients in the upper left region of a DCT transform represent the image contour and background. The aforementioned frequency domain semantic feature vector may be a low-dimensional vector obtained by compressing and encoding the image's frequency domain features, representing the semantic information of the image.

[0032] In some optional implementations of certain embodiments, the execution entity may perform low- and mid-frequency component extraction and semantic enhancement operations on each frequency domain coefficient matrix in the frequency domain coefficient matrix set to generate a corresponding frequency domain semantic feature vector set, which may include the following steps: The first step, for each frequency domain coefficient matrix in the above set of frequency domain coefficient matrices, is to perform the following steps: Sub-step one involves extracting low-frequency components from the aforementioned frequency domain coefficient matrix to generate a low-frequency coefficient matrix. This low-frequency coefficient matrix can be a matrix composed of the low-frequency components extracted from the frequency domain coefficient matrix, representing the main structure and stable background of the image. In practice, first, the frequency range of the low-frequency components is determined (e.g., the upper left corner region of the DCT matrix); then, coefficients for the corresponding region are extracted from the frequency domain coefficient matrix. Finally, the low-frequency coefficient matrix is ​​generated. For example, 256 coefficients from the upper left 16×16 region of a 64×64 DCT matrix are extracted to form a 16×16 low-frequency matrix.

[0033] Sub-step two involves extracting intermediate frequency (IF) components from the aforementioned frequency domain coefficient matrix to generate an IF coefficient matrix. This IF coefficient matrix can be a matrix composed of the IF portion extracted from the frequency domain coefficient matrix, representing the image's detail texture and motion variations. In practice, first, the frequency range of the IF components is defined (e.g., the central annular region of the DCT matrix). Then, the coefficients of this region are extracted. Finally, the IF coefficient matrix is ​​generated. 768 coefficients from the 16×16 to 32×32 annular region are extracted from the 64×64 DCT matrix to form the corresponding IF matrix.

[0034] Sub-step three involves extracting high-frequency components from the aforementioned frequency domain coefficient matrix to generate a high-frequency coefficient matrix, and then performing noise suppression processing on this high-frequency coefficient matrix. The high-frequency coefficient matrix can be a set of frequency coefficients extracted from the image's frequency domain representation, corresponding to image details, texture noise, and rapidly changing parts. For example, in an 8×8 coefficient block of a DCT transform, the coefficients in the lower right corner region constitute the high-frequency coefficient matrix, primarily including image edge details and compression noise. In practice, first, the remaining high-frequency region coefficients are extracted. Then, threshold filtering or wavelet denoising methods are applied to suppress noise.

[0035] Sub-step four involves performing semantic enhancement operations on the aforementioned low-frequency coefficient matrix and mid-frequency coefficient matrix to obtain enhanced low-frequency components and enhanced mid-frequency components. The enhanced low-frequency components can be obtained by enhancing the low-frequency coefficient matrix, thus strengthening the semantic expression of the main structure. The enhanced mid-frequency components can be obtained by enhancing the mid-frequency coefficient matrix, thus strengthening the expression of detailed textures and changing regions. In practice, firstly, contrast stretching is performed on the low-frequency coefficient matrix to enhance the main structure. Then, edge enhancement is performed on the mid-frequency coefficient matrix to highlight details. Finally, the enhanced low-frequency and mid-frequency components are obtained.

[0036] Sub-step five involves fusing the enhanced low-frequency and mid-frequency components to obtain a preliminary semantic feature matrix. This preliminary semantic feature matrix can be a matrix formed by fusing the enhanced low-frequency and mid-frequency components, encompassing the complete semantic features of the image. For example, the enhanced 8×8 low-frequency matrix and the 8×8 mid-frequency matrix can be concatenated to form a 16×8 feature matrix. In practice, the enhanced low-frequency and mid-frequency components are first concatenated along the feature dimension. Then, information is integrated using weighted fusion or an attention mechanism. Finally, the preliminary semantic feature matrix is ​​generated.

[0037] Sub-step six involves compressing and encoding the preliminary semantic feature matrix to generate the frequency domain semantic feature vector corresponding to the frequency domain coefficient matrix. In practice, first, the preliminary semantic feature matrix is ​​reduced in dimensionality (e.g., by PCA or an autoencoder); then, the reduced feature vector is normalized; finally, a compact frequency domain semantic feature vector is generated.

[0038] The second step is to define the frequency domain semantic feature vectors as a set of frequency domain semantic feature vectors. In practice, firstly, the frequency domain semantic feature vectors corresponding to each frame are collected. Then, these vectors are organized in chronological order. Finally, a set of frequency domain semantic feature vectors is formed.

[0039] Step 104: Based on the frequency domain semantic feature vector set, determine the similarity of the frequency domain semantic feature vectors corresponding to every two adjacent frames in the image frame set, so as to generate an inter-frame correlation measurement sequence.

[0040] In some embodiments, the execution entity can determine the similarity of the frequency domain semantic feature vectors corresponding to every two adjacent frames in the image frame set based on the frequency domain semantic feature vector set, in order to generate an inter-frame correlation measurement sequence. The adjacent frames can be two images that are temporally adjacent within the image frame set. The inter-frame correlation measurement sequence can be a sequence formed by arranging the similarity evaluation results of each pair of adjacent frames in the image frame set in chronological order. For example, the inter-frame correlation measurement sequence can be [0.95, 0.12, 0.88, ...], representing that the similarity between frames 1 and 2 is 0.95, and the similarity between frames 2 and 3 is 0.12.

[0041] In some optional implementations of certain embodiments, the execution entity may determine the similarity of the frequency domain semantic feature vectors corresponding to every two adjacent frames in the image frame set based on the frequency domain semantic feature vector set, in order to generate an inter-frame correlation measurement sequence, which may include the following steps: The first step, for each pair of adjacent frames corresponding to the frequency domain semantic feature vectors in the above frequency domain semantic feature vector set, is to perform the following steps: Sub-step one involves determining the cosine similarity between the semantic feature vectors corresponding to the adjacent frames to generate initial similarity information. This initial similarity information can be the inter-frame similarity value obtained directly through cosine similarity calculation without any adjustment. In practice, firstly, the semantic feature vectors corresponding to two adjacent frames are extracted. Then, the ratio of the dot product of the two vectors to their magnitude is calculated. Finally, an initial similarity value ranging from [-1, 1] is obtained as the initial similarity information.

[0042] Sub-step two involves correcting the initial similarity information based on the differences in low- and mid-frequency energy distribution among the semantic feature vectors of adjacent frames, yielding corrected similarity information. The aforementioned differences in low- and mid-frequency energy distribution refer to the difference in energy concentration at different frequency bands (e.g., low and mid frequencies) in the frequency domain between the two images. For example, if one frame has 70% low-frequency energy and the other has 50%, the difference is 20%. The corrected similarity information is the final similarity value adjusted for energy distribution differences based on the initial similarity information. In practice, firstly, the proportion of low- and mid-frequency energy in the feature vectors of the two frames is calculated. Then, the absolute difference in energy proportions is calculated as a correction factor. Finally, the corrected similarity is obtained by subtracting the weighted correction factor from the initial similarity. For example, if the low-frequency energy proportions of the two frames are 65% and 70% respectively, with a difference of 5%, the initial similarity of 0.92 minus 0.05 × 0.3 = 0.015 yields a corrected similarity of 0.905.

[0043] Step three involves comparing the corrected similarity with a preset similarity threshold to obtain a similarity classification result. The preset similarity threshold can be a pre-defined critical value used to determine the level of inter-frame similarity. The similarity classification result can be a category label assigned to the similarity value based on the threshold. For example, a similarity of 0.8 is classified as "high similarity," and 0.3 is classified as "low similarity." In practice, first, preset high similarity thresholds (e.g., 0.85) and low similarity thresholds (e.g., 0.4) are read. Then, the corrected similarity is compared with the thresholds. Finally, it is classified as "high similarity," "moderate similarity," or "low similarity."

[0044] Sub-step four involves combining the corrected similarity information with the similarity classification results to form an inter-frame correlation measure. This inter-frame correlation measure can be a complete correlation description including the corrected similarity values ​​and their classification results. In practice, first, a data structure is created. Then, the corrected similarity information and similarity classification results are stored in this data structure. Finally, the complete inter-frame correlation measure is generated.

[0045] The second step involves arranging the inter-frame correlation metrics generated sequentially into an inter-frame correlation metric sequence. In practice, this is done by first collecting the inter-frame correlation metrics for each pair of adjacent frames, then sorting them according to the temporal order of the frame pairs, and finally forming a complete inter-frame correlation metric sequence.

[0046] Step 105: Based on the inter-frame correlation measurement sequence, use the pre-trained semantic change prediction model to generate semantic change probability information for each image frame, so as to obtain a semantic change probability information set.

[0047] In some embodiments, the execution entity can generate semantic change probability information for each image frame based on the inter-frame correlation measurement sequence and a pre-trained semantic change prediction model, thereby obtaining a semantic change probability information set. The pre-trained semantic change prediction model can be a machine learning model (lightweight temporal model) trained on historical data to predict whether the semantic content of video frames has changed. For example, the pre-trained semantic change prediction model can be a lightweight temporal neural network (based on...) Transformer architecture ) ，This can include: an input layer (which receives a temporal feature vector composed of inter-frame correlation measures from the current frame and the k frames before and after it), a hidden layer (usually a fully connected layer or a one-dimensional convolutional layer, used to capture local temporal patterns), and an output layer (which uses a fully connected layer with a sigmoid activation function to map the features to a scalar between 0 and 1. The output is the initial semantic change probability, representing the likelihood that a semantic change has occurred in the frame). This semantic change probability information can be a numerical value output by the semantic change prediction model, representing the probability that a frame has undergone a significant semantic change. For example, an output probability value of 0.85 indicates that there is an 85% probability that the frame contains new semantic content.

[0048] In some optional implementations of certain embodiments, the execution entity can generate semantic change probability information for each image frame based on the inter-frame correlation measurement sequence and a pre-trained semantic change prediction model to obtain a semantic change probability information set, which may include the following steps: The first step is to perform the following steps for each inter-frame correlation metric in the above inter-frame correlation metric sequence: Sub-step one involves combining the inter-frame correlation metrics of the current frame and its adjacent frames before and after it into a temporal feature vector to generate a contextual feature representation of the current frame. This temporal feature vector can be a feature vector formed by concatenating the correlation metrics of the current frame and multiple frames before and after it in chronological order. The contextual feature representation can be a feature representation that includes the temporal contextual information of the current frame. For example, the contextual feature representation can be a 5-dimensional vector composed of the similarity scores of the two frames before and after the current frame. In practice, first, the size of the context window is determined (e.g., one frame before and one frame after, for a total of three frames). Then, the correlation metrics of the current frame and its adjacent frames are extracted. Finally, the correlation metrics are concatenated into a temporal feature vector in chronological order.

[0049] Sub-step two involves inputting the aforementioned contextual feature representation into the aforementioned semantic change prediction model to generate initial semantic change probability information. This initial semantic change probability information can be the raw probability values ​​directly output by the semantic change prediction model without post-processing.

[0050] Sub-step three involves adjusting the temporal weights of the initial semantic change probability information based on the temporal position information of the current frame within the aforementioned image frame set, to generate the adjusted semantic change probability. The adjusted semantic change probability can be a probability value obtained by adjusting the initial probability after considering factors such as temporal position weights. In practice, firstly, weights are calculated based on the temporal position of the image frame (e.g., higher weights in the middle of the video, lower weights at both ends). Then, the initial probability is multiplied by this weight. Finally, the adjusted probability value is obtained.

[0051] Sub-step four involves normalizing the adjusted semantic change probabilities to generate the semantic change probability information corresponding to the current frame. In practice, Min-Max or Softmax normalization methods can be applied to normalize the adjusted semantic change probabilities.

[0052] The second step is to arrange the final semantic change probability information corresponding to each image frame in frame order to obtain a semantic change probability information set. In practice, firstly, the semantic change probability information of each image frame is collected. Then, it is arranged in the original frame order. Finally, a complete semantic change probability information set is formed.

[0053] Step 106: Based on the semantic change probability information set, perform dynamic frame extraction on the image frame set to generate a keyframe set.

[0054] In some embodiments, the execution entity may perform dynamic frame extraction on the image frame set based on the semantic change probability information set to generate a keyframe set. The dynamic frame extraction operation may be a process of adaptively deciding whether to retain or discard video frames based on the semantic change probability information and real-time load. The keyframe set may be a collection of image frames that, after dynamic frame extraction, are fed into a large model for inference. For example, a 1000-frame video may ultimately generate a set including 150 keyframes.

[0055] In addressing the technical challenges mentioned above, the application scenario—real-time intelligent analysis of multiple video streams, such as video content review in urban security monitoring centers or cloud platforms—often presents the following challenges: during periods of sudden high traffic, multiple video streams flood in simultaneously, overloading large model inference servers, wasting computing resources, and increasing response time. Considering the specific requirements of this application scenario—high concurrency adaptability and elastic scheduling of computing resources—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity may perform dynamic frame extraction on the image frame set based on the semantic change probability information set to generate a keyframe set, which may include the following steps: The first step is to perform the following steps for each semantic change probability information in the above semantic change probability information set: Sub-step one involves generating a resource load metric by acquiring the current available computing resource status information. This information includes, for example, quantified data of real-time available computing resources, such as GPU utilization and memory idle rate. The resource load metric can be a scalar value of load level obtained by normalizing or comprehensively calculating the current available computing resource status information. For example, the resource load metric can be calculated by combining GPU utilization and memory usage, resulting in a load metric of 0.72 (range 0-1). In practice, firstly, the system API is called to obtain real-time GPU utilization, memory usage, and other information. Then, this information is normalized to the [0, 1] interval. Finally, a comprehensive resource load metric is calculated using a weighted formula. For example, with a GPU utilization of 90% and a memory usage of 80%, a weighted calculation (GPU weight 0.7, memory weight 0.3) yields a load metric of 0.87.

[0056] Sub-step two involves generating a basic keyframe determination result based on the aforementioned semantic change probability information and a preset static probability threshold. The preset static probability threshold can be a pre-defined fixed probability threshold used to determine whether a frame is a keyframe. The basic keyframe determination result can be a preliminary determination based solely on whether the semantic change probability exceeds the static threshold. In practice, first, the semantic change probability P of the current frame is read. Then, it is compared with the preset static probability threshold T. Finally, if P ≥ T, it is determined to be a basic keyframe; otherwise, it is a non-keyframe. Finally, the basic keyframe determination result is generated.

[0057] Sub-step three involves querying a predefined resource-threshold mapping relationship based on the aforementioned resource load indicators to generate threshold adjustment coefficients. This predefined resource-threshold mapping relationship can be a rule or function describing the correspondence between the resource load indicators and the threshold adjustment magnitude. For example, the predefined resource-threshold mapping relationship could be: when load > 0.8, coefficient = 1.2 (increase the threshold); when load < 0.3, coefficient = 0.8 (decrease the threshold). These threshold adjustment coefficients can be calculated based on the resource load indicators and the mapping relationship, and are used to scale the static threshold. In practice, first, based on the resource load indicators; then, querying a predefined resource-threshold mapping relationship; and finally, outputting the corresponding threshold adjustment coefficients.

[0058] Sub-step four involves determining the preliminary dynamic probability threshold based on the aforementioned threshold adjustment coefficient and the aforementioned static probability threshold. This preliminary dynamic probability threshold can be the initially adjusted threshold obtained by multiplying the static threshold by the threshold adjustment coefficient. In practice, first, the static threshold and the static probability threshold are obtained. Then, the two are multiplied together. Finally, the preliminary dynamic threshold is obtained. Sub-step five involves smoothing and correcting the preliminary dynamic probability threshold based on the semantic change probability information of multiple consecutive image frames prior to the current time, thereby generating a dynamic probability threshold. This dynamic probability threshold can be the actual threshold used for determining the current frame after temporally smoothing the preliminary dynamic threshold. In practice, firstly, preliminary dynamic thresholds for the current frame and the previous N frames (e.g., N=4) are obtained. Then, these values ​​are subjected to moving average or median filtering. Finally, the smoothed dynamic probability threshold is output.

[0059] Sub-step six: Based on the aforementioned semantic change probability information and dynamic probability threshold, generate a resource-aware keyframe determination result. This resource-aware keyframe determination result can be a result combining the frame's semantic change probability and the dynamic threshold, taking into account the current system load. In practice, first, the semantic change probability information P and the smoothed dynamic threshold T of the current frame are read. Then, P and T are directly compared. Finally, if P ≥ T, it is determined to be a "resource-aware keyframe"; otherwise, it is determined to be a "non-resource-aware keyframe".

[0060] Sub-step seven involves determining the scene's dynamic change trend information based on the aforementioned semantic change probability information and the corresponding semantic change probability information of adjacent frames. This scene dynamic change trend information can be based on the probability sequence of the current frame and its neighboring frames, determining whether the scene content change is accelerating, decelerating, or stable. In practice, firstly, the semantic change probabilities of the current frame and each of the preceding and following M frames (e.g., M=2) are obtained to form a short temporal window. Then, the first-order difference or linear regression slope of the probability sequence within this window is calculated. Finally, the trend is determined as "actively rising," "stable," or "decreasing" based on the sign and magnitude of the slope.

[0061] Sub-step eight: Based on the aforementioned scene dynamics change information and the aforementioned resource-aware keyframe determination results, determine the keyframe determination result. The aforementioned keyframe determination result can be the final result regarding whether to retain the frame after comprehensively considering the resource-aware determination result and the scene dynamics trend. In practice, first, check the resource-aware determination result. If it is "yes," it is usually directly determined as a keyframe. If it is "no," the scene dynamics trend is introduced for arbitration: if the trend is "significantly active and rising," the resource-aware result may be overturned, and it is still determined as a keyframe to prevent missed detection; if the trend is stable or declining, the "non-keyframe" determination is maintained.

[0062] The second step involves generating a keyframe set based on the keyframe determination results for each image frame. In practice, firstly, all images whose final determination result is a "keyframe" are selected from the image frame set. Then, these keyframes are arranged according to their original timestamp order. Finally, a structured keyframe set is formed. The above-described steps, as an inventive point of this disclosure, solve the technical problem mentioned in the background section: "During periods of sudden high traffic, multiple video streams flood in simultaneously, causing overload on large model inference servers, wasting computing resources, and increasing response time." The reasons for this technical problem are as follows: a fixed-threshold frame extraction strategy cannot perceive the real-time system load; when resources are scarce, it may lose critical information due to excessive frame extraction, or waste computing power when resources are abundant. This invention, by introducing a resource-aware dynamic threshold adjustment mechanism, saves computing resources and reduces inference latency.

[0063] Step 107: Based on the keyframe set and the semantic change probability information corresponding to the keyframe set, dynamically schedule the computing resources of the large model to perform differentiated inference tasks.

[0064] In some embodiments, the execution entity can dynamically schedule the computational resources of the large model based on the keyframe set and the semantic change probability information corresponding to the keyframe set, to execute differentiated inference tasks. The computational resources can be the hardware resources required for large model inference, such as the computing power, video memory, and bandwidth of a GPU / NPU. The differentiated inference tasks can be model inference processes with different complexities and computational costs allocated according to the importance of the large model's input data. For example, the differentiated inference tasks could use a large ViT model for deep analysis of keyframes and a small model for fast processing of minor frames.

[0065] In addressing the aforementioned technical challenges in implementing solutions, the application scenario—a large-scale cloud-edge collaborative intelligent traffic flow video analysis system—often presents the following technical issues: the inability to differentiate and adaptively allocate computing resources based on the importance of frame content. This leads to insufficient analysis of key frames or excessive computation of minor frames, resulting in low resource utilization and increased computational costs. Considering the specific requirements of this application scenario—precise on-demand allocation of computing resources, high-precision assurance for critical tasks, and maximization of overall efficiency—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity can dynamically schedule the computational resources of the large model based on the keyframe set and the semantic change probability information corresponding to the keyframe set to perform differentiated inference tasks, which may include the following steps: The first step is to perform the following steps based on the semantic change probability information corresponding to each keyframe in the above keyframe set: Sub-step one: Based on the aforementioned semantic change probability information, determine the semantic importance level of the keyframes. The semantic importance level of the keyframes can be determined by classifying keyframes into different importance levels based on information such as semantic change probability. In practice, first, read the semantic change probability value P of the current keyframe. Then, query a predefined threshold range (e.g., high: P≥0.8, medium: 0.5≤P<0.8, low: P<0.5). Finally, map the P value to the corresponding "high," "medium," or "low" importance level.

[0066] Sub-step two involves determining the corresponding target inference path based on the aforementioned semantic importance levels. These target inference paths include: a full-scale deep inference path, a lightweight model inference path, and a cross-frame reuse inference path. Each target inference path can be a pre-defined model processing flow and resource configuration scheme for keyframes of different importance levels. The full-scale deep inference path can call the largest model with the largest parameter scale and deepest layers to perform complete and high-precision inference on the input frame. For example, using the Swin Transformer-Huge model for panoramic semantic segmentation of keyframes. The lightweight model inference path can call a small, efficient model that has been compressed or pruned for fast but slightly lower-precision inference. For example, using the EfficientNet-B0 model for object classification of keyframes. The cross-frame reuse inference path can reuse the computation results of historically similar frames without running the full model, obtaining the inference result for the current frame through feature alignment and incremental computation. For example, if the current frame is similar to a frame from 10 seconds ago, its high-level features are reused, with only a lightweight network for fine-tuning. In practice, firstly, based on the importance levels determined in the previous step... Then, the predefined mapping rules are queried (e.g., high -> full deep inference path, medium -> lightweight model inference path, low -> cross-frame reuse inference path). Finally, the corresponding target inference path is assigned to the frame.

[0067] Sub-step three involves dynamically allocating large-scale model computing resources based on the semantic importance level, corresponding to either the full inference path or the lightweight model inference path, to execute the corresponding inference task. The currently available computing resource status can be a real-time measure of remaining computing power at the scheduling moment. The corresponding large-scale model computing resources can be dynamically allocated hardware resources matched to the task requirements for the target inference path. In practice, first, the currently available computing resource status (currently available GPU memory and computing power) is obtained. Then, based on the path requirements (full inference requires a large amount of resources, lightweight inference requires fewer resources), a quota is dynamically allocated from the resource pool. Finally, the corresponding model is loaded, keyframes are input, and inference is executed.

[0068] Sub-step four, in response to the cross-frame reuse inference path corresponding to the semantic importance level, performs the following steps: The first sub-step involves obtaining semantic feature information of similar frames from the previously inferred historical keyframes. These previously inferred historical keyframes can be keyframes that have already been processed and whose intermediate features or final results have been saved; for example, a frame processed 5 seconds ago, whose features have been cached at layer 12 of the ViT model. The similar frames can be historical keyframes that are highly similar to the current keyframe in visual content or semantics. The semantic feature information can be high-dimensional vectors or feature maps representing the semantics of the historical similar frames, extracted from the model's intermediate or output layers. In practice, firstly, the most similar frame is found in the previously inferred historical keyframes of the current frame through feature retrieval. Then, the pre-calculated semantic feature information of that similar frame is read from the cache.

[0069] The second sub-step involves aligning the aforementioned semantic feature information with the aforementioned keyframes in a spatiotemporal manner to obtain aligned feature information. This aligned feature information can be derived by geometrically or semantically transforming the semantic features of historical frames to align them with the spatial or semantic structure of the current frame. In practice, firstly, the optical flow field or feature correlation between the current frame and similar frames is calculated. Then, the semantic features of historical frames are spatially deformed or weighted and fused using the optical flow field or feature correlation between the current frame and similar frames. Finally, a feature representation aligned with the spatial structure of the current frame is generated as the aligned feature information. For example, a RAFT optical flow network can be used to estimate motion, and the feature maps of historical frames can be bilinearly sampled based on the optical flow field and aligned to the current frame.

[0070] The third sub-step involves inputting the aligned feature information into a pre-trained feature reuse inference model for incremental inference to execute the corresponding inference task. This pre-trained feature reuse inference model can be a lightweight network that receives the aligned feature information and outputs the incremental inference result for the current frame. For example, it could be a small Transformer network. The pre-trained feature reuse inference model may include: an input layer (aligned feature information), a feature fusion layer (composed of multiple convolutional or fully connected layers), and an output layer (fully connected layer + Softmax). In practice, first, the aligned feature information is input into the pre-trained feature reuse inference model. Then, the aligned feature reuse inference model performs fine-tuning and supplementary calculations based on the aligned features. Finally, the final inference result for the current frame is output.

[0071] The above-described steps, as an inventive point of this disclosure, solve the technical problem mentioned in the background art: "The inability to differentiate and adaptively schedule computing resources based on the importance of frame content leads to insufficient analysis of key frames or excessive computation of minor frames, resulting in low resource utilization and increased computational costs." The reasons for this technical problem are as follows: Existing scheduling strategies only complete frame screening but do not further differentiate processing intensity based on the semantic value of frames; all key frames are allocated the highest priority computing resources. This invention, by establishing a multi-path reasoning and dynamic resource scheduling mechanism based on semantic importance, achieves an optimal balance between in-depth analysis of key frames and rapid processing of minor frames, saving a significant amount of unnecessary computational overhead and significantly improving system throughput and response speed.

[0072] The above embodiments of this disclosure have the following beneficial effects: By using the large model computational resource scheduling method based on frequency domain correlation in some embodiments of this disclosure, the computational redundancy and computational resource consumption of large models processing video streams are reduced, and the response time is shortened. Specifically, the reason for excessive computational resource consumption and response time is that existing methods perform undifferentiated complete model inference on adjacent frames with highly similar content in video sequences, failing to effectively utilize the temporal redundancy characteristics between frames, resulting in a large amount of computational power being wasted on repetitive or invalid feature extraction. Based on this, the large model computational resource scheduling method based on frequency domain correlation in some embodiments of this disclosure firstly, in response to the large model receiving a request information for the original video stream inference task, preprocesses the original video stream to generate a set of image frames to be processed. The original video stream is decoded and standardized into a unified format image frame sequence, eliminating interference from video encoding differences and resolution inconsistencies, providing regular input data for subsequent frequency domain analysis, and ensuring the stability and repeatability of the processing flow. Then, frequency domain transformation is performed on each image frame in the above image frame set to generate a corresponding set of frequency domain coefficient matrices. The image frames are transformed from the spatial domain to the frequency domain, and pixel information is reconstructed into coefficients of different frequency components. High-frequency noise and low-frequency semantic features are removed, providing a data foundation for subsequent semantic layering. Then, low- and mid-frequency components are extracted and semantic enhancement is performed on each frequency coefficient matrix in the aforementioned frequency domain coefficient matrix set to generate a corresponding frequency domain semantic feature vector set. High-frequency noise components are suppressed, and the structural and contour features of the mid- and low-frequency bands are strengthened to improve the anti-interference ability of semantic features, making the inter-frame similarity calculation more focused on stable semantic content rather than instantaneous interference. Next, based on the aforementioned frequency domain semantic feature vector set, the similarity of the frequency domain semantic feature vectors corresponding to every two adjacent frames in the aforementioned image frame set is determined to generate an inter-frame correlation measurement sequence. By calculating the cosine similarity of the frequency domain features of adjacent frames, the continuity and degree of change of the sequence content are quantified, establishing an objective indicator of inter-frame semantic association and providing data support for change detection. Finally, based on the aforementioned inter-frame correlation measurement sequence, a pre-trained semantic change prediction model is used to generate semantic change probability information corresponding to each image frame to obtain a semantic change probability information set. A pre-trained semantic change prediction model identifies semantic change patterns from inter-frame correlation measurement sequences, proactively predicting the probability of key events and achieving intelligent semantic change detection, providing a reliable basis for frame extraction decisions. Secondly, based on the aforementioned semantic change probability information set, dynamic frame extraction is performed on the aforementioned image frame set to generate a keyframe set. The frame extraction density is adaptively adjusted according to the semantic change probability, reducing redundant frames in static scenes and retaining keyframes in dynamic scenes, minimizing the number of frames processed while ensuring semantic integrity. Finally, based on the aforementioned keyframe set and the corresponding semantic change probability information, the computational resources of the large model are dynamically scheduled to execute differentiated inference tasks.By allocating computing resources of different scales according to the semantic importance of keyframes, fine-grained scheduling of computing load is achieved. While ensuring in-depth analysis of keyframes, the overall efficiency is greatly improved through lightweight inference and feature reuse, reducing the waste of computing resources and thus reducing inference response time.

[0073] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a large-scale model computational resource scheduling device based on frequency domain correlation. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this large-scale computational resource scheduling device based on frequency domain correlation can be specifically applied to various electronic devices.

[0074] like Figure 2 As shown, a large-scale model computational resource scheduling device 200 based on frequency domain correlation includes: an acquisition unit 201, a frequency domain transformation unit 202, an extraction and enhancement unit 203, a determination unit 204, a generation unit 205, a frame extraction unit 206, and a dynamic scheduling unit 207. The preprocessing unit 201 is configured to: in response to receiving a request for an inference task on the original video stream, preprocess the original video stream to generate a set of image frames to be processed. The frequency domain transformation unit 202 is configured to: perform a frequency domain transformation on each image frame in the image frame set to generate a corresponding set of frequency domain coefficient matrices. The extraction and enhancement unit 203 is configured to: perform low- and mid-frequency component extraction and semantic enhancement operations on each frequency domain coefficient matrix in the frequency domain coefficient matrix set to generate a corresponding set of frequency domain semantic feature vectors. The determination unit 204 is configured to: based on the frequency domain semantic feature vector set, determine the similarity of the frequency domain semantic feature vectors corresponding to every two adjacent frames in the image frame set to generate an inter-frame correlation measurement sequence. The generation unit 205 is configured to generate semantic change probability information for each image frame based on the aforementioned inter-frame correlation measurement sequence and a pre-trained semantic change prediction model, thereby obtaining a semantic change probability information set. The frame extraction unit 206 is configured to perform dynamic frame extraction on the aforementioned image frame set based on the aforementioned semantic change probability information set, thereby generating a keyframe set. The dynamic scheduling unit 207 is configured to dynamically schedule the computational resources of the aforementioned large model based on the aforementioned keyframe set and the semantic change probability information corresponding to the aforementioned keyframe set, in order to perform differentiated inference tasks.

[0075] It is understandable that the units and references described in the large-scale computational resource scheduling device 200 based on frequency domain correlation are... Figure 1 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the large-scale computational resource scheduling device 200 based on frequency domain correlation and the units contained therein, and will not be repeated here.

[0076] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0077] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0078] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0079] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0080] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0081] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0082] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs. When the aforementioned one or more programs are executed by the electronic device, the electronic device causes the following actions: In response to the large model receiving a request for an inference task on the original video stream, the electronic device preprocesses the original video stream to generate a set of image frames to be processed; performs a frequency domain transformation on each image frame in the set of image frames to generate a corresponding set of frequency domain coefficient matrices; performs low- and mid-frequency component extraction and semantic enhancement operations on each frequency domain coefficient matrix in the set of frequency domain coefficient matrices to generate a corresponding set of frequency domain semantic feature vectors; determines the similarity of the frequency domain semantic feature vectors corresponding to every two adjacent frames in the set of image frames based on the set of frequency domain semantic feature vectors to generate an inter-frame correlation measurement sequence; generates semantic change probability information corresponding to each image frame using a pre-trained semantic change prediction model based on the inter-frame correlation measurement sequence to obtain a set of semantic change probability information; performs dynamic frame extraction on the set of image frames based on the set of semantic change probability information to generate a set of key frames; and dynamically schedules the computing resources of the large model based on the set of key frames and the semantic change probability information corresponding to the set of key frames to execute differentiated inference tasks.

[0083] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0085] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a frequency domain transformation unit, an extraction and enhancement unit, a determination unit, a generation unit, a frame extraction unit, and a dynamic scheduling unit. The names of these units do not necessarily limit the specific unit itself; for example, the acquisition unit may also be described as "a unit that, in response to a large model receiving a request for an inference task on the original video stream, preprocesses the original video stream to generate a set of image frames to be processed."

[0086] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0087] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for scheduling large-scale computational resources based on frequency domain correlation, comprising: In response to the large model receiving a request for an inference task on the original video stream, the original video stream is preprocessed to generate a set of image frames to be processed. Perform a frequency domain transformation on each image frame in the image frame set to generate a corresponding set of frequency domain coefficient matrices; For each frequency domain coefficient matrix in the frequency domain coefficient matrix set, perform low- and mid-frequency component extraction and semantic enhancement operations to generate a corresponding frequency domain semantic feature vector set; Based on the frequency domain semantic feature vector set, the similarity of the frequency domain semantic feature vectors corresponding to every two adjacent frames in the image frame set is determined to generate an inter-frame correlation measurement sequence. Based on the inter-frame correlation measurement sequence, a pre-trained semantic change prediction model is used to generate semantic change probability information for each image frame, so as to obtain a semantic change probability information set. Based on the semantic change probability information set, a dynamic frame extraction operation is performed on the image frame set to generate a keyframe set; Based on the keyframe set and the semantic change probability information corresponding to the keyframe set, the computing resources of the large model are dynamically scheduled to execute differentiated inference tasks. The step of dynamically scheduling the computational resources of the large model based on the keyframe set and the semantic change probability information corresponding to the keyframe set to execute differentiated inference tasks includes: For the semantic change probability information corresponding to each keyframe in the keyframe set, the following steps are performed: Based on the semantic change probability information, the semantic importance level of the key frame is determined; The corresponding target inference path is determined based on the semantic importance level, wherein the target inference path includes: full-scale deep inference path, lightweight model inference path and cross-frame reuse inference path; In response to the semantic importance level corresponding to the full inference path or the lightweight model inference path, the corresponding large-scale model computing resources are dynamically allocated according to the current available computing resources to execute the corresponding inference task. In response to the cross-frame reuse inference path corresponding to the semantic importance level, the following steps are performed: Semantic feature information of similar frames to the key frame from the historical key frames that have been reasoned out; The semantic feature information is spatiotemporally aligned with the key frame to obtain the aligned feature information; The aligned feature information is input into a pre-trained feature reuse inference model for incremental inference to execute the corresponding inference task.

2. The method according to claim 1, wherein, In response to the large model receiving a request for an inference task on the original video stream, the original video stream is preprocessed to generate a set of image frames to be processed, including: The original video stream is decoded and sampled at a fixed frame rate to generate an initial image frame sequence; Each image frame in the initial image frame sequence is subjected to size normalization and color space normalization to generate an image frame set.

3. The method according to claim 1, wherein, Performing a frequency domain transformation on each image frame in the image frame set to generate a corresponding frequency domain coefficient matrix set includes: For each image frame in the image frame set, perform the following steps: The image frame is segmented to generate multiple regular image blocks; Perform a discrete cosine transform on each of the plurality of regular image blocks to obtain the frequency domain coefficient submatrix corresponding to each image block; Each frequency domain coefficient submatrix is ​​normalized to generate a processed submatrix corresponding to each frequency domain coefficient submatrix; The processed sub-matrices corresponding to the image frame are arranged and combined according to their spatial positions to generate the frequency domain coefficient matrix corresponding to the image frame; Each frequency domain coefficient matrix is ​​defined as a set of frequency domain coefficient matrices.

4. The method according to claim 1, wherein, The step of extracting low- and mid-frequency components and performing semantic enhancement operations on each frequency domain coefficient matrix in the set of frequency domain coefficient matrices to generate a corresponding set of frequency domain semantic feature vectors includes: For each frequency domain coefficient matrix in the set of frequency domain coefficient matrices, perform the following steps: Low-frequency component extraction is performed on the frequency domain coefficient matrix to generate a low-frequency coefficient matrix; The frequency domain coefficient matrix is ​​subjected to mid-frequency band component extraction to generate a mid-frequency coefficient matrix; High-frequency band component extraction is performed on the frequency domain coefficient matrix to generate a high-frequency coefficient matrix, and noise suppression processing is performed on the high-frequency coefficient matrix; Perform semantic enhancement operations on the low-frequency coefficient matrix and the mid-frequency coefficient matrix to obtain enhanced low-frequency components and enhanced mid-frequency components; The enhanced low-frequency component and the enhanced mid-frequency component are fused to obtain a preliminary semantic feature matrix; The preliminary semantic feature matrix is ​​compressed and encoded to generate the frequency domain semantic feature vector corresponding to the frequency domain coefficient matrix; Each frequency domain semantic feature vector is determined as a frequency domain semantic feature vector set.

5. The method according to claim 1, wherein, The step of determining the similarity of frequency domain semantic feature vectors corresponding to every two adjacent frames in the image frame set based on the frequency domain semantic feature vector set, in order to generate an inter-frame correlation measurement sequence, includes: For each pair of adjacent frames corresponding to the frequency domain semantic feature vectors in the aforementioned frequency domain semantic feature vector set, the following steps are performed: Determine the cosine similarity between the semantic feature vectors corresponding to the adjacent frames to generate initial similarity information; Based on the difference in low- and mid-frequency energy distribution of the domain semantic feature vectors corresponding to the adjacent frames, the initial similarity information is corrected to obtain corrected similarity information; The corrected similarity is compared with a preset similarity threshold to obtain the similarity classification result; The corrected similarity information is combined with the similarity classification result to form an inter-frame correlation measure; The inter-frame correlation metrics generated in frame order are arranged into an inter-frame correlation metric sequence.

6. The method according to claim 1, wherein, The step involves generating semantic change probability information for each image frame based on the inter-frame correlation measurement sequence and using a pre-trained semantic change prediction model to obtain a semantic change probability information set, including: For each inter-frame correlation measure in the inter-frame correlation measure sequence, perform the following steps: The inter-frame correlation measures of the current frame and the adjacent frames before and after the current frame are combined into a temporal feature vector to generate the context feature representation of the current frame. The contextual feature representation is input into the semantic change prediction model to generate initial semantic change probability information; Based on the temporal position information of the current frame in the image frame set, the initial semantic change probability information is temporally weighted to generate the adjusted semantic change probability. The adjusted semantic change probability is normalized to generate semantic change probability information corresponding to the current frame. The final semantic change probability information corresponding to each image frame is arranged in frame order to obtain the semantic change probability information set.

7. A large-scale computational resource scheduling device based on frequency domain correlation, comprising: The preprocessing unit is configured to preprocess the original video stream in response to the large model receiving a request for an inference task for the original video stream, in order to generate a set of image frames to be processed. The frequency domain transformation unit is configured to perform a frequency domain transformation on each image frame in the image frame set to generate a corresponding set of frequency domain coefficient matrices; The extraction and enhancement unit is configured to perform low- and mid-frequency component extraction and semantic enhancement operations on each frequency domain coefficient matrix in the set of frequency domain coefficient matrices to generate a corresponding set of frequency domain semantic feature vectors. The determining unit is configured to determine the similarity of the frequency domain semantic feature vectors corresponding to every two adjacent frames in the image frame set based on the frequency domain semantic feature vector set, so as to generate an inter-frame correlation measurement sequence. The generation unit is configured to generate semantic change probability information for each image frame based on the inter-frame correlation measurement sequence and using a pre-trained semantic change prediction model, so as to obtain a semantic change probability information set. The frame extraction unit is configured to perform dynamic frame extraction on the image frame set based on the semantic change probability information set to generate a keyframe set. A dynamic scheduling unit is configured to dynamically schedule the computational resources of the large model based on the keyframe set and the semantic change probability information corresponding to the keyframe set, in order to execute differentiated inference tasks. The dynamic scheduling unit is further configured to include: For the semantic change probability information corresponding to each keyframe in the keyframe set, the following steps are performed: Based on the semantic change probability information, the semantic importance level of the key frame is determined; The corresponding target inference path is determined based on the semantic importance level, wherein the target inference path includes: full-scale deep inference path, lightweight model inference path and cross-frame reuse inference path; In response to the semantic importance level corresponding to the full inference path or the lightweight model inference path, the corresponding large-scale model computing resources are dynamically allocated according to the current available computing resources to execute the corresponding inference task. In response to the cross-frame reuse inference path corresponding to the semantic importance level, the following steps are performed: Semantic feature information of similar frames to the key frame from the historical key frames that have been reasoned out; The semantic feature information is spatiotemporally aligned with the key frame to obtain the aligned feature information; The aligned feature information is input into a pre-trained feature reuse inference model for incremental inference to execute the corresponding inference task.

8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images