Algorithm configuration method and device for video stream, program product and storage medium

Abstract

This application provides a method for configuring video stream algorithms, comprising: extracting visual features, spatiotemporal features, and semantic features of the input video stream; fusing these features to generate a first scene feature of the video stream; and generating a configuration algorithm for the video stream based on the first scene feature and a pre-constructed knowledge graph representing the relationship between scene features and algorithms. The technical solution of this application can effectively solve the problems of high configuration cost, poor scene adaptability, knowledge transfer gaps, and low configuration efficiency of video stream algorithms in smart city scenarios in existing technologies.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and more specifically, to a video stream algorithm configuration method and apparatus, program product, and storage medium. Background Technology

[0002] In the field of video surveillance for smart cities, the introduction of artificial intelligence technology has provided strong support for urban management and public safety.

[0003] Existing technologies have significant shortcomings in video stream algorithm configuration, which limits the efficiency and effectiveness of smart city video analytics. Specifically, existing algorithm configuration methods have the following key problems: (1) High configuration cost: In the early stages of a project, it is necessary to analyze and manually configure algorithms for a large number of video streams (e.g., tens of thousands of video streams), which is not only time-consuming and labor-intensive but also prone to omissions. This reliance on manual methods leads to a sharp increase in configuration costs, especially when the scale of video streams is large, the workload of configuration increases exponentially. (2) Poor scene adaptability: Current algorithm configuration rules are relatively fixed, and once set up, they are difficult to adapt to dynamic changes in the scene. For example, when objects or events within the camera coverage area change (e.g., shop locations are adjusted, trash cans are replaced), existing configuration methods require manual intervention to reconfigure the algorithm, lacking a mechanism to automatically adapt to scene changes, resulting in reduced algorithm matching accuracy and response speed. (3) Knowledge transfer gap: Algorithm configuration logic relies on the experience of specific personnel, which is difficult to standardize and transfer within the platform, resulting in high learning costs for new maintenance personnel and difficulties in maintenance and adjustment in the later stages of the project. Furthermore, as video surveillance scenarios continue to change, the lack of an effective mechanism to update and optimize algorithm configuration rules results in configuration rules lagging behind actual needs. Summary of the Invention

[0004] This application provides a method and apparatus for configuring video stream algorithms, a program product, and a storage medium, to at least solve the problems of high configuration cost, poor scene adaptability, knowledge transfer gaps, and low configuration efficiency of video stream algorithms in smart city scenarios in the prior art.

[0005] According to one embodiment of this application, a video stream algorithm configuration method is provided, including:

[0006] Extract visual, spatiotemporal, and semantic features from the input video stream;

[0007] The visual features, the spatiotemporal features, and the semantic features are fused to generate the first scene features of the video stream;

[0008] Based on the first scene features and a pre-constructed knowledge graph representing the relationship between scene features and algorithms, a configuration algorithm for the video stream is generated.

[0009] In an exemplary embodiment, the configuration algorithm for generating the video stream based on the first scene features and the knowledge graph includes: determining the degree of similarity between the first scene features and the second scene features of the historical video stream according to the association relationship between the scene features and the algorithm; and generating the configuration algorithm for the video stream based on the degree of similarity and the historical configuration algorithm of the historical video stream.

[0010] In an exemplary embodiment, the feature fusion of the visual features, the spatiotemporal features, and the semantic features includes: dimensional alignment of the visual features, the spatiotemporal features, and the semantic features; concatenating the dimensionally aligned features to generate an initial fused feature; inputting the initial fused feature into a pre-trained target model and outputting the first scene feature; wherein the target model is a model obtained by iteratively training the initial model based on a multi-head attention mechanism using the historical video stream.

[0011] In an exemplary embodiment, the training process of the target model includes: acquiring historical initial fusion features by splicing together visual features, spatiotemporal features, and semantic features based on historical video streams; constructing a joint loss function based on the visual features, the spatiotemporal features, and the semantic features; inputting the historical initial fusion features into the initial model, and using the joint loss function to calculate the difference between the scene features output by the initial model and the actual scene features of the historical video stream, thereby obtaining the loss value of the initial model; updating the parameters of the initial model based on the loss value to obtain the target model, wherein the loss value of the target model satisfies a preset threshold.

[0012] In an exemplary embodiment, constructing a joint loss function based on the visual features, the spatiotemporal features, and the semantic features includes: assigning preset weight coefficients to the loss functions corresponding to the visual features, the spatiotemporal features, and the semantic features respectively; and using the sum of the values ​​obtained by multiplying each loss function by its corresponding weight coefficient as the joint loss function.

[0013] In an exemplary embodiment, generating the configuration algorithm for the video stream based on the similarity degree and the historical configuration algorithm of the historical video stream includes: sorting according to the similarity degree and sequentially selecting a set number of historical configuration algorithms corresponding to historical video streams from high to low; filtering the historical configuration algorithms according to the frequency of occurrence of each algorithm in the historical configuration algorithms to generate the configuration algorithm for the video stream.

[0014] In an exemplary embodiment, the configuration algorithm for generating the video stream based on the first scene features and a pre-constructed knowledge graph representing the association between scene features and algorithms further includes: determining a reward function based on the effective trigger rate, computing power utilization rate, and conflict penalty, and optimizing the configuration algorithm for generating the video stream using a deep Q-network; wherein the effective trigger rate represents the proportion of real events triggered by the algorithm, the computing power utilization rate represents the ratio of used video memory to total video memory, and the conflict penalty represents the deduction of points when the exclusion relationship between configuration algorithms is triggered.

[0015] In an exemplary embodiment, after generating the configuration algorithm for the video stream, the method further includes: determining whether there is a resource conflict problem when multiple algorithms are deployed based on the correlation between the scene features and the algorithm; if so, further filtering the configuration algorithm for the video stream based on a preset resource priority ranking to obtain the resource conflict configuration algorithm for the video stream.

[0016] In one exemplary embodiment, the knowledge graph is updated based on the execution results of the configuration algorithm and the federated learning mechanism.

[0017] In an exemplary embodiment, the step of extracting visual features and spatiotemporal features from the input video stream includes: extracting visual features from the video stream based on the SE attention module and a detection layer for small targets, and outputting the visual features; and extracting spatiotemporal features from a continuous number of frames of the video stream using the TimeSformer model, and outputting the spatiotemporal features.

[0018] In one exemplary embodiment, the method further includes: fine-tuning the target model based on a meta-learning fast fine-tuning model to achieve the recognition of specific scene features.

[0019] According to another embodiment of this application, a video stream algorithm configuration device is provided, including a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor. When the first processor executes the first computer program, it performs the following operations: extracting visual features, spatiotemporal features, and semantic features of the input video stream; performing feature fusion on the visual features, the spatiotemporal features, and the semantic features to generate a first scene feature of the video stream; and generating a configuration algorithm for the video stream based on the first scene feature and a pre-constructed knowledge graph representing the relationship between scene features and algorithms.

[0020] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0021] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0022] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0023] By applying the technical solution of this embodiment, the visual, spatiotemporal, and semantic features of the video stream can be extracted to comprehensively and accurately understand the video stream content. These features are then fused to generate the first scene features of the video stream, enhancing the model's ability to understand complex scenes and enabling it to more precisely distinguish different scene types. Finally, based on the first scene features and a pre-built knowledge graph, this method generates a configuration algorithm for the video stream. The knowledge graph stores the relationships between scene features and algorithms, allowing algorithm selection to be based not only on the current scene but also on historical data and experience, thus improving the intelligence level of algorithm configuration. Applying the technical solution of this invention significantly reduces the cost of video stream algorithm configuration in smart city scenarios, improves scene adaptability, solves the problem of knowledge transfer gaps, and greatly improves configuration efficiency, achieving automation and intelligence in algorithm configuration and providing strong technical support for smart city construction and management. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the hardware environment for a video stream algorithm configuration method according to an embodiment of this application;

[0025] Figure 2 This is a flowchart of a video stream algorithm configuration method according to an embodiment of this application;

[0026] Figure 3 This is a flowchart illustrating another video stream algorithm configuration method according to an embodiment of this application;

[0027] Figure 4 This is a schematic flowchart illustrating the training process of a target model according to an embodiment of this application;

[0028] Figure 5 This is a structural block diagram of a video stream algorithm configuration device according to an embodiment of this application. Detailed Implementation

[0029] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0031] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a schematic diagram of the hardware environment for a video stream algorithm configuration method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to a video stream algorithm configuration method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0034] This embodiment provides a method for configuring video stream algorithms. Figure 2 This is a flowchart of a video stream algorithm configuration method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0035] Step S202: Extract the visual features, spatiotemporal features, and semantic features of the input video stream.

[0036] Optionally, the input video stream can be a sequence of consecutive video image frames captured by various cameras, and the input video stream can be a video stream with tens of thousands of streams. Visual features can be information such as objects, colors, and textures extracted from the video images, used to describe the content and scene of the video. Spatiotemporal features can be a combination of time-series information and spatial layout information of the video images, used to capture dynamic changes and object movement trajectories in the video. Semantic features can be information describing the meaning of the video scene extracted through text analysis of camera names or video context.

[0037] Optionally, this method utilizes technologies such as computer vision, natural language processing, and deep learning to extract visual, spatiotemporal, and semantic features of the video stream, thereby comprehensively and accurately understanding the video stream content.

[0038] For example, when a smart city monitoring system receives a video stream from a commercial street, the system first automatically extracts visual features from the video, such as identifying images of trash cans and shops. Then, it extracts spatiotemporal features, such as changes in the opening times of shops and the movement of trash cans. At the same time, it extracts semantic features from the camera names, such as "main road of the commercial street".

[0039] Optionally, visual and spatiotemporal features of the input video stream are extracted, including: visual feature extraction of the video stream based on the SE attention module and a detection layer for small targets, and outputting visual features; spatiotemporal feature extraction of a continuous number of frames of video stream using the TimeSformer model, and outputting spatiotemporal features.

[0040] In this embodiment, dynamic frame extraction and motion detection preprocessing of the input video stream effectively reduce the computational load of subsequent processing while ensuring the preservation of key information. The core technology lies in utilizing an improved YOLOv5s model, embedding an attention (Squeeze-and-Excitation, SE) module after its backbone network, which significantly improves the accuracy of small object detection in videos, particularly excelling in handling small but important scene elements such as trash cans and shop signs. Furthermore, a dedicated detection layer for small objects is added, further enhancing the model's sensitivity and recognition ability for small-sized objects.

[0041] For example, an SE (Squeeze-and-Excitation) module is inserted after layer C3 of the backbone network. This module recalibrates the channel weights of the feature maps through a channel attention mechanism, enhancing the model's focus on important channel features and suppressing irrelevant channel features. The computation flow of the SE module is as follows:

[0042]

[0043] Where u is the output feature map of layer C3, GAP is global average pooling, W1 and W2 are the weights of the fully connected layer, δ is the ReLU activation function, and ⊙ is element-wise multiplication.

[0044] The original three detection heads of YOLOv5s are expanded to four, and a detection layer is added for small targets (area less than 32×32 pixels). The output bounding box coordinates (x, y, w, h), confidence score c, and category label (e.g., "trash can" or "shop") are output.

[0045] Optionally, the TimeSformer model is used to process five consecutive frames of images in the video stream. This not only captures visual features but also effectively extracts spatiotemporal features, including changes in object position and dynamic behavior, by modeling spatial and temporal attention mechanisms. This provides comprehensive information for understanding and predicting complex scenes in video streams.

[0046] For example, to capture dynamic changes in a video sequence (such as the opening status of a shop or the movement of a trash can), the TimeSformer model is used to process five consecutive frames to model the spatiotemporal relationship. The image size of each frame is adjusted to 224×224, and the features of each frame are extracted using the VisionTransformer. Then, the features of the five frames are concatenated into a spatiotemporal sequence.

[0047] Optionally, to combine the camera name and visual content, the CLIP (Contrastive Language-Image Pretraining) model is used to achieve image-text semantic alignment. Image features are extracted using ResNet-50, and the camera name text (e.g., "Business Street-001") is input into BERT to output text features.

[0048] Through the above processing flow, this embodiment can automatically identify and characterize scene features in video streams, laying a solid data foundation for subsequent dynamic algorithm recommendations and knowledge base updates, and greatly improving the efficiency and accuracy of scene analysis for tens of thousands of video streams. Of course, in other possible implementation paths not described in detail in this embodiment, the accuracy and speed of feature extraction can also be further improved by adjusting model parameters or optimizing the algorithm structure to adapt to broader or more challenging application scenarios.

[0049] Step S204: Perform feature fusion on visual features, spatiotemporal features, and semantic features to generate the first scene features of the video stream.

[0050] In this embodiment, the first scene feature can be a scene feature representation obtained after feature fusion processing.

[0051] Optionally, feature fusion is performed on visual features, spatiotemporal features, and semantic features, including: dimensional alignment of visual features, spatiotemporal features, and semantic features; concatenation of dimensionally aligned features to generate initial fused features; inputting the initial fused features into a pre-trained target model to output first scene features; wherein the target model is a model obtained by iteratively training the initial model based on a multi-head attention mechanism using historical video streams.

[0052] In this embodiment, multimodal information from visual features, spatiotemporal features, and semantic features is deeply fused to construct a unified scene understanding model. First, dimensional alignment technology ensures that the three types of features are in the same vector space. Then, these features are concatenated to form an initial fused feature encompassing visual details, dynamic changes, and semantic information. Subsequently, this initial fused feature is input into a pre-trained target model. This model, based on historical video stream data, undergoes iterative training through a multi-head attention mechanism, capturing the interactions and correlations between features, and ultimately outputting the first scene feature, achieving accurate scene recognition. This technical solution not only effectively integrates multi-source information, improving the accuracy and comprehensiveness of scene understanding, but also significantly enhances the system's adaptability and generalization performance through the model's deep learning capabilities. This ensures that when facing complex and ever-changing smart city scenarios, it can efficiently generate and recommend optimal algorithm combinations, while optimizing resource allocation, reducing computational waste, and achieving a dual improvement in real-time performance and decision support.

[0053] Optionally, the final output of the first scene features can be achieved through a feedforward network and layer normalization.

[0054] Step S206: Based on the first scene features and the pre-built knowledge graph representing the relationship between scene features and algorithms, generate a configuration algorithm for the video stream.

[0055] Optionally, a knowledge graph can be a data structure used to describe the relationships between entities; in this solution, it is used to store and query the association rules between scenes and algorithms. Configuring the algorithm can be a process of automatically selecting appropriate AI algorithms for real-time detection and recognition based on the scene characteristics of the video stream.

[0056] In this embodiment, based on the first scene features and the pre-built knowledge graph, the method generates a configuration algorithm for the video stream. The knowledge graph stores the relationship between scene features and algorithms, so that the selection of algorithms is not only based on the current scene, but also refers to historical data and experience, thereby improving the intelligence level of algorithm configuration.

[0057] For example, the knowledge graph construction process is as follows: Knowledge graph entities are defined, including scene entities, algorithm entities, and relationships. Scene entities: Represent the scene attributes of the video stream, including the following sub-entities: Spatial attributes: camera ID, geographical location (latitude and longitude), coverage area (e.g., "main road of a commercial street"); Temporal attributes: active time periods (e.g., "daily 18:00-22:00"); Object attributes: list of detected objects (e.g., {trash cans: 2, shops: 5, pedestrian traffic: high}); Event attributes: historical trigger events (e.g., "garbage overflow alarm 10 times / week"). Algorithm entities: Represent available AI algorithms, including the following sub-entities: Functional tags: such as "garbage overflow detection," "street vendor identification," "license plate recognition"; Resource requirements: memory usage (e.g., ≥4GB), computing power requirements (e.g., single-path inference time ≤800ms); Performance metrics: historical accuracy (e.g., 92%), false alarm rate (e.g., 0.5 times / day). Association Relationship: Define the matching rules between the scene and the algorithm, including: Strong association: When object A exists in the scene, algorithm B must be deployed (e.g., "≥1 trash can → overflow detection"); Weak association: When object A exists in the scene, it is recommended to deploy algorithm B (e.g., "≥3 shops → street vendor identification"); Exclusion relationship: Algorithm B and algorithm C cannot be deployed at the same time (e.g., "high memory algorithm A + high memory algorithm B → insufficient memory").

[0058] The Neo4j graph database is used for storage, where nodes represent entities (scenes / algorithms) and edges represent relationships. For example: Scene node S (attributes: {geographic location: commercial street-001, objects: trash cans: 2, shops: 5}); Algorithm node A (attributes: {functional tags: overflow detection, memory requirement: 4GB}); Edge S→A (relationship: strong association, confidence 0.95).

[0059] Optionally, a configuration algorithm for generating video streams based on first scene features and knowledge graphs, such as... Figure 3 As shown, the process includes the following steps:

[0060] Step S302 determines the similarity between the first scene feature and the second scene feature of the historical video stream based on the correlation between scene features and the algorithm.

[0061] Step S304: Generate a configuration algorithm for the video stream based on similarity and historical configuration algorithms of historical video streams.

[0062] In this embodiment, the relationship between scene features and algorithms can be a mapping between scene features of video streams (such as visual features indicating the presence of trash cans or shops) and the AI ​​algorithms to be deployed (such as garbage overflow detection or street vendor identification) in smart city scene analysis. This relationship defines which algorithms should be recommended and deployed when specific scene features are detected. Historical video streams can refer to video stream data that has been processed and analyzed in the past in the algorithm automatic configuration system. This data is used to build and train a knowledge graph linking scene features and algorithms, accumulating historical configuration experience. Second scene features can be scene features corresponding to historical video streams. Similarity can be a metric of the similarity between current scene features and historical scene features in a dynamic algorithm recommendation engine. Historical configuration algorithms can refer to algorithms that have been deployed based on scene features in historical video stream processing. These algorithms are stored in the knowledge graph as historical configuration experience for algorithm recommendation in future scenarios. Video stream configuration algorithms refer to a combination of AI algorithms generated by the dynamic algorithm recommendation engine and deployed to the video analysis platform based on the scene feature analysis results of the current video stream, used for real-time monitoring and analysis of video content.

[0063] In this embodiment, when generating a configuration algorithm for a video stream based on the first scene features and the knowledge graph, the system first calculates the similarity between the first scene features of the current video stream and the second scene features of historical video streams stored in the knowledge graph to determine their proximity in terms of scene elements, spatiotemporal dynamics, and semantic context. Then, based on this similarity and the historical configuration algorithms corresponding to the historical video streams, a recommended list of configuration algorithms for the current video stream is generated using collaborative filtering technology. This process utilizes the scene-algorithm association knowledge contained in the knowledge graph to optimize the accuracy and real-time performance of algorithm configuration, ensuring efficient matching and execution of algorithms in different scenarios. In practical implementation, the system can quickly identify and adapt to scene changes, such as automatically adjusting the algorithm configuration when detecting the emergence of new objects or event patterns, reducing the need for manual intervention and improving the overall efficiency of smart city scene monitoring. Of course, this embodiment also considers matching strategies for other scene features and algorithm configurations, such as dynamically optimizing recommendation strategies through reinforcement learning to cope with more complex and ever-changing real-world environments, further improving the intelligence level of algorithm deployment and resource utilization efficiency.

[0064] Optionally, the configuration algorithm for generating the video stream based on the similarity and historical configuration algorithms of historical video streams includes: sorting according to the similarity and selecting a set number of historical configuration algorithms corresponding to historical video streams in descending order; filtering the historical configuration algorithms according to the frequency of occurrence of each algorithm in the historical configuration algorithms to generate the configuration algorithm for the video stream.

[0065] In this embodiment, a configuration algorithm for the video stream is generated based on similarity and historical configuration algorithms of historical video streams. The specific implementation process includes the following steps: First, by calculating the scene features of the current video stream and the scene features of historical video streams, the algorithms are sorted according to their similarity and selected sequentially from high to low. This serves as an initial recommendation for collaborative filtering. Subsequently, the historical configuration algorithms are filtered based on their frequency of occurrence to generate the configuration algorithm for the video stream, ensuring that the recommended algorithm combination has been verified as effective in historical scenarios. This process can dynamically combine scene changes and historical configuration experience to achieve intelligent and adaptive algorithm recommendations, effectively improving the accuracy and efficiency of algorithm deployment and reducing resource waste and processing delays caused by improper algorithm configuration.

[0066] For example, in a smart city, the method of empowering a video platform with artificial intelligence algorithms is as follows: Assume a county-level city with 10,000 video streams. Due to the high cost of GPU computing resources, the project only purchased 5 algorithm inference servers equipped with 8 Nvidia T4 cards each. Existing technologies for video frame extraction, server allocation, and algorithm matching implement the following: Under a centralized processing architecture in a government cloud center, 2 dedicated decoding and frame extraction servers and 3 dedicated algorithm inference servers are used. After the video is framed into images, it is sent to the algorithm for detection and recognition. Depending on the dynamic resource scheduling optimization strategy of each server, approximately 1200-1920 concurrent inference streams can be processed. If each video stream is framed and distributed to all algorithm services for inference in a polling manner, it will lead to increased processing latency and wasted computing resources. Alternatively, based on the specific application scenario of the video stream, appropriate algorithm services can be matched according to human experience. For example, if a camera is positioned where there are both trash cans and many street shops, an algorithm for overflowing trash is needed to promptly notify the sanitation department for cleaning. Simultaneously, an algorithm for street vendors is needed to report the incident to the urban management department. This algorithmic matching method involves a significant workload in the early stages of a project. In this embodiment, the first scene features are directly used as the input S to the dynamic algorithm recommendation engine. Historical scene feature vectors are stored in a knowledge graph and consist of feature extraction results from historical video streams. Based on historical configuration data, an initial recommendation list is generated through item-based collaborative filtering.

[0067] Calculate the similarity between scene S and historical scene S':

[0068]

[0069] Where O is the set of objects, I(S,o) represents the number of objects o in scene S (1 if they exist, 0 otherwise), and I(S',o) represents the number of objects o in scene S'. Assign weights to object o (e.g., trash can weight 0.7, shop weight 0.5). Calculate the similarity between the features of the first scene and the features of the second scene, select the top K historical scenes with the highest similarity, and statistically analyze the algorithms frequently deployed in these scenes to create an initial recommendation list (e.g., Top 5 algorithm).

[0070] By applying the technical solution of this embodiment, the visual, spatiotemporal, and semantic features of the video stream can be extracted to comprehensively and accurately understand the video stream content. These features are then fused to generate the first scene features of the video stream, enhancing the model's ability to understand complex scenes and enabling it to more precisely distinguish different scene types. Finally, based on the first scene features and a pre-built knowledge graph, this method generates a configuration algorithm for the video stream. The knowledge graph stores the relationships between scene features and algorithms, allowing algorithm selection to be based not only on the current scene but also on historical data and experience, thus improving the intelligence level of algorithm configuration. Applying the technical solution of this invention significantly reduces the cost of video stream algorithm configuration in smart city scenarios, improves scene adaptability, solves the problem of knowledge transfer gaps, and greatly improves configuration efficiency, achieving automation and intelligence in algorithm configuration and providing strong technical support for smart city construction and management.

[0071] In one exemplary embodiment, Figure 4 This is a flowchart illustrating the training process of a target model according to an embodiment of this application, as shown below. Figure 4 As shown, the training process of the above target model includes:

[0072] Step S402: Obtain the initial historical fusion features by splicing together the visual features, spatiotemporal features and semantic features of the historical video stream.

[0073] Step S404: Construct a joint loss function based on visual features, spatiotemporal features and semantic features.

[0074] Step S406: Input the historical initial fusion features into the initial model, and use the joint loss function to calculate the difference between the scene features output by the initial model and the actual scene features of the historical video stream to obtain the loss value of the initial model.

[0075] Step S408: Update the parameters of the initial model based on the loss value to obtain the target model, wherein the loss value of the target model meets a preset threshold.

[0076] In this embodiment, the optimization of the target model's training process hinges on integrating visual, spatiotemporal, and semantic features from historical video streams to form initial historical fusion features. These initial features are then used to guide model parameter updates through a designed joint loss function. Specifically, multimodal features extracted from historical video streams are acquired and concatenated to construct initial historical fusion features, which serve as the input basis for model training. The joint loss function is designed to measure the difference between model predictions and actual scene features. By inputting the initial historical fusion features into the initial model, the deviation between the output scene features and the actual scene features from the historical video stream is calculated, yielding the initial model's loss value. During training, model parameters are iteratively updated based on this loss value until the target model's loss value meets a preset threshold. This training process not only fully considers the multidimensional nature of scene features, enhancing the model's generalization ability, but also effectively guides model learning through a refined loss function design, improving the accuracy and efficiency of algorithm recommendations. In subsequent implementations, the target model can more accurately identify and match scene features and algorithms, achieving rational resource allocation, reducing false positives and false negatives, and optimizing the video analysis and response mechanism of smart cities.

[0077] Optionally, a joint loss function based on visual features, spatiotemporal features, and semantic features is constructed, including: assigning preset weight coefficients to the loss functions corresponding to visual features, spatiotemporal features, and semantic features respectively; and using the sum of the values ​​of each loss function multiplied by its corresponding weight coefficient as the joint loss function.

[0078] In this embodiment, to ensure the consistency of features across modalities and the accuracy of scene recognition, a joint loss function is designed and adopted. This function comprises three components: visual detection loss, spatiotemporal contrast loss, and semantic alignment loss. By assigning preset weight coefficients to each feature loss, a balanced optimization of visual, spatiotemporal, and semantic features is achieved. Specifically, in the loss function, the visual detection loss measures the accuracy of target object detection; the spatiotemporal contrast loss, through a contrastive learning method, ensures the similarity between spatiotemporal features of similar scenes and the differences between different scenes; the semantic alignment loss utilizes the CLIP model to guarantee the semantic consistency between camera names and visual content. The three loss functions are multiplied by their respective weight coefficients and then summed to obtain the joint loss function value. This design enables the system to comprehensively consider and optimize the multi-dimensional representation of video stream scene features in an end-to-end manner, thereby significantly improving the accuracy and real-time performance of scene recognition when processing large-scale video streams. It effectively balances algorithm performance and resource consumption, achieves accurate matching between scene features and AI algorithms, and reduces the overall system configuration cost and mismatch rate. Furthermore, this embodiment also takes into account the long-tail demand in open scenarios, and enhances the system's ability to identify and configure unknown scenario types through federated learning and unsupervised clustering techniques.

[0079] The visual detection loss uses CIoU (Complete Intersection over Union) loss as the bounding box regression loss, combined with classification loss and confidence loss:

[0080]

[0081] in, These are the weighting coefficients.

[0082] The spatiotemporal contrast loss function is:

[0083]

[0084] in, =0.5 is the marginal parameter, (i,j) is the positive sample pair, and (i,k) is the negative sample pair. The joint loss is the weighted sum of the components.

[0085] Optionally, the configuration algorithm for generating video streams based on the first scene features and a pre-built knowledge graph representing the relationship between scene features and algorithms also includes: determining the reward function based on the effective trigger rate, computing power utilization rate, and conflict penalty, and optimizing the configuration algorithm for generating video streams using a deep Q-network; wherein, the effective trigger rate represents the proportion of real events triggered by the algorithm, the computing power utilization rate represents the ratio of used video memory to total video memory, and the conflict penalty represents the deduction of points when the exclusion relationship between configuration algorithms is triggered.

[0086] In this embodiment, the process of generating a dynamically configured algorithm based on the first scene features of the video stream and a pre-constructed knowledge graph representing the relationship between scene features and algorithms further incorporates a strategy optimization mechanism using a deep Q-network. Specifically, the reward function is determined through metrics such as effective trigger rate, computing power utilization, and conflict penalty. The effective trigger rate measures the algorithm's ability to identify real events, computing power utilization reflects the effective use of resources, and conflict penalty aims to avoid potential exclusionary relationships between algorithms. This enables the dynamic algorithm recommendation engine to intelligently select the optimal algorithm combination based on real-time computing resources and scene features, ensuring efficient algorithm operation and accurate event response while avoiding resource waste and algorithm conflicts. Employing a deep Q-network to dynamically adjust the algorithm recommendation strategy not only improves the deployment efficiency and matching accuracy of the algorithm but also automatically adjusts the recommendation list based on scene changes and algorithm execution effects. This achieves intelligent and automated algorithm configuration, significantly enhancing system flexibility and resource optimization, and providing a more efficient and accurate algorithm deployment solution for video analysis in smart cities.

[0087] Optionally, after generating the configuration algorithm for the video stream, the method further includes: determining whether there is a resource conflict problem when deploying multiple algorithms based on the correlation between scene characteristics and algorithms; if so, further filtering the configuration algorithm for the video stream based on a preset resource priority ranking to obtain the resource conflict configuration algorithm for the video stream.

[0088] In this embodiment, after generating the configuration algorithm for the video stream, the system actively detects whether there are resource conflicts when deploying multiple algorithms in the relationship between scene features and algorithms. When a resource conflict is detected, the system further filters the configuration algorithms for the video stream according to a preset resource priority ranking, aiming to ensure that emergency algorithms (such as fireworks detection) are prioritized, followed by regulatory algorithms (such as street vendor identification), and finally statistical algorithms (such as passenger flow analysis). This mechanism ensures that, under limited computing power, the optimization of algorithm combinations not only considers the urgency of scene requirements but also takes into account the rationality of resource allocation, effectively avoiding service interruptions or performance degradation caused by resource contention between algorithms, and achieving efficient and stable algorithm deployment. Especially when processing multiple concurrent video streams, this conflict resolution strategy can significantly improve the overall responsiveness and resource utilization efficiency of the system, ensuring the real-time performance and accuracy of critical applications.

[0089] Optionally, the knowledge graph can be updated based on the execution results of the configuration algorithm and the federated learning mechanism.

[0090] In this embodiment, updating the knowledge graph based on the configuration algorithm execution results and the federated learning mechanism further improves the system's adaptive capabilities and generalization performance. Specifically, the system dynamically adjusts the scene-algorithm association rules and confidence levels in the knowledge graph by collecting algorithm execution data and manual correction feedback in real time and utilizing online learning, ensuring the accuracy and timeliness of the recommendation strategy. In cross-county scenarios, the model parameters generated locally by each node are aggregated to the central server through encrypted federated learning, generating a globally optimized model, and then redistributed to local nodes. This achieves model fine-tuning and optimization under multi-domain data collaboration, enhances the system's ability to identify and configure long-tail scenarios, reduces the number of samples required for adapting to new scenarios, and improves the overall efficiency and accuracy of algorithm deployment. Through this mechanism, the knowledge base can continuously evolve, better adapt to scenario changes, reduce configuration errors, and improve the rational allocation and utilization of computing resources.

[0091] Optionally, cluster the first scene features based on preset clustering parameters to identify unlabeled scene features.

[0092] In this embodiment, an unsupervised clustering algorithm is used to automatically discover dense regions in the feature space, including unlabeled scene categories, based on preset clustering parameters such as neighborhood radius and minimum sample size for the first scene features. This process can effectively capture scene changes in the video stream that have not yet been clearly defined or labeled, such as the formation of temporary vendor areas or the sudden occurrence of specific events. By identifying these unlabeled scene features, the training data of the target model is annotated and optimized, enabling the target model to quickly adapt to unknown or changing environments. The implementation effect is reflected in the fact that for tens of thousands of video streams, the system can automatically discover about 15% of the unlabeled scenes, and the automatic configuration time of a single video stream is shortened to less than 10 seconds, significantly improving the overall configuration efficiency and accuracy.

[0093] Optionally, the target model can be fine-tuned based on meta-learning to achieve the recognition of features in a specific scene.

[0094] In this embodiment, when there are scenarios with unique attributes, such as trash cans of a specific style in a county town or localized shop types, this system adopts a meta-learning framework, which only requires a small amount of sample data (such as 10-50 labeled images) to complete the adaptive fine-tuning of the model in a short time, significantly reducing the algorithm adaptation cost and time consumption for new scenarios.

[0095] It should be noted that, through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described above in the various embodiments of this application.

[0096] This embodiment also provides a video stream algorithm configuration device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0097] Figure 5 This is a structural block diagram of a video stream algorithm configuration device according to an embodiment of this application, such as... Figure 5 As shown, the device includes a first memory 52, a first processor 54, and a first computer program 5202 stored in the first memory 52 and executable on the first processor 54. When the first processor 54 executes the first computer program 5202, it performs the following operations: extracting visual features, spatiotemporal features, and semantic features of the input video stream; performing feature fusion on the visual features, the spatiotemporal features, and the semantic features to generate a first scene feature of the video stream; and generating a configuration algorithm for the video stream based on the first scene feature and a pre-built knowledge graph representing the relationship between scene features and algorithms.

[0098] When the first processor 54 executes the first computer program 5202, it also performs the following operations: determining the degree of similarity between the first scene feature and the second scene feature of the historical video stream based on the relationship between the scene feature and the algorithm; and generating the configuration algorithm of the video stream based on the degree of similarity and the historical configuration algorithm of the historical video stream.

[0099] When the first processor 54 executes the first computer program 5202, it also performs the following operations: dimensional alignment of the visual features, the spatiotemporal features, and the semantic features; concatenation of the dimensionally aligned features to generate an initial fusion feature; inputting the initial fusion feature into a pre-trained target model and outputting the first scene feature; wherein the target model is a model obtained by iteratively training the initial model based on the multi-head attention mechanism using the historical video stream.

[0100] When the first processor 54 executes the first computer program 5202, it also performs the following operations: acquiring historical initial fusion features spliced ​​together from visual features, spatiotemporal features, and semantic features of historical video streams; constructing a joint loss function based on the visual features, spatiotemporal features, and semantic features; inputting the historical initial fusion features into the initial model, using the joint loss function to calculate the difference between the scene features output by the initial model and the actual scene features of the historical video stream, to obtain the loss value of the initial model; updating the parameters of the initial model based on the loss value to obtain the target model, wherein the loss value of the target model satisfies a preset threshold.

[0101] When the first processor 54 executes the first computer program 5202, it also performs the following operations: assigning preset weight coefficients to the loss functions corresponding to the visual features, the spatiotemporal features, and the semantic features respectively; and using the sum of the loss functions multiplied by their respective weight coefficients as the joint loss function.

[0102] When the first processor 54 executes the first computer program 5202, it also performs the following operations: sorting according to the similarity and selecting a set number of historical configuration algorithms corresponding to historical video streams from high to low; filtering the historical configuration algorithms according to the number of times each algorithm appears in the historical configuration algorithms, and generating the configuration algorithm for the video stream.

[0103] When the first processor 54 executes the first computer program 5202, it also performs the following operations: determining the reward function based on the effective trigger rate, computing power utilization rate, and conflict penalty, and optimizing the configuration algorithm for generating the video stream using a deep Q-network; wherein, the effective trigger rate represents the proportion of real events triggered by the algorithm, the computing power utilization rate represents the ratio of used video memory to total video memory, and the conflict penalty represents the deduction of points when the exclusion relationship between the configuration algorithms is triggered.

[0104] When the first processor 54 executes the first computer program 5202, it also performs the following operations: determining whether there is a resource conflict problem when deploying multiple algorithms based on the correlation between the scene features and the algorithm; if so, further filtering the configuration algorithm of the video stream based on the preset resource priority sorting to obtain the resource conflict configuration algorithm of the video stream.

[0105] When the first processor 54 executes the first computer program 5202, it also performs the following operation: updating the knowledge graph based on the execution result of the configuration algorithm and the federated learning mechanism.

[0106] When the first processor 54 executes the first computer program 5202, it also performs the following operations: extracting visual features from the video stream based on the SE attention module and the detection layer for small targets, and outputting the visual features; and extracting spatiotemporal features from a continuous number of frames of the video stream using the TimeSformer model, and outputting the spatiotemporal features.

[0107] When the first processor 54 executes the first computer program 5202, it also performs the following operation: fine-tuning the target model based on meta-learning fast fine-tuning model to achieve the recognition of features of a specific scene.

[0108] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0109] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0110] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0111] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0112] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0113] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0114] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0115] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0116] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0117] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for configuring an algorithm for a video stream, characterized in that, The method includes: Extract visual, spatiotemporal, and semantic features from the input video stream; The visual features, the spatiotemporal features, and the semantic features are fused to generate the first scene features of the video stream; Based on the first scene features and a pre-constructed knowledge graph representing the relationship between scene features and algorithms, a configuration algorithm for the video stream is generated.

2. The method according to claim 1, characterized in that, The configuration algorithm for generating the video stream based on the first scene features and the knowledge graph includes: The similarity between the first scene feature and the second scene feature of the historical video stream is determined based on the correlation between the scene features and the algorithm. The configuration algorithm for the video stream is generated based on the similarity and the historical configuration algorithm of the historical video stream.

3. The method according to claim 1, characterized in that, The feature fusion of the visual features, the spatiotemporal features, and the semantic features includes: The visual features, spatiotemporal features, and semantic features are dimensionally aligned, and the dimensionally aligned features are concatenated to generate an initial fused feature. The initial fusion features are input into the pre-trained target model, and the first scene features are output; wherein, the target model is a model obtained by iteratively training the initial model based on the multi-head attention mechanism using the historical video stream.

4. The method according to claim 3, characterized in that, The training process of the target model includes: The initial historical fusion features are obtained by splicing together visual, spatiotemporal, and semantic features based on historical video streams; Construct a joint loss function based on the visual features, the spatiotemporal features, and the semantic features; The historical initial fusion features are input into the initial model, and the difference between the scene features output by the initial model and the actual scene features of the historical video stream is calculated using the joint loss function to obtain the loss value of the initial model. The parameters of the initial model are updated based on the loss value to obtain the target model, wherein the loss value of the target model satisfies a preset threshold.

5. The method according to claim 4, characterized in that, The construction of the joint loss function based on the visual features, the spatiotemporal features, and the semantic features includes: The loss functions corresponding to the visual features, the spatiotemporal features, and the semantic features are each assigned a preset weight coefficient; The sum of the products of each loss function multiplied by its corresponding weight coefficient is used as the joint loss function.

6. The method according to claim 2, characterized in that, The configuration algorithm for generating the video stream based on the similarity and the historical configuration algorithm of the historical video stream includes: Based on the similarity, the historical configuration algorithms corresponding to a set number of historical video streams are selected from high to low. The historical configuration algorithms are filtered based on the frequency of each algorithm in the historical configuration algorithms to generate the configuration algorithm for the video stream.

7. The method according to claim 1, characterized in that, The configuration algorithm for generating the video stream based on the first scene features and a pre-constructed knowledge graph representing the association between scene features and algorithms further includes: The reward function is determined based on the effective trigger rate, computing power utilization rate, and conflict penalty. A deep Q-network is used to optimize the configuration algorithm for generating the video stream. The effective trigger rate represents the proportion of real events triggered by the algorithm, the computing power utilization rate represents the ratio of used video memory to total video memory, and the conflict penalty represents the deduction of points when the exclusion relationship between the configuration algorithms is triggered.

8. The method according to claim 1, characterized in that, Following the configuration algorithm for generating the video stream, the method further includes: Based on the correlation between the scenario characteristics and the algorithm, determine whether there is a resource conflict problem when deploying multiple algorithms; If such an algorithm exists, the configuration algorithm for the video stream is further filtered based on the preset resource priority sorting to obtain the resource conflict configuration algorithm for the video stream.

9. The method according to claim 1, characterized in that, The extraction of visual and spatiotemporal features from the input video stream includes: Visual features are extracted from the video stream based on the SE attention module and a detection layer for small targets, and the visual features are output. The TimeSformer model is used to extract spatiotemporal features from a continuous number of frames in the video stream, and the spatiotemporal features are output.

10. The method according to claim 1, characterized in that, Also includes: The knowledge graph is updated based on the execution results of the configuration algorithm and the federated learning mechanism.

11. The method according to claim 3, characterized in that, Also includes: The target model is fine-tuned based on meta-learning to achieve the recognition of features in specific scenarios.

12. A video stream algorithm configuration device, characterized in that, It includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor. When the first processor executes the first computer program, it performs the following operations: Extract visual, spatiotemporal, and semantic features from the input video stream; The visual features, the spatiotemporal features, and the semantic features are fused to generate the first scene features of the video stream; Based on the first scene features and a pre-constructed knowledge graph representing the relationship between scene features and algorithms, a configuration algorithm for the video stream is generated.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 11.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 11.

Images