Key frame generation method and device based on multi-agent cooperation, and equipment

Abstract

This disclosure provides a keyframe generation method based on multi-agent collaboration, comprising: preprocessing raw video data to obtain standardized frame data; performing fusion processing on multi-dimensional features extracted from the standardized frame data to generate a unified state vector; using multiple independent agents to perform feature analysis on corresponding dimensions based on the unified state vector to output keyframe probabilities for corresponding dimensions and uncertainties corresponding to the keyframe probabilities; performing weighted fusion on the keyframe probabilities to obtain fusion probability and comprehensive uncertainty, wherein the weights in the weighted fusion are based on the uncertainty; and performing trigger judgment based on the fusion probability and comprehensive uncertainty to generate a keyframe trigger signal to generate a keyframe.

Description

Technical Field

[0001] This disclosure relates to the field of storage technology, specifically to a method and apparatus for generating keyframes based on multi-agent collaboration, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the rapid development of applications such as urban security, traffic management, park monitoring, and industrial production supervision, video surveillance systems have been deployed on a large scale. Surveillance video is typically a long-term, continuous video stream, resulting in massive amounts of data and high costs for storage, transmission, and subsequent analysis.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a method and apparatus for generating keyframes based on multi-agent collaboration, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a keyframe generation method based on multi-agent collaboration is provided, comprising: performing preprocessing on raw video data to obtain standardized frame data; performing fusion processing on multi-dimensional features extracted from the standardized frame data to generate a unified state vector; using multiple independent agents to perform feature analysis on corresponding dimensions based on the unified state vector to output keyframe probabilities for corresponding dimensions and uncertainties corresponding to the keyframe probabilities; performing weighted fusion on the keyframe probabilities to obtain fusion probability and comprehensive uncertainty, wherein the weights in the weighted fusion are based on the uncertainty; and performing trigger judgment based on the fusion probability and comprehensive uncertainty to generate a keyframe trigger signal to generate a keyframe.

[0006] According to one aspect of this disclosure, a keyframe generation device based on multi-agent collaboration is provided, comprising: a video input module for preprocessing raw video data to obtain standardized frame data; a feature encoding module, communicatively connected to the video input module, for performing fusion processing on multi-dimensional features extracted from the standardized frame data to generate a unified state vector; a multi-agent collaborative decision-making module, communicatively connected to the feature encoding module, for performing feature analysis of corresponding dimensions based on the unified state vector by multiple independent agents to output keyframe probabilities of corresponding dimensions and uncertainties corresponding to the keyframe probabilities; an uncertainty fusion module, communicatively connected to the multi-agent collaborative decision-making module, for performing weighted fusion of keyframe probabilities to obtain fusion probability and comprehensive uncertainty, wherein the weights in the weighted fusion are based on uncertainty; and a joint triggering decision-making module, communicatively connected to the uncertainty fusion module, for performing trigger judgment based on the fusion probability and comprehensive uncertainty to generate a keyframe trigger signal to generate a keyframe.

[0007] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described above.

[0008] According to another aspect of this disclosure, a computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above-described method when executed by a processor.

[0009] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0010] By utilizing one or more embodiments provided in this disclosure, multi-agent multi-dimensional collaborative decision-making can cover all-dimensional video features. Combined with uncertainty-driven dynamic weighted fusion and joint triggering mechanisms, the accuracy of key frame determination in complex scenes can be effectively improved, the risk of missed detection and false detection can be reduced, and the differentiated needs of dynamic monitoring scenarios can be adapted.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 This is a schematic diagram of the architecture of a keyframe generation system 10 based on multi-agent collaboration in some embodiments of this disclosure.

[0014] Figure 2 A schematic block diagram of a video input module 200 according to some embodiments of the present disclosure is shown.

[0015] Figure 3 A schematic block diagram of a feature encoding module 300 according to some embodiments of the present disclosure is shown.

[0016] Figure 4 A schematic block diagram of a multi-agent cooperative decision-making module 400 according to some embodiments of the present disclosure is shown.

[0017] Figure 5 A schematic block diagram of an uncertainty fusion module 500 according to some embodiments of the present disclosure is shown.

[0018] Figure 6 A schematic block diagram of a joint triggering decision module 600 according to some embodiments of the present disclosure is shown.

[0019] Figure 7 A schematic block diagram of a self-reflective closed-loop module 700 according to some embodiments of the present disclosure is shown.

[0020] Figure 8 A schematic block diagram of a memory update module 800 according to some embodiments of the present disclosure is shown.

[0021] Figure 9 A flowchart illustrating a keyframe generation method 900 based on multi-agent collaboration provided in an embodiment of this disclosure is shown.

[0022] Figure 10 A schematic block diagram of a keyframe generation apparatus 1000 based on multi-agent collaboration provided in an embodiment of this disclosure is shown.

[0023] Figure 11 A structural block diagram of an exemplary electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0026] The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0027] It should be noted that, in any part of this disclosure involving the collection, storage, use, transmission, and processing of data, each stage strictly adheres to the laws, regulations, industry standards, and regulatory requirements of the data source, usage location, and relevant countries and regions to ensure the legality and compliance of data activities. In the collection stage, the purpose, method, and scope of collection are clearly communicated to the data subject in a prominent manner. Collection is conducted only after obtaining the data subject's legal authorization, ensuring that the collection process follows the "minimum necessary" principle and does not exceed the scope of data collection. In the storage stage, storage periods are limited, and data is promptly deleted or anonymized / encrypted after the storage purpose is achieved. In the usage stage, a strict data security protection mechanism is implemented, using field-level desensitization technology and processing the original data according to preset desensitization rules. For different types of data, multiple desensitization strategies, such as data generalization, data anonymization, and data encryption, are employed to effectively mitigate the risk of sensitive information leakage and ensure that all data used is securely processed and desensitized, comprehensively protecting the rights and interests of data subjects and data security. In the transmission and processing stages, the confidentiality and security of data are ensured during transmission and processing.

[0028] As mentioned earlier, with the rapid development of application scenarios such as urban security, traffic management, park monitoring, and industrial production supervision, video surveillance systems have been deployed on a large scale. The long-term continuous video streams collected by such systems have the characteristics of huge data volume, which brings high cost pressure to the storage, transmission, and subsequent intelligent analysis of video data. Therefore, key frame generation technology, as a key technology to reduce redundant video data while retaining core event information, has become a core link in video surveillance systems.

[0029] Currently, existing keyframe generation solutions mainly include keyframe extraction methods based on inter-frame difference or motion variation, video summarization methods based on feature distance or clustering analysis, single-model scoring methods based on deep learning, and encoder-controlled keyframe generation methods. Most mainstream video surveillance systems and intelligent security platforms employ these single strategies or rule-driven strategies for keyframe control. However, these existing technologies still have insurmountable drawbacks: methods based on inter-frame difference or fixed thresholds rely solely on low-level pixel or motion variation information, failing to identify high-level semantic changes and easily resulting in missed keyframes in complex monitoring scenarios; single-model scoring methods rely solely on the output of a single decision model, failing to identify the uncertainty of model predictions and easily leading to keyframe omissions or misselections in scenarios with unstable model predictions; existing keyframe generation mechanisms are mostly one-time decision structures, lacking decision quality feedback and self-correction mechanisms after keyframe generation; and encoder-controlled methods often use fixed GOP lengths or static rule triggers, failing to dynamically adjust keyframe density according to real-time scene changes, easily resulting in excessive redundant keyframes in stable scenes and insufficient keyframes in abnormal scenes. Overall, existing technologies generally suffer from core defects such as a single decision-making mechanism, lack of uncertainty modeling, lack of multi-dimensional collaborative decision-making capabilities, and lack of closed-loop feedback and dynamic adjustment capabilities.

[0030] To address the aforementioned problems in the existing technology, this disclosure provides a keyframe generation method and apparatus based on multi-agent collaboration. The aim is to achieve dynamic, adaptive, and error-correctable optimization of keyframe generation in monitoring scenarios by constructing a multi-agent collaborative decision-making mechanism, an uncertainty-driven dynamic weight allocation mechanism, a joint triggering mechanism integrating probability and uncertainty, and a self-reflective closed-loop optimization mechanism. This improves the accuracy of keyframe determination in complex scenarios, reduces the omission of abnormal events, and balances video information integrity with storage resource consumption.

[0031] Figure 1 This is a schematic diagram of the architecture of a keyframe generation system 10 based on multi-agent collaboration in some embodiments of this disclosure.

[0032] A keyframe generation system 10 based on multi-agent collaboration includes a video input module 200, a feature encoding module 300, a multi-agent collaborative decision-making module 400, an uncertainty fusion module 500, a joint triggering decision-making module 600, a self-reflective closed-loop module 700, and a memory update module 800. The self-reflective closed-loop module 700 may include a self-reflective evaluation module and a frame interpolation adjustment module.

[0033] The output of the video input module 200 is communicatively connected to the input of the feature encoding module 300, and is used to output preprocessed standardized frame data to the feature encoding module 300. The output of the feature encoding module 300 is communicatively connected to the input of the multi-agent collaborative decision-making module 400, and is used to output a unified state vector to the multi-agent collaborative decision-making module 400. The output of the multi-agent collaborative decision-making module 400 is communicatively connected to the input of the uncertainty fusion module 500, and is used to output the probability and corresponding uncertainty of the key frames generated by each agent to the uncertainty fusion module 500. The output of the uncertainty fusion module 500 is communicatively connected to the input of the joint triggering decision-making module 600, and is used to output the fusion probability and comprehensive uncertainty to the joint triggering decision-making module 600. The output of the joint triggering decision-making module 600 outputs the key frame generation result, and at the same time, the output of the joint triggering decision-making module 600 is also communicatively connected to the input of the self-reflective closed-loop module 700, so as to input the key frame generation result into the self-reflective closed-loop module 700 for quality evaluation and adjustment.

[0034] The output of the self-reflective closed-loop module 700 is communicatively connected to the feedback inputs of the feature encoding module 300, the multi-agent collaborative decision-making module 400, and the uncertainty fusion module 500, respectively, to feed back the adjusted parameters after quality assessment to the corresponding modules, thereby achieving closed-loop optimization of the decision-making process. The memory update module 800 is bidirectionally communicatively connected to the self-reflective closed-loop module 700, the multi-agent collaborative decision-making module 400, and the joint triggering decision-making module 600, respectively, to record and update historical state information during the keyframe generation process, providing historical data references for the decision-making of each module.

[0035] The multi-agent collaborative decision-making module 400 incorporates multiple independent agents, including a scene change agent, a semantic understanding agent, an anomaly detection agent, and a motion modeling agent. Each agent receives the unified state vector output by the feature encoding module 300 in parallel, performs feature analysis in the corresponding dimension, and independently outputs the keyframe probability and uncertainty in the corresponding dimension.

[0036] In some embodiments of this disclosure, the execution flow of the keyframe generation method based on multi-agent collaboration can be divided into seven stages according to the execution sequence. The execution logic and parallel / serial rules of each stage are as follows: Stage 1 is video input and preprocessing, which can be processed serially. The video access unit receives the video stream, the frame sampling unit samples the data, the time synchronization unit assigns a timestamp, the image standardization process is performed, and the frame data is encapsulated in sequence. Stage 2 is feature encoding, some steps of which can be executed in parallel. First, the visual backbone encoding unit extracts basic features serially to provide input for subsequent steps. Then, the semantic feature extraction, motion feature extraction, and anomaly feature extraction are executed in parallel. Finally, the feature fusion unit generates a unified state vector. Stage 3 is multi-agent decision-making, which can be executed in parallel. The scene change agent, semantic understanding agent, anomaly detection agent, and motion modeling agent can process the state vector in parallel, each independently outputting the probability p and uncertainty U. Then, the agent output aggregation unit collects the outputs of all agents. Stage 4 is uncertainty fusion, which can be processed serially. The weight calculation unit calculates the weights based on the uncertainty, and the fusion calculation unit generates the uncertainty fusion. and Phase five involves joint triggering decision-making, which can be processed sequentially, allowing triggering decision values ​​to be calculated for each threshold judgment unit in turn. and threshold Compare and generate trigger signals; Stage 6 is a self-reflective closed loop, which can be conditionally triggered, with the trigger condition being that a keyframe is triggered. After startup, the system can sequentially predict features of subsequent frames for the prediction unit, calculate reconstruction error, and evaluate error. When the reconstruction error is greater than... At this time, the frame interpolation control unit can perform frame interpolation, threshold adjustment, and weight adjustment operations; stage seven is memory update, which can be executed every frame and can update keyframe positions, density statistics, and Agent historical uncertainty. It can be understood that the closed-loop feedback update frequency can be set as follows: a self-reflective evaluation can be performed immediately after each keyframe trigger, and the memory state can be updated every frame; closed-loop adjustment is the threshold... The correction of agent weights can take effect immediately when the frame interpolation control unit determines that adjustments are needed, and can be applied to the processing of the next frame.

[0037] Figure 2 A schematic block diagram of a video input module 200 according to some embodiments of the present disclosure is shown. In this embodiment, the multi-agent keyframe generation system may include a video input module 200, which can be used to receive raw video data and output standardized frame data. In some embodiments, the video input module 200 may include a video access unit 210, a frame sampling unit 220, a time synchronization unit 230, an image normalization processing unit 240, and a frame data encapsulation unit 250.

[0038] By way of example and not limitation, the video access unit 210 can be used to receive video stream data from at least one video source, which may include, but is not limited to, webcam video streams, local video files, edge-cached video queues, and distributed video forwarding nodes. The video access unit 210 can parse the video stream into a continuous sequence of video frames. It is understood that, as used herein, when referring to raw video data, it may refer to at least one of visible light video data, infrared video data, and depth camera video data. This disclosure supports simultaneous input from multiple video sources.

[0039] In some embodiments, the frame sampling unit 220 can be used to perform time-dimensional sampling processing on a video frame sequence. In one embodiment, the frame sampling unit 220 can sample the video stream at equal intervals according to a preset frame rate. In another embodiment, the frame sampling unit 220 can dynamically adjust the sampling interval based on the image difference results between adjacent frames. When the inter-frame variation is greater than a set threshold, the sampling density can be increased; when the inter-frame variation is less than the set threshold, the sampling density can be decreased. The frame sampling unit 220 can output the frame sequence to be processed. Additionally or alternatively, the inter-frame difference value can be calculated as follows: Let the image of the t-th frame be... The image of frame t-1 is The inter-frame difference value can be calculated as follows: Where H and W are the image height and width, respectively, and the calculation can be performed in grayscale space. As an example, and not a limitation, the frame sampling unit 220 can employ a dual-threshold mechanism to set the inter-frame difference threshold. and ,when When this is the case, it can be identified as a high-variation range, and the sampling density can be increased; when When the range is low, the sampling density can be reduced; when At this time, the current sampling interval can be maintained, where and It can be preset according to the video scene; the typical value can be set to... , It is understandable that the sampling density of frame sampling unit 220 can be set with upper and lower limits, where the minimum sampling interval, i.e., the maximum density, can be set to... A frame refers to sampling one frame at a time; the maximum sampling interval, or minimum density, can be set to... Frames, N is a configurable parameter, typically 30, meaning at least 1 frame is sampled per second, calculated at 30fps. In the example, the sampling interval adjustment step can be set as follows: When hour, ,in This is the encryption step size, typically 2; when hour, ,in This is the sparse step size, typically 2; otherwise, it can be set to... .

[0040] In some implementations, the time synchronization unit 230 can be used to assign a unified timestamp identifier to the sampled video frames. It is understood that the timestamp identifier may include an absolute time identifier, a relative frame sequence number identifier, or an internal time offset within the video stream, and the time-synchronized frame data can serve as a time reference for subsequent feature calculations.

[0041] In some embodiments, the image normalization processing unit 240 can be used to perform format unification processing on video frames. This processing may include resolution scaling, aspect ratio correction, brightness normalization, image denoising, and color space conversion. In one embodiment, the resolution can be unified to a fixed-size tensor format. In another embodiment, the resolution can be scaled proportionally and then edge-filled. As an example, and not a limitation, the specific parameters for image normalization processing can be set as follows: the target size for resolution scaling can be uniformly scaled to 224×224 or 256×256 pixels, depending on the input requirements of the backend feature encoding model. When the aspect ratio of the original frame is inconsistent with the target, it can be scaled proportionally and then zero-value-filled on the shorter side; the brightness normalization range can be set to normalize pixel values ​​to... The range is calculated by dividing the original pixel value by 255, followed by mean-variance standardization, which can be achieved by subtracting the channel mean. Divide by the channel standard deviation The color space conversion target format can be uniformly converted to RGB three-channel format. If the input is a grayscale image, it can be copied to three channels. If the input is YUV or other formats, it can be converted to RGB. Image denoising can be performed using Gaussian filtering with a kernel size of 3×3. It performs mild noise reduction and can be selectively enabled in low-light or high-noise scenes.

[0042] In some implementations, the frame data encapsulation unit 250 can be used to encapsulate the processed image frame along with corresponding time information and source information into a structured data object. The structured data object may include at least frame identifier information, timestamp information, image data tensor, and video source identifier information. The structured data object can be used as input to the feature encoding module 300. As an example, and not a limitation, the specific field format and data type of the structured data object may be as follows: FrameDataObject { frame_id: int64 / / Unique frame identifier, globally incrementing integer timestamp: float64 / / Timestamp, in seconds, with a precision of milliseconds (e.g., 1234.567) image_tensor: float32[C, H, W] / / Normalized image tensor, C=3 (RGB channels), H and W are the target resolution source_id: string / / Video source identifier, such as camera ID or hash value of file path }

[0043] Among them, frame_id is of type int64, which supports frame numbering of long video streams without overflow; timestamp is of type float64, which supports high-precision time representation; image_tensor is a float32 three-dimensional tensor with dimensions in the order of [channel, height, width]; source_id is of type string, used to distinguish the source in multi-source video scenarios.

[0044] Figure 3 A schematic block diagram of a feature encoding module 300 according to some embodiments of the present disclosure is shown. In this embodiment, the feature encoding module 300 may be connected to a video input module 200 and may be used to convert the frame data into a multidimensional state vector. In some embodiments, the feature encoding module 300 may include a visual backbone encoding unit 310, a semantic feature extraction unit 320, a motion feature extraction unit 330, an anomaly feature extraction unit 340, and a feature fusion unit 350.

[0045] In some embodiments, the visual backbone coding unit 310 can be used to perform convolutional feature extraction or Transformer feature extraction on the input image frame, and the visual backbone coding unit 310 can output a basic visual feature map. In one embodiment, the visual backbone coding unit 310 can adopt a deep convolutional network structure. In another embodiment, the visual backbone coding unit 310 can adopt a visual Transformer structure.

[0046] In some implementations, the semantic feature extraction unit 320 may be connected to the visual backbone encoding unit 310 and may be used to extract semantic representations based on basic visual features. The semantic feature extraction unit 320 may include at least one object detection subunit, an instance segmentation subunit, and a scene classification subunit. The semantic feature extraction unit 320 may output a semantic vector representation.

[0047] In some embodiments, the motion feature extraction unit 330 can be used to extract temporal variation features between video frames. The motion feature extraction unit 330 may include at least one optical flow calculation subunit, an inter-frame difference unit, and a temporal memory subunit. In one embodiment, the temporal memory subunit may include a recurrent neural network structure or a gated memory structure, which can be used to establish inter-frame dependencies. The motion feature extraction unit 330 can output a motion feature vector.

[0048] In some implementations, the abnormal feature extraction unit 340 can be used to calculate the degree of abnormality of the current frame in the feature space. The abnormal feature extraction unit 340 may include at least one feature reconstruction error calculation unit, a prediction distribution offset calculation unit, and a confidence entropy calculation unit. The abnormal feature extraction unit 340 can output an abnormality index.

[0049] In some embodiments, the feature fusion unit 350 can receive basic visual features, semantic features, motion features, and anomaly features respectively. The feature fusion unit 350 can combine these features to generate a unified state vector. In one embodiment, the feature fusion unit 350 can use a concatenation method followed by processing through a multilayer perceptron network. In another embodiment, the feature fusion unit 350 can use an attention weight allocation method to assign different weights to different features. The feature fusion unit 350 can output a state vector, which can be used as input to the multi-agent collaborative decision-making module 400.

[0050] Figure 4 A schematic block diagram of a multi-agent collaborative decision-making module 400 according to some embodiments of the present disclosure is shown. In this embodiment, the multi-agent collaborative decision-making module 400 can be connected to a feature encoding module 300, and can be used to generate keyframe decision probabilities and corresponding uncertainties based on the state vector, and can output them to a subsequent uncertainty fusion module 500. In some embodiments, the multi-agent collaborative decision-making module 400 may include a scene change agent 410, a semantic understanding agent 420, an anomaly detection agent 430, a motion modeling agent 440, and an agent output aggregation unit 450.

[0051] In some embodiments, the scene change agent 410 can be used to analyze low-level visual change features of video frames. The scene change agent 410 may include an input interface for receiving state vectors; a feature processing unit for calculating inter-frame change indices; and an output unit for generating scene change probability values. and corresponding uncertainty In one embodiment, the feature processing unit can generate a scene change probability value by calculating the pixel or feature differences between the current frame and historical keyframes. As an example, and not a limitation, the probability calculation method of the scene change agent 410 can be set as follows: calculating the scene change probability based on the feature differences between the current frame and historical keyframes, where the feature vector of the current frame is... The feature vector of the most recent keyframe is ,but , ,in For the Sigmoid function, This is the scaling factor. The offset can be a learnable parameter. Uncertainty calculation for the scene-changing agent 410 can be performed using the MC Dropout method, which allows for T forward inferences on the same input, yielding T probability predictions. The uncertainty can be set as the prediction variance: .

[0052] In some implementations, the semantic understanding agent 420 can be used to identify high-level semantic information of video frames. The semantic understanding agent 420 may include an input interface for receiving semantic features from a state vector; a semantic analysis unit for determining the presence and behavioral characteristics of important objects in the scene; and an output unit for generating semantically relevant keyframe probability values. and corresponding uncertainty In one embodiment, the semantic analysis unit can detect abnormal object or important target behavior and convert it into keyframe trigger probabilities. As an example, and not a limitation, the probability calculation method of the semantic understanding agent 420 can be set as follows: based on semantic feature vectors mapped through a fully connected layer and then activated by a Sigmoid function, where... For semantic feature vectors, and For learnable parameters, The uncertainty calculation of the semantic understanding agent 420 can be performed using the information entropy of the prediction distribution, and the semantic analysis unit can output the probability distribution of K semantic categories. ,but Divisible by Normalize to make .

[0053] In some embodiments, the anomaly detection agent 430 can be used to determine abnormal events in video frames. The anomaly detection agent 430 may include an input interface for receiving abnormal features in a state vector and an anomaly scoring unit for generating anomaly probability values ​​based on the degree of feature anomaly. Uncertainty evaluation unit, which can be used to generate the corresponding uncertainty. In one implementation, the anomaly score can be calculated based on the prediction reconstruction error or the confidence entropy. As an example, and not a limitation, the probability calculation method of the anomaly detection agent 430 can be set as follows: based on the feature reconstruction error, let the input feature be... The autoencoder reconstructed output is ,but , The uncertainty of the anomaly detection agent 430 can be calculated using the confidence entropy method, and the anomaly scoring network can output binary classification probabilities of anomaly / normal. ,but Normalization to Range, i.e., divided by .

[0054] In some embodiments, the motion modeling agent 440 can be used to analyze the motion characteristics and temporal changes of objects in a video. The motion modeling agent 440 may include an input interface for receiving motion characteristics from a state vector and a temporal analysis unit for generating motion stability or abnormal motion probabilities. Uncertainty output unit, which can be used to generate uncertainties in motion analysis. In one embodiment, the temporal analysis unit can process continuous frame sequences via a short-time memory unit or a gated loop unit. As an example, and not a limitation, the probability calculation method of the motion modeling agent 440 can be configured such that the temporal analysis unit, i.e., the gated loop unit, can process continuous frame motion feature sequences and output hidden states. It can be mapped through a fully connected layer. The uncertainty calculation for motion modeling agent 440 can be performed using the MC Dropout method, similar to the scenario-changing agent 410, to calculate the prediction variance after T inferences. .

[0055] It is understandable that all probability values This can be guaranteed by the Sigmoid function; all uncertainties. This can be guaranteed through normalization. Boundary value handling rules can be set as follows: when... When the value is completely determined, the agent's weight can take the maximum value; when... When the time is completely uncertain, the agent's weight can take the minimum value; when When the value is close to the extreme value of 0 or 1, no additional truncation is required, and the results of multiple agents can be balanced by the subsequent fusion module.

[0056] In some embodiments, the agent output aggregation unit 450 can be used to receive the probabilities and uncertainties of the outputs of each agent, and can output them uniformly to the uncertainty fusion module 500. The output aggregation unit may include a probability collection interface, which can be used to receive... Uncertainty collection interface, which can be used to receive... The output encapsulation unit can be used to generate a multi-agent output data structure, StateAgentPacket, which may be, for example: StateAgentPacket { frame_id agent_probabilities {P_scene, P_sem, P_abn, P_motion} agent_uncertainties {U_scene, U_sem, U_abn, U_motion} }, The multi-agent output data can be used as input to the uncertainty fusion module 500 to generate the final keyframe trigger decision.

[0057] In this embodiment, the multi-agent collaborative decision-making module 400 can be easily understood to have the following characteristics: multi-dimensional information judgment, each agent can independently analyze features of different dimensions, including low-level visual changes, high-level semantics, abnormal features and motion information; uncertainty output, each agent can output not only probability values, but also corresponding uncertainties, which can be used for subsequent fusion and dynamic weighting; modular interface, each agent can be connected to the feature encoding module 300 and the fusion module through a unified data interface; scalability, additional agents can be added to expand new feature judgment dimensions, such as a nighttime anomaly agent and a weather status agent.

[0058] Figure 5 A schematic block diagram of an uncertainty fusion module 500 according to some embodiments of the present disclosure is shown. In this embodiment, the uncertainty fusion module 500 can be connected to a multi-agent collaborative decision-making module 400, and can be used to receive the keyframe probabilities and corresponding uncertainties output by each agent, and can dynamically calculate a comprehensive decision value based on the uncertainties, which can be output to a joint triggering decision-making module 600. In some embodiments, the uncertainty fusion module 500 may include a probability input interface 510, an uncertainty input interface 520, a weight calculation unit 530, a fusion calculation unit 540, and an output encapsulation unit 550.

[0059] In some embodiments, the probability input interface 510 can be used to receive keyframe generation probabilities output by multiple agents, and may include... The interface can transmit the received probability vector to the fusion computing unit 540.

[0060] In some embodiments, the uncertainty input interface 520 can be used to receive uncertainties output by multiple agents, and may include... The interface can transmit the received uncertainty vector to the weight calculation unit 530.

[0061] In some embodiments, the weight calculation unit 530 can be used to calculate the corresponding weight based on the uncertainty of each agent, and can be used to weight the output probabilities of each agent. In one embodiment, the weight calculation unit 530 can calculate the weight of each agent based on the reciprocal of the uncertainty. In another embodiment, the weights can be dynamically adjusted using normalization or gating mechanisms. As an example and not a limitation, the complete formula for weight calculation can be as follows: Let the first... The uncertainty of each agent is: , The corresponding weight is ,in For extremely small positive numbers, a typical value can be set to... This can be used to prevent division by zero anomalies. It is understandable that the boundary value handling rules for this weight calculation method can be set as follows: when a certain agent... At that time, due to The existence of the weight is The normalized value, that is, the agent can obtain extremely high weight, almost exclusive; when an agent When, the weight is The normalized value, that is, the agent can obtain a lower weight; when all agents When all are equal, the weights of each agent can be equal, that is... Weight normalization can guarantee .

[0062] In some implementations, the fusion computing unit 540 can receive the probabilities and corresponding weights of each agent and can calculate the overall keyframe probability. In one embodiment, the fusion computing unit 540 may generate a combined probability using a weighted summation method. In another embodiment, an attention mechanism or a gated fusion method may be used to dynamically weight the probabilities. As an example and not a limitation, the fusion probability... The complete formula for weighted calculation can be set as follows: ,Right now ,in For normalized weights based on uncertainty, The keyframe probabilities output by each agent, due to And each ,therefore Additional or alternative uncertainties, combined The specific calculation can be performed using the weighted variance method, and the formula can be set as follows: The first term is the weighted average of the uncertainties of each agent, which reflects the uncertainty of each agent itself; the second term is the weighted variance of the probability predictions of each agent, that is, the degree of divergence among agents, which reflects the degree of disagreement among multiple agents. This is the divergence weighting coefficient; a typical value can be set to... It can be used to balance the contributions of two items. When all agents are highly consistent and their individual uncertainties are low, It can approach 0; when there is a large divergence among agents or their individual uncertainties are high, It can approach 1.

[0063] In some implementations, the output encapsulation unit 550 can encapsulate the fusion result into a structured data object, FusionPacket, which may be, for example: FusionPacket { frame_id fusion_probability P_fusion combined_uncertainty U_fusion }, The data object can be output to the joint triggering decision module 600 and used for final keyframe determination.

[0064] Figure 6 A schematic block diagram of a joint triggering decision module 600 according to some embodiments of the present disclosure is shown. In this embodiment, the joint triggering decision module 600 may be connected to an uncertainty fusion module 500 and can be used to generate keyframe triggering decisions based on fusion probabilities and uncertainties. In some embodiments, the joint triggering decision module 600 may include a decision input interface 610, a threshold judgment unit 620, and a decision output unit 630.

[0065] In some embodiments, the decision input interface 610 can be used to receive the fusion probability and uncertainty from the uncertainty fusion module 500, which may include the fusion probability. Comprehensive uncertainty .

[0066] In some implementations, the threshold determination unit 620 can calculate the trigger decision value based on the fusion probability and uncertainty. And can be compared with a preset threshold. Comparison. In one embodiment, the trigger decision value It can be generated by a weighted combination of fusion probability and comprehensive uncertainty. In another embodiment, the threshold can be dynamically adjusted according to the historical memory state. .when If a keyframe trigger signal is generated, it can be used; otherwise, a keyframe may not be generated. This is an example, not a limitation, of the trigger decision value. The calculation formula can be set as follows: ,in For fusion probability, To account for the overall uncertainty, This is the uncertainty weighting coefficient, and a typical value can be set to... , The range of values ​​can be set to That is, when At that time threshold The setting must match this range. Understandably, the threshold... The initial value can be set to It is applicable to general monitoring scenarios, and its dynamic adjustment logic and formula are as follows: Adjustment can be made based on the keyframe density statistics provided by the memory update module 800. Let the keyframe density within the current time window be... That is, the number of keyframes per unit time, and the target density is ,but ,in To adjust the learning rate, a typical value can be set to... When the actual density That is, when there are too many keyframes It can be increased, and the trigger threshold can be raised; when the actual density That is, when there are too few keyframes, The threshold can be reduced, thus lowering the triggering threshold; at the same time, the threshold... Value range constraints can be set. ,in , This can prevent excessive threshold offset from causing system failure.

[0067] In some implementations, the decision output unit 630 can be used to generate a keyframe trigger signal, and can encapsulate the frame identifier and trigger state into a structured data TriggerPacket, which may be, for example: TriggerPacket { frame_id trigger_flag } The TriggerPacket can be used by the subsequent self-reflective closed-loop module 700 for quality verification and frame interpolation operations.

[0068] Figure 7A schematic block diagram of a self-reflective closed-loop module 700 according to some embodiments of the present disclosure is shown. In this embodiment, the self-reflective closed-loop module 700 can be connected to a joint triggering decision module 600, and can be used to evaluate the quality of selected keyframes after keyframe generation, and can perform frame interpolation or adjustment based on the evaluation results, thus forming a closed-loop optimization. In some embodiments, the self-reflective closed-loop module 700 may include a prediction unit 710, a reconstruction error calculation unit 720, an error evaluation unit 730, a frame interpolation control unit 740, and a state update unit 750.

[0069] In some implementations, the prediction unit 710 can predict feature representations for several subsequent frames based on the current keyframe and historical state information. In one implementation, the prediction unit 710 can use an autoregressive network or a gated recurrent unit for prediction.

[0070] In some embodiments, the reconstruction error calculation unit 720 can be used to compare the predicted frame with the real frame and calculate an error value to quantify the completeness of the current keyframe's representation of the video content. In one embodiment, the error value can be the Euclidean distance or cosine similarity of the feature vectors. In another embodiment, an uncertainty enhancement index can be calculated by combining the difference in reconstruction probability distribution. As an example and not a limitation, the formula for calculating the reconstruction error can be as follows: Let the feature vector of the current keyframe be... Prediction unit 710 predicts subsequent... The prediction features of the frame are The real number The characteristics of a frame are The single-frame reconstruction error can then be set to... In the prediction window Inside, To predict the step size, a typical value can be set to... The overall reconstruction error can be set to The value range can be set to Since feature vectors are usually processed by L2 normalization, single-frame error Comprehensive reconstruction error Alternatively, if cosine similarity is used, the single-frame error can be set to... ,but , .

[0071] In some implementations, the error evaluation unit 730 can determine whether the keyframes fully express subsequent video information based on the reconstruction error. When the reconstruction error exceeds a set threshold... If the current keyframe is insufficient to support subsequent video reconstruction or event backtracking, it can be determined that the current keyframe is inadequate. As an example, not a limitation, the error threshold... The initial value can be set to Based on empirical values ​​obtained from L2 normalized feature vectors, the dynamic adjustment rule can be as follows: adaptive adjustment can be performed based on statistical information of historical reconstruction errors. ,in For the most recent Average reconstruction error of sub-keyframes This is the sliding window size; a typical value can be set to... ; For the most recent The standard deviation of the reconstruction error; This is the sensitivity coefficient, and a typical value can be set to... ;at the same time Value range constraints can be set. ,in , This adaptive mechanism prevents excessively low thresholds from causing frequent frame interpolation or excessively high thresholds from causing frame interpolation failure. It can automatically adjust according to the complexity of the video content, especially in scenarios with rapidly changing content. It can automatically rise in stable environments. It can be automatically reduced.

[0072] In some implementations, the frame interpolation control unit 740 may perform the following operations based on the error assessment results: add supplementary keyframes within the current time window; dynamically adjust the threshold of the joint triggering decision module 600. To increase the trigger probability; the weights of the 400 agents in the multi-agent collaborative decision-making module can be adjusted to enhance sensitivity to high-uncertainty regions. As an example, and not a limitation, the specific selection rules for frame interpolation can be as follows: Regarding the selection of the frame interpolation position, when the reconstruction error... At this time, you can select to add frames within the time window between the current keyframe and the next keyframe. The specific selection strategy can be set to select frames within the time window. Inside, Given the time window length, the frame that contributes the most to the reconstruction error can be selected as the supplementary keyframe, i.e. The maximum number of frames that can be interpolated in a single interpolation operation can be set to... When the number of frames to be interpolated exceeds When the error is the largest, the first option can be selected. Frame; Length of the time window for frame interpolation search This can be related to the sampling interval; a typical value can be set to... That is, twice the current sampling interval, and Frame; the frame interpolation termination condition can be set to recalculate the reconstruction error after frame interpolation, if Then you can stop frame interpolation; if the number of frames interpolated has reached... If the conditions are still not met, the frame interpolation can be stopped and the interval recorded as a high-uncertainty region for subsequent threshold adjustment. Alternatively, the frame interpolation control unit 740 may adjust the trigger threshold. The adjustment step size and upper / lower bound rules for agent weights can be as follows: Regarding the threshold... The adjustment is made when the frame interpolation control unit 740 determines that the trigger probability needs to be increased. ,in To adjust the threshold step size, a typical value can be set to... The upper limit of the threshold can be set to The lower limit can be set to Regarding the adjustment of agent weights, when the frame interpolation control unit 740 determines that it is necessary to enhance the sensitivity to high uncertainty regions, it can increase the weight bias for agents with higher uncertainty. The formula can be set as follows: ,in To adjust the step size for weights, a typical value can be set to... , Let be the mean of the uncertainties of all agents. This is an indicator function; it returns 1 when the condition is true and 0 otherwise. It can be renormalized after adjustment. The lower limit of the weight of a single agent can be set to This ensures that each agent has at least a minimum say, and the maximum weight of a single agent can be set to [value missing]. This can prevent a single intelligent agent from dominating decision-making.

[0073] In some implementations, the state update unit 750 can update the system's historical memory state, which may include the position of the most recent keyframe, the density statistics of the most recent keyframe, the historical uncertainty information of each agent, and the historical information of reconstruction error. The updated state can be used to generate the next keyframe, enabling continuous closed-loop optimization.

[0074] Figure 8 A schematic block diagram of a memory update module 800 according to some embodiments of the present disclosure is shown. In this embodiment, the memory update module 800 can be connected to the self-reflective closed-loop module 700, and can be used to record historical state information during the keyframe generation process. It can also provide historical references for the multi-agent collaborative decision-making module 400 and the joint triggering decision-making module 600, enabling continuous optimization of the system state. In some embodiments, the memory update module 800 may include a keyframe position recording unit 810, a keyframe density statistics unit 820, an agent historical uncertainty recording unit 830, and a state update interface 840.

[0075] In some embodiments, the keyframe location recording unit 810 can be used to record the time position or frame number of the generated keyframes in the video stream. In one embodiment, the keyframe location recording unit 810 can store each keyframe identifier in timestamp order and form a keyframe sequence list. In another embodiment, it can be grouped and stored by time window or scene segment, supporting local keyframe analysis. As an example and not a limitation, the detailed parameters of keyframe location and density statistics can be as follows: the time window length can be set to... A frame, approximately 10 seconds at 30fps, can be stored using a sliding window approach; the storage structure can employ a circular buffer, capable of storing (frame_id, timestamp) tuples for keyframes; the maximum buffer capacity can be set to... There are 100 records; if the limit is exceeded, the oldest record will be overwritten.

[0076] In some implementations, the keyframe density statistics unit 820 can be used to calculate the distribution density of keyframes within a certain time window based on the recorded keyframe positions. The statistical information may include the number of keyframes per unit time window, the mean keyframe interval, and the variance of the keyframe interval. The keyframe density statistics results can be used by the self-reflective closed-loop module 700 and the joint triggering decision module 600 to dynamically adjust the keyframe trigger threshold and the weights of multiple agents. It is understood that the keyframe density statistics can be updated once per frame and can be updated synchronously as the sliding window moves.

[0077] In some implementations, the agent historical uncertainty recording unit 830 can be used to record the uncertainty information output by each agent during the keyframe determination process. The recorded information may include the uncertainty value of each agent in each frame, the historical maximum / minimum / average uncertainty, and the trend change of uncertainty. This information can be used to dynamically adjust the weights of each agent in the fusion module and can assist in determining keyframe coverage and system stability. As an example, and not a limitation, the statistical window for agent historical uncertainty can be set to the most recent... The uncertainty record of a frame can maintain an independent circular buffer for each agent, which can store (frame_id, ) tuple, the buffer size can be set to It can also maintain the mean, variance, maximum and minimum values ​​within the window in real time, and can use incremental update method without traversal.

[0078] In some implementations, the state update interface 840 can be used to update the keyframe position, density statistics, and agent uncertainty information to the system state storage unit. The updated state information can be used as a reference for the input history memory of the multi-agent collaborative decision-making module 400, for the joint triggering decision-making module 600 to dynamically adjust thresholds, and for the self-reflective closed-loop module 700 to generate frame supplementation and optimization strategies. The memory update module 800 can form a closed-loop feedback system with the self-reflective closed-loop module 700 and the multi-agent collaborative decision-making module 400, enabling adaptive optimization of keyframe determination. It is understood that the state update frequency can be set as follows: keyframe position records can be updated each time a keyframe is triggered, i.e., event-driven; density statistics can be updated once per frame; agent uncertainty records can be updated once per frame. Feedback logic may include: using the agent's historical uncertainty statistics, i.e., mean and trend, as additional contextual information for each agent's input, allowing the agent to perceive its own historical performance and refer to the historical state during inference in the next frame; and using the keyframe density statistics results, i.e., the current window density... The density change trend is transmitted to the threshold judgment unit 620, which can be used to dynamically adjust the threshold. Historical error information can be transmitted to the error evaluation unit 730, which can be used to dynamically adjust the error threshold. .

[0079] Figure 9 The diagram shows a flowchart of a keyframe generation method 900 based on multi-agent collaboration provided in this disclosure. This method 900 can be applied to the keyframe generation system based on multi-agent collaboration described in this disclosure and can be deployed in edge computing devices, GPU servers, cloud video analysis platforms, or embedded AI cameras to achieve adaptive keyframe generation of monitoring video streams in scenarios such as urban security, traffic management, park monitoring, and industrial production supervision.

[0080] In this embodiment, the keyframe generation method 900 based on multi-agent collaboration includes steps 910 to 950, and the specific details of each step are as follows.

[0081] Step 910: Perform preprocessing on the raw video data to obtain standardized frame data. In some embodiments, this step can be performed by the video input module described in this disclosure. The raw video data may include, but is not limited to, video streams from network cameras, local video files, edge-cached video queues, and video data output by distributed video forwarding nodes. Optionally, the raw video data may include at least one of visible light video data, infrared video data, and depth camera video data, supporting simultaneous input from multiple video sources. Exemplarily, the preprocessing flow of this step may include: parsing the raw video data into a continuous video frame sequence, performing time-dimensional sampling on the continuous video frame sequence, assigning a unified timestamp identifier to the sampled video frames to synchronize the video frames in time, performing image standardization processing on the time-synchronized video frames, and finally encapsulating the standardized video frames and corresponding metadata into structured frame data to obtain standardized frame data, providing a unified and standardized input for subsequent feature extraction.

[0082] Step 920: Perform fusion processing on the multi-dimensional features extracted from the standardized frame data to generate a unified state vector. In some embodiments, this step can be performed by the feature encoding module described in this disclosure. For example, features in four dimensions—basic visual features, high-level semantic features, temporal motion features, and feature space anomaly features—can be extracted from the standardized frame data. The basic visual features can be extracted using a deep convolutional network or a visual Transformer structure. The high-level semantic features can be extracted using at least one of object detection, instance segmentation, and scene classification. The temporal motion features can be extracted using at least one of optical flow calculation, inter-frame differencing, and temporal memory modeling. The feature space anomaly features can be calculated using at least one of feature reconstruction error, prediction distribution offset, and confidence entropy. In some embodiments, fusion processing can be performed on the extracted multi-dimensional features. The fusion processing can use multi-dimensional features concatenated by a multi-layer perceptual network, or dynamically weighted and fused based on attention weight allocation, ultimately generating a unified state vector that can be recognized by multiple agents.

[0083] Step 930 involves multiple independent agents performing feature analysis on corresponding dimensions based on the unified state vector to output keyframe probabilities for those dimensions, and the corresponding uncertainties. In some embodiments, this step can be executed by the multi-agent collaborative decision-making module described in this disclosure. The multiple independent agents may include a scene change agent, a semantic understanding agent, an anomaly detection agent, and a motion modeling agent. Each agent receives the unified state vector in parallel and analyzes it from four independent dimensions: low-level visual changes, high-level semantic information, abnormal events, and motion temporal changes. Each agent independently outputs keyframe probabilities for the corresponding dimension, and the uncertainty used to measure the agent's confidence in the keyframe determination. Optionally, the keyframe probabilities and uncertainties output by all agents can be encapsulated as multi-agent output structured data for subsequent fusion calculations.

[0084] Step 940: Weighted fusion of the keyframe probabilities is performed to obtain a fusion probability and a comprehensive uncertainty, wherein the weights in the weighted fusion are based on the uncertainty. In some embodiments, this step can be performed by the uncertainty fusion module described in this disclosure. For example, the initial weights can be calculated based on the reciprocal of the uncertainty of each agent, and the initial weights can be normalized to obtain the final fusion weights of each agent. Then, the keyframe probabilities of each agent are weighted and summed based on the final fusion weights to obtain the fusion probability. At the same time, the comprehensive uncertainty can be calculated by weighted variance, and the comprehensive uncertainty reflects both the uncertainty of each agent itself and the degree of decision divergence among multiple agents.

[0085] Step 950: Based on the fusion probability and the comprehensive uncertainty, a trigger judgment is made to generate a keyframe trigger signal to generate the keyframe. In some embodiments, this step can be executed by the joint trigger decision module described in this disclosure. For example, a weighted combination of the fusion probability and the comprehensive uncertainty can be performed to generate a trigger decision value. The trigger decision value is compared with a keyframe trigger threshold. When the trigger decision value is greater than the keyframe trigger threshold, a keyframe trigger signal is generated, and the current frame is marked as a keyframe, thus completing the generation of the keyframe. Optionally, the keyframe trigger threshold can be a fixed threshold or an adaptive threshold dynamically adjusted based on historical state information.

[0086] The keyframe generation method 900 based on multi-agent collaboration described in this embodiment effectively overcomes the core defects of existing keyframe generation schemes, such as a single decision-making mechanism, lack of uncertainty modeling and multi-dimensional collaborative decision-making capabilities, and inability to adapt to dynamic monitoring scenarios. It achieves beneficial technical effects including, but not limited to, the following: First, by using multiple independent agents to analyze video frame information in parallel from different dimensions, it simultaneously covers low-level visual changes and high-level semantic, anomaly, and motion information, avoiding the judgment failure problem of a single decision model in complex monitoring scenarios. This significantly improves the robustness and accuracy of keyframe judgment, reducing missed keyframe detections in important scenarios such as abnormal events and minor violations. Second, through an uncertainty-driven dynamic weight allocation mechanism, the decision weights are dynamically adjusted based on the agent's judgment confidence, reducing the uncertainty... By leveraging the influence of the decision-making body on the final result, and combining the joint triggering judgment of fusion probability and comprehensive uncertainty, this method takes into account both the degree of scene change and the risk of model prediction. When the uncertainty of model prediction is high, it automatically increases the triggering sensitivity of key frames, avoiding the omission of important information due to insufficient model confidence, and significantly reducing the risk of misjudgment and missed detection of key frames. Thirdly, the modular design of the entire process of this method has strong scalability and scene adaptability, and can be compatible with multi-channel and multi-modal video source input. At the same time, it provides a standardized process and data foundation for subsequent self-reflective quality assessment and closed-loop parameter optimization. Under the premise of ensuring the complete preservation of key video information, it can adaptively adapt to the differentiated needs of stable and abnormal scenarios, effectively reducing the amount of redundant data in monitoring videos and reducing the resource consumption of video storage, transmission and subsequent analysis.

[0087] Embodiments of other aspects of this disclosure are described below.

[0088] In some embodiments, after generating a keyframe trigger signal, the information expression quality of the generated keyframes is evaluated. If the evaluation result does not meet preset requirements, supplementary keyframes are generated within the corresponding time window. Thus, through the quality verification and supplementary frame mechanism after keyframe generation, the keyframe selection result is self-corrected, avoiding the problem that keyframes are insufficient to support subsequent video reconstruction or event backtracking, and improving the completeness of keyframe expression of video content.

[0089] In some embodiments, the information representation quality assessment includes: predicting the feature representation of a predetermined number of subsequent frames based on the current keyframe and historical state information; calculating the reconstruction error between the predicted feature representation and the actual frame feature representation; comparing the reconstruction error with a predetermined error threshold, and completing the information representation quality assessment based on the comparison result. When the reconstruction error is greater than the predetermined error threshold, the assessment result is determined to not meet the predetermined requirements; when the reconstruction error is less than or equal to the predetermined error threshold, the assessment result is determined to meet the predetermined requirements. Thus, through feature prediction and quantitative calculation of reconstruction error, combined with a clear determination standard of the predetermined threshold, the expressive power of keyframes for subsequent video content is accurately measured, providing a quantifiable, reproducible, and clearly defined basis for quality assessment.

[0090] In some embodiments, the reconstruction error includes the Euclidean distance, cosine similarity, or probability distribution difference of the feature vectors; the feature representation of the subsequent preset number of frames is predicted through an autoregressive network or a gated recurrent unit. This clarifies the calculation method of the reconstruction error and the implementation path of feature prediction, ensuring the accuracy of quality assessment and operational efficiency, and improving the engineering feasibility of the solution.

[0091] In some embodiments, after completing the information expression quality assessment, the control parameters for keyframe generation are adjusted based on the quantification results of the reconstruction error and the comparison results between the reconstruction error and a preset error threshold. Thus, by using explicit quality assessment quantification results to inversely optimize the keyframe generation parameters, closed-loop process optimization is achieved, enabling the system to adaptively adjust its decision logic according to the actual expression effect of the video content.

[0092] In some embodiments, adjusting the control parameters for keyframe generation includes performing at least one of the following operations: dynamically adjusting the keyframe trigger threshold based on the keyframe distribution density, reducing the keyframe generation frequency in stable scenarios, and increasing the keyframe trigger probability in abnormal scenarios; dynamically adjusting the fusion weights of each agent based on historical uncertainty data of each agent to enhance sensitivity to high uncertainty regions; and optimizing the error threshold corresponding to the information expression quality assessment based on historical reconstruction error data. Thus, full-link dynamic optimization is achieved from three aspects: trigger sensitivity, decision weights, and evaluation criteria, balancing storage resource consumption and video information integrity, while eliminating redundant constraints on parameter adjustment logic and ensuring a logical closed loop in the technical solution.

[0093] In some embodiments, the method further includes recording and updating historical state information during the keyframe generation process. This historical state information includes at least one of the following: the time position and frame number of the generated keyframes, the keyframe distribution density within a preset time window, historical uncertainty data for each agent, and historical reconstruction error data corresponding to the keyframe quality assessment. This clarifies the core recording dimensions of the historical state information, providing comprehensive and accurate data support for subsequent adaptive decision-making adjustments throughout the entire process, ensuring the rationality and accuracy of the system's adaptive optimization.

[0094] In some embodiments, preprocessing is performed on the raw video data, including: parsing the raw video data into a continuous sequence of video frames; performing time-dimensional sampling on the continuous video frame sequence; assigning a unified timestamp identifier to the sampled video frames for time synchronization; performing image normalization processing on the time-synchronized video frames; and encapsulating the normalized video frames and corresponding metadata into structured frame data to obtain the normalized frame data. Thus, through the standardized preprocessing flow, data differences between different video sources and acquisition environments are eliminated, providing a unified and standardized input for subsequent feature extraction and agent decision-making.

[0095] In some embodiments, time-dimensional sampling is performed at equal intervals and a fixed frame rate. This provides a basic sampling scheme with low computational cost and simple structure, adaptable to deployment on edge devices with low computing power, and ensuring stable and predictable sampling.

[0096] In some embodiments, temporal sampling is dynamic density sampling based on the difference results between adjacent frames, including: increasing the sampling density in response to determining that the inter-frame change amplitude is greater than a preset threshold; and decreasing the sampling density in response to determining that the inter-frame change amplitude is less than a preset threshold. Thus, increasing the sampling density in high-change scenarios ensures information integrity, while decreasing the sampling density in low-change scenarios reduces computational redundancy, balancing efficiency and information integrity.

[0097] In some embodiments, the multi-dimensional features include basic visual features, high-level semantic features, temporal motion features, and feature space anomaly features of the standardized frame data. Thus, by clearly defining the multi-dimensional features, the visual, semantic, motion, and anomaly information of the video frames is comprehensively covered, avoiding judgment failures caused by a single feature dimension and improving the accuracy of keyframe judgment in complex scenes.

[0098] In some embodiments, the fusion process includes concatenating multi-dimensional features through a multi-layer perceptual network, or dynamically weighting and fusing the extracted features based on attention weight allocation. This provides two preferred feature fusion implementation paths, adapting to different computing power environments, ensuring effective fusion of multi-dimensional features, and generating a unified state vector that accurately represents the full-dimensional information of the current video frame.

[0099] In some embodiments, multiple independent agents are used, including a scene change agent, a semantic understanding agent, an anomaly detection agent, and a motion modeling agent. The scene change agent analyzes low-level visual change features of video frames and outputs the probability and uncertainty of keyframes corresponding to scene changes. The semantic understanding agent identifies high-level semantic information of video frames and outputs the probability and uncertainty of semantically related keyframes. The anomaly detection agent determines abnormal events in video frames and outputs the probability and uncertainty of keyframes corresponding to abnormal events. The motion modeling agent analyzes the motion features and temporal changes of targets in the video and outputs the probability and uncertainty of keyframes corresponding to motion anomalies. Thus, through parallel multi-dimensional analysis by four independent agents, multi-dimensional collaborative judgment is achieved, avoiding the decision bias of a single model and significantly improving the robustness and accuracy of keyframe selection.

[0100] In some embodiments, the keyframe probabilities and uncertainties output by all agents are encapsulated into multi-agent output structured data. This structured data includes the current frame identifier, the keyframe probability set for each agent, and the uncertainty set for each agent. Thus, through standardized structured data encapsulation, unified management and transmission of agent output data are achieved, improving the efficiency and stability of data interaction between modules.

[0101] In some embodiments, the keyframe probabilities are weighted and fused, including: using the reciprocal of the uncertainty output by each agent as the initial weight for that agent; performing normalization on all initial weights to obtain the final fused weight for each agent; and performing a weighted summation on the keyframe probabilities of each agent based on the final fused weight to obtain the fused probability. Thus, through the dynamic weight allocation mechanism based on the reciprocal of uncertainty, agents with higher confidence levels receive greater decision weights, reducing the impact of unreliable agents on the final decision, enhancing the system's awareness of prediction risks, and reducing the risk of keyframe omissions and misjudgments.

[0102] In some embodiments, triggering a keyframe trigger signal based on the fusion probability and the overall uncertainty includes: performing a weighted combination of the fusion probability and the overall uncertainty to generate a trigger decision value; comparing the trigger decision value with a keyframe trigger threshold; and generating the keyframe trigger signal when the trigger decision value is greater than the keyframe trigger threshold. Thus, by using a joint trigger function of fusion probability and uncertainty, the final decision simultaneously considers the degree of scene change and the risk level of model prediction, avoiding the omission of important information due to insufficient model confidence.

[0103] In some embodiments, the keyframe trigger threshold is a fixed threshold or an adaptive threshold that is dynamically adjusted based on historical state information. This simultaneously covers both the basic scheme with a fixed threshold and the optimized scheme with an adaptive threshold, balancing the versatility of the solution with scenario adaptability, and meeting the differentiated needs of different monitoring scenarios.

[0104] In some embodiments, the raw video data includes at least one of visible light video data, infrared video data, and depth camera video data. This expands the range of applicable video sources for the solution, making it suitable for multimodal monitoring scenarios such as visible light surveillance, infrared night vision, and depth sensing, significantly improving the solution's scenario coverage and engineering application value.

[0105] Figure 10 A schematic block diagram of a keyframe generation apparatus 1000 based on multi-agent collaboration provided in an embodiment of the present disclosure is shown. The apparatus 1000 is used to execute the keyframe generation method based on multi-agent collaboration described in any embodiment of the present disclosure.

[0106] In some embodiments, a keyframe generation apparatus 1000 based on multi-agent collaboration includes: The video input module 1010 is used to perform preprocessing on the raw video data to obtain standardized frame data; The feature encoding module 1020 is communicatively connected to the video input module 1010 and is used to perform fusion processing on multi-dimensional features extracted from standardized frame data to generate a unified state vector. The multi-agent collaborative decision-making module 1030 is communicatively connected to the feature encoding module 1020. It is used to perform feature analysis of the corresponding dimension by multiple independent agents based on a unified state vector, so as to output the key frame probability of the corresponding dimension and the uncertainty corresponding to the key frame probability. Uncertainty fusion module 1040 is communicatively connected to multi-agent collaborative decision-making module 1030 and is used to perform weighted fusion of the key frame probabilities to obtain fusion probability and comprehensive uncertainty, wherein the weights in the weighted fusion are based on the uncertainty. The joint triggering decision module 1050 is communicatively connected to the uncertainty fusion module 1040 and is used to make triggering judgments based on the fusion probability and the comprehensive uncertainty, and generate a key frame triggering signal to generate the key frame.

[0107] In some embodiments, the device 1000 further includes a keyframe quality assessment module, which is communicatively connected to the joint triggering decision module 1050. This module is used to assess the information expression quality of the generated keyframes after a keyframe trigger signal is generated. The information expression quality assessment includes: predicting the feature representation of a preset number of subsequent frames based on the current keyframe and historical state information; calculating the reconstruction error between the predicted feature representation and the actual frame feature representation; comparing the reconstruction error with a preset error threshold; and completing the information expression quality assessment based on the comparison result. When the reconstruction error is greater than the preset error threshold, the assessment result is determined to not meet the preset requirements; when the reconstruction error is less than or equal to the preset error threshold, the assessment result is determined to meet the preset requirements. When the assessment result does not meet the preset requirements, additional keyframes are generated within the corresponding time window.

[0108] In some embodiments, the device 1000 further includes a parameter adjustment module, which is communicatively connected to the keyframe quality assessment module. After completing the information expression quality assessment, the parameter adjustment module adjusts the control parameters for keyframe generation based on the quantification result of the reconstruction error and the comparison result between the reconstruction error and a preset error threshold.

[0109] In some embodiments, adjusting the control parameters for keyframe generation includes performing at least one of the following operations: dynamically adjusting the keyframe triggering threshold based on the keyframe distribution density, reducing the keyframe generation frequency in stable scenarios, and increasing the keyframe triggering probability in abnormal scenarios; dynamically adjusting the fusion weights of each agent based on the historical uncertainty data of each agent to enhance sensitivity to high uncertainty regions; and optimizing the error threshold corresponding to the information expression quality assessment based on historical reconstruction error data.

[0110] In some embodiments, the device 1000 further includes a memory update module, which is communicatively connected to the other functional modules in the device, and is used to record and update historical state information during the key frame generation process. The historical state information includes at least one of the following: the time position and frame number of the generated key frame, the key frame distribution density within a preset time window, the historical uncertainty data of each agent, and the historical reconstruction error data corresponding to the key frame quality assessment. The historical state information is then used for decision adjustments in subsequent key frame generation.

[0111] In some embodiments, the video input module 1010 includes a video access unit, a frame sampling unit, a time synchronization unit, an image normalization processing unit, and a frame data encapsulation unit; The video access unit is used to parse raw video data into a continuous sequence of video frames; The frame sampling unit is used to perform temporal sampling on a continuous sequence of video frames; The time synchronization unit is used to assign a unified timestamp identifier to the sampled video frames in order to synchronize the video frames in time. The image normalization processing unit is used to perform image normalization processing on the time-synchronized video frames; The frame data encapsulation unit is used to encapsulate the standardized video frames and corresponding metadata into structured frame data, thereby obtaining the standardized frame data.

[0112] In some embodiments, the feature encoding module 1020 includes a visual backbone encoding unit, a semantic feature extraction unit, a motion feature extraction unit, an anomaly feature extraction unit, and a feature fusion unit; The visual backbone coding unit, semantic feature extraction unit, motion feature extraction unit, and anomaly feature extraction unit are used to extract basic visual features, high-level semantic features, temporal motion features, and feature space anomaly features from standardized frame data, respectively. The feature fusion unit is used to perform fusion processing on the extracted multi-dimensional features. The fusion processing includes multi-dimensional features spliced ​​by multi-layer perceptron processing, or dynamic weighted fusion of each extracted feature based on attention weight allocation, to generate a unified state vector.

[0113] In some embodiments, the multi-agent collaborative decision-making module 1030 includes a scene change agent, a semantic understanding agent, an anomaly detection agent, a motion modeling agent, and an agent output aggregation unit; The scene change agent, semantic understanding agent, anomaly detection agent, and motion modeling agent are used to output the keyframe probability and uncertainty of the corresponding dimensions, respectively. The agent output aggregation unit is used to encapsulate the keyframe probabilities and uncertainties of all agent outputs into multi-agent output structured data.

[0114] In some embodiments, the uncertainty fusion module 1040 includes a probability input interface, an uncertainty input interface, a weight calculation unit, and a fusion calculation unit; The probability input interface is used to receive keyframe probabilities, and the uncertainty input interface is used to receive uncertainties. The weight calculation unit is used to take the reciprocal of the uncertainty output by each agent as the initial weight, and perform normalization processing on the initial weight to obtain the final fusion weight; The fusion computing unit is used to perform a weighted summation of the keyframe probabilities of each agent based on the final fusion weights to obtain the fusion probability.

[0115] In some embodiments, the joint triggering decision module 1050 includes a decision input interface, a threshold judgment unit, and a decision output unit; The decision input interface is used to receive the fusion probability and the overall uncertainty; The threshold judgment unit is used to perform a weighted combination of fusion probability and comprehensive uncertainty to generate a trigger decision value, and compare the trigger decision value with the key frame trigger threshold; The decision output unit is used to generate a keyframe trigger signal when the trigger decision value is greater than the keyframe trigger threshold.

[0116] In some embodiments, the device 1000 is deployed in an edge computing device, a GPU server, a cloud video analytics platform, or an embedded AI camera. It should be understood that... Figure 10 The various modules or units of the device 1000 shown can be connected with the reference. Figure 9 The steps in method 900 described correspond to each other. Therefore, the operations, features, and advantages described above for method 900 also apply to apparatus 1000 and its included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0117] Although specific functions have been discussed with reference to specific modules above, it should be noted that the functions of the units discussed in this article can be divided into multiple units, and / or at least some functions of multiple units can be combined into a single unit.

[0118] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described above.

[0119] According to another aspect of this disclosure, a computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above-described method when executed by a processor.

[0120] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0121] See Figure 11 The present invention describes a structural block diagram of an electronic device 1100 that can serve as a storage node, client agent, dedicated computing node, or server, etc., as an example of hardware devices applicable to various aspects of the present disclosure. The electronic device can be different types of computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0122] like Figure 11 As shown, the electronic device 1100 may include at least one processor 1101, working memory 1102, input unit 1104, display unit 1105, speaker 1106, storage unit 1107, communication unit 1108 and other output units 1109 that are capable of communicating with each other via system bus 1103.

[0123] Processor 1101 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 1101 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Processor 1101 may be configured to acquire and execute computer-readable instructions stored in working memory 1102, storage unit 1107, or other computer-readable media, such as program code of operating system 1102a, program code of application program 1102b, etc.

[0124] Working memory 1102 and storage unit 1107 are examples of computer-readable storage media for storing instructions executed by processor 1101 to perform the various functions described above. Working memory 1102 may include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, storage unit 1107 may include hard disk drives, solid-state drives, removable media including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Working memory 1102 and storage unit 1107 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 1101 as a specific machine configured to perform the operations and functions described in the examples herein.

[0125] Input unit 1104 can be any type of device capable of inputting information to electronic device 1100. Input unit 1104 can receive input digital or character information and generate key signal input related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone and / or remote control. Output unit can be any type of device capable of presenting information, and can include, but is not limited to, display unit 1105, speaker 1106 and other output units 1109. Other output units 1109 can include, but are not limited to, video / audio output terminals, vibrators and / or printers. Communication unit 1108 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth™ devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0126] The application program 1102b in the working memory 1102 can be loaded to execute the various methods and processes described above, for example... Figure 9 Boxes 910 to 950 are shown in the diagram. For example, in some embodiments, the multi-agent cooperative keyframe generation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1107. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 1100 via storage unit 1107 and / or communication unit 1108. When the computer program is loaded and executed by processor 1101, one or more steps of the multi-agent cooperative keyframe generation method described above can be performed. Alternatively, in other embodiments, processor 1101 can be configured to perform the multi-agent cooperative keyframe generation method by any other suitable means (e.g., by means of firmware).

[0127] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0129] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0131] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0132] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0133] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0134] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A keyframe generation method based on multi-agent cooperation, comprising: Preprocessing is performed on the raw video data to obtain standardized frame data; The multi-dimensional features extracted from the standardized frame data are fused to generate a unified state vector; Multiple independent intelligent agents perform feature analysis on the corresponding dimension based on the unified state vector to output the key frame probability of the corresponding dimension and the uncertainty corresponding to the key frame probability. The keyframe probabilities are weighted and fused to obtain a fused probability and a comprehensive uncertainty, wherein the weights in the weighted fusion are based on the uncertainty; Triggering judgment is made based on the fusion probability and the comprehensive uncertainty to generate a keyframe trigger signal, thereby generating the keyframe.

2. The method of claim 1, wherein, After generating the keyframe trigger signal, the information expression quality of the generated keyframe is evaluated. If the evaluation result does not meet the preset requirements, a supplementary keyframe is generated within the corresponding time window.

3. The method of claim 2, wherein, The information presentation quality assessment includes: Based on the current keyframe and historical state information, predict the feature representation of a preset number of subsequent frames; Calculate the reconstruction error between the predicted feature representation and the true frame feature representation; The reconstruction error is compared with a preset error threshold, and the information representation quality assessment is completed based on the comparison result, including: In response to determining that the reconstruction error is greater than a preset error threshold, it is determined that the evaluation result does not meet the preset requirements; In response to determining that the reconstruction error is less than or equal to a preset error threshold, the evaluation result is deemed to meet the preset requirement.

4. The method of claim 3, wherein, The reconstruction error includes the Euclidean distance, cosine similarity, or probability distribution difference of the feature vectors; the feature representation of the predicted subsequent preset number of frames is implemented through an autoregressive network or a gated recurrent unit.

5. The method of claim 2, wherein, After completing the information expression quality assessment, the control parameters for keyframe generation are adjusted based on the quantification results of the reconstruction error and the comparison results between the reconstruction error and the preset error threshold.

6. The method according to claim 5, wherein, The adjustment of the control parameters for keyframe generation includes performing at least one of the following operations: Based on the keyframe distribution density, the keyframe triggering threshold is dynamically adjusted to reduce the keyframe generation frequency in stable scenarios and increase the keyframe triggering probability in abnormal scenarios. Based on the historical uncertainty data of each agent, the fusion weights of each agent are dynamically adjusted to enhance the sensitivity to high uncertainty regions. Based on the reconstructed historical error data, the error threshold corresponding to the information expression quality assessment is optimized.

7. The method according to claim 3 further includes recording and updating historical state information during the keyframe generation process, wherein the historical state information includes at least one of the following: the time position and frame number of the generated keyframe, the keyframe distribution density within a preset time window, the historical uncertainty data of each agent, and the historical reconstruction error data corresponding to the keyframe quality assessment.

8. The method according to claim 1, wherein, The preprocessing of the raw video data includes: The original video data is parsed into a continuous sequence of video frames; Perform time-dimensional sampling on the continuous video frame sequence; A unified timestamp is assigned to the sampled video frames to synchronize the video frames in time. Perform image normalization processing on the time-synchronized video frames; The standardized video frames and their corresponding metadata are encapsulated into structured frame data to obtain the standardized frame data.

9. The method according to claim 8, wherein, The time dimension sampling is performed at equal intervals with a fixed frame rate.

10. The method according to claim 8, wherein, The temporal dimension sampling is a dynamic density sampling based on the difference results between adjacent frames, including: In response to determining that the inter-frame variation amplitude is greater than a preset threshold, the sampling density is increased; In response to determining that the inter-frame variation amplitude is less than a preset threshold, the sampling density is reduced.

11. The method according to claim 1, wherein, The multi-dimensional features include the basic visual features, high-level semantic features, temporal motion features, and feature space anomaly features of the standardized frame data.

12. The method according to claim 11, wherein, The fusion process includes multi-dimensional features spliced ​​together by a multi-layer perceptual network, or dynamic weighted fusion of the extracted features based on attention weight allocation.

13. The method according to claim 1, wherein, The multiple independent intelligent agents include a scene change intelligent agent, a semantic understanding intelligent agent, an anomaly detection intelligent agent, and a motion modeling intelligent agent; The scene change agent is used to analyze the low-level visual change features of video frames and output the key frame probability and uncertainty corresponding to the scene change. The semantic understanding agent is used to identify high-level semantic information of video frames and output the semantically related key frame probability and uncertainty. The anomaly detection agent is used to determine abnormal events in video frames and output the key frame probability and uncertainty corresponding to the abnormal event. The motion modeling agent is used to analyze the motion characteristics and temporal changes of targets in the video, and outputs the keyframe probability and uncertainty corresponding to motion anomalies.

14. The method according to claim 1, wherein, The keyframe probabilities and uncertainties output by all agents are encapsulated into multi-agent output structured data, which includes the current frame identifier, the keyframe probability set of each agent, and the uncertainty set of each agent.

15. The method according to claim 1, wherein, The weighted fusion of keyframe probabilities includes: The reciprocal of the uncertainty output by each agent is used as the initial weight corresponding to that agent; Normalize all initial weights to obtain the final fusion weights for each agent; Based on the final fusion weight, the keyframe probabilities of each agent are weighted and summed to obtain the fusion probability.

16. The method according to claim 1, wherein, The step of generating a keyframe trigger signal based on the fusion probability and overall uncertainty includes: A weighted combination is applied to the fusion probability and the overall uncertainty to generate a trigger decision value; The trigger decision value is compared with the keyframe trigger threshold. When the trigger decision value is greater than the keyframe trigger threshold, the keyframe trigger signal is generated.

17. The method according to claim 16, wherein, The keyframe trigger threshold is either a fixed threshold or an adaptive threshold that is dynamically adjusted based on historical state information.

18. The method according to claim 1, wherein, The raw video data includes at least one of visible light video data, infrared video data, and depth camera video data.

19. A keyframe generation device based on multi-agent cooperation, comprising: The video input module is used to perform preprocessing on the raw video data to obtain standardized frame data; The feature encoding module is communicatively connected to the video input module and is used to perform fusion processing on the multi-dimensional features extracted from the standardized frame data to generate a unified state vector. A multi-agent collaborative decision-making module is communicatively connected to the feature encoding module. It is used to perform feature analysis of the corresponding dimension based on the unified state vector by multiple independent agents, so as to output the key frame probability of the corresponding dimension and the uncertainty corresponding to the key frame probability. An uncertainty fusion module, which is communicatively connected to the multi-agent collaborative decision-making module, is used to perform weighted fusion of the keyframe probabilities to obtain a fusion probability and a comprehensive uncertainty, wherein the weights in the weighted fusion are based on the uncertainty. The joint triggering decision module is communicatively connected to the uncertainty fusion module and is used to make a triggering judgment based on the fusion probability and the comprehensive uncertainty, and generate a key frame triggering signal to generate the key frame.

20. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores a computer program that, when executed by the at least one processor, implements the method according to any one of claims 1-18.

21. A non-transitory computer-readable storage medium storing a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-18.

22. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-18.

Images