Predefined behavior identification method, system and device based on audiovisual material

Abstract

The application discloses a kind of based on audiovisual material's pre-defined behavior identification method, system and device, it is related to computer vision technical field, the method includes: when the reliability degree of edge preliminary detection result is not high, using end cloud semantic alignment encoder to frame coding is carried out to region of interest sequence, forms semantic token sequence;Semantic token includes base layer and enhancement layer;End cloud semantic alignment encoder and the token mapping dimension increasing module of cloud end are pre-completed end-to-end joint training, the output distribution of token mapping dimension increasing module is matched with the input distribution of cloud end big model;Enhancement layer is buffered in edge, base layer is uploaded to cloud end, cloud end uses token mapping dimension increasing module to base layer and is dimensioned, big model first uses base layer and carries out first level review, when the confidence of first level review is low, to edge request obtains enhancement layer and carries out second level review.The application can improve the real-time performance, accuracy of pre-defined behavior detection in audiovisual material audit.

Description

Technical Field

[0001] This invention relates to the field of computer vision and audiovisual material review technology, and in particular to a method, system and device for identifying predefined behaviors based on audiovisual materials. Background Technology

[0002] Currently, with the increasing proportion of audiovisual materials in case review and evidence verification processes, the ability to quickly and accurately review these materials and cross-verify them with on-site investigation records, witness testimonies, and other evidence to form a complete chain of evidence is a crucial basis for decision-making when taking appropriate measures. However, the current review of audiovisual materials mainly relies on prosecutors' experience to manually determine specific behaviors. This is inefficient, especially when the amount of audiovisual material is large and its sources are scattered, consuming a significant amount of prosecutors' review time. Furthermore, relying on manual experience to simultaneously cross-reference multiple types of materials often requires repeated verification, demanding high levels of experience and ability, and inexperienced prosecutors are prone to errors and omissions.

[0003] As computer vision-based behavior recognition technologies mature, they become a viable alternative to inefficient manual review. However, current solutions generally face a core contradiction in their architectural design: how to simultaneously meet the real-time requirements of detection and the high accuracy requirements of recognition. Specifically: (1) Pure edge computing solution: To meet the timeliness requirements, lightweight models are usually deployed on edge devices. This type of solution has fast response and low bandwidth consumption, but the model has limited capabilities. In complex scenarios (such as severe occlusion, sudden changes in light, blurred behavior, etc.), the accuracy drops significantly, leading to frequent false alarms (such as judging playful fighting as specific harmful behavior) and missed alarms.

[0004] (2) Traditional cloud-edge collaborative solution: In order to improve accuracy, some solutions attempt to directly upload video clips or image frames corresponding to suspected events to the cloud for in-depth analysis by large models. Although this architecture can improve accuracy, transmitting raw image or video data indiscriminately will consume a huge amount of uplink bandwidth, and the high computational latency of large cloud models will slow down the overall system response, which cannot meet the need for rapid response to the identification of specific behaviors.

[0005] (3) Fixed threshold detection scheme: Contact actions are identified by key point detection and fixed threshold. This scheme represents another common approach. However, the preset fixed recognition threshold cannot adapt to all scenarios (such as the different intensity of normal interaction between campus and shopping mall). When there are many people and the detection boxes overlap, it is difficult to effectively distinguish between general collisions and specific harmful behaviors.

[0006] (4) Solutions relying on manual review: Some solutions use audio and video fusion detection, but require operators to make a secondary judgment before triggering an alert. This solution attempts to introduce manual judgment into automatic detection to ensure accuracy, but it introduces new problems: poor timeliness and questionable reliability. The manual review process severely slows down the response speed, and the mechanism may fail at night or during unattended periods. At the same time, its accuracy relies excessively on the operator's personal experience and lacks stable assurance. Summary of the Invention

[0007] This invention provides a method for identifying predefined behaviors based on audiovisual materials at the edge, which simultaneously improves the real-time performance, accuracy, and efficiency of predefined behavior detection in audiovisual material review, while also rationally utilizing bandwidth. The method includes: A pre-trained target detection model is used to perform predefined behavior detection on audiovisual data streams and output preliminary category determination results; the target detection model is used to locate regions of interest from audiovisual data and determine preliminary category determination results based on regions of interest. Assess the reliability of the preliminary classification results; When the reliability level is higher than the set threshold, the final detection result is output based on the preliminary category determination result. When the reliability level is not higher than the set threshold, a region of interest sequence is formed based on the sequential audiovisual data of multiple frames before and after the preliminary category determination result. A pre-deployed edge-cloud semantic alignment encoder is used to encode the region of interest sequence frame by frame, forming a semantic word sequence. The semantic word includes a base layer and an enhancement layer. The base layer is a first semantic representation, and the enhancement layer is a second semantic representation. The dimension of the first semantic representation is smaller than that of the second semantic representation. Dimensions The edge-cloud semantic alignment encoder and the cloud-based lexical mapping upscaling module have completed end-to-end joint training in advance, and the output distribution of the lexical mapping upscaling module matches the input distribution of the large cloud model. The enhancement layer is cached at the edge, while the base layer is uploaded to the cloud. This allows the cloud to: use a lexical mapping dimensionality-upgrading module to upgrade the base layer, obtaining upgraded lexical units; input these upgraded lexical units into the large model, and output a verification conclusion back to the edge; the large model first uses the upgraded lexical units of the base layer for a first-level verification of category determination. When the confidence level of the first-level verification is lower than a preset arbitration threshold, the cloud requests the enhancement layer from the edge, performs a second-level verification based on the upgraded lexical units of the base layer and the enhancement layer, and outputs a verification conclusion.

[0008] This invention provides a cloud-based method for identifying predefined behaviors based on audiovisual materials, which simultaneously improves the real-time performance, accuracy, and efficiency of predefined behavior detection in audiovisual material review, while also rationally utilizing bandwidth. The method includes: The base layer receives the semantic word sequence uploaded by the edge terminal; the semantic word sequence is obtained by the edge terminal using the end-to-cloud semantic alignment encoder to encode the region of interest sequence frame by frame, and the semantic word sequence also includes an enhancement layer cached at the edge terminal; the region of interest sequence is formed based on the previous and next consecutive frames of audiovisual data associated with the preliminary category determination result when the edge terminal performs predefined behavior detection on the audiovisual data stream and the reliability of the output preliminary category determination result is not higher than a set threshold. The basic layer is upgraded using a lexical mapping dimensionality-upgrading module to obtain upgraded lexical units; the lexical mapping dimensionality-upgrading module and the end-to-end semantic alignment encoder are pre-trained together, and the output distribution of the lexical mapping dimensionality-upgrading module matches the input distribution of the large cloud model; The first-level review of category determination involves directly inputting upgraded word units into the cloud-based large model without decoding and restoration. When the confidence level of the first-level review is lower than the preset arbitration threshold, a request is made to the edge to obtain the enhancement layer; The second-level verification is performed based on the upgraded lexical units of the base layer and enhancement layer, and the verification results are output and sent back to the edge.

[0009] This invention also provides a predefined behavior recognition system based on audiovisual materials, including the aforementioned edge and cloud terminals.

[0010] This invention also provides a predefined behavior recognition device based on audiovisual materials, which simultaneously improves the real-time performance, accuracy, and efficiency of predefined behavior detection in audiovisual material review, while reasonably utilizing bandwidth. This device is applied at the edge and includes: The preliminary detection module is used to perform predefined behavior detection on the audiovisual data stream using a pre-trained target detection model and output a preliminary category determination result; the target detection model is used to locate the region of interest from the audiovisual data and determine the preliminary category determination result based on the region of interest. The preliminary test result evaluation module is used to assess the reliability of the preliminary category determination results; The results output module is used to output the final detection result based on the preliminary category determination result when the reliability level is higher than the set threshold. The semantic lexical encoding module is used to form a region of interest (ROI) sequence based on multiple consecutive frames of audiovisual data associated with the preliminary category judgment results, when the reliability level is not higher than a set threshold. The ROI sequence is then encoded frame-by-frame using a pre-deployed edge-cloud semantic alignment encoder to form a semantic lexical sequence. The semantic lexical sequence includes a base layer and an enhancement layer. The base layer is a first semantic representation, and the enhancement layer is a second semantic representation. The dimension of the first semantic representation is smaller than the dimension of the second semantic representation. The edge-cloud semantic alignment encoder and the cloud-based lexical mapping dimensionality enhancement module are pre-trained end-to-end, and the output distribution of the lexical mapping dimensionality enhancement module matches the input distribution of the large cloud model. The edge-cloud communication module is used to cache the enhancement layer at the edge and upload the base layer to the cloud, so that the cloud: uses the lexical mapping dimensionality-upgrading module to upgrade the base layer to obtain upgraded lexical units; inputs the upgraded lexical units into the large model, and outputs the verification conclusion back to the edge; the large model first uses the upgraded lexical units of the base layer to perform a first-level verification of category determination. When the confidence of the first-level verification is lower than a preset arbitration threshold, the cloud requests the enhancement layer from the edge, performs a second-level verification based on the upgraded lexical units of the base layer and the enhancement layer, and outputs the verification conclusion.

[0011] This invention also provides a predefined behavior recognition device based on audiovisual materials, which simultaneously improves the real-time performance, accuracy, and efficiency of predefined behavior detection in audiovisual material review, while reasonably utilizing bandwidth. This device is applied in the cloud and includes: The data receiving module is used to receive the base layer of semantic word sequence uploaded by the edge terminal; the semantic word sequence is obtained by the edge terminal using the end-to-cloud semantic alignment encoder to encode the region of interest sequence frame by frame, and the semantic word sequence also includes an enhancement layer cached at the edge terminal; the region of interest sequence is formed based on the previous and next consecutive frames of audiovisual data associated with the preliminary category judgment result when the edge terminal performs predefined behavior detection on the audiovisual data stream and the reliability of the output preliminary category judgment result is not higher than a set threshold. The dimensionality-upgrading module is used to upgrade the base layer using the lexical mapping dimensionality-upgrading module to obtain upgraded lexical units; the lexical mapping dimensionality-upgrading module and the end-to-end semantic alignment encoder are pre-trained together, and the output distribution of the lexical mapping dimensionality-upgrading module matches the input distribution of the large cloud model; The composite judgment module is used to input the upgraded words into the large model for the first-level review of category judgment; when the confidence of the first-level review is lower than the preset arbitration threshold, it requests the edge end to obtain the enhancement layer; based on the upgraded words of the base layer and the enhancement layer, the second-level review is performed, and the review conclusion is output and sent back to the edge end.

[0012] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for identifying predefined behaviors based on audiovisual materials.

[0013] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for identifying predefined behaviors based on audiovisual materials.

[0014] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for identifying predefined behaviors based on audiovisual materials.

[0015] In this embodiment of the invention, a preliminary predefined behavior determination is first performed at the edge using an object detection model. If the preliminary category determination result is reliable, the result is directly output; if the preliminary category determination result is unreliable, it is determined that a review request needs to be made to the large model in the cloud. First, the region of interest sequence is extracted, and then the region of interest sequence is encoded frame by frame using a pre-deployed edge-cloud semantic alignment encoder to form a semantic lexical sequence. This semantic lexical sequence includes a base layer and an enhancement layer. The edge-cloud semantic alignment encoder and the lexical mapping dimensionality-upgrading module in the cloud are pre-trained end-to-end. The output distribution of the lexical mapping dimensionality-upgrading module matches the input distribution of the large model in the cloud. The enhancement layer is cached at the edge, and the base layer is uploaded to the cloud. In the cloud, the lexical mapping dimensionality-upgrading module is used to upgrade the base layer, and the upgraded lexical sequence does not require decoding and other operations. It is directly input into the large model. The large model first uses the base layer for a first-level review. When the confidence of the first-level review is lower than a preset arbitration threshold, the cloud requests the enhancement layer from the edge and performs a second-level review based on the base layer and the enhancement layer. In this embodiment of the invention, data is uploaded to the cloud only when the initial judgment result at the edge is unreliable, and not all data is uploaded to the cloud. Instead, a base layer and an enhancement layer are encoded. Initially, only the base layer is uploaded to the cloud. When the large cloud model determines that an enhancement layer is needed, the enhancement layer is then uploaded to the cloud, thus rationally utilizing bandwidth and helping to improve processing efficiency. Simultaneously, this embodiment of the invention establishes a decoding-free direct connection path. By encoding visual information at the edge into semantic visual units aligned with the embedding space of the large cloud model, the visual information at the edge can be directly understood by the large cloud visual language model after only one encoding. The cloud performs zero decoding and reconstruction of the uploaded data and zero secondary feature extraction, directly improving the real-time performance, accuracy, and efficiency of predefined behavior detection in audiovisual material review. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart illustrating the predefined behavior identification method based on audiovisual materials in an embodiment of the present invention. Figure 2 This is a schematic diagram of interface alignment distillation training in an embodiment of the present invention; Figure 3 This is a schematic diagram of closed-loop self-iteration in an embodiment of the present invention; Figure 4 This is a schematic diagram of adaptive adjustment of entropy gate threshold in an embodiment of the present invention; Figure 5 This is a specific example diagram of the predefined behavior identification method based on audiovisual materials in an embodiment of the present invention; Figure 6 This is another flowchart illustrating the predefined behavior identification method based on audiovisual materials in an embodiment of the present invention; Figure 7 This is a schematic diagram of a predefined behavior recognition device based on audiovisual materials in an embodiment of the present invention; Figure 8 This is another schematic diagram of the predefined behavior recognition device based on audiovisual materials in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0018] To facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order.

[0019] In summary, existing technologies suffer from the following structural defects when addressing the core issue of "collaboration between real-time edge detection and high-precision cloud-based verification": Cloud-based verification is costly: Existing cloud-edge collaboration solutions generally adopt the raw data reporting method of "transmitting images and videos", which results in huge bandwidth consumption and high transmission latency, severely restricting the feasibility of large cloud models participating in real-time verification.

[0020] The review trigger strategy is rigid: existing solutions lack an effective quantification and intelligent decision-making mechanism for the uncertainty of edge detection results. They either blindly report all suspected events, wasting resources, or rely on fixed thresholds or manual judgment, lacking adaptive capabilities.

[0021] The capability gap between edge and cloud models is as follows: the detection boxes or low-level features output by edge models are difficult to be directly and effectively utilized by large cloud models. As a result, cloud verification still needs to rely on the original image data, which needs to be decoded after uploading. Large cloud models cannot consume this data directly.

[0022] The system lacks self-iteration and consistency maintenance capabilities: existing systems are all "open-loop" structures, which cannot use the high-quality labels generated by cloud verification to continuously optimize the edge model, resulting in the solidification of the edge model's performance limit; even if the model is updated, there is no mechanism to maintain the consistency of the representation space on both the edge and cloud sides, and the edge-cloud interface will gradually become mismatched as the model evolves.

[0023] Therefore, this invention proposes a method and apparatus for predefined behavior identification based on audiovisual materials, which solves the following technical problems: (1) Interface problem: The visual features at the edge and the large model embedding space in the cloud are naturally heterogeneous. Existing solutions either transmit the original image (high bandwidth cost) or transmit the compressed representation that needs to be decoded (the large model cannot be consumed directly and must be decoded and restored and the features extracted a second time, which introduces double loss). (2) Zero adaptation problem: Fine-tuning and inserting adaptation layers to modify large models means high cloud training costs and version management burden, and damages the universality and upgradeability of large models, which is unacceptable from an engineering perspective. (3) Consistency maintenance problem: If the edge encoder continues to learn and evolve, its output distribution will inevitably drift, the semantic interface between the edge and the cloud will gradually become mismatched, and the open-loop system cannot maintain the long-term consistency of the edge-cloud representation space.

[0024] The predefined behavior identification method and apparatus based on audiovisual materials in this embodiment of the invention are specifically a predefined behavior detection and identification method and apparatus based on edge-cloud feature space interface alignment, which can achieve the following: Establish a direct connection without decoding: By encoding visual information at the edge as semantic visual tokens that are aligned with the embedding space of the large cloud model, the visual information at the edge can be directly understood by the large cloud visual language model with frozen parameters after only one encoding. The cloud requires zero adaptation and modification to the large model, zero decoding and restoration of uploaded data, and zero secondary feature extraction. Establish an interface-aligned distillation method: Train an edge encoder that does not have independent recognition capabilities and is dedicated to the visual front end of large cloud models, so that its output token is consistent with the teacher's embedding space in three aspects: geometric structure, statistical distribution and downstream understanding behavior. Establish a long-term maintenance mechanism for the consistency of the representation space: use cloud verification results to select high-value samples to drive the incremental evolution of the edge encoder, while use distribution offset quantization monitoring and lightweight calibration to ensure that the edge evolution does not damage the end-cloud interface; and reuse the entropy gating on-demand scheduling mechanism under the constraints of the above interface alignment architecture, triggering cloud verification only when the uncertainty at the edge exceeds the limit, so as to achieve a comprehensive balance between bandwidth consumption, response latency and verification accuracy.

[0025] First, the technical terms used in the embodiments of this invention will be explained.

[0026] Semantic Visual Token: A low-dimensional vector sequence that is aligned with the input space of the visual embedding layer of the cloud-based visual language large model, obtained by the edge-cloud semantic alignment encoder in this embodiment of the invention. It represents the semantic features of key objects (ROIs) in the image rather than the original pixel data, has a very low data volume, and can be directly processed by the cloud-based large model without any decoding or reconstruction.

[0027] Scalable Semantic Token: A hierarchical semantic feature representation consisting of a base layer and an enhancement layer. The base layer is a low-dimensional, compact representation that carries core semantics such as target category, overall pose, and scene context, and is uploaded by default. The enhancement layer is a high-dimensional, fine-grained representation that carries subtle actions, texture features, and interaction details. It is temporarily stored at the edge and is only requested for uploading as needed when the amount of information for cloud verification is insufficient, allowing the transmission bandwidth to adaptively scale with the difficulty of the samples.

[0028] Edge-Cloud Semantic Alignment Encoder: Deployed at the edge, this lightweight encoder uses the visual encoder of a large multimodal model in the cloud as its teacher model and is trained through interface alignment distillation. It does not contain an independent classification head and does not independently perform behavior recognition tasks. Its sole function is to encode visual information into semantic visual tokens that the large cloud model can directly understand, thus acting as a "dedicated visual front-end" for the large cloud model.

[0029] Entropy Gating Module: A logic module at the edge used to assess the uncertainty of detection results. By calculating the entropy value of the detection probability distribution, it decides whether the sample needs to be uploaded to the cloud for verification.

[0030] Token Mapping Upscaling Module: Deployed at the front end of the large cloud model, this module is responsible for mapping and aligning low-dimensional semantic tokens uploaded from the edge to the dimensions required for the visual embedding layer input of the large cloud model. During the distillation training phase, this module is jointly trained end-to-end with the edge-cloud semantic alignment encoder, and its output is used directly as the input to the visual embedding layer of the large model without image decoding and reconstruction.

[0031] Cloud-based large-scale model: A multimodal large-scale language model with extremely strong semantic understanding and logical reasoning capabilities. In this embodiment of the invention, the parameters of the cloud-based large-scale model are frozen throughout the process, without fine-tuning, insertion of adaptation layers, or modification of the structure.

[0032] Interface-aligned distillation: This invention proposes a dedicated distillation method for feature space interface alignment. Unlike traditional model compression distillation, which aims to have the student model independently replace the teacher model to complete the task, the training objective of interface-aligned distillation is to ensure that the projected features of the student model output are consistent with the teacher model's embedding space in terms of geometric structure, statistical distribution, and downstream understanding behavior.

[0033] Direct connection without decoding: The semantic visual token uploaded from the edge is directly injected into the visual embedding layer of the visual language big model in the cloud to complete the data path for inference without undergoing image reconstruction, pixel decoding or secondary feature extraction.

[0034] Cross-modal semantic anchors: The category prototype vectors generated by the cloud-based visual language large model text encoder for behavioral category text are used as semantic reference points in the feature space; by constraining the similarity distribution structure of student features relative to each anchor point to be consistent with the teacher features, cross-modal semantic structure alignment is achieved.

[0035] Conformal Gating: A statistical gating mechanism based on conformal prediction. The (1-α) quantile of the inconsistency score is calculated using a calibration sample set to determine the cloud-based review trigger threshold, ensuring that the trigger rate has a provable statistical coverage guarantee without relying on manual threshold adjustment based on experience.

[0036] Predefined behavior: refers to a category of target behavior that is pre-configured according to the application scenario and needs to be detected and identified by the system; for example, in a judicial review scenario, predefined behavior can be a specific harmful behavior. In this embodiment of the invention, "specific behavior," "specific harmful behavior," and "predefined behavior" have the same meaning and all refer to the aforementioned pre-configured category of target behavior.

[0037] Figure 1 This is a flowchart illustrating the predefined behavior identification method based on audiovisual materials in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes: Step 101: Use a pre-trained target detection model to perform predefined behavior detection on the audiovisual data stream and output preliminary category determination results; the target detection model is used to locate the region of interest from the audiovisual data and determine the preliminary category determination results based on the region of interest. Step 102: Assess the reliability of the preliminary category determination results; Step 103: When the reliability level is higher than the set threshold, output the final detection result based on the preliminary category determination result; Step 104: When the reliability level is not higher than the set threshold, form a region of interest sequence based on the consecutive audiovisual data of multiple frames before and after the preliminary category determination result; Step 105: Encode the region of interest sequence frame by frame using a pre-deployed edge-cloud semantic alignment encoder to form a semantic word sequence; the semantic word includes a base layer and an enhancement layer, the base layer is a first semantic representation, and the enhancement layer is a second semantic representation, the dimension of the first semantic representation is smaller than the dimension of the second semantic representation; the edge-cloud semantic alignment encoder and the word mapping dimensionality enhancement module in the cloud are pre-trained end-to-end, and the output distribution of the word mapping dimensionality enhancement module matches the input distribution of the large model in the cloud; Step 106: Cache the enhancement layer at the edge and upload the base layer to the cloud, so that the cloud: uses the lexical mapping dimensionality-upgrading module to upgrade the base layer to obtain upgraded lexical units; inputs the upgraded lexical units into the large model and outputs the verification conclusion back to the edge; the large model first uses the upgraded lexical units of the base layer to perform the first-level verification of category determination. When the confidence of the first-level verification is lower than the preset arbitration threshold, the cloud requests the edge to obtain the enhancement layer, performs the second-level verification based on the upgraded lexical units of the base layer and the enhancement layer, and outputs the verification conclusion.

[0038] The following is a detailed description of the predefined behavior identification method based on audiovisual materials in the embodiments of the present invention.

[0039] Step 1: Preliminary edge detection and entropy calculation.

[0040] Before using a pre-target detection model to perform predefined behavior detection on the audiovisual data stream, video frames are acquired and preprocessed. Edge devices continuously acquire the video stream and perform necessary preprocessing on the video frames, such as size scaling and illumination normalization.

[0041] Then, object detection and region of interest (ROI) extraction are performed. The preprocessed image frames are input into a pre-trained lightweight object detection model, which locates key targets (such as people and objects) in the image and generates one or more ROIs. For each ROI, the object detection model outputs a preliminary category determination. This preliminary category determination may include, for example, multiple categories, the confidence score for each category, and the corresponding object tracking identifier. Specifically, the object detection model may include an object localization branch and a behavior classification branch: the object localization branch locates key targets in the image and generates ROIs, while the behavior classification branch outputs the confidence scores for each candidate behavior category based on the visual features of the ROIs, serving as the preliminary category determination.

[0042] Next, a reliability assessment is performed on the preliminary category determination results. This can be based on confidence levels or an assessment of uncertainty quantification.

[0043] In one embodiment, the reliability level is quantified by information entropy; step 102, which evaluates the reliability level of the preliminary category determination result, may include: Calculate the information entropy using the following formula: (1) in, Information entropy (calculated as a base-2 logarithm). The normalized confidence score for the i-th category predicted by the object detection model, where n represents the number of initial category determinations; The information entropy is compared with a set threshold to assess the reliability of the preliminary category determination result; the set threshold is calibrated offline using a conformal prediction method.

[0044] In this example, an entropy gating module is constructed to perform entropy gating decision-making. Based on the input data stream, the confidence score distribution output by the target detection model can be obtained. The entropy gating module receives the confidence score distribution output by the target detection model and calculates the information entropy of the distribution according to the formula (1) above. The higher the information entropy value H, the lower the confidence of the target detection model in the detection result, that is, the greater the uncertainty.

[0045] In one embodiment, the set threshold is calibrated offline using the following conformal prediction method: A calibration sample set covering various typical scenarios is collected at the edge and input into the target detection model. The information entropy H of each sample calculated according to formula (1) is used as the non-consistency score (the higher the entropy value, the greater the uncertainty of the target detection model in judging the sample). The (1-α) quantile of the non-consistency score of the calibration sample set is taken as the trigger threshold (i.e., the set threshold), where α is the preset misjudgment tolerance (e.g., 0.05). Thus, the cloud review trigger rate has a provably guaranteed coverage in a statistical sense. The error rate of the conclusion directly determined at the edge is bounded by α, does not rely on manual experience to adjust the threshold, and can adapt to scene changes as the calibration set is updated.

[0046] Scenario 1: Low entropy (H ≤ set threshold): This indicates that the target detection model has a high degree of confidence in the results (e.g., clearly identifying it as normal activity). Edge detection results are directly adopted without triggering a cloud-based review process to save resources.

[0047] Scenario 2: High entropy (H > set threshold): This indicates high uncertainty in the detection result (e.g., an ambiguous posture between normal behavior and a specific behavior), posing a risk of misjudgment. In this case, the gating decision is "report," and the process proceeds to the next step.

[0048] Step 2: Temporal semantic alignment, token generation, and accompanying tag upload.

[0049] In one embodiment, when the reliability level is not higher than a set threshold, forming a region of interest sequence based on the sequential audiovisual data of multiple frames before and after the preliminary category determination result may include: associating the regions of interest of the same target in K consecutive frames before and after the current frame with the target tracking identifier to form a temporal region of interest sequence, where K is a positive integer; Furthermore, the semantic lexical sequence includes temporal lexical groups. The semantic lexical sequence is formed by encoding the region of interest sequence frame by frame using a pre-deployed end-to-cloud semantic alignment encoder. This can include: using a pre-deployed end-to-cloud semantic alignment encoder to aggregate the semantic lexical groups of each frame in the temporal region of interest sequence into temporal lexical groups after encoding them in temporal order and superimposing their temporal positions. The process of uploading the base layer to the cloud includes uploading the base layer of semantic lexical units of each frame in the temporal lexical group to the cloud.

[0050] In practice, firstly, Regions of Interest (ROIs) are cropped and associated with multiple frames. Based on the target tracking identifiers associated with the target in the preliminary category determination results, a sequence of ROIs is formed from the consecutive audiovisual data of the preceding and following frames. According to the ROI coordinates output by the target detection model, the corresponding regions are cropped from the original image frames. At the same time, based on the target tracking identifiers, the ROIs of the same target in the K consecutive frames before and after the current frame are associated to form a temporal ROI sequence.

[0051] Then, frame-by-frame hierarchical semantic alignment encoding is performed. The temporal ROI sequence is encoded frame-by-frame using a pre-deployed edge-cloud semantic alignment encoder to form a semantic token sequence. This semantic token sequence includes a base layer and an enhancement layer. The base layer is a first semantic representation, such as a low-dimensional compact semantic representation, and the enhancement layer is a second semantic representation, such as a high-dimensional fine-grained semantic representation.

[0052] For example, each frame of the ROI image in the temporal ROI sequence is input into the edge-cloud semantic alignment encoder, and encoded frame by frame into a scalable semantic token aligned with the embedding space of the large model in the cloud. This scalable semantic token adopts a hierarchical structure: the base layer is a low-dimensional compact semantic representation (e.g., 64-dimensional), carrying core semantics such as target category, overall pose, and scene context; the enhancement layer is a high-dimensional fine-grained semantic representation (e.g., 192-dimensional), carrying subtle actions, texture features, and interaction details. The base layer and the enhancement layer can be concatenated in the embedding space to restore the complete semantic token, and the sum of their dimensions equals the dimension of the standard semantic token (e.g., 256-dimensional).

[0053] Before uploading to the cloud, temporal token aggregation is performed: the K-frame semantic token sequence is arranged in chronological order, superimposed with temporal position encoding, and aggregated into a temporal token group. The identification of specific behaviors is inherently temporally dependent: "intense play" and "specific harmful behavior" in a single frame are highly similar in posture, but their action evolution trajectories (direction of force, duration pattern, and interaction rhythm) have distinguishable differences. The temporal token group enables the large cloud model to use this difference for discrimination.

[0054] In step 106, the enhancement layer is cached at the edge, while the base layer is uploaded to the cloud. By default, the edge first uploads the base layer of each frame's token, and the enhancement layer is temporarily stored in the local cache at the edge, waiting for the cloud to request it as needed.

[0055] In one embodiment, caching the enhancement layer at the edge and uploading the base layer to the cloud may include: uploading the base layer and metadata tags to the cloud so that the cloud: compiles the metadata tags into structured text hints, which are then input into the large model along with the upgraded lexical units; the metadata tags include preliminary category determination results, confidence levels, and entropy values.

[0056] In this example, the edge device only uploads the basic layer and metadata tags of each frame's token by default. The metadata tags include, but are not limited to, the entropy value of the sample, the preliminary category identified by the edge device, and its confidence level. The enhancement layer is temporarily stored in the local cache of the edge device, waiting for the cloud to request it as needed.

[0057] Furthermore, in one embodiment, the number of semantic units of the base layer uploaded to the cloud is positively correlated with the information entropy. The higher the information entropy, the more semantic units of the base layer are uploaded, and the higher the number of frames and spatial granularity are retained.

[0058] For example, uploading the base layer to the cloud includes an adaptive token budget, where the number of uploaded base layer tokens adjusts adaptively with the entropy value. The number of uploaded base layer semantic tokens is directly proportional to the entropy value: the higher the entropy value, the larger the allocated token budget, and the higher the number of frames and spatial granularity retained; when the entropy value is low, the token budget is compressed. The uploaded content does not contain the original image, and the data volume is only a fraction of that of the original image.

[0059] Step 3: Cloud-based large model verification without decoding.

[0060] Token mapping upscaling: After receiving the low-dimensional token sequence, the cloud first sends it to the token mapping upscaling module. This module has been jointly trained end-to-end with the edge-cloud semantic alignment encoder during the distillation training phase, and its output distribution falls within the input distribution of the large model's visual embedding layer. The upscaled tokens are not decoded and restored from the image, but are directly used as the input of the large model's visual embedding layer.

[0061] Confidence Context Injection: The cloud compiles the metadata tags uploaded from the edge into structured text hints, which are then input into the large model along with the upgraded token. For example, when the metadata indicates that the initial edge judgment has low confidence and high entropy, the text hint is: "The initial edge judgment tends to be a specific behavior but the uncertainty is high. Please focus on checking the force exertion method and the interaction rhythm between the two parties to determine whether it constitutes a specific behavioral event." This mechanism allows the large model to explicitly know the uncertainty state of the edge during inference, and to adopt a more cautious inference strategy for low-quality input.

[0062] Multimodal reasoning: A parameter-frozen visual language big model performs cross-modal reasoning on temporal token groups and structured text prompts, analyzing subtle actions, character relationships, temporal evolution, and scene context.

[0063] Two-level review and on-demand enhancement layer requests: The cloud-based large model first performs a first-level review based solely on the base layer token group. If the cloud confidence score output by the first-level review is higher than a preset arbitration threshold, a conclusion is directly output. If the confidence score is insufficient, the cloud sends an enhancement layer request to the edge. This enhancement layer request carries the frame index and region range required by the cloud-based large model. The edge uses this information to send back the corresponding enhancement layer token. The cloud then concatenates the base layer and enhancement layer tokens to restore the complete semantic token before performing the second-level review. Thus, simple samples complete the review with minimal bandwidth, while only difficult samples trigger incremental transmission, achieving a two-level review protocol where bandwidth scales with sample difficulty.

[0064] Structured conclusion output, criterion location feedback and edge arbitration: The visual language big model outputs structured review conclusions, including the identification category ("is a predefined behavior", "is not a predefined behavior" or "cannot be determined"), cloud confidence and key criterion summary, and outputs criterion location information, namely the key frame index and key token index on which the conclusion is based.

[0065] In one embodiment, the method further includes: receiving structured review conclusions and criterion location information transmitted back from the cloud; the criterion location information includes the key frame index and key semantic term index on which the review conclusion is based, and superimposing the time position and image region corresponding to the key criterion on the audiovisual data.

[0066] For example, the structured review conclusions and criterion location information mentioned above are sent to the edge end via the feedback module. The edge end arbitration module generates the final determination according to the preset fusion strategy: when the confidence level in the cloud is higher than the initial confidence level in the edge end, the cloud conclusion shall prevail; otherwise, the fusion shall be weighted according to the confidence level. At the same time, the time location and image area corresponding to the key criteria are superimposed and marked on the original video for reviewers to verify, so that the review conclusions are interpretable and the criteria can be fixed.

[0067] Figure 2This is a schematic diagram of interface-aligned distillation training in an embodiment of the present invention. The following is in conjunction with... Figure 2 This paper introduces the model training method (interface-aligned distillation) in the embodiments of the present invention.

[0068] The purpose of distillation in this embodiment of the invention is not model compression (i.e., to allow student models to complete recognition tasks independently), but to train a semantic "translator" so that the tokens generated at the edge can be aligned with the internal representation depth of the large cloud model in the feature space, thereby being directly understood by the large cloud model.

[0069] In one embodiment, the large model includes a visual encoder; the visual encoder is used to map input image features to semantic features; The visual encoder is used as the teacher model, and the convolutional neural network or visual transformer ViT is used as the student model. The edge-cloud semantic alignment encoder is obtained through knowledge distillation. The input of the edge-cloud semantic alignment encoder is the region of interest sequence, and the output is the semantic lexical units after encoding and dimensionality reduction.

[0070] For example, when selecting a teacher model, the visual encoder portion of a large visual language model deployed in the cloud is chosen as the teacher model. This visual encoder can map an input image to high-dimensional semantic features. ,in The dimension of the visual embedding layer of the teacher model (e.g., 1024 dimensions).

[0071] When constructing the student model, a lightweight convolutional neural network or Vision Transformer is designed as the student model, i.e., an edge-cloud semantic alignment encoder. Its input is a cropped ROI image, and its output is a low-dimensional semantic visual token sequence. The student model does not contain any classification head or task head, structurally excluding the possibility of it independently performing recognition tasks.

[0072] In one embodiment, the edge-cloud semantic alignment encoder and the cloud-based lexical mapping upscaling module are jointly trained end-to-end in the following manner: Construct a projection head with a multilayer perceptron structure; the output dimension of the projection head matches the input dimension requirement of the large model visual embedding layer. During the knowledge distillation training phase, the lexical mapping upscaling module is connected to the training loop to achieve end-to-end joint optimization of the lexical mapping upscaling module, the student model, and the projection head.

[0073] In this embodiment, a projection head with a multilayer sensor structure was constructed. It contains at least one hidden layer and a LayerNorm normalization layer, and a projection head. Mapping low-dimensional student features back to the same dimensional space as teacher features: (2) in, , The semantic tokens output by the student model and the semantic tokens mapped by the projection head are respectively.

[0074] Projector head output dimensions It strictly matches the input dimension requirements of the visual embedding layer of the cloud-based visual language large model, and the output feature distribution is aligned with the mean and variance of the teacher model feature distribution after statistical normalization.

[0075] The Token mapping upscaling module is integrated into the training loop during the distillation training phase, and is jointly optimized end-to-end with the student model and the projection head to ensure that its output distribution falls within the input distribution of the large model visual embedding layer in the cloud.

[0076] In one embodiment, the method further includes: during the knowledge distillation training phase, constraining the semantic lexical generation target based on the Information Bottleneck principle in the following manner: While minimizing the mutual information between the generated semantic lexical and the input region of interest, the goal is to maximize the mutual information retention of the semantic lexical on the output features of the teacher model.

[0077] In this example, the generation objective of semantic tokens can be formalized using the information bottleneck principle: under the constraint of minimizing the token's information content I(T;X_roi) of the input ROI, maximize the preservation of the token's feature information content I(T;F_teacher) of the teacher model. That is, the optimization objective is min I(T;X_roi) - β·I(T;F_teacher), where β is a tradeoff coefficient, T is the semantic lexical unit generated by the student model, X_roi is the input region of interest image, F_teacher is the feature output by the teacher model, I(T;X_roi) is the mutual information between T and X_roi, I(T;F_teacher) represents the mutual information between T and F_teacher, and min denotes minimization. This formalization shows that the base layer of the hierarchical token corresponds to a semantic backbone with high mutual information and low bitrate, while the enhancement layer corresponds to progressively supplemented detailed information. From an information theory perspective, this ensures the design legitimacy of "minimum bandwidth carrying maximum semantics" and provides a theoretical basis for entropy-adaptive token budgeting.

[0078] In one embodiment, the method further includes: during the knowledge distillation training phase, calculating multiple losses using the outputs of the teacher model and the student model, weighting and fusing the various losses to obtain the total loss, iteratively updating the student model parameters based on the total loss, and iterating repeatedly until training is completed; the multiple losses include, but are not limited to, semantic alignment loss, behavioral consistency loss, and anchor structure loss.

[0079] For example, by inputting the same batch of images into the teacher model and the student model respectively, we can obtain... and The semantic alignment loss is expressed as follows: (3) in, The first term represents the semantic alignment loss. The mean squared error loss constrains the geometric alignment of the feature space; the second term For distribution alignment loss, Maximum Mean Discrepancy (MMD) or Contrastive Learning Loss (InfoNCE) is used to constrain the probability distributions of teacher and student features to be consistent. , The output obtained by inputting the same batch of images into the teacher model and the student model respectively. To balance the hyperparameters.

[0080] The behavioral consistency loss is represented as follows: (4) in, The loss represents the behavioral consistency, and KL() is the KL divergence calculation function. ( ) represents the softmax function. , The unnormalized raw scores (logits) of the teacher model and student model after projection are input into the cloud-based large model language decoder. The temperature coefficient is used to ensure that edge tokens, after being processed in the cloud, trigger semantic understanding behaviors consistent with the teacher model.

[0081] Further, cross-modal semantic anchor structure distillation is introduced: the text descriptions of each behavior category (such as "specific harmful behavior", "normal play", "undetermined") are input into the text encoder of the cloud-based large model to generate a set of category prototype vectors as semantic anchors; that is, the anchors are the category prototype vectors generated by the large model from the text descriptions of each behavior category. The anchor structure loss is represented by: KL divergence constraining the structure of two relative similarity distributions to be consistent; the two relative similarity distributions include the similarity distribution of teacher visual features relative to each anchor and the similarity distribution of student visual features relative to each anchor.

[0082] During implementation, the similarity distributions of teacher visual features and student visual features relative to each anchor point are calculated separately. The KL divergence constraint ensures that the structures of the two relative similarity distributions are consistent, and this is denoted as the anchor point structure loss. This loss does not directly constrain the absolute coordinates of the features, but rather constrains the relative structure of the features in the semantic anchor reference system, aligning student features with teacher features at the level of cross-modal semantic relations, and making it more robust to perturbations in the absolute position of features caused by changes in edge viewpoints and lighting.

[0083] The total loss function is expressed as follows: (5) The parameters of the student model and projector head are updated through backpropagation, while the teacher model parameters are frozen. , These are the weights.

[0084] In one embodiment, after knowledge distillation training is completed, interface alignment consistency verification is performed. Interface alignment consistency verification includes: processing the same batch of test images through both the student path and the teacher path, then inputting them into a large model with the same parameters; calculating the consistency rate of the output conclusions of the two paths; if the consistency rate exceeds a preset proportion (e.g., 95%), deployment of the student model is allowed; otherwise, knowledge distillation training is repeated. The student path includes encoding semantic lexical sequences and lexical mapping dimensionality upscaling, while the teacher path includes teacher model encoder processing.

[0085] Through the feature space alignment training described above, the student model is trained into a dedicated encoder capable of "translating" edge visual information into the semantic language of the large cloud model. This edge-cloud semantic alignment encoder does not have independent recognition capabilities; its sole function is to provide standardized visual semantic input for the large cloud model.

[0086] Furthermore, this embodiment of the invention proposes a fourth step: maintaining consistency of the representation space and closed-loop self-iteration.

[0087] In one embodiment, the audiovisual materials data that have been reviewed in the cloud and their associated information are stored in an edge-cloud collaborative database; the associated information includes the original region of interest sequence, semantic word sequence, cloud review conclusion labels, and reference features output by the teacher model; When the preset period is reached or the number of stored samples reaches the preset threshold, an incremental training set is constructed based on the stored samples. The multiple losses are reused and elastic weights are added to consolidate the EWC regularization term, and the edge-cloud semantic alignment encoder and projection head are jointly fine-tuned.

[0088] Furthermore, after joint fine-tuning, the method may also include: A standard test lexicon set is uploaded to the cloud so that the cloud can detect the input distribution offset of the lexicon mapping dimensionality upgrade module. When the input distribution offset exceeds the offset threshold, the cloud performs a lightweight calibration on the lexicon mapping dimensionality upgrade module. The lightweight calibration only updates the batch normalization parameters. And / or, adjust the set threshold according to the high-entropy sample trigger rate feedback that triggers cloud review. When the trigger rate is continuously higher than the upper limit of the trigger rate, the set threshold is increased; when the trigger rate is continuously lower than the lower limit of the trigger rate, the set threshold is decreased.

[0089] 1. Establish an edge-cloud collaborative database to store high-value samples. All high-entropy samples verified by the cloud (i.e., samples where H > a set threshold) and their associated information, including the original ROI image, are included. Token sequences generated at the edge Cloud-based review conclusion label and the reference features output by the teacher model. As a "high-value sample pair" ( , , , Stored in an edge-cloud collaborative database.

[0090] The screening criteria for high-value samples include: entropy value exceeding a dynamically set threshold; inconsistency between cloud verification conclusions and edge detection results; and cloud verification confidence level higher than a preset standard.

[0091] 2. Edge Model Incremental Update: The edge model update process is triggered periodically (e.g., weekly) or when the accumulated number of high-value samples reaches a preset threshold (e.g., 1000 samples). Data preparation: Extract recent samples from the high-value sample library, weight them according to time decay (recent samples have higher weight), and construct an incremental training set; Joint optimization: Fix the parameters of the large model in the cloud and jointly fine-tune the semantic alignment encoder (student model) and the projection head at the edge; the fine-tuning uses the same composite loss function of the interface alignment distillation to ensure that the output of the evolved encoder still meets the alignment constraints with the embedding layer; Loss Design: Using the total loss function In addition, an Elastic Weight Consolidation (EWC) regularization term is added to prevent the model from forgetting its mastered ability to discriminate simple scenes when adapting to new and complex scenes. (6) in, This represents the updated total loss function. For the diagonal elements of the Fisher information matrix, The parameters to be updated are the current model parameters (specifically, the parameters of the edge-cloud semantic alignment encoder and the projection head). These are the historically optimal parameters. This represents the regularity strength.

[0092] 3. Cloud-based Token Mapping Upgrade Module Synchronous Calibration: After the edge model is updated, the input distribution of the cloud-based Token Mapping Upgrade Module may shift. The following mechanism maintains consistency between the edge and cloud: When the edge model is updated, a batch of standard test tokens are uploaded synchronously. The cloud-based Token mapping upscaling module detects input distribution offsets. If the offset exceeds the offset threshold, it triggers a lightweight adaptive calibration of the Token mapping upscaling module (only updating the batch normalization parameters of the mapping layer).

[0093] 4. Closed-loop formation and performance evolution: Reference Figure 3 This forms a complete closed loop: "initial screening of the margins → entropy-gated decision-making and routing → cloud-based decoding-free verification and precise judgment → accumulation of high-value samples → evolution of the margin model → MMD offset monitoring, lightweight BN (Batch Normalization) calibration, and dynamic adjustment of entropy-gated thresholds." Figure 3 This is a schematic diagram of closed-loop self-iteration in an embodiment of the present invention. Figure 4 This is a schematic diagram of adaptive adjustment of entropy gate threshold in an embodiment of the present invention, for reference. Figure 4 The horizontal axis represents the system running time, and the vertical axis represents the high-entropy sample trigger rate. When the trigger rate is continuously higher than the upper limit of the target interval (i.e., the upper limit of the trigger rate), the entropy threshold is increased (i.e., the set threshold is set). When the trigger rate is continuously lower than the lower limit of the target interval (i.e., the lower limit of the trigger rate), the entropy threshold is decreased.

[0094] The methods, apparatus, and systems constructed according to the embodiments of the present invention possess long-term self-iterative capabilities: As runtime increases, the edge model's ability to discriminate complex scenes continues to improve; The trigger rate of high-entropy samples gradually decreases, and the consumption of cloud-based review resources decreases. Overall performance is constantly approaching the level of large cloud models, while maintaining real-time responsiveness at the edge.

[0095] Figure 5 This is a specific example diagram of a predefined behavior identification method based on audiovisual materials in an embodiment of the present invention, as shown below. Figure 5 As shown, this method demonstrates the overall implementation process, including: Step 1: Video frame acquisition and preprocessing; Step 2: Target detection and region of interest extraction, output including confidence distribution; Step 3: Entropy-gated decision-making, calculate information entropy H; Step 4: Determine whether the information entropy H is not higher than the set threshold; Step 5: If the information entropy H is not higher than the set threshold, directly adopt the edge detection result and do not trigger cloud review; end.

[0096] Step 6: If the information entropy H is higher than the set threshold, ROI pruning is associated with multiple frames, and the ROI sequence is output. Step 7: Semantic compression and tokenization to generate a semantic visual token sequence; Step 8: Transfer the token data to the cloud; Step 9: Begin cloud-based review and perform token mapping upgrade; Step 10: Large model verification, output verification results, and end.

[0097] Figure 6 This is another flowchart illustrating the predefined behavior identification method based on audiovisual materials in an embodiment of the present invention, as shown below. Figure 6 The diagram illustrates a flowchart of a predefined behavior identification method based on audiovisual materials applied in the cloud. This method includes: Step 601: Receive the base layer of the semantic lexical sequence uploaded by the edge terminal; the semantic lexical sequence is obtained by the edge terminal using the end-to-cloud semantic alignment encoder to encode the region of interest sequence frame by frame, and the semantic lexical sequence also includes an enhancement layer cached at the edge terminal; the region of interest sequence is formed based on the previous and next consecutive frames of audiovisual data associated with the preliminary category determination result when the edge terminal performs predefined behavior detection on the audiovisual data stream and the reliability of the output preliminary category determination result is not higher than a set threshold. Step 602: Use the lexical mapping dimensionality-upgrading module to upgrade the dimensionality of the base layer to obtain upgraded lexical units; the lexical mapping dimensionality-upgrading module and the end-to-end semantic alignment encoder have completed end-to-end joint training in advance, and the output distribution of the lexical mapping dimensionality-upgrading module matches the input distribution of the large cloud model; Step 603: Input the upgraded word units directly into the cloud-based large model without decoding or restoration for the first-level review of category determination; Step 604: When the confidence level of the first-level review is lower than the preset arbitration threshold, request the edge end to obtain the enhancement layer; Step 605: Perform a second-level review based on the upgraded lexical units of the base layer and enhancement layer, and output the review results back to the edge.

[0098] The process of requesting an enhancement layer from the edge device includes: sending an enhancement layer request to the edge device; the enhancement layer request carries the frame index and region range required by the large cloud model, so that the edge device can send back the corresponding enhancement layer according to the enhancement layer request.

[0099] This invention also provides a predefined behavior recognition system based on audiovisual materials, including the aforementioned edge and cloud terminals.

[0100] In summary, this invention constructs an intelligent verification link aligned with the edge-cloud feature space interface, from edge visual perception to cloud semantic understanding. This predefined behavior recognition system based on audiovisual materials consists of an edge-deployed target detection model, an entropy gating module, an edge-cloud semantic alignment encoder (including a temporal aggregation processing unit), and a cloud-deployed token mapping and dimensionality enhancement module, a parameter-frozen visual language large model, a structured conclusion parsing and feedback module, an edge-cloud collaborative database, and a distribution offset monitoring and calibration module. The core feature of the system's data path is direct connection without decoding: the semantic visual token generated at the edge is directly used as input to the visual embedding layer of the cloud-based visual language large model after network transmission and mapping dimensionality enhancement; the entire link eliminates image reconstruction, pixel decoding, and secondary feature extraction steps.

[0101] The following example illustrates a video evidence review scenario. Assume the system has pre-deployed a lightweight object detection model and an edge-cloud semantic alignment encoder on edge devices, and deployed a token mapping dimensionality enhancement module, a parameter-frozen visual language model, and related modules on the cloud.

[0102] Implementation process: 1. Edge detection and decision-making: The reviewers imported a surveillance video clip into the system. The video was fed frame by frame into the edge device. In one frame, two people were seen making physical contact.

[0103] The edge target detection model successfully located two people and output a preliminary confidence distribution, for example: [Normal interaction: 0.45, Specific behavior: 0.40, Other: 0.15].

[0104] The entropy gating module calculates the entropy value of this distribution. Since the probability distribution is relatively even (with high uncertainty), the calculated entropy value H exceeds the preset threshold (for example, if the threshold is set to 1.0, the H calculated according to formula (1) is about 1.46).

[0105] The system determines that the current test result is unreliable and triggers a cloud-based review process.

[0106] 2. Temporal semantic alignment token generation: The system clips the ROI of the current frame based on the detection box coordinates and associates the ROIs of the same two people in five consecutive frames before and after the target tracking identifier to form a temporal ROI sequence. Each ROI image frame is fed into an edge-cloud semantic alignment encoder, where it is encoded frame by frame into a 256-dimensional visual token sequence. This sequence is then overlaid with temporal location encoding and aggregated into a temporal token group. Simultaneously, metadata tags (entropy value, initial edge classification, and confidence level) are generated. The data volume of this token group is only a fraction of that of the original image.

[0107] 3. Cloud-based review and output: The edge device sends the base layer and metadata tags of the time-series token group to the cloud, while the enhancement layer token is temporarily stored in the local cache of the edge device.

[0108] The cloud-based Token mapping and dimensionality-upgrading module receives the sequence and maps it to 1024 dimensions to match the input dimension of the cloud-based visual language large model visual embedding layer. The upgraded Token is directly injected into the large model visual embedding layer without image decoding and restoration.

[0109] The cloud-based metadata tags are compiled into text prompts: "The initial edge judgment tends to indicate a specific behavior, but the uncertainty is high (entropy value 1.46). Please combine the temporal action evolution, focus on checking the action force method and the interaction rhythm of both parties, and judge whether the interaction between the two people in the image constitutes a specific behavior?" Together with the upgraded token, they are sent into the visual language big model.

[0110] The large model comprehensively analyzes subtle visual cues such as arm postures, body force exertion, interaction rhythms, and facial expressions of the two individuals in the time-series token group, and outputs a structured review conclusion: the category is identified as "non-predefined behavior, belonging to intense playful fighting", with a cloud confidence level of 0.93, and the key criterion summary is "the actions of both parties show symmetrical playful rhythms, without unilateral oppressive force".

[0111] 4. Marginal Arbitration and Final Determination: The system arbitration module compared the cloud-based confidence score of 0.93 with the initial edge detection confidence score of 0.40, adopted the cloud-based conclusion according to the fusion strategy, and ultimately marked the video segment as "normal," thus avoiding a potential false alarm. The entire review process was completed in just a few seconds, and the original image was not transmitted, decoded, or modified, and the parameters of the large model were not altered.

[0112] If the confidence level of the first-level review is insufficient (e.g., below 0.80), the cloud will request the edge to send back the enhancement layer token of the corresponding frame, and after splicing and restoring, perform the second-level review. In this example, the confidence level of the first-level review of 0.93 has reached the arbitration threshold, and the enhancement layer request was not triggered, so the actual amount of data transmitted was further reduced.

[0113] 5. Example of closed-loop evolution: This sample was stored in the high-value sample library because it met the triple criteria (high entropy, inconsistency between edge and cloud conclusions, and high confidence in the cloud). When the sample library accumulated to 1000 samples, an incremental update was triggered: the training set was constructed by time decay weighting, the composite alignment loss was used and an EWC regularization term was added to fine-tune the encoder and projector. After the update, the standard test token set was uploaded, and the measured MMD offset was 0.08, which is lower than the calibration threshold of 0.15, so BN calibration was not triggered. After the system ran continuously for three months, the high entropy trigger rate dropped from the initial 12% to 5%, which is lower than the lower limit of the target range. The entropy threshold was lowered by one level according to the feedback rules, and the system's ability to autonomously identify complex scenarios continued to improve.

[0114] This invention proposes a dedicated feature space alignment mechanism for edge-cloud semantic collaboration in its predefined behavior recognition method based on audiovisual materials. Distillation is a deep interface alignment of the feature space, aiming to train a "visual semantic translator" rather than a "model substitute." In terms of purpose: traditional distillation aims for student models to complete tasks independently; however, in this invention, the student model does not independently undertake recognition tasks. Its sole function is to encode visual information into semantic tokens that the cloud-based large model can directly understand. In terms of structural differences: the student model in this invention does not contain any classification head or task head, structurally excluding the possibility of it independently performing recognition tasks. A dedicated projection head is introduced to achieve dimensional alignment, and the projection head output strictly matches the input specifications of the visual embedding layer of the cloud-based large model. Furthermore, the token mapping dimensionality-upgrading module is jointly optimized end-to-end with the student model during the training phase. In terms of loss and validation differences: in addition to feature space geometric alignment (MSE), distribution alignment loss (MMD / InfoNCE) and behavior consistency loss (KL divergence, temperature coefficient) are added, and the "consistency rate of conclusions between student paths and teacher paths on the same frozen large model" is used as the pre-deployment acceptance criterion.

[0115] This invention proposes a heterogeneous collaborative reasoning architecture of "edge semantic abstraction - cloud-based direct connection without decoding" in its method for identifying predefined behaviors based on audiovisual materials. Instead of transmitting original images, compressed images, or traditional handcrafted features at the edge, this architecture transmits a semantic token that is deeply aligned with the feature space of the large cloud model. The cloud directly injects this token into the parameter-frozen embedding layer of the large visual language model for verification and reasoning without image decoding. This architecture enables the large cloud model to access edge visual input with zero adaptation cost. Specifically: First, compared to transmitting images, the data volume in this invention is reduced by tens to hundreds of times; second, compared to transmitting compressed images, this invention transmits semantic features rather than pixel information, eliminating the need for image decoding in the cloud; third, compared to transmitting "traditional features," the token in this invention is directly aligned with the internal representation space of the large cloud model, eliminating the need for feature re-extraction in the cloud. This architecture allows the large cloud model to participate in real-time tasks with extremely low communication costs.

[0116] In the predefined behavior identification method based on audiovisual materials in this invention embodiment, a closed-loop self-iterative mechanism with the maintenance of representation space consistency as the main line is proposed. This mechanism not only includes an entropy-gated driven "on-demand reporting" mechanism, but also further uses the results of cloud-based review as high-value training samples. Incremental learning is fed back to the edge for self-iterative optimization, and quantitative monitoring and lightweight calibration maintain the long-term consistency of the edge-cloud representation space: First, high-value sample screening: training samples are screened based on three criteria: entropy value, conclusion consistency, and cloud confidence. Second, elastic weight consolidation: preventing catastrophic forgetting in incremental learning. Third, distribution offset quantitative monitoring and mapping module calibration: edge-cloud interface drift is monitored using MMD offset quantization; when the limit is exceeded, only the batch normalization parameters of the mapping module are calibrated. Fourth, trigger rate feedback adjustment: the high-entropy sample trigger rate is used as a controlled variable to adjust the entropy gating threshold, ensuring stable convergence of the cloud review load. This allows the system's performance ceiling to be continuously "lifted" by the large cloud model as it runs, possessing long-term evolution capabilities and breaking through the performance ceiling of existing static systems.

[0117] In the predefined behavior identification method based on audiovisual materials in this invention embodiment, a scalable semantic token and a two-level review protocol are also proposed. Building upon the aforementioned interface alignment architecture, this invention embodiment further designs the semantic token into a layered structure of a base layer and an enhancement layer, and includes a two-level review protocol: by default, only the base layer is uploaded to complete the first-level review; when the confidence level in the cloud is insufficient, the enhancement layer is requested to perform the second-level review as needed; combined with entropy-adaptive token budgeting and conformal gating statistical threshold calibration, the transmission bandwidth adaptively scales with the sample difficulty, and the review trigger rate has a provably guaranteed statistical coverage.

[0118] Furthermore, this embodiment of the invention introduces a temporal token aggregation mechanism, which associates the same target ROI with K consecutive frames, encodes each frame and then superimposes temporal and positional codes, aggregating them into temporal token groups at the feature level. This allows the large cloud model to distinguish ambiguous behaviors in a single frame (such as "intense play" and "specific behavior") using action evolution trajectories. This mechanism performs aggregation in the semantic feature space, eliminating the need for re-encoding images in the cloud.

[0119] In this embodiment of the invention, conformal prediction statistical gating is implemented by using the (1-α) quantile of the non-consistency scores of the calibration sample set to calibrate the entropy gating trigger threshold, so that the cloud review trigger rate has a provable statistical coverage guarantee, and the error rate of the conclusion directly identified at the edge is bounded by a preset tolerance α, and can adapt to scene changes as the calibration set is updated.

[0120] In this embodiment of the invention, the criterion location feedback is as follows: while the cloud-based large model outputs structured conclusions, it also feeds back the key frame index and key token index on which the conclusions are based. The edge end overlays and marks the time position and image region corresponding to the key criteria onto the original video, so that the review conclusions can be interpreted and the criteria can be fixed, which facilitates evidence verification and cross-examination in judicial review scenarios.

[0121] This invention also provides a device for identifying predefined behaviors based on audiovisual materials, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the method for identifying predefined behaviors based on audiovisual materials, the implementation of this device can refer to the implementation of the method for identifying predefined behaviors based on audiovisual materials; repeated details will not be elaborated further.

[0122] Figure 7 This is a schematic diagram of a predefined behavior recognition device based on audiovisual materials in an embodiment of the present invention, such as... Figure 7 As shown, the device 700 is applied to the edge, and the device 700 is implemented using the aforementioned predefined behavior recognition method based on audiovisual data applied to the edge. The device 700 includes: The preliminary detection module 701 is used to perform predefined behavior detection on the audiovisual data stream using a pre-trained target detection model and output a preliminary category determination result; the target detection model is used to locate the region of interest from the audiovisual data and determine the preliminary category determination result based on the region of interest. The preliminary test result evaluation module 702 is used to evaluate the reliability of the preliminary category determination results; The result output module 703 is used to output the final detection result based on the preliminary category judgment result when the reliability level is higher than the set threshold. The semantic lexical encoding module 704 is used to form a region of interest sequence based on the preliminary category judgment results of consecutive audiovisual data from multiple frames before and after the reliability level is not higher than a set threshold. The region of interest sequence is encoded frame by frame using a pre-deployed edge-cloud semantic alignment encoder to form a semantic lexical sequence. The semantic lexical includes a base layer and an enhancement layer. The base layer is a first semantic representation, and the enhancement layer is a second semantic representation. The dimension of the first semantic representation is smaller than the dimension of the second semantic representation. The edge-cloud semantic alignment encoder and the cloud-based lexical mapping dimensionality enhancement module are pre-trained end-to-end. The output distribution of the lexical mapping dimensionality enhancement module matches the input distribution of the large cloud model. The edge-cloud communication module 705 is used to cache the enhancement layer at the edge and upload the base layer to the cloud, so that the cloud: uses the lexical mapping dimensionality-upgrading module to upgrade the base layer to obtain upgraded lexical units; inputs the upgraded lexical units into the large model, and outputs the verification conclusion back to the edge; the large model first uses the upgraded lexical units of the base layer to perform a first-level verification of category determination. When the confidence of the first-level verification is lower than a preset arbitration threshold, the cloud requests the edge to obtain the enhancement layer, performs a second-level verification based on the upgraded lexical units of the base layer and the enhancement layer, and outputs the verification conclusion.

[0123] In one embodiment, the large model includes a visual encoder; the visual encoder is used to map input image features to semantic features; The visual encoder is used as the teacher model, and the convolutional neural network or visual transformer ViT is used as the student model. The edge-cloud semantic alignment encoder is obtained through knowledge distillation. The input of the edge-cloud semantic alignment encoder is the region of interest sequence, and the output is the semantic lexical units after encoding and dimensionality reduction.

[0124] In one embodiment, the edge-cloud semantic alignment encoder and the cloud-based lexical mapping upscaling module are jointly trained end-to-end in the following manner: A projection head with a multilayer perceptron structure is constructed; the output dimension of the projection head matches the input dimension requirement of the large model visual embedding layer. During the knowledge distillation training phase, the lexical mapping upscaling module is connected to the training loop to achieve end-to-end joint optimization of the lexical mapping upscaling module, the student model, and the projection head.

[0125] In one embodiment, the device: during the knowledge distillation training phase, constrains the semantic lexical generation target based on the information bottleneck principle in the following manner: While minimizing the mutual information between the generated semantic lexical and the input region of interest, the goal is to maximize the mutual information retention of the semantic lexical on the output features of the teacher model.

[0126] In one embodiment, the device: during the knowledge distillation training phase, calculates multiple losses using the outputs of the teacher model and the student model, weights and fuses the various losses to obtain the total loss, iteratively updates the student model parameters based on the total loss, and iterates repeatedly until training is completed; the multiple losses include semantic alignment loss, behavioral consistency loss, and anchor structure loss; The semantic alignment loss is expressed as follows: ; in, Represents semantic alignment loss. The mean squared error loss constrains the geometric alignment of the feature space; For distribution alignment loss, the probability distributions of teacher and student features are constrained to be consistent; , The output obtained by inputting the same batch of images into the teacher model and the student model respectively. To balance the hyperparameters; The behavioral consistency loss is represented as follows: ; in, The loss represents the behavioral consistency, and KL() is the KL divergence calculation function. ( ) represents the softmax function. , The unnormalized raw scores of the teacher and student models, after projection, are input into the cloud-based large-scale model language decoder. Temperature coefficient; The anchor point structure loss is represented by: constraining the structure of the two relative similarity distributions to be consistent with KL divergence; the two relative similarity distributions include the similarity distribution of teacher visual features relative to each anchor point and the similarity distribution of student visual features relative to each anchor point; the anchor point is the category prototype vector generated by the large model from the text description of each behavior category.

[0127] In one embodiment, the preliminary category determination result includes multiple categories, the confidence level of each category, and the target tracking identifier corresponding to each category; the region of interest sequence includes a temporal region of interest sequence; and the semantic lexical sequence includes temporal tuples. The semantic lexical encoding module 704 is specifically used for: Based on the target tracking identifier, the regions of interest (ROIs) of the same target in K consecutive frames before and after the current frame are associated to form a temporal ROI sequence, where K is a positive integer; The semantic units of each frame in the temporal region of interest sequence are aggregated into temporal units by superimposing temporal position encodings in a pre-deployed edge-cloud semantic alignment encoder. The end-to-cloud communication module 705 is specifically used to upload the basic layer of semantic lexical units of each frame in the temporal lexical unit to the cloud.

[0128] In one embodiment, the preliminary category determination result includes multiple categories and the confidence level of each category; the reliability level is quantified by information entropy. The preliminary test result evaluation module 702 is specifically used for: Calculate the information entropy using the following formula: ; in, Represents information entropy. The normalized confidence score for the i-th category predicted by the object detection model, where n represents the number of initial category determinations; The information entropy is compared with a set threshold to assess the reliability of the preliminary category determination result; the set threshold is calibrated offline using a conformal prediction method.

[0129] In one embodiment, the end-to-cloud communication module 705 is specifically used for: The base layer and metadata tags are uploaded to the cloud so that the cloud can: compile the metadata tags into structured text hints, which are then input into the large model along with the upgraded lexical units; the metadata tags include preliminary category determination results, confidence levels, and entropy values.

[0130] In one embodiment, the number of semantic units in the base layer uploaded to the cloud is positively correlated with the information entropy; the higher the information entropy, the more semantic units in the base layer are uploaded, and the higher the number of frames and spatial granularity are retained.

[0131] In one embodiment, the device 700 further includes a determination result processing module, used for: Receive structured review conclusions and criterion location information transmitted back from the cloud; the criterion location information includes the keyframe index and key semantic term index on which the review conclusion is based; The time location corresponding to the key criteria is superimposed and marked on the audiovisual data along with the image region.

[0132] In one embodiment, the device 700 further includes a closed-loop iteration module for: The audiovisual materials data and their associated information that have been reviewed in the cloud are stored in the end-to-cloud collaborative database; the associated information includes the original region of interest sequence, semantic word sequence, cloud review conclusion labels, and reference features output by the teacher model; When the preset period is reached or the number of stored samples reaches the preset threshold, an incremental training set is constructed based on the stored samples. The multiple losses are reused and elastic weights are added to consolidate the EWC regularization term, and the edge-cloud semantic alignment encoder and projection head are jointly fine-tuned.

[0133] In one embodiment, the closed-loop iteration module is further configured to, after the joint fine-tuning: A standard test lexicon set is uploaded to the cloud so that the cloud can detect the input distribution offset of the lexicon mapping dimensionality upgrade module. When the input distribution offset exceeds the offset threshold, the cloud performs a lightweight calibration on the lexicon mapping dimensionality upgrade module. The lightweight calibration only updates the batch normalization parameters. And / or, adjust the set threshold according to the high-entropy sample trigger rate feedback that triggers cloud review. When the trigger rate is continuously higher than the upper limit of the trigger rate, the set threshold is increased; when the trigger rate is continuously lower than the lower limit of the trigger rate, the set threshold is decreased.

[0134] Figure 8 This is another schematic diagram of the predefined behavior recognition device based on audiovisual materials in an embodiment of the present invention, as shown below. Figure 8 As shown, the device 800 is applied in the cloud, and the device 800 is implemented using the aforementioned cloud-based predefined behavior recognition method based on audiovisual materials. The device 800 includes: The data receiving module 801 is used to receive the base layer of the semantic word sequence uploaded by the edge terminal; the semantic word sequence is obtained by the edge terminal using the end-to-cloud semantic alignment encoder to encode the region of interest sequence frame by frame, and the semantic word sequence also includes an enhancement layer cached at the edge terminal; the region of interest sequence is formed based on the previous and next consecutive frames of audiovisual data associated with the preliminary category judgment result when the edge terminal performs predefined behavior detection on the audiovisual data stream and the reliability of the output preliminary category judgment result is not higher than a set threshold. The dimension-upgrading module 802 is used to upgrade the base layer using the lexical mapping dimension-upgrading module to obtain upgraded lexical units; the lexical mapping dimension-upgrading module and the end-to-end semantic alignment encoder are pre-trained together end-to-end, and the output distribution of the lexical mapping dimension-upgrading module matches the input distribution of the large cloud model; The composite judgment module 803 is used to input the upgraded words directly into the cloud-based big model for the first-level review of category judgment without decoding and restoration; when the confidence of the first-level review is lower than the preset arbitration threshold, it requests the edge end to obtain the enhancement layer; based on the upgraded words of the base layer and the enhancement layer, the second-level review is performed, and the review conclusion is output and sent back to the edge end.

[0135] Specifically, the composite determination module 803 is used to: send an enhancement layer request to the edge end; the enhancement layer request carries the frame index and region range required by the large cloud model, so that the edge end can send back the corresponding enhancement layer according to the enhancement layer request.

[0136] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for identifying predefined behaviors based on audiovisual materials.

[0137] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for identifying predefined behaviors based on audiovisual materials.

[0138] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for identifying predefined behaviors based on audiovisual materials.

[0139] The embodiments of the present invention have the following significant beneficial effects: 1. Zero-adaptation access to large cloud-based models: The parameters of the large cloud-based visual language model are frozen throughout the process, without fine-tuning, inserting adaptation layers, or changing the structure, making it plug-and-play; when upgrading the large model version, only the edge encoder needs to be retrained, with zero changes in the cloud, and the engineering cost and compliance risk are significantly lower than any other solution that requires modification of the large model; 2. Direct connection without decoding eliminates double loss: The cloud does not need to decode and restore the image or re-extract features, which avoids the erosion of verification accuracy by "decoding distortion + secondary feature extraction deviation" and further reduces end-to-end verification latency; 3. Verification accuracy does not decrease with compression rate: Distribution alignment loss ensures that edge tokens and teacher embedding space are consistent at the statistical distribution level, behavioral consistency loss ensures that the large model's understanding of edge tokens is consistent with the direct processing of the original image, and downstream consistency verification makes the effect quantifiable and acceptable. 4. Improved temporal discrimination: The temporal token group explicitly presents the action evolution trajectory to the large model, which solves the ambiguity between similar behaviors such as "playing" and "predefined behavior" in a single frame and makes up for the lack of temporal modeling capability of pure edge lightweight solutions. 5. Long-term evolution without mismatch: MMD offset monitoring and BN lightweight calibration ensure that the edge model continues to evolve without damaging the edge-cloud semantic interface. Trigger rate feedback adjustment makes the cloud verification load converge as the edge capability is enhanced. The overall system performance approaches the level of large cloud models and remains stable in the long term.

[0140] 6. Bandwidth scalable with sample difficulty: The tiered token and two-level review protocol enable simple samples to be reviewed with minimal bandwidth, while only difficult samples trigger incremental transmission of the enhancement layer. Combined with the entropy-adaptive token budget, the average bandwidth consumption is further reduced compared to the fixed token scheme, and the review accuracy is not compromised.

[0141] 7. Interpretable conclusions and fixed criteria: The criterion positioning and feedback mechanism enables cloud-based review conclusions to include key frames and key area markers, allowing reviewers to directly verify the judgment basis of the large model, thus meeting the requirements of judicial review scenarios for interpretability of conclusions and verifiability of evidence.

[0142] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This invention is described in terms of flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowcharts and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying predefined behaviors based on audiovisual materials, characterized in that, Applied to the edge, the method includes: A pre-trained target detection model is used to perform predefined behavior detection on audiovisual data streams and output preliminary category determination results; the target detection model is used to locate regions of interest from audiovisual data and determine preliminary category determination results based on regions of interest. Assess the reliability of the preliminary classification results; When the reliability level is higher than the set threshold, the final detection result is output based on the preliminary category determination result. When the reliability level is not higher than the set threshold, a region of interest sequence is formed based on the sequential audiovisual data of multiple frames before and after the preliminary category determination result. A pre-deployed edge-cloud semantic alignment encoder is used to encode the region of interest sequence frame by frame to form a semantic word sequence. The semantic word includes a base layer and an enhancement layer. The base layer is a first semantic representation, and the enhancement layer is a second semantic representation. The dimension of the first semantic representation is smaller than the dimension of the second semantic representation. The edge-cloud semantic alignment encoder and the word mapping dimensionality enhancement module in the cloud are pre-trained end-to-end. The output distribution of the word mapping dimensionality enhancement module matches the input distribution of the large model in the cloud. The enhancement layer is cached at the edge, while the base layer is uploaded to the cloud. This allows the cloud to: use a lexical mapping dimensionality-upgrading module to upgrade the base layer, obtaining upgraded lexical units; input these upgraded lexical units into the large model, and output a verification conclusion back to the edge; the large model first uses the upgraded lexical units of the base layer for a first-level verification of category determination. When the confidence level of the first-level verification is lower than a preset arbitration threshold, the cloud requests the enhancement layer from the edge, performs a second-level verification based on the upgraded lexical units of the base layer and the enhancement layer, and outputs a verification conclusion.

2. The method as described in claim 1, characterized in that, The large model includes a visual encoder; the visual encoder is used to map input image features into semantic features; The visual encoder is used as the teacher model, and the convolutional neural network or visual transformer ViT is used as the student model. The edge-cloud semantic alignment encoder is obtained through knowledge distillation. The input of the edge-cloud semantic alignment encoder is the region of interest sequence, and the output is the semantic lexical units after encoding and dimensionality reduction.

3. The method as described in claim 2, characterized in that, The edge-cloud semantic alignment encoder and the cloud-based lexical mapping upscaling module are jointly trained end-to-end in the following manner: A projection head with a multi-layer perceptron structure is constructed; the output dimension of the projection head matches the input dimension requirements of the large model's visual embedding layer. During the knowledge distillation training phase, the lexical mapping upscaling module is connected to the training loop to achieve end-to-end joint optimization of the lexical mapping upscaling module, the student model, and the projection head.

4. The method as described in claim 2, characterized in that, Also includes: During the knowledge distillation training phase, the generation target of semantic lexical units is constrained based on the information bottleneck principle in the following manner: While minimizing the mutual information between the generated semantic lexical and the input region of interest, the goal is to maximize the mutual information retention of the semantic lexical on the output features of the teacher model.

5. The method as described in claim 2, characterized in that, Also includes: During the knowledge distillation training phase, multiple losses are calculated using the outputs of the teacher model and the student model. The total loss is obtained by weighted fusion of the various losses. The parameters of the student model are iteratively updated based on the total loss. This process is repeated until training is complete. The multiple losses include semantic alignment loss, behavioral consistency loss, and anchor structure loss. The semantic alignment loss is expressed as follows: ； in, Represents semantic alignment loss. The mean squared error loss constrains the geometric alignment of the feature space; For distribution alignment loss, the probability distributions of teacher and student features are constrained to be consistent; , The output obtained by inputting the same batch of images into the teacher model and the student model respectively. To balance the hyperparameters; The behavioral consistency loss is represented as follows: ； in, The loss represents the behavioral consistency, and KL() is the KL divergence calculation function. ( ) represents the softmax function. , The unnormalized raw scores of the teacher and student models, after projection, are input into the cloud-based large-scale model language decoder. Temperature coefficient; The anchor point structure loss is represented by: constraining the structure of the two relative similarity distributions to be consistent with KL divergence; the two relative similarity distributions include the similarity distribution of teacher visual features relative to each anchor point and the similarity distribution of student visual features relative to each anchor point; the anchor point is the category prototype vector generated by the large model from the text description of each behavior category.

6. The method as described in claim 1, characterized in that, The preliminary category determination results include multiple categories, the confidence level of each category, and the target tracking identifier corresponding to each category; the region of interest sequence includes a temporal region of interest sequence; When the reliability level is not higher than a set threshold, a region of interest sequence is formed from consecutive frames of audiovisual data associated with the preliminary category determination results, including: Based on the target tracking identifier, the regions of interest (ROIs) of the same target in K consecutive frames before and after the current frame are associated to form a temporal ROI sequence, where K is a positive integer; The semantic lexical sequence includes temporal lexical groups. A pre-deployed edge-cloud semantic alignment encoder is used to encode the region of interest sequence frame by frame to form the semantic lexical sequence, which includes: The semantic units of each frame in the temporal region of interest sequence are aggregated into temporal units by superimposing temporal position encodings in a pre-deployed edge-cloud semantic alignment encoder. Uploading the base layer to the cloud includes uploading the base layer of semantic lexical units of each frame in the temporal lexical group to the cloud.

7. The method as described in claim 6, characterized in that, The enhancement layer is cached at the edge, while the base layer is uploaded to the cloud, including: The base layer and metadata tags are uploaded to the cloud so that the cloud can: compile the metadata tags into structured text hints, which are then input into the large model along with the upgraded lexical units; the metadata tags include preliminary category determination results, confidence levels, and entropy values.

8. The method as described in claim 1, characterized in that, The degree of reliability is quantified by information entropy; Assess the reliability of the preliminary classification results, including: Calculate the information entropy using the following formula: ； in, Represents information entropy. The normalized confidence score for the i-th category predicted by the object detection model, where n represents the number of initial category determinations; The information entropy is compared with a set threshold to assess the reliability of the preliminary category determination result; the set threshold is calibrated offline using a conformal prediction method.

9. The method as described in claim 8, characterized in that, The number of semantic units in the base layer uploaded to the cloud is positively correlated with information entropy.

10. The method as described in claim 5, characterized in that, Also includes: Receive structured review conclusions and criterion location information transmitted back from the cloud; the criterion location information includes the keyframe index and key semantic term index on which the review conclusion is based; The time location corresponding to the key criteria is superimposed and marked on the audiovisual data along with the image region.

11. The method as described in claim 10, characterized in that, Also includes: The audiovisual data and related information that have been verified in the cloud are stored in the end-to-cloud collaborative database; The associated information includes the original region of interest sequence, semantic word sequence, cloud-based review conclusion labels, and reference features output by the teacher model; When the preset period is reached or the number of stored samples reaches the preset threshold, an incremental training set is constructed based on the stored samples. The multiple losses are reused and elastic weights are added to consolidate the EWC regularization term, and the edge-cloud semantic alignment encoder and projection head are jointly fine-tuned.

12. The method as described in claim 11, characterized in that, Following the joint fine-tuning, the following is also included: A standard test lexicon set is uploaded to the cloud so that the cloud can detect the input distribution offset of the lexicon mapping dimensionality upgrade module. When the input distribution offset exceeds the offset threshold, the cloud performs a lightweight calibration on the lexicon mapping dimensionality upgrade module. The lightweight calibration only updates the batch normalization parameters. And / or, adjust the set threshold according to the high-entropy sample trigger rate feedback that triggers cloud review. When the trigger rate is continuously higher than the upper limit of the trigger rate, the set threshold is increased; when the trigger rate is continuously lower than the lower limit of the trigger rate, the set threshold is decreased.

13. A method for identifying predefined behaviors based on audiovisual materials, characterized in that, Applied to the cloud, the method includes: The base layer receives the semantic word sequence uploaded by the edge terminal; the semantic word sequence is obtained by the edge terminal using the end-to-cloud semantic alignment encoder to encode the region of interest sequence frame by frame, and the semantic word sequence also includes an enhancement layer cached at the edge terminal; the region of interest sequence is formed based on the previous and next consecutive frames of audiovisual data associated with the preliminary category determination result when the edge terminal performs predefined behavior detection on the audiovisual data stream and the reliability of the output preliminary category determination result is not higher than a set threshold. The basic layer is upgraded using a lexical mapping dimensionality-upgrading module to obtain upgraded lexical units; the lexical mapping dimensionality-upgrading module and the end-to-end semantic alignment encoder are pre-trained together, and the output distribution of the lexical mapping dimensionality-upgrading module matches the input distribution of the large cloud model; The first-level verification of class determination is to input the upgraded word units into the large model; When the confidence level of the first-level review is lower than the preset arbitration threshold, a request is made to the edge to obtain the enhancement layer; The second-level verification is performed based on the upgraded lexical units of the base layer and enhancement layer, and the verification results are output and sent back to the edge.

14. The method as described in claim 13, characterized in that, Request the enhancement layer from the edge, including: An enhancement layer request is sent to the edge device; the enhancement layer request carries the frame index and region range required by the large model in the cloud, so that the edge device can send back the corresponding enhancement layer according to the enhancement layer request.

15. A predefined behavior recognition system based on audiovisual materials, characterized in that, It includes an edge terminal and a cloud terminal; the edge terminal is used to perform the method as described in any one of claims 1 to 12; the cloud terminal is used to perform the method as described in any one of claims 13 to 14.

16. A device for predefined behavior recognition based on audiovisual materials, characterized in that, Applied to the edge, the device includes: The preliminary detection module is used to perform predefined behavior detection on the audiovisual data stream using a pre-trained target detection model and output a preliminary category determination result; the target detection model is used to locate the region of interest from the audiovisual data and determine the preliminary category determination result based on the region of interest. The preliminary test result evaluation module is used to assess the reliability of the preliminary category determination results; The results output module is used to output the final detection result based on the preliminary category determination result when the reliability level is higher than the set threshold. The semantic lexical encoding module is used to form a region of interest (ROI) sequence based on multiple consecutive frames of audiovisual data associated with the preliminary category judgment results, when the reliability level is not higher than a set threshold. The ROI sequence is then encoded frame-by-frame using a pre-deployed edge-cloud semantic alignment encoder to form a semantic lexical sequence. The semantic lexical sequence includes a base layer and an enhancement layer. The base layer is a first semantic representation, and the enhancement layer is a second semantic representation. The dimension of the first semantic representation is smaller than the dimension of the second semantic representation. The edge-cloud semantic alignment encoder and the cloud-based lexical mapping dimensionality enhancement module are pre-trained end-to-end, and the output distribution of the lexical mapping dimensionality enhancement module matches the input distribution of the large cloud model. The edge-cloud communication module is used to cache the enhancement layer at the edge and upload the base layer to the cloud, so that the cloud: uses the lexical mapping dimensionality-upgrading module to upgrade the base layer to obtain upgraded lexical units; inputs the upgraded lexical units into the large model, and outputs the verification conclusion back to the edge; the large model first uses the upgraded lexical units of the base layer to perform a first-level verification of category determination. When the confidence of the first-level verification is lower than a preset arbitration threshold, the cloud requests the enhancement layer from the edge, performs a second-level verification based on the upgraded lexical units of the base layer and the enhancement layer, and outputs the verification conclusion.

17. A device for predefined behavior recognition based on audiovisual materials, characterized in that, The device, applied in the cloud, includes: The data receiving module is used to receive the base layer of semantic word sequence uploaded by the edge terminal; the semantic word sequence is obtained by the edge terminal using the end-to-cloud semantic alignment encoder to encode the region of interest sequence frame by frame, and the semantic word sequence also includes an enhancement layer cached at the edge terminal; the region of interest sequence is formed based on the previous and next consecutive frames of audiovisual data associated with the preliminary category judgment result when the edge terminal performs predefined behavior detection on the audiovisual data stream and the reliability of the output preliminary category judgment result is not higher than a set threshold. The dimensionality-upgrading module is used to upgrade the base layer using the lexical mapping dimensionality-upgrading module to obtain upgraded lexical units; the lexical mapping dimensionality-upgrading module and the end-to-end semantic alignment encoder are pre-trained together, and the output distribution of the lexical mapping dimensionality-upgrading module matches the input distribution of the large cloud model; The composite judgment module is used to input the upgraded words into the large model for the first-level review of category judgment; when the confidence of the first-level review is lower than the preset arbitration threshold, it requests the edge end to obtain the enhancement layer; based on the upgraded words of the base layer and the enhancement layer, the second-level review is performed, and the review conclusion is output and sent back to the edge end.

18. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 14.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 14.

Images