End-cloud collaborative alarm review decision-making method based on multi-modal semantic understanding

Abstract

The invention belongs to the technical field of artificial intelligence and edge computing, and particularly discloses an end-cloud collaborative alarm review decision-making method based on multi-modal semantic understanding, which comprises the following steps of: filtering alarm information reported by an end side; the filtering comprises alarm triggering judgment based on a time interval and repeated alarm judgment based on a detection frame overlapping area; performing multi-modal semantic review on the alarm retained after filtering based on the target frame coordinate and the associated context metadata, and calculating to obtain a dynamic review confidence coefficient; the context metadata at least comprises an illumination condition, a weather condition and motion blur information; and according to the dynamic review confidence, executing a classification decision to output a final alarm review result. According to the method, dynamic and intelligent re-checking of the alarm can be realized at the cloud, and semantic rationality of target appearance and quality influence of the current imaging environment are comprehensively considered, so that the accuracy of alarm judgment is greatly improved, and the effect of effectively reducing the false alarm rate of the alarm is realized.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and edge computing technology, and more specifically, relates to an edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding. Background Technology

[0002] Currently, visual analytics-based intelligent monitoring systems are widely used in edge-cloud collaborative architectures. However, limited by local computing resources, intelligent devices deployed on the edge can typically only perform preliminary identification and alarms based on target detection boxes. The depth of their identification results is limited, leading to a high false alarm rate in real-world applications. To improve alarm accuracy, a common solution is to upload edge-side alarms to the cloud for verification. However, existing cloud-side verification methods generally focus on solutions at the larger scale and more complex algorithm level, and often rely on fixed confidence thresholds or preset static rules for judgment. These methods struggle to adapt to complex, dynamically changing environments such as lighting changes, weather conditions, and target motion blur. They lack effective understanding and fusion of the semantic information of the target's context, thus limiting their false alarm suppression effectiveness when dealing with the diversity of real-world scenarios, and the overall alarm accuracy of the system still has significant room for improvement.

[0003] Therefore, how to effectively reduce the false alarm rate is an urgent problem to be solved. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this application is to provide an edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding, which can effectively reduce the false alarm rate.

[0005] To achieve the above objectives, firstly, this application provides a cloud-edge collaborative alarm review decision-making method based on multimodal semantic understanding, comprising the following steps: S10, filter the alarm information reported by the end side. The filtering includes alarm trigger determination based on time interval and duplicate alarm determination based on the overlapping area of ​​the detection box. S20: For the alarms retained after filtering in step S10, perform multimodal semantic verification based on their target box coordinates and associated contextual metadata, and calculate the dynamic verification confidence level; the contextual metadata includes at least lighting conditions, weather conditions, and motion blur information; S30, based on the dynamic verification confidence level, perform a classification decision to output the final alarm verification result.

[0006] The edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding provided in this application has the following effects: By pre-filtering alarms reported from the edge based on time interval and spatial overlap, the computing load on the cloud can be effectively reduced. Furthermore, multimodal semantic understanding is introduced into the filtered alarms, upgrading the review basis from a single visual confidence level to a joint evaluation model that integrates visual evidence, scene rationality, and environmental reliability. Through the synergistic effect of the above technical features, this method can achieve dynamic and intelligent review of alarms in the cloud, comprehensively considering the semantic rationality of the target's appearance and the quality impact of the current imaging environment, thereby significantly improving the accuracy of alarm judgment and effectively reducing the false alarm rate.

[0007] As a further preferred embodiment, in step S10, the alarm triggering determination based on the time interval specifically includes: Receive the current alarm and its timestamp at a specific video monitoring point. ; Get the timestamp of the last alarm ; Calculate the time interval ΔT = - ; If ΔT is greater than the pre-configured first alarm time interval threshold α, then the subsequent deep review process is triggered; otherwise, the review of the current alarm is skipped.

[0008] As a further preferred embodiment, in step S10, the determination of repeated alarms based on the overlapping area of ​​the detection boxes specifically includes: Obtain the coordinates of two adjacent potential trigger target detection boxes belonging to a certain video monitoring point. The coordinates of the first detection box (box1) are... The coordinates of the second detection box (box2) are: ,in These represent the X-coordinate of the top left corner, the Y-coordinate of the top left corner, the X-coordinate of the bottom right corner, and the Y-coordinate of the bottom right corner of the first detection box, respectively. These represent the X-coordinate of the top left corner, the Y-coordinate of the top left corner, the X-coordinate of the bottom right corner, and the Y-coordinate of the bottom right corner of the second detection box, respectively. Calculate the intersection area of ​​the two detection boxes. ; Calculate the area of ​​each of the two detection boxes. ; like The overlap region determination threshold γ is greater than the pre-configured threshold value. If the product of these is used, it is considered a duplicate alarm and is filtered out.

[0009] As a further preferred option, the intersection area of ​​the two detection boxes is calculated. Specifically: Calculate the coordinates of the top left corner of the intersection region. , ),in , ; Calculate the coordinates of the lower right corner of the intersection region. , ),in , ; If satisfied ,but ,otherwise .

[0010] As a further preferred embodiment, in step S20, the step of performing multimodal semantic verification and calculating dynamic verification confidence specifically includes: S21, Multimodal Feature Extraction, including: S211, Visual feature extraction: Based on the target box coordinates B, the corresponding target area image block is cropped from the original image data associated with the alarm information. The image block is adjusted to a fixed size and the pixel value is normalized. Then, it is input into the visual Transformer model for feature encoding to obtain the visual feature vector V(B). S212, Semantic feature extraction: The context metadata C is converted into scene description text in natural language form, and input into the text encoder of the CLIP model for feature encoding to obtain the semantic feature vector L(C); S213, Environmental feature extraction: The illumination conditions, weather conditions and motion blur information in the context metadata C are encoded respectively, and the encoded feature vectors are concatenated and input into a multilayer perceptron (MLP) for processing to obtain the environmental feature vector E(C). S22, Cross-modal attention fusion: The visual feature vector V(B), semantic feature vector L(C), and environmental feature vector E(C) are fused to obtain a fused feature vector. ; S23, Dynamic confidence calculation: The fused feature vector... Input another multilayer perceptron (MLP) to calculate the initial existence plausibility score of the target. The intensity of light interference is obtained by mapping the environmental feature vector E(C). and motion blur intensity And combined with pre-configured illumination interference weighting coefficients and motion blur interference weight coefficient Regarding the above Environmental reliability calibration was performed to obtain dynamic verification confidence levels. .

[0011] As a further preferred embodiment, in step S23, the dynamic verification confidence level... Calculated using the following formula: .

[0012] As a further preferred embodiment, the illumination interference weighting coefficient The value range is 0.1 to 0.3; the motion blur interference weight coefficient The value range is 0.05 to 0.2.

[0013] As a further preferred embodiment, in step S30, the execution of the classification decision specifically includes: If the confidence level is dynamically reviewed Greater than or equal to the high confidence threshold If so, then an acknowledgment alarm will be output; If the confidence level is dynamically reviewed Less than And greater than or equal to the low confidence threshold Then, it is further determined whether the time interval between the current alarm and the previous alarm is less than the false alarm threshold. If so, output a false alarm; otherwise, output a low-priority alarm that requires manual review. If the confidence level is dynamically reviewed Less than If so, the output will show "Not approved". in, , and All of these are pre-configured threshold parameters.

[0014] As a further preferred option, the high confidence threshold The value range is 0.8 to 0.9, and the low confidence threshold is... The value range is 0.65 to 0.75, and the false alarm threshold for the time interval is... The value ranges from 1 second to 3 seconds.

[0015] Secondly, this application provides an edge-cloud collaborative alarm review system, including: End-side devices are used to collect alarm information and generate target bounding box coordinates and context metadata; A cloud-based server is used to execute the methods described in any of the above statements and output the verification results.

[0016] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0017] Figure 1This is a flowchart of the edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding provided in this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] like Figure 1 As shown, this application provides a cloud-edge collaborative alarm review decision-making method based on multimodal semantic understanding, including steps S10 to S30, which are detailed below: Step S10: Filter the alarm information reported by the terminal side. The filtering includes alarm trigger determination based on time interval and duplicate alarm determination based on the overlapping area of ​​the detection box.

[0020] This step determines whether to trigger a deep review in the cloud by judging whether the time interval between consecutive alarms exceeds a threshold, thus avoiding the repeated processing of similar alarms generated in a short period of time. At the same time, by calculating the overlap area of ​​two adjacent potential alarm target detection boxes and comparing it with a threshold, redundant alarms targeting the same target can be filtered out. The two working together can significantly reduce the number of alarms requiring deep processing on the cloud side.

[0021] Step S20: For the alarms retained after filtering in step S10, perform multimodal semantic verification based on their target box coordinates and associated contextual metadata, and calculate the dynamic verification confidence. The contextual metadata includes at least lighting conditions, weather conditions, and motion blur information.

[0022] This step achieves a deep understanding and evaluation of alarms by fusing visual target features with contextual semantics containing environmental information. Specifically, it extracts the target's appearance features and combines them with scene descriptions and environmental interference factors to comprehensively assess the reasonableness and reliability of the target's existence in the current context, thereby generating a more accurate and dynamic confidence score, providing a core basis for subsequent decision-making.

[0023] Step S30: Based on the dynamic review confidence level, perform classification decision to output the final alarm review result.

[0024] This step, based on the calculated dynamic confidence level, can employ multi-level thresholds for adjudication. Specifically, alarms can be categorized into different classes, such as high-confidence confirmed alarms, low-priority alarms requiring further analysis, or alarms directly classified as false alarms. This achieves automated and refined classification and output of alarm results, thereby improving the efficiency and accuracy of system processing.

[0025] The edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding provided in this application has the following effects: By pre-filtering alarms reported from the edge based on time interval and spatial overlap, the computing load on the cloud can be effectively reduced. Furthermore, multimodal semantic understanding is introduced into the filtered alarms, upgrading the review basis from a single visual confidence level to a joint evaluation model that integrates visual evidence, scene rationality, and environmental reliability. Through the synergistic effect of the above technical features, this method can achieve dynamic and intelligent review of alarms in the cloud, comprehensively considering the semantic rationality of the target's appearance and the quality impact of the current imaging environment, thereby significantly improving the accuracy of alarm judgment and effectively reducing the false alarm rate.

[0026] In one embodiment, the technical solution to achieve the above objective can be as follows: This embodiment provides a dynamic semantic verification algorithm, the core of which lies in realizing intelligent verification and classification decision-making for end-side alarms through parameterized modeling and multimodal semantic fusion. The algorithm includes the following steps: Step S1: Initial Intelligent Judgment Determine whether to trigger a deep cloud-side review based on the time interval between the current alarm and the previous alarm at a specific video monitoring point received from the cloud:

[0027] in The threshold for the first alarm interval (range: 3~10 seconds); the subsequent multimodal semantic review process is triggered only when F=1, otherwise the AI ​​call for the current alarm is skipped to avoid duplicate processing.

[0028] Step S2: Dynamic filtering of overlapping regions For the same target in two adjacent potential alarm triggers belonging to a certain video monitoring point, calculate two detection boxes. and The overlapping area. Wherein coordinate information These represent the X and Y coordinates of the top left corner and the bottom right corner of the first detection box, respectively. The coordinate information for 2 has a similar meaning.

[0029] When two detection boxes have overlapping areas, their intersection is also a rectangle; Top left corner of the intersection: ; The top left x-coordinate is used The vertical axis is represented by... express; Bottom right corner of the intersection: ; The bottom right x-coordinate is used The vertical axis is represented by... express; Valid intersection conditions: ; Therefore, the overlapping area (intersection of detection boxes) is:

[0030] If the following formula is met, it is determined to be a duplicate alarm and filtered out:

[0031] in Let the area be the area of ​​the two detection boxes. Threshold for determining overlapping areas (value range: 0.6~0.85).

[0032] This approach does not employ an IOU (Intersection over Union)-based scheme. The main reason is that when two detection boxes are given, if they partially overlap, the IOU scheme, because its union area is larger than the area of ​​either detection box, makes it less likely to satisfy the filtering criteria for the same intersection area. In actual alarm applications, when the intersection area of ​​two detection boxes covers a certain proportion of a larger detection box, it likely indicates that both have captured the main feature region of the target, and can be considered duplicate detection of the same target, triggering filtering.

[0033] Step 3: Multimodal semantic verification The target bounding box coordinates reported by the input side Coordinate information These represent the X and Y coordinates of the top-left and bottom-right corners of the target bounding box, respectively, and their associated contextual metadata. : The contextual metadata includes the time, location, lighting environment, weather conditions, and motion blur of the warning event.

[0034] 1. Multimodal feature extraction Through visual branches Feature extraction: based on the coordinates of the target bounding box Image patches of the target area are cropped from the original panoramic surveillance image, then resized to a fixed size (e.g., 224x224) and pixel values ​​are normalized. These patches are then input into a ViT (Vision Transformer) model for feature encoding, resulting in... It is a high-dimensional dense vector (e.g., 768-dimensional) that encodes deep semantic information such as the appearance, shape, texture, key components (e.g., bulldozer bucket, reflective safety vest, facial features) and pose of the target.

[0035] Through language branches Generate scene description: Transform structured context metadata The text is converted into a natural scene description. Then it is input into the CLIP model's text encoder (usually a Transformer) to obtain its sentence vector representation. It is a high-dimensional vector aligned with the visual feature space. It encodes a semantic description of the current overall scene, such as "a raw material transfer warehouse at night, dimly lit, with unknown weather, and a blurry image." It leverages world knowledge gained from CLIP pre-training on large-scale image-text pairs to provide a contextual semantic background for visual features.

[0036] Scene structured coding For the categorical variables "light" (e.g., "dim", "normal", "overexposed") and "weather" (e.g., "sunny", "rainy", "foggy"), one-hot encoding or embedding encoding is performed. For the ordered variable "motion_blur" (e.g., "none", "slight", "severe"), it is mapped to the [0,1] interval. The encoded feature vectors are concatenated, and then a shallow multilayer perceptron (MLP) (e.g., 2-3 layers) is used for nonlinear transformation and dimensionality reduction. The resulting... It is a low- or medium-dimensional dense vector. It specifically and accurately encodes environmental interference factors that directly affect image quality and the difficulty of visual recognition, while stripping away semantic information such as time and location, which facilitates subsequent quantization calibration.

[0037] 2. Cross-modal attention fusion

[0038] By actively "querying" and fusing relevant contextual information using visual features, a unified scene perception feature is formed.

[0039] Computational visual features With context features The dot product similarity of each element is used to calculate the similarity weights (after Softmax) and apply them to the... The weighted sum is obtained. Visual feature V(B) is enhanced by extracting the most relevant parts from the context. For example, if the visual feature is blurry, it will give higher weight to the "motion blur" component in E(C) and the "dim lighting" description in L(C). It is a visual-contextual joint feature. It not only includes the appearance of the target itself, but also incorporates the understanding of its surrounding environment, such as "a humanoid target that may appear blurry in dim light", as the basis for subsequent judgment of the "reasonableness" of its existence.

[0040] 3. Behavioral rationality scoring and environmental calibration

[0041] Fusion features The input is another multilayer perceptron (MLP) (typically a binary classification or regression head), whose weights are learned on the training data and used to capture complex patterns of "whether the appearance of a certain visual target is reasonable in a certain context." The resulting... It is a scalar between 0 and 1, representing the initial confidence level of the object's existence based on scene semantics, without considering specific image quality degradation. For example, detecting a "forklift" in a "warehouse". It will be very high; a giraffe was detected in the "bedroom". It will be extremely low.

[0042]

[0043] Light interference intensity It is from The values ​​mapped from metadata, such as (dim -> 0.8, normal -> 0.2, overexposed -> 0.6), represent the degree of image distortion caused by unsatisfactory lighting conditions. Motion blur intensity. These are numerical values ​​directly mapped from the metadata (none -> 0, slight -> 0.4, severe -> 0.9), representing the degree to which the target's motion causes image blurring.

[0044] The light interference weighting coefficient represents the maximum reduction rate of confidence level due to light interference intensity. Based on numerous experiments, a value range of 0.1 to 0.3 is recommended, indicating that this factor can reduce confidence level by up to 10% to 30%. The motion blur interference weight coefficient is the maximum reduction rate of confidence due to motion blur intensity. The effective value range is generally 0.05~0.2. Its influence is usually weaker than that of illumination because modern algorithms have a certain robustness to non-extreme blur.

[0045] Environmental reliability factor This is a multiplier of ≤1, directly quantifying the impact of the current imaging environment quality on the reliability of the visual algorithm. The final result... It is the final verification confidence level of the algorithm. It is based on semantic reasonableness. A comprehensive evaluation was conducted on two dimensions: perceived reliability and environmental factors. A high The objective implies that: 1) its appearance in this scenario is reasonable; 2) the current image quality is sufficient to support the reliability of this judgment. Only... Only when the reported target exceeds the preset threshold should the final alarm be triggered.

[0046] Step S4: Outcome Classification Decision Based on the dynamic confidence level assessment results, a three-level decision is executed: High confidence level: -like If so, then output "Device alarm"; Low confidence level requires further analysis: - like ,but like If the output is "Device false alarm (repeatedly occurs within a short period of time, but with low confidence, it is likely a false alarm);", then the output will be "Device false alarm (repeatedly occurs within a short period of time, but with low confidence, it is likely a false alarm)". Otherwise, output "low priority alarm" (manual review or observation is recommended); Low confidence level: - like If the output is "Device not verified", then the output will be "Device not verified".

[0047] in The high confidence threshold (range: 0.8~0.9). The low confidence threshold (range: 0.65~0.75) The false alarm threshold for the time interval (range: 1~3 seconds).

[0048] The following is a specific implementation example of this application: Taking the "personnel / vehicle intrusion" in a smart park as an example, this illustrates the dynamic parameter configuration mechanism in the algorithm. All threshold parameters support dynamic updates via OTA.

[0049] Scenario 1: High-risk perimeter area For high-risk areas such as perimeter walls, a rapid response and strict verification strategy is adopted.

[0050] Set a shorter first-report interval ( =3 seconds), ensuring that the first alarm quickly triggers a deep review in the cloud.

[0051] Use a higher overlap filtering threshold ( =0.75), requiring the target boxes to overlap significantly to be considered a duplicate alarm, thus avoiding missed alarms.

[0052] This scenario is particularly important for nighttime monitoring, so the weighting of light interference needs to be increased. =0.25); When severe weather occurs, increase the motion blur weight ( =0.15), to strengthen compensation for environmental disturbances.

[0053] Set a strict confidence threshold ( =0.85, =0.7), only high-confidence events trigger direct alarms.

[0054] Set a shorter low-confidence false alarm detection interval ( =2 seconds), quickly suppressing periodic false alarms such as the sound of wind and grass.

[0055] Scene 2: Ordinary park roads For low-risk areas such as internal roads, strategies to suppress false alarms and reduce interference are adopted.

[0056] Set a longer first-report interval ( =10 seconds), reducing the frequency of cloud-based review.

[0057] Use a lower overlap filtering threshold ( =0.6), loosely filtering similar targets in consecutive frames.

[0058] Using basic environmental interference weights ( =0.1, =0.1), usually day / night light conditions are relatively stable.

[0059] Set a lenient confidence threshold ( =0.75, =0.65), more targets are a reasonable phenomenon.

[0060] Set a longer interval for judging low-confidence false alarms ( =5 seconds), to avoid misjudging reasonable targets that appear briefly (such as temporary parking) as false alarms.

[0061] Scenario 3: Single-factor experiment on open flame monitoring in an electric vehicle shed (simulating a single lane) in the park. First notification interval setting =5 seconds and settings =15 seconds, with other configurations remaining unchanged, the number of concurrent cloud testing review tasks (i.e., the number of computing power application queued tasks) is reduced from 4 to 1.

[0062] Overlapping region determination threshold setting =0.8 and settings =0.5, with other configurations remaining unchanged, the number of concurrent cloud testing review tasks (i.e., the number of computing power application queued tasks) is reduced from 7 to 2.

[0063] Note: Changes in the number of concurrent connections have relative optimization significance, but their absolute values ​​are affected by several other factors and are only valid within the current experimental configuration.

[0064] The algorithm provided in this embodiment achieves the following beneficial effects through the above-described parameterized design and multimodal semantic fusion: 1. By introducing a multimodal semantic fusion design, the basis for verifying alarms from on-site devices is upgraded from a single visual confidence level to a joint probability model of (visual evidence × scenario rationality × environmental reliability), thereby significantly improving the accuracy of alarms and directly improving the user experience.

[0065] 2. The concept of 'first alert' is introduced to assess the reasonable frequency of alarms and quantify their utilization. 'Overlapping area' filters redundant alarm targets. The combination of these two designs significantly saves cloud computing power consumption.

[0066] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A cloud-edge collaborative alarm review decision-making method based on multimodal semantic understanding, characterized in that, Includes the following steps: S10, filter the alarm information reported by the end side. The filtering includes alarm trigger determination based on time interval and duplicate alarm determination based on the overlapping area of ​​the detection box. S20: For the alarms retained after filtering in step S10, perform multimodal semantic verification based on their target box coordinates and associated contextual metadata, and calculate the dynamic verification confidence level; the contextual metadata includes at least lighting conditions, weather conditions, and motion blur information. S30, based on the dynamic verification confidence level, perform a classification decision to output the final alarm verification result.

2. The edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding as described in claim 1, characterized in that, In step S10, the alarm trigger determination based on time interval specifically includes: Receive current alarms and their timestamps ; Get the timestamp of the last alarm ; Calculate the time interval ΔT = - ; If ΔT is greater than the pre-configured first alarm time interval threshold α, then the subsequent deep review process is triggered; otherwise, the review of the current alarm is skipped.

3. The edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding as described in claim 1, characterized in that, In step S10, the determination of repeated alarms based on the overlapping area of ​​the detection boxes specifically includes: Obtain the coordinates of two adjacent potential trigger target detection boxes. The coordinates of the first detection box (box1) are: The coordinates of the second detection box (box2) are: ,in These represent the X-coordinate of the top left corner, the Y-coordinate of the top left corner, the X-coordinate of the bottom right corner, and the Y-coordinate of the bottom right corner of the first detection box, respectively. These represent the X-coordinate of the top left corner, the Y-coordinate of the top left corner, the X-coordinate of the bottom right corner, and the Y-coordinate of the bottom right corner of the second detection box, respectively. Calculate the intersection area of ​​the two detection boxes. ; Calculate the area of ​​each of the two detection boxes. ; like The overlap region determination threshold γ is greater than the pre-configured threshold value. If the product of these is used, it is considered a duplicate alarm and is filtered out.

4. The edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding as described in claim 3, characterized in that, Calculate the intersection area of ​​the two detection boxes. Specifically: Calculate the coordinates of the top left corner of the intersection region. , ),in , ; Calculate the coordinates of the lower right corner of the intersection region. , ),in , ; If satisfied ,but ,otherwise .

5. The edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding as described in claim 1, characterized in that, In step S20, the step of performing multimodal semantic verification and calculating dynamic verification confidence specifically involves: S21, Multimodal Feature Extraction, including: S211, Visual feature extraction: Based on the target box coordinates B, the corresponding target area image block is cropped from the original image data associated with the alarm information. The image block is adjusted to a fixed size and the pixel value is normalized. Then, it is input into the visual Transformer model for feature encoding to obtain the visual feature vector V(B). S212, Semantic feature extraction: The context metadata C is converted into scene description text in natural language form, and input into the text encoder of the CLIP model for feature encoding to obtain the semantic feature vector L(C); S213, Environmental feature extraction: The illumination conditions, weather conditions and motion blur information in the context metadata C are encoded respectively, and the encoded feature vectors are concatenated and input into a multilayer perceptron (MLP) for processing to obtain the environmental feature vector E(C). S22, Cross-modal attention fusion: The visual feature vector V(B), semantic feature vector L(C), and environmental feature vector E(C) are fused to obtain a fused feature vector. ; S23, Dynamic confidence calculation: The fused feature vector... Input another multilayer perceptron (MLP) to calculate the initial existence plausibility score of the target. The intensity of light interference is obtained by mapping the environmental feature vector E(C). and motion blur intensity And combined with pre-configured illumination interference weighting coefficients and motion blur interference weight coefficient Regarding the above Environmental reliability calibration was performed to obtain dynamic verification confidence levels. .

6. The edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding as described in claim 5, characterized in that, In step S23, the dynamic verification confidence level Calculated using the following formula: 。 7. The edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding as described in claim 5, characterized in that, The light interference weighting coefficient The value range is 0.1 to 0.3; the motion blur interference weight coefficient The value range is 0.05 to 0.

2.

8. The edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding as described in claim 1, characterized in that, In step S30, the classification decision is specifically performed as follows: If the confidence level is dynamically reviewed Greater than or equal to the high confidence threshold If so, an acknowledgment alarm will be output; If the confidence level is dynamically reviewed Less than And greater than or equal to the low confidence threshold Then, it is further determined whether the time interval between the current alarm and the previous alarm is less than the false alarm threshold. If so, output a false alarm; otherwise, output a low-priority alarm that requires manual review. If the confidence level is dynamically reviewed Less than If so, the output will show "Not approved". in, , and All of these are pre-configured threshold parameters.

9. The edge-cloud collaborative alarm review decision-making method based on multimodal semantic understanding as described in claim 8, characterized in that, The high confidence threshold The value range is 0.8 to 0.9, and the low confidence threshold is... The value range is 0.65 to 0.75, and the false alarm threshold for the time interval is... The value ranges from 1 second to 3 seconds.

10. A cloud-edge collaborative alarm review system, characterized in that, include: End-side devices are used to collect alarm information and generate target bounding box coordinates and context metadata; A cloud-based server is used to execute the method as described in any one of claims 1 to 9 and output the verification results.

Images