Open scene zero sample anomaly detection method, system and device with semantic dynamic evolution capability and medium

Abstract

The invention discloses an open scene zero sample anomaly detection method, system and device with semantic dynamic evolution capability and a medium. The method comprises the following steps: acquiring a feature vector of scene data through a multi-modal feature extractor; projecting the feature vector to a pre-trained prototype embedding space, performing similarity comparison with a prototype vector, and judging abnormality based on a dynamic threshold value; when the confidence index shows that matching is unreliable, a double-layer semantic evolution mechanism is triggered, reciprocal learning loss is further introduced to relieve the centralization problem of a prototype embedding space, and symmetry and robustness of a similarity relation are ensured. According to the invention, autonomous perception, accurate description and continuous evolution of unknown abnormal types in an open scene are realized without a large amount of manual intervention, the detection precision, the adaptive ability and the stability of the system are effectively improved, and the method can be widely applied to the technical fields of intelligent transportation, smart cities, medical image diagnosis, industrial internet and the like.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data analysis technology, and in particular to a method, system, device, and medium for zero-shot anomaly detection in open scenarios with semantic dynamic evolution capabilities. The method, system, device, and medium can be widely applied in fields such as intelligent transportation, smart cities, industrial internet, mobile robots, and medical imaging diagnosis. Background Technology

[0002] In relevant application areas, real-time monitoring of various abnormal events is crucial for system security and status assessment. AI algorithms have been widely used in abnormal event detection tasks. These algorithms rely on massive amounts of labeled data to train models, analyze multimodal data online, identify common abnormal events, and then report the identification results to a manual review stage to trigger an alarm.

[0003] However, in some fields (such as smart cities and the industrial internet), anomalies inherently possess open characteristics, requiring continuous identification of new features. In other fields (such as intelligent transportation roads), as circumstances evolve, spaces that were originally considered closed worlds are gradually transforming into quasi-open scenarios, leading to a proliferation of new events (such as road collapses, animal intrusions, drone crashes, and LiDAR interference). Because these anomalies generally lack labeled data, traditional AI algorithms are not highly sensitive to them, reducing the timeliness and accuracy of anomaly detection.

[0004] In summary, the technical problems existing in the relevant fields need to be improved. Summary of the Invention

[0005] In view of this, this patent proposes an open-scene zero-sample anomaly detection method, system, device and medium with semantic dynamic evolution capability.

[0006] A zero-shot anomaly detection method for open scenes with semantic dynamic evolution capability, the method comprising: Step S1. Extract the data features of the scene to be detected and obtain the feature vector through processing. F ; Step S2. Transfer the feature vector F Projected into the pre-trained prototype embedding space, and compared with multiple prototype vectors in that prototype embedding space. c A similarity comparison is performed to obtain a similarity score. ; Step S3. Based on the similarity score Similarity dynamic threshold To make a comparison, if Then it is determined that an anomaly exists; Step S4. Calculate the confidence level; if the result is less than the threshold... When the semantic evolution process is triggered, at least one prototype vector in the prototype embedding space is updated.

[0007] Furthermore, the pre-training method for the prototype embedding space in step S2 includes the following steps: Step S21. Construct a training dataset, which includes normal samples and anomalous samples of multiple known categories, each anomalous sample containing unimodal or multimodal data; Step S22. Initialize the prototype embedding space, which contains multiple prototype vectors. c Each prototype vector c It corresponds to a semantic concept and is used as a known anomaly category; Step S23. The training process of the prototype embedding space includes a prototype contrastive learning mechanism, a domain adaptation calibration mechanism, and a reciprocal learning mechanism. The training of the prototype embedding space and the feature extraction network is completed by iteratively optimizing the total loss function, whereby the total loss function satisfies...

[0008] in, For prototype comparison loss, For domain adaptation loss, For mutual inverse learning loss, This is the regularization factor term for the feature extraction network. , , These are the weight coefficients for training the prototype embedding space; Step S24. Repeat step S23 until the model converges, and the trained prototype embedding space is obtained.

[0009] Furthermore, the prototype contrast loss in step S23 is obtained as follows: For each training sample, calculate the similarity between the feature representation of the training sample and the positive prototype vector, and calculate the similarity between the feature representation of the training sample and other negative prototype vectors, and then calculate the cross-entropy loss. In the single-modal case, the positive prototype vector comes from the text description of the same modality, and the cross-entropy loss satisfies...

[0010] in, For feature vectors Data; The text prototype vector generated by the text encoder. The positive prototype vector that matches the current feature vector; It is a cosine approximation calculation, specifically meaning: ; This is the temperature coefficient used in prototype comparison to adjust the degree of attention given to difficult samples; The total number of prototype vectors. This represents the number of items compared in a batch. In the multimodal case, the positive prototype vector comes from another modality, and the cross-entropy loss satisfies

[0011] in, and They are the feature vectors The data vectors of the first and second modes; and They are the prototype vectors. The data vectors of the first and second modes, and These are the positive prototype vectors that match the current feature vector.

[0012] Furthermore, the domain adaptation calibration mechanism in step S23 reduces the distribution difference between training and test data through the Wasserstein distance domain adaptation mechanism. This Wasserstein distance domain adaptation mechanism calculates the feature distribution distance between the source and target domains using a discriminator and ensures the Lipschitz continuity of the discriminator through a gradient penalty term. The domain adaptation loss satisfies:

[0013] in, The number of modes for domain adaptation enhancement, For the first m Feature extraction function for each modality For the first m A discriminator function for each modality. For the first m The scores obtained by inputting the results of modal feature extraction into the corresponding discriminator; It is the source domain of the Wasserstein distance domain adaptation mechanism. m The expected score of a modality after its features are processed by a discriminator. It is the target domain in the Wasserstein distance domain adaptation mechanism. m The expected score of a modality after its features are processed by a discriminator; These are the weights for the gradient penalty, used to balance the distance loss and the gradient penalty loss. The gradient penalty term is calculated at the interpolation point. The expectation of the square of the difference between the gradient norm of the discriminator and 1 is used to force the discriminator to satisfy 1-Lipschitz continuity.

[0014] Furthermore, the reciprocal learning mechanism in step S23 alleviates the bias problem in prototype matching by simultaneously optimizing the similarity calculation in both the feature-prototype and prototype-feature directions, thereby improving the model's ability to identify samples of unknown categories. Specific implementation methods include: For single-modal data, the inverse learning loss is...

[0015] in, For the first i Sample label The corresponding text prototype vector; For multimodal data, the inverse learning loss is...

[0016] in, The comparison loss from the first modality features to the second modality prototype data satisfies...

[0017] The comparison loss from the second modality features to the first modality prototype data satisfies

[0018] and For the first i Sample label The corresponding first and second mode prototype vectors.

[0019] Further, the similarity score of step S2 is calculated. The dynamic threshold for similarity with step S3 "Entering" includes: For unimodal data, similarity score The calculation method satisfies

[0020] Similarity dynamic threshold The calculation satisfies

[0021] For multimodal data, similarity score The calculation method satisfies

[0022] Similarity dynamic threshold The calculation satisfies

[0023] in, For cosine similarity calculation, The baseline threshold is set based on historical data. Scaling factor To calculate the standard deviation of similarity sequences, Weighting coefficients to balance the contributions of the first and second modes.

[0024] Furthermore, for single-modal data, similarity score The calculation method satisfies

[0025] For multimodal data, similarity score The calculation method satisfies

[0026] The confidence threshold Designed for

[0027] in, For the past N Average confidence level of each effective detection For this N The confidence standard deviation of the test. To balance the weighting coefficients of the contributions from the first and second modes, This is the scaling factor; If and If all values ​​are set to zero, the system will no longer perform semantic evolution. That is, steps S1, S2, and S3 constitute an independent zero-sample anomaly detection process, which runs without executing step S4.

[0028] Furthermore, the semantic evolution process in step S4 includes: Step S41. Real-time template update closed loop; Step S42. Close the loop of training the policy network.

[0029] Furthermore, the real-time template update closed loop is executed during the inference phase, including: Step S411. Construct a state space, which includes the current semantic template vector, the historical confidence sequence, and the entropy value of the current prediction distribution; Step S412. Input the state space into the policy network. Output the original update action action; Step S413. Perform the original update action. The actions are filtered using a causal mask and adjusted using a time decay factor to obtain the adjusted actions. ; Step S414. Perform the adjusted action. Perform semantic projection normalization to obtain the normalized action. Ensure that the updated template vector lies within the unit sphere; Step S415. Actions after applying the aforementioned specifications Update the semantics of the current event description.

[0030] Furthermore, the state space of step S411 The mathematical representation of is

[0031] in, For the current template The generated dimensional vector, Before the corresponding semantic template time window Historical confidence of frames For the current template The predicted distribution entropy value.

[0032] Furthermore, the policy network in step S412 It is a multilayer perceptron (MLP) with input dimension . ,in For template vector dimensions, The time window size for the historical confidence series, with the output dimension being... The hidden layers contain 256 and 128 neurons respectively, using the GeLU activation function, while the output layer uses the tanh activation function to constrain the action range. Within.

[0033] Furthermore, the causal mask in step S413 is generated based on a time sliding window, satisfying...

[0034] in, The size of the time-sliding window, in frames.

[0035] Furthermore, the implementation of the semantic projection normalization operation in step S414 satisfies

[0036] in, Let be the radius of a unit sphere, where This represents the number of dimensions of the semantic vector.

[0037] Furthermore, the policy network training loop is executed periodically during the training phase to optimize the policy network. parameters ,include: Step S421. Collect state-action-reward trajectory data generated by the real-time template update loop; Step S422. Optimization algorithm and reward function based on proximal policy Calculate the policy gradient; Step S423. Update the policy network using the policy gradient. The parameters are adjusted to improve the long-term benefits of its generated template update actions.

[0038] Furthermore, the policy network parameter update in step S422 uses the Proximal Policy Optimization (PPO) algorithm, and the loss function satisfies...

[0039] in, The probability ratio of the new strategy to the old strategy satisfies

[0040] in, As the dominance function, the This refers to the cutting range.

[0041] Furthermore, the reward function of the proximal optimization algorithm in step S422 Designed for

[0042] in, , , These are the weight coefficients used in training the policy network. For anomaly detection recall rate, To predict the distribution entropy value, To adjust and standardize the actions L2 norm penalty.

[0043] An open-scene zero-shot anomaly detection system with semantic dynamic evolution capability, the system comprising: The data acquisition and feature extraction module is used to acquire multimodal sensing data of the scene to be detected and process the data to obtain feature vectors. F ; The prototype comparison and anomaly decision-making module, connected to the data acquisition and feature extraction module, is used to process the feature vector. FProjected into a pre-trained prototype embedding space, along with multiple prototype vectors. c Similarity scores are obtained by performing similarity comparisons. And based on the similarity score and a dynamic threshold Compare to determine if any anomalies exist; A semantic dynamic evolution engine, connected to the prototype comparison and anomaly decision module, is used to trigger a semantic evolution process to update at least one prototype vector in the prototype embedding space when the confidence index calculated based on the similarity score is lower than the confidence threshold.

[0044] A computer device, the device comprising: At least one processor; At least one memory for storing at least one program; When the program is executed by the processor, the processor performs the method described above.

[0045] A computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the above-described method.

[0046] Beneficial effects The embodiments of this application include at least the following beneficial effects: 1) Through reinforcement learning-driven dynamic template evolution, the system can autonomously generate or optimize semantic descriptions of unknown events based on real-time perceived data, significantly improving the detection capability and semantic generalization of unknown anomalies, and realizing true open-world learning.

[0047] 2) By introducing causal masking and time decay factor, the template update process is strictly constrained within the recently effective context window, which effectively suppresses historical noise interference and semantic drift, enhances system stability, and effectively prevents outdated or accidental false detection events (such as nighttime reflection misjudged as reverse movement) from polluting the long-term template quality.

[0048] 3) By reasonably introducing a reciprocal learning mechanism, the model is forced to learn symmetrical similarity relationships, fundamentally alleviating the Hubness problem. This effectively prevents a few prototypes from becoming "hub points" while most prototypes are "marginalized" in the embedding space, ensuring that all prototypes (including those evolving for new categories) are matched fairly. This effectively improves the zero-sample retrieval success rate for scarce categories, constructs a more balanced and robust embedding space, and further enhances the overall fairness and reliability of the system.

[0049] 4) By decomposing the evolutionary process into lightweight, real-time small closed loops and resource-intensive, periodic large closed loops, the system places the main computational load on the non-real-time training phase, achieving a high-efficiency balance between computational efficiency and model performance, and adapting to edge deployment. Meanwhile, the lightweight Wasserstein domain adaptation module further ensures the model's robustness in different deployment scenarios, enabling it to run stably on resource-constrained edge devices. Attached Figure Description

[0050] Figure 1 The overall methodology and workflow for zero-sample anomaly detection; Figure 2 The modules of a zero-sample anomaly detection system; Figure 3 The prototype embedding space training method flow; Figure 4 The semantic template evolution method flow under the dual closed loop; Figure 5 This refers to the process of causal masking and projection normalization. Figure 6 This is a system framework diagram. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application.

[0052] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0053] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0055] In related technical fields, real-time monitoring of various abnormal events is crucial for system security and status assessment. AI algorithms have been widely applied in abnormal event detection tasks. These algorithms rely on massive amounts of labeled data to train models, analyze multimodal data online, identify common abnormal events, and then report the identification results to a manual review stage to trigger an alarm.

[0056] However, in some fields (such as smart cities and the industrial internet), anomalies inherently possess open characteristics, requiring continuous identification of new features. In other fields (such as intelligent transportation roads), as circumstances evolve, spaces that were originally considered closed worlds are gradually transforming into quasi-open scenarios, leading to a proliferation of new events (such as road collapses, animal intrusions, drone crashes, and LiDAR interference). Because these anomalies generally lack labeled data, traditional AI algorithms are not highly sensitive to them, reducing the timeliness and accuracy of anomaly detection.

[0057] Zero-shot learning is a relatively new AI paradigm that has emerged in recent years. By leveraging the semantic relationships (such as attributes, word vectors, or knowledge graphs) between known and unknown categories, models can identify or classify categories not seen during the training phase without requiring labeled samples of the target category. Its introduction not only solves the problem of sparse labeling of unknown anomalies but also avoids the repetitive labeling and model training of unknown events required by traditional methods, thereby effectively reducing the operational costs of various AI systems.

[0058] However, zero-shot anomaly detection technology still faces three core challenges in open scenarios. First, there's the issue of frequency domain drift and the semantic gap—models relying solely on fixed semantic templates and prototype libraries struggle to dynamically adapt to unknown event types (such as new types of traffic accidents or sudden geological disasters), leading to a sharp performance drop when scenarios change (such as lighting, weather, and sensor differences), and a significant mismatch between visual features and semantic descriptions. Second, there's the issue of historical noise accumulation and semantic drift—traditional template update mechanisms lack modeling of temporal causality; outdated or erroneous detection results can interfere with future states, causing template descriptions to gradually deviate from the original semantics (e.g., shifting "road debris" from large spills to include fallen leaves and pebbles), ultimately resulting in increased false alarm rates and template failure. Finally, there's the hubness problem—in high-dimensional prototype embedding spaces, some prototype vectors become "hubs," over-matched as the nearest neighbor for most queries, while other prototypes are rarely hit, severely compromising the fairness and accuracy of zero-shot classification, causing the model to completely fail for some hidden, unknown anomaly categories. Existing solutions, such as static reinforcement learning or simple contrastive learning, cannot systematically solve the above problems simultaneously.

[0059] The open-scene zero-shot anomaly detection method with dynamic semantic evolution capability provided in this application relates to the fields of artificial intelligence and data analysis technology. The open-scene zero-shot anomaly detection method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be an intelligent robot, smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing anomaly detection methods based on zero-shot prototype contrastive learning, but is not limited to the above forms.

[0060] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0061] Terminology Explanation Causal masking is a key constraint technique in deep learning used to handle causal dependencies in time series data, especially in Transformer models and reinforcement learning. Its core objective is to ensure that the model relies only on current and previous elements when processing a sequence, avoiding the leakage of future information, thereby correctly modeling temporal relationships.

[0062] Semantic projection refers to the process of mapping text, images, or other modalities of data into a feature space with semantic structure. Its core objective is to transform the original data into a vector representation that captures its semantic information through mathematical transformations (such as embedding and encoding), thereby enabling tasks such as similarity measurement, classification, and retrieval within that space.

[0063] Proximal Policy Optimization (PPO) is a reinforcement learning algorithm based on policy gradients, aiming to improve training stability by limiting the magnitude of policy updates. Its core idea is to constrain the update range between the old and new policies by introducing a pruning mechanism (PPO-clip) or a penalty term (PPO-penalty), preventing the policy from collapsing due to excessive deviation. This algorithm combines importance sampling and advantage function estimation, utilizing historical empirical data to optimize the current policy. Simultaneously, it ensures the rationality of updates through a trust region method, supports offline batch data updates, has high sample efficiency, and low sensitivity to hyperparameters, making it suitable for both continuous and discrete action spaces.

[0064] In the PPO optimization process, the advantage function measures the additional benefit of taking a certain action relative to the average performance of the current policy in a specific state. It provides directional guidance for policy updates by comparing the immediate value of an action with the long-term expected value of the state, helping the algorithm to efficiently adjust its policy. Specific Implementation See Figures 1-6 A zero-shot anomaly detection method for open scenes with semantic dynamic evolution capability, the method includes: Step S1. Extract the data features of the scene to be detected and obtain the feature vector through processing. F ; Step S2. Convert the feature vector F Projected into the pre-trained prototype embedding space, and compared with multiple prototype vectors in that prototype embedding space. c A similarity comparison is performed to obtain a similarity score. ; Step S3. Based on similarity score Similarity dynamic threshold To make a comparison, if Then it is determined that an anomaly exists; Step S4. Calculate the confidence level; if the result is less than the threshold... When this occurs, the semantic evolution process is triggered, updating at least one prototype vector in the prototype embedding space.

[0066] The pre-training method for the prototype embedding space in step S2 includes the following steps: Step S21. Construct a training dataset. The training dataset includes normal samples and anomalous samples of multiple known categories. Each anomalous sample contains single-modal data or multi-modal data (such as video streams and LiDAR sensing point cloud data). Step S22. Initialize the prototype embedding space, which contains multiple prototype vectors. c Each prototype vector c It corresponds to a semantic concept and is used as a known anomaly category; Step S23. The training process of the prototype embedding space includes a prototype contrastive learning mechanism, a domain adaptation calibration mechanism, and a reciprocal learning mechanism. The training of the prototype embedding space and the feature extraction network is completed by iteratively optimizing the total loss function, which satisfies the following conditions:

[0067] in, For prototype comparison loss, For domain adaptation loss, For mutual inverse learning loss, This is the regularization factor term for the feature extraction network. , , These are the weight coefficients for training in the prototype embedding space; Step S24. Repeat step S23 until the model converges, and the trained prototype embedding space is obtained.

[0068] The prototype contrast loss in step S23 is obtained as follows: For each training sample, the similarity between the feature representation of the training sample and the positive prototype vector is calculated, as well as the similarity between the feature representation of the training sample and other negative prototype vectors, and then the cross-entropy loss is calculated.

[0069] In the single-modal case, the positive prototype vectors come from text descriptions of the same modality, and the cross-entropy loss satisfies...

[0070] in, For feature vectors Data; The text prototype vector generated by the text encoder. The positive prototype vector that matches the current feature vector; It is a cosine approximation calculation, specifically meaning: ; This is the temperature coefficient used in prototype comparison to adjust the degree of attention given to difficult samples; The total number of prototype vectors. This represents the number of items compared in a batch.

[0071] In the multimodal case, the positive prototype vector comes from another modality, and the cross-entropy loss satisfies...

[0072] in, and They are the feature vectors The data vectors of the first and second modes; and They are the prototype vectors. The data vectors of the first and second modes, and These are the positive prototype vectors that match the current feature vector.

[0073] The domain adaptation calibration mechanism in step S23 reduces the distribution difference between training and test data through the Wasserstein distance domain adaptation mechanism. The Wasserstein distance domain adaptation mechanism calculates the feature distribution distance between the source and target domains through the discriminator and ensures the Lipschitz continuity of the discriminator through a gradient penalty term. The domain adaptation loss satisfies:

[0074] in, The number of modes for domain adaptation enhancement, For the first m Feature extraction function for each modality For the first m A discriminator function for each modality. For the first m The scores obtained by inputting the results of modal feature extraction into the corresponding discriminator; It is the source domain of the Wasserstein distance domain adaptation mechanism. m The expected score of a modality after its features are processed by a discriminator. It is the target domain in the Wasserstein distance domain adaptation mechanism. m The expected score of a modality after its features are processed by a discriminator; These are the weights for the gradient penalty, used to balance the distance loss and the gradient penalty loss. The gradient penalty term is calculated at the interpolation point. The expectation of the square of the difference between the gradient norm of the discriminator and 1 is used to force the discriminator to satisfy 1-Lipschitz continuity.

[0075] The reciprocal learning mechanism in step S23 alleviates the bias problem in prototype matching by simultaneously optimizing similarity calculations in both the feature-prototype and prototype-feature directions, thereby improving the model's ability to identify samples of unknown categories. Specific implementation methods include: For single-modal data, the inverse learning loss is...

[0076] in, For the first i Sample label The corresponding text prototype vector.

[0077] For multimodal data, the inverse learning loss is...

[0078] in, The comparison loss from the first modality features to the second modality prototype data satisfies...

[0079] The comparison loss from the second modality features to the first modality prototype data satisfies

[0080] and For the first i Sample label The corresponding first and second mode prototype vectors.

[0081] Calculate the similarity score in step S2. The dynamic threshold for similarity with step S3 "Entering" includes: For unimodal data, similarity score The calculation method satisfies

[0082] Similarity dynamic threshold The calculation satisfies

[0083] For multimodal data, similarity score The calculation method satisfies

[0084] Similarity dynamic threshold The calculation satisfies

[0085] in, For cosine similarity calculation, The baseline threshold is set based on historical data. Scaling factor To calculate the standard deviation of similarity sequences, Weighting coefficients to balance the contributions of the first and second modes.

[0086] For unimodal data, similarity score The calculation method satisfies

[0087] For multimodal data, similarity score The calculation method satisfies

[0088] The confidence threshold Designed for

[0089] in, For the past N Average confidence level of each effective detection For this N The confidence standard deviation of the test. To balance the weighting coefficients of the contributions from the first and second modes, This is the scaling factor.

[0090] If and If all values ​​are set to zero, the system will no longer perform semantic evolution. That is, steps S1, S2, and S3 constitute an independent zero-sample anomaly detection process, which runs without executing step S4.

[0091] The semantic evolution process in step S4 includes: Step S41. Real-time template update closed loop; Step S42. Close the loop of training the policy network.

[0092] The real-time template update closed loop is executed during the inference phase, including: Step S411. Construct the state space, which includes the current semantic template vector, the historical confidence sequence, and the entropy value of the current prediction distribution; Step S412. Input the state space into the policy network. Output the original update action action; Step S413. Perform the original update action. The actions are filtered using a causal mask and adjusted using a time decay factor to obtain the adjusted actions. ; Step S414. Adjust the action Perform semantic projection normalization to obtain the normalized action. Ensure that the updated template vector lies within the unit sphere; Step S415. Actions after applying the standard Update the semantics of the current event description.

[0093] State space of step S411 The mathematical representation of is

[0094] in, For the current template The generated dimensional vector, Before the corresponding semantic template time window Historical confidence of frames For the current template The predicted distribution entropy value.

[0095] Policy network in step S412 It is a multilayer perceptron (MLP) with input dimension . ,in For template vector dimensions, The time window size for the historical confidence series, with the output dimension being... The hidden layers contain 256 and 128 neurons respectively, using the GeLU activation function, while the output layer uses the tanh activation function to constrain the action range. Within.

[0096] The causal mask in step S413 is generated based on a time sliding window and satisfies...

[0097] in, The size of the time-sliding window, in frames.

[0098] Furthermore, the implementation of the semantic projection normalization operation in step S414 satisfies

[0099] in, Let be the radius of a unit sphere, where This represents the number of dimensions of the semantic vector.

[0100] The policy network training loop is executed periodically during the training phase to optimize the policy network. parameters ,include: Step S421. Collect state-action-reward trajectory data generated by the real-time template update loop; Step S422. Optimization algorithm and reward function based on proximal policy Calculate the policy gradient; Step S423. Update the policy network using policy gradients. The parameters are adjusted to improve the long-term benefits of its generated template update actions.

[0101] Step S422 updates the policy network parameters using the Proximal Policy Optimization (PPO) algorithm, with the loss function satisfying...

[0102] in, The probability ratio of the new strategy to the old strategy satisfies

[0103] in, For the dominant function, This refers to the cutting range.

[0104] The reward function of the proximal optimization algorithm in step S422 Designed for

[0105] in, , , These are the weight coefficients used in training the policy network. For anomaly detection recall rate, To predict the distribution entropy value, To adjust and standardize the actions L2 norm penalty.

[0106] An open-scene zero-shot anomaly detection system with semantic dynamic evolution capability, the system as follows: Figure 6 As shown, it includes: The data acquisition and feature extraction module is used to acquire multimodal sensing data of the scene to be detected and process the data to obtain feature vectors. F ; The prototype comparison and anomaly decision-making module, connected to the data acquisition and feature extraction module, is used to process the feature vector. F Projected into a pre-trained prototype embedding space, along with multiple prototype vectors. c Similarity scores are obtained by performing similarity comparisons. And based on the similarity score and a dynamic threshold Compare to determine if any anomalies exist; A semantic dynamic evolution engine, connected to the prototype comparison and anomaly decision module, is used to trigger a semantic evolution process to update at least one prototype vector in the prototype embedding space when the confidence index calculated based on the similarity score is lower than the confidence threshold.

[0107] A computer device, the device comprising: At least one processor; At least one memory for storing at least one program; When the program is executed by the processor, the processor performs the above-described method.

[0108] A computer-readable storage medium storing a computer program, characterized in that the computer program implements the above-described method when executed by a processor.

[0109] Real-world application scenarios Application 1: Drone fire inspection In forest fire patrols, drones equipped with visual sensors monitor large areas. However, due to the diverse forms of fire (such as open flames, smoke, and hotspots) and the limited computing resources of drones, online semantic evolution is not supported. This embodiment uses a pre-trained fixed prototype library for zero-shot anomaly detection.

[0110] Assume the initially solidified semantic template is as follows:

[0111] Fixed prototype library example: / / Weak fire, stable, dense smoke, ground / / Open flame, rapid, light smoke, tree canopy / / Open flame, deflagration, thick smoke, middle layer Testing process: When the drone observes the feature vector ,and The one with the highest similarity was identified as "open flame in the tree crown," triggering an alarm.

[0112] The beneficial effects achieved by this embodiment include: By solidifying the prototype library, significant computational overhead can be reduced, meeting the requirements for edge environment deployment; In environments with limited resources for drone terminals, real-time detection of common fire types can be basically achieved.

[0113] Application 2: Identifying the type of debris spilled by vehicles on rural roads When driving on rural roads, debris (such as cardboard boxes or plastic sheets) may suddenly appear behind a vehicle, making accurate classification difficult with traditional target detection methods. This embodiment improves detection accuracy by tracking the characteristics of the debris and updating the event description in real time.

[0114] Assuming the semantic template is at this time

[0115] The specific implementation process is as follows: (1) Trigger: A white plastic bucket was detected, and the initial template matching degree was low (Confidence=0.18). (2) Reasoning: After analyzing the visual and point cloud features, the strategy network outputs the original data. Action [+0.8, -0.3, -0.9, +0.1]; (3) Constraint: Causal masking ensures that this evolution is not affected by the "cardboard box" that was falsely detected 5 minutes ago; (4) Projection: Semantic projection prevents excessive growth of vector magnitude; (5) Update: ; (6) Iteration: After 3 similar iterations, the template converges to [0.75, -0.25, -0.85, 0.15].

[0116] The final semantic interpretation was "a rigid, relatively light, highly irregularly shaped, slightly reflective object." This description matched the object "plastic bucket" quite accurately, increasing the confidence level to 0.65. Based on this, the vehicle decision-making system determined that it was safe to detour, rather than brake urgently.

[0117] The beneficial effects achieved by this embodiment include: Causal masks prevent interference from past false detections (such as vehicle shadows); Template evolution was completed in a short period of time, resulting in a significant reduction in false positives.

[0118] Application 3: Complex anomaly detection and fault diagnosis of rotating machinery In high-end manufacturing, a large centrifugal compressor exhibited a combination of fault symptoms: rotor imbalance and poor bearing lubrication. These two faults show similarities in their vibration spectra, making them easily confused, and their interaction generates new characteristics that are difficult to describe using traditional single-dimensional methods. This embodiment demonstrates how the system evolves from an initial coarse description to precise diagnosis, and ultimately autonomously derives a key new dimension for distinguishing between these combined faults.

[0119] The specific technical implementation is as follows: (1) The initial template is designed based on expert experience, and is assumed to be:

[0120] Before evolution, its characteristic output may be interpreted as a general description of "high vibration, high high frequency noise, high temperature, and high data noise", but this cannot determine the primary and secondary causes. Further manual diagnosis is needed to determine whether the primary cause is "rotor imbalance" or "poor bearing lubrication".

[0121] (2) By continuously learning from a large amount of similar composite fault data, a key discriminative feature hidden outside the initial dimensions can be discovered—"sideband stability" (sideband stability for rotor imbalance, while sideband drift for poor lubrication). By combining and reinterpreting existing dimensions, the policy network further "creates" dimensions to describe this feature.

[0122] For example, The physical meaning of high-frequency energy has been functionally modified and expanded by the policy network. It no longer simply represents "energy magnitude," but is internalized by the system as a composite characteristic: "the coupling strength and stability between high-frequency energy and the dominant frequency." A higher value indicates a stronger correlation with poor bearing lubrication.

[0123] If a bearing lubrication failure occurs at this point, the characteristic output might be interpreted as "the main frequency amplitude is slightly out of range, but exhibits typical high-frequency coupling instability with weak temperature correlation." This is a precise and identifiable description of "poor bearing lubrication accompanied by slight rotor imbalance."

[0124] (3) Through multiple iterations of optimization, the dimension of the semantic template will be reinterpreted by the policy network as:

[0125] The beneficial effects achieved by this embodiment include: Through manual description and continuous learning and correction, the original one-sided and vague description can gradually evolve into a comprehensive and clear description. This process significantly reduces the manual annotation costs required by existing methods.

[0126] Application 4: Mobile robots determine obstacles in open scenes through online semantic evolution. A mobile robot navigates in an open environment and encounters unknown obstacles (such as a floor lamp). This embodiment applies online semantic evolution to non-anomaly detection tasks, enabling environmental interaction and decision-making. This is an application of online semantic evolution in non-anomaly detection.

[0127] Assume the evolutionary direction of semantic template generation and the decisions it drives are shown in the table below:

[0128] The final semantic interpretation is: "A movable, structurally stable, impenetrable object that may be an electrical appliance." This description allows the robot not only to know what it is, but also how to interact with it, achieving a closed loop from perception to action. This represents a significant leap forward compared to traditional knowledge graph approaches that can only label things like "floor lamp."

[0129] The beneficial effects achieved by this embodiment include: This patent's semantic evolution technology system can be applied beyond the field of anomaly detection. Through end-to-end learning, the robot's success rate in interacting with unknown obstacles has been greatly improved.

[0130] Other embodiments and extended applications of the present invention It should be noted that one of the core innovations of this invention lies in its powerful zero-sample recognition capability. Even without activating the semantic dynamic evolution module (i.e., without executing step S4 in claim 1), the system can still effectively handle unknown anomalies in most open scenarios through the prototype comparison-based feature matching mechanism (step S2) described in this invention. For example, in a perimeter security scenario in an industrial park with a relatively stable prototype library, the system can achieve efficient intrusion detection using only steps S1-S3. This basic operating mode, due to its high computational efficiency and fast response speed, also has significant industrial application value.

[0131] Furthermore, the semantic dynamic evolution process proposed in this invention (corresponding to step S4 in claim 1) is itself a general and decoupled semantic representation learning tool. Its core technology lies in utilizing reinforcement learning with temporal causal constraints to optimize semantic vectors, preventing historical noise interference and ensuring the rationality of the evolutionary direction. This module can be independent of the anomaly detection system and ported to other artificial intelligence fields that require dynamic adaptation to new concepts. For example, in an intelligent customer service system, this module can be used to dynamically optimize the semantic template of "user intent" based on user feedback; in continuous learning models, this module can be used to alleviate the problem of catastrophic forgetting. Example 4 of the above application scenario is also an example of this module being applied to non-anomaly detection. This demonstrates that the semantic evolution method has universality beyond the field of anomaly detection.

[0132] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention, and these will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A zero-shot anomaly detection method for open scenes with semantic dynamic evolution capability, characterized in that the method... include: Step S1. Extract the data features of the scene to be detected and obtain the feature vector through processing. F ; Step S2. Transfer the feature vector F Projected into the pre-trained prototype embedding space, and compared with multiple prototype vectors in that prototype embedding space. c A similarity comparison is performed to obtain a similarity score. ; Step S3. Based on the similarity score Similarity dynamic threshold To make a comparison, if Then it is determined that an anomaly exists; Step S4. Calculate the confidence level; if the result is less than the threshold... When the semantic evolution process is triggered, at least one prototype vector in the prototype embedding space is updated.

2. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 1, characterized in that, The pre-training method for the prototype embedding space in step S2 includes the following steps: Step S21. Construct a training dataset, which includes normal samples and anomalous samples of multiple known categories, each anomalous sample containing unimodal or multimodal data; Step S22. Initialize the prototype embedding space, which contains multiple prototype vectors. c Each prototype vector c It corresponds to a semantic concept and is used as a known anomaly category; Step S23. The training process of the prototype embedding space includes a prototype contrastive learning mechanism, a domain adaptation calibration mechanism, and a reciprocal learning mechanism. The training of the prototype embedding space and the feature extraction network is completed by iteratively optimizing the total loss function, whereby the total loss function satisfies... in, For prototype comparison loss, For domain adaptation loss, For mutual inverse learning loss, This is the regularization factor term for the feature extraction network. , , These are the weight coefficients for training the prototype embedding space; Step S24. Repeat step S23 until the model converges, and the trained prototype embedding space is obtained.

3. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 2, characterized in that, The prototype contrast loss in step S23 is obtained as follows: For each training sample, calculate the similarity between the feature representation of the training sample and the positive prototype vector, and calculate the similarity between the feature representation of the training sample and other negative prototype vectors, and then calculate the cross-entropy loss. In the single-modal case, the positive prototype vector comes from the text description of the same modality, and the cross-entropy loss satisfies... in, For feature vectors Data; The text prototype vector generated by the text encoder. The positive prototype vector that matches the current feature vector; It is a cosine approximation calculation, specifically meaning: ; This is the temperature coefficient used in prototype comparison to adjust the degree of attention given to difficult samples; This represents the total number of prototype vectors. This represents the number of items compared in a batch. In the multimodal case, the positive prototype vector comes from another modality, and the cross-entropy loss satisfies in, and They are the feature vectors The data vectors of the first and second modes; and They are the prototype vectors. The data vectors of the first and second modes, and These are the positive prototype vectors that match the current feature vector.

4. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 2, characterized in that, The domain adaptation calibration mechanism in step S23 reduces the distribution difference between training and test data through the Wasserstein distance domain adaptation mechanism. This Wasserstein distance domain adaptation mechanism calculates the feature distribution distance between the source and target domains using a discriminator and ensures the Lipschitz continuity of the discriminator through a gradient penalty term. The domain adaptation loss satisfies: in, The number of modes for domain adaptation enhancement, For the first m Feature extraction function for each modality For the first m A discriminator function for each modality. For the first m The scores obtained by inputting the results of modal feature extraction into the corresponding discriminator; It is the source domain of the Wasserstein distance domain adaptation mechanism. m The expected score of a modality after its features are processed by a discriminator. It is the target domain in the Wasserstein distance domain adaptation mechanism. m The expected score of a modality after its features are processed by a discriminator; These are the weights for the gradient penalty, used to balance the distance loss and the gradient penalty loss. The gradient penalty term is calculated at the interpolation point. The expectation of the square of the difference between the gradient norm of the discriminator and 1 is used to force the discriminator to satisfy 1-Lipschitz continuity.

5. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 2, characterized in that, The reciprocal learning mechanism in step S23 alleviates the bias problem in prototype matching by simultaneously optimizing similarity calculations in both the feature-prototype and prototype-feature directions, thereby improving the model's ability to identify samples of unknown categories. Specific implementation methods include: For single-modal data, the inverse learning loss is... in, For the first i Sample label The corresponding text prototype vector; For multimodal data, the inverse learning loss is... in, The comparison loss from the first modality features to the second modality prototype data satisfies... The comparison loss from the second modality features to the first modality prototype data satisfies and For the first i Sample label The corresponding first and second mode prototype vectors.

6. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 1, characterized in that, Calculate the similarity score in step S2. The dynamic threshold for similarity with step S3 "Entering" includes: For unimodal data, similarity score The calculation method satisfies Similarity dynamic threshold The calculation satisfies For multimodal data, similarity score The calculation method satisfies Similarity dynamic threshold The calculation satisfies in, For cosine similarity calculation, The baseline threshold is set based on historical data. Scaling factor To calculate the standard deviation of similarity sequences, Weighting coefficients to balance the contributions of the first and second modes.

7. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 1, characterized in that, For unimodal data, similarity score The calculation method satisfies For multimodal data, similarity score The calculation method satisfies The confidence threshold Designed for in, For the past N Average confidence level of each effective detection For this N The confidence standard deviation of the test. To balance the weighting coefficients of the contributions from the first and second modes, This is the scaling factor; If and If all values ​​are set to zero, the system will no longer perform semantic evolution. That is, steps S1, S2, and S3 constitute an independent zero-sample anomaly detection process, which runs without executing step S4.

8. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 1, characterized in that, The semantic evolution process in step S4 includes: Step S41. Real-time template update closed loop; Step S42. Close the loop of training the policy network.

9. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 8, characterized in that, The real-time template update closed loop is executed during the inference phase, including: Step S411. Construct a state space, which includes the current semantic template vector, the historical confidence sequence, and the entropy value of the current prediction distribution; Step S412. Input the state space into the policy network. Output the original update action action; Step S413. Perform the original update action. The actions are filtered using a causal mask and adjusted using a time decay factor to obtain the adjusted actions. ; Step S414. Perform the adjusted action. Perform semantic projection normalization to obtain the normalized action. Ensure that the updated template vector lies within the unit sphere; Step S415. Actions after applying the aforementioned specifications Update the semantics of the current event description.

10. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 9, characterized in that, The state space of step S411 The mathematical representation of is in, For the current template The generated dimensional vector, Before the corresponding semantic template time window Historical confidence of frames For the current template The predicted distribution entropy value.

11. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 9, characterized in that, The policy network in step S412 It is a multilayer perceptron (MLP) with input dimension . ,in For template vector dimensions, The time window size for the historical confidence series, with the output dimension being... The hidden layers contain 256 and 128 neurons respectively, using the GeLU activation function, while the output layer uses the tanh activation function to constrain the action range. Within.

12. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 9, characterized in that, The causal mask in step S413 is generated based on a time sliding window and satisfies... in, The time-based sliding window size is expressed in frames.

13. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 9, characterized in that, The implementation of the semantic projection normalization operation in step S414 satisfies in, Let be the radius of a unit sphere, where The number of dimensions of the semantic vector.

14. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 8, characterized in that, The policy network training loop is executed periodically during the training phase to optimize the policy network. parameters ,include: Step S421. Collect state-action-reward trajectory data generated by the real-time template update loop; Step S422. Optimization algorithm and reward function based on proximal policy Calculate the policy gradient; Step S423. Update the policy network using the policy gradient. The parameters are adjusted to improve the long-term benefits of its generated template update actions.

15. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 14, characterized in that, The policy network parameter update in step S422 uses the Proximal Policy Optimization (PPO) algorithm, and the loss function satisfies... in, The probability ratio of the new strategy to the old strategy satisfies in, As the dominance function, the This refers to the cutting range.

16. The open-scene zero-shot anomaly detection method with semantic dynamic evolution capability according to claim 14, characterized in that, The reward function of the proximal optimization algorithm in step S422 Designed for in, , , These are the weight coefficients used in training the policy network. For anomaly detection recall rate, To predict the distribution entropy value, To adjust and standardize the actions L2 norm penalty.

17. An open-scene zero-shot anomaly detection system with semantic dynamic evolution capability, characterized in that, The system includes: The data acquisition and feature extraction module is used to acquire multimodal sensing data of the scene to be detected and process the data to obtain feature vectors. F ; The prototype comparison and anomaly decision-making module, connected to the data acquisition and feature extraction module, is used to process the feature vector. F Projected into a pre-trained prototype embedding space, along with multiple prototype vectors. c Similarity scores are obtained by performing similarity comparisons. And based on the similarity score and a dynamic threshold Compare to determine if any anomalies exist; A semantic dynamic evolution engine, connected to the prototype comparison and anomaly decision module, is used to trigger a semantic evolution process to update at least one prototype vector in the prototype embedding space when the confidence index calculated based on the similarity score is lower than the confidence threshold.

18. A computer device, characterized in that, The device includes: At least one processor; At least one memory for storing at least one program; When the program is executed by the processor, the processor implements the method as described in any one of claims 1 to 16.

19. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 16.

Images