Video abnormal event detection method, electronic device, and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and particularly provides a video abnormal event detection method, an electronic device and a storage medium, and aims to solve the problem of how to accurately monitor abnormal events in a video by using a visual language large model. The method comprises the following steps: acquiring a visual language model which is sequentially pre-trained and post-trained; rewriting first prompt information of an abnormal event into second prompt information conforming to an information expression mode of a pre-training sample; controlling the visual language model to detect the abnormal event in the video according to the second prompt information to obtain a detection result; the post-training comprises the following steps: acquiring a first visual language model which is pre-trained, a video sample and third prompt information of an abnormal event, rewriting the third prompt information into fourth prompt information conforming to the information expression mode of the pre-training sample; and post-training the first visual language model based on the video sample and the fourth prompt information. Based on the above method, whether an abnormal event exists in the video can be effectively detected.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically providing a video anomaly event detection method, electronic device, and storage medium. Background Technology

[0002] In industrial manufacturing, to ensure safety and reliability, it is necessary to monitor and alert on potential anomalies (such as non-compliant workstations, misaligned assembly, and missing parts). With the continuous development of Vision Language Model (VLM) technology, VLMs have been widely applied in scenarios such as image-text understanding and multimodal (image-text) dialogue. Therefore, it is feasible to consider using VLMs to monitor anomalies in industrial manufacturing scenarios. However, current conventional VLMs are mainly pre-trained, general-purpose VLMs designed for common scenarios. Since the sample data used in pre-training does not cover the anomalies that occur in industrial manufacturing scenarios, it is not possible to directly use general-purpose VLMs to monitor anomalies in industrial manufacturing scenarios.

[0003] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention

[0004] This application aims to solve the above-mentioned technical problems, namely, to solve or at least partially solve the following technical problems: how to accurately monitor abnormal events in industrial production and manufacturing scenarios using a large visual language model.

[0005] In a first aspect, this application provides a video anomaly event detection method, comprising:

[0006] Obtain a visual language model that has undergone pre-training and post-training sequentially;

[0007] Obtain the first prompt information of the abnormal event, and rewrite the first prompt information into a second prompt information that conforms to the information expression mode of the pre-trained sample;

[0008] The visual language model is controlled to perform abnormal event detection on the video based on the second prompt information to obtain a detection result, and the detection result is used to indicate whether there is an abnormal event described by the second prompt information in the video;

[0009] The post-training of the visual language model includes:

[0010] Obtain the third prompt information of the pre-trained first visual language model, video samples, and abnormal events, and rewrite the third prompt information into a fourth prompt information that conforms to the information expression mode of the pre-trained samples;

[0011] Based on the video samples and the fourth prompt information, the first visual language model is post-trained for video anomaly event detection to obtain the final visual language model.

[0012] In one technical solution of the above-mentioned video anomaly detection method, there are multiple first visual language models, and the post-training of the first visual language model for video anomaly detection based on the video samples and the fourth prompt information includes:

[0013] Each first visual language model is controlled to perform abnormal event detection on the video sample according to the fourth prompt information to obtain an initial detection result. The initial detection result is used to indicate whether there is an abnormal event described by the fourth prompt information in the video sample.

[0014] The final detection result is determined based on all initial detection results, and the category of the video sample is determined based on the final detection result, the category including positive samples with abnormal events and negative samples without abnormal events;

[0015] Each first visual language model is compared and learned based on the positive and negative samples respectively;

[0016] The knowledge distillation method is used to distill all the first visual language models that have completed contrastive learning into a second visual language model, and the second visual language model is used as the final visual language model.

[0017] In one technical solution of the above-mentioned video abnormal event detection method, the method further includes rewriting the first prompt information or the third prompt information in the following manner:

[0018] Based on a template of a preset information expression method, the descriptive information of the physical state of the abnormal event is obtained according to the first target information;

[0019] The second target information is obtained based on the description information; wherein, when the first target information is the first prompt information, the second target information is the second prompt information, and when the first target information is the second prompt information, the second target information is the fourth prompt information.

[0020] In one technical solution of the above-mentioned video abnormal event detection method, obtaining the second target information based on the description information includes:

[0021] By replacing the words in the description information with synonyms, multiple different candidate information can be obtained;

[0022] The second target information is selected from the multiple different candidate information and the one with the highest semantic similarity to the video sample.

[0023] In one technical solution of the above-mentioned video abnormal event detection method, the template of the preset information expression mode includes at least one of the following:

[0024] Component-level expression templates are used to provide descriptions of different physical states of abnormal events;

[0025] Event-level expression templates are used to describe the logical relationships between different physical states.

[0026] In one technical solution of the above-mentioned video abnormal event detection method, the initial detection result further includes event description information of the abnormal event, and the step of determining the final detection result based on all initial detection results includes:

[0027] A consistency evaluation is performed on the event description information of all initial detection results indicating the presence of the anomalous event in the video sample to obtain a first score value;

[0028] A second score value is obtained by evaluating the consistency of the event description information of all initial detection results indicating that the abnormal event does not exist in the video sample.

[0029] Based on the initial detection results of all first-visual language models, obtain the model divergence degree of all first-visual language models;

[0030] Based on the blurriness of the video samples and the divergence of the model, the uncertainty in determining the final detection result based on all initial detection results is obtained;

[0031] Based on the ratio of the first score to the uncertainty, a first confidence level is obtained that the abnormal event exists in the video sample;

[0032] Based on the ratio of the second score to the uncertainty, a second confidence level is obtained that the abnormal event does not exist in the video sample;

[0033] The final detection result is determined based on the first confidence level and the second confidence level.

[0034] In one technical solution of the above-mentioned video anomaly detection method, determining the final detection result based on the first confidence level and the second confidence level includes:

[0035] The judgment result corresponding to the larger of the first confidence level and the second confidence level is taken as the final detection result, and the larger confidence level is taken as the confidence level of the final detection result; wherein, the judgment result corresponding to the first confidence level is that the abnormal event exists in the video sample, and the judgment result corresponding to the second confidence level is that the abnormal event does not exist in the video sample.

[0036] In one technical solution of the above-mentioned video anomaly detection method, when the final detection result indicates that the anomaly exists in the video sample, the method further includes adjusting the confidence level of the final detection result in the following manner:

[0037] Based on the video samples, obtain early warning clues before the abnormal event occurs, and obtain consequence information after the abnormal event occurs;

[0038] If both the precursor clues and the consequences match the abnormal event, the confidence level of the final detection result is increased; otherwise, the confidence level of the final detection result is decreased.

[0039] In one technical solution of the above-mentioned video anomaly event detection method, the method further includes performing a consistency evaluation on the textual description information of all initial detection results, whether or not the anomaly event exists, through the following method:

[0040] Semantic consistency evaluation is performed on the event description information in the target detection results to obtain a text consistency score. The target detection results are all initial detection results that indicate the presence or absence of the abnormal event.

[0041] The structural consistency of the event description information in the target detection results is evaluated to obtain a structural consistency score.

[0042] The text consistency score and the structural consistency score are weighted and summed to obtain a score value.

[0043] In one technical solution of the above video abnormal event detection method, the fourth prompt information is obtained by rewriting a preset information expression template. The preset information expression template includes a component-level expression template and an event-level expression template. The component-level expression template is used to provide a description of different physical states of abnormal event presentation, and the event-level expression template is used to provide a description of the logical relationship between different physical states.

[0044] The structural consistency assessment of the event description information in the target detection results includes:

[0045] The consistency of the event description information in the target detection results with the description method provided by the component-level expression template is evaluated to obtain a first consistency score;

[0046] The consistency of the event description information in the target detection results with the description method provided by the event-level expression template is evaluated to obtain a second consistency score;

[0047] The first consistency score and the second consistency score are weighted and summed to obtain the structural consistency score.

[0048] In a second aspect, an electronic device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described in any of the technical solutions provided in the first aspect.

[0049] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and executed by a processor to perform the method described in any of the technical solutions provided in the first aspect above.

[0050] The above-described technical solutions of this application have at least one or more of the following beneficial effects:

[0051] In one technical solution of the video anomaly detection method provided in this application, the method can acquire a visual language model that has been pre-trained and post-trained sequentially; acquire first prompt information of anomalies, and rewrite the first prompt information into second prompt information that conforms to the information expression mode of the pre-training samples; control the visual language model to perform anomaly detection on the video based on the second prompt information to obtain a detection result, which is used to indicate whether there is an anomaly described by the second prompt information in the video. The post-training of the visual language model includes: acquiring the pre-trained first visual language model, video samples, and third prompt information of anomalies; rewriting the third prompt information into fourth prompt information that conforms to the information expression mode of the pre-training samples; and performing post-training of the first visual language model for video anomaly detection based on the video samples and the fourth prompt information to obtain the final visual language model.

[0052] In the above implementation scheme, the first visual language model can be understood as a pre-trained, general-purpose visual language model for common scenarios. The training samples used during the pre-training of the first visual language model do not cover the data of the scenario corresponding to the abnormal event to be detected (such as an industrial manufacturing scenario); that is, the data domain of the training samples is different from the data domain of the scenario corresponding to the abnormal event to be detected. By rewriting the third prompt information, it is equivalent to transforming the third prompt information from its original data domain to the data domain of the pre-training samples. Based on this, when post-training the first visual language model, it can accurately understand the fourth prompt information and detect whether the abnormal event described by the fourth prompt information exists in the video sample. Furthermore, when using the post-trained visual language model for abnormal event detection, the prompt information can still be rewritten to conform to the information expression mode of the pre-training samples, and the rewritten prompt information can be used to accurately detect whether an abnormal event exists in the video.

[0053] In another technical solution for implementing the video anomaly detection method provided in this application, there are multiple first visual language models. The first visual language models can be post-trained for video anomaly detection in the following manner: each first visual language model is controlled to perform anomaly detection on video samples based on the fourth prompt information to obtain initial detection results. The initial detection results are used to indicate whether the video samples contain the anomaly described by the fourth prompt information; a final detection result is determined based on all initial detection results, and the category of the video samples is determined based on the final detection results. The categories include positive samples with anomalies and negative samples without anomalies; each first visual language model is compared and learned based on the positive and negative samples; a knowledge distillation method is used to distill all the first visual language models that have completed the comparative learning into a second visual language model, and the second visual language model is used as the final visual language model.

[0054] Based on the above implementation scheme, the final detection result can be determined using all initial detection results obtained from all first visual language models, further improving the accuracy of anomaly detection. Furthermore, based on the final detection results, video samples are classified into positive and negative samples. These positive and negative sample classifications are then used to compare and learn from each first visual language model, further enhancing their anomaly detection capabilities. Additionally, distilling all the first visual language models that have completed comparative learning into a second visual language model allows a single second visual language model to integrate the advantages of all first visual language models, resulting in a more comprehensive visual language model. This model can then effectively detect the presence of anomalies in the video. Attached Figure Description

[0055] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:

[0056] Figure 1 This is a schematic flowchart of the main steps of a video abnormal event detection method in one embodiment of this application;

[0057] Figure 2 This is a schematic diagram of the main steps in obtaining the final visual language model by post-training the first visual language model in one embodiment of this application.

[0058] Figure 3 This is a schematic diagram of the main steps for determining the final detection result based on all initial detection results in one embodiment of this application;

[0059] Figure 4 This is a flowchart illustrating a model post-training method in one embodiment of this application;

[0060] Figure 5 This is a schematic diagram of the main structure of an electronic device in one embodiment of this application. Detailed Implementation

[0061] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0062] First, embodiments of the video anomaly event detection method provided in this application will be described.

[0063] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a video anomaly event detection method in one embodiment of this application. Figure 1 As shown, the video abnormal event detection method in this application embodiment may include the following steps S101 to S103.

[0064] Step S101: Obtain the visual language model that has completed pre-training and post-training in sequence.

[0065] Specifically, a pre-trained first visual language model can be obtained, and then the first visual language model can be post-trained to obtain the final visual language model. This application does not specifically limit the type of the first visual language model in its embodiments.

[0066] In this embodiment of the application, the first visual language model can be post-trained in the following way: First, the third prompt information of video samples and abnormal events can be obtained, the third prompt information can be rewritten into a fourth prompt information that conforms to the information expression mode of the pre-trained samples, and then the first visual language model can be post-trained for video abnormal event detection based on the video samples and the fourth prompt information to obtain the final visual language model.

[0067] The pre-training samples are the training samples used during the pre-training of the visual language model. These training samples do not cover the data of the scenarios corresponding to the abnormal events to be detected in this application; that is, the data domain of the training samples differs from the data domain of the scenarios corresponding to the abnormal events to be detected. For example, the scenario corresponding to the abnormal event to be detected could be an industrial manufacturing scenario. Abnormal events occurring in industrial manufacturing scenarios include, but are not limited to, assembly misalignment / reverse assembly / omission, insufficient tightening, welding defects, and abnormal glue application. Taking the abnormal event of assembly misalignment as an example, the prompt information could be the text "assembly misalignment".

[0068] Taking the abnormal event "screw not tightened" as an example, its third prompt message is "screw not tightened". Since the training samples used during pre-training do not cover industrial production and manufacturing scenarios, the visual language model cannot accurately understand "screw not tightened", and therefore cannot accurately detect whether the abnormal event "screw not tightened" exists in the video sample. In this embodiment of the application, "screw not tightened" can be rewritten into a fourth prompt message that conforms to the information expression mode of the pre-training samples. The fourth prompt message can include at least one of the following: (1) the relative poses between the hand, tool and component are not aligned; (2) the rotation time and number of turns of the screwdriver / electric screwdriver are insufficient; (3) the axis of the tool and the screw are not collinear; (4) the gap of the connecting parts does not converge / micro-displace after tightening; (5) the acoustic / torque waveform is abnormal.

[0069] By rewriting the third prompt, it's equivalent to transferring the third prompt from its original data domain to the data domain of the pre-trained samples. Based on this, the first visual language model can accurately understand the fourth prompt and thus detect whether the abnormal event described by the fourth prompt exists in the video sample.

[0070] Step S102: Obtain the first prompt information of the abnormal event, and rewrite the first prompt information into a second prompt information that conforms to the information expression mode of the pre-trained sample.

[0071] The method for rewriting the first prompt message is the same as the method for rewriting the third prompt message in step S101 above, and will not be repeated here.

[0072] Step S103: Control the perceived language model to perform abnormal event detection on the video based on the second prompt information to obtain the detection result. The detection result is used to indicate whether there is an abnormal event described in the second prompt information in the video.

[0073] Based on the methods described in steps S101 to S103 above, a pre-trained, general-purpose visual language model for common scenarios can be trained to accurately detect abnormal events in videos. Furthermore, the post-trained visual language model, combined with rewritten prompts, can accurately detect the presence of abnormal events in a video.

[0074] The following describes an embodiment of the video abnormal event detection method provided in this application, specifically the method for rewriting the first and third prompt information in the foregoing embodiments.

[0075] In some embodiments of this application, the first target information (i.e., the first prompt information or the third prompt information) can be rewritten through the following steps 11 to 12:

[0076] Step 11: Based on the template of the preset information expression method, obtain the description information of the physical state of the abnormal event based on the first target information.

[0077] The preset information expression templates may include at least one of component-level expression templates and event-level expression templates. The following is an explanation of these two templates.

[0078] (1) Explanation of component-level expression template.

[0079] Component-level expression templates are used to provide descriptions of different physical states of abnormal events. The descriptions provided by component-level expression templates can describe the physical states of abnormal events from at least one dimension: object posture, object interaction state, and environment state.

[0080] For example, based on the component-level expression template, the following physical state description information can be obtained: (1) the relative poses between hand-tool-part are not aligned; (2) the positioning pin is not in the slot or the fixture is not closed; (3) the local strong light of the solder joint is accompanied by mechanical splashing; (4) the glue track is interrupted / width fluctuates; (5) abnormal gap or shaking at the interface; (6) the torque gun indicator light is not lit / the buzzer alarm sounds.

[0081] (2) Explanation of the event-level expression template.

[0082] Event-level expression templates are used to describe the logical relationships between different physical states. The descriptions provided by event-level expression templates can combine at least one dimension of logic or sequence to describe the logical relationships between different physical states.

[0083] For example, the event-level expression template can obtain the following physical state description information: (1) Fixture not locked → forced pressing → gap appears at the interface; (2) Gun head contacts screw → insufficient rotation angle → torque light not lit; (3) Unstable weld pool → pinhole / cold solder appears after cooling; (4) Adhesive valve stops → adhesive breaks in trajectory → air tightness test fails. Taking description information (1) as an example, this description information includes three physical states (i.e., combinational logic) of fixture not locked, forced pressing, and gap appears at the interface. The order of these states is fixture not locked, forced pressing, and gap appears at the interface.

[0084] In one example, based on the template of the above-mentioned preset information expression method, the following table 1 can be obtained: description information of three abnormal events: "assembly misalignment / not in place", "insufficient tightening / missing tightening", and "welding defects (cold weld / incomplete penetration)".

[0085] Table 1

[0086] Abnormal events Descriptive information obtained from templates based on information representation methods Misalignment / Not in place Misalignment of relative positions between hand, tool, and component; misalignment of key / positioning pin with slot; short insertion force or repeated attempts; failure to trigger clamp / position indicator; abnormal gap or wobbling at interface; the description may also omit the mention of abnormal gap or wobbling at interface. Insufficient tightening / missing tightening The tool head rotates for an insufficient duration / angle after contacting the screw; the tool is tilted; the washer is not compressed or the marking line is not displaced; the torque gun's green light is not lit / the buzzer alarm sounds. The description information may also omit the mention of the torque gun's green light not being lit / the buzzer alarm sounding. Welding defects (incomplete weld / lack penetration) The arc is unstable and flickering, the weld brightness / texture is discontinuous, the weld has obvious holes / slag inclusions / splashes, and the color banding is abnormal after cooling; however, the description information may not include the abnormal color banding after cooling.

[0087] Step 12: Obtain the second target information based on the description information.

[0088] Specifically, this descriptive information can be used as the second target information. Wherein, when the first target information is the first prompt information, the second target information is the second prompt information; when the first target information is the second prompt information, the second target information is the fourth prompt information.

[0089] In some embodiments of this application, the second target information can also be obtained based on the description information through the following steps 121 to 122:

[0090] Step 121: Replace the words in the description information with synonyms to obtain multiple different candidate information. Specifically, synonym clusters for different words can be generated in advance, and then the synonym clusters containing the words in the description information can be obtained, and synonyms can be obtained from the synonym clusters for replacement.

[0091] In one example, four synonym clusters as shown in Table 2 can be pre-generated.

[0092] Table 2

[0093] Synonym Cluster 1 Not in place, not aligned, not locked Synonym Cluster 2 Under-tightening, loose tightening, insufficient torque Synonym Cluster 3 Incomplete soldering, incomplete penetration, cold soldering Synonym Cluster 4 Glue breakage, glue leakage, air bubbles

[0094] Step 122: Select the candidate information with the highest semantic similarity to the video sample from multiple different candidate information sources as the second target information. The candidate information is text information, and the video sample can be understood as image information. In this embodiment, a conventional image-text similarity acquisition method can be used to obtain the semantic similarity between each candidate information source and the video sample, and then the one with the highest semantic similarity is selected as the second target information.

[0095] Based on the method described in steps 121 to 122 above, the descriptive information obtained in step 11 can be expanded, and the most accurate one can be selected from multiple expanded candidate information by using the semantic similarity between the information and the video sample. This is beneficial to improving the accuracy of the visual language model in detecting abnormal events based on the second target information.

[0096] The following description continues with an embodiment of the video abnormal event detection method provided in this application, specifically focusing on the post-training of the first visual language model in step S101.

[0097] In some embodiments of this application, multiple first visual language models are pre-trained, and can be... Figure 2 The following steps S201 to S204 are shown to post-train the first visual language model to obtain the final visual language model.

[0098] Step S201: Control each first visual language model to perform abnormal event detection on the video sample according to the fourth prompt information to obtain the initial detection result. The initial detection result is used to indicate whether there is an abnormal event described in the fourth prompt information in the video sample.

[0099] Step S202: Determine the final detection result based on all initial detection results, and determine the category of the video sample based on the final detection result. The category includes positive samples and negative samples, where positive and negative samples represent the presence and absence of the abnormal event described in the second prompt information, respectively.

[0100] In this embodiment, the first visual language model performs abnormal event detection on video samples. In addition to obtaining the initial detection results, it can also obtain the confidence level (i.e., the degree of reliability) of the initial detection results. When determining the final detection result based on all the initial detection results, the final detection result can be determined based on the confidence level of each initial detection result.

[0101] For example, we can obtain the average first confidence score of all first detection results (i.e., initial detection results indicating the presence of an anomaly), and the average second confidence score of all second detection results (i.e., initial detection results indicating the absence of an anomaly). We can then compare the first and second confidence score averages and select the detection result corresponding to the larger average confidence score average as the final detection result. If the first and second confidence score averages are equal, manual verification can be performed to determine the final detection result. Alternatively, we can select the result with the highest confidence score from all initial detection results as the final detection result. If two initial detection results with contradictory conclusions both have the highest confidence scores, manual verification can be performed to determine the final detection result.

[0102] Step S203: Perform contrastive learning on each first visual language model based on positive and negative samples. The contrastive learning of each first visual language model is independent of each other. In this embodiment, conventional contrastive learning methods can be used to perform contrastive learning on the first visual language models using positive and negative samples. After contrastive learning, the ability of the first visual language models to detect abnormal events in videos can be further improved.

[0103] Step S204: Use knowledge distillation to distill all the first visual language models that have completed comparative learning into a second visual language model, and use the second visual language model as the final visual language model. Compared to the first visual language model, the second visual language model can be a lightweight model, which allows it to be deployed on edge devices (such as industrial control computers) within the scene described by the video to be detected (such as an industrial manufacturing scene). Users in this scene can conveniently check for abnormal events through the edge devices.

[0104] Based on the methods described in steps S201 to S204 above, the final detection result can be determined using all initial detection results obtained from all first visual language models, further improving the accuracy of anomaly detection. Furthermore, by classifying video samples into positive and negative samples according to the final detection result, and using these classifications to compare and learn from each first visual language model, the anomaly detection capability of each first visual language model can be further improved. In addition, distilling all the first visual language models that have completed comparative learning into a second visual language model allows a single second visual language model to integrate the advantages of all first visual language models, resulting in a more comprehensive visual language model. This allows for effective detection of anomalies in the video when using this visual language model for anomaly detection.

[0105] The following describes an embodiment of the video abnormal event detection method provided in this application, specifically the method for determining the final detection result based on all initial detection results in step S202.

[0106] In some embodiments of this application, the initial detection result may further include event description information of the abnormal event, which can be obtained through... Figure 3 The following steps S2021 to S2027, as shown, determine the final detection result based on all initial detection results.

[0107] Step S2021: Perform a consistency evaluation on the event description information of all initial detection results representing the presence of abnormal events in the video sample to obtain the first score value.

[0108] Step S2022: Perform a consistency evaluation on the event description information of all initial detection results indicating that there are no abnormal events in the video sample to obtain a second score value.

[0109] Step S2023: Based on the initial detection results of all first visual language models, obtain the model divergence degree of all first visual language models.

[0110] Model divergence can be understood as the degree of inconsistency between the initial detection results obtained by different first visual language models using the same video samples and fourth cue information. The higher the model divergence, the more inconsistent the initial detection results obtained by these first visual language models are, and vice versa.

[0111] As described in step S202 above, the first visual language model can obtain the confidence level of the initial detection result by detecting abnormal events in the video samples. The initial detection result can be either the presence or absence of abnormal events. Based on this, the prediction confidence level of each first visual language model in predicting the presence of abnormal events can be obtained according to the initial detection results and their confidence levels. Then, the average confidence level of the prediction confidence levels of all first visual language models is obtained, and this average confidence level is used as the model divergence degree.

[0112] Taking a first-person visual language model as an example, when obtaining the prediction confidence score, if the initial detection result of this model is that an anomalous event exists, then the confidence score of the initial detection result is used as the prediction confidence score for the existence of an anomalous event; if the initial detection result of this model is that no anomalous event exists, then the prediction confidence score for the existence of an anomalous event can be calculated based on the confidence score of the initial detection result. For example, if the confidence score is a value between 0 and 1, the difference between the value 1 and the confidence score of the initial detection result can be obtained, and this difference can be used as the prediction confidence score for the existence of an anomalous event.

[0113] Step S2024: Based on the blurriness of the video samples and the model divergence, obtain the uncertainty of determining the final detection result based on all initial detection results.

[0114] Specifically, the higher the blurriness of the video sample and the higher the model divergence, the higher the uncertainty of the final detection result, and vice versa.

[0115] When obtaining uncertainty, the blurriness and model divergence of the video samples can be normalized first, so that the values ​​of blurriness and model divergence are both between 0 and 1. Then, the normalized blurriness and model divergence are weighted and summed, and the result is used as the uncertainty of the final detection result.

[0116] Step S2025: Obtain the first confidence level of the existence of an abnormal event in the video sample based on the ratio of the first score to the uncertainty. Specifically, this ratio is used as the first confidence level.

[0117] Step S2026: Obtain the second confidence level that no abnormal events exist in the video sample based on the ratio of the second score to the uncertainty. Specifically, this ratio is used as the second confidence level.

[0118] Step S2027: Determine the final detection result based on the first confidence level and the second confidence level.

[0119] The higher the confidence level, the more credible the judgment result (existence or non-existence of an abnormal event) is. Therefore, the first and second confidence levels can be compared, and the judgment result (existence or non-existence of an abnormal event) with the higher confidence level can be selected as the final detection result. Alternatively, the larger of the first and second confidence levels can be used as the confidence level of the final detection result.

[0120] In some embodiments of this application, if the final detection result indicates the presence of an abnormal event, the confidence level of the final detection result can be adjusted through the following steps 21 to 22.

[0121] Step 21: Obtain warning signs before the abnormal event occurs based on the video samples, and obtain information about the consequences after the abnormal event occurs.

[0122] Precursor information can be understood as information presented in video samples before an abnormal event occurs, while consequence information can be understood as information presented in video samples after an abnormal event occurs.

[0123] Step 22: Match the abnormal events with the warning clues and consequences information respectively.

[0124] If the precursor clues are the cause of the anomalous event, then the precursor clues match the anomalous event; if the consequence information is the result of the anomalous event, then the consequence information matches the anomalous event. When both the precursor clues and the consequence information match the anomalous event, it indicates a high degree of confidence in the presence of an anomalous event in the video sample, thus increasing the confidence of the final detection result; otherwise, the confidence of the final detection result can be decreased.

[0125] In addition, after reducing the confidence level of the final test result, a prompt message can be output to allow for manual verification to determine whether the final test result is incorrect.

[0126] In one example, the precursor clues and consequences information that match the three abnormal events of assembly misalignment, insufficient tightening, and incomplete soldering / incomplete penetration can be shown in Table 3 below.

[0127] Table 3

[0128] Abnormal events Early warning signs and clues Consequences Information Assembly misalignment Detection of not being in place / not locked Interface gap out of tolerance Insufficient tightening Insufficient gun head angle Torque light not lit / beep Incomplete solder joint / incomplete penetration Unstable weld pool Air tightness / appearance not up to standard

[0129] Based on the methods described in steps 21 to 22 above, causal inference can be performed on abnormal events to further determine whether there are abnormal events in the video samples. Then, the confidence level of the final detection result can be adjusted according to the judgment result to ensure the accuracy of the final detection result.

[0130] In some embodiments of this application, before determining the category of the video sample based on the final detection result in the aforementioned step S202, the confidence level of the final detection result can be compared with a set threshold. If the confidence level is greater than the set threshold, it indicates that the accuracy of the final detection result is very high, and the category of the video sample can be determined based on the final detection result; otherwise, the category of the video sample is not determined based on the final detection result. This application does not specifically limit the value of the set threshold in its embodiments.

[0131] Based on the method described in steps S2021 to S2027 above, the initial detection results obtained from all the first visual language models can be used to obtain the most accurate final detection result, which is beneficial to improving the category accuracy of the video samples obtained from the final detection result.

[0132] The following describes an embodiment of the video abnormal event detection method provided in this application, specifically the method for performing consistency evaluation on the aforementioned steps S2021 and S2022.

[0133] In the embodiments of this application, the consistency of event description information of all initial detection results with or without abnormal events can be evaluated through the following steps 31 to 33.

[0134] Step 31: Perform semantic consistency evaluation on the event description information in the target detection results to obtain a text consistency score; where the target detection results are all initial detection results with or without abnormal events.

[0135] Step 32: Perform a structural consistency assessment on the event description information in the target detection results to obtain a structural consistency score.

[0136] In some embodiments of this application, the fourth cue information used during the post-training of the visual language model is rewritten based on the templates of the preset information expression methods (including component-level and event-level expression templates) in the aforementioned embodiments. Based on this, the structural consistency of the event description information in the object detection results can be evaluated through the following steps 321 to 323:

[0137] Step 321: Evaluate the consistency of the text description information in the target detection results with the description method provided by the component-level expression template to obtain a first consistency score. Step 322: Evaluate the consistency of the text description information in the target detection results with the description method provided by the event-level expression template to obtain a second consistency score. Step 323: Calculate the weighted sum of the first and second consistency scores to obtain a structural consistency score. Based on this embodiment, the consistency of the information structure of event description information can be comprehensively evaluated from the description methods provided by the component-level and event-level expression templates, which helps improve the accuracy of the structural consistency score.

[0138] Step 33: Calculate a weighted sum of the text consistency score and the structural consistency score to obtain a total score. When the target detection result includes all initial detection results containing anomalies, the score is the aforementioned first score; when the target detection result includes all initial detection results without anomalies, the score is the aforementioned second score.

[0139] Based on steps 31 to 33 above, the consistency of the event description information of the initial detection results can be comprehensively evaluated from the two dimensions of semantic consistency and structural consistency, which is conducive to improving the accuracy of consistency evaluation.

[0140] The following is in conjunction with the appendix Figure 4 The post-training method of the model provided in the embodiments of this application will be described.

[0141] like Figure 4As shown, in this embodiment, N pre-trained first visual language models are used. First, video samples and first prompts for abnormal events are acquired. These first prompts are then input into a semantic transfer processor to rewrite the information and obtain second prompts. The semantic transfer processor can implement the methods described in steps 11 to 12 of the aforementioned embodiments. Then, video samples and second prompts are input into each of the first visual language models to detect abnormal events and obtain initial detection results. Based on all initial detection results, a final detection result is determined. Finally, the confidence level of this final detection result is determined through causal inference and result confidence analysis. The methods for determining the final detection result based on all initial detection results and the methods for causal inference and result confidence analysis are the same as the methods in steps S2021 to S2027 of the aforementioned embodiments, and will not be repeated here. Figure 4 As shown, through causal inference and result confidence analysis, the final detection result was determined to be "unlocked screws and misaligned assembly" in the video sample, with a confidence level of 80%. After obtaining the final detection result, the positive and negative categories of the video sample were determined based on the final detection result.

[0142] By repeating the above steps, positive and negative categories of multiple video samples can be obtained. Then, the first visual language model is compared and learned based on the positive and negative samples. The knowledge distillation method is used to distill all the first visual language models that have completed the comparative learning into the second visual language model, and the second visual language model is used as the final visual language model.

[0143] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0144] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0145] Another aspect of this application provides a computer-readable storage medium.

[0146] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium may be configured to store a program for performing the video anomaly detection method of the above-described method embodiments. This program may be loaded and run by a processor to implement the above-described method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium may be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0147] Another aspect of this application provides an electronic device.

[0148] In one embodiment of an electronic device according to this application, the electronic device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the methods described in any of the above embodiments. See Appendix Figure 5 , Figure 5 The example illustrates a memory and processor connected via a bus communication connection.

[0149] The electronic devices described in this application may include, but are not limited to, mobile phones, tablets, desktop computers, laptops, handheld computers, notebook computers, in-vehicle devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc., and the embodiments of this application do not limit them.

[0150] In the description of this application, "processor" can include hardware, software, or a combination of both. A processor can be a central processing unit, microprocessor, graphics processor, digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as a magnetic disk, hard disk, optical disk, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B.

[0151] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A method for detecting abnormal events in video, characterized in that, The method includes: Obtain a visual language model that has undergone pre-training and post-training sequentially; Obtain the first prompt information of the abnormal event, and rewrite the first prompt information into a second prompt information that conforms to the information expression mode of the pre-trained sample; The visual language model is controlled to perform abnormal event detection on the video based on the second prompt information to obtain a detection result, and the detection result is used to indicate whether there is an abnormal event described by the second prompt information in the video; The post-training of the visual language model includes: Obtain the third prompt information of the pre-trained first visual language model, video samples, and abnormal events, and rewrite the third prompt information into a fourth prompt information that conforms to the information expression mode of the pre-trained samples; Based on the video samples and the fourth prompt information, the first visual language model is post-trained for video anomaly event detection to obtain the final visual language model.

2. The method according to claim 1, characterized in that, The first visual language model comprises multiple models. The post-training of the first visual language model based on the video samples and the fourth cue information for video anomaly event detection includes: Each first visual language model is controlled to perform abnormal event detection on the video sample according to the fourth prompt information to obtain an initial detection result. The initial detection result is used to indicate whether there is an abnormal event described by the fourth prompt information in the video sample. The final detection result is determined based on all initial detection results, and the category of the video sample is determined based on the final detection result, the category including positive samples with abnormal events and negative samples without abnormal events; Each first visual language model is compared and learned based on the positive and negative samples respectively; The knowledge distillation method is used to distill all the first visual language models that have completed contrastive learning into a second visual language model, and the second visual language model is used as the final visual language model.

3. The method according to claim 1, characterized in that, The method further includes rewriting the first prompt message or the third prompt message in the following ways: Based on a template of a preset information expression method, the descriptive information of the physical state of the abnormal event is obtained according to the first target information; The second target information is obtained based on the description information; wherein, when the first target information is the first prompt information, the second target information is the second prompt information, and when the first target information is the second prompt information, the second target information is the fourth prompt information.

4. The method according to claim 3, characterized in that, The step of obtaining the second target information based on the description information includes: By replacing the words in the description information with synonyms, multiple different candidate information can be obtained; The second target information is selected from the multiple different candidate information and the one with the highest semantic similarity to the video sample.

5. The method according to claim 3, characterized in that, The template for the preset information expression method includes at least one of the following: Component-level expression templates are used to provide descriptions of different physical states of abnormal events; Event-level expression templates are used to describe the logical relationships between different physical states.

6. The method according to claim 2, characterized in that, The initial detection results also include event description information of the abnormal events. Determining the final detection result based on all initial detection results includes: A consistency evaluation is performed on the event description information of all initial detection results indicating the presence of the anomalous event in the video sample to obtain a first score value; A second score value is obtained by evaluating the consistency of the event description information of all initial detection results indicating that the abnormal event does not exist in the video sample. Based on the initial detection results of all first-visual language models, obtain the model divergence degree of all first-visual language models; Based on the blurriness of the video samples and the divergence of the model, the uncertainty in determining the final detection result based on all initial detection results is obtained; Based on the ratio of the first score to the uncertainty, a first confidence level is obtained that the abnormal event exists in the video sample; Based on the ratio of the second score to the uncertainty, a second confidence level is obtained that the abnormal event does not exist in the video sample; The final detection result is determined based on the first confidence level and the second confidence level.

7. The method according to claim 6, characterized in that, Determining the final detection result based on the first confidence level and the second confidence level includes: The judgment result corresponding to the larger of the first confidence level and the second confidence level is taken as the final detection result, and the larger confidence level is taken as the confidence level of the final detection result; wherein, the judgment result corresponding to the first confidence level is that the abnormal event exists in the video sample, and the judgment result corresponding to the second confidence level is that the abnormal event does not exist in the video sample.

8. The method according to claim 7, characterized in that, When the final detection result indicates that the abnormal event exists in the video sample, the method further includes adjusting the confidence level of the final detection result in the following manner: Based on the video samples, obtain early warning clues before the abnormal event occurs, and obtain consequence information after the abnormal event occurs; If both the precursor clues and the consequences match the abnormal event, the confidence level of the final detection result is increased; otherwise, the confidence level of the final detection result is decreased.

9. The method according to claim 6, characterized in that, The method further includes a consistency assessment of the textual description information of all initial detection results, whether or not the anomalous event exists, by means of: Semantic consistency evaluation is performed on the event description information in the target detection results to obtain a text consistency score. The target detection results are all initial detection results that indicate the presence or absence of the abnormal event. The structural consistency of the event description information in the target detection results is evaluated to obtain a structural consistency score. The text consistency score and the structural consistency score are weighted and summed to obtain a score value.

10. The method according to claim 9, characterized in that, The fourth prompt information is obtained by rewriting a preset information expression template. The preset information expression template includes a component-level expression template and an event-level expression template. The component-level expression template is used to provide a description of different physical states of abnormal events, and the event-level expression template is used to provide a description of the logical relationship between different physical states. The structural consistency assessment of the event description information in the target detection results includes: The consistency of the event description information in the target detection results with the description method provided by the component-level expression template is evaluated to obtain a first consistency score; The consistency of the event description information in the target detection results with the description method provided by the event-level expression template is evaluated to obtain a second consistency score; The first consistency score and the second consistency score are weighted and summed to obtain the structural consistency score.

11. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the video anomaly detection method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the video anomaly event detection method according to any one of claims 1 to 10.

Images