Target video driven abnormality recognition method, electronic device and storage medium

Abstract

The present invention provides a target video-driven anomaly recognition method, electronic device and storage medium, relating to the field of anomaly recognition technology. The method inputs the feature vector of the target video, the feature vector list corresponding to the target device parameter list set, and the total set of related data sets corresponding to the target device parameter list set obtained by recalling the target video and the target device parameter list set in the device database into a preset large prediction model to obtain anomaly recognition results, and comprehensively utilizes multiple data sources. Among them, the device database is constructed based on the feature vector of the specified device parameter list group, the feature vector of the historical video corresponding to the specified device parameter list group, the status label corresponding to the historical video and the abnormal parameter name list corresponding to the historical video. It contains rich domain knowledge, can provide necessary data support for the anomaly recognition task, and is conducive to improving the accuracy of the obtained anomaly recognition results.

Description

Technical Field

[0001] The present invention relates to the technical field of anomaly recognition, and in particular to a target video driven anomaly recognition method, electronic equipment and storage medium. Background Art

[0002] In recent years, with the development of natural language processing technology, large language models have been widely used in various fields. They are not limited to text processing. They are also combined with other technologies such as computer vision models to form multimodal systems. Multimodal systems can process various types of data (such as images, videos, text, vectors, etc.) and combine these data through multimodal fusion technology to perform more complex recognition tasks, such as anomaly recognition tasks. Multimodal systems combine the capabilities of large language models and computer vision models, and can comprehensively analyze multimodal data and output anomaly recognition results.

[0003] In the prior art, an image or video corresponding to an area is input into a multimodal system, and an abnormality recognition result output by the multimodal system is obtained. The user determines whether there is an abnormality in the area corresponding to the image or video based on the abnormality recognition result. The abnormality includes but is not limited to safety hazards, operational errors, etc.

[0004] However, the above method also has the following technical problems:

[0005] Although multimodal systems can provide more information than a single modality, they may still lack sufficient domain knowledge or contextual understanding in some cases, which can affect the accuracy of judgments. For example, in a chemical production plant, specific operating procedures and safety regulations may not be easily fully understood and applied by general models. Therefore, today's multimodal systems can only simply determine whether there are anomalies in the area based on images or videos to generate anomaly recognition results, resulting in low accuracy. Summary of the Invention

[0006] In view of the above technical problems, the technical solution adopted by the present invention is:

[0007] According to a first aspect of the present invention, a method for identifying anomalies driven by a target video is provided, the method comprising the following steps:

[0008] S001. For each designated device, obtain a number of historical videos corresponding to the designated device and a designated device parameter list group corresponding to each historical video.

[0009] S002. Clustering the feature vectors of all designated device parameter list groups corresponding to the designated device to obtain several intermediate clusters corresponding to the designated device, and using the vector corresponding to the center position of the intermediate cluster as the intermediate center vector corresponding to the designated device.

[0010] S003. If the characteristic vector of the specified device parameter list group is in the intermediate cluster corresponding to the intermediate center vector, the characteristic vector of the historical video corresponding to the specified device parameter list group is used as the key vector corresponding to the intermediate center vector, the status label corresponding to the historical video is used as the status label corresponding to the key vector, and the abnormal parameter name list corresponding to the historical video is used as the abnormal parameter name list corresponding to the key vector.

[0011] S004. Store the intermediate center vector corresponding to the designated device, the key vector corresponding to the intermediate center vector, the state label corresponding to the key vector, and the abnormal parameter name list corresponding to the key vector in a device database.

[0012] S005. Input the feature vector of SP, the feature vector list corresponding to MB, and the total set of related data sets corresponding to MB into the preset large language model to obtain the abnormality recognition result corresponding to SP output by the preset large language model, wherein the total set of related data sets corresponding to MB is obtained by recalling SP and MB in the device database, SP is the target video, and MB is the target device parameter list set corresponding to SP.

[0013] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the aforementioned method.

[0014] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned method when executing the computer program.

[0015] The present invention has at least the following beneficial effects:

[0016] The present invention provides a target video driven anomaly recognition method, electronic device and storage medium, the method can obtain a number of historical videos corresponding to a specified device and a specified device parameter list group corresponding to each historical video, build a device database based on the feature vector of the specified device parameter list group, the feature vector of the historical video corresponding to the specified device parameter list group, the status label corresponding to the historical video and the abnormal parameter name list corresponding to the historical video, input the feature vector of the target video, the feature vector list corresponding to the target device parameter list set and the total set of related data sets corresponding to the target device parameter list set obtained by recalling the target video and the target device parameter list set in the device database into a preset large prediction model to obtain anomaly recognition results. It can be seen that in the present invention, the device database is based on the target device parameter list set. The method is constructed by the feature vector of a specified device parameter list group, the feature vector of the historical video corresponding to the specified device parameter list group, the status label corresponding to the historical video and the list of abnormal parameter names corresponding to the historical video. It contains rich domain knowledge. The total set of related data sets corresponding to the target device parameter list set is a set recalled from the device database, which can provide necessary data support for the abnormality recognition task. The feature vector of the target video, the feature vector list corresponding to the target device parameter list set and the total set of related data sets corresponding to the target device parameter list set are input into the preset large prediction model to obtain the abnormality recognition result. It comprehensively utilizes multiple data sources, rather than simply judging whether there is an abnormality in the area based on the image or video to generate the abnormality recognition result, which is conducive to improving the accuracy of the obtained abnormality recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 The present invention provides a flowchart of a method for identifying anomalies driven by a target video. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar tasks and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] An embodiment of the present invention provides a method for identifying anomalies driven by a target video, the method comprising the following steps: Figure 1 As shown:

[0022] S001. For each designated device, several historical videos corresponding to the designated device and a designated device parameter list group corresponding to each historical video are obtained. The designated device parameter list group includes a designated device parameter list corresponding to each parameter of the designated device. The designated device parameter list includes specific parameter values ​​of the parameters within each second of the historical video. Each historical video corresponds to a status label and a list of abnormal parameter names.

[0023] Specifically, the designated device is a device located in the target area.

[0024] Specifically, the historical video corresponding to the designated device is a video collected before the current time point that can present the designated device.

[0025] Specifically, the duration of the historical video is the same as the duration of the target video.

[0026] Specifically, the status label includes a normal label and an abnormal label.

[0027] Specifically, when the status label corresponding to the historical video is a normal label, it means that the sub-video corresponding to the specified device in the historical video is consistent with the standard video corresponding to the specified device parameter list group, and the sub-video corresponding to the specified device is a partial video in the historical video that only presents the specified device.

[0028] Specifically, when the status tag corresponding to the historical video is an abnormal tag, it means that the sub-video corresponding to the specified device in the historical video is inconsistent with the standard video corresponding to the specified device parameter list group, and there may be a fault or improper operation.

[0029] Specifically, when the status label corresponding to the historical video is a normal label, the abnormal parameter name list corresponding to the historical video is NULL.

[0030] Specifically, when the status label corresponding to the historical video is an abnormal label, the abnormal parameter name list corresponding to the historical video includes several abnormal parameter names, and the abnormal parameter names are names of parameters that cause the historical video to be abnormal.

[0031] S002. Clustering the feature vectors of all designated device parameter list groups corresponding to the designated device to obtain several intermediate clusters corresponding to the designated device, and using the vector corresponding to the center position of the intermediate cluster as the intermediate center vector corresponding to the designated device, wherein, based on the vector similarity between the feature vectors of the designated device parameter list groups corresponding to the designated device, clustering the feature vectors of all designated device parameter list groups corresponding to the designated device to obtain several intermediate clusters corresponding to the designated device and using the vector corresponding to the center position of the intermediate cluster as the intermediate center vector corresponding to the designated device, and the intermediate cluster corresponding to the designated device includes the feature vectors of several designated device parameter list groups corresponding to the designated device; it can be understood as: using the vector similarity between the feature vectors of the designated device parameter list groups corresponding to the designated device as the distance metric in the preset clustering algorithm, and using the preset clustering algorithm to cluster the feature vectors of all designated device parameter list groups corresponding to the designated device to obtain several intermediate clusters corresponding to the designated device.

[0032] Specifically, the vector corresponding to the center position of the middle cluster is the average value of all eigenvectors in the middle cluster.

[0033] Specifically, the designated device parameter list group corresponding to the designated device can be understood as the designated device parameter list group corresponding to the historical video corresponding to the designated device.

[0034] Specifically, the feature vector of the specified device parameter list group is a vector obtained by splicing the feature vectors of each specified device parameter list according to the order of the specified device parameter list in the specified device parameter list group, wherein the vector values ​​in the feature vector of the specified device parameter list correspond one-to-one to the parameter values ​​in the specified device parameter list.

[0035] S003. If the feature vector of the specified device parameter list group is in the intermediate cluster corresponding to the intermediate center vector, the feature vector of the historical video corresponding to the specified device parameter list group is used as the key vector corresponding to the intermediate center vector, the status label corresponding to the historical video is used as the status label corresponding to the key vector, and the abnormal parameter name list corresponding to the historical video is used as the abnormal parameter name list corresponding to the key vector.

[0036] S004. Store the intermediate center vector corresponding to the designated device, the key vector corresponding to the intermediate center vector, the state label corresponding to the key vector, and the abnormal parameter name list corresponding to the key vector in a device database.

[0037] Through the above steps, based on the vector similarity between the feature vectors of the specified device parameter list groups corresponding to the specified device, the feature vectors of all the specified device parameter list groups corresponding to the specified device are clustered to obtain several intermediate clusters corresponding to the specified device and the vector corresponding to the center position of the intermediate cluster is used as the intermediate center vector corresponding to the specified device. If the feature vector of the specified device parameter list group is in the intermediate cluster corresponding to the intermediate center vector, the feature vector of the historical video corresponding to the specified device parameter list group is used as the key vector corresponding to the intermediate center vector, the status label corresponding to the historical video is used as the status label corresponding to the key vector, and the abnormal parameter name list corresponding to the historical video is used as the abnormal parameter name list corresponding to the key vector. The intermediate center vector corresponding to the specified device, the key vector corresponding to the intermediate center vector, the status label corresponding to the key vector, and the abnormal parameter name list corresponding to the key vector are stored in the device database. Clustering can reduce redundant information and save storage space. The status label corresponding to the key vector and the abnormal parameter name list corresponding to the key vector are stored in the device database, so that the device database contains rich and detailed domain knowledge, which can provide data support for the abnormality recognition task and is conducive to improving the accuracy of the obtained abnormality recognition results.

[0038] S005. Input the feature vector of SP, the feature vector list corresponding to MB, and the total set of related data sets corresponding to MB into the preset large language model to obtain the abnormality recognition result corresponding to SP output by the preset large language model, wherein the total set of related data sets corresponding to MB is obtained by recalling SP and MB in the device database, SP is the target video, and MB is the target device parameter list set corresponding to SP. Those skilled in the art know that any method of obtaining the feature vector of a video in the prior art falls within the scope of protection of the present invention and will not be repeated here.

[0039] Optionally, the device database is initially NULL.

[0040] Specifically, before step S005, the process further includes step S1:

[0041] S1. Whenever the preset duration is reached, the video captured by the target video capture device within the preset duration is used as SP and MB is obtained. The MB includes a target device parameter list group corresponding to several target devices. The target device parameter list group includes a target device parameter list corresponding to each parameter of the target device. The target device parameter list includes specific parameter values ​​of the parameters within each second of the SP. The preset duration is a duration pre-set by those skilled in the art according to actual needs, for example: 1 minute, 2 minutes, which will not be repeated here.

[0042] Specifically, the parameters of a device refer to the specific performance indicators of the device during operation, such as power, speed, accuracy, temperature, etc.

[0043] Specifically, the target video acquisition device is a video acquisition device set in a target area. In a specific embodiment, the target area may be a chemical production workshop.

[0044] Furthermore, several video acquisition devices are set up in the target area.

[0045] Specifically, the target device is a device located in the video acquisition area of ​​the target video acquisition device. In a specific embodiment, the device located in the video acquisition area of ​​the target video acquisition device can be chemical production equipment (for example: reactors, material transmission devices, polymerization kettles, distillation towers, extraction towers) and detection equipment (for example: gas concentration detection equipment, temperature detection equipment, pressure detection equipment, noise detection equipment, particulate matter detection equipment).

[0046] Specifically, each target device is equipped with at least one sensor for collecting parameters of the target device in real time. For example, a reactor is equipped with a temperature sensor, a pressure sensor, and a liquid level sensor.

[0047] Specifically, the target video acquisition device is equipped with a data receiving port, and the data receiving port is used to receive parameters transmitted from the target device and its configured sensors.

[0048] In a specific embodiment, step S1 includes: whenever a specific time point is reached, taking the video captured by the target video capture device between the specific time point and a previous specific time point as the SP and obtaining the MB.

[0049] Specifically, the feature vector list corresponding to the MB includes the feature vector of each target device parameter list group in the MB.

[0050] Specifically, the feature vector of the target device parameter list group is a vector obtained by splicing the feature vectors of each target device parameter list according to the order of the target parameter lists in the target device parameter list group, wherein the vector values ​​in the feature vectors of the target device parameter list correspond one-to-one to the parameter values ​​in the target device parameter list.

[0051] Specifically, the total set of related data sets corresponding to the MB includes the related data sets corresponding to each target device parameter list group in the MB, and the related data sets corresponding to the target device parameter list group include: a related video feature vector list corresponding to the target device parameter list group, a related device feature vector list corresponding to the target device parameter list group, and a status label list corresponding to the related video feature list, wherein the related video feature vector list includes several related video feature vectors, the related device feature vector list includes several related device feature vectors, and the status label list includes a status label corresponding to each related video feature vector.

[0052] Specifically, the abnormality identification result corresponding to the SP includes an abnormality judgment label, which includes no abnormality and abnormality. When the abnormality judgment label is abnormality, the abnormality identification result also includes an abnormal device identifier.

[0053] Furthermore, when the abnormality judgment label in the abnormality identification result corresponding to the SP is "no abnormality", it means that all target devices corresponding to the SP have no abnormality.

[0054] Furthermore, when the abnormality judgment label in the abnormality identification result corresponding to the SP is abnormality exists, it means that the target device corresponding to the abnormal device identifier has an abnormality, for example: the temperature of the target device is too high and there is a safety hazard.

[0055] In a specific embodiment, the preset large language model is a general large language model, such as GPT and BERT.

[0056] In a specific embodiment, the preset large language model is a special model obtained by those skilled in the art after optimizing and adjusting the general large language model through technical means such as fine-tuning for the abnormality recognition task.

[0057] Through the above steps, the feature vector of the target video, the feature vector list corresponding to the target device parameter list set, and the total set of related data sets corresponding to the target device parameter list set are input into the preset large prediction model to obtain the anomaly recognition result. It comprehensively utilizes multiple data sources, rather than simply judging whether there is an anomaly in the area based on the image or video to generate the anomaly recognition result, which is conducive to improving the accuracy of the obtained anomaly recognition result.

[0058] Specifically, in step S005, the process of recalling the SP and MB in the device database to obtain a total set of related data sets corresponding to the MB is performed, and obtaining the related data sets corresponding to the target device parameter list group according to the SP, the target device corresponding to the target device parameter list group, and the device database includes the following steps S0051-S0055:

[0059] S0051. Obtain a list of intermediate center vectors R={R1, R2, ..., R} corresponding to a designated device that is the same device as the target device. g ,……,R h} , R g is the gth intermediate center vector corresponding to the designated device that is the same device as the target device, where g ranges from 1 to h, and h is the number of intermediate center vectors corresponding to the designated device that is the same device as the target device.

[0060] S0052. Obtain B and R g The first average vector similarity K between g , where B is the feature vector of the target device parameter list group.

[0061] Specifically, step S0052 includes the following steps ac:

[0062] a. Decompose B to obtain the first sub-vector list D = {D1, D2, ..., D j ,……,D n}, D j is the first subvector corresponding to the j-th parameter of the target device in B, where j ranges from 1 to n, and n is the number of parameters of the target device. The first subvector corresponding to the j-th parameter of the target device can be understood as the eigenvector of the target device parameter list corresponding to the j-th parameter of the target device.

[0063] b. R g Decompose to obtain the intermediate subvector list Q g ={Q g1 , Q g2 ,……,Q gj ,……,Q gn}, Q gj R g The intermediate sub-vector corresponding to the jth parameter of the target device in R g The intermediate sub-vector corresponding to the j-th parameter of the target device in R can be understood as: the j-th parameter of the target device in R g The average value of all key feature vectors in the corresponding middle cluster, where the jth parameter of the target device is in R g The corresponding key feature vector in the corresponding intermediate cluster is a feature vector of the designated device parameter list corresponding to the j-th parameter of the target device in the designated device parameter list group corresponding to the key vector in the intermediate cluster.

[0064] c. According to D j and Q gj Get K g , where K g Meet the following conditions:

[0065] K g =∑ n j＝1 U gj / n,U gj D j With Q gj The vector similarity between .

[0066] Specifically, the greater the vector similarity, the more similar the two vectors are.

[0067] Through the above steps, with the parameters of the target device as the splitting dimension, the characteristic vector of the target device parameter list group is split into the first sub-vector corresponding to the parameters of the target device, and the intermediate center vector is split into the intermediate sub-vector corresponding to the parameters of the target device. Based on the vector similarity between the first sub-vector and the intermediate sub-vector corresponding to the parameters of the target device, the average vector similarity between the characteristic vector and the intermediate center vector of the target device parameter list group is obtained, which can smooth out the influence of individual abnormal parameter values.

[0068] S0053, if K g ≥C 0 , then R g As the relevant device feature vector corresponding to the target device parameter list group, R g The corresponding key vector is used as the relevant video feature vector corresponding to the target device parameter list group, and the state label corresponding to the key vector is used as the state label corresponding to the relevant video feature vector, where C 0 It is a preset vector similarity threshold. Those skilled in the art know that the preset vector similarity threshold is a value less than 1 pre-set by those skilled in the art according to actual needs, for example, 0.8, 0.9, 0.95, and will not be repeated here.

[0069] S0054, if K1<C 0 , K2<C 0 ,……,K g <C 0 ,……,K h <C 0 , then based on R g The corresponding key vectors and the corresponding exception parameter name list get B and R g The second average vector similarity L between g .

[0070] Specifically, in step S0054, based on R g The corresponding key vectors and the corresponding exception parameter name list get B and R g The second average vector similarity L between g The method includes the following steps S0061-0065:

[0071] S0061、Get R g The corresponding key vector corresponds to the abnormal parameter name list set M g ={M g1 , M g2 ,……,M gr ,……,M gs(g)}, M gr R g The list of abnormal parameter names corresponding to the rth key vector, r ranges from 1 to s(g), s(g) is R g The number of corresponding key vectors.

[0072] S0062, M g1 , M g2 ,……,M gr ,……,M gs(g) The union of R g Corresponding important parameter name list N g ={N g1 , N g2 ,……,N gk ,……,N gt(g)}, N gk R g The corresponding kth important parameter name, k ranges from 1 to t(g), t(g) is R g The number of corresponding important parameter names.

[0073] S0063, according to N gk Get the jth parameter of the target device in R g The first importance weight P in the corresponding middle cluster jg , P jg Meet the following conditions:

[0074] When the parameter name of the jth parameter of the target device is the same as N gk If the same, let P jg =1+N 0 gk , N 0 gk M g1 , M g2 ,……,M gr ,……,M gs(g) Including N gk The number of exception parameter name lists with the same exception parameter name; when the parameter name of the jth parameter of the target device is the same as N g1 , N g2 ,……,N gk ,……,N gt(g) If they are all different, let Pjg =1.

[0075] S0064, P 1g , P 2g ,……,P jg ,……,P ng Perform normalization to obtain P jg The corresponding normalized value, and P jg The corresponding normalized value is used as the jth parameter of the target device in R g The corresponding second importance weight W in the corresponding middle cluster jg .

[0076] Specifically, the larger the second importance weight is, the more important the corresponding parameter is.

[0077] S0065, according to D j 、W jg and Q gj Get L g , L g Meet the following conditions:

[0078] L g =∑ n j＝1 (W jg ×U gj ) / ∑ n j＝1 W jg .

[0079] Through the above steps, a set of abnormal parameter name lists corresponding to the key vectors corresponding to the intermediate central vector corresponding to the designated device that is the same device as the target device is obtained, and the union of all abnormal parameter name lists in the abnormal parameter name list set is used as the important parameter name list corresponding to the intermediate central vector. According to the parameter names, important parameter names and the number of abnormal parameter name lists containing important parameter names of the parameters of the target device, the first importance weights corresponding to the parameters of the target device in the intermediate cluster are obtained, and the first importance weights corresponding to all the parameters of the target device in the intermediate cluster are normalized to obtain the second importance weights corresponding to the parameters of the target device in the intermediate cluster, eliminating the dimensional difference in weights between different parameters. Based on the second importance weights corresponding to the parameters of the target device in the intermediate cluster and the vector similarity between the first sub-vector corresponding to the parameters of the target device and the intermediate sub-vector, the second average vector similarity between the vector features of the target device parameter list group and the intermediate central vector is obtained, taking into account the importance of the parameters themselves, so that the data set is correlated according to the second average vector similarity, which is conducive to improving the accuracy of the obtained related data set.

[0080] S0055, if L g ≥C0 , then R g As the relevant device feature vector corresponding to the target device parameter list group, R g The corresponding key vector is used as the relevant video feature vector corresponding to the target device parameter list group. If L1<C 0 , L2<C 0 ,……,L g <C 0 ,……,L h <C 0 , then L1, L2, ..., L g ,……,L h The intermediate center vector corresponding to the largest second average vector similarity is used as the relevant device feature vector corresponding to the target device parameter list group, and the key vector corresponding to the intermediate center vector is used as the relevant video feature vector corresponding to the target device parameter list group, wherein the state label corresponding to the key vector is used as the state label corresponding to its corresponding relevant video feature vector.

[0081] Through the above steps, the intermediate center vector corresponding to the designated device that is the same device as the target device is obtained, and the first average vector similarity between the feature vector of the target device parameter list group and the intermediate center vector is obtained. When the first average vector similarity is not less than the preset vector similarity, it means that the feature vector of the target device parameter list group is very similar to the intermediate center vector. It can be understood that the target device parameter list group and the designated device parameter list group corresponding to the feature vector in the intermediate cluster corresponding to the intermediate center vector are also very similar. Therefore, the intermediate center vector is used as the relevant device feature vector corresponding to the target device parameter list group, the key vector corresponding to the intermediate center vector is used as the relevant video feature vector corresponding to the target device parameter list group, and the status label corresponding to the key vector is used as the status label corresponding to its corresponding relevant video feature vector, which can provide necessary data support for the abnormality recognition task; if the first average vector similarity between the feature vector of the target device parameter list group and any intermediate center vector is less than the preset vector similarity threshold, then the second average vector similarity between the feature vector of the target device parameter list group and the intermediate center vector is obtained. When the second average vector similarity is not less than the preset vector similarity, it means that the feature vector of the target device parameter list group is very similar to the intermediate center vector. It can be understood that the target device parameter list group and the designated device parameter list group corresponding to the feature vector in the intermediate cluster corresponding to the intermediate center vector are also very similar. Therefore, the intermediate center vector is used as the relevant device feature vector corresponding to the target device parameter list group, the key vector corresponding to the intermediate center vector is used as the relevant video feature vector corresponding to the target device parameter list group, and the status label corresponding to the key vector is used as the status label corresponding to its corresponding relevant video feature vector; it can provide necessary data support for the abnormality recognition task; otherwise, the intermediate center vector corresponding to the largest second average vector similarity is used as the relevant device feature vector corresponding to the target device parameter list group, the key vector corresponding to the intermediate center vector is used as the relevant video feature vector corresponding to the target device parameter list group, and the status label corresponding to the key vector is used as the status label corresponding to its corresponding relevant video feature vector; it can provide necessary data support for the abnormality recognition task; it is beneficial to improve the accuracy of the obtained abnormality recognition results.

[0082] The present invention also provides a specific embodiment, in which the relevant data sets corresponding to the target device parameter list group include: a relevant video feature vector list corresponding to the target device parameter list group, a relevant device feature vector list corresponding to the target device parameter list group, and a status label list corresponding to the relevant device feature vector list, wherein the relevant video feature vector list includes several relevant video feature vectors, the relevant device feature vector list includes several relevant device feature vectors, and the status label list includes a status label corresponding to each relevant device feature vector.

[0083] Specifically, when the status label corresponding to the relevant device feature vector is a normal label, it indicates that there is no abnormality in the relevant device feature vector.

[0084] Specifically, when the status label corresponding to the relevant device feature vector is an abnormal label, it indicates that the relevant device feature vector is abnormal, for example, the vector value in the relevant device feature vector is too large.

[0085] Before step S005, the following steps S01-S04 are included to build a device database:

[0086] S01. Clustering the feature vectors of all historical videos corresponding to the specified device to obtain several first center vectors corresponding to the specified device, wherein, based on the vector similarity between the feature vectors of the historical videos corresponding to the specified device, clustering the feature vectors of all historical videos corresponding to the specified device to obtain several first clusters corresponding to the specified device and using the vector corresponding to the center position of the first cluster as the first center vector corresponding to the specified device, the first cluster corresponding to the specified device includes the feature vectors of several historical videos corresponding to the specified device; it can be understood as: using the vector similarity between the feature vectors of the historical videos corresponding to the specified device as the distance metric in the preset clustering algorithm, and using the preset clustering algorithm to cluster the feature vectors of all historical videos corresponding to the specified device to obtain several first clusters corresponding to the specified device. Those skilled in the art know that the preset clustering algorithm is a clustering algorithm that does not require the pre-specified number of clusters and can use vector similarity as a distance metric, such as: hierarchical clustering method. Any method of obtaining vector similarity between vectors in the prior art, such as: all fall within the scope of protection of the present invention, such as: cosine similarity, Euclidean distance, which will not be repeated here.

[0087] Specifically, the vector corresponding to the center position of the first cluster is the average value of all eigenvectors in the first cluster.

[0088] Specifically, before step S01, the following steps are also included: for each designated device, several historical videos corresponding to the designated device and a designated device parameter list group corresponding to each historical video are obtained, the designated device parameter list group includes a designated device parameter list corresponding to each parameter of the designated device, and the designated device parameter list includes specific parameter values ​​of the parameters within each second of the historical video, wherein each designated device parameter list group corresponds to a status label.

[0089] Specifically, when the status label corresponding to the specified device parameter list group is an abnormal label, it means that the specified device parameter list group is inconsistent with the standard parameter list group corresponding to the specified device in the historical video corresponding to the specified device parameter list group, and there may be a fault or improper operation.

[0090] Specifically, when the status label corresponding to the designated device parameter list group is a normal label, it indicates that the designated device parameter list group is consistent with the standard parameter list group corresponding to the designated device in the historical video corresponding to the designated device parameter list group.

[0091] S02. If the feature vector of the historical video is in the first cluster corresponding to the first central vector, the feature vector of the specified device parameter list group corresponding to the historical video is used as the intermediate vector corresponding to the first central vector, and the status label corresponding to the specified device parameter list group is used as the status label corresponding to the intermediate vector.

[0092] S03, clustering all intermediate vectors corresponding to the first central vector, obtaining the second central vector corresponding to the first central vector and the state label corresponding to the second central vector, including the following steps S031-S032:

[0093] S031. Based on the vector similarity between the intermediate vectors corresponding to the first central vector, clustering all the intermediate vectors corresponding to the first central vector to obtain several second clusters corresponding to the first central vector, wherein the second cluster corresponding to the first central vector includes several intermediate vectors corresponding to the first central vector; this can be understood as: using the vector similarity between the intermediate vectors corresponding to the first central vector as the distance metric in the preset clustering algorithm, and using the preset clustering algorithm to cluster all the intermediate vectors corresponding to the first central vector to obtain several second clusters corresponding to the first central vector.

[0094] S032. If the state labels corresponding to the intermediate vectors in the second cluster corresponding to the first center vector are not completely consistent, the second cluster is divided into two sub-clusters according to the state labels, and the second cluster is deleted and the two sub-clusters are used as two new second clusters corresponding to the first center vector; if the state labels corresponding to the intermediate vectors in the second cluster corresponding to the first center vector are completely consistent, the vector corresponding to the center position of the second cluster is used as the second center vector corresponding to the first center vector, and the state label corresponding to any intermediate vector in the second cluster is used as the state label corresponding to the second center vector.

[0095] Specifically, the second cluster is divided into two subclusters according to the state labels. One subcluster includes all intermediate vectors in the second cluster whose state labels are normal labels, and the other subcluster includes all intermediate vectors in the second cluster whose state labels are abnormal labels.

[0096] Specifically, the vector corresponding to the center position of the second cluster is the average value of all intermediate vectors in the second cluster.

[0097] Through the above steps, when the state labels corresponding to the intermediate vectors in the second cluster corresponding to the first central vector are not completely consistent, only one state label cannot represent the state labels corresponding to all the intermediate vectors in the second cluster. Therefore, the second cluster is divided into two sub-clusters according to the state labels, and the second cluster is deleted and the two sub-clusters are used as two new second clusters corresponding to the first central vector. The state labels corresponding to all the intermediate vectors in the new second clusters are the same. Only one state label can be used to represent the state labels corresponding to all the intermediate vectors in the new second clusters, which is convenient for storage. There is no need to store the state labels corresponding to all the intermediate vectors in the second cluster in the device database. This can effectively reduce redundant information, reduce the amount of data storage, save storage space, make the device database more concise and efficient, and facilitate subsequent query and use.

[0098] S04: Store the first central vector corresponding to the designated device, the second central vector corresponding to the first central vector, and the state label corresponding to the second central vector in a device database.

[0099] Through the above steps, the feature vectors of all historical videos corresponding to the specified device are first clustered according to the vector similarity between the feature vectors of the historical videos corresponding to the specified device, and the feature vectors of similar historical videos are clustered into the same first cluster, and the first cluster is represented by the first central vector. The feature vectors of the specified device parameter list group corresponding to the historical videos corresponding to the feature vectors in the first cluster are used as the intermediate vectors corresponding to the first central vector of the first cluster, and the status labels corresponding to the specified device parameter list group are used as the status labels corresponding to the intermediate vectors thereof. Then, based on the vector similarity between the intermediate vectors corresponding to the first central vector, similar intermediate vectors are clustered into the same second cluster to obtain the second central vector corresponding to the first central vector and the status labels corresponding to the second central vector. The first central vector corresponding to the specified device, the second central vector corresponding to the first central vector, and the status labels corresponding to the second central vector are stored in the device database, so that the device database contains rich domain knowledge and can provide data support for the anomaly recognition task. In addition, through multi-level (first cluster, second cluster) cluster analysis, redundant information can be effectively reduced, the amount of data storage can be reduced, storage space can be saved, and the device database can be made more concise and efficient, facilitating subsequent query and use.

[0100] In step S005, the process of recalling the SP and MB in the device database to obtain a total set of related data sets corresponding to the MB is performed, and the process of obtaining the related data sets corresponding to the target device parameter list group according to the SP, the target device parameter list group in the MB, the target device corresponding to the target device parameter list group, and the device database includes the following steps S051-S054:

[0101] S051, obtain a third center vector list A corresponding to a designated device that is the same device as the target device = {A1, A2, ..., A i ,……,A m}, A i is the i-th third center vector corresponding to the specified device that is the same device as the target device, where i ranges from 1 to m, m is the number of third center vectors corresponding to the specified device that is the same device as the target device, and the third center vector corresponding to the specified device is the second center vector corresponding to the first center vector corresponding to the specified device.

[0102] S052. Get B and A i The average vector similarity C between i .

[0103] Specifically, step S052 includes the following steps S0521-S0523:

[0104] S0521. Decompose B to obtain the first sub-vector list D={D1, D2, ..., D j ,……,D n}.

[0105] S0522, A i Decompose to obtain the second sub-vector list E i ={E i1 , E i2 ,……,E ij ,……,E in}, E ij A i The second subvector corresponding to the jth parameter of the target device in A i The second sub-vector corresponding to the j-th parameter of the target device in A can be understood as: the j-th parameter of the target device in A i The average value of all the corresponding specified feature vectors in the corresponding second cluster, where the jth parameter of the target device is in A i The corresponding designated feature vector in the corresponding second cluster is a feature vector of the designated device parameter list corresponding to the j-th parameter of the target device in the designated device parameter list group corresponding to the intermediate vector in the second cluster.

[0106] S0523, according to D j and E ij Get C i , where C i Meet the following conditions:

[0107] C i =∑ n j＝1 F ij / n,Fij D j With E ij The vector similarity between .

[0108] Through the above steps, with the parameters of the target device as the splitting dimension, the characteristic vector of the target device parameter list group is split into a first sub-vector corresponding to the parameters of the target device, and the third central vector is split into a second sub-vector corresponding to the parameters of the target device. Based on the vector similarity between the first sub-vector and the second sub-vector corresponding to the parameters of the target device, the average vector similarity between the characteristic vector of the target device parameter list group and the third central vector is obtained, which can smooth out the influence of individual abnormal parameter values.

[0109] S053, if C i ≥C 0 , then A i As the relevant device feature vector corresponding to the target device parameter list group, A i The corresponding state label is A i The state label corresponding to the corresponding related device feature vector is A i The corresponding first center vector is used as the relevant video feature vector corresponding to the target device parameter list group.

[0110] Through the above steps, the average vector similarity between the feature vector of the target device parameter list group and the third center vector corresponding to the designated device which is the same device as the target device is obtained. If the average similarity between the feature vector of the target device parameter list group and the third center vector is not less than the preset vector similarity threshold, it means that the feature vector of the target device parameter list group is very similar to the third center vector. It can be understood that the target device parameter list group and the designated device parameter list group corresponding to the feature vector in the second cluster corresponding to the third center vector are also similar. Therefore, the third center vector is used as the relevant device feature vector corresponding to the target device parameter list group, the status label corresponding to the third center vector is used as the status label corresponding to the relevant device feature vector, and the first center vector corresponding to the third center vector is used as the relevant video feature vector corresponding to the target device parameter list group. This can provide necessary data support for the abnormality recognition task, which is conducive to improving the accuracy of the obtained abnormality recognition results.

[0111] S054、If C1<C 0 , C2<C 0 ,……,C i <C 0 ,……,C m <C 0 , then obtaining the relevant data set corresponding to the target device parameter list group according to SP, including the following steps S0541-S0544:

[0112] S0541. Obtain a target sub-video corresponding to the target device from the SP, wherein the target sub-video is a portion of the video in the SP that only presents the target device. Those skilled in the art know that any method of obtaining a sub-video from a video in the prior art falls within the scope of protection of the present invention and will not be described in detail here.

[0113] S0542. Obtain a first center vector list G corresponding to a designated device that is the same device as the target device = {G1, G2, ..., G e ,……,G f}, where G e is the e-th first center vector corresponding to the designated device that is the same device as the target device, where the value of e ranges from 1 to f, and f is the number of first center vectors corresponding to the designated device that is the same device as the target device.

[0114] S0543, obtain the feature vector of the target sub-video and G e The vector similarity H between e .

[0115] S0544, if H e ≥C 0 , then G e As the relevant video feature vector corresponding to the target device parameter list group, G e The corresponding second center vector is used as the relevant device feature vector corresponding to the target device parameter list group; if H1<C 0 , H2<C 0 ,……,H e <C 0 ,……,H f <C 0 , then H1, H2, ..., H e ,……,H f The first center vector corresponding to the largest vector similarity is used as the relevant video feature vector corresponding to the target device parameter list group, and the second center vector corresponding to the first center vector is used as the relevant device feature vector corresponding to the target device parameter list group, wherein the state label corresponding to the second center vector is used as the state label corresponding to its corresponding relevant device feature vector.

[0116] Through the above steps, if the average similarity between the feature vector of the target device parameter list group and any third central vector is less than the preset vector similarity threshold, it means that the feature vector of the target device parameter list group is not so similar to these third central vectors, and the data related to the target device parameter list group cannot be obtained based on the third central vector. At this time, the target sub-video corresponding to the target device is obtained from the target video, and the vector similarity between the feature vector of the target sub-video and the first central vector corresponding to the designated device that is the same device as the target device is obtained. When the vector similarity is not less than the preset vector similarity threshold, it means that the feature vector of the target sub-video is very similar to the first central vector. It can be understood that the historical video corresponding to the feature vector in the first cluster corresponding to the target sub-video and the first central vector is also similar. Therefore, the first central vector is used as the relevant video feature vector corresponding to the target device parameter list group, the second central vector corresponding to the first central vector is used as the relevant device feature vector corresponding to the target device parameter list group, and the status label corresponding to the second central vector is used as the status label corresponding to its corresponding relevant device feature vector; it can provide necessary data support for the abnormality recognition task. If the vector similarity between the feature vector of the target sub-video and the first center vector corresponding to the designated device that is the same device as the target device is less than the preset vector similarity threshold, it means that the feature vector of the target sub-video is not so similar to these first center vectors. In this case, compared to being unable to provide necessary data support for the abnormality recognition task, taking the first center vector corresponding to the largest vector similarity threshold as the relevant video feature vector corresponding to the target device parameter list group, taking the second center vector corresponding to the first center vector as the relevant device feature vector corresponding to the target device parameter list group, and taking the status label corresponding to the second center vector as the status label corresponding to its corresponding relevant device feature vector is the best choice. It can provide necessary data support for the abnormality recognition task and is conducive to improving the accuracy of the obtained abnormality recognition results.

[0117] The present invention also provides a specific embodiment, wherein step S005 includes: inputting SP, MB and the associated data set corresponding to MB into a preset large language model to obtain the abnormality recognition result corresponding to SP output by the preset large language model, wherein the associated data set corresponding to MB includes an associated data subset corresponding to each target device parameter list group in MB, and the associated data subset corresponding to the target device parameter list group is a data set recalled from the device database based on SP and the target device parameter list group, including: an associated video set corresponding to the target device parameter list group, an associated parameter list group set corresponding to the target device parameter list group and a status label list corresponding to the associated parameter list group set, wherein the associated video set includes several associated videos, the associated parameter list group set includes several associated parameter list groups, and the status label list includes a status label corresponding to each associated parameter list group.

[0118] After step S04, the following step S05 is also included to build a device database:

[0119] S05. Store the historical videos corresponding to each designated device and the designated device parameter list group corresponding to the historical videos in the device database, and establish an association relationship between the first center vector and the historical videos corresponding to the feature vector in the first cluster corresponding to the first center vector, and establish an association relationship between the second center vector and the designated device parameter list group corresponding to the intermediate vector in the second cluster corresponding to the second center vector.

[0120] Step S053 includes: if C i ≥C 0 , then it will be combined with A i The designated device parameter list group with an associated relationship is used as the associated parameter list group corresponding to the target device parameter list group, and the status tag corresponding to the designated device parameter list group is used as the status tag corresponding to the associated parameter list group. i The historical videos having an associated relationship with the corresponding first central vector are used as associated videos corresponding to the target device parameter list group.

[0121] Step S0544 includes: if H e ≥C 0 , then it will be combined with G e The historical videos with related relationships will be used as the related videos corresponding to the target device parameter list group and will be used with G e The designated device parameter list group with an associated relationship corresponding to the second center vector is used as the associated parameter list group corresponding to the target device parameter list group; if H1<C 0 , H2<C 0 ,……,H e <C 0 ,……,H f <C0 , then it will be combined with H1, H2, ..., H e ,……,H f The historical video associated with the first center vector corresponding to the largest vector similarity is used as the associated video corresponding to the target device parameter list group, and the specified device parameter list group associated with the second center vector corresponding to the first center vector is used as the associated parameter list group corresponding to the target device parameter list group, wherein the status label corresponding to the specified device parameter list group is used as the status label corresponding to its corresponding associated parameter list group.

[0122] Through the above steps, the historical videos corresponding to the designated device and the designated device parameter list groups corresponding to the historical videos are also stored in the device database, and an association relationship is established between the first central vector and the historical videos corresponding to the feature vector in the first cluster corresponding to the first central vector, and an association relationship is established between the second central vector and the designated device parameter list groups corresponding to the intermediate vector in the second cluster corresponding to the second central vector, thereby enriching the content of the device database. According to the target video and the target device parameter list group, the data set recalled in the device database includes: an associated video set corresponding to the target device parameter list group, an associated parameter list group set corresponding to the target device parameter list group, and a status label list corresponding to the associated parameter list group set, wherein the associated video set includes several associated videos, the associated parameter list group set includes several associated parameter list groups, and the status label list includes a status label corresponding to each associated parameter list group, which can provide more comprehensive data support for the abnormality recognition task and is conducive to improving the accuracy of the obtained abnormality recognition results.

[0123] The present invention also provides a specific embodiment, in which the associated data subset corresponding to the target device parameter list group is a data set recalled from the device database based on the SP and the target device parameter list group, including: an associated video set corresponding to the target device parameter list group, an associated parameter list group set corresponding to the target device parameter list group, and a status label list corresponding to the associated video set, wherein the associated video set includes several associated videos, the associated parameter list group set includes several associated parameter list groups, and the status label list includes a status label corresponding to each associated video.

[0124] After step S004, the following step S0041 is also included to build a device database:

[0125] S0041. Store the historical videos corresponding to each designated device and the designated device parameter list group corresponding to the historical videos in the device database, and establish an association relationship between the intermediate center vector and the designated device parameter list group corresponding to the feature vector in the intermediate cluster corresponding to the intermediate center vector, and establish an association relationship between the key vector and the historical video corresponding to the key vector.

[0126] Step S0053 includes: if K g ≥C 0 , then it will be combined with R g The specified device parameter list group with an associated relationship is used as the associated parameter list group corresponding to the target device parameter list group, and will be used with R g The historical video with the associated relationship of the corresponding key vector is used as the associated video corresponding to the target device parameter list group, and the status label corresponding to the historical video is used as the status label corresponding to the corresponding associated video.

[0127] Step S0055 includes: if L g ≥C 0 , then it will be combined with R g The specified device parameter list group with an associated relationship is used as the associated parameter list group corresponding to the target device parameter list group, and will be used with R g The historical video with the corresponding key vector is used as the associated video corresponding to the target device parameter list group; if L1<C 0 , L2<C 0 ,……,L g <C 0 ,……,L h <C 0 , then it will be combined with L1, L2, ..., L g ,……,L h The designated device parameter list group that is associated with the intermediate center vector corresponding to the largest second average vector similarity is used as the associated parameter list group corresponding to the target device parameter list group, and the historical video that is associated with the key vector corresponding to the intermediate center vector is used as the associated video corresponding to the target device parameter list group, wherein the status label corresponding to the historical video is used as the status label corresponding to its corresponding associated video.

[0128] Through the above steps, the historical videos corresponding to each designated device and the designated device parameter list group corresponding to the historical videos are stored in the device database, and an association relationship is established between the intermediate center vector and the designated device parameter list group corresponding to the feature vector in the intermediate cluster corresponding to the intermediate center vector, and an association relationship is established between the key vector and the historical video corresponding to the key vector, thereby enriching the content of the device database. According to the target video and the target device parameter list group, the data set recalled in the device database includes: an associated video set corresponding to the target device parameter list group, an associated parameter list group set corresponding to the target device parameter list group, and a status label list corresponding to the associated video set, wherein the associated video set includes several associated videos, the associated parameter list group set includes several associated parameter list groups, and the status label list includes a status label corresponding to each associated video, which can provide more comprehensive data support for the abnormality recognition task and is conducive to improving the accuracy of the obtained abnormality recognition results.

[0129] The present invention also provides a specific embodiment, which includes steps S11-S12 after step S1:

[0130] S11. According to the preset text corresponding to the SP and the target video acquisition device, search the text database for a recall text set corresponding to the SP, where the recall text set includes several recall texts. Those skilled in the art know that any method of searching for recall information in a database in the prior art falls within the scope of protection of the present invention and will not be described in detail here.

[0131] Specifically, the recalled text set corresponding to the SP can be understood as a set of all texts in the text database that are related to the SP and the preset text corresponding to the target video acquisition device.

[0132] Specifically, the preset text corresponding to the target video acquisition device includes preset question prompt text, such as: "Please check whether there is any abnormality in the device in the video"; "Please identify whether each device in the video is operating normally"; "Please check whether there are any safety hazards in the device in the video."

[0133] In a specific embodiment, the preset text corresponding to the target video capture device also includes text presenting information related to the video capture area of ​​the target video capture device, for example: text presenting the range of the video capture area of ​​the target video capture device, and text presenting the device name, device attributes, device purpose, and device location within the video capture area of ​​the target video capture device.

[0134] Specifically, the text database includes relevant text for each specified device, such as: operating manual, preset risk treatment plan, safe operation steps, fault records, personnel duty roster, clothing standard manual, etc.

[0135] S12. Input the preset text corresponding to the target video acquisition device, the recalled text set corresponding to SP, the feature vector of SP, the feature vector list corresponding to MB, and the total set of related data sets corresponding to MB into the preset large language model to obtain the abnormality recognition result corresponding to SP output by the preset large language model.

[0136] In a specific embodiment, step S12 includes: inputting the preset text corresponding to the target video acquisition device, the recalled text set corresponding to the SP, the SP, the MB and the associated data set corresponding to the MB into the preset large language model to obtain the abnormality recognition result corresponding to the SP output by the preset large language model.

[0137] Through the above steps, according to the preset text corresponding to the target video and the target video acquisition device, the recall text set corresponding to the target video is searched in the text database, and the preset text corresponding to the target video acquisition device, the recall text set corresponding to the target video, the feature vector of the target video, the feature vector list corresponding to the target device parameter list set, and the total set of related data sets corresponding to the target device parameter list set are input into the preset large language model to obtain the abnormality recognition result. In combination with the preset text and the recall text set corresponding to the target video acquisition device, more domain expertise can be introduced to help the large language model better understand and apply the background information of the specific field, provide the large language model with rich context information, and help improve the accuracy of the obtained abnormality recognition results.

[0138] The present invention also provides a specific embodiment, which includes the following step S10 after step S005:

[0139] S10. Construct a historical abnormality recognition result data combination corresponding to the SP based on the abnormality recognition result corresponding to the SP, and insert the historical abnormality recognition result data combination corresponding to the SP into the historical abnormality recognition result data set corresponding to the target video acquisition device corresponding to the SP, wherein the historical abnormality recognition result data combination includes the abnormality recognition result and the initial device parameter list set corresponding to the abnormality recognition result, and the initial device parameter list set corresponding to the abnormality recognition result is the target device parameter list set used to obtain the abnormality recognition result.

[0140] Specifically, if the abnormality judgment label in the abnormality recognition result in the historical abnormality recognition result data combination is no abnormality, and the time point when the historical abnormality recognition result data combination is inserted into the historical abnormality recognition result data set is closest to the current time point, then the initial device parameter list set in the historical abnormality recognition data combination is used as the key device parameter list set corresponding to its corresponding target video acquisition device at the current time point.

[0141] Through the above steps, after obtaining the abnormality recognition result, a historical abnormality recognition result data combination is constructed based on the abnormality recognition result, and the historical abnormality recognition result data combination is inserted into the historical abnormality recognition result data set corresponding to the target video acquisition device. Furthermore, the key device parameter list set corresponding to the target video acquisition device at the current time point is obtained, which can manage and update the historical abnormality recognition result data set of the device, and dynamically adjust the key device parameter list set based on the latest abnormality-free recognition result, which is helpful for user management and query.

[0142] After step S1 and before step S005, the following steps S100-S300 are included:

[0143] S100. Obtain data similarity XS between MB and a set of key device parameter lists corresponding to the target video acquisition device at the current time point. Those skilled in the art are aware that any method for obtaining data similarity between two data sets in the prior art falls within the scope of protection of the present invention and will not be described in detail herein.

[0144] S200, when XS ≥ XS 0 When the target device parameter list group in SP and MB is obtained, the steps S051-S054 are executed to obtain the relevant data set corresponding to the target device parameter list group according to the target device parameter list group in SP and MB, the target device corresponding to the target device parameter list group and the first device database, XS 0 It is a preset data similarity threshold, where those skilled in the art know that the preset data similarity threshold is a value less than 1 pre-set by those skilled in the art according to actual needs, for example: 0.8, 0.58, 0.9, 0.95, which will not be repeated here.

[0145] Specifically, before step S1 , the method further includes: executing steps S01 - S04 to construct a first device database.

[0146] Specifically, the greater the data similarity, the more similar the key device parameter list set corresponding to the MB and the target video acquisition device at the current time point is.

[0147] S300, when XS<XS 0 When the target device parameter list group is obtained, steps S0051-S0055 are executed to obtain the relevant data set corresponding to the target device parameter list group according to the target device parameter list group in SP and MB, the target device corresponding to the target device parameter list group and the second device database.

[0148] Specifically, before step S1 , the method further includes: executing steps S001 - S004 to construct a second device database.

[0149] Through the above steps, a first device database and a second device database are constructed to obtain data similarity between the target device parameter list set and the key device parameter list set corresponding to the target video acquisition device at the current time point. When the data similarity is higher than the preset data similarity threshold, it means that the target device parameter list set and the key device parameter list set corresponding to the target video acquisition device at the current time point are very similar, and there may be no abnormality. S051-S054 are executed to obtain the relevant data set corresponding to the target device parameter list group, and the relevant data set is obtained in combination with the target device parameter list set and the target video. Otherwise, it means that the target device parameter list set and the key device parameter list set corresponding to the target video acquisition device at the current time point are not similar, and it is impossible to judge whether there is an abnormality. At this time, S0051-S0055 are executed to obtain the relevant data set corresponding to the target device parameter list group, mainly relying on the target device parameter list set to obtain the relevant data set. The method of obtaining the relevant data set can be flexibly selected, which is conducive to improving the accuracy of obtaining the relevant data set and avoiding unnecessary calculation and resource consumption.

[0150] The present invention also provides a specific embodiment, including step S1 and steps S101-S104:

[0151] S101. Input SP and MB into the preset large language model to obtain the target object data set output by the preset large language model, wherein the target object data set includes a target object data list corresponding to several target objects, and the target object data list includes the personnel type corresponding to the target object, the device type of the intermediate device corresponding to the target object, the device name of the intermediate device corresponding to the target object, the working status of the intermediate device corresponding to the target object, and the initial intention corresponding to the target object, wherein the target object is an object in SP, the intermediate device corresponding to the target object is the target device in SP with the smallest straight-line distance to the target object, and the target object can be understood as a person in SP.

[0152] Specifically, intent can be understood as the name of the operation performed on the equipment, such as: checking the equipment, opening the cover of the equipment, adding materials to the equipment, performing daily maintenance, checking the status of the equipment, maintaining and recording logs, and leaving after confirming that the equipment is fault-free.

[0153] S102. Obtain a preset intent tree corresponding to the device type of the intermediate device corresponding to the target object. The structure of the preset intent tree has 6 layers. The first-layer node represents the device type, the second-layer node represents the device name of the specified device corresponding to the device type, the third-layer node represents the working status of the specified device, the fourth-layer node represents the personnel type related to the working status of the specified device, the fifth-layer node represents the original intent corresponding to the personnel type, and the sixth-layer node represents the target intent corresponding to the original intent. Each fifth-layer node has only one child node; for example: the first-layer node is a temperature sensor, the second-layer node is sensor A, sensor B, and sensor C, the third-layer node is normal operation, failure, and waiting for repair, the fourth-layer node is equipment maintenance personnel and equipment inspection personnel, the fifth-layer node is for performing daily maintenance and checking equipment status, and the sixth-layer node is for completing maintenance, recording logs, and leaving after confirming that the equipment is fault-free.

[0154] S103: Determine the final intent of the target object according to the target object data list and the nodes in the preset intent tree corresponding to the device type of the intermediate device corresponding to the target object.

[0155] Specifically, step S103 includes the following steps S1031-S1035:

[0156] S1031. When the device name of the intermediate device corresponding to the target object is the same as the device name represented by the second-layer node in the preset intent tree, the second-layer node is used as the second-layer key node.

[0157] S1032: When the working status represented by the child node of the second-layer key node is the same as the working status of the intermediate device corresponding to the target object, use the child node as the third-layer key node.

[0158] S1033: When the personnel type represented by the child node of the third-layer key node is the same as the personnel type corresponding to the target object, the child node is used as the fourth-layer key node.

[0159] S1034. When the original intent represented by the child node of the fourth-layer key node is the same as the initial intent corresponding to the target object, the child node is used as the fifth-layer key node.

[0160] S1035. Use the target intent represented by the child node of the fifth-layer key node as the final intent of the target object corresponding to the target object data list.

[0161] Through the above steps, the data stored in the target object data list is matched one by one with the nodes in the preset intention tree corresponding to the device type of the intermediate device corresponding to the target object to determine the final intention of the target object, which is conducive to improving the accuracy of the determined final intention.

[0162] S104: Input the target object data list corresponding to the target object, the final intention of the target object, and the recall information corresponding to the target object into the preset large language model to obtain the abnormality recognition result corresponding to the target object output by the preset large language model.

[0163] In a specific embodiment, step S104 includes: inputting the SP, the target object data list corresponding to the target object, the final intention of the target object and the recall information corresponding to the target object into the preset large language model to obtain the abnormal recognition result corresponding to the target object output by the preset large language model.

[0164] Specifically, the abnormality recognition results corresponding to the target object include normal and abnormal.

[0165] Furthermore, when the abnormality identification result corresponding to the target object is normal, it indicates that there is no abnormality in the operation of the target object in the SP.

[0166] Furthermore, when the abnormality identification result corresponding to the target object is abnormal, it indicates that the operation of the target object in the SP is abnormal, which may pose a security risk.

[0167] Specifically, the recall information corresponding to the target object includes video recall information, which is a standard operation video searched from the device database based on the target object data list corresponding to the target object and the final intention of the target object. The device database includes several standard operation videos corresponding to each specified device, and each standard operation video corresponds to a real intention. When the intermediate device corresponding to the target object is the same device as the specified device, and the final intention of the target object is the same as the real intention corresponding to the standard operation video corresponding to the specified device, the standard operation video is used as video recall information.

[0168] In a specific embodiment, the recall information corresponding to the target object includes text recall information, which is a standard operation manual searched from a text database based on the target object data list corresponding to the target object and the final intention of the target object, wherein the text database includes several standard operation manuals corresponding to each specified device, and each standard operation manual corresponds to a real intention. When the intermediate device corresponding to the target object is the same device as the specified device, and the final intention of the target object is the same as the real intention corresponding to the standard operation manual corresponding to the specified device, the standard operation manual is used as text recall information.

[0169] Through the above steps, the target video and the target device parameter list set are input into the preset large language model to obtain the target object data set, and the final intention of the target object is determined according to the preset intention tree corresponding to the device type of the intermediate device corresponding to the target object. The target object data list corresponding to the target object, the final intention of the target object and the recall information corresponding to the target object are input into the preset large language model to obtain the abnormality recognition result corresponding to the target object output by the preset large language model. The recall information corresponding to the target object can provide necessary context information or data support for the abnormality recognition task, and the target object data list corresponding to the target object, the final intention of the target object and the recall information corresponding to the target object are input into the preset large language model to obtain the abnormality recognition result. The comprehensive utilization of multiple data sources does not simply determine whether the target object has an abnormality based on the image or video to generate the abnormality recognition result, which is conducive to improving the accuracy of the obtained abnormality recognition result.

[0170] The present invention also provides a specific embodiment, which includes the following steps S010-S020 after step S101 and before step S104 to determine the final intention of the target object:

[0171] S010. Obtain a preset knowledge graph, which includes several triples. The first entity in the triple is the preset device type, the second entity is the target intent, and the relationship is a combination of relevant data corresponding to the preset device type and the target intent, including: the device name of the specified device corresponding to the preset device type, the working status of the specified device, the personnel type related to the working status, and the original intent corresponding to the personnel type. For example, the first entity is a temperature sensor, the second entity is completing maintenance and recording logs, and the relationship is a combination of relevant data corresponding to the temperature sensor and completing maintenance and recording logs, including sensor A, fault, equipment maintenance personnel, and performing daily maintenance.

[0172] Optionally, the preset device type is a device type obtained by deduplicating the device types of all specified devices.

[0173] Specifically, in a relevant data combination, the number of device name, working status, personnel type and original intention is 1.

[0174] S020. If, in the target object data list, the device type of the intermediate device corresponding to the target object is the same as the first entity in the triple, and the intermediate data combination corresponding to the target object data list is completely consistent with the relationship in the triple, then the second entity in the triple is taken as the final intention of the target object, wherein the intermediate data combination corresponding to the target object data list includes: the personnel type corresponding to the target object, the device name of the intermediate device corresponding to the target object, the working status of the intermediate device corresponding to the target object, and the initial intention corresponding to the target object.

[0175] Specifically, the intermediate data combination corresponding to the target object data list is completely consistent with the relationship in the triplet, which can be understood as: the personnel type corresponding to the target object in the intermediate data combination is the same as the personnel type related to the working status in the relationship, the device name of the intermediate device corresponding to the target object in the intermediate data combination is the same as the device name of the designated device corresponding to the preset device type in the relationship, the working status of the intermediate device corresponding to the target object in the intermediate data combination is the same as the working status of the designated device in the relationship, and the initial intention corresponding to the target object in the intermediate data combination is the same as the original intention corresponding to the personnel type in the relationship.

[0176] Through the above steps, the structured knowledge graph helps to improve the efficiency and accuracy of data analysis. According to the target object data list and the entities and relationships in the preset knowledge graph, the ultimate intention of the target object can be determined. The accurate matching and identification of the ultimate intention of the target object can be achieved quickly, which can improve the efficiency of determining the ultimate intention of the target object.

[0177] Specifically, the videos in the above embodiments (target video, historical video, target sub-video, associated video, standard operation video) can be replaced by their corresponding image groups, which include one frame of image per second of their corresponding videos.

[0178] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store a computer program related to a method in the method embodiment. The computer program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0179] An embodiment of the present invention further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in the above embodiment when executing the computer program.

[0180] An embodiment of the present invention further provides a computer program product comprising program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0181] The present invention provides a target video driven anomaly recognition method, electronic device and storage medium, the method can obtain a number of historical videos corresponding to a specified device and a specified device parameter list group corresponding to each historical video, build a device database based on the feature vector of the specified device parameter list group, the feature vector of the historical video corresponding to the specified device parameter list group, the status label corresponding to the historical video and the abnormal parameter name list corresponding to the historical video, input the feature vector of the target video, the feature vector list corresponding to the target device parameter list set and the total set of related data sets corresponding to the target device parameter list set obtained by recalling the target video and the target device parameter list set in the device database into a preset large prediction model to obtain anomaly recognition results. It can be seen that in the present invention, the device database is based on the target device parameter list set. The method is constructed by the feature vector of a specified device parameter list group, the feature vector of the historical video corresponding to the specified device parameter list group, the status label corresponding to the historical video and the list of abnormal parameter names corresponding to the historical video. It contains rich domain knowledge. The total set of related data sets corresponding to the target device parameter list set is a set recalled from the device database, which can provide necessary data support for the abnormality recognition task. The feature vector of the target video, the feature vector list corresponding to the target device parameter list set and the total set of related data sets corresponding to the target device parameter list set are input into the preset large prediction model to obtain the abnormality recognition result. It comprehensively utilizes multiple data sources, rather than simply judging whether there is an abnormality in the area based on the image or video to generate the abnormality recognition result, which is conducive to improving the accuracy of the obtained abnormality recognition results.

[0182] Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A target video driven anomaly recognition method, characterized in that: The method comprises the following steps: S001. For each designated device, obtain a number of historical videos corresponding to the designated device and a designated device parameter list group corresponding to each historical video; S002. Clustering the feature vectors of all designated device parameter list groups corresponding to the designated device to obtain several intermediate clusters corresponding to the designated device, and using the vector corresponding to the center position of the intermediate cluster as the intermediate center vector corresponding to the designated device; S003. If the feature vector of the specified device parameter list group is in the middle cluster corresponding to the middle center vector, the feature vector of the historical video corresponding to the specified device parameter list group is used as the key vector corresponding to the middle center vector, the state label corresponding to the historical video is used as the state label corresponding to the key vector, and the abnormal parameter name list corresponding to the historical video is used as the abnormal parameter name list corresponding to the key vector; S004. Store the intermediate center vector corresponding to the designated device, the key vector corresponding to the intermediate center vector, the state label corresponding to the key vector, and the abnormal parameter name list corresponding to the key vector in the device database; S005. Input the feature vector of SP, the feature vector list corresponding to MB, and the total set of related data sets corresponding to MB into the preset large language model to obtain the abnormality recognition result corresponding to SP output by the preset large language model, wherein the total set of related data sets corresponding to MB is obtained by recalling SP and MB in the device database, SP is the target video, and MB is the target device parameter list set corresponding to SP.

2. The target video driven anomaly recognition method according to claim 1, characterized in that: Before step S005, the method further includes step S1: S1. Whenever the preset duration is reached, the video captured by the target video capture device within the preset duration is used as SP and MB is obtained. The MB includes a target device parameter list group corresponding to several target devices. The target device parameter list group includes a target device parameter list corresponding to each parameter of the target device. The target device parameter list includes the specific parameter values ​​of the parameters in each second of the SP.

3. The target video driven anomaly recognition method according to claim 2, characterized in that: The duration of the historical video is the same as that of the target video.

4. The target video driven anomaly recognition method according to claim 2, characterized in that: The feature vector list corresponding to MB includes the feature vector of each target device parameter list group in MB. The feature vector of the target device parameter list group is a vector obtained by splicing the feature vectors of each target device parameter list according to the order of the target parameter lists in the target device parameter list group, wherein the vector values ​​in the feature vectors of the target device parameter list correspond one-to-one to the parameter values ​​in the target device parameter list.

5. The target video driven anomaly recognition method according to claim 4, characterized in that: The total set of related data sets corresponding to MB includes the related data sets corresponding to each target device parameter list group in MB, and the related data sets corresponding to the target device parameter list group include: a related video feature vector list corresponding to the target device parameter list group, a related device feature vector list corresponding to the target device parameter list group, and a status label list corresponding to the related video feature list, wherein the related video feature vector list includes several related video feature vectors, the related device feature vector list includes several related device feature vectors, and the status label list includes a status label corresponding to each related video feature vector.

6. The target video driven anomaly recognition method according to claim 5, characterized in that: In step S005, the process of recalling the SP and MB in the device database to obtain a total set of related data sets corresponding to the MB is performed, and the process of obtaining the related data sets corresponding to the target device parameter list group according to the SP, the target device parameter list group in the MB, the target device corresponding to the target device parameter list group, and the device database includes the following steps S0051-S0055: S0051. Obtain a list of intermediate center vectors R={R1, R2, ..., R} corresponding to a designated device that is the same device as the target device. g ,……,R h} , R g is the gth intermediate center vector corresponding to the specified device that is the same device as the target device, where g ranges from 1 to h, and h is the number of intermediate center vectors corresponding to the specified device that is the same device as the target device; S0052. Obtain B and R g The first average vector similarity K between g , where B is the characteristic vector of the target device parameter list group; S0053, if K g ≥C 0 , then R g As the relevant device feature vector corresponding to the target device parameter list group, R g The corresponding key vector is used as the relevant video feature vector corresponding to the target device parameter list group, and the state label corresponding to the key vector is used as the state label corresponding to the relevant video feature vector, where C 0 is the preset vector similarity threshold; S0054, if K1<C 0 , K2<C 0 ,……,K g <C 0 ,……,K h <C 0 , then based on R g The corresponding key vectors and the corresponding exception parameter name list get B and R g The second average vector similarity L between g ; S0055, if L g ≥C 0 , then R g As the relevant device feature vector corresponding to the target device parameter list group, R g The corresponding key vector is used as the relevant video feature vector corresponding to the target device parameter list group. If L1<C 0 , L2<C 0 ,……,L g <C 0 ,……,L h <C 0 , then L1, L2, ..., L g ,……,L h The intermediate center vector corresponding to the largest second average vector similarity is used as the relevant device feature vector corresponding to the target device parameter list group, and the key vector corresponding to the intermediate center vector is used as the relevant video feature vector corresponding to the target device parameter list group, wherein the state label corresponding to the key vector is used as the state label corresponding to its corresponding relevant video feature vector.

7. The target video driven anomaly recognition method according to claim 6, characterized in that: Step S0052 includes the following steps ac: a. Decompose B to obtain the first sub-vector list D = {D1, D2, ..., D j ,……,D n }, D j is the first subvector corresponding to the jth parameter of the target device in B, where j ranges from 1 to n, and n is the number of parameters of the target device; b. R g Decompose to obtain the intermediate subvector list Q g ={Q g1 , Q g2 ,……,Q gj ,……,Q gn }, Q gj R g The intermediate sub-vector corresponding to the j-th parameter of the target device; c. According to D j and Q gj Get K g , where K g Meet the following conditions: K g =∑ n j＝1 U gj / n,U gj D j With Q gj The vector similarity between .

8. The target video driven anomaly recognition method according to claim 7, characterized in that: In step S0054, based on R g The corresponding key vectors and the corresponding exception parameter name list get B and R g The second average vector similarity L between g The method includes the following steps S0061-0065: S0061、Get R g The corresponding key vector corresponds to the abnormal parameter name list set M g ={M g1 , M g2 ,……,M gr ,……,M gs(g) }, M gr R g The list of abnormal parameter names corresponding to the rth key vector, r ranges from 1 to s(g), s(g) is R g the number of corresponding key vectors; S0062, M g1 , M g2 ,……,M gr ,……,M gs(g) The union of R g Corresponding important parameter name list N g ={N g1 , N g2 ,……,N gk ,……,N gt(g) }, N gk R g The corresponding kth important parameter name, k ranges from 1 to t(g), t(g) is R g The number of corresponding important parameter names; S0063, according to N gk Get the jth parameter of the target device in R g The first importance weight P in the corresponding middle cluster jg , P jg Meet the following conditions: When the parameter name of the jth parameter of the target device is the same as N gk If the same, let P jg =1+N 0 gk , N 0 gk M g1 , M g2 ,……,M gr ,……,M gs(g) Including N gk The number of exception parameter name lists with the same exception parameter name; when the parameter name of the jth parameter of the target device is the same as N g1 , N g2 ,……,N gk ,……,N gt(g) If they are all different, let P jg =1; S0064, P 1g , P 2g ,……,P jg ,……,P ng Perform normalization to obtain P jg The corresponding normalized value, and P jg The corresponding normalized value is used as the jth parameter of the target device in R g The corresponding second importance weight W in the corresponding middle cluster jg ; S0065, according to D j 、W jg and Q gj Get L g , L g Meet the following conditions: L g =∑ n j＝1 (W jg ×U gj ) / ∑ n j＝1 W jg 。 9. A non-transitory computer-readable storage medium, characterized in that The storage medium stores a computer program, which is loaded and executed by a processor to implement the target video driven anomaly recognition method according to any one of claims 1 to 8.

10. An electronic device comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the target video drive anomaly identification method as described in any one of claims 1 to 8 when executing the computer program.

Images