Abnormality recognition method driven by target video, electronic equipment and storage medium

Abstract

The invention provides a target video driven exception recognition method, electronic equipment and a storage medium, and relates to the technical field of exception recognition. The method comprises the following steps: extracting feature vectors of a target video; and inputting a feature vector list corresponding to the target device parameter list set and a total set of related data sets corresponding to the target device parameter list set obtained by recalling in a device database according to the target video and the target device parameter list set into a preset large oracle model to obtain an anomaly recognition result. According to the method, multiple data sources are comprehensively utilized, and an equipment database is constructed based on feature vectors of a specified equipment parameter list group, feature vectors of historical videos corresponding to the specified equipment parameter list group, state tags corresponding to the historical videos and an abnormal parameter name list corresponding to the historical videos and contains rich domain knowledge; necessary data support can be provided for the anomaly recognition task, and the accuracy of the obtained anomaly recognition result can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of anomaly recognition, and particularly to an anomaly recognition method, an electronic device, and a storage medium driven by a target video. Background Art

[0002] In recent years, with the development of natural language processing technology, large language models have been widely applied in various fields, not only limited to text processing, but also combined with other technologies such as computer vision models to form multimodal systems. Multimodal systems can process various types of data (such as images, videos, texts, vectors, etc.), and combine these data through multimodal fusion technology to perform more complex recognition tasks, such as anomaly recognition tasks. Multimodal systems combine the capabilities of large language models and computer vision models, and can comprehensively analyze multimodal data to output anomaly recognition results.

[0003] In the prior art, an image or video corresponding to a region is input into a multimodal system to obtain an anomaly recognition result output by the multimodal system, and a user determines whether there is an anomaly in the region corresponding to the image or video according to the anomaly recognition result, where the anomaly includes but is not limited to situations such as potential safety hazards and operation errors.

[0004] However, the above method also has the following technical problems:

[0005] Although multimodal systems can provide more information than single modalities, they may still lack sufficient domain knowledge or context understanding ability in some cases, thus affecting the accuracy of judgment. For example, in a chemical production workshop, specific operation procedures and safety specifications may not be fully understood and applied by general models. Therefore, current multimodal systems can only simply judge whether there is an anomaly in a region based on an image or video to generate an anomaly recognition result, and the accuracy of the obtained anomaly recognition result is relatively low. Summary of the Invention

[0006] In view of the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] According to a first aspect of the present invention, there is provided an anomaly recognition method driven by a target video, the method including the following steps:

[0008] S001. For each specified device, obtain a plurality of historical videos corresponding to the specified device and a list group of specified device parameters corresponding to each historical video.

[0009] S002. Cluster the feature vectors of all the list groups of specified device parameters corresponding to the specified device to obtain a plurality of intermediate clusters corresponding to the specified device, and use the vector corresponding to the central position of the intermediate cluster as the intermediate central vector corresponding to the specified device.

[0010] S003. If the feature vector of the specified device parameter list group is in the intermediate cluster corresponding to the intermediate center vector, then use the feature vector of the historical video corresponding to the specified device parameter list group as the key vector corresponding to the intermediate center vector, use the status label corresponding to the historical video as the status label corresponding to the key vector, and use the list of abnormal parameter names corresponding to the historical video as the list of abnormal parameter names corresponding to the key vector.

[0011] S004. Store the intermediate center vector corresponding to the specified device, the key vector corresponding to the intermediate center vector, the status label corresponding to the key vector, and the list of abnormal parameter names corresponding to the key vector in the device database.

[0012] S005. Input the feature vector of the SP, the list of feature vectors corresponding to the MB, and the total set of relevant data sets corresponding to the MB into a preset large language model to obtain the abnormal recognition result corresponding to the SP output by the preset large language model. Among them, recall according to the SP and the MB in the device database to obtain the total set of relevant data sets corresponding to the MB. The SP is the target video, and the MB is the set of target device parameter lists corresponding to the SP.

[0013] According to a second aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the foregoing method.

[0014] According to a third aspect of the present invention, there is provided an electronic device including: a processor, a memory, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the foregoing method is implemented.

[0015] The present invention has at least the following beneficial effects:

[0016] The present invention provides a method for anomaly recognition driven by a target video, an electronic device, and a storage medium. The method can obtain a plurality of historical videos corresponding to a specified device and a list group of specified device parameters corresponding to each historical video, construct a device database based on the feature vectors of the list group of specified device parameters, the feature vectors of the historical videos corresponding to the list group of specified device parameters, the status labels corresponding to the historical videos, and the list of anomaly parameter names corresponding to the historical videos. The total set of the feature vector of the target video, the list of feature vectors corresponding to the target device parameter list set, and the relevant data sets corresponding to the target device parameter list set retrieved from the device database according to the target video and the target device parameter list set is input into a preset large prediction model to obtain an anomaly recognition result. It can be seen that in the present invention, the device database is constructed based on the feature vectors of the list group of specified device parameters, the feature vectors of the historical videos corresponding to the list group of specified device parameters, the status labels corresponding to the historical videos, and the list of anomaly parameter names corresponding to the historical videos. It contains rich domain knowledge. The total set of the relevant data sets corresponding to the target device parameter list set is a set retrieved from the device database, which can provide necessary data support for the anomaly recognition task. Inputting the feature vector of the target video, the list of feature vectors corresponding to the target device parameter list set, and the total set of the relevant data sets corresponding to the target device parameter list set into the preset large prediction model to obtain an anomaly recognition result comprehensively utilizes multiple data sources, rather than simply judging whether there is an anomaly in a region based on an image or a video to generate an anomaly recognition result, which is beneficial to improving the accuracy of the obtained anomaly recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a flowchart of a method for anomaly recognition driven by a target video provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0020] It should be noted that the terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present invention are used to distinguish similar tasks and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or server comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0021] An embodiment of the present invention provides an abnormal recognition method driven by a target video. The method includes the following steps, as Figure 1 shown:

[0022] S001. For each specified device, obtain a number of historical videos corresponding to the specified device and a group of specified device parameter lists corresponding to each historical video. The group of specified device parameter lists includes a specified device parameter list corresponding to each parameter of the specified device. The specified device parameter list includes the specific parameter values of the parameters within each second of the historical video. Wherein, each historical video corresponds to a status label and a list of abnormal parameter names.

[0023] Specifically, the specified device is a device located in the target area.

[0024] Specifically, the historical video corresponding to the specified device is a video that can present the specified device and was collected before the current time point.

[0025] Specifically, the duration of the historical video is the same as the duration of the target video.

[0026] Specifically, the status label includes a normal label and an abnormal label.

[0027] Specifically, when the status label corresponding to the historical video is a normal label, it means that the sub-video corresponding to the specified device in the historical video is consistent with the standard video corresponding to the group of specified device parameter lists. The sub-video corresponding to the specified device is the part of the historical video that only presents the specified device.

[0028] Specifically, when the status label corresponding to the historical video is an abnormal label, it means that the sub-video corresponding to the specified device in the historical video is inconsistent with the standard video corresponding to the group of specified device parameter lists, and there may be a fault or improper operation.

[0029] Specifically, when the status label corresponding to the historical video is a normal label, the list of abnormal parameter names corresponding to the historical video is NULL.

[0030] Specifically, when the status label corresponding to the historical video is an abnormal label, the list of abnormal parameter names corresponding to the historical video includes several abnormal parameter names, and the abnormal parameter name is the name of the parameter that causes the historical video to be abnormal.

[0031] S002. Cluster the feature vectors of all specified device parameter list groups corresponding to the specified device to obtain several intermediate clusters corresponding to the specified device, and use the vector corresponding to the central position of the intermediate cluster as the intermediate central vector corresponding to the specified device. Among them, based on the vector similarity between the feature vectors of the specified device parameter list groups corresponding to the specified device, cluster the feature vectors of all specified device parameter list groups corresponding to the specified device to obtain several intermediate clusters corresponding to the specified device and use the vector corresponding to the central position of the intermediate cluster as the intermediate central vector corresponding to the specified device. The intermediate clusters corresponding to the specified device include the feature vectors of several specified device parameter list groups corresponding to the specified device; it can be understood that: use the vector similarity between the feature vectors of the specified device parameter list groups corresponding to the specified device as the distance metric in the preset clustering algorithm, and use the preset clustering algorithm to cluster the feature vectors of all specified device parameter list groups corresponding to the specified device to obtain several intermediate clusters corresponding to the specified device.

[0032] Specifically, the vector corresponding to the central position of the intermediate cluster is the average value of all feature vectors in the intermediate cluster.

[0033] Specifically, the specified device parameter list group corresponding to the specified device can be understood as the specified device parameter list group corresponding to the historical video corresponding to the specified device.

[0034] Specifically, the feature vector of the specified device parameter list group is a vector obtained by splicing the feature vectors of each specified device parameter list in the order of the specified device parameter list in the specified device parameter list group, where the vector values in the feature vector of the specified device parameter list correspond one-to-one with the parameter values in the specified device parameter list.

[0035] S003. If the feature vector of the specified device parameter list group is in the intermediate cluster corresponding to the intermediate central vector, use the feature vector of the historical video corresponding to the specified device parameter list group as the key vector corresponding to the intermediate central vector, use the status label corresponding to the historical video as the status label corresponding to the key vector, and use the list of abnormal parameter names corresponding to the historical video as the list of abnormal parameter names corresponding to the key vector.

[0036] S004. Store the intermediate central vector corresponding to the specified device, the key vector corresponding to the intermediate central vector, the status label corresponding to the key vector, and the list of abnormal parameter names corresponding to the key vector in the device database.

[0037] Through the above steps, based on the vector similarity between the feature vectors of the specified device parameter list groups corresponding to the specified device, cluster the feature vectors of all the specified device parameter list groups corresponding to the specified device to obtain several intermediate clusters corresponding to the specified device, and use the vector corresponding to the center position of the intermediate cluster as the intermediate center vector corresponding to the specified device. If the feature vector of the specified device parameter list group is in the intermediate cluster corresponding to the intermediate center vector, then use the feature vector of the historical video corresponding to the specified device parameter list group as the key vector corresponding to the intermediate center vector, use the status label corresponding to the historical video as the status label corresponding to the key vector, use the list of abnormal parameter names corresponding to the historical video as the list of abnormal parameter names corresponding to the key vector, and store the intermediate center vector corresponding to the specified device, the key vector corresponding to the intermediate center vector, the status label corresponding to the key vector, and the list of abnormal parameter names corresponding to the key vector in the device database. It is possible to reduce redundant information through clustering, save storage space, store the status label corresponding to the key vector and the list of abnormal parameter names corresponding to the key vector in the device database, so that the device database contains rich and detailed domain knowledge, can provide data support for the abnormal recognition task, and is conducive to improving the accuracy of the obtained abnormal recognition results.

[0038] S005. Input the feature vector of SP, the list of feature vectors corresponding to MB, and the total set of relevant data sets corresponding to MB into a preset large language model to obtain the abnormal recognition result corresponding to SP output by the preset large language model. Among them, recall according to SP and MB in the device database to obtain the total set of relevant data sets corresponding to MB. SP is the target video, and MB is the set of target device parameter lists corresponding to SP. Those skilled in the art know that any method for obtaining the feature vector of a video in the prior art belongs to the protection scope of the present invention and will not be elaborated here.

[0039] Optionally, the device database is initially NULL.

[0040] Specifically, before step S005, there is also step S1:

[0041] S1. Whenever the preset duration is reached, use the video collected by the target video acquisition device within the preset duration as SP and obtain MB. MB includes several target device parameter list groups corresponding to the target device. Each target device parameter list group includes the target device parameter list corresponding to each parameter of the target device. The target device parameter list includes the specific parameter values of the parameters within each second of SP. The preset duration is the duration preset by those skilled in the art according to actual needs. For example: 1 minute, 2 minutes, which will not be elaborated here.

[0042] Specifically, the parameters of the device refer to the specific performance indicators during the operation of the device, such as power, speed, accuracy, temperature, etc.

[0043] Specifically, the target video acquisition device is a video acquisition device set in the target area. In a specific embodiment, the target area can be a chemical production workshop.

[0044] Furthermore, several video acquisition devices are set in the target area.

[0045] Specifically, the target device is a device located within the video acquisition area of the target video acquisition device. In a specific embodiment, the devices located within the video acquisition area of the target video acquisition device can be chemical production devices (e.g., reaction kettles, material transfer devices, polymerization kettles, distillation towers, extraction towers) and detection devices (e.g., gas concentration detection devices, temperature detection devices, pressure detection devices, noise detection devices, particulate matter detection devices).

[0046] Specifically, each target device is equipped with at least one sensor for real-time acquisition of the parameters of the target device. For example, the reaction kettle is equipped with a temperature sensor, a pressure sensor, and a liquid level sensor.

[0047] Specifically, the target video acquisition device is equipped with a data receiving port, and the data receiving port is used to receive the parameters transmitted from the target device and its configured sensors.

[0048] In a specific embodiment, step S1 includes: whenever a specific time point is reached, taking the video captured by the target video acquisition device between the specific time point and the previous specific time point as the SP and obtaining the MB.

[0049] Specifically, the feature vector list corresponding to the MB includes the feature vectors of each target device parameter list group in the MB.

[0050] Specifically, the feature vector of the target device parameter list group is a vector obtained by splicing the feature vectors of each target device parameter list in the order of the target parameter list in the target device parameter list group, where the vector values in the feature vector of the target device parameter list correspond one-to-one with the parameter values in the target device parameter list.

[0051] Specifically, the total set of relevant data sets corresponding to MB includes the relevant data sets corresponding to each target device parameter list group in MB. The relevant data set corresponding to the target device parameter list group includes: the relevant video feature vector list corresponding to the target device parameter list group, the relevant device feature vector list corresponding to the target device parameter list group, and the status label list corresponding to the relevant video feature list. Among them, the relevant video feature vector list includes several relevant video feature vectors, the relevant device feature vector list includes several relevant device feature vectors, and the status label list includes the status label corresponding to each relevant video feature vector.

[0052] Specifically, the anomaly recognition result corresponding to SP includes an anomaly judgment label. The anomaly judgment label includes no anomaly and presence of anomaly. When the anomaly judgment label is presence of anomaly, the anomaly recognition result further includes an anomaly device identifier.

[0053] Furthermore, when the anomaly judgment label in the anomaly recognition result corresponding to SP is no anomaly, it means that all target devices corresponding to SP have no anomaly.

[0054] Furthermore, when the anomaly judgment label in the anomaly recognition result corresponding to SP is presence of anomaly, it means that the target device corresponding to the anomaly device identifier has an anomaly. For example, the temperature of the target device is too high or there are safety hazards.

[0055] In a specific embodiment, the pre-set large language model is a general large language model, such as GPT, BERT.

[0056] In a specific embodiment, the pre-set large language model is a dedicated model obtained by a person skilled in the art through technical means such as fine-tuning on the basis of a general large language model and optimizing and adjusting for the anomaly recognition task.

[0057] Through the above steps, the feature vector of the target video, the feature vector list corresponding to the target device parameter list set, and the total set of relevant data sets corresponding to the target device parameter list set are input into the pre-set large prediction model to obtain the anomaly recognition result. A variety of data sources are comprehensively utilized, rather than simply judging whether there is an anomaly in the area only based on images or videos to generate the anomaly recognition result, which is beneficial to improving the accuracy of the obtained anomaly recognition result.

[0058] Specifically, in the step S005 of recalling in the device database according to SP and MB to obtain the total set of relevant data sets corresponding to MB, the relevant data set corresponding to the target device parameter list group is obtained according to SP, the target device parameter list group in MB, the target device corresponding to the target device parameter list group, and the device database, including the following steps S0051 - S0055:

[0059] S0051. Obtain the intermediate central vector list R = {R1, R2, ……, R g , ……, R h} , R g is the g-th intermediate central vector corresponding to the specified device that is the same device as the target device, where g ranges from 1 to h, and h is the number of intermediate central vectors corresponding to the specified device that is the same device as the target device.

[0060] S0052. Obtain the first average vector similarity K g between B and R g , where B is the feature vector of the target device parameter list group.

[0061] Specifically, step S0052 includes the following steps a - c:

[0062] a. Decompose B to obtain the first sub - vector list D = {D1, D2, ……, D j , ……, D n}, where D j is the first sub - vector corresponding to the j - th parameter of the target device in B, and j ranges from 1 to n, where n is the number of parameters of the target device; the first sub - vector corresponding to the j - th parameter of the target device can be understood as: the feature vector of the target device parameter list corresponding to the j - th parameter of the target device.

[0063] b. Decompose R g to obtain the intermediate sub - vector list Q g = {Q g1 , Q g2 , ……, Q gj , ……, Q gn}, where Q gj is the intermediate sub - vector corresponding to the j - th parameter of the target device in R g ; the intermediate sub - vector corresponding to the j - th parameter of the target device in R g can be understood as: the average value of all key feature vectors corresponding to the j - th parameter of the target device in the intermediate cluster corresponding to R g , where the key feature vectors corresponding to the j - th parameter of the target device in the intermediate cluster corresponding to R g are the feature vectors of the specified device parameter list corresponding to the j - th parameter of the target device in the specified device parameter list group corresponding to the key vectors in the intermediate cluster.

[0064] c. Obtain K j according to D gj and Q g , where K g meets the following conditions:

[0065] K g =∑ n j＝1 U gj / n,U gj D j With Q gj The vector similarity between .

[0066] Specifically, the greater the vector similarity, the more similar the two vectors are.

[0067] Through the above steps, the parameters of the target device are used as the splitting dimensions, the characteristic vector of the target device parameter list group is split into the first sub-vector corresponding to the parameters of the target device, and the intermediate center vector is split into the intermediate sub-vector corresponding to the parameters of the target device. Based on the vector similarity between the first sub-vector and the intermediate sub-vector corresponding to the parameters of the target device, the average vector similarity between the characteristic vector and the intermediate center vector of the target device parameter list group is obtained, which can smooth out the influence of individual abnormal parameter values.

[0068] S0053, if K g ≥C 0 , then R g As the relevant device feature vector corresponding to the target device parameter list group, R g The corresponding key vector is used as the relevant video feature vector corresponding to the target device parameter list group, and the state label corresponding to the key vector is used as the state label corresponding to the relevant video feature vector, where C 0 It is a preset vector similarity threshold. Those skilled in the art know that the preset vector similarity threshold is a value less than 1 pre-set by those skilled in the art according to actual needs, for example, 0.8, 0.9, 0.95, which will not be repeated here.

[0069] S0054, if K1<C 0 , K2<C 0 , ..., K g <C 0 , ..., K h <C 0 , then based on R g The corresponding key vector corresponding to the abnormal parameter name list obtains B and R g The second average vector similarity L between g .

[0070] Specifically, in step S0054, based on R g The corresponding key vector corresponding to the abnormal parameter name list obtains B and R g The second average vector similarity L between g The method comprises the following steps S0061-0065:

[0071] S0061. Obtain R g The set M of lists of names of abnormal parameters corresponding to the corresponding key vectors g ={M g1 , M g2 , ……, M gr , ……, M gs(g)}, where M gr is the list of names of abnormal parameters corresponding to the r-th key vector corresponding to R, and the value range of r is from 1 to s(g), and s(g) is the number of key vectors corresponding to R g . g

[0072] S0062. Take the union of M g1 , M g2 , ……, M gr , ……, M gs(g) as the list N of names of important parameters corresponding to R g ={N g , N g1 , ……, N g2 , ……, N gk , ……, N gt(g)}, where N gk is the k-th name of an important parameter corresponding to R, and the value range of k is from 1 to t(g), and t(g) is the number of names of important parameters corresponding to R g . g

[0073] S0063. Obtain the first importance weight P gk corresponding to the j-th parameter of the target device in the corresponding intermediate cluster of R g , and P jg , P jg meets the following conditions:

[0074] When the name of the j-th parameter of the target device is the same as N gk , let P jg =1 + N 0 gk , where N 0 gk is the number of lists of names of abnormal parameters in M g1 , M g2 , ……, M gr , ……, M gs(g) that include the same name of the abnormal parameter as N gk ; when the name of the j-th parameter of the target device is different from N g1 , N g2 , ……, N gk , ……, N gt(g) are all different, let P​​jg = 1.

[0075] S0064. Normalize P 1g , P 2g , ……, P jg , ……, P ng to obtain the normalized value of P jg , and use the normalized value corresponding to P jg as the second importance weight W g corresponding to the j-th parameter of the target device in the intermediate cluster corresponding to R jg .

[0076] Specifically, the greater the second importance weight, the more important the corresponding parameter.

[0077] S0065. Obtain L j , W jg and Q gj to obtain L g , where L g meets the following condition:

[0078] L g = ∑ n j＝1 (W jg × U gj ) / ∑ n j＝1 W jg .

[0079] Through the above steps, obtain the set of lists of abnormal parameter names corresponding to the key vectors corresponding to the intermediate center vectors of the specified device that is the same device as the target device. Take the union of all the lists of abnormal parameter names in the set of lists of abnormal parameter names as the list of important parameter names corresponding to the intermediate center vector. According to the parameter names of the parameters of the target device, the list of important parameter names, and the number of lists of abnormal parameter names containing the important parameter names, obtain the first importance weight corresponding to the parameters of the target device in the intermediate cluster. Normalize the first importance weights corresponding to all the parameters of the target device in the intermediate cluster to obtain the second importance weight corresponding to the parameters of the target device in the intermediate cluster, eliminating the dimensional difference of the weights between different parameters. Based on the second importance weight corresponding to the parameters of the target device in the intermediate cluster, obtain the second average vector similarity between the vector features of the target device parameter list group and the intermediate center vector considering the vector similarity between the first sub-vector and the intermediate sub-vector corresponding to the parameters of the target device, taking into account the importance degree of the parameters themselves, so as to improve the accuracy of the obtained relevant data set according to the relevant data set related to the second average vector similarity.

[0080] S0055. If L g ≥ C0 , then use R g as the relevant device feature vector corresponding to the target device parameter list group, and use the R g corresponding key vector as the relevant video feature vector corresponding to the target device parameter list group. If L1 < C 0 , L2 < C 0 , ……, L g < C 0 , ……, L h < C 0 , then use the middle center vector corresponding to the largest second average vector similarity among L1, L2, ……, L g , ……, L h as the relevant device feature vector corresponding to the target device parameter list group, and use the key vector corresponding to the middle center vector as the relevant video feature vector corresponding to the target device parameter list group. Among them, use the status label corresponding to the key vector as the status label corresponding to its corresponding relevant video feature vector.

[0081] Through the above steps, obtain the intermediate center vector corresponding to the specified device that is the same device as the target device, and obtain the first average vector similarity between the feature vector of the target device parameter list group and the intermediate center vector. When the first average vector similarity is not less than the preset vector similarity, it indicates that the feature vector of the target device parameter list group is very similar to the intermediate center vector. It can be understood that the target device parameter list group is also very similar to the specified device parameter list group corresponding to the feature vector in the intermediate cluster corresponding to the intermediate center vector. Therefore, use the intermediate center vector as the relevant device feature vector corresponding to the target device parameter list group, use the key vector corresponding to the intermediate center vector as the relevant video feature vector corresponding to the target device parameter list group, and use the status label corresponding to the key vector as the status label corresponding to its corresponding relevant video feature vector, which can provide necessary data support for the anomaly recognition task; if the first average vector similarity between the feature vector of the target device parameter list group and any intermediate center vector is less than the preset vector similarity threshold, then obtain the second average vector similarity between the feature vector of the target device parameter list group and the intermediate center vector. When the second average vector similarity is not less than the preset vector similarity, it indicates that the feature vector of the target device parameter list group is very similar to the intermediate center vector. It can be understood that the target device parameter list group is also very similar to the specified device parameter list group corresponding to the feature vector in the intermediate cluster corresponding to the intermediate center vector. Therefore, use the intermediate center vector as the relevant device feature vector corresponding to the target device parameter list group, use the key vector corresponding to the intermediate center vector as the relevant video feature vector corresponding to the target device parameter list group, and use the status label corresponding to the key vector as the status label corresponding to its corresponding relevant video feature vector; it can provide necessary data support for the anomaly recognition task; otherwise, use the intermediate center vector corresponding to the maximum second average vector similarity as the relevant device feature vector corresponding to the target device parameter list group, use the key vector corresponding to the intermediate center vector as the relevant video feature vector corresponding to the target device parameter list group, and use the status label corresponding to the key vector as the status label corresponding to its corresponding relevant video feature vector; it can provide necessary data support for the anomaly recognition task; which is beneficial to improving the accuracy of the obtained anomaly recognition result.

[0082] The present invention also provides a specific embodiment. Among them, the relevant data set corresponding to the target device parameter list group includes: a list of relevant video feature vectors corresponding to the target device parameter list group, a list of relevant device feature vectors corresponding to the target device parameter list group, and a list of status labels corresponding to the list of relevant device feature vectors. Among them, the list of relevant video feature vectors includes several relevant video feature vectors, the list of relevant device feature vectors includes several relevant device feature vectors, and the list of status labels includes the status label corresponding to each relevant device feature vector.

[0083] Specifically, when the status label corresponding to the relevant device feature vector is a normal label, it indicates that there is no abnormality in the relevant device feature vector.

[0084] Specifically, when the status label corresponding to the relevant device feature vector is an abnormal label, it indicates that there is an abnormality in the relevant device feature vector. For example, the vector value in the relevant device feature vector is too large.

[0085] Before step S005, the following steps S01 - S04 are included to construct a device database:

[0086] S01. Cluster the feature vectors of all historical videos corresponding to a specified device to obtain several first central vectors corresponding to the specified device. Among them, based on the vector similarity between the feature vectors of the historical videos corresponding to the specified device, cluster the feature vectors of all historical videos corresponding to the specified device to obtain several first clusters corresponding to the specified device, and use the vector corresponding to the central position of the first cluster as the first central vector corresponding to the specified device. The first cluster corresponding to the specified device includes the feature vectors of several historical videos corresponding to the specified device. It can be understood that the vector similarity between the feature vectors of the historical videos corresponding to the specified device is used as the distance metric in the preset clustering algorithm, and the preset clustering algorithm is used to cluster the feature vectors of all historical videos corresponding to the specified device to obtain several first clusters corresponding to the specified device. Those skilled in the art know that the preset clustering algorithm is a clustering algorithm that does not require specifying the number of clusters in advance and can use vector similarity as the distance metric. For example, the hierarchical clustering method, any method for obtaining the vector similarity between vectors in the prior art, such as: all belong to the protection scope of the present invention, such as: cosine similarity, Euclidean distance, which will not be elaborated here.

[0087] Specifically, the vector corresponding to the central position of the first cluster is the average value of all feature vectors in the first cluster.

[0088] Specifically, before step S01, the following steps are further included: For each specified device, obtain several historical videos corresponding to the specified device and the specified device parameter list group corresponding to each historical video. The specified device parameter list group includes the specified device parameter list corresponding to each parameter of the specified device. The specified device parameter list includes the specific parameter values of the parameters within each second of the historical video. Among them, each specified device parameter list group corresponds to a status label.

[0089] Specifically, when the status label corresponding to the specified device parameter list group is an abnormal label, it indicates that the specified device parameter list group is inconsistent with the standard device parameter list group corresponding to the specified device in the historical video corresponding to the specified device parameter list group, and there may be a fault or improper operation.

[0090] Specifically, when the status label corresponding to the specified device parameter list group is a normal label, it indicates that the specified device parameter list group is consistent with the standard parameter list group corresponding to the specified device in the historical video corresponding to the specified device parameter list group.

[0091] S02. If the feature vector of the historical video is in the first cluster corresponding to the first central vector, then use the feature vector of the specified device parameter list group corresponding to the historical video as the intermediate vector corresponding to the first central vector, and use the status label corresponding to the specified device parameter list group as the status label corresponding to the intermediate vector.

[0092] S03. Cluster all the intermediate vectors corresponding to the first central vector to obtain the second central vector corresponding to the first central vector and the status label corresponding to the second central vector, including the following steps S031 - S032:

[0093] S031. Based on the vector similarity between the intermediate vectors corresponding to the first central vector, cluster all the intermediate vectors corresponding to the first central vector to obtain several second clusters corresponding to the first central vector. Among them, the second clusters corresponding to the first central vector include several intermediate vectors corresponding to the first central vector. It can be understood that the vector similarity between the intermediate vectors corresponding to the first central vector is used as the distance metric in the preset clustering algorithm, and the preset clustering algorithm is used to cluster all the intermediate vectors corresponding to the first central vector to obtain several second clusters corresponding to the first central vector.

[0094] S032. If the status labels corresponding to the intermediate vectors in the second cluster corresponding to the first central vector are not completely consistent, then divide the second cluster into two sub - clusters according to the status labels, delete the second cluster, and use the two sub - clusters as two new second clusters corresponding to the first central vector. If the status labels corresponding to the intermediate vectors in the second cluster corresponding to the first central vector are completely consistent, then use the vector corresponding to the central position of the second cluster as the second central vector corresponding to the first central vector, and use the status label corresponding to any intermediate vector in the second cluster as the status label corresponding to the second central vector.

[0095] Specifically, when dividing the second cluster into two sub - clusters according to the status labels, one sub - cluster includes all the intermediate vectors with normal labels in the second cluster, and the other sub - cluster includes all the intermediate vectors with abnormal labels in the second cluster.

[0096] Specifically, the vector corresponding to the central position of the second cluster is the average value of all the intermediate vectors in the second cluster.

[0097] Through the above steps, when the state labels corresponding to the intermediate vectors in the second cluster corresponding to the first central vector are not completely consistent, a single state label cannot represent the state labels corresponding to all the intermediate vectors in the second cluster. Therefore, the second cluster is divided into two sub-clusters according to the state labels, and the second cluster is deleted while the two sub-clusters are used as two new second clusters corresponding to the first central vector. The state labels corresponding to all the intermediate vectors in the new second clusters are the same, and a single state label can be used to represent the state labels corresponding to all the intermediate vectors in the new second clusters, which is convenient for storage. There is no need to store the state labels corresponding to all the intermediate vectors in the second cluster in the device database, which can effectively reduce redundant information, reduce the data storage volume, save storage space, make the device database more concise and efficient, and facilitate subsequent query and use.

[0098] S04. Store the first central vector corresponding to the specified device, the second central vector corresponding to the first central vector, and the state label corresponding to the second central vector in the device database.

[0099] Through the above steps, first, cluster the feature vectors of all historical videos corresponding to the specified device according to the vector similarity between the feature vectors of the historical videos corresponding to the specified device, cluster the feature vectors of similar historical videos into the same first cluster, and represent the first cluster with the first central vector; use the feature vectors of the specified device parameter list group corresponding to the historical videos corresponding to the feature vectors in the first cluster as the intermediate vectors corresponding to the first central vector corresponding to the first cluster, and use the state label corresponding to the specified device parameter list group as the state label corresponding to its corresponding intermediate vector. Then, according to the vector similarity between the intermediate vectors corresponding to the first central vector, cluster the similar intermediate vectors into the same second cluster to obtain the second central vector corresponding to the first central vector and the state label corresponding to the second central vector; store the first central vector corresponding to the specified device, the second central vector corresponding to the first central vector, and the state label corresponding to the second central vector in the device database, so that the device database contains rich domain knowledge, can provide data support for the anomaly recognition task, and through multi-level (first cluster, second cluster) clustering analysis, can effectively reduce redundant information, reduce the data storage volume, save storage space, make the device database more concise and efficient, and facilitate subsequent query and use.

[0100] In the step of recalling in the device database according to SP and MB in step S005 to obtain the total set of the relevant data set corresponding to MB, obtain the relevant data set corresponding to the target device parameter list group according to SP, the target device parameter list group in MB, the target device corresponding to the target device parameter list group, and the device database, including the following steps S051 - S054:

[0101] S051. Obtain the third central vector list A = {A1, A2, ……, A i , ……, A m} corresponding to the specified device that is the same device as the target device. A i is the i-th third central vector corresponding to the specified device that is the same device as the target device, where the value of i ranges from 1 to m, and m is the number of third central vectors corresponding to the specified device that is the same device as the target device. The third central vector corresponding to the specified device is the second central vector corresponding to the first central vector corresponding to the specified device.

[0102] S052. Obtain the average vector similarity C i between B and A i .

[0103] Specifically, step S052 includes the following steps S0521 - S0523:

[0104] S0521. Decompose B to obtain the first sub-vector list D = {D1, D2, ……, D j , ……, D n}.

[0105] S0522. Decompose A i to obtain the second sub-vector list E i = {E i1 , E i2 , ……, E ij , ……, E in}, where E ij is the second sub-vector corresponding to the j-th parameter of the target device in A i ; the second sub-vector corresponding to the j-th parameter of the target device in A i can be understood as: the average value of all specified feature vectors corresponding to the j-th parameter of the target device in the second cluster corresponding to A i , where the specified feature vectors corresponding to the j-th parameter of the target device in the second cluster corresponding to A i are the feature vectors of the specified device parameter list corresponding to the j-th parameter of the target device in the specified device parameter list group corresponding to the intermediate vector in the second cluster.

[0106] S0523. Obtain C j according to D ij and E i , where C i meets the following conditions:

[0107] C i = ∑ n j＝1 F ij / n, Fij is D j and E ij the vector similarity between them

[0108] Through the above steps, taking the parameters of the target device as the splitting dimension, splitting the feature vectors of the target device parameter list group into the first sub-vectors corresponding to the parameters of the target device, splitting the third central vector into the second sub-vectors corresponding to the parameters of the target device, and based on the vector similarity between the first sub-vectors and the second sub-vectors corresponding to the parameters of the target device, obtaining the average vector similarity between the feature vectors of the target device parameter list group and the third central vector, which can smooth out the influence of individual abnormal parameter values

[0109] S053. If C i ≥ C 0 , then take A i as the relevant device feature vector corresponding to the target device parameter list group, take the status label corresponding to A i as the status label corresponding to the relevant device feature vector corresponding to A i , and take the first central vector corresponding to A i as the relevant video feature vector corresponding to the target device parameter list group

[0110] Through the above steps, obtaining the average vector similarity between the feature vectors of the target device parameter list group and the third central vector corresponding to the specified device of the same device as the target device. If the average similarity between the feature vectors of the target device parameter list group and the third central vector is not less than the preset vector similarity threshold, it indicates that the feature vectors of the target device parameter list group are very similar to the third central vector. It can be understood that the target device parameter list group is also similar to the specified device parameter list group corresponding to the feature vectors in the second cluster corresponding to the third central vector. Therefore, taking the third central vector as the relevant device feature vector corresponding to the target device parameter list group, taking the status label corresponding to the third central vector as the status label corresponding to the relevant device feature vector, and taking the first central vector corresponding to the third central vector as the relevant video feature vector corresponding to the target device parameter list group can provide necessary data support for the anomaly recognition task and is conducive to improving the accuracy of the obtained anomaly recognition results

[0111] S054. If C1 < C 0 , C2 < C 0 , ……, C i < C 0 , ……, C m < C 0 , then obtain the relevant data set corresponding to the target device parameter list group according to SP, including the following steps S0541 - S0544

[0112] S0541. Obtain the target sub - video corresponding to the target device from the SP. Here, the target sub - video is the part of the video in the SP that only presents the target device. Those skilled in the art know that any method of obtaining a sub - video from a video in the prior art falls within the protection scope of the present invention and will not be elaborated here.

[0113] S0542. Obtain the first central vector list G = {G1, G2, ……, G e , ……, G f} corresponding to the specified device that is the same device as the target device, where G e is the e - th first central vector corresponding to the specified device that is the same device as the target device, and the value of e ranges from 1 to f, where f is the number of the first central vectors corresponding to the specified device that is the same device as the target device.

[0114] S0543. Obtain the vector similarity H e between the feature vector of the target sub - video and G e .

[0115] S0544. If H e ≥C 0 , then take G e as the relevant video feature vector corresponding to the target device parameter list group, and take the second central vector corresponding to G e as the relevant device feature vector corresponding to the target device parameter list group; if H1 < C 0 , H2 < C 0 , ……, H e <C 0 , ……, H f <C 0 , then take the first central vector corresponding to the maximum vector similarity among H1, H2, ……, H e , ……, H f as the relevant video feature vector corresponding to the target device parameter list group, and take the second central vector corresponding to the first central vector as the relevant device feature vector corresponding to the target device parameter list group, where the status label corresponding to the second central vector is used as the status label corresponding to the relevant device feature vector.

[0116] Through the above steps, if the average similarity between the feature vector of the target device parameter list group and any one of the third central vectors is less than the preset vector similarity threshold, it indicates that the feature vector of the target device parameter list group is not very similar to these third central vectors, and data related to the target device parameter list group cannot be obtained based on the third central vectors. At this time, the target sub-video corresponding to the target device is obtained from the target video, and the vector similarity between the feature vector of the target sub-video and the first central vector corresponding to the specified device of the same device as the target device is obtained. When the vector similarity is not less than the preset vector similarity threshold, it indicates that the feature vector of the target sub-video is very similar to the first central vector, which can be understood as the target sub-video is also similar to the historical videos corresponding to the feature vectors in the first cluster corresponding to the first central vector. Therefore, the first central vector is used as the relevant video feature vector corresponding to the target device parameter list group, the second central vector corresponding to the first central vector is used as the relevant device feature vector corresponding to the target device parameter list group, and the status label corresponding to the second central vector is used as the status label corresponding to its corresponding relevant device feature vector; it can provide necessary data support for the anomaly recognition task. If the vector similarity between the feature vector of the target sub-video and the first central vector corresponding to the specified device of the same device as the target device is less than the preset vector similarity threshold, it indicates that the feature vector of the target sub-video is not very similar to these first central vectors. In this case, compared with not being able to provide necessary data support for the anomaly recognition task, it is the best choice to use the first central vector corresponding to the maximum vector similarity threshold as the relevant video feature vector corresponding to the target device parameter list group, the second central vector corresponding to the first central vector is used as the relevant device feature vector corresponding to the target device parameter list group, and the status label corresponding to the second central vector is used as the status label corresponding to its corresponding relevant device feature vector, which can provide necessary data support for the anomaly recognition task and is beneficial to improving the accuracy of the obtained anomaly recognition results.

[0117] The present invention also provides a specific embodiment, wherein step S005 includes: inputting SP, MB, and the associated dataset corresponding to MB into a preset large language model to obtain the anomaly recognition result corresponding to SP output by the preset large language model. The associated dataset corresponding to MB includes an associated data subset corresponding to each target device parameter list group in MB. The associated data subset corresponding to the target device parameter list group is a dataset recalled from the device database according to SP and the target device parameter list group, including: an associated video set corresponding to the target device parameter list group, an associated parameter list group set corresponding to the target device parameter list group, and a status label list corresponding to the associated parameter list group set. The associated video set includes several associated videos, the associated parameter list group set includes several associated parameter list groups, and the status label list includes the status label corresponding to each associated parameter list group.

[0118] After step S04, the following step S05 of constructing a device database is further included:

[0119] S05. Store the historical video corresponding to each specified device and the specified device parameter list group corresponding to the historical video in the device database, and establish an association relationship between the first central vector and the historical video corresponding to the feature vector in the first cluster corresponding to the first central vector, and establish an association relationship between the second central vector and the specified device parameter list group corresponding to the intermediate vector in the second cluster corresponding to the second central vector.

[0120] Step S053 includes: If C i ≥C 0 , then regard the specified device parameter list group having an association relationship with A i as the associated parameter list group corresponding to the target device parameter list group, and regard the status label corresponding to the specified device parameter list group as the status label corresponding to the associated parameter list group, and regard the historical video having an association relationship with the first central vector corresponding to A i as the associated video corresponding to the target device parameter list group.

[0121] Step S0544 includes: If H e ≥C 0 , then regard the historical video having an association relationship with G e as the associated video corresponding to the target device parameter list group, and regard the specified device parameter list group having an association relationship with the second central vector corresponding to G e as the associated parameter list group corresponding to the target device parameter list group; if H1 < C 0 , H2 < C 0 , ……, H e <C 0 , ……, H f <C0 , then the historical video associated with the first central vector corresponding to the maximum vector similarity among H1, H2, ……, H e , ……, H f is used as the associated video corresponding to the target device parameter list group. The specified device parameter list group associated with the second central vector corresponding to the first central vector is used as the associated parameter list group corresponding to the target device parameter list group. Among them, the status label corresponding to the specified device parameter list group is used as the status label corresponding to its associated parameter list group.

[0122] Through the above steps, the historical video corresponding to the specified device and the specified device parameter list group corresponding to the historical video are also stored in the device database. An association relationship is established between the first central vector and the historical video corresponding to the feature vector in the first cluster corresponding to the first central vector. An association relationship is established between the second central vector and the specified device parameter list group corresponding to the intermediate vector in the second cluster corresponding to the second central vector. This enriches the content of the device database, so that the data set recalled from the device database according to the target video and the target device parameter list group includes: the associated video set corresponding to the target device parameter list group, the associated parameter list group set corresponding to the target device parameter list group, and the status label list corresponding to the associated parameter list group set. Among them, the associated video set includes several associated videos, the associated parameter list group set includes several associated parameter list groups, and the status label list includes the status label corresponding to each associated parameter list group, which can provide more comprehensive data support for the anomaly recognition task and is conducive to improving the accuracy of the obtained anomaly recognition results.

[0123] The present invention also provides a specific embodiment. Among them, the associated data subset corresponding to the target device parameter list group is the data set recalled from the device database according to SP and the target device parameter list group, including: the associated video set corresponding to the target device parameter list group, the associated parameter list group set corresponding to the target device parameter list group, and the status label list corresponding to the associated video set. Among them, the associated video set includes several associated videos, the associated parameter list group set includes several associated parameter list groups, and the status label list includes the status label corresponding to each associated video.

[0124] After step S004, the following step S0041 is also included to construct the device database:

[0125] S0041. Store the historical video corresponding to each specified device and the specified device parameter list group corresponding to the historical video in the device database, and establish an association relationship between the intermediate central vector and the specified device parameter list group corresponding to the feature vector in the intermediate cluster corresponding to the intermediate central vector, and establish an association relationship between the key vector and the historical video corresponding to the key vector.

[0126] Step S0053 includes: If K g ≥C 0 , then take the specified device parameter list group associated with R g as the associated parameter list group corresponding to the target device parameter list group, take the historical video associated with the key vector corresponding to R g as the associated video corresponding to the target device parameter list group, and take the status label corresponding to the historical video as the status label corresponding to its corresponding associated video.

[0127] Step S0055 includes: If L g ≥C 0 , then take the specified device parameter list group associated with R g as the associated parameter list group corresponding to the target device parameter list group, and take the historical video associated with the key vector corresponding to R g as the associated video corresponding to the target device parameter list group; if L1 < C 0 , L2 < C 0 , ……, L g < C 0 , ……, L h < C 0 , then take the specified device parameter list group associated with the intermediate center vector corresponding to the largest second average vector similarity among L1, L2, ……, L g , ……, L h as the associated parameter list group corresponding to the target device parameter list group, and take the historical video associated with the key vector corresponding to the intermediate center vector as the associated video corresponding to the target device parameter list group, where the status label corresponding to the historical video is taken as the status label corresponding to its corresponding associated video.

[0128] Through the above steps, the historical videos corresponding to each designated device and the list group of designated device parameters corresponding to the historical videos are stored in the device database, and an association relationship is established between the intermediate center vector and the list group of designated device parameters corresponding to the feature vectors in the intermediate cluster corresponding to the intermediate center vector, and an association relationship is established between the key vector and the historical video corresponding to the key vector, enriching the content of the device database. So, according to the target video and the list group of target device parameters, the data set recalled from the device database includes: the associated video set corresponding to the list group of target device parameters, the set of associated parameter list groups corresponding to the list group of target device parameters, and the status label list corresponding to the associated video set. Among them, the associated video set includes several associated videos, the set of associated parameter list groups includes several associated parameter list groups, and the status label list includes the status label corresponding to each associated video, which can provide more comprehensive data support for the anomaly recognition task and is beneficial to improving the accuracy of the obtained anomaly recognition results.

[0129] The present invention also provides a specific embodiment, which includes steps S11 - S12 after step S1:

[0130] S11. According to the SP and the preset text corresponding to the target video acquisition device, search in the text database to recall the recall text set corresponding to the SP. The recall text set includes several recall texts. It should be noted that for those skilled in the art, any method of searching in the database to recall information in the prior art belongs to the protection scope of the present invention and will not be elaborated here.

[0131] Specifically, the recall text set corresponding to the SP can be understood as a set composed of all texts in the text database related to the SP and the preset text corresponding to the target video acquisition device.

[0132] Specifically, the preset text corresponding to the target video acquisition device includes preset question prompt texts, such as: "Please detect whether there are anomalies in the devices in the video"; "Please identify whether the operation of each device in the video is normal"; "Please detect whether there are safety hazards in the devices in the video".

[0133] In a specific embodiment, the preset text corresponding to the target video acquisition device further includes texts presenting information related to the video acquisition area of the target video acquisition device, such as: texts presenting the range of the video acquisition area of the target video acquisition device, texts presenting the device names, device attributes, device uses, and device locations within the video acquisition area of the target video acquisition device.

[0134] Specifically, the text database includes relevant texts of each designated device, such as: operation manuals, preset risk handling plans, safety operation steps, fault records, personnel duty schedules, clothing specification manuals, etc.

[0135] S12. Input the preset text corresponding to the target video capture device, the recall text set corresponding to the SP, the feature vector of the SP, the list of feature vectors corresponding to the MB, and the total set of the relevant data sets corresponding to the MB into the preset large language model to obtain the anomaly recognition result corresponding to the SP output by the preset large language model.

[0136] In a specific embodiment, step S12 includes: Input the preset text corresponding to the target video capture device, the recall text set corresponding to the SP, the SP, the MB, and the associated data set corresponding to the MB into the preset large language model to obtain the anomaly recognition result corresponding to the SP output by the preset large language model.

[0137] Through the above steps, according to the target video and the preset text corresponding to the target video capture device, search in the text database to recall the recall text set corresponding to the target video. Input the preset text corresponding to the target video capture device, the recall text set corresponding to the target video, the feature vector of the target video, the list of feature vectors corresponding to the target device parameter list set, and the total set of the relevant data sets corresponding to the target device parameter list set into the preset large language model to obtain the anomaly recognition result. Combining the preset text corresponding to the target video capture device and the recall text set can introduce more domain expertise, help the large language model better understand and apply the background information of a specific domain, provide rich context information for the large language model, and is beneficial to improving the accuracy of the obtained anomaly recognition result.

[0138] The present invention also provides a specific embodiment, which includes the following step S10 after step S005:

[0139] S10. Construct a historical anomaly recognition result data combination corresponding to the SP based on the anomaly recognition result corresponding to the SP, and insert the historical anomaly recognition result data combination corresponding to the SP into the historical anomaly recognition result data set corresponding to the target video capture device corresponding to the SP. Among them, the historical anomaly recognition result data combination includes the anomaly recognition result and the initial device parameter list set corresponding to the anomaly recognition result, and the initial device parameter list set corresponding to the anomaly recognition result is the target device parameter list set used to obtain the anomaly recognition result.

[0140] Specifically, if the anomaly judgment label in the anomaly recognition result in the historical anomaly recognition result data combination is no anomaly, and the time point when the historical anomaly recognition result data combination is inserted into the historical anomaly recognition result data set is the closest to the current time point, then use the initial device parameter list set in the historical anomaly data combination as the key device parameter list set corresponding to its corresponding target video capture device at the current time point.

[0141] Through the above steps, after obtaining the anomaly recognition result, a historical anomaly recognition result data combination is constructed based on the anomaly recognition result, and the historical anomaly recognition result data combination is inserted into the historical anomaly recognition result dataset corresponding to the target video acquisition device. Further, a key device parameter list set corresponding to the target video acquisition device at the current time point is obtained, which can manage and update the historical anomaly recognition result dataset of the device, and dynamically adjust the key device parameter list set based on the latest anomaly-free recognition result, which is helpful for users to manage and query.

[0142] After step S1 and before step S005, the following steps S100 - S300 are included:

[0143] S100. Obtain the data similarity XS between MB and the key device parameter list set corresponding to the target video acquisition device at the current time point. Among them, those skilled in the art know that any method for obtaining the data similarity between two data sets in the prior art belongs to the protection scope of the present invention and will not be elaborated here.

[0144] S200. When XS ≥ XS 0 Execute steps S051 - S054 to obtain the relevant data set corresponding to the target device parameter list group according to the target device parameter list group in SP and MB, the target device corresponding to the target device parameter list group, and the first device database. XS 0 is a preset data similarity threshold. Among them, those skilled in the art know that the preset data similarity threshold is a value less than 1 preset by those skilled in the art according to actual needs. For example: 0.8, 0.58, 0.9, 0.95, and will not be elaborated here.

[0145] Specifically, before step S1, it also includes: executing steps S01 - S04 to construct the first device database.

[0146] Specifically, the greater the data similarity, the more similar MB is to the key device parameter list set corresponding to the target video acquisition device at the current time point.

[0147] S300. When XS < XS 0 Execute steps S0051 - S0055 to obtain the relevant data set corresponding to the target device parameter list group according to the target device parameter list group in SP and MB, the target device corresponding to the target device parameter list group, and the second device database.

[0148] Specifically, before step S1, it also includes: executing steps S001 - S004 to construct the second device database.

[0149] Through the above steps, the first device database and the second device database are constructed, and the data similarity between the target device parameter list set and the key device parameter list set corresponding to the target video capture device at the current time point is obtained. When the data similarity is greater than the preset data similarity threshold, it indicates that the target device parameter list set is very similar to the key device parameter list set corresponding to the target video capture device at the current time point, and there may be no abnormality. Then, execute S051 - S054 to obtain the relevant data set corresponding to the target device parameter list group, and combine the target device parameter list set and the target video to obtain the relevant data set. Otherwise, it indicates that the target device parameter list set is not similar to the key device parameter list set corresponding to the target video capture device at the current time point, and it is impossible to basically determine whether there is an abnormality. At this time, execute S0051 - S0055 to obtain the relevant data set corresponding to the target device parameter list group, mainly relying on the target device parameter list set to obtain the relevant data set, which can flexibly select the method for obtaining the relevant data set, is beneficial to improving the accuracy of obtaining the relevant data set, and avoids unnecessary calculations and resource consumption.

[0150] The present invention also provides a specific embodiment, including steps S1 and steps S101 - S104:

[0151] S101. Input SP and MB into a preset large language model to obtain a target object data set output by the preset large language model. The target object data set includes target object data lists corresponding to several target objects. The target object data list includes the personnel type corresponding to the target object, the device type of the intermediate device corresponding to the target object, the device name of the intermediate device corresponding to the target object, the working state of the intermediate device corresponding to the target object, and the initial intention corresponding to the target object. Among them, the target object is the object in SP, and the intermediate device corresponding to the target object is the target device with the smallest straight-line distance between the target object and the target object in SP. The target object can be understood as the person in SP.

[0152] Specifically, the intention can be understood as the operation name of the operation performed on the device. For example: view the device, open the lid of the device, add materials to the device, perform daily maintenance, check the device status, maintain and record logs, and leave after confirming that the device is fault-free.

[0153] S102. Obtain the preset intention tree corresponding to the device type of the intermediate device corresponding to the target object. The structure of the preset intention tree has 6 layers. The first-layer node represents the device type, the second-layer node represents the device name of the specified device corresponding to the device type, the third-layer node represents the working state of the specified device, the fourth-layer node represents the personnel type related to the working state of the specified device, the fifth-layer node represents the original intention corresponding to the personnel type, and the sixth-layer node represents the target intention corresponding to the original intention. Among them, each fifth-layer node has only one child node; for example: the first-layer node is a temperature sensor, the second-layer nodes are sensor A, sensor B, and sensor C, the third-layer nodes are normal operation, failure, and waiting for repair, the fourth-layer nodes are equipment maintenance personnel and equipment inspection personnel, the fifth-layer nodes are perform daily maintenance and check the device status, and the sixth-layer nodes are complete maintenance and record the log and leave after confirming that the device is fault-free.

[0154] S103. Determine the final intention of the target object according to the target object data list and the nodes in the preset intention tree corresponding to the device type of the intermediate device corresponding to the target object.

[0155] Specifically, step S103 includes the following steps S1031 - S1035:

[0156] S1031. When the device name of the intermediate device corresponding to the target object is the same as the device name represented by the second-layer node in the preset intention tree, use the second-layer node as the second-layer key node.

[0157] S1032. When the working state represented by the child node of the second-layer key node is the same as the working state of the intermediate device corresponding to the target object, use the child node as the third-layer key node.

[0158] S1033. When the personnel type represented by the child node of the third-layer key node is the same as the personnel type corresponding to the target object, use the child node as the fourth-layer key node.

[0159] S1034. When the original intention represented by the child node of the fourth-layer key node is the same as the initial intention corresponding to the target object, use the child node as the fifth-layer key node.

[0160] S1035. Use the target intention represented by the child node of the fifth-layer key node as the final intention of the target object corresponding to the target object data list.

[0161] Through the above steps, the data stored in the target object data list is matched one by one with the nodes in the preset intention tree corresponding to the device type of the intermediate device corresponding to the target object to determine the final intention of the target object, which is beneficial to improving the accuracy of the determined final intention.

[0162] S104. Input the target object data list corresponding to the target object, the final intention of the target object, and the recall information corresponding to the target object into a preset large language model to obtain the anomaly recognition result corresponding to the target object output by the preset large language model.

[0163] In a specific embodiment, step S104 includes: Inputting the SP, the target object data list corresponding to the target object, the final intention of the target object, and the recall information corresponding to the target object into a preset large language model to obtain the anomaly recognition result corresponding to the target object output by the preset large language model.

[0164] Specifically, the anomaly recognition result corresponding to the target object includes normal and abnormal.

[0165] Further, when the anomaly recognition result corresponding to the target object is normal, it indicates that the operation of the target object in the SP is normal.

[0166] Further, when the anomaly recognition result corresponding to the target object is abnormal, it indicates that the operation of the target object in the SP is abnormal and there may be potential safety hazards.

[0167] Specifically, the recall information corresponding to the target object includes video recall information, and the video recall information is the standard operation video searched from the device database according to the target object data list corresponding to the target object and the final intention of the target object. Among them, the device database includes several standard operation videos corresponding to each designated device, and each standard operation video corresponds to a real intention. When the intermediate device corresponding to the target object is the same as the designated device, and the final intention of the target object is the same as the real intention corresponding to the standard operation video of the designated device, the standard operation video is used as the video recall information.

[0168] In a specific embodiment, the recall information corresponding to the target object includes text recall information, and the text recall information is the standard operation manual searched from the text database according to the target object data list corresponding to the target object and the final intention of the target object. Among them, the text database includes several standard operation manuals corresponding to each designated device, and each standard operation manual corresponds to a real intention. When the intermediate device corresponding to the target object is the same as the designated device, and the final intention of the target object is the same as the real intention corresponding to the standard operation manual of the designated device, the standard operation manual is used as the text recall information.

[0169] Through the above steps, the target video and the target device parameter list set are input into a preset large language model to obtain a target object data set. According to the preset intention tree corresponding to the device type of the intermediate device corresponding to the target object, the final intention of the target object is determined. The target object data list corresponding to the target object, the final intention of the target object, and the recall information corresponding to the target object are input into the preset large language model to obtain the abnormal recognition result corresponding to the target object output by the preset large language model. The recall information corresponding to the target object can provide necessary context information or data support for the abnormal recognition task. Moreover, inputting the target object data list corresponding to the target object, the final intention of the target object, and the recall information corresponding to the target object into the preset large language model to obtain the abnormal recognition result comprehensively utilizes multiple data sources, rather than simply judging whether the target object is abnormal based on images or videos to generate the abnormal recognition result, which is beneficial to improving the accuracy of the obtained abnormal recognition result.

[0170] The present invention also provides a specific embodiment. After step S101 and before step S104, the following steps S010 - S020 are included to determine the final intention of the target object:

[0171] S010. Obtain a preset knowledge graph, where the preset knowledge graph includes several triples. The first entity in the triple is the preset device type, the second entity is the target intention, and the relationship is the relevant data combination corresponding to the preset device type and the target intention, including: the device name of the specified device corresponding to the preset device type, the working state of the specified device, the personnel type related to the working state, and the original intention corresponding to the personnel type. For example: the first entity is a temperature sensor, the second entity is to complete maintenance and record logs, and the relationship is the relevant data combination corresponding to the temperature sensor and completing maintenance and recording logs, including sensor A, failure, equipment maintenance personnel, and performing daily maintenance.

[0172] Optionally, the preset device type is the device type obtained by de-duplicating the device types of all specified devices.

[0173] Specifically, in a relevant data combination, the number of the device name, working state, personnel type, and original intention is all 1.

[0174] S020. If in the target object data list, the device type of the intermediate device corresponding to the target object is the same as the first entity in the triple, and the intermediate data combination corresponding to the target object data list is completely consistent with the relationship in the triple, then use the second entity in the triple as the final intention of the target object. Among them, the intermediate data combination corresponding to the target object data list includes: the personnel type corresponding to the target object, the device name of the intermediate device corresponding to the target object, the working state of the intermediate device corresponding to the target object, and the initial intention corresponding to the target object.

[0175] Specifically, the intermediate data combination corresponding to the target object data list is completely consistent with the relationship in the triple. It can be understood that: the personnel type corresponding to the target object in the intermediate data combination is the same as the personnel type related to the working status in the relationship; the device name of the intermediate device corresponding to the target object in the intermediate data combination is the same as the device name of the specified device corresponding to the preset device type in the relationship; the working status of the intermediate device corresponding to the target object in the intermediate data combination is the same as the working status of the specified device in the relationship; the initial intention corresponding to the target object in the intermediate data combination is the same as the original intention corresponding to the personnel type in the relationship.

[0176] Through the above steps, the structured knowledge graph helps to improve the efficiency and accuracy of data analysis. According to the target object data list and the entities and relationships in the preset knowledge graph, the final intention of the target object can be determined, and the exact matching and recognition of the final intention of the target object can be quickly achieved, which can improve the efficiency of determining the final intention of the target object.

[0177] Specifically, the videos (target video, historical video, target sub-video, associated video, standard operation video) in the above embodiments can all be replaced by their corresponding image groups, and each image group includes one frame image of each second of its corresponding video.

[0178] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store a computer program related to a method in the method embodiment. The computer program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0179] An embodiment of the present invention also provides an electronic device, including: a processor, a memory, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method provided in the above embodiment is implemented.

[0180] An embodiment of the present invention also provides a computer program product, which includes program code. When the program product runs on an electronic device, the program code is used to cause the electronic device to execute the steps in the method according to various exemplary embodiments of the present invention described above in this specification.

[0181] The present invention provides an abnormal recognition method, an electronic device and a storage medium driven by a target video. The method can obtain a plurality of historical videos corresponding to a specified device and a list group of specified device parameters corresponding to each historical video, construct a device database based on the feature vectors of the list group of specified device parameters, the feature vectors of the historical videos corresponding to the list group of specified device parameters, the status labels corresponding to the historical videos and the list of abnormal parameter names corresponding to the historical videos, and input the feature vector of the target video, the list of feature vectors corresponding to the target device parameter list set, and the total set of relevant data sets corresponding to the target device parameter list set retrieved in the device database into a preset large prediction model to obtain an abnormal recognition result. It can be seen that in the present invention, the device database is constructed based on the feature vectors of the list group of specified device parameters, the feature vectors of the historical videos corresponding to the list group of specified device parameters, the status labels corresponding to the historical videos and the list of abnormal parameter names corresponding to the historical videos, which contains rich domain knowledge. The total set of relevant data sets corresponding to the target device parameter list set is a set retrieved from the device database, which can provide necessary data support for the abnormal recognition task. Inputting the feature vector of the target video, the list of feature vectors corresponding to the target device parameter list set, and the total set of relevant data sets corresponding to the target device parameter list set into the preset large prediction model to obtain an abnormal recognition result comprehensively utilizes multiple data sources, rather than simply judging whether there is an abnormality in a region based on an image or a video to generate an abnormal recognition result, which is beneficial to improving the accuracy of the obtained abnormal recognition result.

[0182] Although some specific embodiments of the present invention have been described in detail by way of examples, those skilled in the art should understand that the above examples are only for illustration and not for limiting the scope of the present invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A target video driven anomaly recognition method, characterized in that: The method comprises the following steps: S001. For each designated device, a plurality of historical videos corresponding to the designated device and a designated device parameter list group corresponding to each historical video are obtained; S002, clustering the feature vectors of all designated device parameter list groups corresponding to the designated device to obtain several intermediate clusters corresponding to the designated device, and taking the vector corresponding to the center position of the intermediate cluster as the intermediate center vector corresponding to the designated device; S003. If the feature vector of the specified device parameter list group is in the intermediate cluster corresponding to the intermediate center vector, the feature vector of the historical video corresponding to the specified device parameter list group is used as the key vector corresponding to the intermediate center vector, the state label corresponding to the historical video is used as the state label corresponding to the key vector, and the abnormal parameter name list corresponding to the historical video is used as the abnormal parameter name list corresponding to the key vector; S004. Store the intermediate center vector corresponding to the designated device, the key vector corresponding to the intermediate center vector, the state label corresponding to the key vector, and the abnormal parameter name list corresponding to the key vector in the device database; S005. Input the feature vector of SP, the feature vector list corresponding to MB, and the total set of related data sets corresponding to MB into the preset large language model to obtain the abnormality recognition result corresponding to SP output by the preset large language model, wherein recall is performed in the device database according to SP and MB to obtain the total set of related data sets corresponding to MB, SP is the target video, and MB is the target device parameter list set corresponding to SP.

2. The target video driven abnormality recognition method according to claim 1, characterized in that: Before step S005, the method further includes step S1: S1. Whenever a preset duration is reached, the video captured by the target video capture device within the preset duration is used as SP and MB is obtained. MB includes a target device parameter list group corresponding to several target devices. The target device parameter list group includes a target device parameter list corresponding to each parameter of the target device. The target device parameter list includes specific parameter values ​​of the parameters within each second of SP.

3. The target video driven abnormality identification method according to claim 2, characterized in that: The duration of the historical video is the same as that of the target video.

4. The target video driven abnormality identification method according to claim 2, characterized in that: The feature vector list corresponding to MB includes the feature vector of each target device parameter list group in MB. The feature vector of the target device parameter list group is a vector obtained by concatenating the feature vectors of each target device parameter list according to the order of the target parameter lists in the target device parameter list group, wherein the vector values ​​in the feature vectors of the target device parameter list correspond one-to-one to the parameter values ​​in the target device parameter list.

5. The target video driven abnormality recognition method according to claim 4, characterized in that: The total set of related data sets corresponding to MB includes related data sets corresponding to each target device parameter list group in MB, and the related data sets corresponding to the target device parameter list group include: a related video feature vector list corresponding to the target device parameter list group, a related device feature vector list corresponding to the target device parameter list group, and a status label list corresponding to the related video feature list, wherein the related video feature vector list includes several related video feature vectors, the related device feature vector list includes several related device feature vectors, and the status label list includes a status label corresponding to each related video feature vector.

6. The target video driven abnormality identification method according to claim 5, characterized in that: In step S005, the total set of related data sets corresponding to MB is obtained by recalling the device database according to SP and MB, and the related data sets corresponding to the target device parameter list group are obtained according to SP, the target device parameter list group in MB, the target device corresponding to the target device parameter list group, and the device database, including the following steps S0051-S0055: S0051. Obtain an intermediate center vector list R={R1, R2, . . . , R g , ..., R h} , R g is the g-th intermediate center vector corresponding to the specified device that is the same device as the target device, where g ranges from 1 to h, and h is the number of intermediate center vectors corresponding to the specified device that is the same device as the target device; S0052. Obtain B and R g The first average vector similarity K between g , where B is the characteristic vector of the target device parameter list group; S0053, if K g ≥C 0 , then R g As the relevant device feature vector corresponding to the target device parameter list group, R g The corresponding key vector is used as the relevant video feature vector corresponding to the target device parameter list group, and the state label corresponding to the key vector is used as the state label corresponding to the relevant video feature vector, where C 0 is the preset vector similarity threshold; S0054, if K1<C 0 , K2<C 0 , ..., K g <C 0 , ..., K h <C 0 , then based on R g The corresponding key vector corresponding to the abnormal parameter name list obtains B and R g The second average vector similarity L between g ; S0055, if L g ≥C 0 , then R g As the relevant device feature vector corresponding to the target device parameter list group, R g The corresponding key vector is used as the relevant video feature vector corresponding to the target device parameter list group. If L1<C 0 , L2<C 0 , ..., L g <C 0 , ..., L h <C 0 , then L1, L2, ..., L g , ..., L h The intermediate center vector corresponding to the largest second average vector similarity is used as the relevant device feature vector corresponding to the target device parameter list group, and the key vector corresponding to the intermediate center vector is used as the relevant video feature vector corresponding to the target device parameter list group, wherein the state label corresponding to the key vector is used as the state label corresponding to the corresponding relevant video feature vector.

7. The target video driven abnormality recognition method according to claim 6, characterized in that: Step S0052 includes the following steps ac: a. Decompose B to obtain the first sub-vector list D = {D1, D2, ..., D j , ..., D n }, D j is the first subvector corresponding to the jth parameter of the target device in B, where j ranges from 1 to n, and n is the number of parameters of the target device; b. R g Decompose to obtain a list of intermediate subvectors Q g = {Q g1 , Q g2 , ..., Q gj , ..., Q gn }, Q gj For R g The intermediate sub-vector corresponding to the jth parameter of the target device in; c. According to D j and Q gj Get K g , where K g Meet the following conditions: K g =∑ n j＝1 U gj / n,U gj D j With Q gj The vector similarity between .

8. The target video driven abnormality recognition method according to claim 7, characterized in that: In step S0054, based on R g The corresponding key vector corresponding to the abnormal parameter name list obtains B and R g The second average vector similarity L between g The method comprises the following steps S0061-0065: S0061. Get R g The corresponding key vector corresponds to the abnormal parameter name list set M g ={M g1 , M g2 , ..., M gr , ..., M gs(g) },M gr For R g The list of abnormal parameter names corresponding to the rth key vector. The value of r ranges from 1 to s(g), where s(g) is R g The number of corresponding key vectors; S0062, M g1 , M g2 , ..., M gr , ..., M gs(g) The union of R g Corresponding important parameter name list N g = {N g1 , N g2 , ..., N gk , ..., N gt(g) },N gk For R g The corresponding kth important parameter name, k ranges from 1 to t(g), t(g) is R g The number of corresponding important parameter names; S0063, according to N gk Get the jth parameter of the target device in R g The first importance weight P in the corresponding intermediate cluster jg , P jg Meet the following conditions: When the parameter name of the jth parameter of the target device matches N gk If the same, let P jg =1+N 0 gk , N 0 gk M g1 , M g2 , ..., M gr , ..., M gs(g) Including N gk The number of exception parameter name lists with the same exception parameter name; when the parameter name of the jth parameter of the target device is the same as N g1 , N g2 , ..., N gk , ..., N gt(g) If they are not the same, let P jg =1; S0064, P 1g , P 2g , ..., P jg , ..., P ng Perform normalization to obtain P jg The corresponding normalized value, and P jg The corresponding normalized value is taken as the jth parameter of the target device in R g The corresponding second importance weight W in the corresponding middle cluster jg ; S0065, according to D j , W jg and Q gj Get L g , L g Meet the following conditions: L g =∑ n j＝1 (W jg ×U gj ) / ∑ n j＝1 W jg 。 9. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the target video driven anomaly identification method as described in any one of claims 1-8.

10. An electronic device, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for identifying anomalies of a target video drive as described in any one of claims 1 to 8 is implemented.

Images