Automatic semantic description method for action video data of equipment-associated electric power operating personnel

Abstract

The invention discloses an automatic semantic description method for action video data of equipment-associated electric power operating personnel, and belongs to the technical field of action description of the electric power operating personnel. The method solves the problems that an existing semantic description method is low in accuracy, cannot cover all key actions of power operators and needs to limit the action rhythm of the operators, so that the operation efficiency is low. According to the method, training data are divided according to equipment in a training stage, a model is trained for each type of equipment, each model participates in semantic output in a decision-making stage, all semantics are screened based on equipment association description center vectors and distance weighting, and the semantic output is improved. And finally, an automatic semantic description result of the action video data of the electric power operating personnel is obtained, semantic description requirements of all key actions of the electric power operating personnel can be covered, and the method is suitable for semantic description of all actions. The method can be applied to automatic semantic description of the action video data of the electric power operating personnel.

Description

Technical Field

[0001] The present invention belongs to the technical field of power operator action description, and in particular relates to an automatic semantic description method for power operator action video data based on device association. Background Art

[0002] With the rapid development of my country's power industry, the number of power equipment has gradually increased, and the maintenance and management tasks of power equipment have gradually increased. Power operations are dangerous, and it is necessary to record the operator's action video data during the operation. The traditional method is to directly record the operator's video data, which has a large amount of data to be recorded, is not easy to retrieve, and is not easy to find illegal and unsafe parts. Converting the power operator's action video data into text semantic description can, on the one hand, reduce the difficulty of searching and finding violations, and reduce the storage burden. On the other hand, due to the introduction of semantics, it can be further connected with modern multimodal models and large language models to form a more in-depth artificial intelligence decision support function, significantly improving the management efficiency of power operations. Therefore, it is of great significance to carry out automated semantic description of power operator action video data.

[0003] The main technical approaches currently used for the task of automatically describing the semantics of video data of power workers' actions are:

[0004] (1) Strictly define the operation steps and operation processes for a device. The operator's actions need to match each process, and the degree of matching or mismatching for each step is recorded in text form. This model is easy to implement, but the operation is relatively rigid. In actual work, the camera and equipment positions have an impact on this matching. In addition, the strict restrictions on the operator's action rhythm lead to low operating efficiency. Especially in a computer room with a large number of devices, this technology usually causes confusion and is difficult to implement.

[0005] (2) Introducing some large-scale models for action video description, such as Tarsier, ActionCLIP, and CLIP4Clip. These models are characterized by flexible operation and rich output content. However, the normal operation process on some devices may be a very dangerous and illegal process on other devices. This will cause the training data for building these models to have extremely conflicting descriptions, resulting in model training failure. At the same time, when facing multiple devices, simply measuring distance and constructing output content based on attributes will also cause these models to tend to the most obvious and highest matching output. These high matching degrees may be simply due to the operator's angle, the color of the work clothes worn, and the training data being similar, resulting in serious description omissions, and ultimately resulting in the output semantic description failing to cover the key action content of the power operator, thus losing practical application value.

[0006] In summary, in order to solve the problems of low accuracy of existing semantic description methods, inability to cover all key actions of power workers, and low operating efficiency caused by the need to limit the rhythm of operators' actions, it is necessary to propose a comprehensive, easier to implement, and practical application value method to automatically associate the semantic description results with the matching output of the power workers' actions. Summary of the Invention

[0007] The purpose of the present invention is to solve the problems of low accuracy of existing semantic description methods, inability to cover all key actions of power workers, and low operating efficiency caused by the need to limit the operator's action rhythm, and to propose an automated semantic description method for power worker action video data based on device association.

[0008] The technical solution adopted by the present invention to solve the above technical problems is: a method for automatically describing the semantics of equipment-related power worker action video data, the method specifically comprising the following steps:

[0009] Step S1: Inputting a semantic description dataset ZYDB of video data of electric power workers' actions, obtaining a list ZYDBSBNameList of equipment names operated by electric power workers in ZYDB, and obtaining the number ZYDBSBNum of elements in the list ZYDBSBNameList of equipment names operated by electric power workers;

[0010] Obtain the first list ModelList1 of textual description models of power operation personnel action videos associated with the equipment; obtain the second list ModelList2 of textual description models of power operation personnel action videos associated with the equipment;

[0011] Step S2: establishing a vectorization module WBSLH for integrating the name of the power operation equipment and the action text. The input of the module WBSLH is the name of the equipment to be vectored WBSLHInput1 and the action text to be vectored WBSLHInput2. The output of the module WBSLH is the vectorization result WBSLHOutput.

[0012] Step S3: Use module WBSLH to create a device association description center vector list CenterList;

[0013] Step S4: Input a video data TestData of an operator's action, and use the list ModelList1 and the list ModelList2 to convert the input video data TestData into a device distance weighted candidate text description list BXTextList;

[0014] Step S5: Complete the output of the automated semantic description result according to the device association description center vector list CenterList and the device distance weighted candidate text description list BXTextList.

[0015] Furthermore, the specific process of step S1 is as follows:

[0016] Step S101: Input the semantic description dataset ZYDB of the power worker action video data. The dataset ZYDB contains the following fields:

[0017] DBSP: video data of power workers’ actions;

[0018] DBSB: the name of the operated power device, which is represented by a character string;

[0019] DBDSCSS: Description of the upper body movements of an electric worker when operating electric equipment. The description of the upper body movements is a string.

[0020] DBDSCXS: Description of the lower body movements of an electric worker when operating electric equipment. The description of the lower body movements is a string.

[0021] Step S102: Create a list ZYDBSBNameList of device names operated by power workers in the data set ZYDB, where each element in the list ZYDBSBNameList corresponds to a character string;

[0022] Step S103: Add all DBSBs in the data set ZYDB to ZYDBSBNameList;

[0023] Step S104: Delete duplicate elements in the list ZYDBSBNameList, and keep only one of each element;

[0024] Step S105: Obtain the number of elements in the list of names of devices operated by power workers in the data set ZYDB: ZYDBSBNum = the number of elements in ZYDBSBNameList;

[0025] Step S106: Create a first list ModelList1 of textual description models of power worker action videos associated with equipment, and initialize the number of elements in the list ModelList1 to 0;

[0026] Create the second list ModelList2 of the textual description model of the power worker action video associated with the equipment, and initialize the number of elements in the list ModelList2t to 0;

[0027] Step S107: establish the action video text description model construction process counter InitCounter=1;

[0028] Step S108, in the process of constructing the action video text description model, the first temporary variable InitTemp1 = the InitCounterth element of the list ZYDBSBNameList;

[0029] Step S109, upper body description model VideoBERTSS = trained first ActionCLIP model;

[0030] Select all data in the dataset ZYDB whose DBSB field is equal to InitTemp1, use the DBSP field in the selected data as the input of the first ActionCLIP model, use the DBDSCSS field in the selected data as the output of the first ActionCLIP model, and use the selected data to train the first ActionCLIP model;

[0031] Step S110, lower body description model VideoBERTXS = trained second ActionCLIP model;

[0032] Select all data in the dataset ZYDB whose DBSB field is equal to InitTemp1, use the DBSP field in the selected data as the input of the second ActionCLIP model, use the DBDSCXS field in the selected data as the output of the second ActionCLIP model, and use the selected data to train the second ActionCLIP model;

[0033] Step S111, add VideoBERTSS to the list ModelList1; add VideoBERTXS to the list ModelLis2t;

[0034] Step S112, set InitCounter=InitCounter+1;

[0035] Step S113: If InitCounter is less than or equal to ZYDBSBNum, go to step S108; otherwise, go to step S114;

[0036] Step S114, step S1 ends.

[0037] Furthermore, the specific process of step S2 is as follows:

[0038] Step S201: establishing a vectorization module WBSLH for integrating the name of the power operation equipment and the action text. The input of the module WBSLH is the name of the equipment to be vectorized WBSLHInput1 and the action text to be vectorized WBSLHInput2.

[0039] Step S202, WBSLHPos=finds the position of WBSLHInput1 in ZYDBSBNameList;

[0040] Step S203: Create the first temporary vector WBSLHVector1 of the vectorization module = create a vector of dimension ZYDBSBNum, with all contents of the vector WBSLHVector1 being zeros;

[0041] Step S204, set the value of the WBSLHPos th element of the vector WBSLHVector1 to 0.1;

[0042] Step S205: Create the second temporary vector WBSLHVector2 of the vectorization module = convert WBSLHInput2 into a vector using the bert-base-chinese model;

[0043] Step S206, creating a vectorization result WBSLHOutput = merging WBSLHVector1 and WBSLHVector2 into one vector;

[0044] Step S207: Output WBSLHOutput.

[0045] Furthermore, the specific process of step S3 is as follows:

[0046] Step S301: Create a device association description center vector list CenterList. Each element of CenterList contains four fields:

[0047] CenterVector1: the first center vector;

[0048] Centerstd1: standard deviation of the first center vector;

[0049] CenterVector2: the second center vector;

[0050] Centerstd2: standard deviation of the second center vector;

[0051] Step S302, device association center vector construction counter SBCenterCounter=1;

[0052] Step S303: The device association center vector constructs a first temporary variable SBCenterTemp1 = the SBCenterCounterth element of the list ZYDBSBNameList;

[0053] Step S304: Select all elements in the data set ZYDB whose DBSB field is equal to SBCenterTemp1; use the module WBSLH to process the selected elements, that is, the input WBSLHInput1 of the module WBSLH is equal to SBCenterTemp1, and the input WBSLHInput2 of the module WBSLH is the DBDSCSS of each selected element;

[0054] Device Association Description Vector List 1 SBCenterVector1 = the output of the selected DBDSCSS of each element through the module WBSLH;

[0055] Step S305: Select all elements in the data set ZYDB whose DBSB field is equal to SBCenterTemp1; use the module WBSLH to process the selected elements, that is, the input WBSLHInput1 of the module WBSLH is equal to SBCenterTemp1, and the input WBSLHInput2 of the module WBSLH is the DBDSCXS of each selected element;

[0056] The second list of device association description vectors SBCenterVector2 = the output of each selected element DBDSCXS through the module WBSLH;

[0057] Step S306: Create a new element CenterListNew for the list CenterList;

[0058] CenterListNew's CenterVector1 is equal to the dimension-by-dimension mean of all vectors in the list SBCenterVector1, and CenterListNew's CenterVector2 is equal to the dimension-by-dimension mean of all vectors in the list SBCenterVector2.

[0059] Step S307: Calculate the Cartesian distance between each vector in the list SBCenterVector1 and CenterVector1, and then calculate the standard deviation of all Cartesian distance values;

[0060] CenterListNew's Centerstd1 field = calculated standard deviation;

[0061] Step S308: Calculate the Cartesian distance between each vector in the list SBCenterVector2 and CenterVector2, and then calculate the standard deviation of all Cartesian distance values;

[0062] CenterListNew's Centerstd2 field = calculated standard deviation;

[0063] Step S309: Set SBCenterCounter=SBCenterCounter+1;

[0064] Step S310: If SBCenterCounter is less than or equal to ZYDBSBNum, go to step S303; otherwise, go to step S311;

[0065] Step S311 and step S3 end.

[0066] Furthermore, the specific process of step S4 is as follows:

[0067] Step S401: inputting a video data TestData of an operator's action;

[0068] Step S402: Create a device distance weighted candidate text description list BXTextList, which contains the following fields:

[0069] BXText1: Descriptive text information for the operator's upper body movements;

[0070] BXText2: Descriptive text information for the operator's lower body movements;

[0071] BXVecter1: vector data describing the worker's upper body movements;

[0072] BXVecter2: Vector data describing the lower body movements of workers;

[0073] Distance1: distance description variable of upper body movement;

[0074] Distance2: distance description variable of lower body movement;

[0075] Distance: a variable describing the comprehensive distance between people;

[0076] JQ: device distance weighted value;

[0077] Step S403: Create a counter BXCounter=1 for the candidate text description list;

[0078] Step S404: Create the first temporary variable BXTemp1 in the candidate text description list. Use the BXCounterth model in the list ModelList1 to process the action video data TestData to obtain the text description of the action video data TestData.

[0079] Step S405 , the candidate text description list creates a second temporary variable BXTemp2 = using the BXCounter th model in the list ModelList2 to process the action video data TestData, and obtain the text description of the action video data TestData;

[0080] Step S406: Create a third temporary variable BXTemp3 in the candidate text description list = BXCounter th element of ZYDBSBNameList;

[0081] Step S407: Create a fourth temporary variable BXTemp4 in the candidate text description list. Process the variable using the module WBSLH. The WBSLHInput1 and WBSLHInput2 of the module WBSLH are equal to BXTemp3 and BXTemp1, respectively.

[0082] Step S408: Create the fifth temporary variable BXTemp5 in the candidate text description list. Process the variable using the module WBSLH. The WBSLHInput1 of the module WBSLH is equal to BXTemp3, and the WBSLHInput2 is equal to BXTemp2.

[0083] Step S409: Use the Yolo neural network to identify whether there is a device named BXTemp3 in the action video data TestData;

[0084] If there is only one device named BXTemp3 in the action video data TestData, the alternative text description list creates a sixth temporary variable BXTemp6 = obtains the distance between the position of the device named BXTemp3 in the action video data TestData and the position of the operator;

[0085] If there are multiple devices named BXTemp3 in the action video data TestData, the alternative text description list creates a sixth temporary variable BXTemp6 = the minimum distance between the location of each device named BXTemp3 and the location of the operator;

[0086] If there is no device named BXTemp3 in the action video data TestData, the alternative text description list creates a sixth temporary variable BXTemp6 = the width of the action video data TestData × 10;

[0087] Step S410, updating the sixth temporary variable BXTemp6'=BXTemp6 / W, where W represents the width of the action video data TestData;

[0088] Step S411: Add a new element to the list BXTextList. The BXText1 field of this element is BXTemp1, the BXText2 field is BXTemp2, the BXVecter1 field is BXTemp4, the BXVecter2 field is BXTemp5, the Distance1 field is 0, the Distance2 field is 0, the Distance field is 0, and the JQ field is BXTemp6'.

[0089] Step S412: Set BXCounter=BXCounter+1;

[0090] Step S413: If BXCounter is less than or equal to ZYDBSBNum, go to step S404; otherwise, go to step S414.

[0091] Step S414 and step S4 end.

[0092] Furthermore, the specific process of step S5 is as follows:

[0093] Step S501, automation description counter MSCounter=1;

[0094] Step S502 , automatically describe the first temporary variable MSTemp1 = BXVecter1 of the MSCounterth element in the list BXTextList;

[0095] Step S503 , automatically describe the second temporary variable MSTemp2 = BXVecter2 of the MSCounterth element in the list BXTextList;

[0096] Step S504, automatically describe the third temporary variable MSTemp3 = JQ of the MSCounterth element in the list BXTextList;

[0097] Step S505: Set the value of the MSCounter dimension of MSTemp1 to JQ;

[0098] Step S506: Set the value of the MSCounter dimension of MSTemp2 to JQ;

[0099] Step S507 , automatically describe the fourth temporary variable MSTemp4 = calculate the Cartesian distance between MSTemp1 and CenterVector1 of the MSCounter th element in the list CenterList;

[0100] Step S508 , automatically describe the fifth temporary variable MSTemp5 = calculate the Cartesian distance between MSTemp2 and the CenterVector2 of the MSCounter th element in the list CenterList;

[0101] Step S509, automatically describe the sixth temporary variable MSTemp6 = Centerstd1 of the MSCounterth element in the list CenterList;

[0102] Step S510, automatically describe the seventh temporary variable MSTemp7 = Centerstd2 of the MSCounterth element in the list CenterList;

[0103] Step S511, update MSTemp4'=(MSTemp4-2×MSTemp6) / (4×MSTemp6);

[0104] Step S512, update MSTemp5'=(MSTemp5-2×MSTemp7) / (4×MSTemp7);

[0105] Step S513: Set the Distance1 field of the list BXTextList to MSTemp4', the Distance2 field to MSTemp5', and the Distance field to (0.7*MSTemp4'+0.3*MSTemp5');

[0106] Step S514: Set MSCounter=MSCounter+1;

[0107] Step S515: If MSCounter is less than or equal to ZYDBSBNum, go to step S502; otherwise, go to step S516.

[0108] Step S516: Create a text description list SSSXList. SSSXList is a list of character strings and initially contains only 0 elements.

[0109] Create a device description list SSSBList, which is a list of strings and initially contains only 0 elements;

[0110] Step S517, the first temporary variable SX1 used for screening = find the element position with the smallest value of Distance1 in the list BXTextList;

[0111] Step S518: Add the element at the SX1th position of ZYDBSBNameList to SSSBList, add BXText1 of the element at the SX1th position of BXTextList to SSSXList, and add BXText2 of the element at the SX1th position of BXTextList to SSSXList;

[0112] Step S519, the second temporary variable SX2 used for screening = find the element position with the smallest Distance2 value in the list BXTextList;

[0113] Step S520: Add the element at the SX2th position of ZYDBSBNameList to SSSBList, add BXText1 of the element at the SX2th position of BXTextList to SSSXList, and add BXText2 of the element at the SX2th position of BXTextList to SSSXList;

[0114] Step S521, the third temporary variable SX3 used for screening = find the element position with the smallest Distance value in the list BXTextList;

[0115] Step S522: add the element at the SX3th position of ZYDBSBNameList to SSSBList, add BXText1 of the element at the SX3th position of BXTextList to SSSXList, and add BXText2 of the element at the SX3th position of BXTextList to SSSXList;

[0116] Step S523: Delete all duplicate elements in SSSBList, that is, only keep one of each element;

[0117] Step S524: Delete all duplicate elements in SSSXList, that is, only keep one of each element;

[0118] Step S525, the fourth temporary variable SX4 for screening = synthesizes all elements in SSSBList into a string, with each element separated by ",";

[0119] Step S526, the fifth temporary variable SX5 for screening = synthesizes all elements in SSSXList into a string, with each element separated by ",";

[0120] Step S527, final result string ResultString = the combined result of the strings "for device:" SX4 and "operator performed:" SX5;

[0121] Step S528: Output the final result string ResultString as the automatic semantic description result of the action video data TestData, and the entire algorithm ends.

[0122] The beneficial effects of the present invention are:

[0123] The present invention proposes a method for automated semantic description of video data of power worker actions based on device association. During the training phase, the method divides the training data according to the device and trains a model for each device. During the decision phase, each model participates in semantic output and all semantics are screened based on the device association description center vector and distance weighting. Ultimately, the automated semantic description results of the power worker action video data are obtained. This method solves the low accuracy problem of traditional semantic description methods and can cover the semantic description requirements of all key actions of power workers, and is applicable to the semantic description of all actions. At the same time, the automated semantic description of power worker action video data can be achieved without restricting the operator's action rhythm, avoiding affecting operational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] Figure 1 It is a flow chart of the method for automatic semantic description of equipment-related power worker action video data of the present invention. DETAILED DESCRIPTION

[0125] Specific implementation method 1: Combination Figure 1 This embodiment describes a method for automatically describing the semantics of video data of electric power workers' actions based on equipment association, and the method specifically includes the following steps:

[0126] Step S1: Inputting a semantic description dataset ZYDB of video data of electric power workers' actions, obtaining a list ZYDBSBNameList of equipment names operated by electric power workers in ZYDB, and obtaining the number ZYDBSBNum of elements in the list ZYDBSBNameList of equipment names operated by electric power workers;

[0127] Obtain the first list ModelList1 of textual description models of power operation personnel action videos associated with the equipment; obtain the second list ModelList2 of textual description models of power operation personnel action videos associated with the equipment;

[0128] Step S2: establishing a vectorization module WBSLH for integrating the name of the power operation equipment and the action text. The input of the module WBSLH is the name of the equipment to be vectored WBSLHInput1 and the action text to be vectored WBSLHInput2. The output of the module WBSLH is the vectorization result WBSLHOutput.

[0129] Step S3: Use module WBSLH to create a device association description center vector list CenterList;

[0130] Step S4: Input a video data TestData of an operator's action, and use the list ModelList1 and the list ModelList2 to convert the input video data TestData into a device distance weighted candidate text description list BXTextList;

[0131] Step S5: Complete the output of the automated semantic description result according to the device association description center vector list CenterList and the device distance weighted candidate text description list BXTextList.

[0132] The method of the present invention is equivalent to purifying samples and outputs in the training and prediction stages, significantly reducing the difficulty of constructing semantic samples of power operator actions and reducing conflicts, avoiding the use of a single model for prediction, and effectively preventing some conditions with high matching degrees (such as the operator's clothing being consistent with the sample) from masking the true action description; by introducing the device-associated description center vector, the description content is measured at the dual levels of semantic matching and device matching, while the device distance weighting takes into account the spatial relationship between the device and the operator in the video data, making the method suitable for real production environments with a large number of devices and non-fixed shooting angles, making the method of the present invention a more comprehensive method that is easier to implement and has practical application value.

[0133] Specific embodiment 2: This embodiment differs from specific embodiment 1 in that the specific process of step S1 is as follows:

[0134] Step S101: Input the semantic description dataset ZYDB of the power worker action video data. The dataset ZYDB contains the following fields:

[0135] DBSP: video data of power workers' actions. Each piece of action video data contains 1 to 30 seconds of video content of power workers operating power equipment.

[0136] DBSB: the name of the operated power device, which is represented by a character string;

[0137] DBDSCSS: Description of the upper body movements of power workers when operating power equipment. The description of the upper body movements is a string, which describes the upper body movements of power workers in one sentence.

[0138] DBDSCXS: Description of the lower body movements of power workers when operating power equipment. The description of the lower body movements is a string, which describes the lower body movements of power workers in one sentence.

[0139] Step S102: Create a list ZYDBSBNameList of device names operated by power workers in the data set ZYDB, where each element in the list ZYDBSBNameList corresponds to a character string;

[0140] Step S103: Add all DBSBs in the data set ZYDB to ZYDBSBNameList;

[0141] Step S104: Delete duplicate elements in the list ZYDBSBNameList, and keep only one of each element;

[0142] Step S105: Obtain the number of elements in the list of names of devices operated by power workers in the data set ZYDB: ZYDBSBNum = the number of elements in ZYDBSBNameList;

[0143] Step S106: Create a first list ModelList1 of textual description models of power worker action videos associated with equipment, and initialize the number of elements in the list ModelList1 to 0;

[0144] Create the second list ModelList2 of the textual description model of the power worker action video associated with the equipment, and initialize the number of elements in the list ModelList2t to 0;

[0145] Step S107: establish the action video text description model construction process counter InitCounter=1;

[0146] Step S108, in the process of constructing the action video text description model, the first temporary variable InitTemp1 = the InitCounterth element of the list ZYDBSBNameList;

[0147] Step S109, upper body description model VideoBERTSS = trained first ActionCLIP model;

[0148] Select all data in the dataset ZYDB whose DBSB field is equal to InitTemp1, use the DBSP field in the selected data as the input of the first ActionCLIP model, use the DBDSCSS field in the selected data as the output of the first ActionCLIP model, and use the selected data to train the first ActionCLIP model;

[0149] Step S110, lower body description model VideoBERTXS = trained second ActionCLIP model;

[0150] Select all data in the dataset ZYDB whose DBSB field is equal to InitTemp1, use the DBSP field in the selected data as the input of the second ActionCLIP model, use the DBDSCXS field in the selected data as the output of the second ActionCLIP model, and use the selected data to train the second ActionCLIP model;

[0151] Step S111, add VideoBERTSS to the list ModelList1; add VideoBERTXS to the list ModelLis2t;

[0152] Step S112, set InitCounter=InitCounter+1;

[0153] Step S113: If InitCounter is less than or equal to ZYDBSBNum, go to step S108; otherwise, go to step S114;

[0154] Step S114, step S1 ends.

[0155] Other steps and parameters are the same as those in the first embodiment.

[0156] Specific embodiment three: This embodiment differs from specific embodiment one or two in that the specific process of step S2 is as follows:

[0157] Step S201: establishing a vectorization module WBSLH for integrating the name of the power operation equipment and the action text. The input of the module WBSLH is the name of the equipment to be vectorized WBSLHInput1 and the action text to be vectorized WBSLHInput2.

[0158] Step S202, WBSLHPos=finds the position of WBSLHInput1 in ZYDBSBNameList;

[0159] Step S203: Create the first temporary vector WBSLHVector1 of the vectorization module = create a vector of dimension ZYDBSBNum, with all contents of the vector WBSLHVector1 being zeros;

[0160] Step S204, set the value of the WBSLHPos th element of the vector WBSLHVector1 to 0.1;

[0161] Step S205: Create the second temporary vector WBSLHVector2 of the vectorization module = use the bert-base-chinese model (the model can be directly downloaded on HuggingFace) to convert WBSLHInput2 into a vector (WBSLHInput2 here is the action text to be vectorized corresponding to the WBSLHPos element in the list ZYDBSBNameList);

[0162] Step S206, creating a vectorization result WBSLHOutput = merging WBSLHVector1 and WBSLHVector2 into one vector;

[0163] Step S207: Output WBSLHOutput.

[0164] Other steps and parameters are the same as those in the first or second embodiment.

[0165] Specific embodiment 4: This embodiment differs from any one of specific embodiments 1 to 3 in that the specific process of step S3 is as follows:

[0166] Step S301: Create a device association description center vector list CenterList. Each element of CenterList contains four fields:

[0167] CenterVector1: the first center vector;

[0168] Centerstd1: standard deviation of the first center vector;

[0169] CenterVector2: the second center vector;

[0170] Centerstd2: standard deviation of the second center vector;

[0171] Step S302, device association center vector construction counter SBCenterCounter=1;

[0172] Step S303: The device association center vector constructs a first temporary variable SBCenterTemp1 = the SBCenterCounterth element of the list ZYDBSBNameList;

[0173] Step S304: Select all elements in the data set ZYDB whose DBSB field is equal to SBCenterTemp1; use the module WBSLH to process the selected elements, that is, the input WBSLHInput1 of the module WBSLH is equal to SBCenterTemp1, and the input WBSLHInput2 of the module WBSLH is the DBDSCSS of each selected element;

[0174] Device Association Description Vector List 1 SBCenterVector1 = the output of the selected DBDSCSS of each element through the module WBSLH;

[0175] Step S305: Select all elements in the data set ZYDB whose DBSB field is equal to SBCenterTemp1; use the module WBSLH to process the selected elements, that is, the input WBSLHInput1 of the module WBSLH is equal to SBCenterTemp1, and the input WBSLHInput2 of the module WBSLH is the DBDSCXS of each selected element;

[0176] The second list of device association description vectors SBCenterVector2 = the output of each selected element DBDSCXS through the module WBSLH;

[0177] Step S306: Create a new element CenterListNew for the list CenterList;

[0178] CenterListNew's CenterVector1 is equal to the dimension-by-dimension mean of all vectors in the list SBCenterVector1 (that is, each element in the list ZYDBSBNameList corresponds to multiple action videos, and the vectors corresponding to each action video are averaged). CenterListNew's CenterVector2 is equal to the dimension-by-dimension mean of all vectors in the list SBCenterVector2.

[0179] Step S307: Calculate the Cartesian distance between each vector in the list SBCenterVector1 and CenterVector1, and then calculate the standard deviation of all Cartesian distance values;

[0180] CenterListNew's Centerstd1 field = calculated standard deviation;

[0181] Step S308: Calculate the Cartesian distance between each vector in the list SBCenterVector2 and CenterVector2, and then calculate the standard deviation of all Cartesian distance values;

[0182] CenterListNew's Centerstd2 field = calculated standard deviation;

[0183] Step S309: Set SBCenterCounter=SBCenterCounter+1;

[0184] Step S310: If SBCenterCounter is less than or equal to ZYDBSBNum, go to step S303; otherwise, go to step S311;

[0185] Step S311 and step S3 end.

[0186] The other steps and parameters are the same as those in the first to third embodiments.

[0187] Specific embodiment 5: This embodiment differs from any one of specific embodiments 1 to 4 in that the specific process of step S4 is as follows:

[0188] Step S401: inputting a video data TestData of an operator's action;

[0189] Step S402: Create a device distance weighted candidate text description list BXTextList, which contains the following fields:

[0190] BXText1: Descriptive text information for the operator's upper body movements;

[0191] BXText2: Descriptive text information for the operator's lower body movements;

[0192] BXVecter1: vector data describing the worker's upper body movements;

[0193] BXVecter2: Vector data describing the lower body movements of workers;

[0194] Distance1: distance description variable of upper body movement;

[0195] Distance2: distance description variable of lower body movement;

[0196] Distance: a variable describing the comprehensive distance between people;

[0197] JQ: device distance weighted value;

[0198] Step S403: Create a counter BXCounter=1 for the candidate text description list;

[0199] Step S404: Create the first temporary variable BXTemp1 in the candidate text description list. Use the BXCounterth model in the list ModelList1 to process the action video data TestData to obtain the text description of the action video data TestData.

[0200] Step S405 , the candidate text description list creates a second temporary variable BXTemp2 = using the BXCounter th model in the list ModelList2 to process the action video data TestData, and obtain the text description of the action video data TestData;

[0201] Step S406: Create a third temporary variable BXTemp3 in the candidate text description list = BXCounter th element of ZYDBSBNameList;

[0202] Step S407: Create a fourth temporary variable BXTemp4 in the candidate text description list. Process the variable using the module WBSLH. The WBSLHInput1 and WBSLHInput2 of the module WBSLH are equal to BXTemp3 and BXTemp1, respectively.

[0203] Step S408: Create the fifth temporary variable BXTemp5 in the candidate text description list. Process the variable using the module WBSLH. The WBSLHInput1 of the module WBSLH is equal to BXTemp3, and the WBSLHInput2 is equal to BXTemp2.

[0204] Step S409: Use the Yolo neural network to identify whether there is a device named BXTemp3 in the action video data TestData;

[0205] If there is only one device named BXTemp3 in the action video data TestData, the alternative text description list creates a sixth temporary variable BXTemp6 = obtains the distance between the position of the device named BXTemp3 in the action video data TestData and the position of the operator (i.e., the number of pixels between the position of the device and the position of the operator);

[0206] If there are multiple devices named BXTemp3 in the action video data TestData, the alternative text description list creates a sixth temporary variable BXTemp6 = the minimum distance between the location of each device named BXTemp3 and the location of the operator (i.e., the distance from each device to the operator is obtained separately, and the minimum distance is obtained by comparison);

[0207] If there is no device named BXTemp3 in the action video data TestData, the alternative text description list creates a sixth temporary variable BXTemp6 = the width of the action video data TestData (ie, the number of pixels in the width direction) × 10;

[0208] Step S410, updating the sixth temporary variable BXTemp6'=BXTemp6 / W, where W represents the width of the action video data TestData;

[0209] Step S411: Add a new element to the list BXTextList. The BXText1 field of this element is BXTemp1, the BXText2 field is BXTemp2, the BXVecter1 field is BXTemp4, the BXVecter2 field is BXTemp5, the Distance1 field is 0, the Distance2 field is 0, the Distance field is 0, and the JQ field is BXTemp6'.

[0210] Step S412: Set BXCounter=BXCounter+1;

[0211] Step S413: If BXCounter is less than or equal to ZYDBSBNum, go to step S404; otherwise, go to step S414.

[0212] Step S414 and step S4 end.

[0213] The other steps and parameters are the same as those in the first to fourth embodiments.

[0214] Specific embodiment 6: This embodiment differs from any one of specific embodiments 1 to 5 in that the specific process of step S5 is as follows:

[0215] Step S501, automation description counter MSCounter=1;

[0216] Step S502 , automatically describe the first temporary variable MSTemp1 = BXVecter1 of the MSCounterth element in the list BXTextList;

[0217] Step S503 , automatically describe the second temporary variable MSTemp2 = BXVecter2 of the MSCounterth element in the list BXTextList;

[0218] Step S504, automatically describe the third temporary variable MSTemp3 = JQ of the MSCounterth element in the list BXTextList;

[0219] Step S505: Set the value of the MSCounter dimension of MSTemp1 to JQ;

[0220] Step S506: Set the value of the MSCounter dimension of MSTemp2 to JQ;

[0221] Step S507 , automatically describe the fourth temporary variable MSTemp4 = calculate the Cartesian distance between MSTemp1 and CenterVector1 of the MSCounter th element in the list CenterList;

[0222] Step S508 , automatically describe the fifth temporary variable MSTemp5 = calculate the Cartesian distance between MSTemp2 and the CenterVector2 of the MSCounter th element in the list CenterList;

[0223] Step S509, automatically describe the sixth temporary variable MSTemp6 = Centerstd1 of the MSCounterth element in the list CenterList;

[0224] Step S510, automatically describe the seventh temporary variable MSTemp7 = Centerstd2 of the MSCounterth element in the list CenterList;

[0225] Step S511, update MSTemp4'=(MSTemp4-2×MSTemp6) / (4×MSTemp6);

[0226] Step S512, update MSTemp5'=(MSTemp5-2×MSTemp7) / (4×MSTemp7);

[0227] Step S513: Set the Distance1 field of the list BXTextList to MSTemp4', the Distance2 field to MSTemp5', and the Distance field to (0.7*MSTemp4'+0.3*MSTemp5');

[0228] Step S514: Set MSCounter=MSCounter+1;

[0229] Step S515: If MSCounter is less than or equal to ZYDBSBNum, go to step S502; otherwise, go to step S516.

[0230] Step S516: Create a text description list SSSXList. SSSXList is a list of character strings and initially contains only 0 elements.

[0231] Create a device description list SSSBList, which is a list of strings and initially contains only 0 elements;

[0232] Step S517, the first temporary variable SX1 used for screening = find the element position with the smallest value of Distance1 in the list BXTextList;

[0233] Step S518: Add the element at the SX1th position of ZYDBSBNameList to SSSBList, add BXText1 of the element at the SX1th position of BXTextList to SSSXList, and add BXText2 of the element at the SX1th position of BXTextList to SSSXList;

[0234] Step S519, the second temporary variable SX2 used for screening = find the element position with the smallest Distance2 value in the list BXTextList;

[0235] Step S520: Add the element at the SX2th position of ZYDBSBNameList to SSSBList, add BXText1 of the element at the SX2th position of BXTextList to SSSXList, and add BXText2 of the element at the SX2th position of BXTextList to SSSXList;

[0236] Step S521, the third temporary variable SX3 used for screening = find the element position with the smallest Distance value in the list BXTextList;

[0237] Step S522: add the element at the SX3th position of ZYDBSBNameList to SSSBList, add BXText1 of the element at the SX3th position of BXTextList to SSSXList, and add BXText2 of the element at the SX3th position of BXTextList to SSSXList;

[0238] Step S523: Delete all duplicate elements in SSSBList, that is, only keep one of each element;

[0239] Step S524: Delete all duplicate elements in SSSXList, that is, only keep one of each element;

[0240] Step S525, the fourth temporary variable SX4 for screening = synthesizes all elements in SSSBList into a string, with each element separated by ",";

[0241] Step S526, the fifth temporary variable SX5 for screening = synthesizes all elements in SSSXList into a string, with each element separated by ",";

[0242] Step S527, the final result string ResultString = the combined result of the strings "for device:" SX4 and "operator performed:" SX5; that is, the string ResultString = for device + SX4 + operator performed + SX5;

[0243] Step S528: Output the final result string ResultString as the automatic semantic description result of the action video data TestData, and the entire algorithm ends.

[0244] The other steps and parameters are the same as those in the first to fifth embodiments.

[0245] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A method for automatically describing the semantics of equipment-related video data of electric power workers' actions, characterized in that: The method specifically comprises the following steps: Step S1: Inputting a semantic description dataset ZYDB of video data of electric power workers' actions, obtaining a list ZYDBSBNameList of equipment names operated by electric power workers in ZYDB, and obtaining the number ZYDBSBNum of elements in the list ZYDBSBNameList of equipment names operated by electric power workers; Obtain the first list ModelList1 of textual description models of power operation personnel action videos associated with the equipment; obtain the second list ModelList2 of textual description models of power operation personnel action videos associated with the equipment; Step S2: establishing a vectorization module WBSLH for integrating the name of the power operation equipment and the action text. The input of the module WBSLH is the name of the equipment to be vectored WBSLHInput1 and the action text to be vectored WBSLHInput2. The output of the module WBSLH is the vectorization result WBSLHOutput. Step S3: Use module WBSLH to create a device association description center vector list CenterList; Step S4: Input a video data TestData of an operator's action, and use the list ModelList1 and the list ModelList2 to convert the input video data TestData into a device distance weighted candidate text description list BXTextList; Step S5: Complete the output of the automated semantic description result according to the device association description center vector list CenterList and the device distance weighted candidate text description list BXTextList.

2. The method for automatic semantic description of equipment-related power worker action video data according to claim 1, characterized in that: The specific process of step S1 is: Step S101: Input the semantic description dataset ZYDB of the power worker action video data. The dataset ZYDB contains the following fields: DBSP: video data of power workers’ actions; DBSB: the name of the operated power device, which is represented by a character string; DBDSCSS: Description of the upper body movements of an electric worker when operating electric equipment. The description of the upper body movements is a string. DBDSCXS: Description of the lower body movements of an electric worker when operating electric equipment. The description of the lower body movements is a string. Step S102: Create a list ZYDBSBNameList of device names operated by power workers in the data set ZYDB, where each element in the list ZYDBSBNameList corresponds to a character string; Step S103: Add all DBSBs in the data set ZYDB to ZYDBSBNameList; Step S104: Delete duplicate elements in the list ZYDBSBNameList, and keep only one of each element; Step S105: Obtain the number of elements in the list of names of devices operated by power workers in the data set ZYDB: ZYDBSBNum = the number of elements in ZYDBSBNameList; Step S106: Create a first list ModelList1 of textual description models of power worker action videos associated with equipment, and initialize the number of elements in the list ModelList1 to 0; Create the second list ModelList2 of the textual description model of the power worker action video associated with the equipment, and initialize the number of elements in the list ModelList2t to 0; Step S107: establish the action video text description model construction process counter InitCounter=1; Step S108, in the process of constructing the action video text description model, the first temporary variable InitTemp1 = the InitCounterth element of the list ZYDBSBNameList; Step S109, upper body description model VideoBERTSS = trained first ActionCLIP model; Select all data in the dataset ZYDB whose DBSB field is equal to InitTemp1, use the DBSP field in the selected data as the input of the first ActionCLIP model, use the DBDSCSS field in the selected data as the output of the first ActionCLIP model, and use the selected data to train the first ActionCLIP model; Step S110, lower body description model VideoBERTXS = trained second ActionCLIP model; Select all data in the dataset ZYDB whose DBSB field is equal to InitTemp1, use the DBSP field in the selected data as the input of the second ActionCLIP model, use the DBDSCXS field in the selected data as the output of the second ActionCLIP model, and use the selected data to train the second ActionCLIP model; Step S111, add VideoBERTSS to the list ModelList1; add VideoBERTXS to the list ModelLis2t; Step S112, set InitCounter=InitCounter+1; Step S113: If InitCounter is less than or equal to ZYDBSBNum, go to step S108; otherwise, go to step S114; Step S114, step S1 ends.

3. The method for automatic semantic description of equipment-related power worker action video data according to claim 2, characterized in that: The specific process of step S2 is: Step S201: establishing a vectorization module WBSLH for integrating the name of the power operation equipment and the action text. The input of the module WBSLH is the name of the equipment to be vectorized WBSLHInput1 and the action text to be vectorized WBSLHInput2. Step S202, WBSLHPos=finds the position of WBSLHInput1 in ZYDBSBNameList; Step S203: Create the first temporary vector WBSLHVector1 of the vectorization module = create a vector of dimension ZYDBSBNum, with all contents of the vector WBSLHVector1 being zeros; Step S204, set the value of the WBSLHPos th element of the vector WBSLHVector1 to 0.1; Step S205: Create the second temporary vector WBSLHVector2 of the vectorization module = convert WBSLHInput2 into a vector using the bert-base-chinese model; Step S206, creating a vectorization result WBSLHOutput = merging WBSLHVector1 and WBSLHVector2 into one vector; Step S207: Output WBSLHOutput.

4. The method for automatic semantic description of equipment-related power worker action video data according to claim 3, characterized in that: The specific process of step S3 is: Step S301: Create a device association description center vector list CenterList. Each element of CenterList contains four fields: CenterVector1: the first center vector; Centerstd1: standard deviation of the first center vector; CenterVector2: the second center vector; Centerstd2: standard deviation of the second center vector; Step S302, device association center vector construction counter SBCenterCounter=1; Step S303: The device association center vector constructs a first temporary variable SBCenterTemp1 = the SBCenterCounterth element of the list ZYDBSBNameList; Step S304: Select all elements in the data set ZYDB whose DBSB field is equal to SBCenterTemp1; use the module WBSLH to process the selected elements, that is, the input WBSLHInput1 of the module WBSLH is equal to SBCenterTemp1, and the input WBSLHInput2 of the module WBSLH is the DBDSCSS of each selected element; Device Association Description Vector List 1 SBCenterVector1 = the output of the selected DBDSCSS of each element through the module WBSLH; Step S305: Select all elements in the data set ZYDB whose DBSB field is equal to SBCenterTemp1; use the module WBSLH to process the selected elements, that is, the input WBSLHInput1 of the module WBSLH is equal to SBCenterTemp1, and the input WBSLHInput2 of the module WBSLH is the DBDSCXS of each selected element; The second list of device association description vectors SBCenterVector2 = the output of each selected element DBDSCXS through the module WBSLH; Step S306: Create a new element CenterListNew for the list CenterList; CenterListNew's CenterVector1 is equal to the dimension-by-dimension mean of all vectors in the list SBCenterVector1, and CenterListNew's CenterVector2 is equal to the dimension-by-dimension mean of all vectors in the list SBCenterVector2. Step S307: Calculate the Cartesian distance between each vector in the list SBCenterVector1 and CenterVector1, and then calculate the standard deviation of all Cartesian distance values; CenterListNew's Centerstd1 field = calculated standard deviation; Step S308: Calculate the Cartesian distance between each vector in the list SBCenterVector2 and CenterVector2, and then calculate the standard deviation of all Cartesian distance values; CenterListNew's Centerstd2 field = calculated standard deviation; Step S309: Set SBCenterCounter=SBCenterCounter+1; Step S310: If SBCenterCounter is less than or equal to ZYDBSBNum, go to step S303; otherwise, go to step S311; Step S311 and step S3 end.

5. The method for automatic semantic description of equipment-related power worker action video data according to claim 4, characterized in that: The specific process of step S4 is as follows: Step S401: inputting a video data TestData of an operator's action; Step S402: Create a device distance weighted candidate text description list BXTextList, which contains the following fields: BXText1: Descriptive text information for the operator's upper body movements; BXText2: Descriptive text information for the operator's lower body movements; BXVecter1: vector data describing the worker's upper body movements; BXVecter2: Vector data describing the lower body movements of workers; Distance1: distance description variable of upper body movement; Distance2: distance description variable of lower body movement; Distance: a variable describing the comprehensive distance between people; JQ: device distance weighted value; Step S403: Create a counter BXCounter=1 for the candidate text description list; Step S404: Create the first temporary variable BXTemp1 in the candidate text description list. Use the BXCounterth model in the list ModelList1 to process the action video data TestData to obtain the text description of the action video data TestData. Step S405 , the candidate text description list creates a second temporary variable BXTemp2 = using the BXCounter th model in the list ModelList2 to process the action video data TestData, and obtain the text description of the action video data TestData; Step S406: Create a third temporary variable BXTemp3 in the candidate text description list = BXCounter th element of ZYDBSBNameList; Step S407: Create a fourth temporary variable BXTemp4 in the candidate text description list. Process the variable using the module WBSLH. The WBSLHInput1 and WBSLHInput2 of the module WBSLH are equal to BXTemp3 and BXTemp1, respectively. Step S408: Create the fifth temporary variable BXTemp5 in the candidate text description list. Process the variable using the module WBSLH. The WBSLHInput1 of the module WBSLH is equal to BXTemp3, and the WBSLHInput2 is equal to BXTemp2. Step S409: Use the Yolo neural network to identify whether there is a device named BXTemp3 in the action video data TestData; If there is only one device named BXTemp3 in the action video data TestData, the alternative text description list creates a sixth temporary variable BXTemp6 = obtains the distance between the position of the device named BXTemp3 in the action video data TestData and the position of the operator; If there are multiple devices named BXTemp3 in the action video data TestData, the alternative text description list creates a sixth temporary variable BXTemp6 = the minimum distance between the location of each device named BXTemp3 and the location of the operator; If there is no device named BXTemp3 in the action video data TestData, the alternative text description list creates a sixth temporary variable BXTemp6 = the width of the action video data TestData × 10; Step S410, updating the sixth temporary variable BXTemp6'=BXTemp6 / W, where W represents the width of the action video data TestData; Step S411: Add a new element to the list BXTextList. The BXText1 field of this element is BXTemp1, the BXText2 field is BXTemp2, the BXVecter1 field is BXTemp4, the BXVecter2 field is BXTemp5, the Distance1 field is 0, the Distance2 field is 0, the Distance field is 0, and the JQ field is BXTemp6'. Step S412: Set BXCounter=BXCounter+1; Step S413: If BXCounter is less than or equal to ZYDBSBNum, go to step S404; otherwise, go to step S414. Step S414 and step S4 end.

6. The method for automatic semantic description of equipment-related power worker action video data according to claim 5, characterized in that: The specific process of step S5 is as follows: Step S501, automation description counter MSCounter=1; Step S502 , automatically describe the first temporary variable MSTemp1 = BXVecter1 of the MSCounterth element in the list BXTextList; Step S503 , automatically describe the second temporary variable MSTemp2 = BXVecter2 of the MSCounterth element in the list BXTextList; Step S504, automatically describe the third temporary variable MSTemp3 = JQ of the MSCounterth element in the list BXTextList; Step S505: Set the value of the MSCounter dimension of MSTemp1 to JQ; Step S506: Set the value of the MSCounter dimension of MSTemp2 to JQ; Step S507 , automatically describe the fourth temporary variable MSTemp4 = calculate the Cartesian distance between MSTemp1 and CenterVector1 of the MSCounter th element in the list CenterList; Step S508 , automatically describe the fifth temporary variable MSTemp5 = calculate the Cartesian distance between MSTemp2 and the CenterVector2 of the MSCounter th element in the list CenterList; Step S509, automatically describe the sixth temporary variable MSTemp6 = Centerstd1 of the MSCounterth element in the list CenterList; Step S510, automatically describe the seventh temporary variable MSTemp7 = Centerstd2 of the MSCounterth element in the list CenterList; Step S511, update MSTemp4'=(MSTemp4-2×MSTemp6) / (4×MSTemp6); Step S512, update MSTemp5'=(MSTemp5-2×MSTemp7) / (4×MSTemp7); Step S513: Set the Distance1 field of the list BXTextList to MSTemp4', the Distance2 field to MSTemp5', and the Distance field to (0.7*MSTemp4'+0.3*MSTemp5'); Step S514: Set MSCounter=MSCounter+1; Step S515: If MSCounter is less than or equal to ZYDBSBNum, go to step S502; otherwise, go to step S516. Step S516: Create a text description list SSSXList. SSSXList is a list of character strings and initially contains only 0 elements. Create a device description list SSSBList, which is a list of strings and initially contains only 0 elements; Step S517, the first temporary variable SX1 used for screening = find the element position with the smallest value of Distance1 in the list BXTextList; Step S518: Add the element at the SX1th position of ZYDBSBNameList to SSSBList, add BXText1 of the element at the SX1th position of BXTextList to SSSXList, and add BXText2 of the element at the SX1th position of BXTextList to SSSXList; Step S519, the second temporary variable SX2 used for screening = find the element position with the smallest Distance2 value in the list BXTextList; Step S520: Add the element at the SX2th position of ZYDBSBNameList to SSSBList, add BXText1 of the element at the SX2th position of BXTextList to SSSXList, and add BXText2 of the element at the SX2th position of BXTextList to SSSXList; Step S521, the third temporary variable SX3 used for screening = find the element position with the smallest Distance value in the list BXTextList; Step S522: add the element at the SX3th position of ZYDBSBNameList to SSSBList, add BXText1 of the element at the SX3th position of BXTextList to SSSXList, and add BXText2 of the element at the SX3th position of BXTextList to SSSXList; Step S523: Delete all duplicate elements in SSSBList, that is, only keep one of each element; Step S524: Delete all duplicate elements in SSSXList, that is, only keep one of each element; Step S525, the fourth temporary variable SX4 for screening = synthesizes all elements in SSSBList into a string, and separates each element with ","; Step S526, the fifth temporary variable SX5 for screening = synthesizes all elements in SSSXList into a string, with each element separated by ","; Step S527, final result string ResultString = string "for device:" SX4, "operator performed the merge result of:" SX5; Step S528: Output the final result string ResultString as the automatic semantic description result of the action video data TestData, and the entire algorithm ends.

Images