Human-computer interaction behavior prediction model training method and device, equipment and storage medium

Abstract

The invention provides a human-computer interaction behavior prediction model training method and device, equipment and a storage medium, and belongs to the technical field of power equipment overhaul, and the method comprises the steps: collecting multi-modal image data of human-computer interaction behaviors at a plurality of moments in an overhaul and maintenance region; performing feature extraction on the multi-modal image data to obtain an interactive behavior feature vector corresponding to each moment; respectively inputting each interactive behavior feature vector and a corresponding standard interactive behavior feature vector in a standard interactive behavior database into a preset similarity algorithm, and outputting a plurality of similarity values; and if each similarity value is greater than a preset threshold value, training a human-computer interaction behavior model according to each interaction behavior feature vector to obtain a human-computer interaction behavior prediction model. Based on the model, whether the on-site man-machine interaction behavior in the actual scene meets the standard operation or not can be predicted and evaluated, potential safety hazards or wrong operation can be found in time, and the working efficiency and safety are improved.

Description

Technical Field

[0001] The present application relates to the technical field of power equipment maintenance, and particularly to a method, device, equipment and storage medium for training a human-computer interaction behavior prediction model. Background Art

[0002] With the rapid development of the power industry in China, the safe and stable operation of power equipment and the problems in the operation process of maintenance personnel have received increasing attention.

[0003] In the related art, a method for evaluating and predicting the health degree of power equipment mainly includes: designing an evaluation index system for power equipment according to common monitoring parameters of power equipment, assigning weights to each characteristic parameter by using the entropy weight method based on a certain amount of monitoring historical data, calculating the weight values of the characteristic parameters and the weight values of subsystems, then evaluating the health degrees of each subsystem and the overall power equipment based on real-time data to form a historical trend sequence of the health degree. Then, the health degree sequence is used for training an SVR regression model to obtain a health degree prediction model, and the health degree at the next moment is calculated; finally, the new health degree data is brought into the model for iterative training and prediction, and the health degree prediction values at subsequent moments are obtained in sequence to form a health degree prediction curve.

[0004] However, the above health degree prediction model cannot predict the human-computer interaction behavior intention at the next moment, that is, it cannot predict the interaction information between maintenance personnel and equipment at future moments, so it is insufficient in standardizing and optimizing human-computer interaction, which limits the further improvement of the power equipment maintenance management level. Summary of the Invention

[0005] The present application provides a method, device, equipment and storage medium for training a human-computer interaction behavior prediction model to solve the deficiencies in the related art.

[0006] In a first aspect, the present application provides a method for training a human-computer interaction behavior prediction model, including:

[0007] Collecting multi-modal image data of human-computer interaction behaviors at multiple moments in the maintenance area;

[0008] Performing feature extraction on the multi-modal image data to obtain an interaction behavior feature vector corresponding to each moment, and the interaction behavior features include one or more of the following: operation position feature, operation action feature, and equipment response situation feature;

[0009] Inputting each interaction behavior feature vector and the corresponding standard vector in the standard interaction behavior database into a preset similarity algorithm respectively, and outputting a plurality of similarity values;

[0010] If all similarity values are greater than a preset threshold, the human-computer interaction behavior model is trained according to each interaction behavior feature vector to obtain a human-computer interaction behavior prediction model.

[0011] In a possible implementation, feature extraction is performed on the multimodal image data to obtain an interaction behavior feature vector corresponding to each moment, including:

[0012] The multimodal image data is preprocessed and time-sliced to obtain a multimodal image frame corresponding to each moment;

[0013] The interaction behavior features in each multimodal image frame are extracted, and the interaction behavior features are converted into a numerical feature group, where the numerical feature group includes multiple numerical values reflecting different dimensional information of the interaction behavior features;

[0014] The numerical feature group is standardized to obtain an interaction behavior feature vector corresponding to each moment.

[0015] In a possible implementation, the standard interaction behavior database includes a set of standard interaction behavior feature vectors with timestamp marks, and the human-computer interaction behavior model includes multiple dimensional information numerical sub-models. Among them, each dimensional information numerical sub-model is applicable to the numerical values reflecting the same dimensional information of the standard interaction behavior features in each standard interaction behavior feature vector;

[0016] The dimensional information numerical sub-model represents obtaining the numerical value of the dimensional information in the standard interaction behavior feature vector corresponding to the timestamp value by inputting the timestamp value.

[0017] In a possible implementation, training the human-computer interaction behavior model according to each interaction behavior feature vector to obtain a human-computer interaction behavior prediction model includes:

[0018] Input each interaction behavior feature vector into a clustering algorithm, and output multiple element arrays with timestamp marks, where each element array with a timestamp mark represents the numerical value reflecting the same dimensional information of the interaction behavior features;

[0019] According to the element array with a timestamp mark, train and adjust the parameters in the corresponding dimensional information numerical sub-model to obtain a dimensional information numerical prediction sub-model;

[0020] Combine each dimensional information numerical prediction sub-model to form a human-computer interaction behavior prediction model.

[0021] In a possible implementation, after training the human-computer interaction behavior model according to each interaction behavior feature vector to obtain a human-computer interaction behavior prediction model, it further includes:

[0022] According to the human-computer interaction behavior prediction model, output the interaction behavior feature vector at a future moment;

[0023] Determine the first standard vector corresponding to the future moment from the standard interaction behavior database;

[0024] Input the interaction behavior feature vector at the future moment and the first standard vector into a preset similarity algorithm, and output the first similarity value;

[0025] If the first similarity value is less than or equal to a preset threshold, predict that the human-computer interaction behavior at the future moment is abnormal.

[0026] In a possible implementation manner, after predicting that the interaction behavior at the future moment is abnormal, it further includes:

[0027] Determine the maintenance plan from the expert knowledge base;

[0028] Send the maintenance plan to the mobile terminal of the maintenance personnel and the display interface of the operation console closest to the equipment to be maintained.

[0029] In a possible implementation manner, the human-computer interaction behavior model refers to

[0030]

[0031] Among them, X1(t)...X n (t) represents the numerical value reflecting different dimensional information of the standard interaction behavior characteristics at time t, a1, b1...q1 represent the parameters in the dimensional information numerical sub-model (t, X1(t)); a2, b2...q2 represent the parameters in the dimensional information numerical sub-model (t, X2(t)); a n , b n ...q n represents the parameters in the dimensional information numerical sub-model (t, X n (t)).

[0032] In a second aspect, the present application provides a human-computer interaction behavior prediction model training device, including: a collection module, an extraction module, a calculation module, and a training module, where

[0033] The collection module is used to collect multi-modal image data of human-computer interaction behaviors at multiple moments in the maintenance area;

[0034] The extraction module is used to extract features from the multi-modal image data to obtain interaction behavior feature vectors, and the interaction behaviors include one or more of the following: operation state features, operation action features, and device response situation features;

[0035] The calculation module is used to input each interaction behavior feature vector and the corresponding standard vector in the standard interaction behavior database into a preset similarity algorithm respectively, and output multiple similarity values;

[0036] The training module is used to train the human - machine interaction behavior model according to the interaction behavior feature vectors if all the similarity values are greater than a preset threshold, so as to obtain a human - machine interaction behavior prediction model.

[0037] In a third aspect, the present application provides a control device, including a memory and a processor. The memory stores program instructions, and the processor is used to call the program instructions in the memory to execute the human - machine interaction behavior prediction model training method according to any one of the first aspects.

[0038] In a fourth aspect, the present application further provides a computer - readable storage medium. Computer - executable instructions are stored in the computer - readable storage medium, and when the computer - executable instructions are executed by a processor, they are used to implement the human - machine interaction behavior prediction model training method according to any one of the first aspects.

[0039] The human - machine interaction behavior prediction model training method, device, equipment and storage medium provided by the present application collect multi - modal image data of human - machine interaction behaviors at multiple moments in the maintenance area, and then extract features from the multi - modal image data to obtain the interaction behavior feature vectors corresponding to each moment, so as to obtain the interaction behavior feature information at each moment; by respectively inputting each interaction behavior feature vector and the corresponding standard interaction behavior feature vector in the standard interaction behavior database into a preset similarity algorithm, multiple similarity values are output to quantitatively analyze whether the actual human - machine interaction behaviors at these moments operate with reference to the standard interaction behavior database.

[0040] Furthermore, if all the similarity values are greater than the preset threshold, it indicates that the currently collected interaction behavior data has high standardization, and the human - machine interaction behavior model can be effectively trained based on these interaction behavior feature vectors to obtain a human - machine interaction behavior prediction model. Based on this model, it is possible to predict and evaluate whether the on - site human - machine interaction behavior in the actual scenario conforms to the standard operation, which is conducive to timely discovering potential safety hazards or incorrect operations and improving work efficiency and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with the present application and used together with the description to explain the principles of the present application.

[0042] Figure 1 It is a schematic flowchart of a human - machine interaction behavior prediction model training method provided by an embodiment of the present application;

[0043] Figure 2 It is a schematic flowchart of another human - machine interaction behavior prediction model training method provided by an embodiment of the present application;

[0044] Figure 3Schematic flowchart of yet another method for training a human-computer interaction behavior prediction model provided by an embodiment of the present application;

[0045] Figure 4 Schematic structural diagram of a device for training a human-computer interaction behavior prediction model provided by an embodiment of the present application;

[0046] Figure 5 Exploded view of a mobile terminal provided by an embodiment of the present application;

[0047] Figure 6 Schematic hardware structure diagram of a control device provided by an embodiment of the present application.

[0048] Description of reference numerals:

[0049] 100 - housing; 200 - display screen; 300 - audio player; 400 - chipset; 500 - microphone; 600 - battery; 700 - linear motor. Detailed implementation manners

[0050] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below with reference to the accompanying drawings in the preferred embodiments of the present application. In the drawings, the same or similar reference numerals denote the same or similar components or components with the same or similar functions throughout. The described embodiments are some, but not all, of the embodiments of the present application. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, but should not be construed as limiting the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts fall within the scope of protection of the present application. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0051] In the description of the present application, it should be noted that unless otherwise clearly defined and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it may be a fixed connection or an indirect connection through an intermediate medium, and it may be the internal communication of two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0052] In the description of the present application, it should be understood that the orientation or positional relationship indicated by the terms "upper", "lower", "front", "rear", "left", "right", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the present application.

[0053] In the description and claims of this application and the above-mentioned drawings, terms such as "first", "second", "third", "fourth", etc. are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described herein, for example, can be implemented in an order other than those illustrated or described herein.

[0054] In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0055] In the related art, a method for evaluating and predicting the health of power equipment mainly includes: designing an evaluation index system for power equipment according to common monitoring parameters of power equipment, assigning weights to each characteristic parameter by using the entropy weight method based on a certain amount of historical monitoring data, calculating the weight values of the characteristic parameters and the weight values of the subsystems, then evaluating the health of each subsystem and the overall power equipment based on real-time data to form a historical trend sequence of the health, and then using the health sequence for training of the SVR regression model to obtain a health prediction model and calculating the health at the next moment; finally, bringing the new health data into the model for iterative training and prediction to sequentially obtain the health prediction values at subsequent moments and form a health prediction curve.

[0056] However, the above-mentioned health prediction model cannot predict the human-computer interaction behavior intention at the next moment, that is, it cannot predict the interaction information between maintenance personnel and equipment at future moments, so it is still insufficient in standardizing and optimizing human-computer interaction, which limits the further improvement of the maintenance management level of power equipment.

[0057] In view of this, this application provides a method, device, equipment and storage medium for training a human-computer interaction behavior prediction model. By collecting multi-modal image data of human-computer interaction behavior at multiple moments in the maintenance area, and then extracting features from the multi-modal image data to obtain an interaction behavior feature vector corresponding to each moment, so as to obtain the interaction behavior feature information at each moment. The interaction behavior feature information reflects the interaction information between maintenance personnel and equipment. The interaction behavior feature may include one or more of the following: operation position feature, operation action feature and equipment response situation feature.

[0058] By inputting each interactive behavior feature vector and the corresponding standard interactive behavior feature vector in the standard interactive behavior database into a preset similarity algorithm respectively, multiple similarity values are output to quantitatively analyze whether the actual human-machine interaction behaviors at these moments operate with reference to the standard interactive behavior database; further, if each similarity value is greater than a preset threshold, it indicates that the currently collected interactive behavior data has high normativity, and the human-machine interaction behavior model can be effectively trained based on these interactive behavior feature vectors. Among them, the human-machine interaction behavior model is a model established based on the standard interactive behavior database and can reflect standard human-machine interaction behaviors. For example, the parameters of the human-machine interaction behavior model are optimized and adjusted through each interactive behavior feature vector to obtain a human-machine interaction behavior prediction model.

[0059] Based on this model, it is possible to predict and evaluate whether the on-site human-machine interaction behaviors in the actual scenario conform to the standard operations, which is conducive to timely discovering potential safety hazards or incorrect operations and improving work efficiency and safety.

[0060] Figure 1 The flowchart of a method for training a human-machine interaction behavior prediction model provided by an embodiment of the present application is shown in Figure 1 and the method includes:

[0061] S101. Collect multi-modal image data of human-machine interaction behaviors at multiple moments in the maintenance area.

[0062] Here, the multi-modal image data refers to a set of image data collected by multiple different imaging technologies or sensors and can describe the same scene or event from multiple perceptual dimensions.

[0063] The multi-modal image data at multiple moments can be in the form of a continuous video stream or a set of static images with timestamp marks.

[0064] The multi-modal image data of human-machine interaction behaviors refers to the set of image information during the interaction between maintenance personnel and equipment collected by various sensors installed in the work area during the maintenance of power equipment. It can include various types of data such as gesture recognition images of maintenance personnel, body posture analysis images, and images of changes in the display panel during equipment response.

[0065] Specifically, in the power equipment maintenance area, multi-modal sensors are installed. The multi-modal sensors include cameras and motion capture devices, and multi-modal image data of human-machine interaction behaviors at multiple moments are collected through the multi-modal sensors.

[0066] Exemplarily, the multi-modal image data encompasses rich information, including the operation position information of maintenance personnel; operation action information, such as the gestures, body postures, tool usage, etc. of maintenance personnel; and equipment response information, such as the feedback content on the display screen, the action responses of mechanical components, etc.

[0067] S102. Extract features from the multi-modal image data to obtain an interaction behavior feature vector corresponding to each moment. The interaction behavior features include one or more of the following: operation position features, operation action features, and equipment response situation features.

[0068] This step can be understood as that for the multi-modal image data collected at each moment, through work such as preprocessing and feature extraction, complex visual information can be transformed into quantitative features that can be used for analysis and prediction. Among them, the quantitative features are the interaction behavior feature vectors corresponding to each moment.

[0069] Here, the interaction behavior feature can be one of the operation position features, operation action features, and equipment response situation features. That is to say, when extracting features from the multi-modal image data, only a single interaction behavior feature needs to be extracted. Thus, the processing flow of the multi-modal image data can be simplified, which is beneficial to reducing the complexity of feature extraction.

[0070] Of course, the interaction behavior features can also be multiple of the operation position features, operation action features, and equipment response situation features. Thus, by considering multiple feature dimensions simultaneously, a relatively rich description of the interaction behavior can be obtained.

[0071] S103. Input each interaction behavior feature vector and the corresponding standard interaction behavior feature vector in the standard interaction behavior database into a preset similarity algorithm respectively, and output multiple similarity values.

[0072] That is to say, by setting a preset similarity algorithm, the similarity value between the interaction behavior feature vector corresponding to each moment and the corresponding standard vector in the standard interaction behavior database is obtained, so as to quantitatively analyze whether the actual human-computer interaction behavior at these moments operates with reference to the standard interaction behavior database, thereby being able to determine whether the human-computer interaction behavior at these moments is correct.

[0073] S104. If each similarity value is greater than a preset threshold, then train the human-computer interaction behavior model according to each of the interaction behavior feature vectors to obtain a human-computer interaction behavior prediction model.

[0074] That is to say, when all similarity values are greater than the preset threshold, it indicates that the interactive behavior characteristics collected at multiple moments have a high consistency with the standard overhaul and maintenance behaviors mapped by the standard interactive behavior database. Based on this, the human-machine interaction behavior model can be trained according to the actual interactive behavior feature vectors at multiple moments, so as to obtain a human-machine interaction behavior prediction model that can reflect the interaction information between maintenance personnel and equipment in the overhaul and maintenance area in the actual scenario.

[0075] This model can generate corresponding interactive behavior feature vectors for future moments to predict the human-machine interaction behavior at future moments, so as to provide a basis for guiding and optimizing the operation behaviors of maintenance personnel, which is conducive to ensuring safety and preventing potential errors.

[0076] The method for training the human-machine interaction behavior prediction model provided by this application effectively trains the human-machine interaction behavior model through each interactive behavior feature vector to obtain the human-machine interaction behavior prediction model. Based on this model, it is possible to predict and evaluate whether the on-site human-machine interaction behavior in the actual scenario conforms to the standard operation, which is conducive to timely discovering potential safety hazards or incorrect operations and improving work efficiency and safety.

[0077] Figure 2 For the process schematic diagram of another method for training the human-machine interaction behavior prediction model provided by the embodiments of this application, refer to Figure 2 , this method includes:

[0078] S201. Collect multi-modal image data of the human-machine interaction behavior in the overhaul and maintenance area at multiple moments.

[0079] It should be noted that the execution process of step S201 can refer to the execution process of S101, which will not be elaborated here.

[0080] S202. Preprocess and perform time slicing on the multi-modal image data to obtain multi-modal image frames corresponding to each moment.

[0081] Specifically, the preprocessing performed on the multi-modal image data may include operations such as denoising, enhancement, cropping, and color correction, so as to ensure that the image quality is suitable for subsequent feature extraction.

[0082] It can be understood that time slicing is to cut a continuous video stream into a series of independent image frames at a set time interval (such as N frames per second), and each frame corresponds to the human-machine interaction behavior at a specific moment.

[0083] As mentioned before, when the multi-modal image data collected at multiple moments is a continuous video stream, multi-modal image frames corresponding to each moment can be obtained through time slicing.

[0084] When the multi-modal image data at multiple moments collected is a set of static images marked with specific timestamps, data collation is performed through time slicing with moments as time intervals to ensure that the data collected by the multi-modal sensor at the same moment can be correspondingly matched, so as to obtain the multi-modal image frames corresponding to each moment.

[0085] S203. Extract the interaction behavior features in each multi-modal image frame, convert the interaction behavior features into a numerical feature group, and the numerical feature group includes multiple numerical values reflecting different dimensional information of the interaction behavior features.

[0086] Here, the extracted interaction behavior features can be one or more of the operation position feature, operation action feature, and device response situation feature.

[0087] Specifically, when extracting the operation position feature, image processing technology can be used for extraction. For the position information of the hand or interaction tool in each frame of image, it can usually be represented by two-dimensional or three-dimensional coordinates (for example, in the screen coordinate system). In this way, the position information at each moment can form a numerical feature group, such as (x, y) or (x, y, z).

[0088] When extracting the operation action feature, a deep learning model can be used for extraction, and the extracted action feature is a numerical feature group with a fixed length, and this data feature group encodes different dimensional information such as action type, direction, and speed.

[0089] When extracting the device response situation feature, image processing technology can be used for extraction, and several different state index quantifications of the device response can be obtained. Thus, the device state at each moment can be represented as a numerical feature group, where each dimensional value corresponds to a specific response index information.

[0090] It should be noted that when any two or three of the above-mentioned extracted interaction behavior features are present, the converted digital feature group is the set of the corresponding numerical feature groups of them respectively.

[0091] S204. Perform standardization processing on the numerical feature group to obtain the interaction behavior feature vector corresponding to each moment.

[0092] This step can be understood as converting each numerical feature into data of a unified scale to eliminate the dimensional difference between different dimensional features. Exemplarily, methods such as zero-mean normalization and maximum-minimum normalization can be used to perform standard processing on the numerical feature group, and this application does not limit this.

[0093] S205. Input each interaction behavior feature vector and the corresponding standard interaction behavior feature vector in the standard interaction behavior database into a preset similarity algorithm respectively, and output multiple similarity values.

[0094] S206. If each similarity value is greater than a preset threshold, the human-computer interaction behavior model is trained according to each interaction behavior feature vector to obtain a human-computer interaction behavior prediction model.

[0095] It should be noted that the execution processes of steps S205 - S206 can refer to the execution processes of S103 - S104, and will not be elaborated here.

[0096] In some embodiments, the standard interaction behavior database includes a set of standard interaction behavior feature vectors with timestamp markings.

[0097] The human-computer interaction behavior model includes multiple dimension information numerical sub-models. Among them, each dimension information numerical sub-model is applicable to the numerical values reflecting the same dimension information of the standard interaction behavior features in each standard interaction behavior feature vector. The dimension information numerical sub-model represents that by inputting a timestamp value, the numerical value of the dimension information in the standard interaction behavior feature vector corresponding to the timestamp value can be obtained.

[0098] It should be noted that for the numerical values of different dimension information represented by one of the operation position feature, operation action feature, and device response situation feature that can be included in the interaction behavior feature vector, or the numerical values of different dimension information represented by any two or three of them, corresponding sets of standard interaction behavior feature vectors are set in the standard interaction behavior database.

[0099] Thus, according to the interaction behavior feature vector, the corresponding set of standard interaction behavior feature vectors with timestamp markings can be determined from the standard interaction behavior database.

[0100] Further, for the human-computer interaction behavior model, for example, for the vector set of the standard interaction behavior feature vector that only includes the numerical values of different dimension information represented by the standard operation position feature, the standard interaction behavior feature vector is expressed as [x, y, z], where x represents the standard value of the maintenance personnel in the X direction in the three-dimensional space, y represents the standard value of the maintenance personnel in the Y direction in the three-dimensional space, and z represents the standard value of the maintenance personnel in the Z direction in the three-dimensional space.

[0101] At this time, the human-computer interaction behavior model includes three dimension information numerical sub-models, that is, the three dimension information numerical sub-models are respectively applicable to the standard numerical values in the X direction, Y direction, and Z direction in each standard interaction behavior feature vector.

[0102] Exemplarily, for the dimension information numerical sub-model applicable to the standard numerical values in the X direction in each standard interaction behavior feature vector, by inputting a timestamp value to it, the x value in the standard interaction behavior feature vector corresponding to the timestamp value is output.

[0103] Further, in a specific example, the human - machine interaction behavior model is trained according to each interaction behavior feature vector to obtain a human - machine interaction behavior prediction model, including:

[0104] S1041. Input each interaction behavior feature vector into a clustering algorithm, and output multiple element arrays with timestamp markings, where each element array with a timestamp marking represents a numerical value reflecting the information of the same dimension of the interaction behavior feature.

[0105] Exemplarily, for an interaction behavior feature vector that only includes numerical values representing different dimension information of the operation position feature, it can be expressed as [x’, y’, z’], where x’ represents the actual value of the maintenance personnel in the X - direction in three - dimensional space, y’ represents the actual value of the maintenance personnel in the Y - direction in three - dimensional space, and z’ represents the actual value of the maintenance personnel in the Z - direction in three - dimensional space.

[0106] The clustering algorithm sorts and groups the numerical values of the dimension information in these interaction behavior feature vectors, and outputs three element arrays with timestamp markings. The three arrays respectively correspond to the sets of the actual values of the maintenance personnel in the X, Y, and Z directions in three - dimensional space at specific timestamps.

[0107] S1042. According to the element arrays with timestamp markings, train and adjust the parameters in the corresponding sub - models of the dimension information numerical values to obtain sub - models for predicting dimension information numerical values.

[0108] This step can be understood as training their respective corresponding sub - models of the dimension information numerical values through different element arrays to obtain multiple sub - models for predicting dimension information numerical values, which is beneficial to ensuring the accuracy and effectiveness of the training.

[0109] S1043. Combine the sub - models for predicting dimension information numerical values to form a human - machine interaction behavior prediction model.

[0110] That is to say, a complete human - machine interaction behavior prediction model is formed through combination. This model covers information of multiple dimensions of the interaction behavior feature, and can predict the corresponding interaction information between the operator and the equipment at different time points, so as to realize the real - time monitoring and predictive analysis of the human - machine interaction behavior in the process of power equipment maintenance.

[0111] It can be understood that during the maintenance process, the interaction information data between the maintenance personnel and the equipment is non - linear. Based on this, in a specific example, the human - machine interaction behavior model refers to

[0112]

[0113] where X1(t)...X n(t) represents a numerical value reflecting different dimensional information of the standard interaction behavior characteristics at time t. a1, b1... q1 represent the parameters in the dimensional information numerical sub-model (t, X1(t)); a2, b2... q2 represent the parameters in the dimensional information numerical sub-model (t, X2(t)); a n , b n ... q n represent the parameters in the dimensional information numerical sub-model (t, X n (t)).

[0114] That is to say, for a certain dimensional information numerical sub-model, by inputting a t value, the corresponding dimensional information numerical value in the standard interaction behavior vector can be obtained.

[0115] Figure 3 is a schematic flowchart of another method for training a human-computer interaction behavior prediction model provided by an embodiment of the present application. Refer to Figure 3 , this method includes:

[0116] S301. Collect multi-modal image data of human-computer interaction behaviors at multiple moments in the maintenance area.

[0117] S302. Extract features from the multi-modal image data to obtain an interaction behavior feature vector corresponding to each moment. The interaction behavior features include one or more of the following: operation position feature, operation action feature, and device response situation feature.

[0118] S303. Input each interaction behavior feature vector and the corresponding standard interaction behavior feature vector in the standard interaction behavior database into a preset similarity algorithm respectively, and output multiple similarity values.

[0119] S304. If each similarity value is greater than a preset threshold, train the human-computer interaction behavior model according to each interaction behavior feature vector to obtain a human-computer interaction behavior prediction model.

[0120] It should be noted that the execution process of steps S301 - S304 can refer to the execution process of S101 - S104, and will not be elaborated here.

[0121] S305. Output an interaction behavior feature vector at a future moment according to the human-computer interaction behavior prediction model.

[0122] This step can be understood as that by inputting a future timestamp value into the human-computer interaction prediction model, the interaction behavior feature vector corresponding to this timestamp value can be predicted and output. This interaction behavior feature vector reflects the human-computer interaction behavior intention, that is, the possible interaction information between the maintenance personnel and the device.

[0123] S306. Determine the first standard vector corresponding to the future moment from the standard interaction behavior database.

[0124] That is to say, by selecting the first standard vector, a reference is provided for the interaction behavior feature vector obtained in the previous step.

[0125] It can be understood that the first standard vector reflects the standard interaction information between the maintenance personnel and the equipment.

[0126] S307. Input the interaction behavior feature vector at the future moment and the first standard vector into a preset similarity algorithm, and output the first similarity value.

[0127] Specifically, the preset similarity algorithm can be a cosine similarity algorithm or a Pearson correlation coefficient algorithm. In this regard, the embodiments of the present application do not impose any limitations.

[0128] Exemplarily, the Pearson correlation coefficient algorithm means

[0129]

[0130] where ρ represents the first similarity value; A represents the interaction behavior feature vector at the future moment; B represents the first standard vector; Cov(A, B) is the sample covariance of A and B; S A and S B are the sample standard deviations of A and B respectively.

[0131] S308. If the first similarity value is less than or equal to a preset threshold, predict that the human-computer interaction behavior at the future moment is abnormal.

[0132] This step can be understood as follows: when the first similarity value is less than or equal to the preset threshold, it indicates that there is a large deviation between the interaction behavior feature vector at the future moment and the first standard vector, predicting that the human-computer interaction behavior at the future moment does not follow the standard operation, thus resulting in an abnormality.

[0133] Exemplarily, by using the cosine similarity algorithm or the Pearson correlation coefficient algorithm, the first similarity value will be between [-1, 1]. The preset threshold can be set to 0.5. When the first similarity value is less than or equal to 0.5, it is determined that the human-computer interaction behavior at the future moment is abnormal.

[0134] Furthermore, when an abnormal human-computer interaction behavior intention occurs during the maintenance operation, in order to avoid losses caused by the maintenance personnel performing wrong actions, certain warning measures need to be taken. In some examples, after predicting that the interaction behavior at the future moment is abnormal, it further includes:

[0135] S3081. Determine the maintenance plan from the expert knowledge base.

[0136] Specifically, a maintenance plan refers to the preparatory matters and standardized operation procedures for equipment maintenance.

[0137] The maintenance plans in the expert database include two categories: regular plans and emergency plans. Among them, the regular plan refers to the maintenance plan formulated for regular maintenance tasks; the emergency plan refers to the standardized operation procedures for emergency measures formulated for emergencies mainly caused by misoperations of maintenance personnel during the maintenance operation process.

[0138] S3082. Send the maintenance plan to the mobile terminals of the maintenance personnel and the display interface of the operation console closest to the equipment to be maintained.

[0139] Thereby, it guides the maintenance personnel to perform a series of operations such as cancellation, shutdown, reset, reporting, etc., so as to reduce the occurrence probability of incorrect operations and improve safety.

[0140] Exemplarily, refer to Figure 4 As shown, the mobile terminal includes a housing 100, a display screen 200, an audio player 300, a chipset 400, a microphone 500, a battery 600, and a linear motor 700; among them, the chipset 400 may include a processor chip, a memory chip, and a communication chip, thereby enabling the reception and storage of the maintenance plan.

[0141] In addition to being able to display the specific content of the maintenance plan through the display screen 200, the mobile terminal also has functions such as sound vibration alarm and voice call through the audio player 300 and the microphone 500. Thereby, it can warn the maintenance personnel to pay attention to the operation by means of sound and vibration alarm.

[0142] Refer to Figure 5 As shown, the present application provides a human-computer interaction behavior prediction model training device 40, including: a collection module 41, an extraction module 42, a calculation module 43, and a training module 44, where:

[0143] The collection module 41 is used to collect multi-modal image data of human-computer interaction behaviors at multiple moments in the maintenance area.

[0144] The extraction module 42 is used to extract features from the multi-modal image data to obtain interaction behavior feature vectors. The interaction behavior features include one or more of the following: operation state features, operation action features, and equipment response situation features.

[0145] The calculation module 43 is used to input each interaction behavior feature vector and the corresponding standard vector in the standard interaction behavior database into a preset similarity algorithm respectively, and output a plurality of similarity values.

[0146] The training module 44 is used to train the human-computer interaction behavior model according to the interaction behavior feature vector to obtain a human-computer interaction behavior prediction model if all similarity values ​​are greater than a preset threshold.

[0147] The human-computer interaction behavior prediction model training device 40 of the embodiment of the present application can execute the technical solution of the human-computer interaction behavior prediction model training method in the above method embodiment, and its implementation principle and technical effect are similar and will not be repeated here.

[0148] Figure 6 This is a schematic diagram of the structure of the control device provided in the embodiment of the present application. Figure 6 As shown, the control device 50 includes: a processor 51 and a memory 52, the memory 52 is used to store computer programs, and the processor 51 is used to execute the computer programs stored in the memory 52 to implement the human-computer interaction behavior prediction model training method shown in any of the above method embodiments.

[0149] Specifically, the processor 51 and the memory 52 can communicate; illustratively, the processor 51 and the memory 52 communicate via a communication bus 53.

[0150] Exemplarily, the control device 50 may further include a communication interface, which may include a transmitter and / or a receiver.

[0151] Exemplarily, the processor 51 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0152] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, the computer executes the above-mentioned human-computer interaction behavior prediction model training method.

[0153] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0156] The above-mentioned integrated units implemented in the form of software functional units can be stored in a computer-readable storage medium. The above-mentioned software functional units stored in a storage medium include several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0157] Those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the foregoing method embodiments and will not be repeated here.

[0158] So far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a human-computer interaction behavior prediction model, characterized in that, Including: Collecting multi-modal image data of human-computer interaction behaviors at multiple moments in the maintenance area; Performing feature extraction on the multi-modal image data to obtain an interaction behavior feature vector corresponding to each moment, where the interaction behavior features include one or more of the following: operation position feature, operation action feature, and device response situation feature; Inputting each of the interaction behavior feature vectors and the corresponding standard interaction behavior feature vectors in the standard interaction behavior database into a preset similarity algorithm respectively, and outputting multiple similarity values; If each of the similarity values is greater than a preset threshold, training the human-computer interaction behavior model according to each of the interaction behavior feature vectors to obtain a human-computer interaction behavior prediction model.

2. The method according to claim 1, wherein The performing feature extraction on the multi-modal image data to obtain an interaction behavior feature vector corresponding to each moment includes: Performing preprocessing and time slicing on the multi-modal image data to obtain multi-modal image frames corresponding to each moment; Extracting the interaction behavior features in each of the multi-modal image frames, and converting the interaction behavior features into a numerical feature group, where the numerical feature group includes multiple numerical values reflecting different dimensional information of the interaction behavior features; Performing standardization processing on the numerical feature group to obtain an interaction behavior feature vector corresponding to each moment.

3. The method according to claim 2, wherein The standard interaction behavior database includes a set of standard interaction behavior feature vectors with timestamp marks; The human-computer interaction behavior model includes multiple dimensional information numerical sub-models, where each of the dimensional information numerical sub-models is applicable to the numerical values reflecting the same dimensional information of the standard interaction behavior features in each of the standard interaction behavior feature vectors; The dimensional information numerical sub-model represents that by inputting a timestamp value, the numerical value of the dimensional information in the set of standard interaction behavior feature vectors corresponding to the timestamp value can be obtained.

4. The method according to claim 3, wherein The training the human-computer interaction behavior model according to each of the interaction behavior feature vectors to obtain a human-computer interaction behavior prediction model includes: Inputting each of the interaction behavior feature vectors into a clustering algorithm, and outputting multiple element arrays with timestamp marks, where each element array with a timestamp mark represents the numerical values reflecting the same dimensional information of the interaction behavior features; Training and adjusting the parameters in the corresponding dimensional information numerical sub-model according to the element array with a timestamp mark to obtain a dimensional information numerical prediction sub-model; Collecting each of the dimensional information numerical prediction sub-models to form the human-computer interaction behavior prediction model.

5. The method according to claim 1, characterized in that, After the training the human-computer interaction behavior model according to each of the interaction behavior feature vectors to obtain a human-computer interaction behavior prediction model, it further includes: Outputting an interaction behavior feature vector at a future moment according to the human-computer interaction behavior prediction model; Determining a first standard vector corresponding to the future moment from the standard interaction behavior database; Inputting the interaction behavior feature vector at the future moment and the first standard vector into the preset similarity algorithm, and outputting a first similarity value; If the first similarity value is less than or equal to the preset threshold, predicting that the human-computer interaction behavior at the future moment is abnormal.

6. The method according to claim 5, characterized in that After the abnormal interaction behavior at the predicted future moment, it further includes: Determine a maintenance plan from the expert knowledge base; Send the maintenance plan to the mobile terminal of the maintenance personnel and the display interface of the operation console closest to the equipment to be maintained.

7. The method according to any one of claims 1 to 6, characterized in that, The human-computer interaction behavior model refers to Among them, X1(t)...X n (t) represents the numerical values reflecting different dimensional information of the standard interaction behavior characteristics at time t, and a1, b1...q1 represent the parameters in the dimensional information numerical sub-model (t, X1(t)); a2, b2...q2 represent the parameters in the dimensional information numerical sub-model (t, X2(t)); a n , b n ...q n represent the parameters in the dimensional information numerical sub-model (t, X n (t)).

8. A training device for a human-computer interaction behavior prediction model, characterized in that, Including: An acquisition module, an extraction module, a calculation module, and a training module; The acquisition module is used to acquire multi-modal image data of human-computer interaction behavior at multiple moments in the maintenance area; The extraction module is used to extract features from the multi-modal image data to obtain an interaction behavior feature vector corresponding to each moment, and the interaction behavior features include one or more of the following: operation position feature, operation action feature, and equipment response situation feature; The calculation module is used to input each of the interaction behavior feature vectors and the corresponding standard interaction behavior feature vectors in the standard interaction behavior database into a preset similarity algorithm respectively, and output multiple similarity values; The training module is used to, if each of the similarity values is greater than a preset threshold, train the human-computer interaction behavior model according to each of the interaction behavior feature vectors to obtain a human-computer interaction behavior prediction model.

9. A control device, characterized in that, It includes a memory and a processor. The memory stores program instructions, and the processor is used to call the program instructions in the memory to execute the human-computer interaction behavior prediction model training method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by the processor, they are used to implement the human-computer interaction behavior prediction model training method according to any one of claims 1 to 7.

Images