Human-computer interaction behavior prediction model training method and device, equipment and storage medium

Abstract

The application provides a human-computer interaction behavior prediction model training method and device, equipment and a storage medium, and belongs to the technical field of power equipment maintenance. The method comprises the following steps: collecting multi-modal image data of human-computer interaction behavior at multiple moments in a maintenance area; performing feature extraction on the multi-modal image data to obtain an interaction behavior feature vector corresponding to each moment; inputting each interaction behavior feature vector and a corresponding standard interaction behavior feature vector in a standard interaction behavior database into a preset similarity algorithm respectively, and outputting multiple similarity values; if each similarity value is greater than a preset threshold, training a human-computer interaction behavior model according to each interaction behavior feature vector to obtain a human-computer interaction behavior prediction model. Based on the model, it can be determined whether the on-site human-computer interaction behavior in an actual scene conforms to a standard operation, potential safety hazards or incorrect operations can be found in time, and work efficiency and safety can be improved.

Description

Technical Field

[0001] This application relates to the field of power equipment maintenance technology, and in particular to a training method, device, equipment and storage medium for a human-computer interaction behavior prediction model. Background Technology

[0002] With the rapid development of my country's power industry, the safe and stable operation of power equipment and the problems in the operation process of maintenance personnel are receiving increasing attention.

[0003] In related technologies, a method for assessing and predicting the health of power equipment mainly includes: designing an assessment index system for power equipment based on common monitoring parameters; assigning weights to each characteristic parameter using the entropy weight method based on a certain amount of historical monitoring data; calculating the weights of the characteristic parameters and the weights of the subsystems; assessing the health of each subsystem and the overall power equipment based on real-time data to form a historical trend sequence of health; then using the health sequence to train an SVR regression model to obtain a health prediction model and calculate the health at the next moment; finally, inputting new health data into the model for iterative training and prediction to obtain the predicted health values ​​for subsequent moments, thus forming a health prediction curve.

[0004] However, the aforementioned health prediction model cannot predict the intention of human-computer interaction behavior at the next moment, that is, it cannot predict the interaction information between maintenance personnel and equipment at future moments. Therefore, it is still insufficient in standardizing and optimizing human-computer interaction, which limits the further improvement of the level of power equipment maintenance and management. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for training a human-computer interaction behavior prediction model to address the shortcomings in related technologies.

[0006] Firstly, this application provides a method for training a human-computer interaction behavior prediction model, including:

[0007] Collect multimodal image data of human-computer interaction behavior at multiple times in the maintenance and repair area;

[0008] Feature extraction is performed on multimodal image data to obtain the interaction behavior feature vector corresponding to each time step. The interaction behavior features include one or more of the following: operation position features, operation action features, and device response features.

[0009] Each interactive behavior feature vector and the corresponding standard vector in the standard interactive behavior database are input into a preset similarity algorithm, which outputs multiple similarity values.

[0010] If all similarity values ​​are greater than the preset threshold, the human-computer interaction behavior model is trained based on the feature vectors of each interaction behavior to obtain the human-computer interaction behavior prediction model.

[0011] In one possible implementation, feature extraction is performed on the multimodal image data to obtain the interaction behavior feature vector corresponding to each time step, including:

[0012] The multimodal image data is preprocessed and time-sliced ​​to obtain the multimodal image frame corresponding to each time step;

[0013] Interactive behavior features are extracted from each multimodal image frame and converted into numerical feature groups, which include multiple values ​​that reflect different dimensions of interactive behavior features.

[0014] The numerical feature set is standardized to obtain the interaction behavior feature vector corresponding to each time step.

[0015] In one possible implementation, the standard interactive behavior database includes a set of standard interactive behavior feature vectors with timestamps, and the human-computer interaction behavior model includes multiple dimensional information numerical sub-models, wherein each dimensional information numerical sub-model is applicable to the numerical values ​​in each standard interactive behavior feature vector that reflect the same dimension of information of the standard interactive behavior features.

[0016] The dimensional information numerical sub-model represents obtaining the numerical value of the dimensional information in the standard interactive behavior feature vector set corresponding to the timestamp value by inputting the timestamp value.

[0017] In one possible implementation, the human-computer interaction behavior model is trained based on the feature vectors of each interaction behavior to obtain a human-computer interaction behavior prediction model, including:

[0018] Each interaction behavior feature vector is input into a clustering algorithm, which outputs multiple arrays of elements with timestamps. Each array of elements with timestamps represents a numerical value that reflects the same dimension of information of the interaction behavior features.

[0019] Based on the array of elements with timestamps, train and adjust the parameters in the corresponding dimensional information numerical sub-model to obtain the dimensional information numerical prediction sub-model.

[0020] The numerical prediction sub-models of information from various dimensions are combined to form a human-computer interaction behavior prediction model.

[0021] In one possible implementation, after training the human-computer interaction behavior model based on the feature vectors of each interaction behavior to obtain the human-computer interaction behavior prediction model, the method further includes:

[0022] Based on the human-computer interaction behavior prediction model, output the interaction behavior feature vector for future moments;

[0023] Determine the first standard vector corresponding to the future moment from the standard interaction behavior database;

[0024] Input the interaction behavior feature vector at future time moments and the first standard vector into a preset similarity algorithm, and output the first similarity value;

[0025] If the first similarity value is less than or equal to the preset threshold, the predicted human-computer interaction behavior at future moments will be abnormal.

[0026] In one possible implementation, after an anomaly occurs in the predicted interaction behavior at a future time, the process further includes:

[0027] A repair plan was determined from the expert knowledge base;

[0028] The maintenance plan is sent to the maintenance personnel's mobile terminal and the control panel display interface closest to the equipment to be maintained.

[0029] In one possible implementation, the human-computer interaction behavior model refers to...

[0030]

[0031] Where X1(t)...X n (t) represents the numerical values ​​of different dimensions reflecting the characteristics of standard interactive behavior at time t, where a1, b1...q1 represent the parameters in the dimensional information numerical sub-model (t, X1(t)); a2, b2...q2 represent the parameters in the dimensional information numerical sub-model (t, X2(t)); a n b n ...q n Represents the dimensional information of the numerical sub-model (t, X) n The parameters in (t)).

[0032] Secondly, this application provides a training device for a human-computer interaction behavior prediction model, comprising: a data acquisition module, an extraction module, a calculation module, and a training module, wherein...

[0033] The acquisition module is used to collect multimodal image data of human-computer interaction behavior at multiple times in the maintenance area;

[0034] The extraction module is used to extract features from multimodal image data to obtain interactive behavior feature vectors. The interactive behavior features include one or more of the following: operation status features, operation action features, and device response features.

[0035] The calculation module is used to input the feature vectors of each interaction behavior and the corresponding standard vectors in the standard interaction behavior database into a preset similarity algorithm, and output multiple similarity values.

[0036] The training module is used to train the human-computer interaction behavior model based on the interaction behavior feature vector if all similarity values ​​are greater than the preset threshold, so as to obtain the human-computer interaction behavior prediction model.

[0037] Thirdly, this application provides a control device, including a memory and a processor, wherein the memory stores program instructions, and the processor is used to call the program instructions in the memory to execute the human-computer interaction behavior prediction model training method as described in any of the first aspects.

[0038] Fourthly, this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the human-computer interaction behavior prediction model training method as described in any of the first aspects.

[0039] The human-computer interaction behavior prediction model training method, device, equipment, and storage medium provided in this application collect multimodal image data of human-computer interaction behavior at multiple times in the maintenance area, then extract features from the multimodal image data to obtain the interaction behavior feature vector corresponding to each time moment, thereby obtaining the interaction behavior feature information at each time moment; by inputting each interaction behavior feature vector and the corresponding standard interaction behavior feature vector in the standard interaction behavior database into a preset similarity algorithm, multiple similarity values ​​are output to quantitatively analyze whether the actual human-computer interaction behavior at these times moment is operated with reference to the standard interaction behavior database.

[0040] Furthermore, if all similarity values ​​are greater than the preset threshold, it indicates that the currently collected interaction behavior data has a high degree of standardization. Based on these interaction behavior feature vectors, the human-computer interaction behavior model can be effectively trained to obtain a human-computer interaction behavior prediction model. Based on this model, it is possible to predict and evaluate whether on-site human-computer interaction behavior in real-world scenarios conforms to standard operating procedures, thereby facilitating the timely detection of potential safety hazards or erroneous operations, and improving work efficiency and safety. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] Figure 1 A flowchart illustrating a human-computer interaction behavior prediction model training method provided in this application embodiment;

[0043] Figure 2 A flowchart illustrating another method for training a human-computer interaction behavior prediction model provided in this application embodiment;

[0044] Figure 3A flowchart illustrating another human-computer interaction behavior prediction model training method provided in this application embodiment;

[0045] Figure 4 A schematic diagram of the structure of a training device for a human-computer interaction behavior prediction model provided in an embodiment of this application;

[0046] Figure 5 An exploded view of a mobile terminal provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the hardware structure of a control device provided in an embodiment of this application.

[0048] Explanation of reference numerals in the attached figures:

[0049] 100 - Housing; 200 - Display screen; 300 - Audio player; 400 - Chipset; 500 - Microphone; 600 - Battery; 700 - Linear motor. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar components or components having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0051] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, an indirect connection through an intermediate medium, or the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0052] In the description of this application, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0053] The terms "first," "second," "third," "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in a sequence other than those illustrated or described herein.

[0054] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0055] In related technologies, a method for assessing and predicting the health of power equipment mainly includes: designing an assessment index system for power equipment based on common monitoring parameters; assigning weights to each characteristic parameter using the entropy weight method based on a certain amount of historical monitoring data; calculating the weights of the characteristic parameters and the weights of the subsystems; assessing the health of each subsystem and the overall power equipment based on real-time data to form a historical trend sequence of health; then using the health sequence to train an SVR regression model to obtain a health prediction model and calculate the health at the next moment; finally, inputting new health data into the model for iterative training and prediction to obtain the predicted health values ​​for subsequent moments, thus forming a health prediction curve.

[0056] However, the aforementioned health prediction model cannot predict the intention of human-computer interaction behavior at the next moment, that is, it cannot predict the interaction information between maintenance personnel and equipment at future moments. Therefore, it is still insufficient in standardizing and optimizing human-computer interaction, which limits the further improvement of the level of power equipment maintenance and management.

[0057] In view of this, this application provides a training method, apparatus, device and storage medium for a human-computer interaction behavior prediction model. By collecting multimodal image data of human-computer interaction behavior at multiple times in the maintenance area, and then extracting features from the multimodal image data, the interaction behavior feature vector corresponding to each time moment is obtained, so as to obtain the interaction behavior feature information at each time moment. The interaction behavior feature information reflects the interaction information between maintenance personnel and equipment. The interaction behavior features may include one or more of the following: operation position features, operation action features and equipment response features.

[0058] By inputting the feature vectors of each interactive behavior and their corresponding standard interactive behavior feature vectors from a standard interactive behavior database into a preset similarity algorithm, multiple similarity values ​​are output to quantitatively analyze whether the actual human-computer interaction behavior at these times is performed with reference to the standard interactive behavior database. Furthermore, if all similarity values ​​are greater than a preset threshold, it indicates that the currently collected interactive behavior data has high standardization, and the human-computer interaction behavior model can be effectively trained based on these interactive behavior feature vectors. This human-computer interaction behavior model is built based on the standard interactive behavior database and reflects standard human-computer interaction behavior. For example, the parameters of the human-computer interaction behavior model can be optimized and adjusted using each interactive behavior feature vector to obtain a human-computer interaction behavior prediction model.

[0059] Based on this model, it is possible to predict and evaluate whether on-site human-computer interaction behavior in real-world scenarios conforms to standard operating procedures, thereby helping to promptly identify potential safety hazards or erroneous operations and improve work efficiency and safety.

[0060] Figure 1 A flowchart illustrating a human-computer interaction behavior prediction model training method provided in this application embodiment is shown below. Figure 1 The method includes:

[0061] S101. Collect multimodal image data of human-computer interaction behavior at multiple times in the maintenance and repair area.

[0062] Here, multimodal image data refers to a collection of image data acquired by various imaging technologies or sensors, which can describe the same scene or event from multiple perceptual dimensions.

[0063] Multimodal image data at multiple times can be in the form of a continuous video stream or a collection of static images with timestamps.

[0064] Multimodal image data of human-computer interaction refers to the collection of image information gathered by various sensors installed in the work area during the maintenance and repair of power equipment, reflecting the interaction between maintenance personnel and equipment. It can include various types of data such as gesture recognition images of maintenance personnel, body posture analysis images, and images of changes in the display panel when the equipment responds.

[0065] In practice, within the power equipment maintenance area, multimodal sensors, including cameras and motion capture devices, are installed to collect multimodal image data of human-computer interaction behavior at multiple moments.

[0066] For example, multimodal image data encompasses a wealth of information, including the location of maintenance personnel; operational actions, such as the gestures, body postures, and tool usage of maintenance personnel; and equipment response information, such as the content displayed on the screen and the action response of mechanical components.

[0067] S102. Extract features from the multimodal image data to obtain the interactive behavior feature vector corresponding to each time moment. The interactive behavior features include one or more of the following: operation position features, operation action features, and device response features.

[0068] This step can be understood as follows: for the multimodal image data collected at each time moment, the complex visual information can be transformed into quantitative features that can be used for analysis and prediction through preprocessing and feature extraction. The quantitative features are the interaction behavior feature vectors corresponding to each time moment.

[0069] Here, the interactive behavior feature can be one of the operation position feature, operation action feature, and device response feature. In other words, when extracting features from multimodal image data, only a single interactive behavior feature needs to be extracted. This simplifies the processing flow of multimodal image data and helps reduce the complexity of feature extraction.

[0070] Of course, interactive behavior features can also be multiple features such as operation location features, operation action features, and device response features. Therefore, a richer description of interactive behavior can be obtained by considering multiple feature dimensions at the same time.

[0071] S103. Input each interactive behavior feature vector and the corresponding standard interactive behavior feature vector in the standard interactive behavior database into the preset similarity algorithm, and output multiple similarity values.

[0072] In other words, by setting a preset similarity algorithm, the similarity value between the interactive behavior feature vector corresponding to each moment and the corresponding standard vector in the standard interactive behavior database is obtained. This allows for quantitative analysis of whether the actual human-computer interaction behavior at these moments is performed with reference to the standard interactive behavior database, thereby determining whether the human-computer interaction behavior at these moments is correct.

[0073] S104. If all similarity values ​​are greater than the preset threshold, then the human-computer interaction behavior model is trained based on the interactive behavior feature vectors to obtain the human-computer interaction behavior prediction model.

[0074] In other words, when all similarity values ​​are greater than a preset threshold, it indicates that the collected interactive behavior features at multiple time points have a high degree of consistency with the standard maintenance behaviors mapped by the standard interactive behavior database. Based on this, the human-computer interaction behavior model can be trained using the actual interactive behavior feature vectors at multiple time points, thereby obtaining a human-computer interaction behavior prediction model that can reflect the interaction information between maintenance personnel and equipment in the maintenance area in actual scenarios.

[0075] This model can generate corresponding interactive behavior feature vectors for future moments, thereby predicting human-computer interaction behavior in the future. This can provide a basis for guiding and optimizing the operation behavior of maintenance personnel, which is conducive to ensuring safety and preventing potential errors.

[0076] The human-computer interaction behavior prediction model training method provided in this application effectively trains the human-computer interaction behavior model through feature vectors of various interaction behaviors to obtain the prediction model. Based on this model, it is possible to predict and evaluate whether on-site human-computer interaction behaviors in actual scenarios conform to standard operations, thereby facilitating the timely detection of potential safety hazards or erroneous operations and improving work efficiency and safety.

[0077] Figure 2 A flowchart illustrating another human-computer interaction behavior prediction model training method provided in this application embodiment is shown below. Figure 2 The method includes:

[0078] S201. Collect multimodal image data of human-computer interaction behavior at multiple times in the maintenance and repair area.

[0079] It should be noted that the execution process of step S201 can refer to the execution process of S101, and will not be repeated here.

[0080] S202. Preprocess and time-slice the multimodal image data to obtain the multimodal image frame corresponding to each time step.

[0081] Specifically, preprocessing of multimodal image data can include operations such as denoising, enhancement, cropping, and color correction, thereby ensuring that the image quality is suitable for subsequent feature extraction.

[0082] Understandably, time slicing involves cutting a continuous video stream into a series of independent image frames at set time intervals (such as N frames per second), with each frame corresponding to a specific moment of human-computer interaction.

[0083] As mentioned earlier, when the multimodal image data collected at multiple times is a continuous video stream, the multimodal image frame corresponding to each time moment can be obtained by time slicing.

[0084] When the multimodal image data collected at multiple times is a set of static images marked with specific timestamps, the data is processed by time slicing with time as the time interval to ensure that the data collected by the multimodal sensor at the same time can be matched accordingly, thereby obtaining the multimodal image frame corresponding to each time.

[0085] S203. Extract the interactive behavior features from each multimodal image frame, and convert the interactive behavior features into a numerical feature group. The numerical feature group includes multiple values ​​that reflect different dimensions of interactive behavior features.

[0086] Here, the extracted interactive behavior features can be one or more of the following: operation location features, operation action features, and device response features.

[0087] Specifically, when extracting operation position features, image processing techniques can be used. The position information of the hand or interactive tool in each frame of the image can usually be represented by two-dimensional or three-dimensional coordinates (e.g., in the screen coordinate system). In this way, the position information at each moment can form a set of numerical features, such as (x,y) or (x,y,z).

[0088] When extracting action features, a deep learning model can be used. The extracted action features are a fixed-length numerical feature set, which encodes information in different dimensions such as action type, direction, and speed.

[0089] When extracting device response features, image processing technology can be used to extract them and obtain several different state indicators of the device response. Thus, the device state at each moment can be represented as a set of numerical features, where each dimension corresponds to a specific response indicator.

[0090] It should be noted that when any two or three of the above-mentioned interactive behavior features are present, the resulting set of numerical features is the set of their respective numerical feature sets.

[0091] S204. Standardize the numerical feature set to obtain the interaction behavior feature vector corresponding to each time step.

[0092] This step can be understood as eliminating the dimensional differences between features of different dimensions by transforming each numerical feature into data of a uniform scale. For example, the numerical feature group can be standardized by methods such as zero mean normalization and maximum-minimum normalization. This application does not limit this.

[0093] S205. Input each interactive behavior feature vector and the corresponding standard interactive behavior feature vector in the standard interactive behavior database into the preset similarity algorithm, and output multiple similarity values.

[0094] S206. If all similarity values ​​are greater than the preset threshold, then the human-computer interaction behavior model is trained based on the feature vectors of each interaction behavior to obtain the human-computer interaction behavior prediction model.

[0095] It should be noted that the execution process of steps S205-S206 can refer to the execution process of S103-S104, and will not be repeated here.

[0096] In some embodiments, the standard interaction behavior database includes a set of standard interaction behavior feature vectors with timestamps.

[0097] The human-computer interaction behavior model includes multiple dimensional information numerical sub-models. Each dimensional information numerical sub-model is applicable to the numerical values ​​of the same dimension of information reflecting the standard interaction behavior features in each standard interaction behavior feature vector. The dimensional information numerical sub-model means that by inputting a timestamp value, the numerical values ​​of the dimensional information in the set of standard interaction behavior feature vectors corresponding to the timestamp value are obtained.

[0098] It should be noted that the interactive behavior feature vector can include numerical values ​​representing different dimensions of information, such as operation location features, operation action features, and device response features, or it can include numerical values ​​representing any two or three different dimensions of information. The standard interactive behavior database has set up corresponding standard interactive behavior feature vector sets.

[0099] Therefore, based on the interaction behavior feature vector, the corresponding set of standard interaction behavior feature vectors with timestamps can be determined from the standard interaction behavior database.

[0100] Furthermore, regarding the human-computer interaction behavior model, for example, the standard interaction behavior feature vector is a vector set that only includes the numerical values ​​of different dimensions represented by the standard operation position features. The standard interaction behavior feature vector is represented as [x,y,z], where x represents the standard value of the maintenance personnel in the X direction in three-dimensional space, y represents the standard value of the maintenance personnel in the Y direction in three-dimensional space, and z represents the standard value of the maintenance personnel in the Z direction in three-dimensional space.

[0101] At this point, the human-computer interaction behavior model includes three-dimensional information numerical sub-models. That is, the three-dimensional information numerical sub-models are respectively applicable to the standard values ​​in the X direction, Y direction, and Z direction of each standard interaction behavior feature vector.

[0102] For example, for a dimensional information numerical sub-model applicable to the standard numerical values ​​in the X direction of each standard interaction behavior feature vector, a timestamp value is input to it to output the x value in the standard interaction behavior feature vector corresponding to that timestamp value.

[0103] Furthermore, in a specific example, the human-computer interaction behavior model is trained based on the feature vectors of each interaction behavior to obtain a human-computer interaction behavior prediction model, including:

[0104] S1041. Input each interaction behavior feature vector into the clustering algorithm and output multiple arrays of elements with timestamps. Each array of elements with timestamps represents a value that reflects the same dimension of information of the interaction behavior features.

[0105] For example, an interactive behavior feature vector that includes only numerical values ​​representing different dimensions of information based on the operation position features can be represented as [x',y',z'], where x' represents the actual value of the maintenance personnel in the X direction in three-dimensional space, y' represents the actual value of the maintenance personnel in the Y direction in three-dimensional space, and z' represents the actual value of the maintenance personnel in the Z direction in three-dimensional space.

[0106] Clustering algorithms are used to sort and group the dimensional information values ​​in the feature vectors of these interactive behaviors, outputting three arrays of elements with timestamps. Each array corresponds to a set of actual values ​​in the X, Y, and Z directions of three-dimensional space for the maintenance personnel at a specific timestamp.

[0107] S1042. Based on the array of elements with timestamps, train and adjust the parameters in the corresponding dimensional information numerical sub-model to obtain the dimensional information numerical prediction sub-model.

[0108] This step can be understood as training corresponding dimensional information numerical sub-models using different element arrays to obtain multiple dimensional information numerical prediction sub-models, which helps ensure the accuracy and effectiveness of training.

[0109] S1043. Combine the numerical prediction sub-models of information in each dimension to form a human-computer interaction behavior prediction model.

[0110] In other words, a complete human-computer interaction behavior prediction model is formed by combining various data. This model covers multiple dimensions of interactive behavior characteristics and can predict the corresponding interaction information between operators and equipment at different points in time, thereby enabling real-time monitoring and predictive analysis of human-computer interaction behavior during the maintenance and repair of power equipment.

[0111] It is understandable that during maintenance and repair, the interaction data between maintenance personnel and equipment is non-linear. Based on this, in the specific example, the human-computer interaction behavior model refers to...

[0112]

[0113] Where X1(t)...X n(t) represents the numerical values ​​of different dimensions reflecting the characteristics of standard interactive behavior at time t, where a1, b1...q1 represent the parameters in the dimensional information numerical sub-model (t, X1(t)); a2, b2...q2 represent the parameters in the dimensional information numerical sub-model (t, X2(t)); a n b n ...q n Represents the dimensional information of the numerical sub-model (t, X) n The parameters in (t)).

[0114] In other words, for a specific dimensional information numerical sub-model, by inputting a t value, the corresponding dimensional information value in the standard interactive behavior feature vector can be obtained.

[0115] Figure 3 A flowchart illustrating another human-computer interaction behavior prediction model training method provided in this application embodiment is shown below. Figure 3 The method includes:

[0116] S301. Collect multimodal image data of human-computer interaction behavior at multiple times in the maintenance and repair area.

[0117] S302. Extract features from the multimodal image data to obtain the interactive behavior feature vector corresponding to each time moment. The interactive behavior features include one or more of the following: operation position features, operation action features, and device response features.

[0118] S303. Input each interactive behavior feature vector and the corresponding standard interactive behavior feature vector in the standard interactive behavior database into the preset similarity algorithm, and output multiple similarity values.

[0119] S304. If all similarity values ​​are greater than the preset threshold, then the human-computer interaction behavior model is trained based on the feature vectors of each interaction behavior to obtain the human-computer interaction behavior prediction model.

[0120] It should be noted that the execution process of steps S301-S304 can refer to the execution process of S101-S104, and will not be repeated here.

[0121] S305. Based on the human-computer interaction behavior prediction model, output the interaction behavior feature vector for future moments.

[0122] This step can be understood as follows: by inputting a future timestamp value into the human-computer interaction prediction model, the interactive behavior feature vector corresponding to that timestamp value can be predicted and output. This interactive behavior feature vector reflects the intention of human-computer interaction behavior, that is, the possible interaction information between maintenance personnel and equipment.

[0123] S306. Determine the first standard vector corresponding to the future moment from the standard interaction behavior database.

[0124] In other words, by selecting the first standard vector, a reference is provided for the interaction behavior feature vector obtained in the previous step.

[0125] Understandably, the first standard vector reflects the standard interaction information between maintenance personnel and equipment.

[0126] S307. Input the interaction behavior feature vector of the future time and the first standard vector into the preset similarity algorithm, and output the first similarity value.

[0127] Specifically, the preset similarity algorithm can be a cosine similarity algorithm or a Pearson correlation coefficient algorithm. This application does not limit the specific algorithm used.

[0128] For example, the Pearson correlation coefficient algorithm refers to...

[0129]

[0130] Where ρ represents the first similarity value; A represents the interaction behavior feature vector at future time points; B represents the first standard vector; Cov(A, B) is the sample covariance of A and B; S A and S B These are the sample standard deviations of A and B, respectively.

[0131] S308. If the first similarity value is less than or equal to the preset threshold, the human-computer interaction behavior at the predicted future time will be abnormal.

[0132] This step can be understood as follows: when the first similarity value is less than or equal to the preset threshold, it indicates that there is a large deviation between the interaction behavior feature vector at future moments and the first standard vector, indicating that the human-computer interaction behavior at future moments has not been operated in accordance with the standard, thus an anomaly has occurred.

[0133] For example, by using the cosine similarity algorithm or the Pearson correlation coefficient algorithm, the first similarity value will be between [-1, 1]. The preset threshold can be set to 0.5. When the first similarity value is less than or equal to 0.5, the human-computer interaction behavior at future moments is judged to be abnormal.

[0134] Furthermore, when abnormal human-machine interaction behavior is predicted during maintenance operations, certain alarm measures need to be taken to prevent maintenance personnel from performing incorrect actions and causing losses. In some examples, after predicting abnormal interaction behavior at a future time, the following measures are also taken:

[0135] S3081. Determine the maintenance plan from the expert knowledge base.

[0136] Specifically, a maintenance plan refers to the preparatory work and standardized operating procedures for equipment maintenance.

[0137] The maintenance plans in the expert database include two categories: routine plans and emergency plans. Routine plans refer to maintenance plans developed for routine maintenance tasks, while emergency plans refer to standardized operating procedures for emergency measures developed for sudden events that occur during maintenance operations, mainly caused by maintenance personnel misoperation.

[0138] S3082. Send the maintenance plan to the maintenance personnel's mobile terminal and the control panel display interface closest to the equipment to be maintained.

[0139] This guides maintenance personnel to perform a series of operations such as cancellation, shutdown, reset, and reporting, thereby reducing the probability of erroneous operations and improving safety.

[0140] For example, see Figure 4 As shown, the mobile terminal includes a housing 100, a display screen 200, an audio player 300, a chipset 400, a microphone 500, a battery 600, and a linear motor 700; wherein, the chipset 400 may include a processor chip, a memory chip, and a communication chip, thereby enabling the reception and storage of maintenance plans.

[0141] In addition to displaying the specific details of the maintenance plan on the display screen 200, the mobile terminal also has functions such as sound and vibration alarm and voice communication through the audio player 300 and microphone 500. Thus, it can use sound and vibration alarms to warn maintenance personnel to pay attention to the operation.

[0142] See Figure 5 As shown, this application provides a human-computer interaction behavior prediction model training device 40, including: a data acquisition module 41, an extraction module 42, a calculation module 43, and a training module 44, wherein:

[0143] The acquisition module 41 is used to acquire multimodal image data of human-computer interaction behavior at multiple times in the maintenance area;

[0144] The extraction module 42 is used to extract features from multimodal image data to obtain interactive behavior feature vectors. The interactive behavior features include one or more of the following: operation status features, operation action features, and device response features.

[0145] The calculation module 43 is used to input the feature vectors of each interactive behavior and the corresponding standard vectors in the standard interactive behavior database into the preset similarity algorithm and output multiple similarity values.

[0146] The training module 44 is used to train the human-computer interaction behavior model based on the interaction behavior feature vector if all similarity values ​​are greater than the preset threshold, so as to obtain the human-computer interaction behavior prediction model.

[0147] The human-computer interaction behavior prediction model training device 40 in this application embodiment can execute the technical solution of the human-computer interaction behavior prediction model training method in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0148] Figure 6 A schematic diagram of the control device provided in an embodiment of this application. (See attached diagram.) Figure 6 As shown, the control device 50 includes a processor 51 and a memory 52. ​​The memory 52 is used to store computer programs, and the processor 51 is used to execute the computer programs stored in the memory 52 to implement the human-computer interaction behavior prediction model training method shown in any of the above method embodiments.

[0149] Specifically, the processor 51 and the memory 52 can communicate; for example, the processor 51 and the memory 52 communicate via a communication bus 53.

[0150] For example, the control device 50 may also include a communication interface, which may include a transmitter and / or a receiver.

[0151] For example, the processor 51 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0152] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the computer executes the above-described human-computer interaction behavior prediction model training method.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0156] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0158] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a human-computer interaction behavior prediction model, characterized in that, include: Collect multimodal image data of human-computer interaction behavior at multiple times in the maintenance and repair area; Feature extraction is performed on the multimodal image data to obtain the interactive behavior feature vector corresponding to each time moment. The interactive behavior features include one or more of the following: operation position features, operation action features, and device response features. Each interactive behavior feature vector and the corresponding standard interactive behavior feature vector in the standard interactive behavior database are input into a preset similarity algorithm to output multiple similarity values. If all the similarity values ​​are greater than the preset threshold, then the human-computer interaction behavior model is trained based on the interaction behavior feature vectors to obtain the human-computer interaction behavior prediction model.

2. The method according to claim 1, characterized in that, The step of extracting features from the multimodal image data to obtain the interaction behavior feature vector corresponding to each time step includes: The multimodal image data is preprocessed and time-sliced ​​to obtain the multimodal image frame corresponding to each time moment; The interactive behavior features are extracted from each of the multimodal image frames, and the interactive behavior features are converted into a numerical feature group, which includes multiple numerical values ​​that reflect different dimensions of the interactive behavior features. The numerical feature set is standardized to obtain the interaction behavior feature vector corresponding to each time step.

3. The method according to claim 2, characterized in that, The standard interactive behavior database includes a set of standard interactive behavior feature vectors with timestamps. The human-computer interaction behavior model includes multiple dimensional information numerical sub-models, wherein each dimensional information numerical sub-model is applicable to the numerical values ​​of the same dimensional information of the standard interaction behavior features in each standard interaction behavior feature vector; The dimensional information numerical sub-model represents obtaining the numerical value of the dimensional information in the standard interactive behavior feature vector set corresponding to the timestamp value by inputting a timestamp value.

4. The method according to claim 3, characterized in that, The step of training the human-computer interaction behavior model based on the feature vectors of each interaction behavior to obtain a human-computer interaction behavior prediction model includes: Each of the interactive behavior feature vectors is input into a clustering algorithm, which outputs multiple arrays of elements with timestamps, wherein each array of elements with timestamps represents a value that reflects the same dimension of information of the interactive behavior feature. Based on the array of elements with timestamps, the parameters in the corresponding dimensional information numerical sub-model are trained and adjusted to obtain the dimensional information numerical prediction sub-model. The human-computer interaction behavior prediction model is formed by combining the numerical prediction sub-models of each dimension.

5. The method according to claim 1, characterized in that, After training the human-computer interaction behavior model based on the feature vectors of each interaction behavior to obtain the human-computer interaction behavior prediction model, the method further includes: Based on the human-computer interaction behavior prediction model, output the interaction behavior feature vector for future moments; Determine the first standard vector corresponding to the future moment from the standard interaction behavior database; The interaction behavior feature vector at the future moment and the first standard vector are input into the preset similarity algorithm, and the first similarity value is output. If the first similarity value is less than or equal to the preset threshold, the predicted human-computer interaction behavior at the future moment is abnormal.

6. The method according to claim 5, characterized in that, After the interactive behavior for predicting future moments becomes abnormal, it also includes: A repair plan was determined from the expert knowledge base; The maintenance plan is sent to the maintenance personnel's mobile terminal and the control panel display interface closest to the equipment to be maintained.

7. The method according to any one of claims 1 to 6, characterized in that, The human-computer interaction behavior model refers to Where X1(t)...X n (t) represents the numerical values ​​of different dimensions reflecting the standard interactive behavior characteristics at time t, where a1, b1...q1 represent the parameters in the dimensional information numerical sub-model (t, X1(t)); a2, b2...q2 represent the parameters in the dimensional information numerical sub-model (t, X2(t)); a n b n ...q n Represents the dimensional information of the numerical submodel (t, X) n The parameters in (t)).

8. A training device for a human-computer interaction behavior prediction model, characterized in that, include: The module comprises an acquisition module, an extraction module, a calculation module, and a training module. The acquisition module is used to acquire multimodal image data of human-computer interaction behavior at multiple times in the maintenance area; The extraction module is used to extract features from the multimodal image data to obtain the interaction behavior feature vector corresponding to each time moment. The interaction behavior features include one or more of the following: operation position features, operation action features, and device response features. The calculation module is used to input each interactive behavior feature vector and the corresponding standard interactive behavior feature vector in the standard interactive behavior database into a preset similarity algorithm, and output multiple similarity values. The training module is used to train the human-computer interaction behavior model based on the interaction behavior feature vectors of each of the aforementioned similarity values ​​if all of them are greater than a preset threshold, thereby obtaining a human-computer interaction behavior prediction model.

9. A control device, characterized in that, The system includes a memory and a processor, wherein the memory stores program instructions, and the processor is used to call the program instructions in the memory to execute the human-computer interaction behavior prediction model training method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the human-computer interaction behavior prediction model training method according to any one of claims 1 to 7.

Images