Terminal equipment remote maintenance method, device, equipment and computer storage medium

Abstract

The present invention provides a method, apparatus, device and computer storage medium for remote maintenance of terminal equipment. By acquiring a target terminal forensic video to be identified, feature encoding is performed on the target terminal forensic video to obtain the target terminal forensic video feature, the target maintenance subject in the target terminal forensic video is obtained, and the first related maintenance subject corresponding to the target maintenance subject is determined. According to the maintenance subject optical flow feature of the first related maintenance subject and the corresponding significant attention variable, the target significant optical flow feature corresponding to the target maintenance subject is determined; the target terminal forensic video feature and the target significant optical flow feature corresponding to the target maintenance subject are subjected to feature aggregation operation to obtain the target aggregated feature, and the maintenance situation inspection information corresponding to the target terminal forensic video is determined according to the target aggregated feature. The present invention can help obtain accurate maintenance situation inspection information.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and computer storage medium for remote maintenance of terminal equipment. Background Art

[0002] With the rapid development of information technology, terminal equipment is increasingly used in various fields, such as computer terminals in corporate offices, automation control terminals in industrial environments, and various detection and treatment equipment terminals in the medical field. The normal operation of these terminal equipment is of vital importance to ensuring the smoothness of business processes, efficient and stable production, and the continuous development of medical services. In the use of terminal equipment, various faults will inevitably occur or regular maintenance will be required. Traditional terminal equipment maintenance methods usually require maintenance personnel to arrive at the equipment site for inspection, repair and maintenance operations. However, this on-site maintenance method has many limitations. On the one hand, in some large enterprises or widely distributed equipment networks, on-site maintenance requires a lot of manpower, material resources and time costs for personnel dispatch and transportation. On the other hand, for terminal equipment in some special environments, such as equipment located in dangerous environments or remote and difficult-to-reach areas, on-site maintenance may face many safety risks or be difficult to achieve at all. In order to solve these problems, remote maintenance technology for terminal equipment has emerged. Remote maintenance technology allows maintenance personnel to perform maintenance operations, fault diagnosis, etc. on equipment through network connections at locations far away from terminal equipment.

[0003] However, the existing terminal equipment remote maintenance technology still faces some challenges. For example, in the remote maintenance process, how to accurately obtain the maintenance status information of the terminal equipment is a key issue. Maintenance personnel need to obtain the status information of the equipment in some way, including the operating status of each component of the equipment, the operation actions of the maintenance personnel, etc., but the existing acquisition methods often have incomplete or inaccurate information. Moreover, in analyzing the acquired information to determine whether the maintenance operation is compliant and whether there are any errors, the existing technical means also have certain limitations. Summary of the invention

[0004] In view of this, the present invention provides a terminal device remote maintenance method, device, equipment and computer storage medium. The technical solution of the present invention is implemented as follows:

[0005] In a first aspect, the present invention provides a method for remote maintenance of a terminal device, the method comprising: obtaining a forensic video of a target terminal to be identified, performing feature encoding on the forensic video of the target terminal, obtaining features of the forensic video of the target terminal, wherein the features of the forensic video of the target terminal include optical flow features of a target video corresponding to the forensic video of the target terminal; wherein the forensic video of the target terminal is a forensic video obtained by forensic shooting of a terminal device to be maintained; obtaining a first related maintenance subject corresponding to the target maintenance subject in the forensic video of the target terminal according to different features of the maintenance situation through such a weighted summation method, a computer device can determine the target maintenance subject in the forensic video of the target terminal according to different features; determining the target significant optical flow features corresponding to the target maintenance subject based on the optical flow features of the maintenance subject of the first related maintenance subject and the corresponding significant attention variables, wherein the significant attention variables are obtained based on the degree of correlation between the optical flow features of the target video and the maintenance relationship feature representation, and the maintenance relationship feature representation is a feature representation representing the correlation of the maintenance subject; performing feature aggregation operation on the features of the forensic video of the target terminal and the target significant optical flow features corresponding to the target maintenance subject to obtain target aggregate features; and determining maintenance situation inspection information corresponding to the forensic video of the target terminal based on the target aggregate features.

[0006] In a second aspect, the present invention provides a terminal equipment remote maintenance device, the device comprising: a video feature encoding module, used to obtain a target terminal forensic video to be identified, feature encode the target terminal forensic video, and obtain target terminal forensic video features, wherein the target terminal forensic video features include target video optical flow features corresponding to the target terminal forensic video; wherein the target terminal forensic video is a forensic video obtained by forensic shooting of the terminal equipment to be maintained; a maintenance subject determination module, used to obtain the target maintenance subject in the target terminal forensic video, and determine the first related maintenance subject corresponding to the target maintenance subject; a significant feature determination module, used to determine the target maintenance subject based on the target terminal forensic video. The maintenance subject optical flow features of the first relevant maintenance subject and the corresponding significant attention variables are used to determine the target significant optical flow features corresponding to the target maintenance subject, wherein the significant attention variables are obtained based on the degree of correlation between the target video optical flow features and the maintenance relationship feature representation, and the maintenance relationship feature representation is a feature representation representing the correlation of the maintenance subjects; an optical flow feature aggregation module is used to perform feature aggregation operations on the target terminal forensic video features and the target significant optical flow features corresponding to the target maintenance subject to obtain target aggregation features; a maintenance situation inspection module is used to determine the maintenance situation inspection information corresponding to the target terminal forensic video based on the target aggregation features.

[0007] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the above-described method when executing the program.

[0008] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the above-described method when executed by a processor.

[0009] In an embodiment of the present invention, the cornerstone algorithm includes a video feature encoding algorithm, a significant feature mining algorithm and a feature aggregation algorithm. The significant feature mining algorithm can determine the example significant optical flow features corresponding to the example maintenance subject based on the optical flow features of the maintenance subject and the corresponding significant attention variables. The significant attention variables are obtained based on the correlation between the optical flow features of the example video and the maintenance relationship feature representation. The maintenance relationship feature representation is a feature representation representing the correlation of the maintenance subject. The significant attention variables are obtained according to the correlation between the optical flow features of the example video and the maintenance relationship feature representation. In this way, the importance of the maintenance subject optical flow features of the relevant maintenance subject to the features of the example maintenance subject can be obtained in combination with the semantics of the example terminal forensic video, so as to determine the significant attention variables, so that the obtained example features can help understand the semantics of the example terminal forensic video, and the video semantic learning ability of the cornerstone algorithm and the video understanding algorithm is improved, which helps to obtain accurate maintenance situation inspection information. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present invention and, together with the specification, are used to explain the technical solutions of the present invention.

[0011] Figure 1 A schematic diagram of the implementation flow of a terminal device remote maintenance method provided in an embodiment of the present invention.

[0012] Figure 2 A schematic diagram of the structure of a remote maintenance device for a terminal device provided in an embodiment of the present invention.

[0013] Figure 3 A hardware entity diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The embodiment of the present invention provides a terminal device remote maintenance method, which can be executed by a processor of a computer device, wherein the computer device may refer to a device with data processing capability, such as a server, a laptop, a tablet computer, a desktop computer, a smart TV, a mobile device, etc.

[0015] Figure 1A schematic diagram of a process flow for implementing a remote maintenance method for a terminal device provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0016] Step S110: Obtain the target terminal forensic video to be identified, perform feature encoding on the target terminal forensic video, and obtain target terminal forensic video features, wherein the target terminal forensic video features include target video optical flow features corresponding to the target terminal forensic video; wherein the target terminal forensic video is a forensic video obtained by forensic shooting of the terminal device to be maintained.

[0017] In an embodiment of the present invention, in a terminal device maintenance scenario, when a certain terminal device is remotely maintained, the computer device will obtain a video obtained by forensic shooting of the terminal device to be maintained, and this video is the target terminal forensic video. For example, in an office network environment of an enterprise, a computer as a terminal device fails, and the on-site maintenance personnel use a camera to shoot the faulty computer device and its surrounding related components, areas that may involve maintenance operations, etc. The video obtained by shooting becomes the target terminal forensic video. This video contains a lot of visual information related to the terminal device status, possible fault locations, and maintenance operations.

[0018] Then, the computer device performs feature encoding on the target terminal forensic video. Specifically, the computer device can use a variety of video feature encoding algorithms to achieve this goal. A feasible technical means is based on the convolutional neural network (CNN) method. For example, in a simple CNN structure, each frame of the video can be regarded as a two-dimensional image, and the convolution kernel in the convolution layer will slide on the image to perform a convolution operation. For a convolution kernel of size M×N and a video frame image of size H×W (where M≤H, N≤W), the calculation formula for the convolution operation is: ,in, is the value of the convolution result at position (i,j), is the pixel value of the video frame image at position (i+m, j+n), is the value of the convolution kernel at position (m,n). Through multiple convolution layers, pooling layers (such as maximum pooling, which takes the maximum value in a small area to reduce the amount of data), and other operations, the computer device can extract various features from the video frame, such as edge features, texture features, etc.

[0019] When processing videos, in addition to processing each frame individually, it is also necessary to consider the relationship between frames, which involves the optical flow features of the target video. Optical flow refers to the speed and direction of movement of pixels in an image. For target terminal forensic video, which contains multiple frames of video data, the optical flow features of the target video can reflect the movement of different objects in the video (such as various components of the terminal equipment, operating tools of maintenance personnel, etc.) between consecutive frames. Computer equipment can obtain optical flow information by calculating the displacement of pixels between adjacent frames. For example, suppose that in the tth frame and the t+1th frame of the target terminal forensic video, the horizontal coordinate of a pixel point changes from Change to , the vertical coordinates are from Change to , then the horizontal optical flow component u and vertical optical flow component v of this pixel can be calculated by the following formula: ; .

[0020] By calculating the optical flow of multiple pixels in the video, the optical flow features of the target video can be obtained. This feature, together with the features extracted separately from the video frames mentioned above (such as texture, edge, etc.), constitutes the target terminal forensic video feature.

[0021] Step S120: Acquire a target maintenance subject in the target terminal evidence collection video, and determine a first related maintenance subject corresponding to the target maintenance subject.

[0022] In an embodiment of the present invention, in the scene presented by the target terminal forensic video, the target maintenance subject is a specific element or object directly related to the maintenance of the terminal device. For example, in the scenario of remote maintenance of a server, the target terminal forensic video captures the server's hardware structure, connection lines, and the hand movements of the maintenance personnel who are performing maintenance operations. The fault location here (such as the flashing area of ​​a hard disk indicator light on a faulty hard disk in the server) or the hands of the maintenance personnel (because the hands are the key part of performing maintenance operations) can be regarded as the target maintenance subject. These target maintenance subjects are of key significance in the judgment of the entire maintenance situation. They are important focus points for subsequent analysis of whether the maintenance operation is correct and whether the fault location is accurate.

[0023] After determining the target maintenance subject, the computer device needs to find the first related maintenance subject corresponding to it. The first related maintenance subject is another element or object that has some association with the target maintenance subject. The computer device can use a target detection algorithm and an association analysis algorithm to implement this operation.

[0024] For target detection algorithms, computer devices can use target detection frameworks in deep learning, such as FasterR-CNN (Region-based Convolutional Neural Network). Faster R-CNN uses a convolutional neural network to extract features from each frame of the target terminal forensic video. Assuming that the input video frame image is I, the feature map after the convolution layer is F. The calculation process involves convolution operations between multiple convolution kernels and image pixels, such as the convolution formula mentioned above, which will not be repeated here. The feature map F is obtained through multiple layers of convolution, and then the region proposal network (RPN) is used to generate region proposals (RegionProposals) that may contain the target on the feature map F. For each region proposal, classification and regression operations are performed through the fully connected layer to determine the category and location of the target (for example, in the server maintenance scenario, it is determined to be a hardware device such as a hard disk or a memory stick, or a target such as the hands of a maintenance person) .

[0025] After the target maintenance subject is determined by the target detection algorithm, the computer device begins to analyze the first related maintenance subject related to it. Taking the server maintenance scenario as an example, if the target maintenance subject is a faulty hard disk, then the first related maintenance subject related to it may include the data cable connecting the hard disk, the power cable, and the hard disk bracket where the hard disk is located. From the perspective of technical implementation, the computer device can use an association analysis algorithm. A simple association analysis method is a method based on distance measurement.

[0026] Assume that in a certain frame of the target terminal forensic video, the center coordinates of the target maintenance subject (such as a hard disk) are , the center coordinates of other elements (such as an end point of a data line) are , the computer device can calculate the Euclidean distance d between them, the calculation formula is: .

[0027] If this distance is less than a certain pre-set threshold (this threshold can be determined based on experience or analysis of different terminal equipment maintenance scenarios), then this element (data cable) can be considered to be the first relevant maintenance subject related to the target maintenance subject (hard disk). In addition to distance-based association analysis, association analysis based on semantic information can also be used. For example, if the target maintenance subject is the hand of a maintenance personnel, then the tool semantically related to the hand's action (such as a screwdriver, if the hand is operating a screwdriver to disassemble the server casing) can also be regarded as the first relevant maintenance subject. The computer device can perform semantic analysis on the elements around the target maintenance subject, such as using a pre-trained semantic classification model (which can be a model pre-trained based on a large-scale image dataset, such as an ImageNet pre-trained model) to classify each element, and then determine the first relevant maintenance subject according to predefined semantic association rules (such as association rules such as "hand-operation-tool").

[0028] In this way, the computer device can accurately determine the target maintenance subject from the target terminal evidence video in step S120, and find the first related maintenance subject related to it. This lays the foundation for further analyzing the maintenance situation based on the relationship between these subjects in the subsequent steps. The determination of these related maintenance subjects helps to build a more complete maintenance scenario model, so that the computer device can analyze and judge whether the entire maintenance operation complies with the specifications, whether there are potential problems, etc. from the perspective of multiple related elements.

[0029] Step S130: Determine the target salient optical flow features corresponding to the target maintenance subject based on the maintenance subject optical flow features of the first related maintenance subject and the corresponding salient attention variables, wherein the salient attention variables are obtained based on the degree of correlation between the target video optical flow features and the maintenance relationship feature representation, and the maintenance relationship feature representation is a feature representation representing the correlation of the maintenance subjects.

[0030] In step S130, in the target terminal evidence video, each maintenance subject (such as the target maintenance subject determined in the previous step and its related first related maintenance subject) has its own motion pattern, which is reflected in the video through optical flow features. The optical flow features of the maintenance subject reflect the motion state of the maintenance subject between video frames, including information such as motion direction and speed. For example, in the scenario of maintaining a network switch, if the target maintenance subject is a faulty port and the first related maintenance subject is the network cable connected to the port, then the maintenance subject optical flow features of the network cable (as the first related maintenance subject) may be reflected as optical flow information such as the direction of pulling out from the port and the speed (such as how fast the pulling out is) generated by the maintenance personnel's operation (such as plugging and unplugging the network cable).

[0031] The significant attention variable is another key concept in this step. It can be understood as a weight that measures the importance of the optical flow features of different maintenance subjects to the target maintenance subject. This variable is obtained based on the correlation between the target video optical flow features and the maintenance relationship feature representation. The maintenance relationship feature representation is a feature representation that represents the correlation of maintenance subjects, which describes the intrinsic connection between maintenance subjects in the maintenance operation scenario. For example, in the above-mentioned network switch maintenance scenario, if the target video optical flow feature shows that there are frequent light and shadow changes near the port within a certain period of time (possibly due to fault flashing or maintenance operations), and this light and shadow change is associated with the plugging and unplugging operation of the network cable (the maintenance relationship feature representation of the network cable), then this degree of association will affect the significant attention variable of the network cable (as the first related maintenance subject) relative to the port (the target maintenance subject).

[0032] When determining the significant attention variable, the computer device can use some statistical analysis methods or machine learning algorithms. For example, the correlation between the target video optical flow feature and the maintenance relationship feature representation can be measured by calculating the covariance. Suppose the target video optical flow feature vector is , the maintenance relationship feature representation vector is , the covariance calculation formula is: ;in, are the means of vectors X and Y, respectively. The covariance value can reflect the correlation between the two to a certain extent, and then determine the significant attention variable according to the pre-set mapping rule. For example, if the covariance value is high, it indicates that the correlation is strong, and a higher significant attention variable can be set, indicating that the first related maintenance subject is more important to the target maintenance subject.

[0033] The process of determining the target significant optical flow features corresponding to the target maintenance subject by the computer device is based on the above maintenance subject optical flow features and significant attention variables. Taking a computer motherboard repair scenario as an example, the target maintenance subject is a capacitor on the motherboard, and the first relevant maintenance subject may be the surrounding circuit lines and heat sinks, etc. If the maintenance subject optical flow features of the heat sink (as the first relevant maintenance subject) show that it has moved during the repair process (perhaps it was moved to better observe the capacitor), its significant attention variable is determined to be a higher value based on the correlation calculation mentioned above (assuming that this is because the movement of the heat sink is closely related to the operational process of capacitor troubleshooting).

[0034] The computer device can determine the target salient optical flow features corresponding to the target maintenance subject by weighted calculation. Suppose the maintenance subject optical flow feature vector of the first related maintenance subject is , and its corresponding significant variable of interest is , then the target saliency optical flow feature vector corresponding to the target maintenance subject is It can be obtained by the following calculation: ;in, This weighted calculation method makes it possible to fully consider the importance of different first-related maintenance subjects (reflected by the significant attention variables) and their own optical flow characteristics when determining the target salient optical flow characteristics of the target maintenance subject.

[0035] Step S140: performing a feature aggregation operation on the target terminal forensic video features and the target saliency optical flow features corresponding to the target maintenance subject to obtain target aggregate features.

[0036] In step S140, the target terminal forensic video features are obtained in step S110, and include various feature information extracted from the target terminal forensic video, which can reflect the status, movement, etc. of each element in the video. For example, when performing remote maintenance on a mobile device, the target terminal forensic video features may include the features of the device screen display content, the features of the device appearance components, and the features of the device surrounding environment in the video. These features are obtained through a specific video feature encoding algorithm, such as using a convolutional neural network (CNN) in deep learning to extract features, and converting the original video data into a computer-processable feature vector through operations such as convolutional layers and pooling layers.

[0037] The target significant optical flow feature corresponding to the target maintenance subject is determined in step S130. It is a feature related to the target maintenance subject obtained by comprehensively considering the maintenance subject optical flow feature of the first related maintenance subject and the corresponding significant attention variable. Taking server maintenance as an example, if the target maintenance subject is a hard disk in the server, the target significant optical flow feature may include the motion features of other components related to the hard disk (such as the data cable and power cable connected to the hard disk as the first related maintenance subject) in the video, and these features are weighted according to their importance to the hard disk maintenance (significant attention variable).

[0038] The purpose of the feature aggregation operation performed by the computer device is to fuse these two different types of features together to obtain a more comprehensive and representative target aggregate feature. The feature aggregation operation can be implemented by a variety of technical means. One feasible method is based on vector operations.

[0039] Assume that the target terminal forensic video feature is represented as a vector , the target saliency optical flow feature corresponding to the target maintenance subject is represented as a vector Computer devices can use simple vector concatenation operations to achieve feature aggregation. For example, concatenate vector U to the back of vector V to obtain a new vector , this new vector W is the target aggregate feature. This vector concatenation operation is simple and direct, and can completely combine the two feature information, so that the subsequent analysis can take into account both the original features in the target terminal forensic video and the significant optical flow features related to the target maintenance subject.

[0040] Another possible technical means is the weighted summation method. The computer device can assign weights to the target terminal forensic video features and the target salient optical flow features corresponding to the target maintenance subject. Let the weight of the target terminal forensic video features be a, and the weight of the target salient optical flow features corresponding to the target maintenance subject be b. Then the target aggregate feature vector It can be calculated by the following formula: (The value range of i is determined according to the actual feature dimension), where v i is the target terminal forensic video feature representation vector, u i is the target saliency optical flow feature representation vector.

[0041] Through this weighted summation method, the computer device can adjust the proportion of different features in the aggregated features according to their importance in judging the maintenance situation. For example, if in a certain terminal equipment maintenance scenario, the target salient optical flow features corresponding to the target maintenance subject are more critical for judging the maintenance situation, then the value of b can be appropriately increased, so that the target salient optical flow features corresponding to the target maintenance subject occupy a more important position in the target aggregated features.

[0042] Whether using vector concatenation or weighted summation, the result of the feature aggregation operation - the target aggregation feature, provides a more comprehensive feature representation for computer equipment that better reflects the maintenance status. This target aggregation feature contains multiple aspects of information in the target terminal forensic video and information closely related to the target maintenance subject, laying a solid foundation for accurately determining the maintenance status inspection information corresponding to the target terminal forensic video in subsequent steps. It enables computer equipment to analyze the maintenance status of terminal equipment from a more comprehensive perspective, avoiding the one-sidedness that may be caused by using a single feature, and improving the accuracy and reliability of maintenance status judgment.

[0043] Step S150: Determine maintenance status inspection information corresponding to the target terminal forensic video based on the target aggregation features.

[0044] In step S150, the target aggregate feature is obtained by the computer device in step S140 by performing a feature aggregation operation on the target terminal forensic video feature and the target saliency optical flow feature corresponding to the target maintenance subject. This target aggregate feature integrates various information in the target terminal forensic video, such as texture, edge and other features extracted from the video frame, and processed information of the optical flow feature related to the target maintenance subject. This information exists in the form of a specific vector or data structure, which can comprehensively reflect the maintenance-related content in the target terminal forensic video.

[0045] The maintenance situation inspection information is intended to determine various situations when the maintenance personnel operate the terminal equipment to be maintained, such as whether the maintenance personnel's maintenance actions are compliant, whether maintenance errors occur, whether the maintenance operation process is complete, etc. The process of computer equipment using target aggregate features to determine these maintenance situation inspection information involves complex data analysis and pattern recognition technology. Computer equipment can use machine learning algorithms to achieve this purpose. For example, a classification algorithm based on a neural network can be used. Assume that there is a pre-trained neural network model, and the input of this model is the target aggregate feature vector. The neural network consists of multiple neuron layers, including an input layer, a hidden layer, and an output layer.

[0046] When the target aggregate feature vector is input to the input layer of the neural network, information is transmitted between neurons through weighted connections. Each neuron performs a weighted summation of the input and performs a nonlinear transformation through an activation function. Suppose the input of the i-th neuron in the input layer is (Here is an element in the target aggregate feature vector), and the corresponding weight is (the weight connecting the i-th neuron in the input layer to the j-th neuron in the hidden layer), and the bias of the j-th neuron in the hidden layer is , then the input of the jth neuron in the hidden layer is It can be calculated by the following formula: , and then through the activation function (such as ReLU function, that is, f(x)=max(0,x)) Perform nonlinear transformation to obtain the output of the hidden layer neurons.

[0047] After being processed by multiple hidden layers, the information is finally passed to the output layer. The number of neurons in the output layer depends on the number of categories of maintenance test information to be predicted. For example, if you want to judge whether the maintenance action is compliant (compliant is 1, non-compliant is 0) and whether there is a maintenance error (error is 1, no error is 0), then the output layer has two neurons.

[0048] The computer device determines the maintenance inspection information based on the output value of the output layer neuron. For example, if the output value of the first neuron in the output layer is close to 1 and the output value of the second neuron is close to 0, it can be judged that the maintenance action is compliant and there is no maintenance error.

[0049] Taking the server maintenance scenario as an example, if the target terminal forensic video is a video of the server maintenance process, the target aggregation features include the relevant features of the various server components (such as hard disks, memory sticks, etc.) during the maintenance process and the motion features of the tools (such as screwdrivers, etc.) used by the maintenance personnel. When this information is input into the neural network model, if the output result shows that the maintenance action is not compliant, it may be because the neural network recognizes that the maintenance personnel's actions when operating the hard disk do not conform to the standard server maintenance process (such as excessive force when plugging and unplugging the hard disk or the wrong operation sequence, etc.). This non-compliant action pattern is identified by the neural network's learning of a large amount of normal and abnormal maintenance operation data.

[0050] Another technical means is based on the decision tree algorithm. The decision tree constructs a decision tree by making conditional judgments on different attributes in the target aggregate features. For example, a decision tree is constructed based on the temperature change characteristics of server components and the speed characteristics of the maintenance personnel's hand movements in the target aggregate features. At each node of the decision tree, a branch judgment is made based on the value of an attribute. Finally, the leaf node represents the result of the maintenance status inspection information. For example, if a leaf node of the decision tree is expressed as "the temperature rises abnormally and the hand movement speed is too fast, which is judged as a maintenance error", when the relevant attributes in the target aggregate features meet the conditions of this leaf node, the computer equipment can be judged as a maintenance error.

[0051] Through these technical means, the computer device can effectively use the target aggregation features to accurately determine the maintenance inspection information corresponding to the target terminal forensic video in step S150, thereby realizing remote monitoring and evaluation of the terminal equipment maintenance. This helps to timely discover problems in the maintenance process and improve the quality and efficiency of terminal equipment maintenance.

[0052] In one implementation, step S130, based on the maintenance subject optical flow feature of the first related maintenance subject and the corresponding significant attention variable, determines the target significant optical flow feature corresponding to the target maintenance subject, including:

[0053] Step S131: obtaining a visual spatiotemporal relationship diagram consisting of a first related maintenance subject and a target maintenance subject;

[0054] Step S132: for a relationship graph maintenance subject in the visual spatiotemporal relationship graph, obtaining a maintenance relationship feature representation indicating the correlation between the relationship graph maintenance subject and an adjacent maintenance subject;

[0055] Step S133: obtaining a feature correlation degree based on the maintenance relationship feature representation and the target video optical flow feature, and determining a significant attention variable corresponding to the adjacent maintenance subject based on the feature correlation degree;

[0056] Step S134: determining the target significant optical flow feature corresponding to the maintenance subject of the relationship graph based on the significant attention variables corresponding to the adjacent maintenance subject and the maintenance subject optical flow feature of the adjacent maintenance subject;

[0057] Step S135: extracting the target salient optical flow features corresponding to the target maintenance subject from the target salient optical flow features corresponding to each relationship graph maintenance subject of the visual spatiotemporal relationship graph.

[0058] In step S131, the computer device obtains a visual spatiotemporal relationship graph consisting of the first relevant maintenance subject and the target maintenance subject. The visual spatiotemporal relationship graph is a data structure that can represent the relationship between different maintenance subjects in the visual space and time dimension of the video. In the scenario of terminal equipment maintenance, for example, when multiple servers in a computer room are maintained, the target maintenance subject may be a specific component that fails in a server, such as a hard disk, and the first relevant maintenance subject may include a data cable and a power cable connected to the hard disk, as well as other server components adjacent to the same cabinet. When the computer device constructs a visual spatiotemporal relationship graph, these maintenance subjects are used as nodes, and the edges between the nodes represent the relationship between them. These relationships can be determined based on a variety of factors, such as spatial proximity, the order of precedence in the maintenance operation process, or functional relevance. From a technical point of view, the computer device can use a graph construction algorithm to create this visual spatiotemporal relationship graph. For example, the various maintenance subject elements in the target terminal forensic video can be traversed based on a depth-first search (DFS) algorithm or a breadth-first search (BFS) algorithm to determine the connection relationship between them, thereby constructing a visual spatiotemporal relationship graph.

[0059] Next is step S132, where for the relationship graph maintenance subject in the visual spatiotemporal relationship graph, the computer device obtains a maintenance relationship feature representation indicating the correlation between the relationship graph maintenance subject and the adjacent maintenance subject. The relationship graph maintenance subject is a node in the visual spatiotemporal relationship graph, that is, it represents the maintenance subject. Taking the previous server maintenance as an example, in the visual spatiotemporal relationship graph, if a relationship graph maintenance subject is a hard disk, its adjacent maintenance subjects may be the data cable and power cable connected to it. The maintenance relationship feature representation aims to quantify the correlation of such adjacent relationships. The computer device can use feature extraction technology to obtain the maintenance relationship feature representation. For example, for spatial relationships, the center coordinate distance between the adjacent maintenance subject and the relationship graph maintenance subject can be calculated and normalized to a specific interval as a maintenance relationship feature representation of a spatial relationship. Assume that the center coordinates of the hard disk are ( ), the center coordinate of one end point of the data line is , then the spatial distance d can be expressed by the formula Calculate, and then normalize d to the interval [0, 1]. This normalized value is a maintenance relationship feature representation of a spatial relationship. For functional relationships, they can be determined based on a predefined server component functional association knowledge base. If the hard disk and the data cable are functionally connected in a data transmission relationship, a specific encoding (such as binary encoding) can be used to represent the maintenance relationship feature representation of this functional relationship.

[0060] Then comes step S133, where the computer device obtains the feature correlation degree based on the maintenance relationship feature representation and the target video optical flow feature, and determines the significant attention variable corresponding to the adjacent maintenance subject based on the feature correlation degree. The target video optical flow feature reflects the motion information in the target terminal forensic video. In the server maintenance scenario, if the target video optical flow feature shows that the light and shadow near the hard disk flicker regularly within a certain period of time (possibly due to data reading and writing or fault prompts), and the maintenance relationship feature representation of the data line (as an adjacent maintenance subject) shows that it is closely related to the hard disk in function (such as responsible for data transmission), then there is a high degree of feature correlation between them. The computer device can use a variety of methods to calculate the feature correlation degree. One feasible method is to calculate the Pearson correlation coefficient. Suppose the maintenance relationship feature representation vector is , the target video optical flow feature vector is , the calculation formula of Pearson correlation coefficient r is: ;in, are the means of vectors X and Y, respectively. Based on this feature correlation, the significant attention variable is determined. For example, a linear mapping relationship can be set. If the value of the feature correlation r is between 0.8 and 1, indicating a high correlation, the significant attention variable can be set to a higher value, such as 0.8; if the value of r is between 0 and 0.2, indicating a low correlation, the significant attention variable can be set to 0.1.

[0061] In step S134, the computer device determines the target significant optical flow feature corresponding to the maintenance subject of the relationship graph based on the significant attention variable corresponding to the adjacent maintenance subject and the maintenance subject optical flow feature of the adjacent maintenance subject. Continuing with the server maintenance as an example, assume that the maintenance subject optical flow feature of the data line (adjacent maintenance subject) shows that it has slight vibrations within a certain period of time (possibly due to the rotation of the fan inside the server or the influence of other devices), and its significant attention variable is 0.6 (obtained based on the correlation with the hard disk determined in the previous step). The computer device can use a weighted summation method to determine the target significant optical flow feature corresponding to the maintenance subject (hard disk) of the relationship graph. Suppose the maintenance subject optical flow feature vector of the data line is , then the relationship graph maintains the target saliency optical flow feature vector corresponding to the subject The formula can be This weighted calculation method takes into account the influence of the adjacent maintenance subject on the maintenance subject of the relationship graph (reflected by the significant attention variable) and the optical flow characteristics of the adjacent maintenance subject itself.

[0062] Finally, step S135 is performed, in which the computer device extracts the target significant optical flow features corresponding to the target maintenance subject from the target significant optical flow features corresponding to each relationship graph maintenance subject of the visual spatiotemporal relationship graph. In the entire visual spatiotemporal relationship graph, there are multiple relationship graph maintenance subjects (such as different server components), and each relationship graph maintenance subject has its corresponding target significant optical flow features. In the previous example, the target significant optical flow features corresponding to the relationship graph maintenance subjects (such as data lines) related to the hard disk (target maintenance subject) have been determined, and now the computer device accurately extracts the target significant optical flow features corresponding to the hard disk itself from these results. This can be achieved by indexing the identifiers of the nodes (relationship graph maintenance subjects) in the visual spatiotemporal relationship graph. For example, each relationship graph maintenance subject is assigned a unique identifier when constructing the visual spatiotemporal relationship graph, and the computer device can search and extract its corresponding target significant optical flow features based on the identifier of the hard disk.

[0063] Through steps S131-S135, the computer device can comprehensively consider multiple factors such as the visual spatiotemporal relationship, correlation, and optical flow characteristics between the maintenance subjects, and accurately determine the target saliency optical flow characteristics corresponding to the target maintenance subject.

[0064] In practical applications, the accurate execution of this series of steps is of great significance for the remote maintenance of terminal equipment. For example, in the maintenance of server clusters in large data centers, due to the large number of devices and complex maintenance operations, accurately determining the target salient optical flow features corresponding to the target maintenance subject can help remote maintenance personnel quickly locate possible problems. If the target maintenance subject is the power supply module of a key server in the server cluster, through this series of steps, the computer equipment can comprehensively consider various information related to other components of the power supply module (such as the circuit switch in the power distribution cabinet, the power cord connected to the server, etc. as the first related maintenance subject), including the spatial relationship, functional relationship, optical flow features and correlation between them, so as to accurately judge the maintenance status of the power supply module. This helps to timely discover potential fault hazards, such as abnormal vibration of the power cord (discovered by optical flow features and related calculations) may indicate an impending circuit connection problem, so as to take corresponding maintenance measures to avoid large-scale failures of the server cluster caused by power problems and ensure the normal operation of the data center.

[0065] The same is true for terminal equipment maintenance in the field of industrial control, such as maintenance of equipment on automated production lines. Assuming that the target maintenance subject is a sensor on a certain production equipment, the first related maintenance subject may include the signal line connecting the sensor, the bracket for installing the sensor, and other related mechanical parts around it. By constructing a visual spatiotemporal relationship graph, the computer device can clearly represent the relationship between these maintenance subjects. Obtaining maintenance relationship feature representation can quantify the correlation between them, such as the signal transmission relationship between the signal line and the sensor, the fixing and supporting relationship of the bracket to the sensor, etc. Based on the target video optical flow feature, the feature correlation degree is calculated, and the significant attention variable is determined. Then, the target significant optical flow feature is calculated, and finally the target significant optical flow feature corresponding to the sensor is extracted. This series of operations can provide a comprehensive basis for judging the maintenance status of the sensor. For example, if the target significant optical flow feature of the sensor shows that the light and shadow changes around it are abnormal (possibly due to a fault or an impending fault), combined with the information of other related maintenance subjects, it can be accurately judged whether it is a problem with the sensor itself or a problem caused by surrounding components (such as loose signal lines), so as to carry out targeted maintenance and improve the production efficiency and stability of the automated production line.

[0066] In other possible scenarios, such as in the maintenance of medical equipment, such as the maintenance of X-ray machines in hospitals, the target maintenance subject may be the tube of the X-ray machine, and the first related maintenance subject includes the high-voltage line that provides power to the tube, the cooling system that cools the tube, and the protective device around the tube. The computer device operates according to the above steps, and when constructing the visual spatiotemporal relationship graph, these components are used as nodes, and edges are constructed according to their physical connections and functional relationships. Obtaining the maintenance relationship feature representation can reflect the power supply relationship between the high-voltage line and the tube, the heat exchange relationship between the cooling system and the tube, etc. The target significant optical flow feature corresponding to the tube is determined by calculating the feature correlation degree and the significant attention variable. If the target significant optical flow feature shows that there are abnormalities in the light and shadow around the tube and the movement of the components, combined with other relevant information, it can be accurately judged whether there is a hidden danger of failure in the tube, such as whether the abnormal operation of the tube is caused by the unstable current of the high-voltage line (obtained through calculation and analysis related to the high-voltage line), or whether the failure of the cooling system (through analysis related to the cooling system) affects the normal operation of the tube. This helps to maintain medical equipment in a timely manner, ensure the normal use of medical equipment, and ensure the quality of medical services.

[0067] In one implementation, the target salient optical flow features corresponding to the maintenance subject of the relationship graph are output based on a salient feature mining algorithm, and the salient feature mining algorithm includes at least one target transformation structure. Based on this, step S134, based on the salient attention variables corresponding to the adjacent maintenance subjects and the maintenance subject optical flow features of the adjacent maintenance subjects, determines the target salient optical flow features corresponding to the maintenance subject of the relationship graph, including:

[0068] Step S1341: the maintenance subject optical flow features and maintenance relationship feature representations of adjacent maintenance subjects are transferred into the target transformation structure for transformation, so as to obtain the first significant optical flow features corresponding to the maintenance subject in the relationship graph;

[0069] Step S1342: Determine the target significant optical flow feature corresponding to the relationship graph maintenance subject based on the first significant optical flow feature corresponding to the relationship graph maintenance subject and the significant attention variable corresponding to the corresponding adjacent maintenance subject.

[0070] In step S1341, the computer device inputs the maintenance subject optical flow features of the adjacent maintenance subjects and the maintenance relationship feature representation into the target transformation structure for transformation, and obtains the first significant optical flow feature corresponding to the maintenance subject of the relationship graph. The target transformation structure is an important part of the significant feature mining algorithm. It can be a hidden layer or a series of computing units, which is designed to perform specific transformation operations on the input features to mine information that is more conducive to determining the target significant optical flow features.

[0071] Taking the server maintenance scenario as an example, assuming that the adjacent maintenance subject is the data cable connecting the server hard disk, the optical flow feature of the maintenance subject may reflect the movement state of the data cable during the maintenance operation, such as the moving direction and speed during the plug-in operation. Assume that the optical flow feature vector of this maintenance subject is At the same time, the maintenance relationship feature representation quantifies the relationship between the data line and the maintenance subject of the relationship graph (such as a hard disk), which may include information such as spatial relationship and functional relationship. Assume that the maintenance relationship feature representation vector is The computer device passes these two vectors into the target transformation structure. In the target transformation structure, a variety of technical means can be used for transformation. One feasible method is linear transformation. For example, let the weight matrix in the target transformation structure be W (dimension is p×(m+n)), and the bias vector be b (dimension is p), where p is the dimension of the transformed feature. Then the first salient optical flow feature vector corresponding to the main body of the transformed relationship graph maintenance is It can be calculated by the following formula: , where [F; R] means concatenating vectors F and R into a new vector by column. This linear transformation re-weights the input features through the weight matrix W and adjusts the offset through the bias vector b to obtain the first salient optical flow features with different representations. This transformation helps to fuse the maintenance subject optical flow features and maintenance relationship feature representations of adjacent maintenance subjects, and mine the potential relationship between them, thereby providing more valuable information for determining the target salient optical flow features corresponding to the maintenance subject of the relationship graph.

[0072] In addition to linear transformation, the target transformation structure can also use nonlinear transformation methods. For example, an activation function such as ReLU (Rectified Linear Unit) can be used in the target transformation structure. After the above linear transformation, each element g' of the resulting vector G' is i Processed by ReLU function, that is, g' 'i =max(0, g' i ), and get the new vector G''=[g''1, g''2, …, g'' p ]. This nonlinear transformation can introduce nonlinear factors, so that the target transformation structure can better fit complex relationships. For example, it can better capture the nonlinear relationship between the maintenance subject optical flow features and the maintenance relationship feature representation, which may be very important in actual terminal equipment maintenance scenarios. For example, in some cases, there may be a nonlinear association between the slight movement of the data line (maintenance subject optical flow features) and the complex functional relationship between it and the hard disk (maintenance relationship feature representation). The role of this association in determining the first significant optical flow feature can be more accurately represented through nonlinear transformation.

[0073] Then comes step S1342, where the computer device determines the target significant optical flow feature corresponding to the relationship graph maintenance subject based on the first significant optical flow feature corresponding to the relationship graph maintenance subject and the significant attention variable corresponding to the corresponding adjacent maintenance subject. The significant attention variable reflects the importance of the adjacent maintenance subject to the relationship graph maintenance subject. Continuing with the server maintenance scenario as an example, if the significant attention variable is w (for example, w=0.6, indicating that the data line is more important to the hard disk), the first significant optical flow feature vector corresponding to the relationship graph maintenance subject (hard disk) is G''=[g''1, g''2,…,g'' p ].

[0074] The computer device can use a weighted calculation method to determine the target saliency optical flow feature vector G=[g1, g2,…, g p The specific calculation formula is: g i =w×g'' i .

[0075] This weighted calculation method enables the target salient optical flow features corresponding to the relationship graph maintenance subject to fully consider the importance of the adjacent maintenance subjects (reflected by the salient attention variables) and the first salient optical flow features after the target transformation structure transformation. For example, if the salient attention variable is high, it means that the adjacent maintenance subject (such as the data line) has a greater impact on the relationship graph maintenance subject (such as the hard disk). Then, when determining the target salient optical flow features, the first salient optical flow features will be included in the calculation with a larger weight. Conversely, if the salient attention variable is low, the weight of the first salient optical flow features in determining the target salient optical flow features is relatively small.

[0076] This calculation method is of great significance in the maintenance of terminal equipment in actual industrial environments. For example, in the equipment maintenance scenario on an automated production line, assume that the maintenance subject of the relationship graph is a key sensor, and the adjacent maintenance subject is the signal line connecting the sensor. The normal operation of the sensor is crucial for the monitoring and control of the entire production line. The maintenance subject optical flow feature of the signal line may reflect the vibration of the signal line during the operation of the equipment (which may be caused by mechanical vibration of the equipment or electromagnetic interference, etc.), and the maintenance relationship feature representation may contain information such as the signal transmission relationship between the signal line and the sensor.

[0077] The first significant optical flow feature corresponding to the sensor is obtained by transforming the maintenance subject optical flow feature and maintenance relationship feature representation of the signal line into the target transformation structure. This first significant optical flow feature may contain deeper relationship information between the signal line and the sensor, such as the potential impact of the vibration of the signal line on the stability of the sensor signal transmission. Then, according to the significant attention variable of the signal line relative to the sensor (this variable may be determined according to the importance of the signal line to the normal operation of the sensor, for example, if the stability of the signal line has a great impact on the accuracy of the sensor's readings, then the significant attention variable will be high), the target significant optical flow feature corresponding to the sensor is obtained by weighted calculation. This target significant optical flow feature can accurately reflect the role of the signal line's impact on the sensor in the overall maintenance assessment of the sensor, and is helpful to timely discover potential maintenance problems, such as whether the slight vibration of the signal line will cause the sensor to fail or inaccurate readings.

[0078] In one implementation, step S1341, the maintenance subject optical flow features and maintenance relationship feature representations of adjacent maintenance subjects are transferred into the target transformation structure for transformation to obtain the first significant optical flow features corresponding to the maintenance subject of the relationship graph, including:

[0079] Step S13411: determining a target integration strategy based on the maintenance subject correlation between the maintenance subject and the adjacent maintenance subjects in the relationship graph, the target integration strategy being feature superposition or feature difference;

[0080] Step S13412: integrating the maintenance subject optical flow features and maintenance relationship feature representations of adjacent maintenance subjects based on the target integration strategy to obtain the integrated feature representation corresponding to the maintenance subject in the relationship graph;

[0081] Step S13413: transform the integrated feature representation through the weights and biases in the target transformation structure to obtain the first significant optical flow feature corresponding to the relationship graph maintenance subject.

[0082] First, in step S13411, the computer device determines the target integration strategy based on the maintenance subject correlation between the maintenance subject in the relationship graph and the adjacent maintenance subject. The target integration strategy is feature superposition or feature difference. In the scenario of terminal device maintenance, the maintenance subject correlation describes the inherent connection between the maintenance subject in the relationship graph and the adjacent maintenance subject in the maintenance operation context. For example, when maintaining a computer system, the maintenance subject in the relationship graph is the central processing unit (CPU) of the computer, and the adjacent maintenance subject is the cooling fan. From the perspective of functional relationship, the function of the cooling fan is to dissipate heat for the CPU, and there is a close connection between the two.

[0083] For such closely related maintenance subjects, the computer device may determine to adopt a target integration strategy of feature superposition. Assume that the optical flow feature vector of the maintenance subject of the cooling fan is , which reflects the motion state of the cooling fan in the video (such as the light and shadow changes caused by the speed change, etc.), and the maintenance relationship feature representation vector is , which contains quantitative information about the functional relationship between the cooling fan and the CPU (such as the relationship between cooling efficiency and CPU temperature). When the feature superposition strategy is adopted, it is to integrate the information in these two features. Adding them directly can more comprehensively reflect the impact of the relationship between the two on the maintenance subject of the relationship graph (CPU).

[0084] On the contrary, if the maintenance subject of the relationship graph is the hard disk of the computer, and the adjacent maintenance subject is an external mobile hard disk (the connection relationship exists only under specific maintenance operations, and the relationship is relatively weak), the computer device may determine to adopt the target integration strategy of feature difference. Assume that the optical flow feature vector of the maintenance subject of the mobile hard disk is , the maintenance relationship feature representation vector is The feature difference strategy is adopted because the difference between the two states may better reflect the information that is meaningful to the hard disk (the maintenance subject of the relationship graph). For example, when an external mobile hard disk is connected or removed, some state changes of the hard disk itself may be reflected by the difference between the two features.

[0085] Next, in step S13412, the computer device integrates the maintenance subject optical flow features and maintenance relationship feature representations of adjacent maintenance subjects based on the target integration strategy to obtain the integrated feature representation corresponding to the maintenance subject in the relationship graph. Continuing with the above example, if feature superposition is used for the CPU and the cooling fan, the integrated feature representation vector The calculation of is as follows: , here we assume , for the elements after n, the optical flow feature elements of the cooling fan maintenance subject can be directly used, that is, when i > n, This integration method combines the motion state characteristics of the cooling fan with the functional relationship characteristics between it and the CPU to form a new feature representation. This feature representation more comprehensively contains the association information between the cooling fan and the CPU, which is of great significance for the subsequent determination of the first significant optical flow feature corresponding to the CPU.

[0086] If the hard disk and the external mobile hard disk use feature difference, assuming m=n, the calculation of the integrated feature representation vector I_{mh} is:

[0087] ;

[0088] This integrated feature representation highlights the status difference information between the hard disk and the external mobile hard disk by calculating the difference between the two features, which helps to analyze the impact of the operation of the external mobile hard disk (such as insertion or removal) on the hard disk, and provides more targeted feature information for the subsequent calculation of the first significant optical flow feature corresponding to the hard disk.

[0089] Finally, in step S13413, the computer device transforms the integrated feature representation through the weights and biases in the target transformation structure to obtain the first significant optical flow feature corresponding to the relationship graph maintenance subject. The target transformation structure is an important component of the significant feature mining algorithm. It can perform complex transformation operations on the input integrated feature representation to mine feature information that better meets the requirements.

[0090] Take a server maintenance scenario as an example. Assume that the maintenance subject of the relationship graph is the memory module in the server, and the adjacent maintenance subject is the memory heat sink. Through the previous steps, the integrated feature representation vector related to the memory heat sink and the memory module is obtained as In the target transformation structure, let the weight matrix be W (dimension is q×p) and the bias vector be b (dimension is q), where q is the dimension of the transformed feature.

[0091] Then, the first significant optical flow feature vector corresponding to the transformed relationship graph maintenance subject (memory module) It can be calculated by the following formula: .

[0092] In one implementation, step S13413, transforming the integrated feature representation by weights and biases in the target transformation structure to obtain the first significant optical flow feature corresponding to the relationship graph maintenance subject, includes:

[0093] Step S134131: obtaining the transformation feature representation corresponding to the adjacent maintenance subject outputted by the previous transformation structure corresponding to the target transformation structure in the significant feature mining algorithm;

[0094] Step S134132: The target transformation structure transforms the integrated feature representation and the transformed feature representation through the first weight and bias to obtain the first significant optical flow feature corresponding to the relationship graph maintenance subject.

[0095] First, in step S134131, the computer device obtains the transformation feature representation corresponding to the adjacent maintenance subject output of the previous transformation structure corresponding to the target transformation structure in the significant feature mining algorithm. In the significant feature mining algorithm, multiple transformation structures (such as hidden layers) are often included, each of which performs a specific transformation operation on the input features, and the output of the previous transformation structure will be used as the input of the next transformation structure. This hierarchical structure helps to gradually mine more complex and meaningful feature representations.

[0096] Taking the server maintenance scenario as an example, assume that the maintenance subject of the relationship graph is a hard disk in the server, and the adjacent maintenance subject is the data line connecting the hard disk. In the salient feature mining algorithm, it has been processed by several layers of transformation structures. The transformation feature representation corresponding to the adjacent maintenance subject (data line) output by the previous transformation structure may already contain the hard disk-related information extracted by the previous level of transformation. For example, this transformation feature representation may reflect the characteristics of the data line related to the stability of hard disk data transmission under different maintenance operations.

[0097] Assume that the transformation feature representation vector corresponding to the adjacent maintenance subject output by the previous transformation structure is , where each element t_i is a value obtained through the previous calculation and transformation, and these values ​​represent certain hard disk-related features of the data line in a specific maintenance scenario. This transformed feature representation is obtained in the previous step by performing a series of processing on the original maintenance subject optical flow features and maintenance relationship feature representation of the data line. For example, it may have undergone operations such as feature integration (such as feature superposition or feature difference strategy in the previous step) and transformation of weights and biases in the previous transformation structure.

[0098] Then, in step S134132, the target transformation structure transforms the integrated feature representation and the transformed feature representation through the first weight and bias to obtain the first significant optical flow feature corresponding to the relationship graph maintenance subject. The target transformation structure is a specific level in the significant feature mining algorithm, which transforms the input features through specific weights and biases.

[0099] Continuing with the example of the server hard disk and data cable, in step S134131, the transformation feature representation vector corresponding to the adjacent maintenance subject (data cable) is obtained. At the same time, in the previous step S13412, the integrated feature representation vector corresponding to the relationship graph maintenance subject (hard disk) is obtained, assuming that .

[0100] In the target transformation structure, let the first weight matrix be (dimension is p×(m+n)), the bias vector is (dimension is p), where p is the dimension of the transformed feature (i.e., the first significant optical flow feature corresponding to the main body of the relationship graph maintenance). Then, the first significant optical flow feature vector corresponding to the main body of the transformed relationship graph maintenance (hard disk) is It can be calculated by the following formula: G'=W1×[I;t]+b1.

[0101] Where [I;t] means concatenating vectors I and T into a new vector by column. The significance of this calculation method is that the integrated feature representation and the transformed feature representation are re-weighted and combined through the first weight matrix W1, so that the target transformation structure can comprehensively consider the directly related features (integrated feature representation) of the current relationship graph maintenance subject (hard disk) and the features (transformed feature representation) of the adjacent maintenance subject (data line) after being processed by the previous transformation structure. Each element w in the weight matrix W1 1ij represents the contribution weight of the elements in the integrated feature representation and the transformed feature representation to the i-th first significant optical flow feature after transformation. For example, if w 112 The larger the value of , the more influential the second element in the integrated feature representation or the transformed feature representation is in determining the first element g'1 in the first significant optical flow feature. The bias vector b1 plays the role of translation adjustment, which can make the transformed feature more in line with the requirements of subsequent processing in terms of numerical range.

[0102] In one implementation, the target salient optical flow features corresponding to the maintenance subject of the relationship graph are output by a salient feature mining algorithm, and the salient feature mining algorithm includes at least one target transformation structure. Based on this, step S133 obtains a feature correlation degree based on the maintenance relationship feature representation and the target video optical flow feature, and determines a salient attention variable corresponding to an adjacent maintenance subject based on the feature correlation degree, including:

[0103] Step S1331: transforming the maintenance relationship feature representation by using the second weight and bias in the target transformation structure to obtain a reference feature representation;

[0104] Step S1332: transforming the target video optical flow feature by using the third weight and bias in the target transformation structure to obtain an index feature representation;

[0105] Step S1333: Obtaining the feature correlation degree based on the reference feature representation and the index feature representation;

[0106] Step S1334: Determine the significant attention variables corresponding to the adjacent maintenance subjects based on the feature correlation degree, and the feature correlation degree is consistent with the change direction of the significant attention variables corresponding to the adjacent maintenance subjects.

[0107] First, in step S1331, the computer device transforms the maintenance relationship feature representation through the second weight and bias in the target transformation structure to obtain a reference feature representation. The target transformation structure is a component of the significant feature mining algorithm, and the second weight and bias contained therein are parameters for performing a specific transformation on the input maintenance relationship feature representation.

[0108] Taking the server maintenance scenario as an example, assume that the maintenance relationship feature representation includes information such as the connection relationship and functional collaboration relationship between different components in the server. For example, for the hard disk in the server and the data cable connected to it, the maintenance relationship feature representation may include information such as the data transmission protocol relationship between the data cable and the hard disk, the stability relationship of the physical connection, etc. These relationships are quantified into vector form and are set as .

[0109] In the target transformation structure, let the second weight matrix be (dimension is m×n), the bias vector is (dimension is m), where m is the dimension of the reference feature representation after transformation. Then, the reference feature representation vector after transformation is It can be calculated by the following formula: , this reference feature representation is a transformation result of the original maintenance relationship feature representation. It re-weights each element in the original feature representation through the weight matrix W2 and adjusts the offset vector b2 to obtain a feature representation that is more suitable for correlation analysis with the target video optical flow feature. For example, if an element r in the original maintenance relationship feature representation i It is related to the data transmission efficiency between the hard disk and the data line. After transformation, k in the reference feature representation j It is possible to combine the data transmission efficiency with other relationship factors into a new representation, which can better reflect the importance of this relationship for subsequent analysis in the entire server maintenance scenario.

[0110] Next is step S1332, where the computer device transforms the target video optical flow feature through the third weight and bias in the target transformation structure to obtain an index feature representation. The target video optical flow feature reflects the motion information in the target terminal forensic video, including the motion direction, speed and other information of each maintenance subject in the video.

[0111] Taking server maintenance as an example, the target video optical flow feature may show the light and shadow changes corresponding to the hard disk read and write operations over a period of time, the movement trajectory of the data line during the plug-in and unplug operation, and other information. Suppose the target video optical flow feature vector is .

[0112] In the target transformation structure, let the third weight matrix be W3 (dimension is q×p), and the bias vector be b3 (dimension is q), where q is the dimension of the transformed index feature representation. Then, the transformed index feature representation vector It can be calculated by the following formula: This transformation converts the target video optical flow features into index feature representations, making them more consistent with the reference feature representations obtained previously in terms of form and semantics, and facilitating the subsequent calculation of the feature correlation between the two. For example, through this transformation, the target video optical flow features that originally represent the changes in the light and shadow of the hard disk and the movement trajectory of the data line are converted into an index feature representation that is more abstract and can better reflect its motion features related to the maintenance relationship, which may highlight the importance of certain key motion patterns in the entire server maintenance operation.

[0113] Then, in step S1333, the computer device obtains the feature correlation degree based on the reference feature representation and the index feature representation. There are many methods for calculating the feature correlation degree, and one commonly used method is cosine similarity.

[0114] Let the reference feature representation vector , index feature representation vector (Assume m=q. If they are not equal, appropriate processing can be performed to make their dimensions consistent.) The formula for calculating the cosine similarity S is: , the value of cosine similarity ranges from -1 to 1. The closer the value is to 1, the more similar the two feature representations are, that is, the higher the correlation is; the closer the value is to -1, the lower the correlation is; and the value of 0 means that the two feature representations are orthogonal, that is, there is no correlation. For example, in the server maintenance scenario, if the cosine similarity of the reference feature representation and the index feature representation is close to 1, it means that there is a strong correlation between the maintenance relationship feature representation (the reference feature representation after transformation) and the target video optical flow feature (the index feature representation after transformation). It may mean that in the maintenance operation of the server, the relationship between the hard disk and the data cable (maintenance relationship feature representation) is highly consistent with their actual motion state in the video (target video optical flow feature). For example, when the data transmission protocol relationship is good (maintenance relationship feature representation), the light and shadow changes and movement trajectories of the hard disk and the data cable in the video (target video optical flow feature) show a normal collaborative working state.

[0115] Finally, in step S1334, the computer device determines the significant attention variable corresponding to the adjacent maintenance subject based on the feature correlation degree, and the feature correlation degree is consistent with the change direction of the significant attention variable corresponding to the adjacent maintenance subject. The significant attention variable is used to measure the importance of the adjacent maintenance subject to the target maintenance subject.

[0116] Continuing with the example of server maintenance, if the feature correlation calculated in the previous steps is high (close to 1), the computer device can set the salient attention variable corresponding to the adjacent maintenance subject (such as the data line) to a higher value. Assuming a linear mapping relationship is set, when the feature correlation S is between 0.8-1, the salient attention variable w can be set to 0.8; when S is between 0.6-0.8, w can be set to 0.6; when S is between 0-0.6, w can be set to 0.2, etc. This setting method enables the salient attention variable to be adjusted according to the actual correlation between the maintenance relationship feature representation and the target video optical flow feature.

[0117] In one implementation, the target terminal forensic video includes multiple video frames, the target terminal forensic video features include a video frame feature set, and the video frame feature set includes video frame features corresponding to each video frame. Based on this, step S140 performs a feature aggregation operation on the target terminal forensic video features and the target saliency optical flow features corresponding to the target maintenance subject to obtain the target aggregated features, including:

[0118] Step S141: performing feature fusion processing on video frame features corresponding to the target video frame based on the target saliency optical flow features corresponding to the target maintenance subject to obtain feature fusion features corresponding to the target video frame;

[0119] Step S142: optimizing the video frame features corresponding to the target video frame in the video frame feature set by fusing the features corresponding to the target video frame to obtain an optimized video frame feature set;

[0120] Step S143: performing a feature aggregation operation on the optimized video frame feature set and the target video optical flow feature through a feature aggregation algorithm to obtain a video frame feature set after the feature aggregation operation and a target video optical flow feature after the feature aggregation operation.

[0121] First, in step S141, the computer device performs feature fusion processing on the video frame features corresponding to the target video frame based on the target salient optical flow features corresponding to the target maintenance subject, and obtains the feature fusion features corresponding to the target video frame. The target terminal forensic video contains multiple video frames, each of which has its corresponding video frame features. These video frame features are obtained by feature encoding the video in step S110, and contain various information in the video frame, such as texture, edge and other features. The target salient optical flow features corresponding to the target maintenance subject are determined in the previous step (such as step S130), which integrates various information related to the target maintenance subject.

[0122] Taking the server maintenance scenario as an example, assuming that the target maintenance subject is the hard disk in the server, and the target video frame is a frame of the internal hardware structure of the server. The video frame features corresponding to the video frame may include the appearance contour features of the hard disk, the position relationship of other surrounding hardware devices relative to the hard disk, and other information. Suppose the video frame feature vector is The target salient optical flow feature corresponding to the target maintenance subject (hard disk) may include the motion features of other components related to hard disk maintenance (such as data cables and power cables connected to the hard disk) in the video and their degree of association with the hard disk. Suppose the target salient optical flow feature vector is .

[0123] When computer equipment performs feature fusion processing, a variety of technical means can be used. A simple method is to directly splice two feature vectors to obtain the feature fusion feature vector corresponding to the target video frame. This splicing method completely combines the video frame features and the target saliency optical flow features, so that the new feature fusion features contain both static information about the target maintenance subject in the video frame (such as appearance, position relationship, etc.) and dynamic information related to the target maintenance subject (such as motion characteristics and association relationships of other parts, etc.).

[0124] Another possible technical approach is weighted summation. The computer device can assign weights to the video frame features and the target saliency optical flow features. Suppose the weight of the video frame feature is , the weight of the target salient optical flow feature is , then the feature fusion feature vector corresponding to the target video frame is By adjusting the weights The value of can emphasize the importance of video frame features and target saliency optical flow features in fusion features according to different maintenance requirements and scenarios. For example, in some cases, if you pay more attention to the static structure information of the target maintenance subject, you can appropriately increase If you pay more attention to the dynamic association information related to the target maintenance subject, you can increase it appropriately. The value of .

[0125] Next is step S142, where the computer device optimizes the video frame features corresponding to the target video frame in the video frame feature set by fusing the features corresponding to the target video frame, and obtains an optimized video frame feature set. The video frame feature set includes the video frame features corresponding to each video frame in the target terminal forensic video.

[0126] Continuing with the above server maintenance scenario as an example, in step S141, the feature fusion feature vector corresponding to the target video frame is obtained. The computer device uses this feature fusion feature to transform the original video frame feature vector Optimize. The optimization method can be direct replacement, that is, the original video frame feature vector Replaced by feature fusion feature vector In this way, the video frame features corresponding to the target video frame in the video frame feature set are updated to feature fusion features that contain more information related to the target maintenance subject.

[0127] Or use weighted update. Let the updated weight be , then the optimized video frame feature vector It can be calculated by the following formula: This weighted update method can integrate the information in the feature fusion feature while retaining part of the feature information of the original video frame, so that the optimized video frame feature can more comprehensively and accurately reflect the relationship between the target video frame and the target maintenance subject. u =0.8, it means that it is more inclined to integrate the information in the feature fusion feature into the original video frame feature. In this way, the optimized video frame feature not only retains some information in the original video frame feature that may still be useful (such as the basic appearance features of the hard disk, etc.), but also adds dynamic and associated information related to the target maintenance subject (such as the associated information of the data cable and the hard disk, etc.).

[0128] Finally, in step S143, the computer device performs a feature aggregation operation on the optimized video frame feature set and the target video optical flow feature through a feature aggregation algorithm to obtain the video frame feature set after the feature aggregation operation and the target video optical flow feature after the feature aggregation operation.

[0129] In the server maintenance scenario, the optimized video frame feature set includes the video frame features corresponding to each video frame optimized in step S142, and these features have incorporated information related to the target maintenance subject. The target video optical flow feature reflects the overall motion information in the target terminal forensic video.

[0130] The feature aggregation algorithm can be implemented in many forms. One feasible method is the aggregation method based on the vector space model. Suppose the video frame feature vector in the optimized video frame feature set is , the target video optical flow feature vector is The computing device may first map these vectors into a common vector space and then perform aggregation operations by calculating the distance or similarity between the vectors.

[0131] For example, the Euclidean distance can be used to measure the difference between vectors. and the target video optical flow feature vector , Euclidean distance It can be calculated by the following formula: , where n is the dimension of the vector (assuming that the video frame feature vector and the target video optical flow feature vector have the same dimension after processing), is the feature vector of the jth video frame, is the jth target video optical flow feature vector. Then, according to the size of the distance, the computer device can aggregate the video frame feature vectors with closer distances and the target video optical flow feature vectors. For example, a distance threshold can be set ,when When Perform aggregation to obtain the aggregated video frame feature vector And the aggregated target video optical flow feature vector .

[0132] This feature aggregation operation integrates the optimized video frame feature set and the target video optical flow feature, so that the video frame feature set and the target video optical flow feature after the feature aggregation operation can more comprehensively reflect the information related to the target maintenance subject in the target terminal forensic video, including static features in the video frame, dynamic features of the target maintenance subject related parts, and the overall optical flow features of the video. These aggregated features provide a richer and more representative feature basis for subsequent steps (such as determining the maintenance inspection information corresponding to the target terminal forensic video in step S150), which helps to improve the accuracy and reliability of the maintenance judgment.

[0133] In one implementation, step S141 performs feature fusion processing on video frame features corresponding to the target video frame based on the target saliency optical flow features corresponding to the target maintenance subject to obtain feature fusion features corresponding to the target video frame, including:

[0134] Step S1411: Based on the target saliency optical flow features corresponding to the target maintenance subject and the video frame features corresponding to the target video frame, feature fusion features corresponding to the target video frame are obtained.

[0135] When the computer device executes step S1411, the target significant optical flow feature corresponding to the target maintenance subject and the video frame feature corresponding to the target video frame are connected head to tail to obtain the feature fusion feature corresponding to the target video frame. The target significant optical flow feature corresponding to the target maintenance subject here contains important information related to the target maintenance subject. For example, in the scenario of maintaining a computer motherboard, if the target maintenance subject is a capacitor on the motherboard, the target significant optical flow feature may reflect the movement of other circuit elements related to the capacitor (such as connected resistors, lines, etc.) in the video and their association with the capacitor.

[0136] The video frame features corresponding to the target video frame are specific information about the video frame extracted from the target terminal forensic video. For example, in this motherboard maintenance example, the video frame features may include information such as the display of the motherboard layout structure in the frame, the appearance outlines of other components around the capacitor, etc.

[0137] When the computer device performs the head-to-tail connection of features, it is assumed that the target saliency optical flow feature vector corresponding to the target maintenance subject is , the video frame feature vector corresponding to the target video frame is Then, the concatenated feature fusion feature vector will be . This feature head-to-tail connection technique is a simple and direct feature fusion method. This connection method enables the new feature fusion feature to retain the information in the target salient optical flow feature and fully include the video frame feature information corresponding to the target video frame. Taking the engine maintenance in industrial equipment maintenance as an example, the target maintenance subject is a key valve of the engine. The target salient optical flow feature may include the flow of the fluid in the pipe connected to the valve (the motion feature reflected by the change of light and shadow) and the motion relationship of other components working with the valve. The video frame feature corresponding to the target video frame may be the appearance of the valve in the frame, the layout of the surrounding structural components, etc. The feature fusion feature obtained by connecting the features head-to-tail can simultaneously reflect the static appearance layout of the valve and the dynamic association information related to it, providing a more comprehensive feature information basis for the subsequent maintenance analysis. This helps the computer equipment to more accurately judge the maintenance status of the target maintenance subject in the subsequent steps, such as judging whether the valve needs maintenance due to abnormal movement of surrounding components or changes in its own appearance structure.

[0138] In one implementation, step S150, determining maintenance status inspection information corresponding to the target terminal forensic video based on the target aggregate feature, includes:

[0139] Step S151: passing the target video optical flow features after the feature aggregation operation into the maintenance behavior recognition algorithm in the converged state to obtain the maintenance behavior recognition information corresponding to the target terminal forensic video.

[0140] The target video optical flow feature is obtained through multiple steps of processing. It integrates various information in the target terminal forensic video, including relevant information of the target maintenance subject, video frame features, and optical flow features. For example, in the scenario of maintaining a server, the target video optical flow feature after feature aggregation may include the motion state of each component of the server (such as hard disk, memory, processor, etc.) during the maintenance process (such as the light and shadow changes when the hard disk is plugged in and out, the movement trajectory when the memory stick is replaced, etc.) and the relationship between these components (such as the optical flow features corresponding to the data interaction between the processor and the memory, etc.).

[0141] Then there is the converged maintenance behavior recognition algorithm. The converged state means that the algorithm has been trained or iterated enough to reach a relatively stable state, and can accurately process the input features and output meaningful results. The maintenance behavior recognition algorithm is an algorithm specifically used to identify various behaviors in the maintenance process of terminal equipment. It can be implemented based on a variety of technologies, such as the neural network algorithm in deep learning.

[0142] When the computer device passes the target video optical flow features after the feature aggregation operation to the maintenance behavior recognition algorithm in this converged state, a series of complex calculation operations will be performed inside the algorithm. Taking the neural network algorithm as an example, it is assumed that this maintenance behavior recognition algorithm is a multilayer perceptron (MLP). The target video optical flow feature vector after the feature aggregation operation is used as the input of the input layer, set as X=[x1, x2,…, x n ].

[0143] In a neural network, the neurons in the input layer pass the input features to the hidden layer. There are weighted connections between the neurons in the hidden layer. Let the weight matrix of the neurons in the hidden layer be W h (dimension is m×n), the bias vector is b h (dimension is m), where m is the number of hidden layer neurons. The input H of the hidden layer neurons i It can be calculated by the following formula:

[0144] ;

[0145] Then, the output of the hidden layer neurons is transformed nonlinearly through the activation function. For example, the commonly used ReLU activation function f(x)=max(0, x) transforms the input of the hidden layer neurons into After the activation function transformation, the output of the hidden layer is obtained .

[0146] The output of the hidden layer will continue to be passed to the next layer (which may be more hidden layers or directly to the output layer). Assume that the weight matrix of the output layer is (dimension is p×m), the bias vector is (dimension is p), where p is the number of neurons in the output layer, and the input of the neurons in the output layer It can be calculated by the following formula:

[0147] ;

[0148] Finally, the output values ​​of the neurons in the output layer constitute the maintenance behavior identification information corresponding to the target terminal forensic video. For example, if there are two neurons in the output layer, the output value of the first neuron indicates whether the maintenance behavior is compliant (close to 1 for compliance, close to 0 for non-compliance), and the output value of the second neuron indicates whether there is a maintenance error (close to 1 for errors, close to 0 for no errors).

[0149] In actual application scenarios, take the equipment maintenance in a computer room as an example. Assume that the target terminal forensic video records the maintenance process of a faulty computer. The target video optical flow feature after feature aggregation operation contains the optical flow information corresponding to the maintenance personnel's operation on computer hardware (such as graphics card, power supply, etc.). When the computer equipment passes this feature to the converged maintenance behavior recognition algorithm, if the output result shows that the maintenance behavior is not compliant and there is a maintenance error, it may be because the algorithm recognizes that the maintenance personnel's operation sequence when plugging and unplugging the graphics card is wrong (the optical flow feature corresponding to this wrong operation is recognized by the algorithm) or the standard safety operation process is not followed when handling the power supply (determined by the optical flow features related to the power supply and the algorithm).

[0150] In one implementation, the method is performed based on a machine learning algorithm, and the method also includes the following algorithm training process:

[0151] Step SA: obtaining an example terminal forensic video and prior maintenance inspection information corresponding to the example terminal forensic video;

[0152] Step SB: passing the example terminal forensic video into a video feature encoding algorithm to obtain an example terminal forensic video feature, wherein the example terminal forensic video feature includes an example video optical flow feature corresponding to the example terminal forensic video;

[0153] Step SC: obtaining an example maintenance subject corresponding to the example terminal evidence collection video, and determining a second related maintenance subject corresponding to the example maintenance subject;

[0154] Step SD: passing the maintenance subject optical flow feature corresponding to the second related maintenance subject into the significant feature mining algorithm, determining the example significant optical flow feature corresponding to the example maintenance subject based on the maintenance subject optical flow feature and the corresponding significant attention variable, the significant attention variable is obtained based on the correlation between the example video optical flow feature and the maintenance relationship feature representation, the maintenance relationship feature representation is a feature representation representing the correlation of the maintenance subject;

[0155] Step SE: passing the example terminal forensic video features and the example saliency optical flow features corresponding to the example maintenance subject into the feature aggregation algorithm to perform feature aggregation operation to obtain example aggregated features;

[0156] Step SF: transform the example aggregated features according to the video understanding algorithm to obtain the example transformation result;

[0157] Step SG: based on the example transformation results and the prior maintenance status inspection information, the algorithm parameters of the video understanding algorithm are modified, or the algorithm parameters of the video understanding algorithm and the cornerstone algorithm are modified, and the cornerstone algorithm includes the video feature encoding algorithm, the saliency feature mining algorithm and the feature aggregation algorithm.

[0158] In the embodiment of the present invention, steps SA-SG in the machine learning algorithm training process are important steps to improve the algorithm performance to accurately perform maintenance status inspection.

[0159] First, in step SA, the computer device obtains the example terminal forensic video and the prior maintenance situation verification information corresponding to the example terminal forensic video. The example terminal forensic video is sample data used to train the algorithm. It is similar to the target terminal forensic video in actual applications and contains visual information of the terminal device in various maintenance scenarios. For example, in a server maintenance training scenario, the example terminal forensic video may be a video record of different types of maintenance operations (such as hardware upgrades, troubleshooting, etc.) on different servers. The prior maintenance situation verification information is the known maintenance situation result corresponding to the example terminal forensic video. For example, for an example terminal forensic video of a server hardware upgrade, the prior maintenance situation verification information may indicate that the upgrade operation is compliant and no mistakes have occurred, and it is clear whether each maintenance action (such as plugging and unplugging a memory stick, installing a new hard disk, etc.) complies with the standard process.

[0160] Next is step SB, where the computer device passes the example terminal forensic video into the video feature encoding algorithm to obtain the example terminal forensic video features, where the example terminal forensic video features include the example video optical flow features corresponding to the example terminal forensic video. The function of the video feature encoding algorithm is to convert the example terminal forensic video into a feature form that can be processed and analyzed by the computer device. For example, a video feature encoding algorithm based on a convolutional neural network (CNN) is used, and the convolution layer of the CNN performs a convolution operation on each frame of the video. Through operations such as multi-layer convolution layers, pooling layers (such as maximum pooling, taking the maximum value in a local area to reduce the amount of data), various features are extracted from the video frame, and then combined with the calculated optical flow features (such as calculating the displacement of pixels between adjacent frames to obtain optical flow information), the example terminal forensic video features are finally obtained.

[0161] Then, in step SC, the computer device obtains the example maintenance subject corresponding to the example terminal forensic video, and determines the second related maintenance subject corresponding to the example maintenance subject. The example maintenance subject is similar to the target maintenance subject in actual applications, and is an element or object in the example terminal forensic video that is directly related to the maintenance operation. For example, in the example of maintaining a network device, the example maintenance subject may be a port of a network switch. The second related maintenance subject is other elements associated with the example maintenance subject. Taking the network switch port as an example, the second related maintenance subject may be a network cable connected to the port, a module where the port is located, etc. The computer device may determine the second related maintenance subject through a target detection algorithm and an association analysis algorithm. For example, using a target detection framework based on deep learning, each frame in the example terminal forensic video is detected, the example maintenance subject and other possible related elements are identified, and then the second related maintenance subject is determined through association analysis (such as association analysis based on distance measurement or semantic information).

[0162] In step SD, the computer device passes the maintenance subject optical flow feature corresponding to the second related maintenance subject into the significant feature mining algorithm, and determines the example significant optical flow feature corresponding to the example maintenance subject based on the maintenance subject optical flow feature and the corresponding significant attention variable, and the significant attention variable is obtained based on the correlation between the example video optical flow feature and the maintenance relationship feature representation, and the maintenance relationship feature representation is a feature representation representing the correlation of the maintenance subject. The maintenance subject optical flow feature reflects the motion state of the second related maintenance subject in the video. Assuming that in the example of maintaining a mobile device, the second related maintenance subject is the battery of the mobile device, and its maintenance subject optical flow feature may reflect the optical flow information such as the moving speed and direction of the battery during the plugging and unplugging process. The significant feature mining algorithm determines the example significant optical flow feature corresponding to the example maintenance subject based on the maintenance subject optical flow feature and the significant attention variable. The computer device can calculate the correlation between the example video optical flow feature and the maintenance relationship feature representation (such as using the Pearson correlation coefficient formula to obtain the significant attention variable, and then determine the example significant optical flow feature by weighted calculation or the like.

[0163] Then, in step SE, the computer device passes the example terminal forensic video features and the example salient optical flow features corresponding to the example maintenance subject into the feature aggregation algorithm for feature aggregation operation to obtain the example aggregated features. The feature aggregation algorithm aims to fuse different types of features together. For example, assuming that the example terminal forensic video feature vector is , the example saliency optical flow feature vector corresponding to the example maintenance subject is , feature aggregation can be performed by vector concatenation to obtain an example aggregated feature vector Or, by weighted summation, let the weight of the example terminal forensic video feature be a, and the weight of the example salient optical flow feature corresponding to the example maintenance subject be b, then the example aggregate feature vector ,in .

[0164] Next is step SF, where the computer device transforms the example aggregate features according to the video understanding algorithm to obtain the example transformation result. The video understanding algorithm is a machine learning algorithm for a specific video processing task, such as an algorithm based on a recurrent neural network (RNN) or its variants (such as a long short-term memory network LSTM or a gated recurrent unit GRU). Taking LSTM as an example, it is capable of processing sequence data (the example aggregate features here can be regarded as a form of sequence data). Let the example aggregate feature sequence be (where t represents the length of the sequence), the cell state in LSTM and hidden state The cell is updated through a series of complex gating mechanisms, including the forget gate, input gate, and output gate. The forget gate determines the The input gate determines which new information is stored in the cell state. In , the output gate determines the information to be output. The example aggregation features are processed through these gating mechanisms to obtain the example transformation results.

[0165] Finally, in step SG, the computer device corrects the algorithm parameters of the video understanding algorithm based on the example transformation results and the prior maintenance inspection information, or corrects the algorithm parameters of the video understanding algorithm and the cornerstone algorithm (including the video feature encoding algorithm, the salient feature mining algorithm, and the feature aggregation algorithm). During the training process, the computer device adjusts the algorithm parameters by comparing the difference between the example transformation results and the prior maintenance inspection information. For example, if an optimization algorithm based on gradient descent is used, let the loss function be L (the loss function can be defined according to the specific task and data characteristics, such as the mean square error loss function ,in is the true value in the prior maintenance condition inspection information, is the predicted value in the example transformation result, and N is the number of samples). The algorithm parameters (such as weights and biases in the neural network) are gradiented according to the loss function, and then the algorithm parameters are updated according to a certain learning rate so that the loss function gradually decreases. If it is found that the prediction results of the video understanding algorithm are significantly different from the prior maintenance status inspection information, it may be necessary to adjust not only the parameters of the video understanding algorithm itself, but also the parameters of the cornerstone algorithm. For example, if it is found that the final maintenance status judgment is wrong due to inaccurate features extracted by the video feature encoding algorithm, it is necessary to adjust the convolution kernel parameters and pooling parameters in the video feature encoding algorithm; if the calculation of the significant attention variable in the significant feature mining algorithm is unreasonable, which affects the determination of the example significant optical flow features, thereby leading to an incorrect maintenance status judgment, it is necessary to adjust the relevant parameters in the significant feature mining algorithm; if the feature aggregation algorithm does not fully utilize the information of each feature when fusing features, resulting in the example aggregation features not accurately reflecting the maintenance status, it is necessary to adjust the weights and other parameters in the feature aggregation algorithm.

[0166] In summary, step SA-SG continuously improves the accuracy and reliability of the algorithm in the remote maintenance of terminal equipment by acquiring sample data, extracting features, determining relevant maintenance subjects, feature mining, feature aggregation, transformation based on video understanding algorithms, and correcting algorithm parameters according to the results. This training process is the key to the effective operation of the machine learning algorithm in the entire terminal equipment remote maintenance method, and lays a solid foundation for accurate maintenance judgment in practical applications.

[0167] Based on the foregoing embodiments, an embodiment of the present invention provides a remote maintenance device for a terminal device. The units included in the device and the modules included in each unit can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0168] Figure 2 A schematic diagram of the structure of a terminal device remote maintenance device provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the terminal equipment remote maintenance device 200 includes:

[0169] The video feature encoding module 210 is used to obtain the target terminal forensic video to be identified, perform feature encoding on the target terminal forensic video, and obtain the target terminal forensic video feature, wherein the target terminal forensic video feature includes the target video optical flow feature corresponding to the target terminal forensic video; wherein the target terminal forensic video is a forensic video obtained by forensic shooting of the terminal device to be maintained;

[0170] A maintenance subject determination module 220 is used to obtain a target maintenance subject in the target terminal evidence collection video and determine a first related maintenance subject corresponding to the target maintenance subject;

[0171] A significant feature determination module 230 is used to determine a target significant optical flow feature corresponding to the target maintenance subject based on the maintenance subject optical flow feature of the first related maintenance subject and the corresponding significant attention variable, wherein the significant attention variable is obtained based on the correlation between the target video optical flow feature and the maintenance relationship feature representation, and the maintenance relationship feature representation is a feature representation representing the correlation of the maintenance subject;

[0172] The optical flow feature aggregation module 240 is used to perform a feature aggregation operation on the target terminal forensic video feature and the target saliency optical flow feature corresponding to the target maintenance subject to obtain a target aggregation feature;

[0173] The maintenance status verification module 250 is used to determine the maintenance status verification information corresponding to the target terminal forensic video based on the target aggregation feature.

[0174] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules provided by the device provided in the embodiment of the present invention can be used to execute the method described in the above method embodiment. For the technical details not disclosed in the device embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding. It should be noted that in the embodiment of the present invention, if the above terminal device remote maintenance method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), disk or optical disk, etc. Various media that can store program codes. In this way, the embodiment of the present invention is not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0175] An embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements part or all of the steps in the above method when executing the program.

[0176] The embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium may be transient or non-transient.

[0177] An embodiment of the present invention provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0178] Figure 3 A hardware entity diagram of a computer device provided by an embodiment of the present invention, such as Figure 3As shown, the hardware entity of the computer device 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

Claims

1. A terminal device remote maintenance method, characterized in that: The method comprises: Obtain a target terminal forensic video to be identified, perform feature encoding on the target terminal forensic video, and obtain target terminal forensic video features, wherein the target terminal forensic video features include target video optical flow features corresponding to the target terminal forensic video; wherein the target terminal forensic video is a forensic video obtained by forensic shooting of the terminal device to be maintained; Acquire a target maintenance subject in the target terminal evidence collection video, and determine a first related maintenance subject corresponding to the target maintenance subject; Acquire a visual spatiotemporal relationship diagram consisting of the first relevant maintenance subject and the target maintenance subject; For a relationship graph maintenance subject in the visual spatiotemporal relationship graph, obtaining a maintenance relationship feature representation indicating a correlation between the relationship graph maintenance subject and an adjacent maintenance subject; Obtaining a feature correlation degree based on the maintenance relationship feature representation and the target video optical flow feature, and determining a significant attention variable corresponding to the adjacent maintenance subject based on the feature correlation degree; Determine the target significant optical flow feature corresponding to the maintenance subject of the relationship graph based on the significant attention variable corresponding to the adjacent maintenance subject and the maintenance subject optical flow feature of the adjacent maintenance subject; Extracting the target salient optical flow features corresponding to the target maintenance subject from the target salient optical flow features corresponding to each relationship graph maintenance subject of the visual spatiotemporal relationship graph, wherein the maintenance relationship feature representation is a feature representation representing the correlation of the maintenance subject; Performing a feature aggregation operation on the target terminal forensic video feature and the target saliency optical flow feature corresponding to the target maintenance subject to obtain a target aggregate feature; Maintenance status inspection information corresponding to the target terminal forensic video is determined based on the target aggregation feature.

2. The method according to claim 1, characterized in that The target significant optical flow feature corresponding to the relationship graph maintenance subject is output based on a significant feature mining algorithm, the significant feature mining algorithm includes at least one target transformation structure, and the significant attention variable corresponding to the adjacent maintenance subject and the maintenance subject optical flow feature of the adjacent maintenance subject are determined to include: The maintenance subject optical flow features of the adjacent maintenance subjects and the maintenance relationship feature representation are transferred into the target transformation structure for transformation, so as to obtain the first significant optical flow features corresponding to the maintenance subject of the relationship graph; Based on the first significant optical flow feature corresponding to the relationship graph maintenance subject and the significant attention variable corresponding to the corresponding adjacent maintenance subject, a target significant optical flow feature corresponding to the relationship graph maintenance subject is determined.

3. The method according to claim 2, characterized in that The step of transferring the maintenance subject optical flow features of the adjacent maintenance subjects and the maintenance relationship feature representation into the target transformation structure for transformation to obtain the first significant optical flow features corresponding to the maintenance subject of the relationship graph includes: Determine a target integration strategy based on the maintenance subject correlation between the maintenance subject in the relationship graph and the adjacent maintenance subjects, wherein the target integration strategy is feature superposition or feature difference; Integrate the maintenance subject optical flow features of the adjacent maintenance subjects and the maintenance relationship feature representation based on the target integration strategy to obtain the integrated feature representation corresponding to the maintenance subject of the relationship graph; The integrated feature representation is transformed by using the weights and biases in the target transformation structure to obtain a first salient optical flow feature corresponding to the relationship graph maintenance subject.

4. The method according to claim 3, characterized in that The step of transforming the integrated feature representation by using the weights and biases in the target transformation structure to obtain the first significant optical flow feature corresponding to the relationship graph maintenance subject includes: Obtaining a transformation feature representation corresponding to the adjacent maintenance subject outputted by a previous transformation structure corresponding to the target transformation structure in the significant feature mining algorithm; The target transformation structure transforms the integrated feature representation and the transformed feature representation through a first weight and a bias to obtain a first significant optical flow feature corresponding to the relationship graph maintenance subject.

5. The method according to claim 1, characterized in that The target significant optical flow feature corresponding to the relationship graph maintenance subject is output by a significant feature mining algorithm, the significant feature mining algorithm includes at least one target transformation structure, the feature correlation degree is obtained based on the maintenance relationship feature representation and the target video optical flow feature, and the significant attention variable corresponding to the adjacent maintenance subject is determined based on the feature correlation degree, including: Transforming the maintenance relationship feature representation by using the second weight and bias in the target transformation structure to obtain a reference feature representation; Transforming the target video optical flow feature by using the third weight and bias in the target transformation structure to obtain an index feature representation; Acquiring a feature correlation degree based on the reference feature representation and the index feature representation; The significant attention variable corresponding to the adjacent maintenance subject is determined based on the feature correlation degree, and the feature correlation degree is consistent with the change direction of the significant attention variable corresponding to the adjacent maintenance subject.

6. The method according to claim 1, characterized in that The target terminal forensic video includes a plurality of video frames, the target terminal forensic video features include a video frame feature set, and the video frame feature set includes video frame features corresponding to each video frame; The step of performing a feature aggregation operation on the target terminal forensic video feature and the target saliency optical flow feature corresponding to the target maintenance subject to obtain the target aggregated feature includes: Based on the target saliency optical flow features corresponding to the target maintenance subject, feature fusion processing is performed on the video frame features corresponding to the target video frame to obtain the feature fusion features corresponding to the target video frame; By fusing features corresponding to the target video frame, optimizing the video frame features corresponding to the target video frame in the video frame feature set, to obtain an optimized video frame feature set; A feature aggregation operation is performed on the optimized video frame feature set and the target video optical flow feature through a feature aggregation algorithm to obtain a video frame feature set after the feature aggregation operation and a target video optical flow feature after the feature aggregation operation.

7. The method according to claim 6, characterized in that The determining, based on the target aggregation feature, maintenance inspection information corresponding to the target terminal forensic video includes: The target video optical flow features after the feature aggregation operation are passed into the maintenance behavior recognition algorithm in the converged state to obtain the maintenance behavior recognition information corresponding to the target terminal forensic video; The step of performing feature fusion processing on the video frame features corresponding to the target video frame based on the target saliency optical flow features corresponding to the target maintenance subject to obtain the feature fusion features corresponding to the target video frame includes: Based on the target saliency optical flow features corresponding to the target maintenance subject and the video frame features corresponding to the target video frame, feature head-to-tail connection is performed to obtain feature fusion features corresponding to the target video frame.

8. The method according to claim 1, characterized in that The method is performed based on a machine learning algorithm, and the method also includes the following algorithm training process: Obtaining an example terminal forensic video and prior maintenance inspection information corresponding to the example terminal forensic video; The example terminal forensic video is passed into a video feature encoding algorithm to obtain an example terminal forensic video feature, wherein the example terminal forensic video feature includes an example video optical flow feature corresponding to the example terminal forensic video; Obtain an example maintenance subject corresponding to the example terminal evidence collection video, and determine a second related maintenance subject corresponding to the example maintenance subject; Passing the maintenance subject optical flow feature corresponding to the second related maintenance subject into the significant feature mining algorithm, determining the example significant optical flow feature corresponding to the example maintenance subject based on the maintenance subject optical flow feature and the corresponding significant attention variable, wherein the significant attention variable is obtained based on the correlation between the example video optical flow feature and the maintenance relationship feature representation, wherein the maintenance relationship feature representation is a feature representation representing the correlation of the maintenance subject; The example terminal forensic video feature and the example salient optical flow feature corresponding to the example maintenance subject are passed into a feature aggregation algorithm to perform a feature aggregation operation to obtain an example aggregated feature; Transform the example aggregated features according to a video understanding algorithm to obtain an example transformation result; Based on the example transformation result and the prior maintenance condition inspection information, the algorithm parameters of the video understanding algorithm are modified, or the algorithm parameters of the video understanding algorithm and the cornerstone algorithm are modified, and the cornerstone algorithm includes the video feature encoding algorithm, the saliency feature mining algorithm and the feature aggregation algorithm.

9. A terminal equipment remote maintenance device, characterized in that: The device comprises: A video feature encoding module is used to obtain a target terminal forensic video to be identified, perform feature encoding on the target terminal forensic video, and obtain target terminal forensic video features, wherein the target terminal forensic video features include target video optical flow features corresponding to the target terminal forensic video; wherein the target terminal forensic video is a forensic video obtained by forensic shooting of the terminal device to be maintained; A maintenance subject determination module, used to obtain a target maintenance subject in the target terminal evidence collection video, and determine a first related maintenance subject corresponding to the target maintenance subject; A significant feature determination module is used to obtain a visual spatiotemporal relationship graph consisting of the first relevant maintenance subject and the target maintenance subject; for the relationship graph maintenance subject in the visual spatiotemporal relationship graph, obtain a maintenance relationship feature representation indicating the correlation between the relationship graph maintenance subject and an adjacent maintenance subject; obtain a feature correlation degree based on the maintenance relationship feature representation and the target video optical flow feature, and determine a significant attention variable corresponding to the adjacent maintenance subject based on the feature correlation degree; determine a target significant optical flow feature corresponding to the relationship graph maintenance subject based on the significant attention variable corresponding to the adjacent maintenance subject and the maintenance subject optical flow feature of the adjacent maintenance subject; extract the target significant optical flow feature corresponding to the target maintenance subject from the target significant optical flow features corresponding to each relationship graph maintenance subject in the visual spatiotemporal relationship graph, wherein the maintenance relationship feature representation is a feature representation representing the correlation of maintenance subjects; An optical flow feature aggregation module is used to perform a feature aggregation operation on the target terminal forensic video feature and the target saliency optical flow feature corresponding to the target maintenance subject to obtain a target aggregation feature; A maintenance status verification module is used to determine maintenance status verification information corresponding to the target terminal forensic video based on the target aggregation feature.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images