AI-based terminal equipment remote maintenance method and system

Abstract

The invention relates to the field of data processing, and particularly provides an AI-based terminal equipment remote maintenance method and system, and the method comprises the steps: obtaining a target maintenance guidance video corresponding to a remote maintenance monitoring video, and enabling the target maintenance guidance video to comprise various types of guidance video contents, based on the importance coefficients of the various types of guidance video contents, determining the correlation degree between the various types of guidance video contents included in the target maintenance guidance video and the remote maintenance monitoring video, and based on the correlation degree between the various types of guidance video contents and the remote maintenance monitoring video, determining the remote maintenance monitoring video; and determining a target type of guidance video content in the multiple types of guidance video contents, and highlighting and marking a video frame corresponding to the remote maintenance monitoring video in the target type of guidance video content. According to the method, more important video content highlighted marks can be provided when the maintenance auxiliary video is displayed, and maintenance personnel are helped to quickly determine a maintenance mode to carry out high-efficiency maintenance.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an AI-based terminal equipment remote maintenance method and system. Background Art

[0002] In the field of remote maintenance of terminal equipment, with the continuous improvement of equipment complexity and the growth of remote maintenance needs, how to efficiently and accurately provide effective maintenance guidance for remote maintenance personnel has become an urgent problem to be solved. Traditional remote maintenance methods often lack systematicity and pertinence when providing maintenance guidance. For example, when facing complex terminal equipment failures, the existing remote maintenance system may simply provide various maintenance-related video materials to maintenance personnel without effectively screening and correlating these video materials. On the one hand, among the numerous maintenance guidance videos, different types of video content have different importance for specific remote maintenance monitoring situations, but most existing methods do not take this into account. This leads to the inability to accurately measure the impact of different types of video content on remote maintenance monitoring when correlating maintenance guidance videos with remote maintenance monitoring videos, making the determination of the degree of correlation inaccurate, which in turn affects the efficiency of maintenance personnel in obtaining key information. On the other hand, the existing remote maintenance system lacks a prominent display of key content when displaying maintenance guidance content. Usually, all types of guidance video content are presented to maintenance personnel without distinction, which makes it difficult for maintenance personnel to quickly locate the most relevant information for the current remote maintenance monitoring, which easily leads to the problem of not highlighting the key points. Maintenance personnel may need to spend a lot of time screening out useful information from a large amount of video content, which not only increases the difficulty of maintenance judgment, but also greatly reduces maintenance efficiency. Summary of the invention

[0003] In view of this, the present invention provides an AI-based terminal device remote maintenance method and system. The technical solution of the embodiment of the present invention is implemented as follows:

[0004] On the one hand, the present invention provides an AI-based remote maintenance method for terminal equipment, which is applied to a remote maintenance system, and the method includes: obtaining a target maintenance instruction video corresponding to a remote maintenance monitoring video, wherein the target maintenance instruction video includes multiple categories of instruction video content; based on the importance coefficients of the multiple categories of instruction video content, determining the degree of association between the various categories of instruction video content included in the target maintenance instruction video and the remote maintenance monitoring video; based on the degree of association between the various categories of instruction video content and the remote maintenance monitoring video, determining a target type of instruction video content in the multiple categories of instruction video content; and highlighting and marking video frames corresponding to the remote maintenance monitoring video in the target type of instruction video content. Among them, the highlighting and marking of the video frames corresponding to the remote maintenance monitoring video in the guidance video content of the target type includes: indexing a plurality of video frames to be marked corresponding to the remote maintenance monitoring video in the guidance video content of the target type; determining a solidification coefficient between the plurality of video frames to be marked according to the distribution timing of the plurality of video frames to be marked in the guidance video content of the target type, the solidification coefficient characterizing the time distribution relationship of the video frames to be marked in the video content, so as to determine whether there is logical coherence; if the solidification coefficient between the plurality of video frames to be marked meets the set requirements, the video frames corresponding to the remote maintenance monitoring video in the guidance video content of the target type are highlighted and marked.

[0005] On the other hand, the present invention provides a remote maintenance system, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above method when executing the program.

[0006] Beneficial effects of the present invention: The method provided by the present invention calculates the degree of association between the various categories of instruction video content included in the target maintenance instruction video and the remote maintenance monitoring video based on the importance coefficients of different types of instruction video content, and determines that the obtained degree of association can reflect the influence of various categories of instruction video content on the remote maintenance monitoring video, thereby improving the accuracy of the determined degree of association. In addition, based on the degree of association between various categories of instruction video content and remote maintenance monitoring video, the target type of instruction video content is determined from the multiple categories of instruction video content included in the target maintenance instruction video, and the video frames corresponding to the remote maintenance monitoring video in the target type of instruction video content are highlighted and marked, and the instruction video content that is more related to the remote maintenance monitoring video can be determined in the target maintenance instruction video, which can improve the correlation between the highlighted and marked content and the remote maintenance monitoring video, and can prevent the defects of not highlighting the key points and affecting the maintenance judgment, resulting in reduced maintenance efficiency, caused by indiscriminately displaying the maintenance guidance in various categories of instruction video content in the target maintenance instruction video. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A schematic diagram of the implementation flow of an AI-based terminal device remote maintenance method provided in an embodiment of the present invention.

[0008] Figure 2 A hardware entity schematic diagram of a remote maintenance system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The embodiment of the present invention provides an AI-based remote maintenance method for terminal devices, which can be executed by a processor of a remote maintenance system. The remote maintenance system can be set on a server, a laptop, a tablet computer, a desktop computer, or other electronic devices with data processing capabilities.

[0010] Figure 1 A schematic diagram of an implementation flow of an AI-based terminal device remote maintenance method provided in an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0011] Step S100: Acquire a target maintenance instruction video corresponding to the remote maintenance monitoring video, wherein the target maintenance instruction video includes instruction video contents of various categories.

[0012] In the embodiment of the present invention, the remote maintenance monitoring video is a video content that monitors and records the operating status, fault manifestations, etc. of the terminal device. For example, when performing remote maintenance on a certain model of server, the remote maintenance monitoring video may record the flashing status of the server indicator light, the operation status of the cooling fan, and the abnormal sounds of certain components, and other visual and audible fault-related information.

[0013] The target maintenance instruction video is a video resource that contains instructional video content of multiple categories. The multiple categories here include but are not limited to fault introduction, tool preparation, maintenance points, maintenance methods, maintenance process, etc. Taking server maintenance as an example, the instructional video content of fault introduction may elaborate on the possible causes of a certain fault (such as memory failure) in the server, such as memory particle damage, gold finger oxidation, etc.; the instructional video content of tool preparation will inform the maintenance personnel of the tools they need to prepare, such as professional tools for detecting memory, anti-static gloves, etc.; the instructional video content of maintenance points will emphasize the key links that need special attention during the maintenance process, such as when dealing with memory problems, pay attention to avoid static electricity from damaging other components; the instructional video content of maintenance methods will show different maintenance methods, such as replacing memory modules or cleaning and repairing gold fingers, etc.; the instructional video content of the maintenance process will fully show the entire operation process from the beginning of maintenance to the completion of repair.

[0014] The remote maintenance system can obtain the target maintenance guidance video in a variety of ways. One way is through the retrieval and matching of the video database. Assume that the remote maintenance system has a large maintenance video database. The videos in this database are all accompanied by detailed label information. These label information covers multiple dimensions such as equipment type, fault type, and maintenance operation type. When receiving the remote maintenance monitoring video, the system first performs a preliminary analysis of the remote maintenance monitoring video to determine the equipment type (such as the server mentioned above) and the preliminary judgment of the fault type (such as the preliminary judgment of memory-related faults based on the flashing of the indicator light). Then, the system searches the video database based on these analysis results, and finds the target maintenance guidance video that matches it by setting retrieval conditions such as equipment type and fault type.

[0015] If the video set in the video database is V={v1,v2,v3,...,v n}, for each video v i , all have corresponding device type labels T(v i ) and fault type label F(v i ) and other label information. For remote maintenance monitoring video, set its device type to T m The initial fault type is F mThen the process of acquiring the target maintenance guidance video can be regarded as finding a video that satisfies T(v j )=T m And F(v j ) and F m Videos with correlation (such as partial matching or inclusion of fault types) v i , these videos i It is the target maintenance guidance video.

[0016] Another implementation method is feature matching based on deep learning algorithms. First, the system extracts features from the remote maintenance monitoring video. For example, it uses a convolutional neural network (CNN) to extract features from image frames in the video to obtain a feature vector X. This feature vector contains the feature representation of key information in the video, such as the appearance of the equipment and the fault phenomenon. At the same time, the same feature extraction operation is performed on each video in the video database to obtain the corresponding feature vector set. Then, by calculating the feature vector X of the remote maintenance monitoring video and the feature vector X of each video in the database i The similarity of, for example, using the cosine similarity formula ,When the similarity exceeds a certain threshold, the corresponding video v_i is determined as the target maintenance guidance video.

[0017] In addition, a hybrid approach can be used to first perform preliminary screening through database tag retrieval to obtain a smaller video subset, and then use a deep learning algorithm in this subset for more accurate feature matching, thereby obtaining the target maintenance guidance video corresponding to the remote maintenance monitoring video. The advantage of this is that it can improve the efficiency of obtaining the target maintenance guidance video while ensuring accuracy, and avoid time-consuming feature matching operations in the entire huge video database.

[0018] Step S200: Based on the importance coefficients of the various categories of instruction video contents, determine the degree of association between the various categories of instruction video contents included in the target maintenance instruction video and the remote maintenance monitoring video.

[0019] In an embodiment of the present invention, multiple categories of instructional video content, for example, cover different aspects such as fault introduction, tool preparation, maintenance points, maintenance methods, and maintenance processes. The importance coefficient is a quantitative value, such as a weight value, which is used to indicate the relative importance of each category of instructional video content in the entire maintenance process. For example, in the remote maintenance of a server, if the type of fault is more complex and difficult to diagnose, then the importance of the fault introduction category in the entire maintenance process may be higher, and its importance coefficient may be set to a relatively large value; and if the maintenance tools are more conventional and easy to obtain, the tool preparation category is relatively less important, and the importance coefficient will be relatively small. The degree of association is an indicator to measure the closeness of the connection between various categories of instructional video content and remote maintenance monitoring videos.

[0020] For remote maintenance monitoring videos, it contains information such as the actual operating status of the terminal equipment, fault symptoms, etc. For example, in the monitoring video, it is seen that a certain hard disk indicator of the server continues to flash abnormally, and the system log shows that there are read and write errors. This is the device status information presented by the remote maintenance monitoring video.

[0021] There are many ways for the remote maintenance system to determine the degree of association. One feasible way is based on content feature analysis. The remote maintenance system can extract features from remote maintenance monitoring videos and various types of guidance video content. For feature extraction of video content, the convolutional neural network (CNN) technology in deep learning can be used. For example, for remote maintenance monitoring videos, the CNN network can extract the state features of equipment components in the video (such as visual features such as component color and shape changes) and fault-related dynamic features (such as flickering frequency, the occurrence pattern of error information, etc.), and combine these features to form a feature vector V rm For each category of instructional video content, CNN technology is also used to extract feature vectors. For example, the feature vector extracted from the instructional video content of the fault introduction category is V fi , tool preparation category is V tp wait.

[0022] Then, the distance or similarity between these feature vectors is calculated to measure the degree of association. A commonly used formula for calculating distance is the Euclidean distance formula , where v rm,i and v fi,i They are the eigenvectors V rm and V fiThe smaller the distance value, the higher the degree of association between the two. For example, if the calculated Euclidean distance between the instructional video content of the fault introduction category and the remote maintenance monitoring video is small, it means that the degree of association between the two is high, that is, the content of the fault introduction is more consistent with the actual monitored fault situation.

[0023] Another way is based on semantic analysis. The remote maintenance system can use natural language processing technology to perform semantic analysis on the text information in the video (such as equipment error information, operation prompts, etc.). For the text information in the remote maintenance monitoring video, it is converted into a semantic vector representation S rm Similarly, the text parts in the guidance video content of various categories are also converted into semantic vectors, such as the semantic vector of the fault introduction category is S fi Etc. This conversion can be achieved using a pre-trained word vector model (such as Word2Vec) combined with a semantic analysis algorithm.

[0024] Afterwards, the degree of association is measured by calculating the cosine similarity between the semantic vectors. The closer the similarity value is to 1, the higher the degree of association between the two. For example, in the maintenance of a server, if the error message in the remote maintenance monitoring video is semantically very similar to the relevant expression in the guidance video content of the fault introduction category, then the calculated cosine similarity will be close to 1, indicating that the degree of association between the two is very high.

[0025] However, the correlation degree obtained by the above method alone does not take into account the importance coefficients of various categories of guidance video content. The introduction of importance coefficients is to give different weights to different categories when determining the correlation degree. For example, assuming that the importance coefficient of the fault introduction category is w fi The initial correlation between the guidance video content of the fault introduction category and the remote maintenance monitoring video obtained by the previous content feature analysis or semantic analysis is r fi (The initial correlation degree is the correlation degree without considering the importance coefficient), then the correlation degree after considering the importance coefficient is r fi The formula R fi =w fi × fi to calculate.

[0026] Different categories of instructional video content have different importance to maintenance work. For example, when dealing with an emergency failure of a server, the instructional video content in the maintenance key points category has a higher importance coefficient due to its critical operation guidance nature. Even if its initial degree of association with the remote maintenance monitoring video in terms of content features or semantics is not the highest, after multiplying by a higher importance coefficient, a higher final degree of association can be obtained, which reflects the importance of considering multiple factors when determining the degree of association. In this way, the remote maintenance system can more accurately determine the degree of association between various categories of instructional video content and remote maintenance monitoring videos, providing a more reasonable and accurate data basis for subsequent steps (such as determining the target type of instructional video content, etc.).

[0027] Step S300: determining a target type of instruction video content from among multiple types of instruction video content based on the degree of association between various types of instruction video content and the remote maintenance monitoring video.

[0028] In the embodiment of the present invention, taking the remote maintenance of the server as an example, it is assumed that the remote maintenance monitoring video shows that the server has frequent crashes, and the CPU usage rate increases abnormally before the crash. The degree of correlation calculated by the remote maintenance system through the previous steps can present different situations. For example, if the instruction video content of the fault introduction category contains explanations about the causes of related faults such as CPU overheating leading to crashes, then its degree of correlation with the remote maintenance monitoring video may be high; if the instruction video content of the tool preparation category mentions the tool preparation related to network detection, but has little relevance to the current CPU-related crash fault, its degree of correlation may be low; if the instruction video content of the maintenance points category emphasizes the key operating points when dealing with CPU heat dissipation problems, then its degree of correlation with the monitoring video may be high; if the instruction video content of the maintenance method category shows the maintenance method of replacing the hard disk, it has a low degree of correlation with CPU-related faults; if the instruction video content of the maintenance process category describes in detail the maintenance process of the CPU heat dissipation module, its degree of correlation may be high. The remote maintenance system can determine the instruction video content of the target type by using a threshold judgment and weight synthesis method.

[0029] For threshold judgment, the remote maintenance system can set a threshold T of correlation degree. The setting of this threshold can be based on previous maintenance experience or obtained through statistical analysis of a large amount of sample data. For example, through the analysis of maintenance cases of different types of server failures, it is found that when the correlation degree reaches 0.6, the corresponding guidance video content is of great help in solving the failure, so the threshold T can be set to 0.6. Then, the remote maintenance system traverses the correlation degree between various categories of guidance video content and remote maintenance monitoring videos. If the correlation degree of a certain category is greater than or equal to this threshold T, then this category is preliminarily determined as the guidance video content of the target category. For example, the correlation degrees of the three categories of fault introduction, maintenance points and maintenance process are 0.7, 0.8 and 0.75 respectively, which are all greater than the threshold of 0.6, so these three categories are preliminarily determined as the guidance video content of the target category. The weights of different categories of guidance video content in the entire maintenance process may be different. Assuming that the weight of the fault introduction category is w1, the weight of the maintenance points category is w2, and the weight of the maintenance process category is w3, these weights can be determined based on factors such as the complexity of the equipment and the commonness of the fault. For example, for a complex device like a server, the weight w1 of the fault description may be set to 0.3, because accurately understanding the cause of the fault is a key step in solving the problem; the weight w2 of the maintenance points is set to 0.4, because grasping the key maintenance points is crucial to a successful repair; the weight w3 of the maintenance process is set to 0.3, because it is the step of putting the maintenance points into practice.

[0030] Then, by the formula (where r1, r2, and r3 are the correlation between the fault description, maintenance points, maintenance process, and remote maintenance monitoring video respectively) calculate the comprehensive score S. If the comprehensive score S exceeds a pre-set comprehensive score threshold S T (For example, S T is set to 0.7), then the corresponding category is determined as the target type of instructional video content.

[0031] In actual situations, different terminal devices and fault types may require different weight and threshold settings. For example, for relatively simple terminal devices, such as ordinary network switches, the fault type may be relatively simple, the tool preparation and repair process may be relatively simple, and the fault introduction and repair points may be more important, so the weight distribution will be different. Moreover, with the development of technology and the accumulation of maintenance experience, these weights and thresholds can also be continuously adjusted and optimized to improve the accuracy of the guidance video content of the remote maintenance system to determine the target type.

[0032] By determining the target type of instructional video content, the remote maintenance system can focus on the most relevant and valuable instructional video categories, thereby improving the efficiency and accuracy of remote maintenance. This helps remote maintenance personnel quickly locate key information in a large number of instructional video contents, reduce the time for searching and filtering information, and perform maintenance operations on terminal equipment more efficiently.

[0033] Step S400: highlighting and marking the video frames corresponding to the remote maintenance monitoring video in the instruction video content of the target type.

[0034] In the embodiment of the present invention, the target type of instruction video content is determined in step S300, which is the category of content that is highly associated with the remote maintenance monitoring video and has the most reference value for the current maintenance task, such as instruction video content of categories such as fault introduction and maintenance points. Video frame is the basic component unit of video, and each video frame contains image information at a specific moment. Highlighting is an operation of specially marking video frames so that specific video frames can be more easily identified when viewing the video.

[0035] Take server remote maintenance as an example. Assume that the remote maintenance monitoring video shows that the server's memory module is overheating, and the target type of guidance video content includes two types of video content: fault introduction (mentioning possible causes of memory overheating, such as heat sink damage, memory frequency setting too high) and maintenance points (such as the need to turn off the power first when dealing with memory overheating, and use professional tools to detect memory temperature sensors, etc.).

[0036] The remote maintenance system analyzes the instructional video content of the target category and finds the video frames corresponding to the memory overheating phenomenon in the remote maintenance monitoring video. For example, in the instructional video content of the fault introduction, there may be a video frame showing the appearance characteristics of the memory module when the heat sink is damaged, and this video frame is related to the appearance characteristics of the overheated memory module shown in the remote maintenance monitoring video; in the instructional video content of the maintenance points, there may be a video frame showing the correct operation method of using professional tools to detect the memory temperature sensor, which is related to solving the memory overheating problem in the remote maintenance monitoring video.

[0037] One feasible approach is based on image feature matching. First, image features are extracted from key frames in the remote maintenance monitoring video (e.g., representative frames containing memory overheating). This can be done using convolutional neural network (CNN) technology, such as the pre-trained VGGNet model. Through this model, the feature vector F of the key frame of the remote maintenance monitoring video can be obtained. rm Then, the same feature extraction operation is performed on each video frame in the target type of guidance video content to obtain the corresponding feature vector set Next, calculate the feature vector F of the key frame of the remote maintenance monitoring video rm The feature vector F of each video frame in the guidance video content of the target category gv The similarity between them. For example, the cosine similarity formula can be used. When the similarity exceeds a certain set threshold T s (This threshold can be determined by testing a large amount of sample data. For example, for videos related to server maintenance, it is found through testing that when the similarity reaches 0.8, there is sufficient correlation between the video frames, then T s It can be set to 0.8), then this video frame in the guidance video content of the target type is determined as the video frame corresponding to the remote maintenance monitoring video, and is highlighted.

[0038] There are many ways to highlight and mark, such as adding a border of a specific color (such as red) to the edge of the video frame, or superimposing a semi-transparent identification icon on the video frame, so that maintenance personnel can recognize these key video frames at a glance when viewing the video.

[0039] Another method is based on semantic analysis. If the target type of instructional video content and remote maintenance monitoring video contain text information (such as text description of the fault in the fault introduction, text description of the operation steps in the maintenance points, etc.), these texts can be semantically analyzed. First, the text information in the video is extracted, and then it is converted into a semantic vector using natural language processing technology. For example, for the relevant text in the remote maintenance monitoring video, the semantic vector S rm , for the text in the target category’s guidance video content, convert it into a semantic vector .

[0040] Next, the similarity between the semantic vectors is calculated, and the cosine similarity formula can also be used. When the similarity exceeds the set threshold, the corresponding video frame is determined as the video frame corresponding to the remote maintenance monitoring video and is highlighted.

[0041] In addition, the highlight annotation can also be optimized based on the time series of the video frames. For example, if there are multiple video frames corresponding to the remote maintenance monitoring video that are continuous in time in the guidance video content of the target category, then a continuous annotation method can be used, such as connecting these continuous video frames with a colored line, or marking the time period of this group of continuous video frames with a special color to indicate that these video frames are logically related and are descriptions of different stages of the same problem or operation process. This can further improve the maintenance personnel's understanding and utilization efficiency of the video content, so that they can better perform remote maintenance operations according to the guidance in the video.

[0042] As an implementation mode, step S400, highlighting and marking the video frames corresponding to the remote maintenance monitoring video in the instruction video content of the target type, includes:

[0043] Step S410: indexing the instruction video content of the target type to obtain a plurality of video frames to be annotated corresponding to the remote maintenance monitoring video;

[0044] Step S420: determining a freezing coefficient between the plurality of video frames to be labeled according to the distribution time sequence of the plurality of video frames to be labeled in the guidance video content of the target category, wherein the freezing coefficient represents the time distribution relationship of the video frames to be labeled in the video content, so as to determine whether they are logically coherent;

[0045] Step S430: If the freezing coefficients between the multiple video frames to be annotated meet the set requirements, the video frames corresponding to the remote maintenance monitoring video in the guidance video content of the target type are highlighted and annotated.

[0046] Step S410 is to index and obtain multiple video frames to be annotated corresponding to the remote maintenance monitoring video in the target type of instruction video content. The purpose of this operation is to preliminarily screen out video frames in the target type of instruction video content that are associated with the remote maintenance monitoring video, laying the foundation for subsequent annotation work.

[0047] First, for the target type of instructional video content, the remote maintenance system needs to process it so that it can accurately index the corresponding video frames. For example, in the remote maintenance scenario of a server, the target type of instructional video content may include the category of maintenance points. Suppose the remote maintenance monitoring video shows a specific error code and the corresponding hardware indicator flashing pattern during the operation of the server.

[0048] The remote maintenance system can adopt a feature matching-based approach. For each frame in the remote maintenance monitoring video, the system can extract multiple features, such as visual features (including visually identifiable features such as hardware appearance, indicator light color and flashing frequency) and data-based features (such as features corresponding to text information such as error codes). For the extraction of visual features, convolutional neural network (CNN) technology can be used, such as the ResNet model, the video frame is input into the model, and the output is a feature vector containing rich visual feature information. For the feature extraction of text information, a word vector model, such as Word2Vec, can be used to convert text such as error codes into vector form.

[0049] At the same time, similar feature extraction operations are performed on the video frames in the target category guidance video content. Then, the remote maintenance system determines whether it is a corresponding video frame by calculating the similarity between these feature vectors. For example, the cosine similarity between the feature vector of the remote maintenance monitoring video frame and the feature vector of the video frame in the target category guidance video content is calculated as follows: , where F rm is the feature vector of the remote maintenance monitoring video frame, F gv is the feature vector of the video frame in the target category guidance video content. When the similarity exceeds a preset threshold (such as 0.8 obtained by testing a large amount of sample data), the video frame in the target category guidance video content is determined as the video frame to be labeled.

[0050] For example, in the instructional video content of maintenance points, if a certain video frame shows the troubleshooting operation corresponding to the same hardware indicator flashing pattern as in the remote maintenance monitoring video, and its similarity with the remote maintenance monitoring video frame calculated by the feature vector exceeds the threshold, then this video frame is indexed as a proposed annotated video frame. In this way, the remote maintenance system can find multiple proposed annotated video frames corresponding to the remote maintenance monitoring video in the instructional video content of the target type, preparing for the subsequent judgment of the relationship between these frames.

[0051] Step S420 determines the solidification coefficients between the multiple video frames to be labeled according to the distribution sequence of the multiple video frames to be labeled in the target type of instructional video content. The solidification coefficient is also called the compact parameter, which characterizes the time distribution relationship of these video frames to be labeled in the video content to determine whether they are logically coherent. After obtaining multiple video frames to be labeled, the remote maintenance system will analyze the time position information of each video frame to be labeled in the target type of instructional video content. For example, in a target type of instructional video content about the server maintenance process, it is assumed that there are three video frames to be labeled, marked as frame A, frame B and frame C. Frame A appears at the 10th second of the video, showing the use of the preliminary detection tool for server faults; frame B appears at the 30th second, showing the preliminary maintenance operation based on the preliminary detection results; frame C appears at the 50th second, showing the operation of further in-depth maintenance.

[0052] In order to determine the freezing coefficient between these frames, the remote maintenance system can adopt a calculation method based on time interval and content coherence. First, the time interval between adjacent video frames to be annotated is calculated, such as the time interval between frame A and frame B is 30-10=20 seconds, and the time interval between frame B and frame C is 50-30=20 seconds.

[0053] Then, the coherence of the content between adjacent frames is analyzed. The analysis of content coherence can be based on the logical relationship of operations or the association of objects in the video frames. For example, from frame A to frame B, the operation is from preliminary detection to preliminary maintenance, and there is a logical progressive relationship; from frame B to frame C, the operation is from preliminary maintenance to in-depth maintenance, and there is also a logical progressive relationship.

[0054] The remote maintenance system can set a quantitative value for the closeness of this logical relationship. Assume that a very close logical relationship (such as the progressive relationship from detection to maintenance mentioned above) is set to 1, and a weak logical relationship is set to 0.5, etc. Then, the solidification coefficient is calculated by combining the time interval and content coherence. For example, let the time interval weight between adjacent frames be w T , the content coherence weight is w c , for frame A and frame B, assuming w T =0.4, w c =0.6, the time interval is 20 seconds, within the set reasonable time interval range (such as 0-30 seconds is a reasonable range), according to a certain mapping relationship (such as within a reasonable range, the shorter the time interval, the higher the score, assuming that the score corresponding to 20 seconds is 0.8), the content coherence is 1, then the solidification coefficient C AB =w T ×0.8+w c ×1=0.4×0.8+0.6×1=0.92. Similarly, the freezing coefficient between frame B and frame C can be calculated. In this way, the remote maintenance system can determine the freezing coefficients between multiple video frames to be annotated according to their distribution sequence in the target type of guidance video content, thereby judging the logical coherence of these video frames in the video content.

[0055] In step S430, the setting requirement is a standard pre-set based on the needs of the maintenance task and the expectation of logical coherence of the video content. For example, in the remote maintenance of the server, if it is desired that the video frames to be annotated have a high logical coherence, the threshold of the coagulation coefficient may be set to 0.8. If the coagulation coefficients between the multiple video frames to be annotated calculated in step S420 are greater than or equal to this threshold (such as the coagulation coefficients between frame A and frame B, frame B and frame C calculated above are greater than 0.8), the remote maintenance system will highlight the video frames corresponding to the remote maintenance monitoring video in the guidance video content of the target type.

[0056] There are many ways to highlight and annotate. One way is to draw a border of a specific color (such as red) around the video frame. The remote maintenance system can use the video editing tool library to add a border drawing instruction of a specific color around the pixel coordinates of the video frame when the corresponding video frame is identified. For example, for a video frame with a resolution of 1920×1080, if it is determined to be a frame that needs to be annotated, the system will add red border drawing instructions at the coordinates of the frame (0,0) to (1920,0), (0,1080) to (1920,1080), (0,0) to (0,1080), and (1920,0) to (1920,1080).

[0057] Another way is to superimpose a semi-transparent identification icon on the video frame. The remote maintenance system can prepare icons representing important information in advance (such as a circular icon with an exclamation mark, indicating that special attention is needed). When it is determined that a video frame needs to be annotated, the icon is superimposed in a semi-transparent form at a specific position of the video frame (such as the center of the video frame). This operation can be achieved through image synthesis technology, where the icon image and the video frame image are synthesized according to a certain transparency ratio.

[0058] In this way, when the solidification coefficients between multiple video frames to be annotated meet the set requirements, the remote maintenance system can accurately highlight the video frames corresponding to the remote maintenance monitoring video in the target type of guidance video content, thereby helping remote maintenance personnel to quickly locate key content and improve maintenance efficiency. This annotation method enables maintenance personnel to more clearly identify the video frame content that is closely related to the current remote maintenance monitoring situation when viewing the guidance video, which is of great significance both in fault diagnosis and maintenance operation guidance. For example, in the process of complex server hardware fault repair, maintenance personnel can quickly find the maintenance operation video frame corresponding to the currently monitored fault phenomenon, and perform accurate maintenance operations according to the annotated video frame content, reducing maintenance time and the possibility of erroneous operations.

[0059] As an implementation mode, step S200, based on the importance coefficients of the multiple categories of instruction video contents, determines the degree of association between the various categories of instruction video contents included in the target maintenance instruction video and the remote maintenance monitoring video, including:

[0060] Step S210: Process the remote maintenance monitoring video to obtain a video frame to be matched;

[0061] Step S220: determining the initial correlation degree between various types of instruction video content and the remote maintenance monitoring video based on the intended matching video frame;

[0062] Step S230: Determine the degree of association between various categories of instruction video content and remote maintenance monitoring video based on the initial degree of association between various categories of instruction video content and remote maintenance monitoring video and the importance coefficients of various categories of instruction video content.

[0063] In step S210, the remote maintenance system can process the remote maintenance monitoring video in a variety of ways to obtain the video frames to be matched. For example, the video frame extraction algorithm can be used to decompose the remote maintenance monitoring video into a series of video frames. For a remote maintenance monitoring video with a duration of T seconds and a frame rate of F frames / second, a total of N=T×F video frames can be obtained through simple time interval sampling.

[0064] After acquiring the video frames, the remote maintenance system does not directly use all video frames as the intended matching video frames. It can further filter according to certain screening criteria. For example, based on the time series screening, if the server failure usually shows obvious characteristics in a specific time period, the remote maintenance system can focus on selecting the video frames in this time period as the intended matching video frames. Or based on the screening triggered by key events, such as when a specific error message or abnormal state is detected in the video (such as a hard disk indicator light on the server starts to flash red), the video frames within a period of time before and after the event are selected as the intended matching video frames.

[0065] In addition, the remote maintenance system can also use content-based video frame extraction technology. For example, image recognition technology is used to identify key equipment components in video frames (such as the server's motherboard, memory module, etc.), and only video frames containing these key components are used as matching video frames. This content-based extraction helps focus on the video content parts that are closely related to the maintenance task and reduce unnecessary calculations and processing.

[0066] In step S220, the remote maintenance system establishes a standard or method for measuring the degree of association. One feasible method is to calculate the similarity based on the feature vector. For the video frame to be matched and the various types of guidance video content, the remote maintenance system extracts their feature vectors respectively.

[0067] For example, for the video frame to be matched, multiple features including visual features (such as the shape, color, texture, etc. of the equipment parts), dynamic features (such as the motion state of the equipment parts, the flashing frequency of the indicator lights, etc.) and text features related to the fault (such as error messages, etc.) can be extracted, and these features can be combined into a feature vector F rm .

[0068] For various types of instructional video content, feature extraction is also performed. Taking the instructional video content of the fault introduction category as an example, the visual feature description of the fault phenomenon and the text feature of the fault cause can be extracted and combined into a feature vector F fi .

[0069] Then, the remote maintenance system can determine the initial correlation degree by calculating the similarity between the two feature vectors. For example, using the cosine similarity formula, the closer the value is to 1, the higher the initial correlation degree between the two; the closer it is to 0, the lower the correlation degree.

[0070] In addition to similarity calculation based on feature vectors, remote maintenance systems can also use methods based on semantic analysis. If the video frames to be matched and the guidance video content contain a large amount of text information (such as operating instructions, fault explanations, etc.), these texts can be converted into semantic vectors. For example, the word vector model (such as Word2Vec) in natural language processing technology is used to map the words in the text into vectors, and then these vectors are combined into semantic vectors. The initial degree of association is determined by calculating the similarity between semantic vectors (the cosine similarity formula can also be used). This method is very effective when processing video content containing rich semantic information, and can accurately capture the association relationship between text content.

[0071] In step S230, the remote maintenance system obtains importance coefficients of various categories of instructional video content. These importance coefficients may be pre-set, for example, based on the experience of maintenance experts or statistical analysis of a large number of maintenance cases. Assume that in server maintenance, the importance coefficient of the fault introduction category is w fi is set to 0.4, because accurate understanding of the cause of the failure is crucial to the entire maintenance process; the importance coefficient of the tool preparation category w tp It is set to 0.2 because it is relatively less important; the importance coefficient of the maintenance point category w rp The importance coefficient of the maintenance process category is set to 0.3; rc Set to 0.1.

[0072] Then, the remote maintenance system calculates the degree of correlation between various categories of guidance video content and remote maintenance monitoring videos according to the formula R=w×r (where R is the final correlation degree, w is the importance coefficient, and r is the initial correlation degree).

[0073] For example, for the fault introduction category, assuming its initial correlation degree r fi The result calculated in step S220 is 0.8, so the final correlation degree R fi =w fi × fi=0.4×0.8=0.32. Similarly, for the tool preparation category, if the initial correlation degree r tp is 0.3, then the final correlation degree R tp =w tp × tp =0.2×0.3=0.06.

[0074] In this way, the remote maintenance system comprehensively considers the initial correlation degree and the importance coefficient, and obtains a correlation degree value that better reflects the actual correlation relationship between various categories of guidance video content and remote maintenance monitoring videos. This calculation method that considers the importance coefficient is of great significance in actual remote maintenance scenarios. For example, in some emergency situations, although the initial correlation degree of the guidance video content of the maintenance points category with the remote maintenance monitoring video may not be the highest, due to its high importance coefficient, the final correlation degree can be ranked in the forefront, which enables the remote maintenance system to treat different categories of guidance video content more reasonably in subsequent operations (such as determining the target type of guidance video content, etc.), thereby improving the efficiency and accuracy of remote maintenance.

[0075] As an implementation manner, step S220, determining the initial correlation degree between various types of instruction video content and the remote maintenance monitoring video based on the intended matching video frame, includes:

[0076] Step S221: indexing various types of guidance video content to obtain a target video frame corresponding to the to-be-matched video frame;

[0077] Step S222: Determine the initial correlation degree between various categories of guidance video content and the remote maintenance monitoring video based on the influence coefficient of the to-be-matched video frame and the influence coefficient of the target video frame corresponding to the to-be-matched video frame in various categories of guidance video content.

[0078] In step S221, for various types of instruction video content, such as fault introduction, tool preparation, maintenance points, maintenance methods, maintenance process, etc., each category contains a series of video information about terminal equipment maintenance. The remote maintenance system first needs to analyze and process these instruction video contents.

[0079] One way is based on image feature recognition. The remote maintenance system can extract image features from each video frame in the instructional video content of the fault introduction category. For example, using convolutional neural network (CNN) technology, such as the ResNet model, feature extraction is performed on the device component image in the video frame to obtain a feature vector containing information such as component shape, color, and texture. Then, the image features of the motherboard capacitor bulge in the video frame to be matched are compared and matched with the video frame feature vector in the instructional video content of the fault introduction category.

[0080] At the same time, for the sound information in the video frame to be matched, the remote maintenance system can use audio feature extraction technology. For example, extract the frequency, amplitude, timbre and other features of the sound, and match these features with the audio information that may exist in the guidance video content of the fault introduction category (such as the audio describing the sound made by the server under a certain fault).

[0081] In this process, the remote maintenance system compares these features to find the video frame that is most similar to the intended matching video frame in terms of image and sound features, and indexes it as the target video frame. For example, if there is a video frame in the guidance video content of the fault introduction category that shows both similar capacitor bulge image features and audio feature descriptions similar to the error sound in the intended matching video frame, then this video frame will be indexed as the target video frame.

[0082] For other categories, such as instructional video content in the maintenance key points category, the remote maintenance system will also index based on the relevant information in the intended matching video frame. If the intended matching video frame shows that the server may be unstable due to capacitor bulging, the remote maintenance system will search for video frames related to maintenance points that involve handling such unstable operation caused by capacitor problems in the instructional video content in the maintenance key points category as the target video frame.

[0083] In step S222, the influence coefficient is a quantitative value used to represent the importance or influence of the intended matching video frame or the target video frame in the respective video. The remote maintenance system first needs to determine the influence coefficients of the intended matching video frame and the target video frame.

[0084] For example, in a remote maintenance monitoring video, if a certain video frame shows a critical failure point of a server (e.g., multiple critical components on the motherboard are abnormal at the same time), then the impact coefficient of this video frame can be set to a higher value. The remote maintenance system can determine this coefficient based on pre-set rules, such as quantifying it based on factors such as the severity of the failure point and the scope of impact on the overall operation of the server. Assume that the impact coefficient of the video frame corresponding to the critical failure point is I rm is set to 0.8.

[0085] For the determination of the influence coefficient of the target video frame, taking the guidance video content of the fault introduction category as an example, if the target video frame accurately explains the detailed fault principle related to the key fault point in the intended matching video frame (this explanation of the principle is very important for understanding and solving the fault), then the influence coefficient of this target video frame I fi It may also be set to a higher value, say 0.7.

[0086] Then, the remote maintenance system can use a weighted calculation method to determine the initial correlation degree. For example, the formula To calculate the initial correlation degree r between the instructional video content of the fault introduction category and the remote maintenance monitoring video.

[0087] Taking the instructional video content of the tool preparation category as an example, if the intended matching video frame shows that a specific tool is needed for repair due to a server failure (such as a special capacitor detection tool), and the target video frame in the instructional video content of the tool preparation category shows the preparation process of this tool. Assume that the influence coefficient I of the intended matching video frame is rm is 0.6 (because tool requirements are part of maintenance but are less important than the fault point itself), and the impact coefficient of the target video frame I tp is 0.5 (because it is only tool preparation, which is less important than the explanation of the fault principle), then the initial correlation degree calculated according to the formula .

[0088] As an implementation manner, the number of target maintenance guidance videos is multiple, and the method may further include:

[0089] Step S10: determining the importance coefficient of each target maintenance guidance video based on the pairing method between the remote maintenance monitoring video and each target maintenance guidance video;

[0090] Step S20: determining the degree of association between each target maintenance instruction video and the remote maintenance monitoring video based on the importance coefficient of each target maintenance instruction video and the degree of association between various categories of instruction video content included in each target maintenance instruction video and the remote maintenance monitoring video;

[0091] Step S30: Arrange the multiple target maintenance guidance videos in order based on the degree of association between each target maintenance guidance video and the remote maintenance monitoring video.

[0092] In step S10, in the context of remote maintenance, the remote maintenance monitoring video contains key information such as the actual operating status and fault phenomenon of the terminal equipment, while the target maintenance guidance video is a video resource corresponding to the remote maintenance monitoring video and containing a variety of maintenance guidance contents. The pairing method reflects the correlation between the two.

[0093] The process of determining the importance coefficient of the target maintenance guidance video by the remote maintenance system involves many considerations. First, the analysis of the pairing method. For example, if the remote maintenance monitoring video shows a hardware failure of the server, such as a hard disk failure (manifested as abnormally slow read and write speeds, frequent bad sectors, etc.), the pairing method of the target maintenance guidance video is to match according to the type of failure. If a target maintenance guidance video is mainly about the overall maintenance of server hardware, and only a small part of it involves content related to hard disk failures, then its pairing with the remote maintenance monitoring video is relatively weak.

[0094] The remote maintenance system can use a content ratio-based approach to determine the importance coefficient. Assuming that the total duration of the target maintenance instruction video is T, and the duration of the fault-related content (here, hard disk failure) in the remote maintenance monitoring video is t, then the importance coefficient I=t / T. For example, a target maintenance instruction video with a total duration of 60 minutes, of which the duration of the hard disk failure-related content is 10 minutes, then its importance coefficient I=0.17.

[0095] Another case is the pairing method based on the depth of logical association. For example, although the content of a target maintenance instruction video involving hard disk failure is not long in the overall length, it has an in-depth logical analysis of the cause of hard disk failure, its correlation with other components, and possible subsequent problems. The remote maintenance system can set a weighting factor w for this deep logical association, the importance coefficient Assuming the weighting factor w=2 (indicating that the value of deep logical association is high), according to the previous example, the importance coefficient .

[0096] In addition, the accuracy of the pairing can also be considered. If the target maintenance guidance video is directly matched based on a specific failure mode in the remote maintenance monitoring video, such as the failure mode of a specific model of hard disk in the remote maintenance monitoring video is accurately matched to a target maintenance guidance video specifically for the failure of that model of hard disk, then the importance coefficient of this target maintenance guidance video can be additionally improved. The remote maintenance system can set an accuracy improvement value a, the importance coefficient Assuming the accuracy improvement value a=0.2, the importance coefficient .

[0097] In step S20, the remote maintenance system performs calculations based on various factors. For example, for a target maintenance instruction video, it contains instruction video content of different categories, such as fault introduction, tool preparation, and maintenance points. Assume that the correlation between these categories of instruction video content and the remote maintenance monitoring video has been determined in step S200, which are r1 (correlation between fault introduction and remote maintenance monitoring video), r2 (correlation between tool preparation and remote maintenance monitoring video), r3 (correlation between maintenance points and remote maintenance monitoring video), etc.

[0098] At the same time, the importance coefficient of this target maintenance guidance video determined in step S10 is I. The remote maintenance system can use a weighted summation method to determine the degree of association R between the target maintenance guidance video and the remote maintenance monitoring video. For example, the formula R=I×(r1×w1+r2×w2+r3×w3+…) can be used, where w1, w2, w3, etc. are the weights of the content of each category of guidance video in the entire target maintenance guidance video.

[0099] This calculation method takes into account the importance of the target maintenance guidance video itself and the degree of correlation between the content of different categories of guidance videos and the remote maintenance monitoring video. It can more comprehensively and accurately reflect the relationship between the target maintenance guidance video and the remote maintenance monitoring video.

[0100] In step S30, after obtaining the correlation degree between each target maintenance instruction video and the remote maintenance monitoring video, the remote maintenance system can perform a sorting operation. For example, assuming there are three target maintenance instruction videos, the correlation degrees between them and the remote maintenance monitoring video are R1=0.396, R2=0.52, and R3=0.28 respectively.

[0101] The remote maintenance system will arrange these target maintenance guidance videos in descending order of relevance. This arrangement is of great significance in the actual remote maintenance process. Maintenance personnel can first view the target maintenance guidance videos with the highest relevance, so that they can more efficiently obtain the maintenance guidance information most relevant to the current remote maintenance monitoring situation.

[0102] As an implementation mode, step S100, obtaining a target maintenance instruction video corresponding to the remote maintenance monitoring video, includes:

[0103] Step S110: Process the remote maintenance monitoring video to obtain a video frame sequence to be matched;

[0104] Step S120: classifying the video frames in the to-be-matched video frame sequence to obtain a plurality of video frame groups;

[0105] Step S130: acquiring target maintenance guidance videos corresponding to a plurality of video frame groups respectively, each video frame group corresponding to a pairing method.

[0106] In step S110, the remote maintenance system parses the video and decomposes the video into a frame-by-frame image sequence, because the video is essentially composed of a series of continuous images. For example, for a remote maintenance monitoring video with a duration of T seconds and a frame rate of F frames / second, it contains a total of N=T×F video frames.

[0107] After obtaining these video frames, the remote maintenance system does not simply treat all of them as the to-be-matched video frame sequences, but rather uses some methods to screen and pre-process these video frames.

[0108] One feasible approach is to filter based on key information. For example, in the remote maintenance monitoring video of the server, if the status of a certain indicator light of the server (such as the hard disk indicator light flashing for a long time) or the appearance of a certain component (such as the memory stick is loose) may indicate a fault state of the device, the remote maintenance system can use image recognition technology to identify the video frames containing these key components or states. Assume that through the image recognition algorithm, it is identified that there are M video frames related to key information in the entire video (M≤N), and these video frames initially constitute part of the video frame sequence to be matched.

[0109] The remote maintenance system can also extract features from video frames to further screen and optimize the sequence of video frames to be matched. For example, for each video frame, its color features, texture features, shape features, etc. can be extracted. Taking color features as an example, the color space of the video frame can be converted from RGB to HSV, and then the hue, saturation, and value values ​​can be extracted as color feature vectors. For texture features, the gray level co-occurrence matrix (GLCM) can be used to calculate texture features, such as contrast, correlation, energy, and homogeneity. For shape features, the edge shape of the object in the video frame can be extracted through an edge detection algorithm (such as the Canny edge detection algorithm), and then the geometric features of the shape, such as the perimeter, area, and circularity, can be calculated.

[0110] Assume that after feature extraction, K (K≤M) video frames are selected according to the set feature threshold (for example, a hue value range in color features, contrast range in texture features, etc.), and these video frames eventually constitute the quasi-matching video frame sequence. This quasi-matching video frame sequence will serve as an important basis for finding the target maintenance guidance video in the subsequent steps.

[0111] In step S120, the basis for classification can be various. For example, classification can be based on equipment components. In the remote maintenance monitoring video of the server, if the video frame sequence to be matched contains relevant video frames of different components of the server, such as the hard disk, memory, and CPU, the remote maintenance system can classify the video frames involving the hard disk into one group, those involving the memory into another group, those involving the CPU into another group, and so on.

[0112] The remote maintenance system can use a classification algorithm based on machine learning to achieve this classification. For example, the support vector machine (SVM) algorithm. For each video frame, the remote maintenance system uses the extracted features (such as the color, texture, shape features mentioned in the previous step) as the input vector, and then classifies it through a pre-trained SVM model. Assuming that the trained SVM model can distinguish n different types of equipment parts, the video frames in the sequence of video frames to be matched can be divided into n video frame groups.

[0113] Another classification basis can be based on the fault type. If in the sequence of video frames to be matched, some video frames show overheating-related phenomena (such as red component color, high-speed operation of the cooling fan, etc.), and some show circuit connection-related problems (such as loose lines, solder joints falling off, etc.), the remote maintenance system can group these video frames separately. This classification method can use a clustering algorithm, such as the K-means clustering algorithm. The remote maintenance system first determines the number of clusters to be clustered (for example, m cluster centers are determined based on the known number of fault types), and then uses the feature vector of the video frame as input to divide the video frame into m video frame groups through the K-means clustering algorithm.

[0114] Through such classification, the remote maintenance system can reasonably group the video frames in the video frame sequence to be matched, so that each group has certain commonalities, which helps to obtain the target maintenance guidance video more specifically in the subsequent steps.

[0115] In step S130, for each video frame group, the remote maintenance system needs to find the corresponding target maintenance guidance video. The remote maintenance system can adopt a database query-based approach. Assume that there is a maintenance guidance video database, in which each target maintenance guidance video has detailed label information, which may include equipment component type, fault type, etc.

[0116] At the same time, this pairing method can also take more factors into account. For example, for a specific model of equipment components or a specific failure mode, the remote maintenance system can add more precise matching conditions when querying. Assuming that the hard disk of a specific model of server fails, the remote maintenance system will query the database for a target maintenance guidance video that contains both "hard disk" and the specific model information.

[0117] If there are a large number of target maintenance instruction videos in the database, the remote maintenance system can further filter them based on other attributes of the videos. For example, based on the update time of the video, the latest updated target maintenance instruction videos are given priority, because the latest videos may contain more accurate and advanced maintenance knowledge. Or based on the rating of the video (if there is a user rating mechanism), target maintenance instruction videos with higher ratings are given priority, which means that these videos may have higher reliability and practicality in actual applications.

[0118] As an implementation mode, step S110, processing the remote maintenance monitoring video to obtain a to-be-matched video frame sequence, includes:

[0119] Step S111: detecting the video semantics of the remote maintenance monitoring video to obtain the video semantic tags of the remote maintenance monitoring video;

[0120] Step S112: performing a video frame splitting operation on the remote maintenance monitoring video, and establishing a to-be-matched video frame sequence based on the video semantic tags and the video frames obtained by the video frame splitting operation.

[0121] In step S111, the video semantics includes various meaning information in the video content. For example, in the remote maintenance monitoring video of the terminal device (such as the server), the video semantics may involve the operating status of the device (normal operation, fault status, etc.), the relationship between the components of the device (such as the impact of a certain component on another component), the description of the fault phenomenon (such as the abnormal performance of a specific component), etc. The remote maintenance system can use a variety of methods to detect video semantics. One feasible method is the combination of natural language processing (NLP) technology and image recognition technology. For text information in the video (such as the error message of the device, the prompt text on the operation interface, etc.), the remote maintenance system can use NLP technology for semantic analysis. For example, first perform text extraction to extract the visible text in the video, which can be achieved through optical character recognition (OCR) technology. Assume that in the remote maintenance monitoring video of the server, the text prompt "hard disk read and write error" is displayed on the operation interface, and it is accurately extracted through OCR technology.

[0122] Then, the extracted text is subjected to lexical analysis, syntactic analysis and other operations. Taking lexical analysis as an example, the text "hard disk read and write error" is decomposed into words such as "hard disk", "read and write", and "error", and their parts of speech are determined. Then, syntactic analysis is performed to determine the grammatical relationship between these words, so as to understand that the semantics of this sentence is about the hard disk having errors in read and write operations. For the image part in the video, the remote maintenance system uses image recognition technology to assist in semantic detection. For example, the image features of device components in the video frame, such as hard disks and memory components in the server, are identified. If the image shows that the indicator light of the hard disk flashes red for a long time, the remote maintenance system can associate this image feature with a predefined semantic concept (such as a manifestation of hard disk failure).

[0123] By integrating the text and image information in the video, the remote maintenance system can construct video semantic tags. For example, in the case of the above server, the video semantic tags may include "hard disk failure - read / write error, hard disk indicator flashing red" and other content. These semantic tags can accurately summarize the key semantic information in the video and provide a basis for subsequent operations.

[0124] In step S112, the remote maintenance system performs a video frame splitting operation, which is a process of splitting the video into separate video frames in chronological order. For example, for a remote maintenance monitoring video with a frame rate of 30 frames per second, if the video duration is 10 seconds, 300 video frames will be obtained through the video frame splitting operation.

[0125] After the video frames are split, the remote maintenance system needs to filter and organize these video frames based on the video semantic tags obtained in the previous step S111 to construct a sequence of video frames to be matched.

[0126] Taking the remote maintenance monitoring video of the server as an example, if the video semantic tag contains "hard disk failure-read / write error, hard disk indicator flashing red", the remote maintenance system will search for video frames related to these semantics in the split video frames. For semantics related to the hard disk, the remote maintenance system will identify video frames that show the appearance of the hard disk, the status of the hard disk indicator, and images that may be related to the hard disk read / write operation (such as the flashing status of the data transfer indicator when the hard disk is reading / writing).

[0127] The remote maintenance system can use feature matching to determine the relevant video frames. For example, for the semantics of the hard disk indicator flashing red, the remote maintenance system pre-stores the image feature templates of the hard disk indicator in normal state and red flashing state. For each video frame, the remote maintenance system extracts the image features of its indicator area and then matches them with the pre-stored feature templates. If the matching degree with the feature template of the red flashing state is high (for example, using cosine similarity calculation, when the similarity exceeds a set threshold, such as 0.8), the video frame is determined to be a video frame related to the semantics of the hard disk indicator flashing red.

[0128] Similarly, for the semantics of hard disk read and write errors, if the video frames display specific images related to hard disk read and write operations (such as the connection status of the hard disk read and write data transmission line, the status of related devices during hard disk read and write, etc.), and these images match the image features related to hard disk read and write errors predefined in the remote maintenance system, these video frames will also be determined as related video frames.

[0129] After such a screening and matching process, the remote maintenance system combines all video frames related to the video semantic tags to form a sequence of quasi-matched video frames. This quasi-matched video frame sequence contains video frames related to key semantic information in the remote maintenance monitoring video. These video frames will serve as an important basis for subsequent search for target maintenance guidance videos, which will help improve the accuracy and efficiency of the entire remote maintenance process.

[0130] As an implementation mode, step S300, based on the degree of association between various types of instruction video content and the remote maintenance monitoring video, determines the target type of instruction video content from the multiple types of instruction video content, including:

[0131] Step S310: determining, from among the multiple categories of instruction video content, instruction video content of a preset category and instruction video content of other categories other than the preset category;

[0132] Step S320: determining the candidate category of instruction video content from the remaining categories of instruction video content based on the degree of association between the instruction video content of various categories and the remote maintenance monitoring video;

[0133] Step S330: determining the instruction video content of the candidate category and the instruction video content of the preset category as the instruction video content of the target category.

[0134] In step S310, the remote maintenance system determines a preset category. The preset category can be set according to factors such as the type of terminal device, common failure modes, or maintenance priorities. For example, in remote maintenance of a server, if based on previous maintenance experience, the two types of content, fault introduction and maintenance points, are crucial for quickly locating and resolving server failures, then the remote maintenance system can set the fault introduction and maintenance points as preset categories.

[0135] The remote maintenance system can determine the preset categories based on statistical analysis. Assume that there is a maintenance case database that records the maintenance of a large number of servers, including the frequency of use of different categories of guidance video content in each case and the degree of contribution to troubleshooting. The remote maintenance system analyzes this database and calculates the frequency of each category in successfully solving the fault. For example, the frequency of the fault introduction category is F fi , the frequency of occurrence of the maintenance point category is F rp , if F fi and F rp If the proportion of all successful cases exceeds a certain threshold (such as 0.6), these two categories are set as preset categories.

[0136] After the preset categories are determined, the remaining categories (such as tool preparation, maintenance methods, maintenance processes, etc.) are determined as the remaining categories of instructional video content outside the preset categories. This division method helps the remote maintenance system to process different types of instructional video content in a targeted manner in subsequent steps, and determine the target type of instructional video content based on their characteristics and the degree of association with the remote maintenance monitoring video.

[0137] In step S320, the remote maintenance system obtains the degree of association between the instruction video content of various categories and the remote maintenance monitoring video. These degrees of association are calculated in the previous steps (such as step S200), indicating the closeness of the connection between each category and the remote maintenance monitoring video.

[0138] Taking the remote maintenance of the server as an example, assuming that the correlation between the instructional video content of the tool preparation category and the remote maintenance monitoring video is r tp , the correlation degree of the maintenance mode category is r rw , the correlation degree of the maintenance process category is r rc .

[0139] The remote maintenance system can set a threshold T for the degree of correlation, which can be determined based on maintenance needs and experience. For example, through the analysis of a large number of server maintenance cases, it is found that when the degree of correlation reaches 0.5, the guidance video content of this category has a certain reference value for solving the current fault. Then, if r tp >0.5, rrw >0.5, r rc >0.5, these categories are preliminarily determined as the guidance video content of the initial candidate categories that meet the relevance requirements. Based on the relevance threshold, more candidate categories can be obtained. For further screening, the remote maintenance system can sort them according to the degree of relevance. For example, the guidance video content of the initial candidate categories is arranged from high to low according to the degree of relevance. Then, according to the preset number n (this n can be set according to actual conditions, such as according to the energy of the maintenance personnel and the urgency of the current maintenance task. If the maintenance personnel have limited time and the task is urgent, n can be set to a smaller value), the categories ranked in front are selected as the guidance video content of the candidate categories. For example, if n=2, and after sorting by the degree of relevance, r tp >r rw >r rc , then the two categories of tool preparation and maintenance methods are determined as the alternative categories of instructional video content.

[0140] In step S330, assuming that the preset categories are fault introduction and maintenance points, and the candidate categories are tool preparation and maintenance methods, the remote maintenance system determines these four categories (fault introduction, maintenance points, tool preparation, maintenance methods) as target types of instructional video content.

[0141] It is of great significance to determine the target type of instructional video content. These target types of instructional video content are screened by the remote maintenance system based on factors such as the degree of relevance to the remote maintenance monitoring video, maintenance experience, and actual maintenance needs. For remote maintenance personnel, these target types of instructional video content can provide the most valuable information to help them perform remote maintenance of servers more efficiently.

[0142] For example, the instructional video content in the fault introduction category allows maintenance personnel to quickly understand the possible causes of server failures, the instructional video content in the maintenance points category points out the key issues that need special attention during the maintenance process, the instructional video content in the tool preparation category tells maintenance personnel what tools they need to prepare, and the instructional video content in the maintenance method category shows how to repair the fault. By focusing on these target types of instructional video content, maintenance personnel can reduce the time spent on irrelevant or low-relevance video content and improve the accuracy and efficiency of maintenance.

[0143] As an implementation manner, step S320, based on the degree of association between various categories of instruction video content and the remote maintenance monitoring video, determines the candidate category of instruction video content from the remaining categories of instruction video content, including:

[0144] Step S321: Based on the correlation between various categories of instruction video content and the remote maintenance monitoring video, determine the initial candidate category of instruction video content whose corresponding correlation is greater than a preset correlation value from the remaining categories of instruction video content;

[0145] Step S322: determining a preset number of initial candidate categories of instruction video content as the candidate categories of instruction video content according to the decreasing direction of the association degree.

[0146] In step S321, in the context of remote maintenance, various types of instructional video content (such as fault introduction, tool preparation, maintenance points, maintenance methods, maintenance process, etc.) and remote maintenance monitoring videos have different degrees of association. The degree of association reflects the closeness of the connection between the two, which is obtained through a series of analyses and calculations (such as determined by feature matching, semantic analysis, etc. in the previous steps).

[0147] The remote maintenance system first needs to determine the preset correlation value. This value can be set based on a variety of factors, such as statistical analysis of historical maintenance data, the experience of maintenance experts, etc. Taking the remote maintenance of servers as an example, suppose the remote maintenance system analyzes a large number of server maintenance cases and finds that when the correlation between a certain category of instructional video content and the remote maintenance monitoring video reaches 0.5, the category of content has a certain reference value for solving the current fault. Then, 0.5 can be set as the preset correlation value.

[0148] Then, the remote maintenance system checks the remaining categories of instructional video content (i.e., categories other than the preset categories determined in step S310) one by one to see how they are related to the remote maintenance monitoring video. For example, in server maintenance, if the preset categories are fault introduction and maintenance points, the remaining categories may include tool preparation, maintenance methods, and maintenance process.

[0149] Assume that the correlation between the instructional video content in the tool preparation category and the remote maintenance monitoring video is r tp , the correlation degree of the maintenance mode category is r rw , the correlation degree of the maintenance process category is r rc If r tp >0.5, r rw >0.5, r rc >0.5, then the three categories of tool preparation, maintenance method and maintenance process are preliminarily determined as the instructional video contents of the initial candidate categories whose corresponding correlation degree is greater than the preset correlation degree value.

[0150] The method of determining the initial candidate category can be based on database query and comparison operations. The remote maintenance system can store the correlation degree values ​​between various categories of instruction video content and remote maintenance monitoring videos in a database. When executing step S321, the correlation degree values ​​of the remaining categories are queried from the database and compared with the preset correlation degree values. If the correlation degree value is greater than the preset correlation degree value, the corresponding category is marked as the initial candidate category.

[0151] After the instruction video content of the initial candidate categories is determined in step S321, there may be multiple categories that meet the condition that the correlation degree is greater than the preset correlation degree value. However, in order to further filter out the most relevant categories, the remote maintenance system needs to select in the direction of decreasing correlation degree. For example, continuing to take server maintenance as an example, assuming that in step S321, the three categories of tool preparation, maintenance method and maintenance process are determined as the instruction video content of the initial candidate categories, and the correlation degrees between them and the remote maintenance monitoring video are r and r, respectively. tp =0.6, r rw =0.55, r rc =0.52.

[0152] The remote maintenance system will sort these initial candidate categories from high to low according to the degree of relevance, that is, in the order of tool preparation, maintenance method, and maintenance process. Then, the final candidate categories are determined according to the preset number n. Assuming n=2, then in the order of decreasing degree of relevance, the two categories of tool preparation and maintenance method are determined as the instructional video content of the candidate categories. The preset number n can be adjusted according to actual conditions. If the current remote maintenance task is urgent and the maintenance personnel may not have much time to view a large amount of instructional video content, then n can be set to a smaller value so as to quickly focus on the most relevant video content. Conversely, if time is relatively abundant, n can be set to a slightly larger value to obtain more potentially useful information.

[0153] The remote maintenance system can use a sorting algorithm to sort in the direction of decreasing association. For example, the bubble sort algorithm compares the sizes of adjacent elements (here, comparing the size of the association) multiple times, and gradually "bubbles" the largest (or smallest) element to one end of the array (here, moving the category with the highest association to the front). After determining the initial order of candidate categories after sorting, the front category is selected as the final candidate category according to the preset number n.

[0154] Through steps S321-S322, the remote maintenance system can more accurately determine the instruction video content of the alternative categories among the instruction video content of the remaining categories, which helps to further screen out the instruction video content that is highly relevant to the remote maintenance monitoring video, and lays the foundation for finally determining the instruction video content of the target category (completed in step S330), thereby improving the efficiency and accuracy of remote maintenance.

[0155] As an implementation mode, step S400, highlighting and marking the video frames corresponding to the remote maintenance monitoring video in the instruction video content of the target type, may also include:

[0156] Step S440: determining a preset category of video frames in a sequence of video frames to be matched; the sequence of video frames to be matched includes video frames obtained by processing the remote maintenance monitoring video;

[0157] Step S450: highlighting and marking video frames corresponding to preset category video frames in the target type of guidance video content.

[0158] In step S440, the to-be-matched video frame sequence is a series of video frames obtained after processing the remote maintenance monitoring video, and these video frames contain important information related to the operation status, fault phenomenon, etc. of the terminal device. The preset category video frame is a video frame category with specific attributes or meanings in this sequence.

[0159] The remote maintenance system can determine the preset category video frames based on a variety of factors and methods. For example, in a remote maintenance scenario of a server, if the focus is on information related to the server hardware status, the remote maintenance system can set video frames that display the appearance and connection status of the server's main hardware components (such as CPU, memory, hard disk, etc.) as preset category video frames.

[0160] The remote maintenance system can be implemented using image recognition technology. For each video frame in the sequence of video frames to be matched, the remote maintenance system will perform image analysis. Taking the identification of the server CPU as an example, the remote maintenance system pre-stores the standard appearance feature model of the CPU, which includes information such as the shape, size, color pattern, and specific identification of the CPU.

[0161] When analyzing the video frame to be matched, the remote maintenance system extracts the image features in the video frame, such as obtaining the edge contour information of the image through the edge detection algorithm, and obtaining the color distribution features through the color histogram analysis. Then, these features are matched with the standard appearance feature model of the CPU. If the matching degree reaches a certain threshold (for example, by calculating the cosine similarity between the feature vectors, when the similarity is greater than 0.8, the match is considered successful), the video frame is marked as belonging to a preset category (here, a preset category related to the CPU) video frame.

[0162] In step S450, the target type of instructional video content is a video content category that has been previously screened out through a series of steps and has a high degree of relevance to the remote maintenance monitoring video and is most instructive for remote maintenance work. Video frames corresponding to the preset category video frames are found in the target type of instructional video content and are highlighted, which helps remote maintenance personnel to obtain key information more quickly and accurately.

[0163] The remote maintenance system can perform highlighting in a variety of ways. One way is to add a border of a specific color (such as red) to the edge of the video frame. The remote maintenance system can achieve this operation through a video editing algorithm. First, determine the correspondence between the video frames in the guidance video content of the target category and the video frames of the preset category, which can be based on image feature matching or semantic analysis (if the video frame contains relevant text descriptions).

[0164] After the corresponding relationship is determined, the remote maintenance system obtains the pixel coordinate information for each corresponding video frame. Assuming the resolution of the video frame is 1920×1080, the remote maintenance system will add red border drawing instructions on the four sides of the video frame (i.e., from coordinates (0,0) to (1920,0), (0,1080) to (1920,1080), (0,0) to (0,1080), and (1920,0) to (1920,1080)).

[0165] Another way to mark is to overlay a semi-transparent identification icon on the video frame. For example, an icon representing important information (such as a circular icon with an exclamation mark) can be overlaid in the center of the video frame. The remote maintenance system first generates this icon image, and then overlays the icon image on the target video frame in a semi-transparent form according to the coordinate system of the video frame. This process can be achieved through image synthesis technology, where the icon image and the video frame image are synthesized according to a set transparency value (such as 50% transparency), so that maintenance personnel can notice these important video frames at a glance when viewing the video.

[0166] As an implementation mode, step S400, highlighting and marking the video frames corresponding to the remote maintenance monitoring video in the instruction video content of the target type, may also include:

[0167] Step S460: determining the remaining categories of video frames other than the preset category of video frames in the sequence of video frames to be matched;

[0168] Step S470: indexing the video frames corresponding to the video frames of other categories in the guidance video content of the target category to obtain a candidate video frame sequence;

[0169] Step S480: determining a preset number of video frames in the candidate video frame sequence for highlighting and marking.

[0170] In step S460, the sequence of video frames to be matched is a series of video frames obtained after the remote maintenance system processes the remote maintenance monitoring video, and these video frames contain information on equipment operation status, fault manifestations, etc. The preset category video frames are video frame categories with specific attributes or specially defined (such as the categories determined in the previous steps based on factors such as hardware components and fault phenomena), and the remaining category video frames are video frames other than these preset categories.

[0171] The remote maintenance system can determine the remaining categories of video frames based on feature recognition and classification technology. For each video frame in the sequence of video frames to be matched, the remote maintenance system extracts its feature vector. These feature vectors can include image features (such as color, shape, texture, etc.), data features (such as displayed values, error codes, etc.) and time series features (the temporal position relationship of the video frame in the entire sequence, etc.).

[0172] For example, for a video frame showing the read / write status of a server hard disk, the remote maintenance system extracts its color features (the color of the hard disk indicator light), shape features (the outline of the hard disk), data features (the numerical display of the hard disk read / write speed), etc., and combines them into a feature vector. Then, by comparing the feature vectors with the feature vectors of the preset category video frames, if the feature vector does not meet the feature definition of the preset category video frames (for example, its feature vector is significantly different from the feature vector of the CPU-related video frames), the video frame is determined as a video frame of the remaining categories.

[0173] In step S470, the target type of instructional video content is the instructional video content category that is highly associated with the remote maintenance monitoring video and is of great significance to remote maintenance, which is selected in the previous steps. For example, it may include categories such as fault introduction, maintenance points, and maintenance methods.

[0174] The remote maintenance system needs to find video frames corresponding to the remaining categories of video frames in the guidance video content of these target categories. Taking server maintenance as an example, if the remaining categories of video frames include video frames related to the server hard disk (such as video frames showing abnormal hard disk read and write status), in the guidance video content of the target categories, the guidance video content of the fault introduction category may have video frames describing the causes of hard disk failure (such as hard disk aging, track damage, etc.), and the guidance video content of the maintenance points category may have video frames about key operations that need to be paid attention to when handling hard disk failures (such as data backup, precautions for replacing hard disks, etc.). These video frames correspond to the hard disk-related video frames in the remaining categories of video frames.

[0175] In order to index these corresponding video frames, the remote maintenance system can use a combination of semantic analysis and image feature matching. For the video frames in the target category of instructional video content, the remote maintenance system first performs semantic analysis. If the video frame contains text descriptions (such as text explanations of fault phenomena, text descriptions of operating steps, etc.), the remote maintenance system uses natural language processing technology to convert these texts into semantic vectors. At the same time, for the remaining categories of video frames, their relevant semantic information is also extracted (such as obtaining semantic information from displayed values, error codes, etc.) and converted into semantic vectors.

[0176] Then, the similarity between the semantic vectors (such as cosine similarity) is calculated to preliminarily screen the possible corresponding video frames. For example, if the calculated semantic similarity exceeds a certain threshold (such as 0.7), these video frames are considered to be semantically related. For the video frames preliminarily screened out by semantic analysis, the remote maintenance system further performs image feature matching. For the image part of the video frame, the remote maintenance system extracts its image feature vector (such as the color, shape, texture and other features mentioned above), and then calculates the similarity of the image feature vector between the video frame in the target category of the guidance video content and the video frames of the remaining categories (cosine similarity can also be used). If the image feature similarity also exceeds a certain threshold (such as 0.6), this video frame in the guidance video content of the target category is determined as the video frame corresponding to the video frames of the remaining categories, and these video frames constitute a candidate video frame sequence.

[0177] In step S480, after obtaining the candidate video frame sequence, since it may contain a large number of video frames, the remote maintenance system needs to further select a portion of the video frames for highlighting and marking, so as to improve the efficiency of remote maintenance personnel in obtaining key information.

[0178] The preset number is a value determined according to actual needs. This value can be set according to factors such as the attention span of remote maintenance personnel, the urgency of the maintenance task, and the complexity of the video content. For example, in an urgent server maintenance task, the remote maintenance personnel may not have much time to view a large number of video frames. In this case, the preset number can be set to a smaller value (such as 5); if the maintenance task is relatively less urgent and the video content is relatively complex, more reference information is required, and the preset number can be set to a larger value (such as 15).

[0179] The method for the remote maintenance system to determine the preset number of video frames to be annotated can be based on multiple strategies. One strategy is based on the importance assessment of the video frames. The remote maintenance system can calculate an importance score for each video frame in the candidate video frame sequence. This score can comprehensively consider the semantic importance and image importance of the video frame.

[0180] For semantic importance, the remote maintenance system can calculate it based on the degree of association between the semantic information contained in the video frame and the key information in the remote maintenance monitoring video. For example, if a video frame accurately describes the cause of a fault that is highly related to the hard disk fault phenomenon in the remote maintenance monitoring video in the guidance video content of the fault introduction category (e.g., the remote maintenance monitoring video shows that the hard disk read and write speed is abnormal, and the video frame explains in detail the principle that the track damage causes the abnormal read and write speed), it will be given a higher semantic importance score.

[0181] For image importance, the remote maintenance system can calculate it based on the degree of match between the image features of the video frame and the key image features in the remote maintenance monitoring video. For example, if there is a video frame in the remote maintenance monitoring video showing obvious signs of physical damage to the hard disk, and a video frame in the candidate video frame sequence shows similar image details of hard disk damage, then this video frame will receive a higher image importance score.

[0182] The total importance score of each video frame is obtained by combining the semantic importance score and the image importance score (for example, a weighted sum can be used, assuming that the semantic importance weight is 0.6 and the image importance weight is 0.4). Then, the video frames in the candidate video frame sequence are sorted from high to low according to the importance score, and a preset number of video frames are selected for highlighting. The method of highlighting is not described here.

[0183] As an implementation mode, step S410, indexing a plurality of video frames to be annotated corresponding to the remote maintenance monitoring video in the guidance video content of the target type, includes:

[0184] Step S411: acquiring a first video frame and a second video frame distributed front and back in the remote maintenance monitoring video;

[0185] Step S412: indexing the guidance video content of the target category to obtain a third video frame corresponding to the first video frame and a fourth video frame corresponding to the second video frame.

[0186] Step S420, determining the freezing coefficients between the plurality of video frames to be labeled according to the distribution sequence of the plurality of video frames to be labeled in the guidance video content of the target category, may include:

[0187] Step S421: determining a freezing coefficient between the third video frame and the fourth video frame based on a distribution timing of the third video frame in the target type of instructional video content and a distribution timing of the fourth video frame in the target type of instructional video content.

[0188] Step S430, if the freezing coefficients between the multiple video frames to be annotated meet the set requirements, highlighting and annotating the video frames corresponding to the remote maintenance monitoring video in the instructional video content of the target type, including:

[0189] Step S431: If the freezing coefficient between the third video frame and the fourth video frame meets the set requirements, a video frame corresponding to the remote maintenance monitoring video is determined in the guidance video content of the target type, and the obtained video frame is highlighted.

[0190] In step S411, the first video frame and the second video frame distributed in the remote maintenance monitoring video are obtained. The remote maintenance monitoring video contains various status information of the terminal device during operation, and the remote maintenance system needs to select specific video frames from it to establish an association with the guidance video content of the target type. The first video frame and the second video frame here are two representative or key position video frames in this video.

[0191] For example, in a remote maintenance monitoring video of a server, if a server fails, the first video frame may be a picture when the failure just begins to show signs, such as the frame in which a certain indicator light of the server begins to flash abnormally. This video frame may contain key information at the initial stage of the failure, such as visual features such as the flashing frequency and color of the indicator light, which may indicate the type of failure or the location of the failure. The second video frame may be a picture when the failure develops to a certain critical stage, such as the frame in which the server begins to display a system error prompt. In addition to the previous abnormal flashing of the indicator light, this frame may also contain information such as a specific error code displayed on the screen.

[0192] The remote maintenance system can obtain these two video frames based on an event detection algorithm. For the event of abnormal flashing of the server indicator light, the remote maintenance system can pre-set a model of the normal indicator light flashing pattern. When the difference between the indicator light flashing pattern in the video frame and the normal pattern exceeds a certain threshold, the video frame is determined as the first video frame. For the second video frame, the remote maintenance system can monitor the text information in the video frame (extracted through optical character recognition technology OCR). When a specific system error prompt text is detected, it is determined as the second video frame.

[0193] In step S412, a third video frame corresponding to the first video frame and a fourth video frame corresponding to the second video frame are indexed in the target type of instruction video content. The target type of instruction video content is a video content category that has been previously screened and determined to be highly associated with the remote maintenance monitoring video and is of great significance to the maintenance work.

[0194] For example, assume that the target category of instructional video content includes two categories: fault introduction and maintenance points. If the first video frame (the server indicator light begins to flash abnormally) is in the remote maintenance monitoring video, the remote maintenance system searches for the corresponding video frame in the instructional video content of the fault introduction category. There may be a video frame that shows a similar abnormal flashing of the indicator light and introduces the possibility of the fault represented by this abnormal flashing. This video frame is the third video frame. Similarly, for the second video frame (a system error prompt appears on the server), there may be a video frame in the instructional video content of the maintenance points category that shows the preliminary maintenance operations for this error prompt. This video frame is the fourth video frame.

[0195] The remote maintenance system can index these corresponding video frames based on feature matching and semantic analysis. For image feature matching, the remote maintenance system extracts image feature vectors for the first video frame and the video frame in the target type of instructional video content. For example, for the flashing indicator light area in the first video frame, the features such as color and flashing frequency are extracted and converted into feature vectors, and the same operation is performed on the video frame in the target type of instructional video content. Then, by calculating the similarity between the feature vectors (such as cosine similarity), when the similarity exceeds a certain threshold (such as 0.8), it is considered to be a corresponding video frame (i.e., the third video frame). For semantic analysis, if the first video frame contains some elements that can be converted into semantic information (such as the fault semantics that may be implied by the flashing indicator light), the text description in the video frame in the target type of instructional video content is semantically analyzed (converted into a semantic vector), and the semantic similarity between the two is calculated to assist in determining the third video frame. The same method is used to determine the fourth video frame corresponding to the second video frame.

[0196] In step S421, based on the distribution timing of the third video frame in the target type of instructional video content and the distribution timing of the fourth video frame in the target type of instructional video content, the freezing coefficient between the third video frame and the fourth video frame is determined. In the target type of instructional video content, the distribution timing of the video frames reflects the logical order and coherence of the content.

[0197] For example, in a guidance video content about the target category of server maintenance, the third video frame appears at the 10th second of the video, showing the operation of preliminary detection of server failure (such as detecting the server memory status), and the fourth video frame appears at the 30th second, showing further maintenance operations based on the preliminary detection results (such as replacing the problematic memory module). The remote maintenance system determines the solidification coefficient between the two video frames based on the distribution time sequence of the two video frames. One way for the remote maintenance system to determine the solidification coefficient can be based on a comprehensive consideration of the time interval and the content logical relationship. First, calculate the time interval T=30-10=20 seconds between the third video frame and the fourth video frame. Then, analyze the content logical relationship between the two video frames. If it is a very reasonable and tight logical process from preliminary detection to maintenance based on the detection results, the remote maintenance system can assign a higher logical relationship value L (for example, L=0.8, the value range can be 0-1, 1 indicates a very tight logical relationship) to this logical relationship.

[0198] Next, the remote maintenance system can calculate the freezing coefficient C according to a pre-set formula, such as C=f(T)×L, where f(T) is a function of the time interval T. Assuming that f(T) is defined as f(T)=1-T / 60 (here 60 is a time reference value set based on experience) when T≤30 seconds, then in this example f(T)=1-20 / 30=2 / 3, so the freezing coefficient C=2 / 3×0.8=8 / 15. This freezing coefficient reflects the degree of logical coherence between the third video frame and the fourth video frame in the guidance video content of the target category.

[0199] In step S431, if the freezing coefficient between the third video frame and the fourth video frame meets the set requirements, the video frame corresponding to the remote maintenance monitoring video is determined in the guidance video content of the target type, and the obtained video frame is highlighted. The set requirements are a standard predetermined according to the actual needs and experience of remote maintenance, which is used to determine whether the logical coherence between video frames is strong enough, so as to determine whether to operate the relevant video frames.

[0200] The process of the remote maintenance system determining the video frames corresponding to the remote maintenance monitoring video is as follows: Since the solidification coefficient between the third video frame and the fourth video frame meets the requirements, it means that they are logically coherent and have a corresponding relationship with the first video frame and the second video frame in the remote maintenance monitoring video. The remote maintenance system will expand and determine other video frames corresponding to the remote maintenance monitoring video based on the third video frame and the fourth video frame. For example, if in the guidance video content of the target type, there are some video frames before the third video frame that are a preliminary introduction to the fault phenomenon (related to the comparison with the normal state before the fault starts in the remote maintenance monitoring video), and there are some video frames after the fourth video frame that are verification of the maintenance results (related to the state after the fault is repaired in the remote maintenance monitoring video), the remote maintenance system will also determine these video frames as video frames corresponding to the remote maintenance monitoring video.

[0201] The method of highlighting these video frames will not be described here. The remote maintenance system superimposes the icon image in a semi-transparent form at the center of the video frame according to the coordinate system of the video frame. Through these highlighting methods, remote maintenance personnel can more easily identify the key video frames corresponding to the remote maintenance monitoring video when viewing the guidance video content of the target type, thereby improving the efficiency and accuracy of remote maintenance.

[0202] As an implementation manner, step S421, based on the distribution timing of the third video frame in the guidance video content of the target category and the distribution timing of the fourth video frame in the guidance video content of the target category, determines the freezing coefficient between the third video frame and the fourth video frame, including:

[0203] Step S4211: determining feature similarity between the third video frame and the fourth video frame based on the distribution timing of the third video frame in the target type of instruction video content and the distribution timing of the fourth video frame in the target type of instruction video content;

[0204] Step S4212: determining a freezing coefficient between the third video frame and the fourth video frame based on the distribution influence coefficients of the third video frame and the fourth video frame and the feature similarity between the third video frame and the fourth video frame.

[0205] In step S4211, in the target type of instruction video content, each video frame contains specific information, and its distribution sequence reflects the logical order of the video content. The remote maintenance system determines the feature similarity by analyzing the positions of the third video frame and the fourth video frame in this sequence and the information they contain.

[0206] For example, in the scenario of remote maintenance of a server, the target type of instructional video content may be a video about the repair of server hardware failures. Assume that the third video frame appears in the early part of this instructional video content, showing the appearance inspection of the server memory module, including the number of memory chips, the status of the gold finger and other appearance features; the fourth video frame appears in the later part, showing the use of tools for further testing of the memory module, such as using a memory test card to test the read and write status of the memory.

[0207] The remote maintenance system first extracts features from the third and fourth video frames. For the third video frame, the extracted features may include image features (such as visual features such as the color, shape, and texture of the memory module), text features (if there are text annotations about the appearance inspection of the memory in the video frame, etc.), and semantic features (such as semantic information about the appearance inspection of the memory in the entire maintenance process obtained by analyzing the relevant text using natural language processing technology). For the fourth video frame, these types of features are also extracted, such as the appearance of the detection tool, the visual features of the connection between the detection card and the memory interface, the text features corresponding to the text annotations of the operation steps, and the semantic features about the detection operation in the maintenance process.

[0208] Then, the remote maintenance system calculates the similarity between these features. For the similarity calculation of image features, the structural similarity index (SSIM) algorithm can be used. SSIM measures the similarity of two images by comparing their brightness, contrast, and structural information. Let the image feature of the third video frame be I3, and the image feature of the fourth video frame be I4, and calculate the SSIM value of the image feature SSIM(I3,I4).

[0209] For the similarity of text features, the text can be converted into a vector representation. For example, a word vector model (such as Word2Vec) is used to map the words in the text into vectors, and then the cosine similarity between the vectors is calculated. Finally, the remote maintenance system combines the similarities of these different types of features to determine the feature similarity between the third video frame and the fourth video frame. A possible comprehensive method is weighted averaging. Assuming that the weights of image features, text features, and semantic features are w1, w2, and w3 respectively (and w1+w2+w3=1), the feature similarity F=w1×SSIM(I3,I4)+w2×cos(T3,T4)+w3×cos(S3,S4). For example, w1=0.4, w2=0.3, w3=0.3. Through the above calculation, we can get SSIM(I3,I4)=0.6, cos(T3,T4)=0.5, cos(S3,S4)=0.7, then the feature similarity F=0.4×0.6+0.3×0.5+0.3×0.7=0.58.

[0210] In step S4212, in the target type of guidance video content, the distribution positions of the third video frame and the fourth video frame have different importance, namely, distribution influence coefficients, which reflect the importance of the video frame in the entire video content logical structure.

[0211] Continuing with the above-mentioned server maintenance instruction video as an example, assuming that the third video frame is in the key part of introducing the initial fault detection steps, its distribution influence coefficient D3 may be set to a higher value, such as D3=0.8, because it is the basis for subsequent maintenance operations and is very important for understanding the entire maintenance process. Although the fourth video frame is also in the maintenance process, it may be relatively less important, and its distribution influence coefficient D4=0.6.

[0212] The remote maintenance system determines the coagulation coefficient C according to a pre-set formula, for example , where F is the feature similarity determined in step S4211. This way of determining the solidification coefficient takes into account the feature similarity of the video frames and their distribution influence coefficients in the video content. The feature similarity measures the similarity between the third video frame and the fourth video frame in content, while the distribution influence coefficient takes into account their importance in the video logical structure. The solidification coefficient obtained by this comprehensive calculation can more comprehensively reflect the relationship between the third video frame and the fourth video frame, and provides an important basis for the subsequent judgment of whether to highlight the video frame corresponding to the remote maintenance monitoring video in the guidance video content of the target type. If the solidification coefficient meets the set requirements (such as being greater than a certain threshold), it means that the two video frames have a strong logical coherence, and then the relevant video frames can be determined and highlighted, which helps remote maintenance personnel to better understand and use the guidance video content for remote maintenance of terminal equipment.

[0213] Figure 2 A hardware entity diagram of a remote maintenance system provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the hardware entity of the remote maintenance system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

[0214] The above description is only an implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A terminal device remote maintenance method based on AI, characterized in that: Applied to a remote maintenance system, the method comprises: Acquire a target maintenance instruction video corresponding to the remote maintenance monitoring video, wherein the target maintenance instruction video includes instruction video contents of multiple categories; Determining the degree of association between the various categories of instruction video content included in the target maintenance instruction video and the remote maintenance monitoring video based on the importance coefficients of the multiple categories of instruction video content; Determining a target type of instruction video content from among the multiple types of instruction video content based on the degree of association between the various types of instruction video content and the remote maintenance monitoring video; Highlighting and marking the video frames corresponding to the remote maintenance monitoring video in the guidance video content of the target type; The step of highlighting and marking the video frames corresponding to the remote maintenance monitoring video in the guidance video content of the target type includes: Indexing in the guidance video content of the target type to obtain a plurality of video frames to be annotated corresponding to the remote maintenance monitoring video; Determining a freezing coefficient between the plurality of video frames to be labeled according to the distribution time sequence of the plurality of video frames to be labeled in the guidance video content of the target type, wherein the freezing coefficient represents the time distribution relationship of the video frames to be labeled in the video content, so as to determine whether there is logical coherence; If the freezing coefficients between the multiple video frames to be annotated meet the set requirements, the video frames corresponding to the remote maintenance monitoring video in the guidance video content of the target type are highlighted and annotated.

2. The method according to claim 1, characterized in that: The step of determining the degree of association between the various categories of instruction video content included in the target maintenance instruction video and the remote maintenance monitoring video based on the importance coefficients of the multiple categories of instruction video content includes: Processing the remote maintenance monitoring video to obtain a video frame to be matched; Determining the initial correlation degree between the various categories of instruction video content and the remote maintenance monitoring video based on the to-be-matched video frame; Based on the initial correlation degree between the various categories of instruction video content and the remote maintenance monitoring video, and the importance coefficient of the multiple categories of instruction video content, the correlation degree between the various categories of instruction video content and the remote maintenance monitoring video is determined.

3. The method according to claim 2, characterized in that The determining, based on the to-be-matched video frame, the initial degree of association between the various categories of instruction video content and the remote maintenance monitoring video comprises: Indexing in the various categories of guidance video content to obtain a target video frame corresponding to the to-be-matched video frame; Based on the influence coefficient of the to-be-matched video frame and the influence coefficient of the target video frame corresponding to the to-be-matched video frame in the various categories of guidance video content, the initial degree of association between the various categories of guidance video content and the remote maintenance monitoring video is determined.

4. The method according to claim 1, characterized in that: The number of the target maintenance guidance videos is multiple, and the method further includes: Determining the importance coefficient of each target maintenance guidance video based on the pairing method between the remote maintenance monitoring video and each target maintenance guidance video; Determine the degree of association between each target maintenance instruction video and the remote maintenance monitoring video based on the importance coefficient of each target maintenance instruction video and the degree of association between various categories of instruction video content included in each target maintenance instruction video and the remote maintenance monitoring video; Arranging the multiple target maintenance guidance videos in order based on the degree of association between each target maintenance guidance video and the remote maintenance monitoring video; The target maintenance instruction video corresponding to the remote maintenance monitoring video is obtained, including: Processing the remote maintenance monitoring video to obtain a to-be-matched video frame sequence; Classifying the video frames in the to-be-matched video frame sequence to obtain a plurality of video frame groups; Target maintenance guidance videos corresponding to the multiple video frame groups are respectively obtained, and each video frame group corresponds to a pairing method.

5. The method according to claim 4, characterized in that The step of processing the remote maintenance monitoring video to obtain a sequence of video frames to be matched includes: Detecting the video semantics of the remote maintenance monitoring video to obtain the video semantic tag of the remote maintenance monitoring video; A video frame splitting operation is performed on the remote maintenance monitoring video, and the to-be-matched video frame sequence is established based on the video semantic tags and the video frames obtained by the video frame splitting operation.

6. The method according to claim 1, characterized in that The step of determining a target type of guidance video content from among the multiple types of guidance video content based on the degree of association between the various types of guidance video content and the remote maintenance monitoring video includes: Determining, among the plurality of categories of instruction video contents, instruction video contents of a preset category and instruction video contents of other categories other than the preset category; Determining the alternative category of instruction video content from the remaining categories of instruction video content based on the degree of association between the various categories of instruction video content and the remote maintenance monitoring video; The guiding video content of the candidate category and the guiding video content of the preset category are determined as the guiding video content of the target category.

7. The method according to claim 6, characterized in that The step of determining the candidate category of instruction video content from the remaining categories of instruction video content based on the degree of association between the various categories of instruction video content and the remote maintenance monitoring video comprises: Based on the correlation degree between the various categories of instruction video content and the remote maintenance monitoring video, determining the initial candidate category of instruction video content whose corresponding correlation degree is greater than a preset correlation degree value from the remaining categories of instruction video content; A preset number of initial candidate categories of instruction video content are determined as the instruction video content of the candidate category according to the direction of decreasing association degree.

8. The method according to claim 1, characterized in that The step of highlighting and marking the video frames corresponding to the remote maintenance monitoring video in the guidance video content of the target type further includes: Determining a preset category of video frames in a sequence of video frames to be matched, wherein the sequence of video frames to be matched includes video frames obtained by processing the remote maintenance monitoring video; Highlighting and marking video frames corresponding to the preset category video frames in the guidance video content of the target category; The step of highlighting and marking the video frames corresponding to the remote maintenance monitoring video in the guidance video content of the target type further includes: Determine the remaining categories of video frames other than the preset category of video frames in the to-be-matched video frame sequence; Indexing video frames corresponding to the remaining categories of video frames in the guidance video content of the target category to obtain a candidate video frame sequence; A preset number of video frames are determined in the candidate video frame sequence for highlighting and marking.

9. The method according to claim 1, characterized in that: The step of indexing the target type's instruction video content to obtain a plurality of to-be-annotated video frames corresponding to the remote maintenance monitoring video comprises: Acquire a first video frame and a second video frame distributed front and back in the remote maintenance monitoring video; Indexing in the guidance video content of the target category to obtain a third video frame corresponding to the first video frame and a fourth video frame corresponding to the second video frame; The step of determining the freezing coefficients between the plurality of video frames to be labeled according to the distribution sequence of the plurality of video frames to be labeled in the guidance video content of the target category comprises: determining a freezing coefficient between the third video frame and the fourth video frame based on a distribution timing of the third video frame in the instruction video content of the target category and a distribution timing of the fourth video frame in the instruction video content of the target category; If the freezing coefficients between the plurality of video frames to be annotated meet the set requirements, highlighting and annotating the video frames corresponding to the remote maintenance monitoring video in the guidance video content of the target type includes: If the coagulation coefficient between the third video frame and the fourth video frame meets the set requirement, determining a video frame corresponding to the remote maintenance monitoring video in the guidance video content of the target type, and highlighting the obtained video frame; Wherein, determining the freezing coefficient between the third video frame and the fourth video frame based on the distribution timing of the third video frame in the instruction video content of the target type and the distribution timing of the fourth video frame in the instruction video content of the target type includes: Determining a feature similarity between the third video frame and the fourth video frame based on a distribution timing of the third video frame in the instruction video content of the target category and a distribution timing of the fourth video frame in the instruction video content of the target category; A freezing coefficient between the third video frame and the fourth video frame is determined based on the distribution influence coefficients of the third video frame and the fourth video frame and the feature similarity between the third video frame and the fourth video frame.

10. A remote maintenance system, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 9 are implemented.

Images