An augmented reality-oriented maintenance guide information extraction and display method
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2026-03-20
- Publication Date
- 2026-06-19
Smart Images
Figure CN122240782A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of equipment maintenance technology, and in particular to a method for extracting and displaying maintenance guidance information for augmented reality. Background Technology
[0002] Existing complex systems are characterized by a wide variety of equipment and a high degree of integration. As a result, when different equipment in a complex system fails and requires repair, the corresponding repair activities are also characterized by a wide variety of types and a high degree of specialization. Consequently, repair personnel often need to consult repair manuals when performing equipment testing and repair, resulting in low repair efficiency.
[0003] Augmented Reality (AR) technology can overlay virtual maintenance guidance information onto the field of vision of maintenance personnel in real time and accurately through virtual-real fusion, guiding maintenance operations in an intuitive and immersive way. It has been proven to significantly improve maintenance efficiency and quality. Therefore, in recent years, the auxiliary application of AR technology in the field of inspection and maintenance has become increasingly widespread. However, the extraction of maintenance guidance information currently used by AR technology heavily relies on manual compilation and annotation, hindering the deployment of AR technology. Summary of the Invention
[0004] The purpose of this application is to provide a method for extracting and displaying maintenance guidance information based on augmented reality, which can automatically extract and display maintenance guidance information, significantly improving maintenance efficiency and quality.
[0005] To achieve the above objectives, this application provides the following solution.
[0006] This application provides a method for extracting and displaying maintenance guidance information for augmented reality, which includes the following steps.
[0007] Based on user input commands, the target maintenance activity text required for the current maintenance activity is selected from the maintenance activity text database; the maintenance activity text database includes multiple maintenance activity texts.
[0008] The similarity between the target maintenance activity text and each sample maintenance activity text in the benchmark database is calculated respectively, and the initial similarity corresponding to each category is determined based on all the similarities; the sample maintenance activity text has a classification label, and the classification label is the category of the sample maintenance activity text.
[0009] For each category, determine whether the target maintenance activity text includes the keywords corresponding to the category, obtain the determination result, and correct the initial similarity corresponding to the category based on the determination result to obtain the corrected similarity corresponding to the category.
[0010] The category with the highest similarity after correction is selected as the category of the target maintenance activity text.
[0011] Based on the category of the target maintenance activity text, information is extracted from the target maintenance activity text to obtain maintenance guidance information.
[0012] The maintenance guidance information is displayed on the human-computer interaction interface of the augmented reality device.
[0013] According to the specific embodiments provided in this application, this application has the following technical effects.
[0014] This application provides a method for extracting and displaying maintenance guidance information for augmented reality. Based on user input commands, it selects the target maintenance activity text required for the current maintenance activity from a maintenance activity text database. It calculates the similarity between the target maintenance activity text and each sample maintenance activity text in the benchmark database. Based on all similarities, it determines the initial similarity for each category. For each category, it determines whether the target maintenance activity text contains the keywords corresponding to that category, obtaining a judgment result. Based on the judgment result, it corrects the initial similarity for each category, obtaining a corrected similarity. The category with the highest corrected similarity is selected as the category of the target maintenance activity text. Based on the category of the target maintenance activity text, it extracts information from the target maintenance activity text to obtain maintenance guidance information, which is then displayed on the human-computer interaction interface of the augmented reality device. This application classifies the target maintenance activity text, automatically extracts and displays maintenance guidance information, eliminating the need for heavy reliance on manual compilation and annotation. This automatic extraction and display of maintenance guidance information significantly improves maintenance efficiency and quality. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is an application environment diagram for a method for extracting and displaying maintenance guidance information based on augmented reality, provided in Embodiment 1 of this application.
[0017] Figure 2 This is a flowchart illustrating a method for extracting and displaying maintenance guidance information for augmented reality, as provided in Embodiment 1 of this application.
[0018] Figure 3This is a schematic diagram of the layout of the human-computer interaction interface provided in Embodiment 1 of this application.
[0019] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 2 of this application. Detailed Implementation
[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0021] Example 1.
[0022] The method for extracting and displaying maintenance guidance information for augmented reality provided in this application can be applied to, for example... Figure 1 The application environment shown is as follows. The terminal communicates with the server via a network. The data storage system stores the data the server needs to process. The data storage system can be set up independently, integrated into the server, or placed in the cloud or on another server. The terminal can send extraction and display requests to the server. Upon receiving the requests, the server, based on user input, selects the target maintenance activity text from the maintenance activity text database; calculates the similarity between the target maintenance activity text and each sample maintenance activity text in the benchmark database; determines the initial similarity for each category based on all similarities; for each category, it determines whether the target maintenance activity text contains the corresponding keywords, obtains the judgment result, and corrects the initial similarity for the category based on the judgment result, obtaining the corrected similarity for the category; selects the category with the highest corrected similarity as the category of the target maintenance activity text; extracts information from the target maintenance activity text based on its category to obtain maintenance guidance information; and displays the maintenance guidance information on the human-computer interaction interface of the augmented reality device. The server can feed back the extracted maintenance guidance information (the result of the extraction and display request) and the display result of the human-computer interaction interface to the terminal.
[0023] In addition, in some embodiments, the method for extracting and displaying maintenance guidance information for augmented reality can also be implemented by a server or a terminal. For example, the terminal can directly process the extraction and display requests to be processed, or the server can obtain the extraction and display requests to be processed from the data storage system and process them.
[0024] In one exemplary embodiment, such as Figure 2 As shown, a method for extracting and displaying maintenance guidance information based on augmented reality is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 The following steps are used as an example of a server in the example.
[0025] Step S1: Based on the user input command, select the target maintenance activity text required for the current maintenance activity from the maintenance activity text database; the maintenance activity text database includes multiple maintenance activity texts.
[0026] Step S2: Calculate the similarity between the target maintenance activity text and each sample maintenance activity text in the benchmark database, and determine the initial similarity corresponding to each category based on all the similarities; the sample maintenance activity text has a classification label, which is the category of the sample maintenance activity text.
[0027] Step S3: For each category, determine whether the target maintenance activity text includes the keywords corresponding to the category, obtain the determination result, and correct the initial similarity corresponding to the category based on the determination result to obtain the corrected similarity corresponding to the category.
[0028] Step S4: Select the category with the highest similarity after correction as the category of the target maintenance activity text.
[0029] Step S5: Based on the category of the target maintenance activity text, extract information from the target maintenance activity text to obtain maintenance guidance information.
[0030] Step S6: Display the maintenance guidance information on the human-computer interaction interface of the augmented reality device.
[0031] By implementing steps S1 to S6 above, this embodiment can automatically extract and display maintenance guidance information, significantly improving maintenance efficiency and quality.
[0032] Currently, the extraction of maintenance guidance information used in AR technology heavily relies on manual compilation and annotation, hindering the deployment of AR technology. Therefore, based on a careful analysis of the characteristics and requirements of maintenance guidance information used in AR technology, the applicant proposes a maintenance guidance information extraction method based on Natural Language Processing (NLP). This method involves data cleaning and extraction, data classification and reasoning of the text data related to equipment maintenance activities, automatically annotating and summarizing the required data. It can match the current data input format of AR technology, reducing the difficulty of automatic data processing in AR technology, improving the efficiency of manual processing, and avoiding subjective bias. Simultaneously, a maintenance guidance information display method is proposed, improving the readability and efficiency of maintenance guidance information, effectively supporting the promotion and application of AR technology in the field of inspection and maintenance.
[0033] Specifically, this embodiment designs the following four steps: Step 1, firstly, clarifies the requirements and standards for inputting maintenance guidance information in AR technology applications, guiding the design of subsequent maintenance guidance information extraction methods; Step 2, secondly, proposes a set of nine major categories of text classification rules covering the entire maintenance process, combining professionalism and applicability to meet the structured needs of maintenance guidance information; Step 3, then, proposes a maintenance guidance information extraction method to extract key information (i.e., maintenance guidance information) from maintenance activity texts, achieving precise and focused extraction of maintenance guidance information; Step 4, finally, proposes a maintenance guidance information display method to simulate augmented maintenance application scenarios, visually demonstrating the effect of batch extraction and application of maintenance guidance information, reflecting the practical application value of the maintenance guidance information extraction method.
[0034] In the above four steps, step 1 establishes the required standards for inputting maintenance guidance information when applying AR technology in maintenance work, providing text cleaning and information extraction specifications for step 3; step 2 proposes a text classification rule system for the entire maintenance guidance process, uniformly defining the structure of key information under each category, providing support for step 3; step 3 proposes a text processing algorithm based on keyword recognition and weighted similarity to perform data processing for maintenance activity texts, mainly including data cleaning, classification reasoning, and annotation extraction, providing a data foundation for step 4; step 4 designs the distribution of maintenance guidance information on the AR display terminal (i.e., augmented reality device) in augmented maintenance application scenarios, realizing the simulated application of the information extracted in step 3, and visually demonstrating the practicality and accuracy of the maintenance guidance information extraction method.
[0035] Step 1: Develop the required standards for maintenance guidance information.
[0036] Paper repair manuals typically contain 70% general descriptions and only 30% specific operational details, requiring technicians to sift through hundreds of pages to find fault codes, component parameters, or disassembly / reassembly procedures. However, by extracting key information using repair guidance methods and displaying keywords on an AR display, information density can be increased, saving time spent searching.
[0037] The extracted maintenance guidance information must meet the following requirements and standards.
[0038] Standard 1: Precise information focus requirement, from full redundancy to minimum necessary information flow.
[0039] (1) Necessity principle: Only display key information within the current operating field of view (such as target components, tool parameters, and operating thresholds), and retain only the necessary and sufficient conditions and core parameters for completing the current operation. Maintenance guidance information should be accurate and efficient. The significance of extracting maintenance manuals lies in avoiding the "information overload" problem of traditional paper maintenance manuals, saving maintenance personnel the time and effort of screening and matching corresponding tasks.
[0040] (2) Task-driven principle: Establish a mapping matrix of "operation action - information element" and set different information feedback for different maintenance operations. For example, the core information of "circuit welding" should include at least exclusive elements such as "solder joint temperature 350℃" and "welding wire diameter 0.8mm". Maintenance personnel directly face the maintenance task, so task-driven maintenance guidance information prompts are the most efficient information focusing.
[0041] Standard 2: Data structuring requirements, from unstructured text to machine-parseable knowledge units.
[0042] Complex maintenance manuals contain core knowledge of equipment maintenance, but due to the lack of a unified data organization format, they are difficult for machines to directly analyze and utilize.
[0043] Text classification of maintenance activities is a crucial logical foundation and organizational method for achieving data structuring. From a data processing perspective, the classification process is essentially an abstraction and summarization of maintenance knowledge. It groups maintenance information with similar attributes or characteristics into categories, providing a clear framework for subsequent structured processing of maintenance guidance information. Once maintenance activities are categorized into areas such as inspection, replacement, and disassembly, data processing for each category can employ relatively fixed patterns and rules. For example, for inspection data, the focus is on extracting structured information such as the inspection object and methods; for replacement data, the emphasis is on identifying the names and models of the parts to be replaced, the replacement steps, and the required tools. This classification-based processing method ensures the accuracy and consistency of data structuring, enabling machines to understand and process maintenance knowledge more efficiently.
[0044] Step 1 clarifies the essential characteristics of maintenance guidance information. Analysis of requirement standard 1 shows that maintenance guidance information needs to be sufficiently concise and critical. This goal guides step 3, which involves cleaning up irrelevant information and establishing part-of-speech relationships to maximize the accuracy and conciseness of the maintenance guidance information. Analysis of requirement standard 2 shows that maintenance guidance information needs to be presented in a structured form. Therefore, step 2 must categorize the maintenance process and establish a structured data format under the categorization rules.
[0045] Step 2: Construct a text classification rule system that covers the entire maintenance guidance process.
[0046] Since repair standards vary across different products and models, current repair work cards, manuals, and other repair guidelines are primarily unstructured free text. Their publication and use lack unified guidelines, failing to meet the conditions for directly extracting structured data. Therefore, text classification and labeling for structured data is a prerequisite for the application of AR technology.
[0047] (1) Development of a standardized text classification rule system for maintenance activities.
[0048] Based on relevant standards and guidelines, this paper clarifies the importance of maintenance objects, tools (maintenance equipment, facilities, and materials), consumables, actions, and other identifying information in maintenance activity texts, and proposes a text classification rule system for maintenance activities determined by key field information. This system focuses on actual operations in maintenance activities, classifying maintenance activity texts into nine categories based on real maintenance behaviors: moving, replacing, inspecting, adjusting, tightening, disassembling, plugging / unplugging, cleaning, and finger operation. This text classification rule system, which focuses on actual maintenance operations, meets the application requirements of AR technology. The specific text classification rule system is shown in Table 1.
[0049] Table 1. Text Classification Rule System for Maintenance Activities
[0050] For example, a maintenance activity text like "Do a visual check of the labyrinth seals on the primary nozzle for damage" contains the keyword "check," which points to category "3" in the text classification rule system, and can be classified as an inspection activity. Another example is a maintenance activity text like "Remove the nut, washer, and bolt from the actuator clevis," which contains the keywords "nut" and "bolt," pointing to category "5" in the text classification rule system, and can be classified as a tightening activity.
[0051] (2) Design and maintain the structured data format of maintenance guidance information.
[0052] Based on the above text classification rule system, and taking into account the feature requirements of maintenance guidance information, key information of maintenance activities, and text classification, the structured data format after the extraction of maintenance guidance information is divided, as shown in Table 2.
[0053] Table 2. Structured data format for maintenance guidance information.
[0054] In this embodiment, the categories include moving, replacing, inspecting, adjusting, twisting, disassembling, plugging and unplugging, cleaning, and finger operation.
[0055] Each category corresponds to a structured data format, which includes key content, keywords, and key parameters.
[0056] For mobile applications, key content includes the mobile target and the mobile subject; keywords include nouns; and key parameters include operational parameters.
[0057] For replacements, key information includes the target, the substitute, and the tools used; keywords include nouns and adjectives; and key parameters include operational parameters.
[0058] For inspections, key content includes inspection objectives, inspection purpose, and tools used; keywords include verbs and nouns; and key parameters include operational parameters.
[0059] For adjustments, key content includes the target object, the purpose of the adjustment, and the tools used; keywords include verbs and nouns; and key parameters include operational parameters.
[0060] For twisting, key information includes the target object, its location, and the tools used; keywords include nouns and adjectives; and key parameters include operating parameters.
[0061] For disassembly, key information includes the object to be disassembled, the disassembly method, and the tools used. Keywords include verbs and nouns, and key parameters include operating parameters.
[0062] For plugging and unplugging, key information includes the objects to be plugged and unplugged, precautions, and tools used. Keywords include verbs and nouns, and key parameters include operating parameters.
[0063] For cleaning, key content includes the object to be cleaned, the cleaning method, and the cleaning supplies. Keywords include verbs and nouns, and key parameters include operational parameters.
[0064] For finger operation, key content includes the object being operated on and the operation method, keywords include nouns, and key parameters include operation parameters.
[0065] Step 2 clarified the classification of maintenance guidance operations and listed keywords, providing guidelines for the maintenance guidance text processing in Step 3. The structured data format pointed the way for the extraction of key information.
[0066] Step 3: Extract maintenance guidance information text based on natural language processing.
[0067] Based on the existing maintenance activity text data, preprocessing is first performed to remove irrelevant information. Then, using the text classification rule system constructed in step 2 as a reference, part-of-speech analysis, dictionary construction, and keyword extraction are performed on the maintenance activity text. The results are used as the data input for the next step of the maintenance guidance information display method.
[0068] At this point, the following steps are designed in this embodiment.
[0069] (i) Based on user input instructions, select the target maintenance activity text required for the current maintenance activity from the maintenance activity text database. The maintenance activity text database includes multiple maintenance activity texts.
[0070] This embodiment performs data cleaning as a preprocessing step on the maintenance activity text data to obtain a maintenance activity text database.
[0071] Existing maintenance activity text data includes maintenance work cards, maintenance manuals, and other maintenance guidelines, all of which are unstructured free text. In practical applications of AR technology, the required input data consists of tagged single-action execution commands. This necessitates removing all information from maintenance work cards, manuals, and other guidelines except for specific maintenance action instructions. This embodiment creates a Python-based regular expression text filtering algorithm for data cleaning of the original maintenance activity text data. The specific implementation process is shown below.
[0072] (1) Clarify the format of specific maintenance action instructions in the maintenance activity text data.
[0073] In the textual data of maintenance activities such as maintenance work cards, maintenance manuals and other maintenance guidelines, each chapter contains a large amount of descriptive and useless text. The specific maintenance action instructions required are often scattered in the maintenance activities of various chapters, and are presented in the format of "label + text".
[0074] (2) Construct a regular expression text filtering algorithm.
[0075] First, define a regular expression `patterns` to match and retain lines that begin with a target label, as shown in Table 3.
[0076] Table 3. Reserved Contents for Regular Expressions
[0077] In Table 3, words can be numbers, letters, or words.
[0078] Then, define a cleaning function that iterates through all files in the specified folder (which contain text data related to maintenance activities), opens the files sequentially, reads all lines of the files into a list `lines`, filters the list `lines` using list comprehensions and regular expressions, and keeps only the lines that match the regular expression `patterns`.
[0079] Finally, the cleaned lines are written to a new file in the specified path.
[0080] The above regular expression text filtering algorithm can automatically traverse all files in a specified folder, perform data cleaning, and save the results, avoiding the complex operation of manually processing useless data.
[0081] In this embodiment, the method for establishing the maintenance activity text database includes: cleaning the maintenance activity text data to obtain the maintenance activity text database.
[0082] The process of cleaning the text data of maintenance activities to obtain a maintenance activity text database includes the following steps.
[0083] (1) Select the first line of the maintenance activity text data as the target line.
[0084] (2) Determine if the target line matches the regular expression. If it does, mark the target line as a retained line; otherwise, mark it as a filtered line. The regular expression is to retain lines that begin with a target label. The target label includes words within parentheses (e.g., (1), (a)) and words within angle brackets (e.g., ...). <1> , (and words within right single parentheses (e.g., 1, a)).
[0085] (3) Determine whether the target line is the last line of the maintenance activity text data. If not, take the next line of the target line as the target line of the next iteration and return to the step of "determine whether the target line matches the regular expression". If yes, remove all filtered lines from the maintenance activity text data to obtain the maintenance activity text database. The maintenance activity text database includes all retained lines, and each retained line is a maintenance activity text.
[0086] (ii) Calculate the similarity between the target maintenance activity text and each sample maintenance activity text in the benchmark database. Determine the initial similarity for each category based on all similarities. The sample maintenance activity text has a classification label, which is the category of the sample maintenance activity text.
[0087] This embodiment first constructs a benchmark database. Since this embodiment aims to achieve maintenance activity text classification through similarity calculation, it is necessary to first construct a benchmark database for maintenance activity text classification. After selecting maintenance activity text examples and manually processing and supplementing them, more than 1,000 sample maintenance activity texts with classification labels are formed, which constitute the benchmark database. Part of the contents of the benchmark database are shown in Table 4.
[0088] Table 4 contains part of the benchmark database.
[0089] This embodiment redesigns part-of-speech tagging (POS) extraction based on maintenance activity text. Maintenance activity text typically contains descriptions of equipment malfunctions, maintenance processes, and replaced parts, resulting in large volumes of rich information. However, manually analyzing and labeling this maintenance activity text is inefficient. Therefore, automated word extraction and POS tagging techniques can significantly improve the efficiency of maintenance activity text classification and labeling.
[0090] Based on the standardized maintenance activity text obtained after data cleaning and processing, an effective solution for automatically extracting keywords from the maintenance activity text is achieved through natural language processing technology, mainly word segmentation and part-of-speech tagging algorithms. This prepares for subsequent classification and annotation of the maintenance activity text. The specific implementation process is as follows.
[0091] (1) Word segmentation and part-of-speech tagging.
[0092] The Natural Language Toolkit (nltk) library is used to implement word segmentation and part-of-speech tagging. The key command is: words1 = pos_tag(word_tokenize(text1)), where text1 is the maintenance activity text, word_tokenize is used to segment the text string into a list of words, which is implemented by nltk using a pre-trained Punkt tokenizer to identify word boundaries, pos_tag is used to perform part-of-speech tagging on the segmented word list, which is implemented by nltk using a pre-trained Penn Treebank part-of-speech tagger to assign a part-of-speech tag to each word, and words1 is the returned result.
[0093] (2) Return the results.
[0094] After the word segmentation and part-of-speech tagging instructions are executed, word1 is returned, which is a list of tuples. Each tuple in the list contains a word and its corresponding part-of-speech tag. As shown in Table 5, it includes some common part-of-speech tags and their definitions.
[0095] Table 5. Definitions of Part-of-Speech Tags
[0096] Finally, this embodiment designs text reasoning and classification based on keyword recognition and weighted similarity. Based on the obtained word segmentation and part-of-speech tagging data, and using the text classification rule system constructed in step 2 as the standard, all maintenance activity texts are integrated and summarized, similarity weighted analysis is performed, and keyword recognition classification is added to obtain highly accurate classification results. This provides directly usable data for AR technology and realizes automated batch processing. The specific implementation process is shown below.
[0097] (1) Construct a dictionary of the importance of maintenance texts.
[0098] Each maintenance activity text contains words of various parts of speech. Based on the frequency statistics of each part of speech in the maintenance workbook writing guidelines and benchmark database, it can be seen that the content of verbs and nouns often plays a decisive role in the classification results of maintenance activity texts. That is, the classification result of most maintenance activity texts can be accurately determined simply by the verbs and nouns that appear in them. Therefore, before determining the classification label through text similarity comparison, it is necessary to consider the influence of different parts of speech. Thus, a maintenance text importance dictionary is constructed, and each part of speech is weighted and assigned a value to reduce the impact of local non-critical content similarity on the classification result and improve the accuracy of the algorithm.
[0099] Maximum likelihood estimation was performed based on over 1000 sample maintenance activity texts from the benchmark database. The contributions of verbs and nouns to the classification information entropy were found to be 49.87% (approximately 0.5) and 31.04% (approximately 0.3), respectively. Therefore, we assume... Let be a set of part-of-speech tags, and define a discrete measure as shown in equation (1).
[0100] (1).
[0101] In equation (1), For part-of-speech tags The weights; It is a set of verbs; It is a set of nouns.
[0102] Discrete measures satisfy the following condition: Normalization: Monotonicity: If , The first part-of-speech tag The weight, For the second part-of-speech tag The weight, For third part-of-speech tags Weights; domain invariance: It is independent of the domain dictionary and can be directly transferred in zero-shot scenarios.
[0103] At this point, in the text importance dictionary, the weight of verbs is 0.5, the weight of nouns is 0.3, and the weight of all other parts of speech is 0.2, thus realizing the allocation of the influence importance of different parts of speech when comparing similarity.
[0104] (2) Comparison of weight assignment based on similarity.
[0105] Based on the maintenance text importance dictionary, the similarity between the maintenance activity text to be classified and the sample maintenance activity text in the benchmark database is calculated. The initial similarity of the maintenance activity text to be classified under each classification label is initially determined, and the initial similarity of each category is obtained.
[0106] Define part-of-speech weighted Jaccard similarity for any two maintenance activity texts. , First, construct a weighted multiset, as shown in equation (2).
[0107] (2).
[0108] In equation (2), For maintenance activity text Weighted multiset; For words in the maintenance activity text; For words in the maintenance activity text The weights are based on words in the maintenance activity text. Part-of-speech tags Sure; For maintenance activity text Weighted multiset.
[0109] Clearly, the weighted multiset contains all the words in the maintenance activity text and the weight of each word.
[0110] Then, we define the weighted intersection and the weighted union, and calculate the sum of the intersection weights and the sum of the union weights, as shown in equation (3) below.
[0111] (3).
[0112] In equation (3), The weighted intersection includes two maintenance activity texts. , Words that appear together in the text; The sum of intersection weights; For words In the maintenance activity text The weight of the word In the maintenance activity text In the middle, the weight is based on the word. The part-of-speech tag is determined as follows: if it is a verb, the weight is 0.5; if it is a noun, the weight is 0.3; otherwise, the weight is 0.2. Text not in maintenance activities In the middle, the weight is 0; For words In the maintenance activity text The weight of the word In the maintenance activity text In the middle, the weight is based on the word. The part-of-speech tag is determined as follows: if it is a verb, the weight is 0.5; if it is a noun, the weight is 0.3; otherwise, the weight is 0.2. Text not in maintenance activities In the middle, the weight is 0; For the weighted union, it includes two maintenance activity texts. , All words that appear in the text; The sum of the union weights.
[0113] Finally, the part-of-speech weighted Jaccard similarity is calculated as shown in equation (4).
[0114] (4).
[0115] In equation (4), For two maintenance activity texts , The part-of-speech weighted Jaccard similarity between them.
[0116] Based on the above process, a set of similarities can be obtained, including the similarity between the maintenance activity text to be classified and each sample maintenance activity text in the benchmark database (i.e., part-of-speech weighted Jaccard similarity). Sample maintenance activity texts belonging to the same category are identified, and the maximum similarity of all sample maintenance activity texts belonging to the same category is taken as the initial similarity of that category. Example feedback is shown in Table 6.
[0117] Table 6. Similarity Results Feedback
[0118] (3) Correction based on keyword recognition.
[0119] Considering that some keywords have a decisive effect on the classification of maintenance activity texts, a correction step based on keyword recognition is added. Specifically, based on the keywords corresponding to each category in the last column of Table 1 given in step 2, the initial similarity of each category is corrected. For each category, if the maintenance activity text to be classified contains the keyword corresponding to that category, the initial similarity of that category is increased by 0.2 for each keyword that appears.
[0120] set up For category The set of corresponding keywords is used to define an indicator function, as shown in equation (5) below.
[0121] (5).
[0122] In equation (5), As a binary variable, when the keyword Satisfying the conditions on the right side of the equals sign (i.e., keywords) Belongs to And also belong to )hour, The value is 1, otherwise, The value is 0; The text is for maintenance activities to be categorized.
[0123] The formula for calculating the corrected similarity is shown in equation (6).
[0124] (6).
[0125] In equation (6), For category The corrected similarity; For category The initial similarity; This is a preset ratio.
[0126] After determining the corrected similarity of each category using the above method, the maintenance activity text to be classified is assigned to the category with the highest corrected similarity, thus completing the automatic classification of the maintenance activity text to be classified.
[0127] In this embodiment, the similarity between the target maintenance activity text and each sample maintenance activity text in the benchmark database is calculated, specifically including the following steps.
[0128] (1) Segment the target maintenance activity text to obtain several first words, and perform part-of-speech tagging on each first word to obtain the first part-of-speech tag of each first word.
[0129] (2) For each sample maintenance activity text in the benchmark database, the sample maintenance activity text is segmented to obtain several second words, and each second word is tagged with part of speech to obtain the second part of speech tag of each second word.
[0130] (3) For each sample maintenance activity text in the benchmark database, calculate the similarity between the target maintenance activity text and the sample maintenance activity text based on the first word and first part-of-speech tag in the target maintenance activity text and the second word and second part-of-speech tag in the sample maintenance activity text.
[0131] The similarity between the target maintenance activity text and the sample maintenance activity text is calculated based on the first word and first part-of-speech tag in the target maintenance activity text and the second word and second part-of-speech tag in the sample maintenance activity text. This process includes the following steps.
[0132] (1) Take the intersection of each first word in the target maintenance activity text and each second word in the sample maintenance activity text to obtain a weighted intersection. For each first target word in the weighted intersection, determine the first weight of the first target word based on the first part-of-speech tag of the first target word, and determine the second weight of the first target word based on the second part-of-speech tag of the first target word. Take the minimum value of the first weight and the second weight of the first target word as the intersection weight of the first target word.
[0133] (2) Take the union of each first word in the target maintenance activity text and each second word in the sample maintenance activity text to obtain a weighted union. For each second target word in the weighted union, if the second target word is a word in the target maintenance activity text, determine the first weight of the second target word based on the first part-of-speech tag of the second target word; otherwise, determine the first weight of the second target word as 0. If the second target word is a word in the sample maintenance activity text, determine the second weight of the second target word based on the second part-of-speech tag of the second target word; otherwise, determine the second weight of the second target word as 0. The maximum value of the first weight of the second target word and the second weight of the second target word is taken as the union weight of the second target word.
[0134] (3) Calculate the sum of the intersection weights of all first target words to obtain the sum of intersection weights, and calculate the sum of the union weights of all second target words to obtain the sum of union weights.
[0135] (4) Calculate the ratio of the intersection weight sum to the union weight sum to obtain the similarity between the target maintenance activity text and the sample maintenance activity text.
[0136] In this embodiment, the initial similarity for each category is determined based on all similarities. Specifically, for each category, the sample maintenance activity texts labeled as the category are grouped into groups corresponding to the category, and the maximum similarity of the sample maintenance activity texts in the group is selected as the initial similarity for the category.
[0137] The similarity between the sample maintenance activity text and the target maintenance activity text is the same as the similarity between the sample maintenance activity text and the target maintenance activity text.
[0138] (iii) For each category, determine whether the target maintenance activity text includes the keywords corresponding to the category, obtain the judgment result, and correct the initial similarity corresponding to the category based on the judgment result to obtain the corrected similarity corresponding to the category.
[0139] In this embodiment, it is determined whether the target maintenance activity text includes the keywords corresponding to the category, and the determination result is obtained. Based on the determination result, the initial similarity corresponding to the category is corrected to obtain the corrected similarity corresponding to the category. Specifically, the following steps are included.
[0140] (1) Determine whether the target maintenance activity text includes the keywords corresponding to the category, and obtain the judgment result.
[0141] (2) If the judgment result is yes, then determine the number of keywords corresponding to the categories included in the target maintenance activity text, calculate the product of the preset ratio and the number, obtain the similarity increase value corresponding to the category, calculate the sum of the initial similarity corresponding to the category and the similarity increase value corresponding to the category, and obtain the corrected similarity corresponding to the category.
[0142] The preset ratio can be 0.2.
[0143] (3) If the judgment result is negative, the initial similarity corresponding to the category is used as the corrected similarity corresponding to the category.
[0144] (iv) Select the category with the highest similarity after correction as the category of the target maintenance activity text.
[0145] (v) Based on the category of the target maintenance activity text, extract information from the target maintenance activity text to obtain maintenance guidance information.
[0146] In this embodiment, based on the category of the target maintenance activity text, information is extracted from the target maintenance activity text to obtain maintenance guidance information. Specifically, this includes: extracting information from the target maintenance activity text according to the structured data format corresponding to the category of the target maintenance activity text to obtain maintenance guidance information. The maintenance guidance information includes key content, keywords, and key parameters in the target maintenance activity text.
[0147] It should be noted that the key content, keywords, and key parameters included in the maintenance guidance information are all specific words. For example, if the keyword is a noun, then the keywords included in the maintenance guidance information are words with the part of speech as nouns.
[0148] Step 4: Design a method for displaying maintenance guidance information.
[0149] This embodiment proposes a method for displaying maintenance guidance information on a human-computer interaction interface, according to... Figure 3 The area division shown is filled in with the maintenance guidance information extracted in step 3. The scenic area is regarded as the actual maintenance object. The task progress area mainly includes three types of information: "current task category", "task progress" and "next task category". The task information area mainly includes the first three types of information specified in the structured data format of maintenance guidance information, namely "key content 1", "key content 2" and "key content 3". The key information area mainly includes the last two types of information specified in the structured data format of maintenance guidance information, namely "keywords" and "operation parameters".
[0150] In this embodiment, the maintenance guidance information is displayed on the human-computer interaction interface of the augmented reality device. The human-computer interaction interface includes a task progress area, a task information area, a key information area, and a view area. The task progress area displays the current task category, task progress, and next task category. The current task category is the category of the target maintenance activity statement currently being executed. The task progress shows the order of the target maintenance activity statements currently being executed and the total number of target maintenance activity statements to be executed in the current maintenance activity. For example, if a total of 3 target maintenance activity statements need to be executed, and the first target maintenance activity statement is currently being executed, then the order is 1, and the task progress can be displayed as 1 / 3. The next task category is the category of the next target maintenance activity statement to be executed. The task information area displays the key content in the maintenance guidance information of the target maintenance activity statement currently being executed. The key information area displays the keywords and key parameters in the maintenance guidance information of the target maintenance activity statement currently being executed. The view area displays the maintenance object actually seen by the maintenance personnel.
[0151] Optionally, the task progress area is located at the top of the human-computer interaction interface, the task information area is located on the left side of the human-computer interaction interface, the key information area is located on the right side of the human-computer interaction interface, and the view area is located in the middle of the human-computer interaction interface.
[0152] Taking the maintenance of common diesel engine damage, such as oil leakage from the hollow bolt of the oil distribution seat, as an example, the hollow bolt needs to be replaced. The main operating steps include: ① Cleaning the oil-leaking bolt and surrounding area; ② Unscrewing the M5 hollow bolt from the oil distribution seat; ③ Installing a new M5 hollow bolt.
[0153] Based on the part-of-speech tagging and classification results, the information extraction of the maintenance activity text is completed according to the structured data format of maintenance guidance information proposed in step 2. The resulting maintenance guidance information is shown in Table 7.
[0154] Table 7. Example of Maintenance Guidance Information
[0155] In this embodiment, the augmented reality device can be AR glasses. By simulating actual application scenarios and inputting the extracted maintenance guidance information into the AR glasses, all key information supporting the current maintenance operation is displayed on the human-computer interaction interface of the AR glasses, demonstrating the practicality and efficiency of the maintenance guidance information extraction method.
[0156] This embodiment designs a maintenance process classification and information extraction algorithm, which realizes data cleaning, extraction, automated annotation and classification, and key information extraction of maintenance activity text, providing high-accuracy structured data support for AR technology.
[0157] In step 1, by clarifying the requirements for the text types of maintenance activities according to the standards, the classification rules for all maintenance activity texts were formulated, and nine major categories and their identification keywords were identified. This standardized the retrieval and classification mechanism for maintenance activity texts, eliminated ambiguities in the classification of maintenance activity texts, and improved the efficiency of information extraction.
[0158] In step 2, a regular expression text filtering algorithm based on Python was developed using natural language processing technology. This algorithm automatically preprocesses the existing irregular and unstructured maintenance activity text data, removes irrelevant information from the maintenance activity text data, and improves the efficiency and accuracy of maintenance activity text data cleaning. Compared with manual screening and impurity removal, it significantly improves the level of automation.
[0159] In step 3, the text of the maintenance activity to be classified was effectively compared with the sample text of the maintenance activity in the benchmark database by assigning similarity weights based on part-of-speech analysis and adding keyword recognition classification. This reduced the uncertainty of manual annotation, eliminated subjective ambiguity in the classification and identification of the maintenance process, and obtained classification results with high accuracy. This provides data that can be directly used for AR technology and realizes the extraction of maintenance guidance information.
[0160] In step 4, all key information supporting the current maintenance operation is displayed on the human-computer interaction interface through information partitioning. A method for displaying maintenance guidance information on the human-computer interaction interface is proposed, proving the practicality and efficiency of the maintenance guidance information extraction method and improving the readability of AR-assisted maintenance information.
[0161] This application also provides an application scenario in which the above-described augmented reality-oriented maintenance guidance information extraction and display method is applied. Specifically, the augmented reality-oriented maintenance guidance information extraction and display method provided in this embodiment can be applied in equipment maintenance scenarios. Equipment maintenance scenarios include an extraction and display stage and a maintenance stage. The extraction and display stage is used to extract and display maintenance guidance information, and the maintenance stage is used to enable maintenance personnel to perform maintenance on the equipment based on the displayed maintenance guidance information. The augmented reality-oriented maintenance guidance information extraction and display method provided in this embodiment belongs to the extraction and display stage.
[0162] Example 2.
[0163] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a method for extracting and displaying maintenance guidance information based on augmented reality.
[0164] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0165] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the augmented reality-oriented maintenance guidance information extraction and display method of Embodiment 1.
[0166] Example 3.
[0167] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the augmented reality-oriented maintenance guidance information extraction and display method of Embodiment 1.
[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of the relevant data are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.
[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0170] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for extracting and displaying maintenance guidance information based on augmented reality, characterized in that, The method for extracting and displaying maintenance guidance information based on augmented reality includes: Based on user input commands, the target maintenance activity text required for the current maintenance activity is selected from the maintenance activity text database; the maintenance activity text database includes multiple maintenance activity texts. The similarity between the target maintenance activity text and each sample maintenance activity text in the benchmark database is calculated, and an initial similarity is determined for each category based on all the similarities; the sample maintenance activity text has a classification label, and the classification label is the category of the sample maintenance activity text; For each category, determine whether the target maintenance activity text includes the keyword corresponding to the category, obtain the determination result, and correct the initial similarity corresponding to the category based on the determination result to obtain the corrected similarity corresponding to the category. The category with the highest similarity after correction is selected as the category of the target maintenance activity text; Based on the category of the target maintenance activity text, information is extracted from the target maintenance activity text to obtain maintenance guidance information; The maintenance guidance information is displayed on the human-computer interaction interface of the augmented reality device.
2. The method for extracting and displaying maintenance guidance information based on augmented reality according to claim 1, characterized in that, The method for establishing the maintenance activity text database includes: cleaning the maintenance activity text data to obtain the maintenance activity text database; This includes data cleaning of the maintenance activity text data to obtain a maintenance activity text database, specifically including: Select the first line of the maintenance activity text data as the target line; Determine whether the target line matches the regular expression. If it does, mark the target line as a retained line; otherwise, mark the target line as a filtered line. The regular expression is to retain lines that begin with a target label, which includes words enclosed in parentheses, words enclosed in angle brackets, and words enclosed in right single parentheses. Determine whether the target line is the last line of the maintenance activity text data. If not, take the next line of the target line as the target line of the next iteration and return to the step of "determine whether the target line matches the regular expression". If yes, remove all the filtered lines from the maintenance activity text data to obtain the maintenance activity text database. The maintenance activity text database includes all the retained lines, and each retained line is a maintenance activity text.
3. The method for extracting and displaying maintenance guidance information based on augmented reality according to claim 1, characterized in that, Calculate the similarity between the target maintenance activity text and each sample maintenance activity text in the benchmark database, specifically including: The target maintenance activity text is segmented into several first words, and each first word is tagged with part-of-speech tags to obtain the first part-of-speech tag for each first word. For each sample maintenance activity text in the benchmark database, the sample maintenance activity text is segmented into several second words, and each second word is tagged with part-of-speech tags to obtain a second part-of-speech tag for each second word; For each sample maintenance activity text in the benchmark database, the similarity between the target maintenance activity text and the sample maintenance activity text is calculated based on the first word and first part-of-speech tag in the target maintenance activity text and the second word and second part-of-speech tag in the sample maintenance activity text.
4. The method for extracting and displaying maintenance guidance information based on augmented reality according to claim 3, characterized in that, Based on the first word and first part-of-speech tag in the target maintenance activity text and the second word and second part-of-speech tag in the sample maintenance activity text, the similarity between the target maintenance activity text and the sample maintenance activity text is calculated, specifically including: For each first word in the target maintenance activity text and each second word in the sample maintenance activity text, take the intersection to obtain a weighted intersection. For each first target word in the weighted intersection, determine the first weight of the first target word based on the first part-of-speech tag of the first target word, determine the second weight of the first target word based on the second part-of-speech tag of the first target word, and take the minimum value of the first weight and the second weight of the first target word as the intersection weight of the first target word. For each first word in the target maintenance activity text and each second word in the sample maintenance activity text, take the union of the two sets to obtain a weighted union. For each second target word in the weighted union, if the second target word is a word in the target maintenance activity text, then determine the first weight of the second target word based on the first part-of-speech tag of the second target word; otherwise, determine the first weight of the second target word as 0. If the second target word is a word in the sample maintenance activity text, then determine the second weight of the second target word based on the second part-of-speech tag of the second target word; otherwise, determine the second weight of the second target word as 0. The maximum value of the first weight and the second weight of the second target word is taken as the union weight of the second target word. Calculate the sum of the intersection weights of all the first target words to obtain the intersection weight sum; calculate the sum of the union weights of all the second target words to obtain the union weight sum. The similarity between the target maintenance activity text and the sample maintenance activity text is obtained by calculating the ratio of the intersection weight sum to the union weight sum.
5. The method for extracting and displaying maintenance guidance information based on augmented reality according to claim 1, characterized in that, The initial similarity for each category is determined based on all the aforementioned similarities, specifically including: For each category, the sample maintenance activity texts labeled with the category are grouped into groups corresponding to the category, and the maximum similarity of the sample maintenance activity texts in the group is selected as the initial similarity corresponding to the category.
6. The method for extracting and displaying maintenance guidance information based on augmented reality according to claim 1, characterized in that, Determine whether the target maintenance activity text contains keywords corresponding to the category, obtain a determination result, and correct the initial similarity corresponding to the category based on the determination result to obtain the corrected similarity corresponding to the category, specifically including: Determine whether the target maintenance activity text contains keywords corresponding to the category, and obtain the determination result; If the judgment result is yes, then determine the number of keywords corresponding to the category included in the target maintenance activity text, calculate the product of the preset ratio and the number to obtain the similarity increase value corresponding to the category, calculate the sum of the initial similarity corresponding to the category and the similarity increase value corresponding to the category to obtain the corrected similarity corresponding to the category; If the judgment result is negative, then the initial similarity corresponding to the category is taken as the corrected similarity corresponding to the category.
7. The method for extracting and displaying maintenance guidance information based on augmented reality according to claim 1, characterized in that, Each category corresponds to a structured data format, which includes key content, keywords, and key parameters. Based on the category of the target maintenance activity text, information is extracted from the target maintenance activity text to obtain maintenance guidance information. Specifically, this includes: extracting information from the target maintenance activity text according to the structured data format corresponding to the category of the target maintenance activity text to obtain maintenance guidance information; the maintenance guidance information includes the key content, keywords, and key parameters in the target maintenance activity text.
8. The method for extracting and displaying maintenance guidance information based on augmented reality according to claim 7, characterized in that, The categories include moving, replacing, inspecting, adjusting, screwing, disassembling, plugging and unplugging, cleaning, and manual operation; For the context of movement, the key content includes the movement target and the movement subject, the keywords include nouns, and the key parameters include operational parameters; Regarding replacement, the key content includes the replacement target, the substitute, and the tools used; the keywords include nouns and adjectives; and the key parameters include operational parameters. For the inspection, the key content includes the inspection target, inspection purpose and tools used, the keywords include verbs and nouns, and the key parameters include operational parameters; For adjustments, the key content includes the target object, the purpose of the adjustment, and the tools used; the keywords include verbs and nouns; and the key parameters include operational parameters. For twisting, the key content includes the target object, its location, and the tools used; the keywords include nouns and adjectives; and the key parameters include operating parameters. For disassembly, the key content includes the object to be disassembled, the disassembly method, and the tools used; the keywords include verbs and nouns; and the key parameters include operating parameters. Regarding plugging and unplugging, the key content includes the objects to be plugged and unplugged, precautions, and tools used; the keywords include verbs and nouns; and the key parameters include operating parameters. For cleaning, the key content includes the object to be cleaned, the cleaning method, and the cleaning supplies; the keywords include verbs and nouns; and the key parameters include operational parameters. For finger operation, the key content includes the operation object and operation method, the keywords include nouns, and the key parameters include operation parameters.
9. The method for extracting and displaying maintenance guidance information based on augmented reality according to claim 7, characterized in that, The human-computer interaction interface includes a task progress area, a task information area, a key information area, and a viewing area; The task progress area is used to display the current task category, task progress, and next task category. The current task category is the category of the target maintenance activity statement that is currently being executed. The task progress is the order of the target maintenance activity statements that are currently being executed and the total number of target maintenance activity statements to be executed in the current maintenance activity. The next task category is the category of the next target maintenance activity statement that is about to be executed. The task information area is used to display key content in the maintenance guidance information of the target maintenance activity statement that is currently being executed; The key information area is used to display keywords and key parameters in the maintenance guidance information of the target maintenance activity statement that is currently being executed; The viewing area is used to display the actual object being repaired as seen by the maintenance personnel.
10. The method for extracting and displaying maintenance guidance information based on augmented reality according to claim 9, characterized in that, The task progress area is located at the top of the human-computer interaction interface, the task information area is located on the left side of the human-computer interaction interface, the key information area is located on the right side of the human-computer interaction interface, and the view area is located in the middle of the human-computer interaction interface.