Intelligent equipment maintenance method and system based on large model

Abstract

The invention relates to the technical field of text processing, and provides an intelligent device maintenance method and system based on a large model, and the method comprises the steps: collecting the multi-modal data of a to-be-maintained intelligent device, and building a large model; determining the entity expression similarity degree of two different small models in the large model, and determining two different small models corresponding to the same entity; determining a first confidence coefficient of each entity relationship, defining a to-be-aligned first small model and a to-be-aligned second small model and determining reference values corresponding to the first small model and the second small model, and aligning all the small models corresponding to the same entity as the to-be-aligned first small model according to the reference values of all the small models corresponding to the same entity as the to-be-aligned first small model; and determining the coincidence degree of the actual operation of the to-be-maintained intelligent equipment, judging whether the to-be-maintained intelligent equipment is abnormal or not according to the coincidence degree, and maintaining the abnormal to-be-maintained intelligent equipment. According to the invention, the equipment maintenance accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to a large model-based intelligent device maintenance method and system. Background Art

[0002] Multimodal data refers to data acquired from different channels and in different forms. This includes text data such as device logs generated during operation, image data such as images captured by surveillance cameras of key components within the device, audio data such as sound signals generated during operation, and numerical sensor data such as operating temperature measured by temperature sensors. Intelligent device maintenance based on multimodal data involves comprehensively utilizing this data from multiple modalities through data processing, analysis, and mining techniques to maintain intelligent devices. This provides a more comprehensive and accurate understanding of device status, improves the accuracy of fault diagnosis, and enables predictive maintenance.

[0003] Traditional methods for intelligent equipment maintenance based on multimodal data do not consider the differences in reference quality of reference data of different modalities. Therefore, reference data of different modalities have the same influence weight on the equipment status evaluation results, which can easily lead to inaccurate reference quality evaluation of the equipment status evaluation results and affect the accuracy of the equipment maintenance evaluation results. Summary of the Invention

[0004] The present invention provides a large-scale model-based intelligent equipment maintenance method and system to solve the problem that different modal data are used to evaluate the reference quality of equipment status, which leads to inaccurate equipment maintenance evaluation results. The technical solutions adopted are as follows: In a first aspect, an embodiment of the present invention provides a method for maintaining an intelligent device based on a large model, the method comprising the following steps: Collect multimodal data of the smart device to be maintained and build a large model. Each small model in the large model corresponds to an entity, and related small models are connected by edges; Each small model in the large model is used as a starting point, and the small models in the large model are traversed. The analysis sequence of the starting point is determined based on the traversal results. The degree of similarity in entity expression between two different small models in the large model is determined based on the position difference of the same small model in the analysis sequence of two different small models in the large model, as well as the number of the same small models. Based on the degree of similarity in entity expression, all two different small models in the large model that correspond to the same entity are determined; According to the number of different entity relationships in the large model contained in the text data, the first confidence of each entity relationship is determined respectively, and any two different small models corresponding to the same entity in the large model are respectively recorded as the first small model to be aligned and the second small model to be aligned, and the reference value of the second small model to be aligned corresponding to the first small model to be aligned is determined according to the result of traversing the large model after directly aligning the first small model to be aligned and the first confidence of the small model contained in the large model traversal result of the second small model to be aligned, and according to the reference value of all small models corresponding to the same entity as the first small model to be aligned, all small models corresponding to the same entity as the first small model to be aligned are aligned; Based on all types of multimodal data and the amount of multimodal data, the compliance level of the actual operation of the smart device to be maintained is determined, and based on the compliance level, whether the smart device to be maintained has an abnormality is determined, and maintenance is performed on the smart device to be maintained if an abnormality occurs.

[0005] Furthermore, the multimodal data includes text data, image data, audio data and numerical sensor data.

[0006] Furthermore, the specific method for determining the analysis sequence of the starting point is: Each small model in the large model is used as a starting point to traverse the small models in the large model to obtain the traversal sequence of the starting point. The subsequence consisting of the small models in the first third of the traversal sequence of the starting point is recorded as the analysis sequence of the starting point.

[0007] Furthermore, the method for obtaining the similarity of entity expressions of two different small models in the large model is: The two different small models in the large model are respectively recorded as the first small model and the second small model, the same small models in the analysis sequence of the first small model and the second small model are recorded as the overlapping small models of the first small model and the second small model, and the number of overlapping small models of the first small model and the second small model is recorded as the first number of the first small model and the second small model; The absolute value of the difference in the order of the overlapping small models of the first small model and the second small model in the analysis sequence of the first small model and the second small model is recorded as the order difference of the overlapping small models, and the cumulative sum of the order differences of all overlapping small models is recorded as the cumulative sum of the orders of the first small model and the second small model; In the analysis sequence of the first small model and the second small model, the number of small models between the first small model of the analysis sequence is less than The number of overlapping small models is recorded as the second number of the first small model and the second small model, where Represents the first parameter of the preset; The ratio of the product of the first quantity and the second quantity of the first small model and the second small model to the sequential cumulative sum is recorded as the entity expression similarity between the first small model and the second small model.

[0008] Furthermore, the specific method of determining all two different small models corresponding to the same entity in the large model according to the similarity of entity expression includes: The normalized value of the entity expression similarity is greater than or equal to The two small models are determined to be different small models corresponding to the same entity, where Indicates the second parameter of the preset.

[0009] Furthermore, the method for obtaining the first confidence of the entity relationship is: Record any entity relationship as a target entity relationship, and record the ratio of the number of target entity relationships contained in the text data to the number of all entity relationships contained in the text data as the first quantity ratio of the target entity relationship; The number of all indirect connection paths between two entities contained in the text data in the large model is recorded as the number of first paths of the target entity relationship; The product of the first quantity ratio of the target entity relationship and the first path quantity of the target entity relationship is recorded as the first confidence of the target entity relationship.

[0010] Furthermore, the specific method for determining the reference value of the second small model to be aligned corresponding to the first small model to be aligned is: Directly perform entity alignment on the entity relationship between the first small model to be aligned and the second small model to be aligned, and record the obtained small model as the aligned small model; traverse the small models in the large model using the aligned small model in the large model as the starting point to obtain a traversal sequence of the aligned small models; record the number of identical small models in the traversal sequence of the second small model to be aligned and the aligned small model as the third number of the second small model to be aligned; Record the cumulative sum of the first confidences of the small models included in the traversal sequence of the second small model to be aligned as the second cumulative sum of the second small model to be aligned; The product of the second accumulated sum of the second small model to be aligned and the third number is recorded as the reference value of the second small model to be aligned corresponding to the first small model to be aligned.

[0011] Furthermore, the method of aligning all small models corresponding to the same entity as the first small model to be aligned according to the reference values ​​of all small models corresponding to the same entity as the first small model to be aligned includes the following specific methods: The cumulative sum of the reference values ​​of all small models corresponding to the same entity of the first small model to be aligned is recorded as the total reference value of the first small model to be aligned, and the ratio of the reference value of the second small model to be aligned to the total reference value of the first small model to be aligned is recorded as the reference value ratio of the second small model to be aligned; According to the reference value ratio of the second small model to be aligned, the entities to which the processed data corresponding to all small models corresponding to the same entity as the first small model to be aligned belong are randomly extracted to achieve alignment of all small models corresponding to the same entity as the first small model to be aligned.

[0012] Furthermore, the method of determining the compliance level of the actual operation of the smart device to be maintained based on all types of multimodal data and the amount of the multimodal data, determining whether the smart device to be maintained is abnormal based on the compliance level, and maintaining the smart device to be maintained if abnormal is present, includes the following specific methods: Calculate the quality score of each type of multimodal data separately; The ratio of the quality score of the multimodal data of the same type in the multimodal data of the smart device to be maintained to the cumulative sum of the quality scores of all the multimodal data is recorded as the quality score ratio of the multimodal data of the same type; the product of the data volume of the multimodal data of the same type and the quality score ratio is recorded as the usage weight of the multimodal data of the same type; The usage weight of each type of multimodal data is used as a reference weight for the status evaluation result of the smart device to be maintained by each type of multimodal data, and the normalized value of the absolute value of the difference between the multimodal data of the smart device to be maintained and the standard value corresponding to the multimodal data is weighted and summed. The difference between the number 1 and the weighted summation result is used as the compliance degree of the actual operation of the smart device to be maintained; When the degree of compliance is less than or equal to a preset compliance threshold, it is determined that the operation of the smart device to be maintained is abnormal, and the similarity between the standard multimodal data corresponding to all problems of the smart device to be maintained in the database and the multimodal data of the smart device to be maintained is calculated. The problem corresponding to the standard multimodal data with the greatest similarity is recorded as the operation problem of the smart device to be maintained, and the standard maintenance method corresponding to the operation problem of the smart device to be maintained is fed back and uploaded; When the compliance level is greater than a preset compliance threshold, it is determined that the operation of the smart device to be maintained is not abnormal and maintenance is not performed.

[0013] In a second aspect, an embodiment of the present invention further provides a large-model-based intelligent device maintenance system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above-mentioned methods when executing the computer program.

[0014] The beneficial effects of the present invention are: This application first directly establishes a large model based on the multimodal data of the smart device to be maintained. Then, considering that the same entity may have multiple different names in the multimodal data of the smart device to be maintained, the same entity may have multiple different expressions in the large model, resulting in errors in the large model. In order to ensure the correctness of the large model, the overlap of related entity relationships of different entities is analyzed, and the similarity of entity expressions of two different small models in the large model is determined. When the degree of overlap of entity relationships is greater, the degree of overlap of entity relationships of entities corresponding to different small models is greater, and the degree of similarity of different entity expressions is greater. According to the similarity of entity expressions, all two different small models corresponding to the same entity in the large model are determined; entities with multiple expressions in the large model are very likely to be important reference data for a certain fault judgment condition of the equipment, and the confidence of reference data under different expressions in indicating faults is different. In order to ensure that the reference data corresponding to the entity is more meaningful for the judgment of equipment failure, the confidence of the relevant entity relationship of the small models corresponding to different expressions is obtained, and the first confidence of each entity relationship is determined respectively. The reference value of the second small model to be aligned corresponding to the first small model to be aligned is determined, and according to the reference value of all small models corresponding to the same entity as the first small model to be aligned, all small models corresponding to the same entity as the first small model to be aligned are aligned; finally, according to all types of multimodal data and the data volume of multimodal data, the degree of compliance of the actual operation of the intelligent device to be maintained is determined, and whether the intelligent device to be maintained has an abnormality is determined according to the degree of compliance, and the intelligent device to be maintained with an abnormality is maintained, so as to solve the problem that the reference quality evaluation of the equipment status evaluation results by different modal data is inaccurate, which makes the equipment maintenance evaluation results inaccurate, and improves the accuracy of equipment maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A flowchart of a large model-based intelligent device maintenance method provided by one embodiment of the present invention; Figure 2 This is a flowchart of an analysis sequence acquisition provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] See also Figure 1 , which shows a flow chart of a large model-based intelligent device maintenance method provided by an embodiment of the present invention, the method comprising the following steps: Step S001: Collect multimodal data of the smart device to be maintained and build a large model. Each small model in the large model corresponds to an entity, and related small models are connected by edges.

[0019] Multimodal data of devices that require equipment maintenance are extracted. The devices that require equipment maintenance are smart devices to be maintained.

[0020] Specifically, the multimodal data selected in this embodiment includes text data, image data, audio data, and numerical sensor data; the text data includes the device log information generated during device operation, and the device log information includes the device operation time, operation content, and error code; the image data is the image captured by the monitoring camera of the key internal components of the device that require equipment maintenance, and the key internal components of the device are determined by those skilled in the art; the audio data is the sound signal generated when the device that requires equipment maintenance is in operation; and the numerical sensor data is the device operating temperature measured by the temperature sensor. In actual application, as other implementation methods, the implementer can decide the data types and number of data types included in the multimodal data according to actual conditions, and this application does not impose any special restrictions.

[0021] Perform knowledge extraction, knowledge fusion, and knowledge storage on multimodal data to build a large model.

[0022] Among them, building a large model based on multimodal data is a well-known technology and will not be repeated here; knowledge extraction includes entity extraction, relationship extraction, and attribute extraction; knowledge fusion includes entity alignment, relationship fusion, and knowledge integration. Specifically, this embodiment performs data cleaning, modality alignment, and data labeling on each type of multimodal data, wherein data cleaning includes denoising and de-dimensionalization. This embodiment adopts the Z-Score standard normalization method for de-dimensionalization and median filtering for denoising; uses text-image pairing for modality alignment; realizes cross-modal entity reference through MEL multimodal entity linking technology, and links text data, image data, audio data, and numerical sensor data to knowledge base entities respectively; combines the pre-trained model BERT with the visual feature extractor ResNet to identify entities and entity types based on text and non-text modalities; and extracts data from text. Extract semantic relationships between entities from the data; establish visual similarity relationships between image entities and text entities; align the extracted entities to the knowledge base and use the MELBench dataset for link verification; treat image data, audio data, and numerical sensor data as independent entities, and associate them equally with text data to support intra-modal relationships; use entities as small models, connect related small models with edges, construct triples, and obtain the adjacency matrix based on the triples. The triples show the semantic relationships between entities, and the matrix can intuitively represent the association relationships between entities; the entities, relationships, and attributes extracted from multimodal data form a unified large model structure as a whole.

[0023] It is understandable that the large model is the main model.

[0024] At this point, the large model is obtained.

[0025] Step S002: Take each small model in the large model as a starting point, traverse the small models in the large model, determine the analysis sequence of the starting point based on the traversal results, determine the degree of similarity in entity expression between two different small models in the large model based on the position difference of the same small model in the analysis sequence of two different small models in the large model, and the number of the same small models, and determine all two different small models in the large model that correspond to the same entity based on the degree of similarity in entity expression.

[0026] The large model contains multiple small models, and the relationships between them can be abstracted as a network. Each small model corresponds to a modality of an entity, which can be text, image, video, etc. When building a large model based on existing knowledge, some small models may express entities with the same modality but not be assigned to the same small model, resulting in errors in the large model. To ensure the correctness of the large model, entity alignment is performed on the entities in the large model by checking the overlap of related entity relationships across different small models.

[0027] Take each small model in the large model as the starting point, and visit the small models in the large model layer by layer starting from the starting point through the BFS breadth-first algorithm. Use the queue to ensure that the vertices visited first are expanded first, realize the traversal of the small models in the large model, obtain the traversal sequence corresponding to the starting point, and record the subsequence composed of the small models in the first third of the traversal sequence corresponding to the starting point as the analysis sequence of the starting point.

[0028] At this point, the analysis sequence of each small model in the large model can be obtained. The analysis sequence acquisition flow chart is as follows: Figure 2 As shown; using the BFS breadth-first algorithm to traverse the small models in the large model and obtain the traversal sequence of the starting point is a well-known technology and will not be described in detail.

[0029] The degree of similarity of entity expression of two different small models in the large model is determined based on the position difference of the same small model in the analysis sequence of two different small models in the large model and the number of the same small models in the analysis sequence of two different small models in the large model.

[0030] The two different small models in the large model are respectively recorded as the first small model and the second small model, the small models that are the same in the analysis sequence of the first small model and the second small model are recorded as the overlapping small models of the first small model and the second small model, and the number of overlapping small models of the first small model and the second small model is recorded as the first number of the first small model and the second small model; the absolute value of the difference in the order of the overlapping small models of the first small model and the second small model in the analysis sequence of the first small model and the second small model is recorded as the order difference of the overlapping small models, and the cumulative sum of the order differences of all overlapping small models is recorded as the cumulative sum of the order of the first small model and the second small model; in the analysis sequence of the first small model and the second small model, the number of small models separated from the first small model of the analysis sequence is less than The number of overlapping small models is recorded as the second number of the first small model and the second small model; the ratio of the product of the first number and the second number of the first small model and the second small model to the cumulative sum of the order is recorded as the degree of similarity in entity expression between the first small model and the second small model.

[0031] It is understandable that Represents the first parameter, which is a preset parameter value. In this embodiment, the value of the first parameter is 10.

[0032] The similarity of entity expression between the first small model and the second small model is used to evaluate the degree of entity relationship overlap between the entities corresponding to the first small model and the second small model. When the degree of entity relationship overlap is greater, the degree of entity relationship overlap between the entities corresponding to the first small model and the second small model is greater. At this time, the similarity of entity expression between the first small model and the second small model is greater.

[0033] The normalized value of the entity expression similarity is greater than or equal to The first small model and the second small model are determined to be different small models corresponding to the same entity; the normalized value of the entity expression similarity is less than The first small model and the second small model are determined to be different small models that do not correspond to the same entity. The normalized value of the similarity of entity expression is greater than or equal to The entity relationship between the first small model and the second small model is merged to achieve entity alignment.

[0034] It should be noted that this embodiment uses the Z-Score standard normalization method to calculate the normalized value. In actual application, the implementer may use other existing methods such as the maximum and minimum normalization method, the sigmoid function, etc. to calculate the normalized value, which is not limited here. It is understandable that Represents the second parameter, which is a preset parameter value. In this embodiment, the value of the second parameter is 0.8.

[0035] The same method can be used to obtain all two different small models corresponding to the same entity in the large model.

[0036] At this point, all two different small models corresponding to the same entity in the large model are obtained.

[0037] Step S003: Determine the first confidence of each entity relationship according to the number of different entity relationships in the large model contained in the text data, and record any two different small models corresponding to the same entity in the large model as the first small model to be aligned and the second small model to be aligned respectively; determine the reference value of the second small model to be aligned corresponding to the first small model according to the result of traversing the large model after directly aligning the first small model and the second small model to be aligned, and the first confidence of the small model contained in the large model traversal result of the second small model to be aligned; align all small models corresponding to the same entity as the first small model to be aligned according to the reference value of all small models corresponding to the same entity as the first small model to be aligned.

[0038] Entities in the large model with multiple expressions are very likely to be important reference data for a certain fault judgment condition of the equipment. The confidence of the reference data in different expressions in indicating the fault varies. In order to ensure that the reference data corresponding to the entity is more meaningful for judging the equipment fault, the confidence of the relevant entity relationship in the text data referenced when obtaining the small models corresponding to different expressions is obtained. Specifically, for entity modalities expressed by multiple small models, the reference value of different small models is evaluated through the performance of each entity relationship in the text data referenced when the small models are built. It can be understood that an entity relationship is a triple consisting of two entities connected by an edge in the large model. When an entity relationship appears in the text data, it is determined that the text data contains an entity relationship.

[0039] According to the number of different entity relationships in the large model contained in the text data, a first confidence level of each entity relationship is determined respectively.

[0040] Any entity relationship is recorded as the target entity relationship, and the ratio of the number of target entity relationships contained in the text data to the number of all entity relationships contained in the text data is counted and recorded as the first quantity ratio of the target entity relationship; the DFS depth-first search algorithm and backtracking mechanism are used to obtain all indirect connection paths between the two entities contained in the target entity relationship, and the number of all indirect connection paths contained in the text data in the large model is recorded as the first path number of the target entity relationship; the product of the first quantity ratio of the target entity relationship and the first path number of the target entity relationship is recorded as the first confidence of the target entity relationship.

[0041] The use of a DFS depth-first search algorithm and a backtracking mechanism to obtain all indirect connection paths between two entities contained in an entity relationship is a well-known technique and will not be described in detail.

[0042] The same method can be used to obtain the first confidence of all entity relationships in the large model.

[0043] The first confidence level of an entity relationship evaluates the reference value of each small model based on the different entity relationships. Furthermore, the reference value of each small model must be evaluated by considering the relationships between the different small models corresponding to the entity and other small models before and after entity alignment. The more similar the entity relationships between the different small models corresponding to the entity before and after entity alignment are, and the greater the first confidence level of the entity relationships between the different small models corresponding to the entity, the greater the reference value of the small model corresponding to the entity in diagnosing equipment failures.

[0044] Any two different small models corresponding to the same entity in the large model are respectively recorded as the first small model to be aligned and the second small model to be aligned.

[0045] Directly perform entity alignment on the entity relationship between the first small model to be aligned and the second small model to be aligned, and record the small model obtained after entity alignment of the first small model to be aligned and the second small model to be aligned as the aligned small model; take the aligned small model in the large model as the starting point, and use the BFS breadth-first algorithm to visit the small models in the large model layer by layer from the starting point, use a queue to ensure that the first visited vertices are expanded first, realize traversal of the small models in the large model, and obtain the traversal sequence corresponding to the aligned small model; record the number of the same small models in the traversal sequence of the second small model to be aligned and the aligned small model as the third number of the second small model to be aligned; record the cumulative sum of the first confidence levels of the small models contained in the traversal sequence of the second small model to be aligned as the second cumulative sum of the second small model to be aligned; record the product of the second cumulative sum of the second small model to be aligned and the third number as the reference value of the second small model to be aligned corresponding to the first small model to be aligned.

[0046] The same method can be used to obtain the reference values ​​of all small models in the large model that correspond to the same entity as the first small model to be aligned.

[0047] All small models corresponding to the same entity as the first small model to be aligned are aligned according to the reference values ​​of all small models corresponding to the same entity as the first small model to be aligned.

[0048] The cumulative sum of the reference values ​​of all small models corresponding to the same entity of the first small model to be aligned is recorded as the total reference value of the first small model to be aligned, and the ratio of the reference value of the second small model to be aligned to the total reference value of the first small model to be aligned is recorded as the reference value ratio of the second small model to be aligned. According to the reference value ratio of the second small model to be aligned, the entities to which the processed data corresponding to all small models corresponding to the same entity of the first small model to be aligned belong are randomly extracted to achieve alignment of all small models corresponding to the same entity of the first small model to be aligned.

[0049] The same method can be used to align all small models corresponding to the same entity.

[0050] At this point, the alignment of all small models corresponding to the same entity is achieved.

[0051] Step S004: Determine the compliance level of the actual operation of the smart device to be maintained based on all types of multimodal data and the amount of multimodal data, determine whether the smart device to be maintained is abnormal based on the compliance level, and maintain the smart device to be maintained if it is abnormal.

[0052] When determining the problem of the smart device to be maintained based on the big model, there may be more than one type of multimodal data that can be referenced. Therefore, the specific type of problem of the smart device to be maintained should be determined together based on different types of multimodal data.

[0053] Calculate the quality score of each type of multimodal data separately.

[0054] Specifically, this embodiment calculates the normalized value of the information entropy of the text data, and records the first score of the text data whose normalized value of the information entropy is greater than or equal to 0.8 as 100 points, the first score of the text data whose normalized value of the information entropy is greater than or equal to 0.6 and less than 0.8 as 80 points, the first score of the text data whose normalized value of the information entropy is greater than or equal to 0.4 and less than 0.6 as 60 points, the first score of the text data whose normalized value of the information entropy is greater than or equal to 0.2 and less than 0.4 as 40 points, and the first score of the text data whose normalized value of the information entropy is less than 0.2 as 20 points; the product of the number of errors in the syntactic structure verification result of the text data and 5 is recorded as the deduction amount of the text data, and the difference between 100 points and the deduction amount of the text data is recorded as the second score of the text data; the average of the first score and the second score of the text data is recorded as the quality score of the text data.

[0055] Specifically, this embodiment uses the VMAF video multi-method evaluation fusion method, adopts the SVR support vector regression model to fuse basic image quality indicators, predicts the overall image quality, and obtains an output score. The larger the output score, the better the image quality. The value range of the output score is greater than 0 and equal to and less than or equal to 100. The output score of the image data is recorded as the quality score of the image data.

[0056] Specifically, this embodiment obtains the quality score of the audio data through the non-reference audio quality assessment framework NISQA, sets the value range of the quality score of the audio data to greater than or equal to 0 and less than or equal to 100, and records the quality score of the audio data as the quality score of the audio data.

[0057] Specifically, in this embodiment, the normalized value of the correlation coefficient between the numerical sensor data and the data acquired by the sensor using the standard calibration source is recorded as the quality score of the numerical sensor data.

[0058] Among them, calculating the information entropy of text data, obtaining the syntactic structure verification results of text data, obtaining the output score through the VMAF video multi-method evaluation fusion method, and obtaining the quality score of audio data through the no-reference audio quality assessment framework NISQA are all well-known technologies and will not be described in detail. In actual application, as other implementation methods, on the basis of achieving the purpose of respectively obtaining the quality scores of text data, image data, audio data, and numerical sensor data, implementers can also use other existing methods to respectively obtain the quality scores of text data, image data, audio data, and numerical sensor data. This application does not impose any special restrictions.

[0059] For any type of multimodal data in the multimodal data of the smart device to be maintained, the ratio of the quality score of the same type of multimodal data to the cumulative sum of the quality scores of all multimodal data is recorded as the quality score ratio of the same type of multimodal data; the product of the data volume of the same type of multimodal data and the quality score ratio is recorded as the usage weight of the same type of multimodal data.

[0060] The usage weight of each type of multimodal data is used as the reference weight of the status evaluation result of each type of multimodal data on the smart device to be maintained. The normalized value of the absolute value of the difference between the multimodal data of the smart device to be maintained and the standard value corresponding to the multimodal data is weighted and summed. The difference between the number 1 and the weighted summation result is used as the degree of compliance of the actual operation of the smart device to be maintained.

[0061] The standard values ​​corresponding to the multimodal data are values ​​preset by those skilled in the art and are determined by experts in this field based on the smart device to be maintained.

[0062] When the degree of compliance is less than or equal to the compliance threshold, the operation of the smart device to be maintained is determined to be abnormal. The similarity between the standard multimodal data corresponding to all issues of the smart device to be maintained in the database and the multimodal data of the smart device to be maintained is calculated. The issue corresponding to the standard multimodal data with the greatest similarity is recorded as the operational issue of the smart device to be maintained. The standard maintenance method corresponding to the operational issue of the smart device to be maintained is fed back and uploaded to provide a maintenance reference for the smart device to be maintained. When the degree of compliance is greater than the preset compliance threshold, the operation of the smart device to be maintained is determined to be normal, and maintenance of the smart device to be maintained is not performed.

[0063] Among them, the compliance threshold is a preset constant value, and in this embodiment, the compliance threshold value is 0.5; the standard multimodal data corresponding to all problems of the smart device to be maintained is data extracted by technical personnel in this field when corresponding problems occur in devices of the same model as the smart device to be maintained; the standard maintenance method corresponding to the operating problems of the smart device to be maintained is determined by the instructions provided by the manufacturer of the smart device to be maintained.

[0064] At this point, the intelligent device maintenance based on the large model is completed.

[0065] Based on the same inventive concept as the above-mentioned method, an embodiment of the present invention also provides a large-model-based intelligent device maintenance system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-mentioned large-model-based intelligent device maintenance methods.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for maintaining intelligent equipment based on a large model, characterized in that: The method comprises the following steps: Collect multimodal data of the smart device to be maintained and build a large model. Each small model in the large model corresponds to an entity, and related small models are connected by edges; Each small model in the large model is used as a starting point, and the small models in the large model are traversed. The analysis sequence of the starting point is determined based on the traversal results. The degree of similarity in entity expression between two different small models in the large model is determined based on the position difference of the same small model in the analysis sequence of two different small models in the large model, as well as the number of the same small models. Based on the degree of similarity in entity expression, all two different small models in the large model that correspond to the same entity are determined; According to the number of different entity relationships in the large model contained in the text data, the first confidence of each entity relationship is determined respectively, and any two different small models corresponding to the same entity in the large model are respectively recorded as the first small model to be aligned and the second small model to be aligned, and the reference value of the second small model to be aligned corresponding to the first small model to be aligned is determined according to the result of traversing the large model after directly aligning the first small model to be aligned and the first confidence of the small model contained in the large model traversal result of the second small model to be aligned, and according to the reference value of all small models corresponding to the same entity as the first small model to be aligned, all small models corresponding to the same entity as the first small model to be aligned are aligned; Based on all types of multimodal data and the amount of multimodal data, the compliance level of the actual operation of the smart device to be maintained is determined, and based on the compliance level, whether the smart device to be maintained has an abnormality is determined, and maintenance is performed on the smart device to be maintained if an abnormality occurs.

2. The large model-based intelligent device maintenance method according to claim 1, characterized in that: The multimodal data includes text data, image data, audio data and numerical sensor data.

3. The large model-based intelligent device maintenance method according to claim 1, characterized in that: The specific method for determining the analysis sequence of the starting point is: Each small model in the large model is used as a starting point to traverse the small models in the large model to obtain the traversal sequence of the starting point. The subsequence consisting of the small models in the first third of the traversal sequence of the starting point is recorded as the analysis sequence of the starting point.

4. The large model-based intelligent device maintenance method according to claim 1, characterized in that: The method for obtaining the similarity of entity expressions of two different small models in the large model is as follows: The two different small models in the large model are respectively recorded as the first small model and the second small model, the same small models in the analysis sequence of the first small model and the second small model are recorded as the overlapping small models of the first small model and the second small model, and the number of overlapping small models of the first small model and the second small model is recorded as the first number of the first small model and the second small model; The absolute value of the difference in the order of the overlapping small models of the first small model and the second small model in the analysis sequence of the first small model and the second small model is recorded as the order difference of the overlapping small models, and the cumulative sum of the order differences of all overlapping small models is recorded as the cumulative sum of the orders of the first small model and the second small model; In the analysis sequence of the first small model and the second small model, the number of small models between the first small model of the analysis sequence is less than The number of overlapping small models is recorded as the second number of the first small model and the second small model, where Indicates the first parameter of the preset; The ratio of the product of the first quantity and the second quantity of the first small model and the second small model to the sequential cumulative sum is recorded as the entity expression similarity between the first small model and the second small model.

5. The large model-based intelligent device maintenance method according to claim 1, characterized in that: The specific method of determining all two different small models corresponding to the same entity in the large model based on the similarity of entity expression includes: The normalized value of the entity expression similarity is greater than or equal to The two small models are determined to be different small models corresponding to the same entity, where Indicates the second parameter of the preset.

6. The large model-based intelligent device maintenance method according to claim 2, characterized in that: The method for obtaining the first confidence of the entity relationship is: Record any entity relationship as a target entity relationship, and record the ratio of the number of target entity relationships contained in the text data to the number of all entity relationships contained in the text data as the first quantity ratio of the target entity relationship; The number of all indirect connection paths between two entities contained in the text data in the large model is recorded as the number of first paths of the target entity relationship; The product of the first quantity ratio of the target entity relationship and the first path quantity of the target entity relationship is recorded as the first confidence of the target entity relationship.

7. The large model-based intelligent device maintenance method according to claim 1, characterized in that: The specific method for determining the reference value of the second small model to be aligned corresponding to the first small model to be aligned is: Directly perform entity alignment on the entity relationship between the first small model to be aligned and the second small model to be aligned, and record the obtained small model as the aligned small model; traverse the small models in the large model using the aligned small model in the large model as the starting point to obtain a traversal sequence of the aligned small models; record the number of identical small models in the traversal sequence of the second small model to be aligned and the aligned small model as the third number of the second small model to be aligned; Record the cumulative sum of the first confidences of the small models included in the traversal sequence of the second small model to be aligned as the second cumulative sum of the second small model to be aligned; The product of the second accumulated sum of the second small model to be aligned and the third number is recorded as the reference value of the second small model to be aligned corresponding to the first small model to be aligned.

8. The method for maintaining intelligent devices based on a large model according to claim 1, characterized in that: The method of aligning all small models corresponding to the same entity as the first small model to be aligned according to the reference values ​​of all small models corresponding to the same entity as the first small model to be aligned includes: The cumulative sum of the reference values ​​of all small models corresponding to the same entity of the first small model to be aligned is recorded as the total reference value of the first small model to be aligned, and the ratio of the reference value of the second small model to be aligned to the total reference value of the first small model to be aligned is recorded as the reference value ratio of the second small model to be aligned; According to the reference value ratio of the second small model to be aligned, the entities to which the processed data corresponding to all small models corresponding to the same entity as the first small model to be aligned belong are randomly extracted to achieve alignment of all small models corresponding to the same entity as the first small model to be aligned.

9. The large model-based intelligent device maintenance method according to claim 1, characterized in that: The method of determining the compliance level of the actual operation of the smart device to be maintained based on all types of multimodal data and the amount of the multimodal data, determining whether the smart device to be maintained is abnormal based on the compliance level, and maintaining the smart device to be maintained if abnormal is present, includes the following specific methods: Calculate the quality score of each type of multimodal data separately; The ratio of the quality score of the multimodal data of the same type in the multimodal data of the smart device to be maintained to the cumulative sum of the quality scores of all the multimodal data is recorded as the quality score ratio of the multimodal data of the same type; the product of the data volume of the multimodal data of the same type and the quality score ratio is recorded as the usage weight of the multimodal data of the same type; The usage weight of each type of multimodal data is used as a reference weight for the status evaluation result of the smart device to be maintained by each type of multimodal data, and the normalized value of the absolute value of the difference between the multimodal data of the smart device to be maintained and the standard value corresponding to the multimodal data is weighted and summed. The difference between the number 1 and the weighted summation result is used as the compliance degree of the actual operation of the smart device to be maintained; When the degree of compliance is less than or equal to a preset compliance threshold, it is determined that the operation of the smart device to be maintained is abnormal, and the similarity between the standard multimodal data corresponding to all problems of the smart device to be maintained in the database and the multimodal data of the smart device to be maintained is calculated. The problem corresponding to the standard multimodal data with the greatest similarity is recorded as the operation problem of the smart device to be maintained, and the standard maintenance method corresponding to the operation problem of the smart device to be maintained is fed back and uploaded; When the compliance level is greater than a preset compliance threshold, it is determined that the operation of the smart device to be maintained is not abnormal and maintenance is not performed.

10. A large model-based intelligent device maintenance system comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

Images