Intelligent device maintenance method and system based on large model

Abstract

The application relates to the technical field of text processing, and discloses an intelligent device maintenance method and system based on a large model, which comprises the following steps: collecting multi-modal data of an intelligent device to be maintained, and establishing a large model; determining the entity expression similarity of two different small models in the large model, and determining all the two different small models corresponding to the same entity; determining the first confidence of each entity relationship, defining a first small model to be aligned and a second small model to be aligned and determining the corresponding reference value, aligning all the small models corresponding to the same entity as the first small model to be aligned according to the reference value of all the small models corresponding to the same entity as the first small model to be aligned; determining the compliance degree of the actual operation of the intelligent device to be maintained, determining whether the intelligent device to be maintained is abnormal according to the compliance degree, and maintaining the abnormal intelligent device to be maintained. The application can improve the accuracy of device maintenance.

Description

Technical Field

[0001] This invention relates to the field of text processing technology, and more specifically to a method and system for maintaining intelligent devices based on large models. Background Technology

[0002] Multimodal data refers to data acquired from different channels and in different formats. This includes text data such as equipment logs generated during equipment operation, image data such as images captured by monitoring cameras on key internal components, audio data such as sound signals during equipment operation, and numerical sensor data such as operating temperature measured by temperature sensors. Smart equipment maintenance based on multimodal data involves comprehensively utilizing this data from multiple modalities, employing data processing, analysis, and mining techniques to achieve maintenance of smart equipment. This allows for a more comprehensive and accurate understanding of equipment status, improves the accuracy of fault diagnosis, and enables predictive maintenance.

[0003] Traditional methods for intelligent equipment maintenance based on multimodal data do not consider the differences in reference quality among different modalities. As a result, the reference data from different modalities have the same weight in influencing the equipment condition evaluation results, which can easily lead to inaccurate evaluation of the reference quality of the equipment condition evaluation results and affect the accuracy of the equipment maintenance evaluation results. Summary of the Invention

[0004] This invention provides a method and system for intelligent equipment maintenance based on a large model, to solve the problem that inaccurate reference quality evaluation of equipment condition evaluation results due to different modal data leads to inaccurate equipment maintenance evaluation results. The specific technical solution adopted is as follows:

[0005] In a first aspect, one embodiment of the present invention provides a method for maintaining intelligent devices based on a large model, the method comprising the following steps:

[0006] Collect multimodal data of the smart devices to be maintained, and build a large model. Each small model in the large model corresponds to an entity, and related small models are connected by edges.

[0007] Each small model in the large model is taken as a starting point. The small models in the large model are traversed. The analysis sequence of the starting point is determined based on the traversal results. The position difference of the same small model in the analysis sequence of two different small models in the large model and the number of the same small model are used to determine the entity expression similarity of two different small models in the large model. Based on the entity expression similarity, all two different small models in the large model that correspond to the same entity are determined.

[0008] Based on the number of different entity relationships in the large model contained in the text data, the first confidence level of each entity relationship is determined. Any two different small models corresponding to the same entity in the large model are denoted as the first small model to be aligned and the second small model to be aligned, respectively. Based on the result of traversing the large model after directly aligning the first small model to be aligned and the second small model to be aligned, and the first confidence level of the small models contained in the traversal result of the large model of the second small model to be aligned, the reference value of the second small model to be aligned corresponding to the first small model is determined. Based on the reference value of all small models corresponding to the same entity as the first small model to be aligned, all small models corresponding to the same entity as the first small model to be aligned are aligned.

[0009] Based on all types of multimodal data and the amount of multimodal data, determine the degree of compliance of the actual operation of the smart device to be maintained, determine whether the smart device to be maintained has an anomaly based on the degree of compliance, and perform maintenance on the smart device to be maintained that has an anomaly.

[0010] Furthermore, the multimodal data includes: text data, image data, audio data, and numerical sensor data.

[0011] Furthermore, the specific method for determining the analysis sequence of the starting point is as follows:

[0012] Each small model in the large model is used as a starting point to traverse the small models in the large model, and the traversal sequence of the starting point is obtained. The subsequence composed of the small models in the first third of the traversal sequence of the starting point is denoted as the analysis sequence of the starting point.

[0013] Furthermore, the method for obtaining the similarity of entity representations between two different smaller models in the larger model is as follows:

[0014] Two different sub-models in the large model are denoted as the first sub-model and the second sub-model, respectively. Sub-models that are the same in the analysis sequence of the first sub-model and the second sub-model are denoted as overlapping sub-models of the first sub-model and the second sub-model. The number of overlapping sub-models of the first sub-model and the second sub-model is denoted as the first number of the first sub-model and the second sub-model.

[0015] The absolute value of the difference between the order of the overlapping small models of the first and second small models in the analysis sequence of the first and second small models is denoted as the order difference of the overlapping small models. The sum of the order differences of all overlapping small models is denoted as the order sum of the first and second small models.

[0016] In the analysis sequence of the first and second small models, the number of small models separated from the first small model in the analysis sequence is less than [a certain number]. The number of overlapping small models is denoted as the second number of the first and second small models, where... This indicates the preset first parameter;

[0017] The ratio of the product of the first and second quantities of the first and second mini-models to the sum of their order is denoted as the entity representation similarity between the first and second mini-models.

[0018] Furthermore, the specific method for determining all two different smaller models corresponding to the same entity in the larger model based on the similarity of entity representations includes:

[0019] The normalized value of the similarity between entity representations is greater than or equal to The two smaller models are determined to be different smaller models corresponding to the same entity, among which, This indicates the preset second parameter.

[0020] Furthermore, the method for obtaining the first confidence level of the entity relationship is as follows:

[0021] Let any one entity relation be denoted as the target entity relation, and let the ratio of the number of target entity relations in the text data to the total number of all entity relations in the text data be denoted as the first proportion of the number of target entity relations.

[0022] The number of all indirect connecting paths between two entities contained in the text data in the large model is denoted as the number of the first path of the target entity relationship.

[0023] The first confidence level of the target entity relationship is the product of the first number of the target entity relationship and the first path number of the target entity relationship.

[0024] Furthermore, the specific method for determining the reference value of the second small model to be aligned corresponding to the first small model to be aligned is as follows:

[0025] The entity relationships between the first small model to be aligned and the second small model to be aligned are directly aligned, and the obtained small model is recorded as the aligned small model; the aligned small model in the large model is used as the starting point to traverse the small models in the large model to obtain the traversal sequence of the aligned small model; the number of the same small models in the traversal sequence of the second small model to be aligned and the aligned small model is recorded as the third number of the second small model to be aligned.

[0026] The sum of the first confidence scores of the small models contained in the traversal sequence of the second small model to be aligned is denoted as the second sum of the second small model to be aligned.

[0027] The product of the second cumulative sum and the third quantity of the second small model to be aligned is denoted as the reference value of the second small model to be aligned corresponding to the first small model to be aligned.

[0028] Furthermore, the specific method for aligning all small models corresponding to the same entity as the first small model to be aligned, based on their reference value, includes:

[0029] The sum of the reference values ​​of all small models corresponding to the same entity of the first small model to be aligned is denoted as the total reference value of the first small model to be aligned. The ratio of the reference value of the second small model to be aligned to the total reference value of the first small model to be aligned is denoted as the reference value ratio of the second small model to be aligned.

[0030] Based on the reference value ratio of the second small model to be aligned, the entities to which the processed data belongs for all small models corresponding to the same entity as the first small model to be aligned are randomly selected, thereby achieving alignment of all small models corresponding to the same entity as the first small model to be aligned.

[0031] Furthermore, the specific methods for determining the degree of compliance of the actual operation of the smart device to be maintained based on all types of multimodal data and the amount of multimodal data, determining whether the smart device to be maintained is abnormal based on the degree of compliance, and maintaining the smart device to be maintained if it is abnormal include:

[0032] Calculate the quality score for each type of multimodal data;

[0033] The ratio of the quality score of the same type of multimodal data in the multimodal data of the smart device to be maintained to the sum of the quality scores of all multimodal data is denoted as the quality score ratio of the same type of multimodal data; the product of the data volume of the same type of multimodal data and the quality score ratio is denoted as the usage weight of the same type of multimodal data.

[0034] The weights used for each type of multimodal data are used as reference weights for the state evaluation results of the smart device to be maintained. The normalized values ​​of the absolute values ​​of the differences between the multimodal data of the smart device to be maintained and the corresponding standard values ​​are weighted and summed. The difference between the number 1 and the weighted summation result is used as the degree of conformity of the actual operation of the smart device to be maintained.

[0035] When the compliance level is less than or equal to the preset compliance threshold, it is determined that the operation of the smart device to be maintained is abnormal. The similarity between the standard multimodal data corresponding to all problems of the smart device to be maintained in the database and the multimodal data of the smart device to be maintained is calculated. The problem corresponding to the standard multimodal data with the highest similarity is recorded as the operation problem of the smart device to be maintained. The standard maintenance method corresponding to the operation problem of the smart device to be maintained is fed back and uploaded.

[0036] When the compliance level is greater than the preset compliance threshold, it is determined that the operation of the smart device to be maintained is not abnormal and no maintenance is performed.

[0037] Secondly, embodiments of the present invention also provide an intelligent device maintenance system based on a large model, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0038] The beneficial effects of this invention are:

[0039] This application first establishes a large model directly based on the multimodal data of the intelligent device to be maintained. Then, considering that the same entity may have multiple different names in the multimodal data of the intelligent device, leading to multiple different representations of the same entity in the large model and causing errors, this application analyzes the overlap of related entity relationships between different entities to determine the similarity of entity representations between two different sub-models within the large model. The greater the overlap of entity relationships, the greater the overlap of entity relationships between entities corresponding to different sub-models, and the greater the similarity of entity representations. Based on the similarity of entity representations, all two different sub-models corresponding to the same entity in the large model are identified. Entities in the large model with multiple representations are highly likely to be important reference data for judging certain fault conditions of the device. The confidence levels of reference data under different representations in indicating faults vary. To ensure that the reference data corresponding to entities is more meaningful for judging equipment faults, the confidence levels of the relevant entity relationships of the small models corresponding to different expression methods are obtained. The first confidence level of each entity relationship is determined, and the reference value of the second small model corresponding to the first small model to be aligned is determined. Based on the reference values ​​of all small models corresponding to the same entity as the first small model to be aligned, all small models corresponding to the same entity as the first small model to be aligned are aligned. Finally, based on all types of multimodal data and the amount of multimodal data, the degree of conformity of the actual operation of the intelligent equipment to be maintained is determined. Based on the degree of conformity, it is determined whether the intelligent equipment to be maintained has malfunctioned, and maintenance is performed on the intelligent equipment to be maintained that has malfunctioned. This addresses the problem of inaccurate reference quality evaluation of equipment status evaluation results by different modal data, which leads to inaccurate equipment maintenance evaluation results, and improves the accuracy of equipment maintenance. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating a method for maintaining intelligent devices based on a large model, provided in one embodiment of the present invention.

[0042] Figure 2 This is a flowchart of the analysis sequence acquisition process provided in one embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Please see Figure 1 The diagram illustrates a flowchart of a smart device maintenance method based on a large model according to an embodiment of the present invention. The method includes the following steps:

[0045] Step S001: Collect multimodal data of the smart device to be maintained and establish a large model. Each small model in the large model corresponds to an entity, and related small models are connected by edges.

[0046] Extract multimodal data of devices that require maintenance; these devices are the intelligent devices to be maintained.

[0047] Specifically, the multimodal data selected in this embodiment includes text data, image data, audio data, and numerical sensor data. The text data includes log information generated during device operation, which includes the device's operation time, operation content, and error codes. The image data consists of images captured by monitoring cameras of key internal components of the device requiring maintenance; these key internal components are identified by those skilled in the art. The audio data is the sound signal generated during device operation requiring maintenance. The numerical sensor data is the device's operating temperature measured by a temperature sensor. In practical applications, as other implementation methods, implementers can determine the types and number of data types included in the multimodal data according to actual circumstances; this application does not impose any special limitations.

[0048] Knowledge extraction, knowledge fusion, and knowledge storage are performed on multimodal data to build a large model.

[0049] Among these, building large models from multimodal data is a well-known technique and will not be elaborated further; knowledge extraction includes entity extraction, relation extraction, and attribute extraction; knowledge fusion includes entity alignment, relation fusion, and knowledge integration. Specifically, this embodiment performs data cleaning, modality alignment, and data annotation for each type of multimodal data. Data cleaning includes denoising and dimensionless removal. This embodiment uses the Z-Score standard normalization method for dimensionless removal and median filtering for denoising; text-image pairing is used for modality alignment; cross-modal entity referencing is achieved through MEL multimodal entity linking technology, linking text data, image data, audio data, and numerical sensor data to knowledge base entities respectively; the pre-trained model BERT and the visual feature extractor ResNet are combined to identify entities and their types based on text and non-text modalities; from text... Semantic relationships between entities are extracted from the data; visual similarity relationships between image entities and text entities are established; the extracted entities are aligned to the knowledge base, and the links are verified using the MELBench dataset; image data, audio data, and numerical sensor data are treated as independent entities and associated equally with text data, supporting intra-modal relationships; entities are used as small models, and related small models are connected by edges to construct triples, and an adjacency matrix is ​​obtained based on the triples, where the triples show the semantic relationships between entities, and the matrix can intuitively represent the association relationships between entities; the entities, relationships, and attributes extracted from multimodal data are collectively formed into a unified large model structure.

[0050] Understandably, the large model is the primary model.

[0051] At this point, the large model is obtained.

[0052] Step S002: Take each small model in the large model as a starting point, traverse the small models in the large model, determine the analysis sequence of the starting point based on the traversal results, determine the entity expression similarity of the two different small models in the large model based on the position difference of the same small model in the analysis sequence of the two different small models in the large model, and the number of the same small models, and determine the entity expression similarity of all two different small models in the large model that correspond to the same entity.

[0053] A large model contains multiple smaller models, and the relationships between these smaller models can be abstracted as a network. Each smaller model corresponds to a modality of an entity, which can be text, image, video, etc. When building a large model using existing knowledge, there may be cases where some smaller models express the same entity modality but are not classified into the same smaller model, leading to errors in the large model. To ensure the correctness of the large model, it is necessary to align the entities in the large model by examining the overlap of related entity relationships between different smaller models.

[0054] Each small model in the large model is taken as a starting point. The Breadth-First Search (BFS) algorithm is used to visit the small models in the large model layer by layer starting from the starting point. A queue is used to ensure that the first visited vertex is expanded first, so as to traverse the small models in the large model and obtain the traversal sequence corresponding to the starting point. The subsequence composed of the first third of the small models in the traversal sequence corresponding to the starting point is denoted as the analysis sequence of the starting point.

[0055] At this point, the analysis sequence for each sub-model within the larger model can be obtained. The flowchart for obtaining the analysis sequence is as follows: Figure 2 As shown, the traversal sequence of the starting point by using the breadth-first search algorithm to traverse the smaller models in the larger model is a well-known technique and will not be elaborated further.

[0056] The similarity of entity representations between two different small models in the large model is determined by the positional differences of the same small models in the analysis sequence of the two different small models in the large model, and the number of the same small models in the analysis sequence of the two different small models in the large model.

[0057] Two distinct sub-models within the large model are designated as the first sub-model and the second sub-model, respectively. Sub-models identical in the analysis sequences of the first and second sub-models are designated as overlapping sub-models. The number of overlapping sub-models is designated as the first quantity of the first and second sub-models. The absolute value of the difference in the order of these overlapping sub-models within the analysis sequences of the first and second sub-models is designated as the order difference of the overlapping sub-models. The sum of all order differences of overlapping sub-models is designated as the order sum of the first and second sub-models. Within the analysis sequences of the first and second sub-models, the number of sub-models separated from the first sub-model in the analysis sequence is less than [a certain value]. The number of overlapping small models is denoted as the first small model and the second small model; the ratio of the product of the first and second small models to the sequential summation is denoted as the entity representation similarity between the first and second small models.

[0058] Understandable, This represents the first parameter, which is a preset parameter value. In this embodiment, the value of the first parameter is 10.

[0059] The similarity of entity representations between the first and second mini-models is used to evaluate the degree of overlap in entity relationships between the entities corresponding to the first and second mini-models. The greater the overlap in entity relationships, the greater the overlap in entity relationships between the entities corresponding to the first and second mini-models, and the greater the similarity in entity representations between the first and second mini-models.

[0060] The normalized value of the similarity between entity representations is greater than or equal to The first and second mini-models are determined to be different mini-models corresponding to the same entity; the normalized value of the entity expression similarity is less than... The first and second mini-models are determined to be different mini-models that do not correspond to the same entity. The normalized value of the similarity in entity representation is greater than or equal to... The entity relationships between the first and second sub-models are merged to achieve entity alignment.

[0061] It should be noted that this embodiment uses the Z-Score standard normalization method to calculate the normalized value. In practical applications, implementers may use other existing methods such as the maximum-minimum normalization method or the sigmoid function to calculate the normalized value, and no limitation is made here; it is understood that... This represents the second parameter, which is a preset parameter value. In this embodiment, the value of the second parameter is 0.8.

[0062] The same method can be used to obtain all two different smaller models that correspond to the same entity in the larger model.

[0063] This completes the process of obtaining two different smaller models that correspond to the same entity within the larger model.

[0064] Step S003: Based on the number of different entity relationships contained in the text data and in the large model, determine the first confidence level of each entity relationship. Record any two different small models corresponding to the same entity in the large model as the first small model to be aligned and the second small model to be aligned, respectively. Based on the result of traversing the large model after directly aligning the first small model to be aligned and the second small model to be aligned, and the first confidence level of the small models contained in the traversal result of the large model of the second small model to be aligned, determine the reference value of the second small model to be aligned corresponding to the first small model. Based on the reference value of all small models corresponding to the same entity as the first small model to be aligned, align all small models corresponding to the same entity as the first small model to be aligned.

[0065] Entities with multiple representations in a large model are highly likely to be important reference data for diagnosing certain equipment faults. The confidence levels of these reference data differ depending on their representation. To ensure that the reference data corresponding to an entity is more meaningful for diagnosing equipment faults, the confidence levels of relevant entity relationships in the text data referenced when acquiring smaller models corresponding to different representations are obtained. Specifically, for entity modalities with multiple smaller models, the reference value of different smaller models is evaluated by examining the performance of each entity relationship in the text data referenced during the construction of these smaller models. It can be understood that an entity relationship is a triple formed by two entities connected by an edge in the large model; when an entity relationship appears in the text data, it is determined that the text data contains an entity relationship.

[0066] Based on the number of different entity relationships contained in the text data and in the large model, determine the first confidence level for each entity relationship.

[0067] Let any entity relationship be designated as the target entity relationship. Calculate the ratio of the number of target entity relationships in the text data to the total number of entity relationships in the text data, and denote this as the first proportion of the target entity relationship. Use the Depth-First Search (DFS) algorithm and backtracking mechanism to obtain all indirect connection paths between the two entities in the target entity relationship. Denote the total number of all indirect connection paths in the large model as the first path count of the target entity relationship. Denote the product of the first proportion of the target entity relationship and the first path count of the target entity relationship as the first confidence level of the target entity relationship.

[0068] The use of the Depth-First Search (DFS) algorithm and backtracking mechanism to obtain all indirect connecting paths between two entities in an entity relationship is a well-known technique and will not be elaborated further.

[0069] The same method can be used to obtain the first confidence level of all entity relationships in a large model.

[0070] The first confidence level of entity relationships is evaluated based on the reference value of each sub-model exhibited by different entity relationships. Furthermore, it is necessary to consider the relationships between the different sub-models corresponding to an entity before and after entity alignment with other sub-models to further evaluate the reference value of each sub-model. The more similar the entity relationships of the different sub-models corresponding to an entity are before and after entity alignment, and the higher the first confidence level of the related entity relationships of the different sub-models corresponding to the entity, the greater the reference value of the sub-models corresponding to the entity for judging equipment faults.

[0071] Any two different small models corresponding to the same entity in the large model are denoted as the first small model to be aligned and the second small model to be aligned.

[0072] The entity relationships between the first and second small models to be aligned are directly aligned. The small model obtained after entity alignment is denoted as the aligned small model. Taking the aligned small model in the large model as the starting point, the Breadth-First Search (BFS) algorithm is used to visit the small models in the large model layer by layer from the starting point. A queue is used to ensure that the first visited vertex is expanded first, thus traversing the small models in the large model and obtaining the traversal sequence corresponding to the aligned small model. The number of identical small models in the traversal sequence of the second small model to be aligned and the aligned small model is denoted as the third quantity of the second small model to be aligned. The sum of the first confidence of the small models contained in the traversal sequence of the second small model to be aligned is denoted as the second sum of the second small model to be aligned. The product of the second sum and the third quantity of the second small model to be aligned is denoted as the reference value of the second small model to be aligned corresponding to the first small model to be aligned.

[0073] The same method can be used to obtain the reference value of all small models in the large model that correspond to the same entity as the first small model to be aligned.

[0074] Based on the reference value of all small models corresponding to the same entity as the first small model to be aligned, align all small models corresponding to the same entity as the first small model to be aligned.

[0075] The sum of the reference values ​​of all small models corresponding to the same entity as the first small model to be aligned is denoted as the total reference value of the first small model to be aligned. The ratio of the reference value of the second small model to be aligned to the total reference value of the first small model to be aligned is denoted as the reference value ratio of the second small model to be aligned. The entities to which the processed data belongs are randomly selected according to the reference value ratio of the second small model to be aligned, so as to achieve alignment of all small models corresponding to the same entity as the first small model to be aligned.

[0076] The same method can be used to align all small models that correspond to the same entity.

[0077] This completes the alignment of all smaller models corresponding to the same entity.

[0078] Step S004: Based on all types of multimodal data and the amount of multimodal data, determine the degree of compliance of the actual operation of the smart device to be maintained, determine whether the smart device to be maintained has an anomaly based on the degree of compliance, and perform maintenance on the smart device to be maintained that has an anomaly.

[0079] When determining the problems of the smart devices to be maintained based on the large model, there may be more than one type of multimodal data that can be referenced. Therefore, the specific types of problems of the smart devices to be maintained should be determined together based on different types of multimodal data.

[0080] Calculate the quality score for each type of multimodal data.

[0081] Specifically, in this embodiment, the normalized value of the information entropy of the text data is calculated. The first score of text data with a normalized information entropy value greater than or equal to 0.8 is recorded as 100 points; the first score of text data with a normalized information entropy value greater than or equal to 0.6 and less than 0.8 is recorded as 80 points; the first score of text data with a normalized information entropy value greater than or equal to 0.4 and less than 0.6 is recorded as 60 points; the first score of text data with a normalized information entropy value greater than or equal to 0.2 and less than 0.4 is recorded as 40 points; and the first score of text data with a normalized information entropy value less than 0.2 is recorded as 20 points. The product of the number of errors in the syntactic structure verification result of the text data and 5 is recorded as the deduction score of the text data. The difference between 100 points and the deduction score of the text data is recorded as the second score of the text data. The average of the first score and the second score of the text data is recorded as the quality score of the text data.

[0082] Specifically, this embodiment uses the VMAF video multi-method evaluation fusion method, employs the SVR support vector regression model to fuse basic image quality indicators, predicts the overall image quality, and obtains an output score. The larger the output score, the better the image quality. The output score ranges from 0 to 100. The output score of the image data is recorded as the quality score of the image data.

[0083] Specifically, in this embodiment, the audio data quality score is obtained through the No Reference Audio Quality Assessment (NISQA) framework. The range of the audio data quality score is set to be greater than or equal to 0 and less than or equal to 100. The audio data quality score is recorded as the audio data quality score.

[0084] Specifically, in this embodiment, the normalized value of the correlation coefficient between the numerical sensor data and the data acquired by the sensor from the standard calibration source is denoted as the quality score of the numerical sensor data.

[0085] The calculation of information entropy of text data, the acquisition of syntactic structure verification results of text data, the acquisition of output scores through the VMAF video multi-method evaluation fusion method, and the acquisition of audio data quality scores through the NISQA (No Reference Audio Quality Assessment) framework are all well-known techniques and will not be elaborated further. In practical applications, as other implementation methods, while achieving the goal of separately acquiring quality scores for text data, image data, audio data, and numerical sensor data, implementers may also employ other existing methods to separately acquire the quality scores for text data, image data, audio data, and numerical sensor data; this application does not impose any special limitations.

[0086] For any type of multimodal data in the multimodal data of the smart device to be maintained, the ratio of the quality score of the multimodal data of the same type to the sum of the quality scores of all multimodal data is denoted as the quality score ratio of the multimodal data of the same type; the product of the data volume of the multimodal data of the same type and the quality score ratio is denoted as the usage weight of the multimodal data of the same type.

[0087] The weights used for each type of multimodal data are used as reference weights for the state evaluation results of the smart device to be maintained. The normalized values ​​of the absolute values ​​of the differences between the multimodal data of the smart device to be maintained and the corresponding standard values ​​are weighted and summed. The difference between the number 1 and the weighted summation result is used as the degree of conformity of the actual operation of the smart device to be maintained.

[0088] The standard values ​​corresponding to the multimodal data are preset values ​​by those skilled in the art and are determined by experts in the art based on the smart device to be maintained.

[0089] When the compliance level is less than or equal to the compliance threshold, the operation of the smart device to be maintained is determined to be abnormal. The similarity between the standard multimodal data corresponding to all problems of the smart device to be maintained in the database and the multimodal data of the smart device to be maintained is calculated. The problem corresponding to the standard multimodal data with the highest similarity is recorded as the operation problem of the smart device to be maintained. The standard maintenance method corresponding to the operation problem of the smart device to be maintained is then uploaded to provide maintenance reference for the smart device to be maintained. When the compliance level is greater than the preset compliance threshold, the operation of the smart device to be maintained is determined not to be abnormal, and maintenance is not performed on the smart device to be maintained.

[0090] The threshold value is a preset constant value, and in this embodiment, the threshold value is 0.5. The standard multimodal data corresponding to all problems of the smart device to be maintained are the data extracted by those skilled in the art when the same model of the smart device to be maintained has the same problem. The standard maintenance method corresponding to the operation problem of the smart device to be maintained is determined by the instruction manual provided by the manufacturer of the smart device to be maintained.

[0091] This completes the maintenance of intelligent devices based on a large model.

[0092] Based on the same inventive concept as the above methods, embodiments of the present invention also provide a large-model-based intelligent device maintenance system, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-described large-model-based intelligent device maintenance methods.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent device maintenance based on a large model, characterized by, The method comprises the following steps: Collecting multi-modal data of the intelligent device to be maintained, establishing a large model, each small model in the large model corresponding to an entity, and the small models associated with each other being connected by edges; Taking each small model in the large model as a starting point respectively, traversing the small models in the large model, determining an analysis sequence of the starting point according to the traversal result, determining the similarity degree of entity expression of two different small models in the large model according to the position difference of the same small model in the analysis sequence and the number of the same small model in the analysis sequence of the two different small models, and determining two different small models corresponding to the same entity in the large model; According to the number of different entity relationships in the large model contained in the text data, determining the first confidence of each entity relationship respectively, taking any two different small models corresponding to the same entity in the large model as a first small model to be aligned and a second small model to be aligned respectively, determining the reference value of the second small model to be aligned corresponding to the first small model to be aligned according to the result of traversing the large model after directly aligning the first small model to be aligned and the second small model to be aligned, and the first confidence of the small models contained in the large model traversal result of the second small model to be aligned, aligning all small models corresponding to the same entity as the first small model to be aligned according to the reference value of all small models corresponding to the same entity as the first small model to be aligned. According to the conformity degree of the actual operation of the intelligent device to be maintained, determining whether the intelligent device to be maintained is abnormal, and maintaining the abnormal intelligent device to be maintained.

2. The large model-based intelligent device maintenance method of claim 1, wherein, The multi-modal data comprises text data, image data, audio data and numerical sensor data. 3.The large model based intelligent device maintenance method of claim 1, wherein, The specific determination method of the analysis sequence of the starting point is: Taking each small model in the large model as a starting point respectively, traversing the small models in the large model, obtaining a traversal sequence of the starting point, and taking the small models in the first third of the traversal sequence of the starting point as a sub-sequence, as the analysis sequence of the starting point. 4.The large model based intelligent device maintenance method of claim 1, wherein, The acquisition method of the similarity degree of entity expression of the two different small models in the large model is: Taking the first small model and the second small model in the large model as the first small model and the second small model respectively, taking the same small models in the analysis sequence of the first small model and the second small model as the coincident small models of the first small model and the second small model, taking the number of the coincident small models of the first small model and the second small model as the first number of the first small model and the second small model; Taking the absolute value of the difference of the order of the coincident small models in the analysis sequence of the first small model and the second small model as the order difference of the coincident small models, and taking the cumulative sum of the order differences of all the coincident small models as the order cumulative sum of the first small model and the second small model; In the analysis sequence of the first small model and the second small model, the number of small models spaced apart from the first small model of the analysis sequence is less than the number of coinciding small models of the first small model and the second small model is recorded as a second number of the first small model and the second small model, wherein represents a preset first parameter; Taking the ratio of the product of the first number and the second number of the first small model and the second small model to the order cumulative sum as the similarity degree of entity expression of the first small model and the second small model. 5.The large model based intelligent device maintenance method of claim 1, wherein, The method for determining two different small models corresponding to the same entity in the large model according to the similarity of entity expressions comprises the following specific steps: The normalized value of the similarity degree of the entity expression is greater than or equal to Two small models are determined as different small models corresponding to the same entity, wherein, Indicates a preset second parameter. 6.The large model based intelligent device maintenance method of claim 2, wherein, The method for obtaining the first confidence of the entity relationship comprises the following specific steps: The ratio of the number of target entity relationships in the text data to the number of all entity relationships in the text data is denoted as the target entity relationship first quantity ratio; The number of indirect connection paths between all two entities in the text data in the large model is denoted as the first path quantity of the target entity relationship. The product of the target entity relationship first quantity ratio and the first path quantity of the target entity relationship is denoted as the first confidence of the target entity relationship.

7. The large model-based intelligent device maintenance method of claim 1, wherein, The specific method for determining the reference value of the to-be-aligned second small model corresponding to the to-be-aligned first small model comprises the following specific steps: The entity relationship of the to-be-aligned first small model and the to-be-aligned second small model is directly aligned, and the obtained small model is denoted as an aligned small model; the aligned small model in the large model is taken as a starting point to traverse the small models in the large model, and a traversal sequence of the aligned small model is obtained; the number of the same small models in the traversal sequence of the to-be-aligned second small model and the aligned small model is denoted as the third quantity of the to-be-aligned second small model; The cumulative sum of the first confidence of the small model contained in the traversal sequence of the to-be-aligned second small model is denoted as the second cumulative sum of the to-be-aligned second small model; The product of the second cumulative sum of the to-be-aligned second small model and the third quantity is denoted as the reference value of the to-be-aligned second small model corresponding to the to-be-aligned first small model. 8.The large model based intelligent device maintenance method of claim 1, wherein, The specific method for aligning all the small models corresponding to the same entity as the to-be-aligned first small model according to the reference values of all the small models corresponding to the same entity as the to-be-aligned first small model comprises the following specific steps: The cumulative sum of the reference values of all the small models corresponding to the same entity as the to-be-aligned first small model is denoted as the reference total value of the to-be-aligned first small model, and the ratio of the reference value of the to-be-aligned second small model to the reference total value of the to-be-aligned first small model is denoted as the reference value ratio of the to-be-aligned second small model; According to the reference value ratio of the to-be-aligned second small model, the entities to which the data processed by all the small models corresponding to the same entity as the to-be-aligned first small model belong are randomly extracted, so as to align all the small models corresponding to the same entity as the to-be-aligned first small model. 9.The large model based intelligent device maintenance method of claim 1, wherein, The specific method for determining the compliance degree of the actual operation of the to-be-maintained intelligent device according to all kinds of multi-modal data and the data quantity of the multi-modal data, determining whether the to-be-maintained intelligent device is abnormal according to the compliance degree, and maintaining the to-be-maintained intelligent device when the to-be-maintained intelligent device is abnormal comprises the following specific steps: The quality score of each kind of multi-modal data is calculated respectively; The ratio of the quality score of the same kind of multi-modal data in the multi-modal data of the to-be-maintained intelligent device to the cumulative sum of the quality scores of all multi-modal data is denoted as the quality score ratio of the same kind of multi-modal data, and the product of the data quantity of the same kind of multi-modal data and the quality score ratio is denoted as the use weight of the same kind of multi-modal data. The use weight of each kind of multi-modal data is taken as the reference weight of the multi-modal data of each kind to the state evaluation result of the smart device to be maintained, the normalized value of the absolute value of the difference between the multi-modal data of the smart device to be maintained and the standard value corresponding to the multi-modal data is weighted and summed, and the difference between the digital 1 and the weighted sum result is taken as the compliance degree of the actual operation of the smart device to be maintained; When the compliance degree is less than or equal to the preset compliance threshold, it is determined that the operation of the smart device to be maintained is abnormal, the similarity between the standard multi-modal data corresponding to all problems of the smart device to be maintained in the database and the multi-modal data of the smart device to be maintained is calculated, the problem corresponding to the standard multi-modal data with the largest similarity is recorded as the operation problem of the smart device to be maintained, and the standard maintenance method corresponding to the operation problem of the smart device to be maintained is fed back; When the compliance degree is greater than the preset compliance threshold, it is determined that the operation of the smart device to be maintained is not abnormal, and no maintenance is performed.

10. A large model-based intelligent device maintenance system comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1-9 when executing the computer program. The processor implements the steps of the method of any one of claims 1-9 when executing the computer program.

Images