Handheld operation and maintenance terminal based on multi-modal large model and operation and maintenance method

Abstract

The invention provides a handheld operation and maintenance terminal based on a multi-modal large model and an operation and maintenance method. The handheld operation and maintenance terminal comprises a multi-mode input module which can receive multi-mode operation and maintenance information input by operation and maintenance personnel; wherein the multi-modal operation and maintenance information comprises at least one of operation and maintenance information in a text form, operation and maintenance information in an image form and operation and maintenance information in a voice form; the multi-modal processing and fusion module is configured to perform feature extraction for each form of information in the multi-modal operation and maintenance information and perform fusion processing on the extracted features so as to convert the features into uniform vector space representation; and the intelligent question and answer engine module is configured with a pre-trained multi-modal large model, takes the received unified vector space representation as the input of the multi-modal large model, obtains an answer which is output by the multi-modal large model and aims at the multi-modal operation and maintenance information, and provides the answer to operation and maintenance personnel. According to the invention, under the condition of no network or weak network, operation and maintenance personnel can automatically solve problems in operation and maintenance offline.

Description

Technical Field

[0001] One or more embodiments of the present invention relate to network communication technology, and more particularly to a handheld operation and maintenance terminal and an operation and maintenance method based on a multimodal large model. Background Art

[0002] In many application scenarios, maintenance personnel are required to arrive at the site for maintenance. For example, equipment in some sites needs to be inspected regularly so that problems can be found and solved in time. For example, maintenance personnel need to inspect a chemical plant regularly to check whether the boiler, heating, pressure equipment and other equipment or environmental conditions (such as temperature, humidity, fog) in the chemical plant are normal.

[0003] In the existing operation and maintenance methods, operation and maintenance personnel usually go to the site for inspection. If any abnormal situation is found, they will take photos and upload them to the cloud through the operation and maintenance software. In this way, the management personnel can understand the abnormal situation through the pictures uploaded in the cloud and take corresponding measures, such as sending maintenance personnel to the site for repair.

[0004] It can be seen that the prior art has the following problems:

[0005] 1. When the operation and maintenance personnel conduct on-site inspections and find problems, they are unable to independently solve the problem offline, resulting in delayed handling of the problem.

[0006] 2. Operation and maintenance personnel must upload relevant photos or texts to the cloud through the Internet. This requires good network conditions at the inspection site. However, in many inspection sites, such as those in deserts or mountains, good Internet conditions are often not available, or even no network is available. In this case, existing technologies cannot achieve operation and maintenance management.

[0007] 3. Problems in the operation and maintenance process can only be solved through the cooperation of operation and maintenance personnel, management personnel, and maintenance personnel, which greatly wastes time and manpower and leads to an increase in operation and maintenance costs. Summary of the invention

[0008] One or more embodiments of the present invention describe a handheld operation and maintenance terminal and an operation and maintenance method based on a multimodal large model, which can solve at least one problem of the prior art.

[0009] According to a first aspect, a handheld operation and maintenance terminal based on a multimodal large model is provided, the handheld operation and maintenance terminal comprising:

[0010] The multimodal input module is capable of receiving multimodal operation and maintenance information input by the operation and maintenance personnel, and sending the multimodal operation and maintenance information to the multimodal processing and fusion module; wherein the multimodal operation and maintenance information includes: at least one of operation and maintenance information in text form, operation and maintenance information in image form, and operation and maintenance information in voice form;

[0011] The multimodal processing and fusion module is configured to extract features from each form of information in the multimodal operation and maintenance information, and to fuse the extracted features to convert them into a unified vector space representation, and send the unified vector space representation to the intelligent question answering engine module;

[0012] The intelligent question-answering engine module is configured with a pre-trained multimodal large model, which takes the received unified vector space representation as the input of the multimodal large model, obtains the answer to the multimodal operation and maintenance information output by the multimodal large model, and provides the answer to the operation and maintenance personnel.

[0013] Further including: mobile optimization and interaction design modules;

[0014] The mobile optimization and interaction design module is configured to integrate the multimodal input module, the multimodal processing and fusion module, and the intelligent question and answer engine module into the handheld operation and maintenance terminal based on the multimodal big model, provide user interface and interaction mode design functions, and perform adaptability adjustments according to the type of operating system and screen size on the handheld operation and maintenance terminal based on the multimodal big model.

[0015] The intelligent question-answering engine module performs the following processing when training the multimodal large model:

[0016] Obtain a faulty device picture set; wherein the faulty device picture set includes pictures of at least two faulty devices;

[0017] Obtain the question and target answer for each faulty device image in the faulty device image set;

[0018] Select the adjustment strategy and hyperparameters of the multimodal large model based on task requirements and available resources;

[0019] Initialize the parameters of the multimodal large model according to the weights of the multimodal large model;

[0020] The multimodal large model is trained using a faulty device picture set, questions and target answers for each faulty device picture in the faulty device picture set, and an adjustment strategy for the multimodal large model.

[0021] The intelligent question answering engine module is further used to quantize and trim the trained multimodal large model, and performs the following processing during the quantization and trimming process:

[0022] Count the corresponding minimum and maximum values ​​in the weights or activation values ​​of the multimodal large model;

[0023] Select the quantization type according to the application scenario and requirements;

[0024] According to the selected quantization type, minimum value and maximum value, the quantization parameter zero point Z and scaling factor S are calculated;

[0025] A quantization operation is performed on the multimodal large model according to the faulty device picture set, the question asked and the target answer for each faulty device picture in the faulty device picture set, wherein the parameters Z and S are used to ensure that the quantized data retains the characteristics and information of the original data to the maximum extent.

[0026] According to a second aspect, an operation and maintenance method of a handheld operation and maintenance terminal based on a multimodal large model is provided, the method comprising:

[0027] Receive multimodal operation and maintenance information input by an operation and maintenance personnel; wherein the multimodal operation and maintenance information includes: at least one of operation and maintenance information in text form, operation and maintenance information in image form, and operation and maintenance information in voice form; and the multimodal operation and maintenance information is used to describe problems found by the operation and maintenance personnel during the inspection process;

[0028] For each form of multimodal operation and maintenance information, feature extraction is performed separately, and various extracted features are fused and converted into a unified vector space representation;

[0029] Inputting the unified vector space representation into a pre-trained multimodal large model to obtain an answer to the multimodal operation and maintenance information output by the multimodal large model;

[0030] Provide answers to multimodal operation and maintenance information to operation and maintenance personnel.

[0031] The training method of the multimodal large model includes:

[0032] Obtain a faulty device picture set; wherein the faulty device picture set includes pictures of at least two faulty devices;

[0033] Obtain the question and target answer for each faulty device image in the faulty device image set;

[0034] Select the adjustment strategy and hyperparameters of the multimodal large model based on task requirements and available resources;

[0035] Initialize the parameters of the multimodal large model according to the weights of the multimodal large model;

[0036] The multimodal large model is trained using a faulty device picture set, questions and target answers for each faulty device picture in the faulty device picture set, and an adjustment strategy for the multimodal large model.

[0037] After obtaining the question and the target answer for each faulty device image in the faulty device image set, the method further includes:

[0038] The faulty device image set is divided into a training set, a validation set, and a test set, wherein each of the training set, the validation set, and the test set includes a faulty device image and a question and a target answer corresponding to the faulty device image;

[0039] The process of training the multimodal large model includes:

[0040] Training the multimodal large model using the faulty device images included in the training set and the questions and target answers corresponding to the faulty device images;

[0041] Using the faulty device images included in the validation set and the questions and target answers corresponding to the faulty device images, regularly evaluate the multimodal large model, and adjust the hyperparameters and / or adjustment strategies of the multimodal large model according to the evaluation results;

[0042] The multimodal large model is evaluated using the faulty device images included in the test set and the questions and target answers corresponding to the faulty device images to obtain a multimodal large model that meets the training objectives.

[0043] The method further comprises:

[0044] The trained multimodal large model is quantized and pruned. The following processing is performed during the quantization and pruning process:

[0045] Count the corresponding minimum and maximum values ​​in the weights or activation values ​​of the multimodal large model;

[0046] Select the quantization type according to the application scenario and requirements;

[0047] According to the selected quantization type, minimum value and maximum value, calculate and obtain the quantization parameter zero point Z and scaling factor S; and

[0048] A quantization operation is performed on the multimodal large model according to the faulty device picture set, the question and the target answer for each faulty device picture in the faulty device picture set, wherein the parameters Z and S are used to ensure that the quantized data retains the characteristics and information of the original data to the maximum extent;

[0049] The multimodal large model obtained after quantization and trimming is deployed to the handheld operation and maintenance terminal.

[0050] The performing of quantization operation on the multimodal large model includes:

[0051] The weights and activation values ​​of the multimodal large model are converted from the original floating point format to the quantized integer format.

[0052] According to a third aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any embodiment of the present invention is implemented.

[0053] The handheld operation and maintenance terminal and operation and maintenance method based on the multimodal large model provided by various embodiments of the present invention have at least the following beneficial effects:

[0054] 1. When the operation and maintenance personnel conduct on-site inspections and find problems, they can input the problems into the handheld operation and maintenance terminal in the form of multimodal operation and maintenance information, such as photos and text descriptions. The handheld operation and maintenance terminal can process the multimodal operation and maintenance information, extract features separately, and use the fused features and the pre-trained multimodal large model to obtain answers to the problems described by the multimodal operation and maintenance information, and directly provide the answers to the operation and maintenance personnel. Therefore, the operation and maintenance personnel do not need to upload the problems found during the inspection to the cloud, but can solve the problems independently offline, so that the problems can be handled as soon as possible, avoiding delayed processing of the problems.

[0055] 2. Operation and maintenance personnel do not need to upload photos or texts describing the problem to the cloud through the Internet. The handheld operation and maintenance terminal can process and provide answers to the problems by itself. Therefore, it can help inspection personnel complete inspection tasks in places with poor network signals or even places without network, such as deserts or mountains, greatly increasing the reliability of the business.

[0056] 3. There is no need for the cooperation of operation and maintenance personnel, management personnel, and maintenance personnel. The operation and maintenance personnel can solve the problems in the operation and maintenance process by using a handheld operation and maintenance terminal, which greatly saves time, manpower, and operation and maintenance costs.

[0057] 4. Operation and maintenance personnel can input operation and maintenance information in multiple modes to describe a problem. For example, when an operation and maintenance personnel finds that a device makes a strange sound, they can input three forms of operation and maintenance information into the handheld operation and maintenance terminal: the video of the device, the temperature and humidity information on site, and the text description of the operation and maintenance personnel. Because the operation and maintenance information used to describe the problem can be input in a multimodal manner, the handheld operation and maintenance terminal can obtain richer description information of the problem from multiple dimensions, which is more conducive to the handheld operation and maintenance terminal to obtain more accurate answers to the problem.

[0058] 5. Handheld operation and maintenance terminals can use multimodal large models to get answers to questions, thereby providing operation and maintenance personnel with a more intelligent, convenient, and personalized information acquisition and interactive experience. For example, through the question-and-answer method, it can help operation and maintenance personnel solve problems encountered during the inspection process.

[0059] 6. The handheld operation and maintenance terminal integrates advanced artificial intelligence technology, which can simultaneously process multimodal data such as images and texts, and realize comprehensive perception and intelligent analysis of operation and maintenance scenarios. It can not only take pictures to obtain images of faulty equipment, quickly locate the fault point through image recognition technology, but also provide fault solutions using text question functions. At the same time, combined with natural language processing and big data analysis, the terminal can deeply understand operation and maintenance needs, provide personalized solutions and predictive maintenance suggestions, effectively prevent potential failures, and reduce operation and maintenance costs. In addition, the design of multimodal interaction makes it easy for non-professionals to get started, broadening the talent boundaries of the operation and maintenance team. In short, the application of this handheld operation and maintenance terminal not only improves the efficiency and accuracy of operation and maintenance, but also promotes the intelligent and automated development of operation and maintenance management, laying a solid foundation for the digital transformation and sustainable development of enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 It is a schematic diagram of a handheld operation and maintenance terminal based on a multimodal large model in one embodiment of the present invention.

[0062] Figure 2 It is a schematic diagram of a handheld operation and maintenance terminal based on a multimodal large model in another embodiment of the present invention.

[0063] Figure 3 It is a flow chart of an operation and maintenance method of a handheld operation and maintenance terminal based on a multimodal large model in one embodiment of the present invention.

[0064] Figure 4 It is a flow chart of an operation and maintenance method of a handheld operation and maintenance terminal based on a multimodal large model in another embodiment of the present invention. DETAILED DESCRIPTION

[0065] The solution provided by the present invention is described below in conjunction with the accompanying drawings.

[0066] First of all, it should be noted that the terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise.

[0067] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0068] Figure 1 FIG. 1 is a schematic diagram of a handheld operation and maintenance terminal based on a multimodal large model in one embodiment of the present invention. Figure 1 , the handheld operation and maintenance terminal includes:

[0069] The multimodal input module 101 is capable of receiving multimodal operation and maintenance information input by the operation and maintenance personnel, and sending the multimodal operation and maintenance information to the multimodal processing and fusion module; wherein the multimodal operation and maintenance information includes: at least one of operation and maintenance information in text form, operation and maintenance information in image form, and operation and maintenance information in voice form; and the multimodal operation and maintenance information is used to describe the problems found by the operation and maintenance personnel during the inspection process;

[0070] The multimodal processing and fusion module 102 is configured to extract features from each form of information in the multimodal operation and maintenance information, and to fuse the extracted features to convert them into a unified vector space representation, and to send the unified vector space representation to the intelligent question answering engine module;

[0071] The intelligent question-answering engine module 103 is configured with a pre-trained multimodal large model, takes the received unified vector space representation as the input of the multimodal large model, obtains the answer to the multimodal operation and maintenance information output by the multimodal large model, and provides the answer to the operation and maintenance personnel.

[0072] It can be seen that the embodiment of the present invention provides a handheld operation and maintenance terminal based on a multimodal large model, and the use of the handheld operation and maintenance terminal can achieve the following beneficial effects:

[0073] 1. When the operation and maintenance personnel conduct on-site inspections and find problems, they can input the problems into the handheld operation and maintenance terminal in the form of multimodal operation and maintenance information, such as photos and text descriptions. The handheld operation and maintenance terminal can process the multimodal operation and maintenance information, extract features separately, and use the fused features and the pre-trained multimodal large model to obtain answers to the problems described by the multimodal operation and maintenance information, and directly provide the answers to the operation and maintenance personnel. Therefore, the operation and maintenance personnel do not need to upload the problems found during the inspection to the cloud, but can solve the problems independently offline, so that the problems can be handled as soon as possible, avoiding delayed processing of the problems.

[0074] 2. Operation and maintenance personnel do not need to upload photos or texts describing the problem to the cloud through the Internet. The handheld operation and maintenance terminal can process and provide answers to the problems by itself. Therefore, it can help inspection personnel complete inspection tasks in places with poor network signals or even places without network, such as deserts or mountains, greatly increasing the reliability of the business.

[0075] 3. There is no need for the cooperation of operation and maintenance personnel, management personnel, and maintenance personnel. The operation and maintenance personnel can solve the problems in the operation and maintenance process by using a handheld operation and maintenance terminal, which greatly saves time, manpower, and operation and maintenance costs.

[0076] 4. Operation and maintenance personnel can input operation and maintenance information in multiple modes to describe a problem. For example, when an operation and maintenance personnel finds that a device makes a strange sound, they can input three forms of operation and maintenance information into the handheld operation and maintenance terminal: the video of the device, the temperature and humidity information on site, and the text description of the operation and maintenance personnel. Because the operation and maintenance information used to describe the problem can be input in a multimodal manner, the handheld operation and maintenance terminal can obtain richer description information of the problem from multiple dimensions, which is more conducive to the handheld operation and maintenance terminal to obtain more accurate answers to the problem.

[0077] 5. Handheld operation and maintenance terminals can use multimodal large models to get answers to questions, thereby providing operation and maintenance personnel with a more intelligent, convenient, and personalized information acquisition and interactive experience. For example, through the question-and-answer method, it can help operation and maintenance personnel solve problems encountered during the inspection process.

[0078] 6. The handheld operation and maintenance terminal integrates advanced artificial intelligence technology, which can simultaneously process multimodal data such as images and texts, and realize comprehensive perception and intelligent analysis of operation and maintenance scenarios. It can not only take pictures to obtain images of faulty equipment, quickly locate the fault point through image recognition technology, but also provide fault solutions using text question functions. At the same time, combined with natural language processing and big data analysis, the terminal can deeply understand operation and maintenance needs, provide personalized solutions and predictive maintenance suggestions, effectively prevent potential failures, and reduce operation and maintenance costs. In addition, the design of multimodal interaction makes it easy for non-professionals to get started, broadening the talent boundaries of the operation and maintenance team. In short, the application of this handheld operation and maintenance terminal not only improves the efficiency and accuracy of operation and maintenance, but also promotes the intelligent and automated development of operation and maintenance management, laying a solid foundation for the digital transformation and sustainable development of enterprises.

[0079] See also Figure 2 ,In one embodiment of the present invention, the handheld operation and maintenance terminal based on the multi-modal large model further includes: a mobile optimization and interaction design module 104;

[0080] The mobile optimization and interaction design module 104 is configured to integrate the multimodal input module 101, the multimodal processing and fusion module 102 and the intelligent question and answer engine module 103 into the handheld operation and maintenance terminal based on the multimodal large model, provide user interface and interaction mode design functions, and perform adaptive adjustments according to the type of operating system and screen size on the handheld operation and maintenance terminal based on the multimodal large model.

[0081] Combine the following Figure 1 and Figure 2 The handheld operation and maintenance terminal based on the multimodal large model in the embodiment of the present invention is described. The handheld operation and maintenance terminal based on the multimodal large model can be applied to daily inspections.

[0082] In one embodiment of the present invention, the multimodal input module 101 may specifically include the following functions:

[0083] (1.1) Receiving multimodal information: The multimodal input module 101 is the entry point for the inspector to interact with the handheld operation and maintenance terminal, and is responsible for receiving various types of information input by the inspector, such as text input (such as keyboard input, handwriting recognition), image input (by taking pictures with a camera or uploading pictures), voice input, etc., so that the inspector can submit multimodal operation and maintenance information describing the problem.

[0084] (1.2) Preprocessing: Before passing the multimodal operation and maintenance information to the subsequent processing module, the multimodal input module 101 may further perform preliminary preprocessing on the input multimodal operation and maintenance information, such as denoising, format conversion, compression, etc., to ensure the accuracy and processability of the multimodal operation and maintenance information.

[0085] In one embodiment of the present invention, the multimodal processing and fusion module 102 specifically includes the following functions:

[0086] (2.1) Information classification: Classify the received multimodal operation and maintenance information and distinguish different types of information such as text, pictures, audio, and video;

[0087] (2.2) Different algorithms and techniques are used to extract features of different types of information. For text information, natural language processing technology is used for word segmentation, part-of-speech tagging, syntactic analysis, etc. For image information, convolutional neural network (CNN) is used to extract image features;

[0088] (2.3) The various extracted features are integrated to form a unified representation so that the subsequent intelligent question-answering engine module 103 can process it.

[0089] In one embodiment of the present invention, the intelligent question answering engine module 103 can specifically complete the following functions through a pre-trained multimodal large model:

[0090] (3.1) Question understanding: deeply understand the unified vector space representation output by the multimodal processing and fusion module 102, and analyze the user's intention and question content;

[0091] (3.2) Knowledge retrieval: Retrieve relevant knowledge from the knowledge base based on the user’s question. The knowledge base includes pre-trained language models, domain knowledge graphs, databases, etc.

[0092] (3.3) Answer generation: Generate accurate answers based on the retrieved knowledge and the user’s questions. The answers can be in text form or in multimedia form such as images.

[0093] In one embodiment of the present invention, the mobile optimization and interaction design module 104 specifically includes the following functions:

[0094] (4.1) Performance optimization: Optimize the system performance based on the characteristics of mobile devices, i.e., handheld operation and maintenance terminals (such as processor performance, memory size, screen size, etc.) to ensure smooth operation on the handheld operation and maintenance terminals;

[0095] (4.2) Interaction design: Design intuitive and easy-to-use user interfaces and interaction methods to reduce user learning costs and improve user experience. This includes layout design, color matching, animation effects, interactive feedback, etc.

[0096] (4.3) Make adaptability adjustments based on different operating systems (such as iOS, Android) and screen sizes (such as mobile phones, tablets) to ensure the stability and consistency of the system on different devices and scenarios.

[0097] As mentioned above, in the embodiment of the present invention, a multimodal large model needs to be pre-trained. When training the multimodal large model, the intelligent question answering engine module 103 performs the following processing:

[0098] Obtain a faulty device picture set; wherein the faulty device picture set includes pictures of at least two faulty devices;

[0099] Obtain the question and target answer for each faulty device image in the faulty device image set;

[0100] Select the adjustment strategy and hyperparameters of the multimodal large model based on task requirements and available resources;

[0101] Initialize the parameters of the multimodal large model according to the weights of the multimodal large model;

[0102] The multimodal large model is trained using a faulty device picture set, questions and target answers for each faulty device picture in the faulty device picture set, and an adjustment strategy for the multimodal large model.

[0103] In order to enable the multimodal large model to perform reasoning using only the chip computing power of the handheld operation and maintenance terminal, i.e., the mobile terminal, and complete the question-answering processing, so that it can be used in a weak network or no network environment, in one embodiment of the present invention, the trained multimodal large model is further quantized and pruned. The intelligent question-answering engine module 103 performs the following processing during the quantization and pruning process:

[0104] Count the corresponding minimum value (min_value) and maximum value (max_value) in the weights or activation values ​​of the multimodal large model;

[0105] Select the quantization type according to the application scenario and requirements;

[0106] According to the selected quantization type, minimum value and maximum value, the quantization parameter zero point Z and scaling factor S are calculated;

[0107] A quantization operation is performed on the multimodal large model according to the faulty device picture set, the question asked and the target answer for each faulty device picture in the faulty device picture set, wherein the parameters Z and S are used to ensure that the quantized data retains the characteristics and information of the original data to the maximum extent.

[0108] Figure 3 FIG. 1 is a flow chart of an operation and maintenance method of a handheld operation and maintenance terminal based on a multimodal large model in one embodiment of the present invention. Figure 3 The method is executed by a handheld operation and maintenance terminal, and the method includes:

[0109] Step 301: receiving multimodal operation and maintenance information input by an operation and maintenance personnel; wherein the multimodal operation and maintenance information includes: at least one of operation and maintenance information in text form, operation and maintenance information in image form, and operation and maintenance information in voice form; and the multimodal operation and maintenance information is used to describe problems found by the operation and maintenance personnel during the inspection process;

[0110] Step 303: extracting features for each form of information in the multimodal operation and maintenance information;

[0111] Step 305: various extracted features are fused to convert them into a unified vector space representation.

[0112] Step 307: Input the unified vector space representation into the pre-trained multimodal large model to obtain the answer to the multimodal operation and maintenance information output by the multimodal large model.

[0113] Step 309: Provide answers to the multimodal operation and maintenance information to the operation and maintenance personnel.

[0114] for Figure 3 The method shown in the figure, wherein the training method of the multimodal large model includes:

[0115] Obtain a faulty device picture set; wherein the faulty device picture set includes pictures of at least two faulty devices;

[0116] Obtain the question and target answer for each faulty device image in the faulty device image set;

[0117] Select the adjustment strategy and hyperparameters of the multimodal large model based on task requirements and available resources;

[0118] Initialize the parameters of the multimodal large model according to the weights of the multimodal large model;

[0119] The multimodal large model is trained using a faulty device picture set, questions and target answers for each faulty device picture in the faulty device picture set, and an adjustment strategy for the multimodal large model.

[0120] In order to further ensure the effect of the training, after obtaining the question and target answer for each faulty device picture in the faulty device picture set, further comprising: dividing the faulty device picture set into a training set, a validation set, and a test set, wherein each of the training set, the validation set, and the test set includes a faulty device picture and the question and target answer corresponding to the faulty device picture;

[0121] Accordingly, the process of training the multimodal large model includes: using the faulty equipment pictures included in the training set and the questions and target answers corresponding to the faulty equipment pictures to train the multimodal large model; using the faulty equipment pictures included in the validation set and the questions and target answers corresponding to the faulty equipment pictures to regularly evaluate the multimodal large model, and adjusting the hyperparameters and / or adjustment strategies of the multimodal large model according to the evaluation results; using the faulty equipment pictures included in the test set and the questions and target answers corresponding to the faulty equipment pictures to evaluate the multimodal large model to obtain a multimodal large model that meets the training objectives.

[0122] The method further comprises:

[0123] The trained multimodal large model is quantized and pruned. The following processing is performed during the quantization and pruning process:

[0124] Count the corresponding minimum and maximum values ​​in the weights or activation values ​​of the multimodal large model;

[0125] Select the quantization type according to the application scenario and requirements;

[0126] According to the selected quantization type, minimum value and maximum value, calculate and obtain the quantization parameter zero point Z and scaling factor S; and

[0127] A quantization operation is performed on the multimodal large model according to the faulty device picture set, the question and the target answer for each faulty device picture in the faulty device picture set, wherein the parameters Z and S are used to ensure that the quantized data retains the characteristics and information of the original data to the maximum extent;

[0128] The multimodal large model obtained after quantization and trimming is deployed to the handheld operation and maintenance terminal.

[0129] In one embodiment of the present invention, performing a quantization operation on a multimodal large model includes:

[0130] The weights and activation values ​​of the multimodal large model are converted from the original floating point format to the quantized integer format.

[0131] The methods of various embodiments of the present invention can at least achieve the six beneficial effects described above.

[0132] It can be seen that before the handheld operation and maintenance terminal is officially used, it is necessary to perform training, deployment and other processing of the multimodal large model. After that, the multimodal large model deployed in the handheld operation and maintenance terminal is used to answer questions during the inspection process. Figure 4 FIG. 1 is a flow chart of an operation and maintenance method of a handheld operation and maintenance terminal based on a multimodal large model in another embodiment of the present invention. Figure 4The method is executed by a handheld operation and maintenance terminal, and the method includes:

[0133] Execute step 401: obtain an equipment operation and maintenance data set to serve as a training data sample for training a multimodal large model.

[0134] The specific implementation process of step 401 may include: obtaining a faulty device picture set; wherein the faulty device picture set includes pictures of at least two faulty devices; obtaining a question and a target answer for each faulty device picture in the faulty device picture set. Dividing the faulty device picture set into a training set, a validation set, and a test set, each of which includes a faulty device picture and the question and target answer corresponding to the faulty device picture

[0135] Execute step 403: fine-tune the pre-trained model to obtain a large multimodal model.

[0136] The specific implementation process of step 403 may include:

[0137] Step 4031: Select a suitable pre-trained model as a multimodal large model according to the nature of the target task and the characteristics of the data set;

[0138] Step 4033: Select an appropriate fine-tuning strategy based on task requirements and available resources. Consider whether to perform full fine-tuning or partial fine-tuning, as well as the level and scope of fine-tuning;

[0139] Step 4035: Determine the hyperparameters in the fine-tuning process, such as learning rate, batch size, number of training rounds, etc. The selection of these hyperparameters has an important impact on the performance and convergence speed of fine-tuning;

[0140] Step 4037: Initialize the parameters of the multimodal large model according to the weights of the pre-trained model. For full fine-tuning, all model parameters will be randomly initialized; for partial fine-tuning, only the parameters of the top layer or a few layers will be randomly initialized;

[0141] Step 4039: Use the prepared training set and fine-tuning strategy to train the model. During the training process, the model parameters are gradually adjusted to minimize the loss function according to the set hyperparameters and optimization algorithm.

[0142] Step 40311: During the training process, use the validation set to regularly evaluate the multimodal large model and adjust the hyperparameters or fine-tune the strategy based on the evaluation results.

[0143] Step 40313: After fine-tuning is completed, the final multimodal large model is evaluated using the test set to obtain the final performance indicators.

[0144] Execute step 405: perform quantitative pruning on the multimodal large model.

[0145] The processing of step 405 is to enable the multimodal large model to be applied to the handheld operation and maintenance terminal as a mobile terminal. The process of step 405 may specifically include:

[0146] Step 4051: Count min_value and max_value: Count the corresponding minimum value (min_value) and maximum value (max_value) in the input data (usually weights or activation values);

[0147] Step 4053: Select the appropriate quantization type according to the specific application scenario and requirements. Common quantization types include symmetric quantization (such as int8) and asymmetric quantization (such as uint8). Symmetric quantization means that the quantized data is centered on 0, which is suitable for situations where the data distribution is relatively uniform; asymmetric quantization allows the data to maintain its original dynamic range after quantization, which is suitable for situations where the data distribution is uneven;

[0148] Step 4055: Calculate and obtain quantization parameters Z (Zero point) and S (Scale) according to the selected quantization type, min_value and max_value;

[0149] Step 4057: Quantize the model based on the calibration data (i.e., the data used to train or validate the model). This step typically involves converting the model's weights and activations from the original floating point format (e.g., FP32) to a quantized integer format (e.g., INT8). During the conversion process, the quantization parameters Z and S calculated previously are used to ensure that the quantized data retains the characteristics and information of the original data as much as possible.

[0150] Execute step 407: deploy relevant operation and maintenance software APP on the handheld operation and maintenance terminal.

[0151] In this step 407, the APP is installed on the mobile terminal, that is, the handheld operation and maintenance terminal, and permissions such as storage and photo taking are released.

[0152] Execute step 409: deploy the multimodal large model to the handheld operation and maintenance terminal.

[0153] The specific process of step 409 includes:

[0154] Turn on USB debugging mode and developer mode on your phone;

[0155] Download the multimodal large model file and use the adb tool to put the multimodal large model file in the specified location of the mobile terminal, i.e. the handheld operation and maintenance terminal;

[0156] Restart the APP of the handheld operation and maintenance terminal.

[0157] Execute step 411: the handheld operation and maintenance terminal takes a photo of the faulty device and asks questions, and the handheld operation and maintenance terminal analyzes the photo of the faulty device and outputs answers to the questions.

[0158] The specific process of step 411 includes:

[0159] The mobile terminal allows taking photos;

[0160] The operation and maintenance personnel use the APP to take photos of the faulty equipment and upload them to the APP for image analysis;

[0161] Wait for the image to be parsed and ask questions about the image;

[0162] Based on the image analysis results and the question description, the multimodal large model performs reasoning and analysis to return the answer to the question.

[0163] It should be noted that the above-mentioned devices are usually implemented on the server side, and can be set on independent servers respectively, or some or all of the devices can be combined and set on the same server. The server can be a single server or a server cluster composed of multiple servers. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. The above-mentioned devices can also be implemented on a computer terminal with strong computing power.

[0164] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute a method in any one of the embodiments in the specification.

[0165] An embodiment of the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method in any embodiment of the specification is implemented.

[0166] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the device of the embodiment of the present invention. In other embodiments of the specification, the above-mentioned device may include more or fewer components than those shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0167] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0168] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, widgets, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0169] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A handheld operation and maintenance terminal based on a multimodal large model, characterized in that: The handheld operation and maintenance terminal includes: The multimodal input module is capable of receiving multimodal operation and maintenance information input by the operation and maintenance personnel, and sending the multimodal operation and maintenance information to the multimodal processing and fusion module; wherein the multimodal operation and maintenance information includes: at least one of operation and maintenance information in text form, operation and maintenance information in image form, and operation and maintenance information in voice form; The multimodal processing and fusion module is configured to extract features from each form of information in the multimodal operation and maintenance information, and to fuse the extracted features to convert them into a unified vector space representation, and send the unified vector space representation to the intelligent question answering engine module; The intelligent question-answering engine module is configured with a pre-trained multimodal large model, which takes the received unified vector space representation as the input of the multimodal large model, obtains the answer to the multimodal operation and maintenance information output by the multimodal large model, and provides the answer to the operation and maintenance personnel.

2. The handheld operation and maintenance terminal based on a multimodal large model according to claim 1 is characterized in that: Further including: Mobile optimization and interaction design module; The mobile optimization and interaction design module is configured to integrate the multimodal input module, the multimodal processing and fusion module, and the intelligent question and answer engine module into the handheld operation and maintenance terminal based on the multimodal big model, provide user interface and interaction mode design functions, and perform adaptability adjustments according to the type of operating system and screen size on the handheld operation and maintenance terminal based on the multimodal big model.

3. The handheld operation and maintenance terminal based on a multimodal large model according to claim 1 is characterized in that: The intelligent question-answering engine module performs the following processing when training the multimodal large model: Obtain a faulty device picture set; wherein the faulty device picture set includes pictures of at least two faulty devices; Obtain the question and target answer for each faulty device image in the faulty device image set; Select the adjustment strategy and hyperparameters of the multimodal large model based on task requirements and available resources; Initialize the parameters of the multimodal large model according to the weights of the multimodal large model; The multimodal large model is trained using a faulty device picture set, questions and target answers for each faulty device picture in the faulty device picture set, and an adjustment strategy for the multimodal large model.

4. The handheld operation and maintenance terminal based on a multimodal large model according to claim 1 is characterized in that: The intelligent question answering engine module is further used to quantize and trim the trained multimodal large model, and performs the following processing during the quantization and trimming process: Count the corresponding minimum and maximum values ​​in the weights or activation values ​​of the multimodal large model; Select the quantization type according to the application scenario and requirements; According to the selected quantization type, minimum value and maximum value, the quantization parameter zero point Z and scaling factor S are calculated; A quantization operation is performed on the multimodal large model according to the faulty device picture set, the question asked and the target answer for each faulty device picture in the faulty device picture set, wherein the parameters Z and S are used to ensure that the quantized data retains the characteristics and information of the original data to the maximum extent.

5. The operation and maintenance method of the handheld operation and maintenance terminal based on the multi-modal large model is characterized in that: The method includes: Receive multimodal operation and maintenance information input by an operation and maintenance personnel; wherein the multimodal operation and maintenance information includes: at least one of operation and maintenance information in text form, operation and maintenance information in image form, and operation and maintenance information in voice form; and the multimodal operation and maintenance information is used to describe problems found by the operation and maintenance personnel during the inspection process; For each form of multimodal operation and maintenance information, feature extraction is performed separately, and various extracted features are fused and converted into a unified vector space representation; Inputting the unified vector space representation into a pre-trained multimodal large model to obtain an answer to the multimodal operation and maintenance information output by the multimodal large model; Provide answers to multimodal operation and maintenance information to operation and maintenance personnel.

6. The method according to claim 5, characterized in that The training method of the multimodal large model includes: Obtain a faulty device picture set; wherein the faulty device picture set includes pictures of at least two faulty devices; Obtain the question and target answer for each faulty device image in the faulty device image set; Select the adjustment strategy and hyperparameters of the multimodal large model based on task requirements and available resources; Initialize the parameters of the multimodal large model according to the weights of the multimodal large model; The multimodal large model is trained using a faulty device picture set, questions and target answers for each faulty device picture in the faulty device picture set, and an adjustment strategy for the multimodal large model.

7. The method according to claim 6, characterized in that After obtaining the question and the target answer for each faulty device image in the faulty device image set, the method further includes: The faulty device image set is divided into a training set, a validation set, and a test set, wherein each of the training set, the validation set, and the test set includes a faulty device image and a question and a target answer corresponding to the faulty device image; The process of training the multimodal large model includes: Training the multimodal large model using the faulty device images included in the training set and the questions and target answers corresponding to the faulty device images; Using the faulty device images included in the validation set and the questions and target answers corresponding to the faulty device images, regularly evaluate the multimodal large model, and adjust the hyperparameters and / or adjustment strategies of the multimodal large model according to the evaluation results; The multimodal large model is evaluated using the faulty device images included in the test set and the questions and target answers corresponding to the faulty device images to obtain a multimodal large model that meets the training objectives.

8. The method according to claim 5, characterized in that The method further comprises: The trained multimodal large model is quantized and pruned. The following processing is performed during the quantization and pruning process: Count the corresponding minimum and maximum values ​​in the weights or activation values ​​of the multimodal large model; Select the quantization type according to the application scenario and requirements; According to the selected quantization type, minimum value and maximum value, calculate and obtain the quantization parameter zero point Z and scaling factor S; and A quantization operation is performed on the multimodal large model according to the faulty device picture set, the question and the target answer for each faulty device picture in the faulty device picture set, wherein the parameters Z and S are used to ensure that the quantized data retains the characteristics and information of the original data to the maximum extent; The multimodal large model obtained after quantization and trimming is deployed to the handheld operation and maintenance terminal.

9. The method according to claim 8, characterized in that The performing of quantization operation on the multimodal large model includes: The weights and activation values ​​of the multimodal large model are converted from the original floating point format to the quantized integer format.

10. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 5 to 9 is implemented.

Images