Large model fine tuning and multi-mode RAG-based Internet of Things equipment operation and maintenance method and memory

Abstract

The invention relates to the technical field of Internet of Things, in particular to an Internet of Things equipment operation and maintenance method based on large model fine tuning and multi-mode RAG, and a memory, and the method comprises the steps: respectively collecting external image information and data information for to-be-diagnosed equipment; generating question information according to the external image information and the data information, and inputting the question information into the Internet of Things large model to obtain feedback information fed back by the Internet of Things large model; and determining whether the to-be-diagnosed equipment is abnormal according to the feedback information. Aiming at the problems of low efficiency and inaccuracy of the manual inspection process of the Internet of Vehicles equipment in the prior art, external image information corresponding to equipment appearance inspection and data information acquired through channels such as a sensor and equipment metadata are introduced; and an Internet of Things large model optimized for the Internet of Things field is introduced to analyze the information, so that the Internet of Things large model effectively judges whether a fault exists or not, the inspection accuracy is improved, only appearance images need to be shot, and the efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to an operation and maintenance method and storage device of Internet of Things based on large model fine-tuning and multi-modal RAG. Background Art

[0002] The Internet of Things refers to a network of physical devices, vehicles, appliances, and other physical objects that are embedded with sensors, software, and network connections that can collect and share data. IoT devices range from simple "smart home" devices such as smart thermostats, wearable devices such as smart watches and RFID-enabled clothing, to complex industrial machinery and transportation systems. Technology experts are even envisioning entire "smart cities" based on IoT technology. The Internet of Things enables these smart devices to communicate with each other and other Internet-enabled devices. Just like smartphones and gateways, a large network of interconnected devices is created that can exchange data and perform various tasks autonomously. Among them, the Internet of Vehicles, that is, IoT technology for automotive platform applications, is a relatively common application area. Retrieval-augmented Generation (RAG) is one of the hottest large model frontier technologies. The retrieval-augmented generation model combines language models and information retrieval technology. Specifically, when the model needs to generate text or answer questions, it first retrieves relevant information from a large collection of documents, and then uses this retrieved information to guide the generation of text, thereby improving the quality and accuracy of predictions.

[0003] In the prior art, during the use of Internet of Vehicles equipment, it is usually necessary to operate and maintain the Internet of Vehicles equipment to meet normal usage requirements.

[0004] For example, Chinese patent CN202310824356.6 discloses an intelligent operation and maintenance system for an Internet of Vehicles system, including: a facility layer for storing multiple types of data; a data acquisition module for collecting data in the facility layer and sending it to a data governance module; the data governance module performs data governance and data cleaning on the data, and saves the regularized data to a data storage module; the data storage module is used to store regularized data; a functional presentation layer for distributing various tasks to a task scheduling module through an API gateway; the task scheduling module is used to accept requests from the API gateway, determine task attributes, and send task attributes to a policy service module. It can proactively and timely discover risks and faults in the high concurrency and low latency scenario requirements of the Internet of Vehicles system, automatically handle solutions, and provide subsequent protection solutions.

[0005] However, in the actual implementation process, the inventors found that this type of technical solution usually relies on the sensor itself to report, or uses manual inspection as a supplement. However, when the sensor itself has limited sampling data or there is a fault, it is difficult to judge all kinds of faults; and manual inspection itself is inefficient and relies on manual experience. Summary of the invention

[0006] In view of the above problems existing in the prior art, a large model fine-tuning and multi-modal RAG IoT device operation and maintenance method and storage device are provided.

[0007] The specific technical solutions are as follows:

[0008] A method for operating and maintaining IoT devices based on large model fine-tuning and multi-modal RAG, comprising:

[0009] Step S1, collecting external image information and data information for the device to be diagnosed;

[0010] Step S2, generating question information according to the external image information and the data information and inputting the question information into the IoT big model to obtain feedback information fed back by the IoT big model;

[0011] Step S3: determining whether the device to be diagnosed has an abnormality according to the feedback information.

[0012] On the other hand, the step S1 comprises:

[0013] Step S11, photographing an external image of the device to be diagnosed, segmenting the external image of the device to be diagnosed to obtain a plurality of mark inspection areas, and collecting the data information of the device to be diagnosed;

[0014] Step S12, classifying the mark inspection area and outputting image description information for describing the mark inspection area;

[0015] Step S13: construct the external image information according to the image description information and the landmark inspection area.

[0016] On the other hand, the step S12 comprises:

[0017] Step S121, identifying the regional image and pixel position of the mark inspection area to determine the inspection position corresponding to the mark inspection area;

[0018] Step S122, retrieving a sample image according to the inspection position, and matching the sample image with the regional image to obtain a classification result;

[0019] Step S123: determining the image description information according to the classification result.

[0020] On the other hand, after step S122 and before executing step S123, the mark inspection area without problems is eliminated according to the classification result, and then the image description information is generated again.

[0021] On the other hand, in the step S1, the data information includes the operator to which the device to be diagnosed belongs, the base station location, the base station type, and the base station number.

[0022] On the other hand, before executing step S1, a process of determining a device to be diagnosed is also included, and the process of determining a device to be diagnosed includes:

[0023] Step S101, when a user feedback work order is triggered, a question-answer model is used to collect user input content, and network connection information of the user's device is obtained;

[0024] Step S102, determining an associated base station of the user equipment according to the user input content and the network connection information;

[0025] Step S103: taking the associated base station as the device to be diagnosed.

[0026] On the other hand, step S2 comprises:

[0027] Step S21, constructing an input vector as the question information according to the external image information and the data information;

[0028] Step S22: input the question information into the IoT big model to obtain the feedback information.

[0029] On the other hand, the step S21 comprises:

[0030] Step S211, performing feature extraction on the external image information to obtain image features, and performing vector embedding on the data information to obtain an embedding vector;

[0031] Step S212: construct the input vector according to the embedding vector and the image feature.

[0032] On the other hand, before executing step S1, a model fine-tuning process is also included, and the model fine-tuning process includes:

[0033] Step S201, collecting operation and maintenance sample data corresponding to the device to be diagnosed;

[0034] Step S202, constructing a training corpus using the operation and maintenance sample data;

[0035] Step S203: Use the training corpus to train the general big model to obtain the Internet of Things big model.

[0036] A memory includes computer instructions. When a computer device reads and executes the computer instructions, the above-mentioned Internet of Things device operation and maintenance method is executed.

[0037] The above technical solution has the following advantages or beneficial effects:

[0038] In order to solve the problem of low efficiency and inaccuracy in the manual inspection process of Internet of Vehicles equipment in the prior art, this solution introduces external image information corresponding to the appearance inspection of the equipment and data information collected through channels such as sensors and equipment metadata, and introduces an Internet of Things big model optimized for the Internet of Things field to analyze the above information, so that the Internet of Things big model can effectively judge whether there is a fault, thereby improving the accuracy of the inspection, and only needs to take appearance images, which improves efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The embodiments of the present invention will be described more fully with reference to the attached drawings, which are provided for illustration and description only and are not intended to limit the scope of the present invention.

[0040] Figure 1 It is an overall schematic diagram of an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of step A1 in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of step A12 in an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of a device determination process in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of step S2 in an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of step S21 in an embodiment of the present invention;

[0046] Figure 7 Schematic diagram of the model fine-tuning process in an embodiment of the present invention DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0049] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0050] The present invention comprises:

[0051] A method for operating and maintaining IoT devices based on large model fine-tuning and multi-modal RAG, such as Figure 1 As shown, including:

[0052] Step S1, collecting external image information and data information for the device to be diagnosed;

[0053] Step S2, generating question information according to the external image information and data information and inputting it into the IoT big model to obtain feedback information fed back by the IoT big model;

[0054] Step S3, determining whether the device to be diagnosed has an abnormality according to the feedback information.

[0055] Specifically, in order to address the problem of inefficiency and inaccuracy in the manual inspection process of Internet of Vehicles equipment in the prior art, in this embodiment, external image information corresponding to the appearance inspection of the equipment and data information collected through channels such as sensors and equipment metadata are introduced, and an Internet of Things big model optimized for the Internet of Things field is introduced to analyze the above information, so that the Internet of Things big model can effectively judge whether there is a fault, thereby improving the accuracy of the inspection, and only needs to take the appearance image, thereby improving efficiency.

[0056] In the implementation process, the above-mentioned IoT device operation and maintenance method is mainly configured as a software embodiment in a computer system, such as an IoT operation and maintenance system, which receives and circulates external operation and maintenance work orders, and then collects external image information and data information to diagnose the device to be diagnosed. Among them, the device to be diagnosed mainly refers to IoT devices, such as IoT base stations, collection sensors, positioning beacons, etc.

[0057] When the IoT operation and maintenance system receives the corresponding operation and maintenance work order, it transfers the operation and maintenance work order to the operation and maintenance personnel so that the operation and maintenance personnel can take photos of the appearance of the equipment to be diagnosed and process them to obtain external image information, and pull the relevant information of the corresponding equipment to be diagnosed from the system according to the equipment to be diagnosed corresponding to the operation and maintenance work order, including the operator to which the equipment to be diagnosed belongs, the base station location, the base station type, and the base station number, as data information.

[0058] Subsequently, the information of the above two dimensions is processed to form question content and then input into the IoT big model, which performs diagnosis to obtain feedback content, thereby determining the actual status of the device to be diagnosed.

[0059] In one embodiment, Figure 2 As shown, step S1 includes:

[0060] Step S11, photographing an external image of the device to be diagnosed, segmenting the external image of the device to obtain a plurality of marker inspection areas, and collecting data information of the device to be diagnosed;

[0061] Step S12, classifying the mark inspection area and outputting image description information for describing the mark inspection area;

[0062] Step S13, constructing external image information according to the image description information and the mark inspection area.

[0063] Specifically, in order to achieve a better description process of external image information, in this embodiment, the external image of the device to be diagnosed is firstly photographed, and then the target recognition model is used to identify the key points in the external image of the device, and multiple mark inspection areas are segmented from the external image of the device. Taking the Internet of Vehicles base station as an example, its key points mainly include wiring harness connectors at multiple different positions. Subsequently, for each mark inspection area, classification and prediction are performed to obtain the status of the mark inspection area, such as whether the connector is intact or loose, and then the image description information used to describe the mark inspection area is output. Finally, the image description information and the mark inspection area are assembled to construct external image information, including the description of the connector at a specific position, the corresponding picture, etc.

[0064] In one embodiment, Figure 3 As shown, step S12 includes:

[0065] Step S121, identifying the regional image and pixel position of the mark inspection area to determine the inspection position corresponding to the mark inspection area;

[0066] Step S122, retrieve a sample image according to the inspection position, and match the sample image with the regional image to obtain a classification result;

[0067] Step S123, determining image description information according to the classification result.

[0068] Specifically, in order to achieve a better description effect, in this embodiment, the pixel position corresponding to the mark inspection area in the external image of the device and the actually intercepted regional image are first input as a two-dimensional vector, so as to classify and determine the inspection position corresponding to the mark inspection area, such as the actual joint type and joint position. Then, sample images are retrieved according to the inspection position, mainly including normal images and typical fault images at the corresponding positions, and the sample images are matched with the regional images to obtain classification results corresponding to different fault types. Finally, the image description information is determined according to the classification results to realize the description process.

[0069] In one embodiment, after step S122 and before executing step S123, the mark inspection area without problems is eliminated according to the classification result, and then the image description information is generated.

[0070] Specifically, in order to achieve faster diagnostic efficiency, in this embodiment, after classification in step S122, the mark inspection areas without problems are eliminated and the image description information is regenerated, which reduces the number of mark points that need to generate image description information and improves diagnostic efficiency.

[0071] In one embodiment, before executing step S1, there is a process for determining the device to be diagnosed, such as Figure 4 As shown, the process of determining the device to be diagnosed includes:

[0072] Step S101, when a user feedback work order is triggered, a question-answer model is used to collect user input content, and network connection information of the user's device is obtained;

[0073] Step S102, determining the associated base station of the user equipment according to the user input content and the network connection information;

[0074] Step S103: taking the associated base station as a device to be diagnosed.

[0075] Specifically, in order to achieve better diagnostic efficiency, in this embodiment, the above-mentioned operation and maintenance system is connected to the user customer service process. When a user feedback work order is received in the customer service system, a question-and-answer model is used to interact with the user to collect user input content, such as whether there is a network failure, the location, etc., and the corresponding diagnostic program is triggered on the user device side to obtain the network connection information of the user device, and then the associated base station of the user device is determined according to the user input content and the network connection information, so as to automatically determine the device to be diagnosed and trigger the operation and maintenance work order.

[0076] In one embodiment, Figure 5 As shown, step S2 includes:

[0077] Step S21, constructing an input vector as question content according to external image information and data information;

[0078] Step S22, input the question content into the IoT big model to obtain feedback information.

[0079] Specifically, in order to achieve better question-asking efficiency, in this embodiment, after obtaining external image information and data information, an input vector is constructed as the question content according to the external image information and data information. The original features are retained by means of the input vector, which is convenient for processing by the IoT large model.

[0080] In one embodiment, Figure 6 As shown, step S21 includes:

[0081] Step S211, extracting features from external image information to obtain image features, and embedding data information into vectors to obtain embedding vectors;

[0082] Step S212: construct an input vector according to the embedding vector and the image features.

[0083] Specifically, in order to achieve better input effects, in this embodiment, a convolutional network is first used for external image information to extract high-dimensional features in the image to obtain image features; and, for data information, an embedding vector is formed through encoding and embedding layers, and finally the two parts are spliced ​​to obtain an input vector.

[0084] In one embodiment, before executing step S1, Figure 7 As shown, the model fine-tuning process is also included, and the model fine-tuning process includes:

[0085] Step S201, collecting operation and maintenance sample data corresponding to the device to be diagnosed;

[0086] Step S202, constructing a training corpus using operation and maintenance sample data;

[0087] Step S203: Use the training corpus to train the general large model to obtain the Internet of Things large model.

[0088] Specifically, in order to achieve a better diagnosis effect on the device status, in this embodiment, a fine-tuning process for the large model is also introduced before online deployment. Specifically, for the device types of multiple IoT devices to be inspected, operation and maintenance sample data corresponding to the devices to be diagnosed are collected in advance, including device appearance pictures, device parameters, diagnosis results and fault information. Subsequently, a training corpus is constructed based on the operation and maintenance sample data, including assembling the above information according to a specific template to form an input vector. The training corpus is then used to train the general large model, and the model is fine-tuned to obtain the IoT large model to achieve better diagnostic effects.

[0089] A memory includes computer instructions. When a computer device reads and executes the computer instructions, the above-mentioned Internet of Things device operation and maintenance method is executed.

[0090] Those skilled in the art will appreciate that various aspects of the present invention, or possible implementations of various aspects, may be specifically implemented as systems, methods, or computer program products. Therefore, various aspects of the present invention, or possible implementations of various aspects, may take the form of complete hardware embodiments, complete software embodiments (including firmware, resident software, etc.), or embodiments of combined software and hardware aspects, all collectively referred to herein as "circuits," "modules," or "systems." In addition, various aspects of the present invention, or possible implementations of various aspects, may take the form of computer program products, which refer to computer instructions stored in a memory.

[0091] The memory may be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable read-only memory (CD-ROM).

[0092] The processor in the computer reads the computer instructions stored in the memory, so that the processor can execute the functional actions specified in each step or the combination of steps in the flowchart; and generate a device for implementing the functional actions specified in each block or the combination of blocks in the block diagram.

[0093] It should be understood that the processor in the computer can be understood as one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components implemented to execute the aforementioned computer instructions.

[0094] Computer instructions can be executed completely on the user's local computer, partially on the user's local computer, as a separate software package, partially on the user's local computer and partially on a remote computer, or completely on a remote computer or server. It should also be noted that in some alternative embodiments, the functions noted in each step in the flow chart or each block in the block diagram may not occur in the order noted in the figure. For example, depending on the functions involved, two steps or two blocks shown in succession may actually be executed roughly simultaneously, or these blocks may sometimes be executed in reverse order.

[0095] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for operating and maintaining IoT devices based on large model fine-tuning and multi-modal RAG, characterized in that: include: Step S1, collecting external image information and data information for the device to be diagnosed; Step S2, generating question information according to the external image information and the data information and inputting the question information into the IoT big model to obtain feedback information fed back by the IoT big model; Step S3: determining whether the device to be diagnosed has an abnormality according to the feedback information.

2. The IoT device operation and maintenance method according to claim 1, characterized in that: The step S1 comprises: Step S11, photographing an external image of the device to be diagnosed, segmenting the external image of the device to be diagnosed to obtain a plurality of mark inspection areas, and collecting the data information of the device to be diagnosed; Step S12, classifying the mark inspection area and outputting image description information for describing the mark inspection area; Step S13: construct the external image information according to the image description information and the landmark inspection area.

3. The IoT device operation and maintenance method according to claim 1, characterized in that: The step S12 comprises: Step S121, identifying the regional image and pixel position of the mark inspection area to determine the inspection position corresponding to the mark inspection area; Step S122, retrieving a sample image according to the inspection position, and matching the sample image with the regional image to obtain a classification result; Step S123: determining the image description information according to the classification result.

4. The IoT device operation and maintenance method according to claim 3, characterized in that: After step S122 and before executing step S123, the mark inspection area without problems is eliminated according to the classification result, and then the image description information is generated again.

5. The IoT device operation and maintenance method according to claim 1, characterized in that: In the step S1, the data information includes the operator to which the device to be diagnosed belongs, the base station location, the base station type, and the base station number.

6. The IoT device operation and maintenance method according to claim 1, characterized in that: Before executing step S1, a process of determining a device to be diagnosed is also included, and the process of determining a device to be diagnosed includes: Step S101, when a user feedback work order is triggered, a question-answer model is used to collect user input content, and network connection information of the user's device is obtained; Step S102, determining an associated base station of the user equipment according to the user input content and the network connection information; Step S103: taking the associated base station as the device to be diagnosed.

7. The IoT device operation and maintenance method according to claim 1, characterized in that: Step S2 includes: Step S21, constructing an input vector as the question information according to the external image information and the data information; Step S22: input the question information into the IoT big model to obtain the feedback information.

8. The IoT device operation and maintenance method according to claim 1, characterized in that: The step S21 comprises: Step S211, performing feature extraction on the external image information to obtain image features, and performing vector embedding on the data information to obtain an embedding vector; Step S212: construct the input vector according to the embedding vector and the image feature.

9. The IoT device operation and maintenance method according to claim 1, characterized in that: Before executing step S1, a model fine-tuning process is also included, and the model fine-tuning process includes: Step S201, collecting operation and maintenance sample data corresponding to the device to be diagnosed; Step S202, constructing a training corpus using the operation and maintenance sample data; Step S203: Use the training corpus to train the general big model to obtain the Internet of Things big model.

10. A memory comprising computer instructions, characterized in that: When the computer device reads and executes the computer instructions, the Internet of Things device operation and maintenance method as described in any one of claims 1 to 9 is executed.

Images