Large-model and multi-mode RAG user equipment fault positioning method and memory

Abstract

The invention relates to the technical field of Internet of Things, in particular to a large-model and multi-mode RAG user equipment fault positioning method and storage.The method comprises the steps that when fault feedback information of user equipment is received, a graphical interface display image of the user equipment is obtained; extracting a target graphic screenshot from the graphic interface display image; and inputting a diagnosis large model according to the target graphic screenshot to obtain feedback content corresponding to a fault type. Aiming at the problem that an operation and maintenance system in the prior art is easy to diagnose fault information due to unclear description in a communication process, in the scheme, a graphical interface display image and a specific UI icon of user equipment are introduced as main diagnosis information, and meanwhile, a large diagnosis model is adopted to analyze the interface icon, so that the diagnosis efficiency is improved. Therefore, effective diagnosis of the fault information is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a method and a memory for locating faults of user equipment of a large model and multi-modal RAG. Background Art

[0002] The Internet of Things refers to a network of physical devices, vehicles, appliances, and other physical objects that are embedded with sensors, software, and network connections that can collect and share data. IoT devices range from simple "smart home" devices such as smart thermostats, wearable devices such as smart watches and RFID-enabled clothing, to complex industrial machinery and transportation systems. Technology experts are even envisioning entire "smart cities" based on IoT technology. The Internet of Things enables these smart devices to communicate with each other and other Internet-enabled devices. Just like smartphones and gateways, a large network of interconnected devices is created that can exchange data and perform various tasks autonomously. Among them, the Internet of Vehicles, that is, IoT technology for automotive platform applications, is a relatively common application area. Retrieval-augmented Generation (RAG) is one of the hottest large model frontier technologies. The retrieval-augmented generation model combines language models and information retrieval technology. Specifically, when the model needs to generate text or answer questions, it first retrieves relevant information from a large collection of documents, and then uses this retrieved information to guide the generation of text, thereby improving the quality and accuracy of predictions.

[0003] In the prior art, in order to enable the user-side devices in the IoT devices to achieve the predetermined design functions, it is usually necessary to design operation and maintenance and after-sales links to help users troubleshoot equipment failures. For example, Chinese patent CN201810837086.1 discloses an automated operation and maintenance management system, including: an instant messaging module for receiving user-side information from a mobile communication device; a semantic analysis module for parsing user-side information to obtain semantic information and matching operation and maintenance commands corresponding to the semantic information; a result feedback module for feedback on the results of executing the operation and maintenance commands; wherein the instant messaging module, the semantic analysis module and the result feedback module constitute data interaction; the user-side information is input by the user on the mobile communication device. The benefit is that a method is provided for users to implement automated operation and maintenance management through natural language on a mobile terminal.

[0004] However, during the actual implementation process, the inventors found that the realization process of this type of technical solution mainly relies on the communication between users and after-sales operation and maintenance personnel. When users cannot correctly describe the fault information, it is easy to cause a decrease in diagnostic accuracy. Summary of the invention

[0005] In view of the above problems existing in the prior art, a method and storage for locating user equipment faults of a large model and multi-modal RAG are provided.

[0006] The specific technical solutions are as follows:

[0007] A method for locating user equipment faults in a large model and multi-modal RAG, comprising:

[0008] Step S1, when receiving fault feedback information of a user equipment, obtaining a graphical interface display image of the user equipment;

[0009] Step S2, extracting a target graphic screenshot from the graphical interface display image;

[0010] Step S3, inputting the target graphic screenshot into the diagnosis model to obtain feedback content corresponding to the fault type.

[0011] On the other hand, the step S1 comprises:

[0012] Step S11, when the fault feedback information is received, a question-answer model is used to prompt the user to take a picture of the display interface of the user device to obtain a real-time display image;

[0013] Step S12, collecting the screen area in the real-time display image to obtain a screen image;

[0014] Step S13, desensitizing the screen image to obtain the graphical interface display image.

[0015] On the other hand, the step S12 comprises:

[0016] Step S121, acquiring a screen lighting area in the real-time display image;

[0017] Step S122, using a pre-configured screen image template to match the screen lighting area to obtain a screen matching template;

[0018] Step S123: using the screen matching template to intercept and correct the real-time display image to obtain the screen image.

[0019] On the other hand, in the step S122, the corresponding screen image template is retrieved according to the model of the user equipment.

[0020] On the other hand, the step S2 comprises:

[0021] Step S21, locating a status bar area in the graphical interface display image;

[0022] Step S22, searching and obtaining an icon area from the status bar area using an icon template;

[0023] Step S23: intercepting the graphical interface display image based on the icon area to obtain the target graphic screenshot.

[0024] On the other hand, the step S3 comprises:

[0025] Step S31, processing the target graphic screenshot to obtain graphic description information corresponding to the target graphic screenshot;

[0026] Step S32: input the graphic description information and the target graphic screenshot into the diagnosis macro model to obtain the feedback content.

[0027] On the other hand, the step S31 comprises:

[0028] Step S311, segmenting the target graphic screenshot to obtain segmented icons;

[0029] Step S312: performing template matching on the segmented icons to obtain the graphic description information corresponding to each segmented icon.

[0030] On the other hand, after executing step S3, the method further includes:

[0031] Step S4, returning the feedback content to the user equipment to instruct the user to adjust the user equipment, and collecting new fault feedback information, and then returning to step S1 until the fault is eliminated.

[0032] On the other hand, before executing step S1, a model fine-tuning process is also included, and the model fine-tuning process includes:

[0033] Step S01, collecting fault diagnosis data corresponding to the user equipment;

[0034] Step S02, constructing a training corpus using the fault diagnosis data;

[0035] Step S03: Use the training corpus to train the general large model to obtain the diagnosis large model.

[0036] A memory includes computer instructions. When a computer device reads and executes the computer instructions, the above-mentioned user equipment fault locating method is executed.

[0037] The above technical solution has the following advantages or beneficial effects:

[0038] In order to solve the problem that the operation and maintenance system in the prior art is prone to being unable to diagnose fault information due to unclear descriptions during the communication process, this solution introduces the graphical interface display image and specific UI icons of the user device as the main diagnostic information. At the same time, a large diagnostic model is used to analyze the interface icons, thereby achieving effective diagnosis of fault information. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The embodiments of the present invention will be described more fully with reference to the attached drawings, which are provided for illustration and description only and are not intended to limit the scope of the present invention.

[0040] Figure 1 is an overall schematic diagram of an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of step S1 of an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of step S12 of an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of step S2 of an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of step S3 of an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of step S31 of an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of step S4 of an embodiment of the present invention;

[0047] Figure 8 Schematic diagram of the model fine-tuning process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0050] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0051] The present invention comprises:

[0052] A method for locating user equipment faults based on a large model and multi-modal RAG, such as Figure 1 As shown, including:

[0053] Step S1, when receiving fault feedback information of the user equipment, obtaining a graphical interface display image of the user equipment;

[0054] Step S2, extracting a target graphic screenshot from the graphical interface display image;

[0055] Step S3, inputting the diagnostic model according to the target graphic screenshot to obtain feedback content corresponding to the fault type.

[0056] Specifically, in order to address the problem that the operation and maintenance system in the prior art is prone to being unable to diagnose fault information due to unclear descriptions during the communication process, in this embodiment, a graphical interface display image and specific UI icons of the user device are introduced as the main diagnostic information. At the same time, a large diagnostic model is used to analyze the interface icons, thereby achieving effective diagnosis of fault information.

[0057] During implementation, the above-mentioned user equipment fault location method is mainly configured as a software implementation in a computer system, such as a user equipment operation and maintenance system, which receives external input operation and maintenance work orders and performs diagnostic analysis and feedback to the user on the corresponding troubleshooting method.

[0058] In one embodiment, Figure 2 As shown, step S1 includes:

[0059] Step S11, when fault feedback information is received, a question-answer model is used to prompt the user to take a picture of the display interface of the user device to obtain a real-time display image;

[0060] Step S12, collecting the screen area in the real-time display image to obtain a screen image;

[0061] Step S13, desensitizing the screen image to obtain a graphical interface display image.

[0062] Specifically, in order to achieve better information collection effects and avoid the problem of inaccurate descriptions during user input, in this embodiment, for fault feedback information, a question-and-answer model is first used to interact with the user to instruct the user to open the corresponding permissions and use other devices to shoot the display interface of the user's device to obtain a real-time display image. This part of the real-time display image is an external image displayed by the user's device through the display screen. Then, the screen area in the real-time display image is collected, and irrelevant information in the background is removed to obtain a screen image. At the same time, considering that the screen image also includes some user personal information, it is necessary to cover the application display interface in the screen image to achieve a desensitization effect, and finally obtain a graphical interface display image.

[0063] In one embodiment, Figure 3 As shown, step S12 includes:

[0064] Step S121, obtaining a screen lighting area in the real-time display image;

[0065] Step S122, using a pre-configured screen image template to match the screen lighting area to obtain a screen matching template;

[0066] Step S123, using a screen matching template to intercept and correct the real-time display image to obtain a screen image;

[0067] In step S122, the corresponding screen image template is retrieved according to the model of the user equipment.

[0068] Specifically, in order to achieve a better recognition effect of the screen area, in this embodiment, the screen lighting area is first obtained from the real-time display image by grayscale value detection. Since the illumination of the screen area itself comes from the light emitted by the backlight module or device, there will be a certain difference in grayscale relative to the ambient light, such as extremely bright in the dark or extremely dark in the sun, so the screen lighting area is easily obtained by grayscale value matching. Then, the screen lighting area is matched with a pre-configured screen image template to obtain a screen matching template. The screen image template is determined according to the metadata of the user device, such as the user device is a specific model of mobile phone, car equipment, etc., and the aspect ratio of the screen is easy to obtain. The screen lighting area is matched according to the aspect ratio, so as to calculate the approximate bevel size to obtain the corresponding screen matching template. On this basis, the screen matching template is used to intercept the real-time display image, and the perspective, tilt, etc. are corrected to obtain the screen image.

[0069] In one embodiment, Figure 4 As shown, step S2 includes:

[0070] Step S21, locating a status bar area in the graphical interface display image;

[0071] Step S22, searching for an icon area in the status bar area using an icon template;

[0072] Step S23: intercepting the graphical interface display image based on the icon area to obtain a target graphic screenshot.

[0073] Specifically, in order to achieve a better matching effect, in this embodiment, after obtaining the graphical interface display image, the height pixels in the image are converted into relative coordinate values, and then the edge of the upper status bar area is determined according to the proportion of the relative coordinate values ​​and the change of the color value, so as to locate and segment the status bar area in the graphical interface display image. Subsequently, for the status bar area, the icon template corresponding to different UI icons is used to search the status bar area, so as to obtain the icon area corresponding to the position of the icon. Finally, multiple icons in the icon area are identified and intercepted to finally determine the target graphic screenshot associated with the status diagnosis.

[0074] In one embodiment, Figure 5 As shown, step S3 includes:

[0075] Step S31, processing the target graphic screenshot to obtain graphic description information corresponding to the target graphic screenshot;

[0076] Step S32, inputting the graphic description information and the target graphic screenshot into the diagnosis macro model to obtain feedback content.

[0077] Specifically, in order to achieve better input effects, in this embodiment, the target graphic screenshot is first classified according to different icon status types, such as high, medium, low, no signal, etc. of the network signal strength icon, and then graphic description information corresponding to the target graphic screenshot is generated according to the classification results. Subsequently, the graphic description information and the target graphic screenshot are input into the diagnostic model together to obtain feedback content.

[0078] In one embodiment, Figure 6 As shown, step S31 includes:

[0079] Step S311, segmenting the target graphic screenshot to obtain segmented icons;

[0080] Step S312: Perform template matching on the segmented icons to obtain graphic description information corresponding to each segmented icon.

[0081] Specifically, in order to achieve a better description effect, in this embodiment, the target graphic screenshot corresponding to the entire icon area is first segmented to obtain segmented icons, and then template matching is performed on each segmented icon to obtain graphic description information corresponding to each segmented icon, including the upper left, lower left, upper right or lower right corner of the car screen, icons appearing in the form of triangles, continuously increasing bar graphs or continuous dots, and the corresponding icon content, and then the graphic description information is generated.

[0082] In one embodiment, Figure 7 As shown, after executing step S3, the method further includes:

[0083] Step S4, returning the feedback content to the user equipment to instruct the user to adjust the user equipment, and collecting new fault feedback information, and then returning to step S1 until the fault is eliminated.

[0084] In one embodiment, Figure 8 As shown, before executing step S1, a model fine-tuning process is also included, and the model fine-tuning process includes:

[0085] Step S01, collecting fault diagnosis data corresponding to user equipment;

[0086] Step S02, constructing a training corpus using fault diagnosis data;

[0087] Step S03, using training corpus to train the general large model to obtain a diagnostic large model.

[0088] Specifically, in order to achieve a better diagnosis effect on the equipment status, in this embodiment, before the actual inspection begins, a fine-tuning process for the large model is also introduced. Specifically, for various types of user equipment that may be involved in the after-sales process, fault diagnosis data corresponding to the user equipment is collected in advance, including equipment appearance pictures, equipment parameters, diagnosis results and fault information. Subsequently, a training corpus is constructed based on the operation and maintenance sample data, including assembling the above information according to a specific template to form an input vector. The training corpus is then used to train the general large model, and the model is fine-tuned to obtain the Internet of Things large model to achieve better diagnostic effects.

[0089] A memory includes computer instructions. When a computer device reads and executes the computer instructions, the above-mentioned Internet of Things device operation and maintenance method is executed.

[0090] Those skilled in the art will appreciate that various aspects of the present invention, or possible implementations of various aspects, may be specifically implemented as systems, methods, or computer program products. Therefore, various aspects of the present invention, or possible implementations of various aspects, may take the form of complete hardware embodiments, complete software embodiments (including firmware, resident software, etc.), or embodiments of combined software and hardware aspects, all collectively referred to herein as "circuits," "modules," or "systems." In addition, various aspects of the present invention, or possible implementations of various aspects, may take the form of computer program products, which refer to computer instructions stored in a memory.

[0091] The memory may be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable read-only memory (CD-ROM).

[0092] The processor in the computer reads the computer instructions stored in the memory, so that the processor can execute the functional actions specified in each step or the combination of steps in the flowchart; and generate a device for implementing the functional actions specified in each block or the combination of blocks in the block diagram.

[0093] It should be understood that the processor in the computer can be understood as one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components implemented to execute the aforementioned computer instructions.

[0094] Computer instructions can be executed completely on the user's local computer, partially on the user's local computer, as a separate software package, partially on the user's local computer and partially on a remote computer, or completely on a remote computer or server. It should also be noted that in some alternative embodiments, the functions noted in each step in the flow chart or each block in the block diagram may not occur in the order noted in the figure. For example, depending on the functions involved, two steps or two blocks shown in succession may actually be executed roughly simultaneously, or these blocks may sometimes be executed in reverse order.

[0095] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for locating user equipment faults in a large model and multi-modal RAG, characterized in that: include: Step S1, when receiving fault feedback information of a user equipment, obtaining a graphical interface display image of the user equipment; Step S2, extracting a target graphic screenshot from the graphical interface display image; Step S3, inputting the target graphic screenshot into the diagnosis model to obtain feedback content corresponding to the fault type.

2. The method for locating a fault of a user equipment according to claim 1, characterized in that: The step S1 comprises: Step S11, when the fault feedback information is received, a question-answer model is used to prompt the user to take a picture of the display interface of the user device to obtain a real-time display image; Step S12, collecting the screen area in the real-time display image to obtain a screen image; Step S13, desensitizing the screen image to obtain the graphical interface display image.

3. The method for locating a fault of a user equipment according to claim 2, characterized in that: The step S12 comprises: Step S121, acquiring a screen lighting area in the real-time display image; Step S122, using a pre-configured screen image template to match the screen lighting area to obtain a screen matching template; Step S123: using the screen matching template to intercept and correct the real-time display image to obtain the screen image.

4. The method for locating a fault of a user equipment according to claim 3, characterized in that: In the step S122, the corresponding screen image template is retrieved according to the model of the user equipment.

5. The method for locating a fault of a user equipment according to claim 1, characterized in that: The step S2 comprises: Step S21, locating a status bar area in the graphical interface display image; Step S22, searching and obtaining an icon area from the status bar area using an icon template; Step S23: intercepting the graphical interface display image based on the icon area to obtain the target graphic screenshot.

6. The method for locating a fault of a user equipment according to claim 1, characterized in that: The step S3 comprises: Step S31, processing the target graphic screenshot to obtain graphic description information corresponding to the target graphic screenshot; Step S32: input the graphic description information and the target graphic screenshot into the diagnosis macro model to obtain the feedback content.

7. The method for locating a fault of a user equipment according to claim 6, characterized in that: The step S31 comprises: Step S311, segmenting the target graphic screenshot to obtain segmented icons; Step S312: performing template matching on the segmented icons to obtain the graphic description information corresponding to each segmented icon.

8. The method for locating a fault of a user equipment according to claim 1, characterized in that: After executing step S3, the method further includes: Step S4, returning the feedback content to the user equipment to instruct the user to adjust the user equipment, and collecting new fault feedback information, and then returning to step S1 until the fault is eliminated.

9. The method for locating a fault of a user equipment according to claim 1, characterized in that: Before executing step S1, a model fine-tuning process is also included, and the model fine-tuning process includes: Step S01, collecting fault diagnosis data corresponding to the user equipment; Step S02, constructing a training corpus using the fault diagnosis data; Step S03: Use the training corpus to train the general large model to obtain the diagnosis large model.

10. A memory comprising computer instructions, characterized in that: When the computer device reads and executes the computer instruction, the user equipment fault locating method according to any one of claims 1 to 9 is executed.

Images