Visual troubleshooting method and system

Abstract

The present application provides a visual troubleshooting method and system, the visual troubleshooting method includes: obtaining troubleshooting operations corresponding to the faulty device according to the device information and fault information of the faulty device; generating troubleshooting visualization components for the faulty device according to the troubleshooting operations; fusing the troubleshooting visualization components with the real device image corresponding to the faulty device to generate a mixed reality image, the mixed reality image is used to display the troubleshooting operations on the real device image corresponding to the faulty device; displaying the mixed reality image on a preset terminal to guide the user to troubleshoot. The method provided by the present application can visually guide the user to troubleshoot, and improve the convenience of the user's self-service troubleshooting.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a visual troubleshooting method and a visual troubleshooting system. Background Art

[0002] Visualization refers to the use of computer graphics and image processing technology to centrally convert data into images or graphics and display them on the screen.

[0003] At present, the application rate of broadband and smart home is constantly increasing. If the current broadband and smart home fail during use, a hotline or Internet page is usually used to trigger the fault diagnosis system for fault diagnosis, and simple online processing methods such as resetting data and remote restart are used to troubleshoot. Obviously, this simple fault handling is no longer sufficient to support complex broadband and smart home failures.

[0004] For example, currently the backend system can only determine whether the device is online and whether it meets the speed requirements. Once the device connection is abnormal, the diagnosis will fail or be interrupted. When the device is self-troubleshooting, if the remote restart fails, it is necessary to rely on manual operation by the customer or on-site engineer visits. In fact, there are not many fault scenarios that can be handled by self-troubleshooting, and the customer experience perception is lacking.

[0005] Therefore, how to improve the convenience of customer self-service troubleshooting has always been a technical problem that technical personnel in this field have been committed to solving.

[0006] The above information disclosed in the Background section is only for enhancement of understanding of the background of the present application and therefore it may contain information that does not constitute the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0007] One purpose of the present application is to provide a visual troubleshooting method that can improve the convenience of customer self-service troubleshooting.

[0008] Another object of the present application is to provide a method for visual troubleshooting using a simple intelligent terminal.

[0009] The third object of the present application is to provide a visual troubleshooting system that can improve the convenience of customer self-service troubleshooting.

[0010] In order to solve the above technical problems, this application adopts the following technical solutions:

[0011] According to one aspect of the present application, the present application provides a visual troubleshooting method, including: obtaining troubleshooting operations corresponding to a faulty device based on device information and fault information of the faulty device; generating a troubleshooting visualization component for the faulty device based on the troubleshooting operations, the troubleshooting visualization component being used to guide users in troubleshooting; fusing the troubleshooting visualization component with a real device image corresponding to the faulty device to generate a mixed reality image, the mixed reality image being used to display the troubleshooting operations on the real device image corresponding to the faulty device; and displaying the mixed reality image on a preset terminal.

[0012] In some embodiments, before obtaining the faulty device information, the method further includes: calling a camera to capture an image of the faulty device, and determining the device information based on the image of the faulty device. The image of the faulty device can be captured using a smart terminal with a camera device, without being restricted by proprietary devices.

[0013] In some embodiments, before obtaining the troubleshooting operation corresponding to the faulty device based on the device information and fault information of the faulty device, the method also includes: obtaining the device information of the faulty device; performing fault pre-diagnosis on the faulty device based on the device information; and determining the fault information of the faulty device based on the result of the fault pre-diagnosis.

[0014] In some embodiments, based on the device information and fault information of the faulty device, obtaining troubleshooting operations corresponding to the faulty device includes: based on the device information and fault information of the faulty device, querying in a device fault database to obtain troubleshooting operations corresponding to the faulty device, wherein the device fault database has troubleshooting operations corresponding to the device information, fault information, and a combination of the device information and fault information.

[0015] In some embodiments, generating a troubleshooting visualization component for a faulty device based on a troubleshooting operation includes: acquiring one or more feedback screens of the faulty device in response to the troubleshooting operation; and generating a troubleshooting visualization component for the faulty device based on the feedback screens.

[0016] In some embodiments, the device fault database contains interaction details of troubleshooting operations on the faulty device; obtaining a feedback screen of the faulty device in response to the troubleshooting operation includes: obtaining interaction details of the faulty device in response to the troubleshooting operation from the device fault database; and generating a feedback screen of the faulty device in response to the troubleshooting operation based on the interaction details.

[0017] In some embodiments, before obtaining troubleshooting operations corresponding to the faulty device based on the device information and fault information of the faulty device, the method also includes: triggering a camera to photograph the faulty device; extracting features of the faulty device in the photographed image; and obtaining device information of the faulty device based on the features.

[0018] In some embodiments, obtaining device information of the faulty device according to the features includes: determining the device information of the faulty device from a device library, wherein the device library has a correspondence between the features of the device and the device information.

[0019] In some embodiments, a troubleshooting visualization component is fused with a real device image corresponding to the faulty device to generate a mixed reality image, including: performing correlation analysis, spatial analysis, and multi-dimensional analysis on the real device image and the troubleshooting visualization component to generate a mixed reality image.

[0020] The present application also provides a visual troubleshooting system, including:

[0021] A troubleshooting operation acquisition module is used to acquire the troubleshooting operation corresponding to the faulty device according to the device information and fault information of the faulty device;

[0022] A visualization component generation module is used to generate a troubleshooting visualization component for a faulty device according to troubleshooting operations, and the troubleshooting visualization component is used to guide users in troubleshooting;

[0023] A mixed reality image generation module, used to fuse the troubleshooting visualization component with the real device image corresponding to the faulty device to generate a mixed reality image, and the mixed reality image is used to display the troubleshooting operation on the real device image corresponding to the faulty device;

[0024] The display module is used to display the mixed reality image on a preset terminal.

[0025] In some embodiments, the system also includes: a shooting module, used to shoot the faulty device to obtain a real image of the faulty device; an image processing module, used to process the real device image of the faulty device to obtain the characteristics of the faulty device; and a device information acquisition module, used to obtain device information corresponding to the faulty device based on the characteristics.

[0026] The present application also provides a smart terminal, which includes at least a camera and a control component, wherein the control component is configured to implement: obtaining troubleshooting operations corresponding to a faulty device based on device information and fault information of the faulty device; generating a troubleshooting visualization component for the faulty device based on the troubleshooting operations, wherein the troubleshooting visualization component is used to guide users in troubleshooting; fusing the troubleshooting visualization component with a real device image corresponding to the faulty device to generate a mixed reality image, wherein the mixed reality image is used to display the troubleshooting operations on the real device image corresponding to the faulty device; and displaying the mixed reality image on a preset terminal; the camera is configured to implement: acquiring an image of the faulty device so that the control component determines device information based on the image of the faulty device.

[0027] It can be seen from the above technical solution that the beneficial effects of this application are:

[0028] In the present application, a troubleshooting visualization component for a faulty device is generated, and the troubleshooting visualization component is merged with a real device image corresponding to the faulty device to generate a mixed reality image, so as to display the troubleshooting operation on the real device image corresponding to the faulty device in the mixed reality image, so as to guide the user to visualize the troubleshooting, thereby improving the convenience of self-service troubleshooting.

[0029] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other objects, features and advantages of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

[0031] Figure 1 1. A schematic system architecture 100 is shown to which a visual troubleshooting method according to an example embodiment may be applied;

[0032] Figure 2 is a flowchart of a visual troubleshooting method according to an example implementation;

[0033] Figure 3 is a flowchart of a visual troubleshooting method according to an example implementation;

[0034] Figure 4 is a flowchart of a specific embodiment of the visual troubleshooting method according to the present application;

[0035] Figure 5 is a schematic diagram showing the structure of a visual troubleshooting system of the present application according to an example implementation;

[0036] Figure 6 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0037] Although the present application can be easily embodied in different forms of embodiments, only some of the specific embodiments are shown in the drawings and described in detail in this specification. It should be understood that this description should be regarded as an exemplary illustration of the principles of the present application and is not intended to limit the present application to that described herein.

[0038] Thus, a feature indicated in this specification will be used to illustrate one of the features of an embodiment of the present application, rather than implying that each embodiment of the present application must have the described feature. In addition, it should be noted that this specification describes many features. Although some features can be combined together to illustrate possible system designs, these features can also be used in other combinations that are not explicitly described. Thus, unless otherwise stated, the described combinations are not intended to be limiting.

[0039] In the embodiments shown in the drawings, the indications of directions (such as up, down, left, right, front and back) used to explain the structure and movement of various elements of the present application are not absolute but relative. These descriptions are appropriate when these elements are in the positions shown in the drawings. If the description of the positions of these elements changes, the indications of these directions also change accordingly.

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present application will be more comprehensive and complete and the concepts of the example embodiments will be fully conveyed to those skilled in the art. The accompanying drawings are only schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0041] The preferred implementation methods of the present application are further described in detail below in conjunction with the drawings of this specification.

[0042] Figure 1 1 shows a schematic system architecture 100 to which a visual troubleshooting method according to an example embodiment can be applied. Figure 1 As shown, the system architecture 100 may include terminal devices 101 , 102 , 103 , a network 104 and a server 105 .

[0043] The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0044] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0045] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0046] The server 105 can provide various services, such as a backend server that supports the web page content displayed on the terminal devices 101, 102, and 103. The backend server can analyze and process the page content according to the received user settings, and feed back the processing results (such as the web page changed according to the user settings) to the terminal device.

[0047] It should be noted that the visual troubleshooting method provided in the embodiment of the present application is generally executed by the terminal devices 101 , 102 , and 103 , and correspondingly, the visual troubleshooting system can be set in the terminal devices 101 , 102 , and 103 .

[0048] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0049] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0050] In the related technologies of this application, the self-service troubleshooting method is usually: when a user encounters a problem with broadband or smart home products, he or she calls the service hotline or reports the problem through the WeChat applet, and the system triggers fault diagnosis and online preprocessing and obtains the execution results. After receiving the results, the operation and maintenance support system displays the diagnosis report and execution result report through the contact points, and provides guidance on online troubleshooting. However, this method is too complicated, and there are not many fault scenarios that can be handled by actual self-service troubleshooting, and the customer experience perception is lacking.

[0051] The purpose of this application is to provide a visual troubleshooting method to improve the convenience of customer self-service troubleshooting.

[0052] Figure 2 FIG. 1 is a flowchart of a visual troubleshooting method according to an example embodiment. Figure 2 As shown, in this embodiment, the visual troubleshooting method at least includes the following steps S210 to S270.

[0053] Step S210: Obtain troubleshooting operations corresponding to the faulty device according to the device information and fault information of the faulty device.

[0054] Faulty equipment is terminal equipment that has lost or reduced its specified function due to a fault.

[0055] The device information includes the device model and terminal device number of the terminal device that caused the failure. The device information may also include intelligent operation instructions corresponding to each device. In an exemplary manner, when the faulty device is a home gateway of a certain model, it may also include intelligent operation instructions for triggering the detection function of the home gateway, collecting diagnostic data and uploading it to a dedicated platform for analysis.

[0056] In one embodiment, the user can self-service and observe the device information of the faulty device and access and input the device information based on the browser or application running on the terminal device to obtain the device information of the faulty device. The user can also automatically obtain the device information of the faulty device based on the browser or application running on the terminal device to run a plug-in for obtaining device information. In another embodiment, the user can also trigger the camera of the terminal device to shoot the faulty device to extract the appearance, label and other information of the faulty device in the shot image, and then obtain the device information of the faulty device.

[0057] Fault information includes the type (code) of the fault generated by the faulty device, the manifestation of the fault, and other information. Specifically, for more complex faults, users can obtain fault information by obtaining device logs based on the browser running on the terminal device or the corresponding plug-in running in the application. For simpler faults, users can also observe and describe the fault information by themselves, so that the backend operation and maintenance personnel can further locate the fault based on the feedback fault information. When a fault occurs, the faulty device can also actively report the fault information to prevent the user from having a low utilization rate of fault diagnosis due to lack of relevant knowledge, and can also greatly reduce the difficulty of user self-service troubleshooting.

[0058] The troubleshooting operation is an operation process for eliminating the fault so that the terminal device can work normally. In one embodiment, according to the obtained device information and fault information, a query can be made in the knowledge base such as the device manual, fault cases and corresponding solution processes to obtain the required troubleshooting operation.

[0059] The knowledge base used may be an equipment fault database, wherein the equipment fault database contains equipment information, fault information, and troubleshooting operations corresponding to a combination of the equipment information and the fault information.

[0060] In one embodiment, the device fault database may include a device library and an instance library.

[0061] The device library is used to store the device information of several devices, which may include device identification, device model, and device intelligent operation instructions, etc., so as to identify the faulty device using the stored device information. The instance library can store several fault cases, several device manuals, and repair manuals corresponding to the fault type (code), etc., so as to obtain the troubleshooting operations of the faulty device.

[0062] Specifically, in one embodiment, step S210, according to the device information and fault information of the faulty device, obtaining the troubleshooting operation corresponding to the faulty device may specifically include the following steps:

[0063] Perform pre-diagnosis of the faulty equipment according to its equipment information;

[0064] According to the results of the pre-diagnosis, the fault information of the faulty equipment is determined.

[0065] Among them, pre-diagnosis is the process of finding the cause of the fault, which includes status monitoring, fault cause analysis, and degradation trend prediction. After obtaining the device information corresponding to the device, the corresponding instructions of the pre-diagnosis can be called according to the device identification, device model, and device intelligent operation instructions corresponding to the device stored in the device library, and the pre-diagnosis results can be obtained, thereby obtaining the cause of the fault and the status of the diagnosed object, in other words, obtaining the fault information. Furthermore, according to the obtained fault information and the fault type and other information in the instance library, the troubleshooting operation can be further obtained to facilitate the user's self-service troubleshooting.

[0066] It should be noted that the amount of device information and fault information that can be stored can be massive, so the troubleshooting operations that can be successfully generated by this implementation can also be massive. Therefore, through this solution, the types of obstacles that users can troubleshoot by themselves can be increased, and the troubleshooting success rate can be improved.

[0067] Step S230: Generate a troubleshooting visualization component for the faulty device according to the troubleshooting operation, and the troubleshooting visualization component is used to guide the user in troubleshooting.

[0068] The troubleshooting visualization component is used to present the troubleshooting process in the form of graphics or images, so that users can perceive the troubleshooting guidance operation through vision.

[0069] A troubleshooting visualization component for a faulty device may be generated based on the initial visualization component. The initial visualization component has preset properties, such as size, color, display style, etc. Each device may have a corresponding initial visualization component, which may include an appearance image of the corresponding device.

[0070] It is easy to understand that when a troubleshooting operation is performed on a faulty device, the faulty device will have one or more interaction details in response to the troubleshooting operation. The one or more real-time interaction details form a feedback screen.

[0071] In one embodiment, a troubleshooting visualization component for a faulty device can be generated based on the relationship information and attribute information of the interaction details. The relationship information includes the display order of the interaction details. The attribute information includes the position, color, display style, etc.

[0072] In one embodiment, generating a troubleshooting visualization component for a faulty device according to the troubleshooting operation may specifically include the following steps:

[0073] Obtaining a feedback screen of the faulty device in response to the troubleshooting operation from the device fault database;

[0074] Generate visual troubleshooting components for faulty equipment based on the feedback screen.

[0075] Specifically, the feedback screen is used to display the interactive scene of the current operation. In one embodiment, the feedback screen of the faulty device in response to the troubleshooting operation can be directly or indirectly obtained from the device fault database, which can specifically include the following steps:

[0076] Acquire interaction details of the faulty device in response to the troubleshooting operation from a device fault database, wherein the device fault database contains interaction details corresponding to the faulty device when the troubleshooting operation is performed on the faulty device;

[0077] Based on the interaction details, a feedback screen is generated showing the faulty device responding to the troubleshooting operation.

[0078] Specifically, the feedback result includes one or more interaction details respectively present in one or more interaction scenarios during the troubleshooting operation. In other words, the interaction details are information that needs to be fed back to the user.

[0079] After troubleshooting the faulty device, the interaction details can be fed back visually and / or aurally for a certain duration. Indicatively, an image corresponding to the interaction details can be directly displayed on the page of the terminal device. It should be noted that the frequency of updating the feedback screen needs to be reduced to reduce disturbance to the user and give the user sufficient time to operate the faulty device. Furthermore, based on the feedback screen, a troubleshooting visualization component for the faulty device is generated to feed back the determined feedback information to the customer through the feedback method used.

[0080] Step S250, merging the troubleshooting visualization component with the real device image corresponding to the faulty device to generate a mixed reality image, and the mixed reality image is used to display the troubleshooting operation on the real device image corresponding to the faulty device.

[0081] The camera of the terminal device can be used to photograph the faulty device, and a device image model can be established to generate a real device image corresponding to the faulty device based on the device image model, and then merged with the troubleshooting visualization component to finally generate a mixed reality image. Specifically, one or more interactive detail images on the feedback screen can be used as foreground images, and the real device image corresponding to the faulty device can be used as the background image. After point-to-point fusion of the images, a mixed reality image is generated. As a result, the interactive details related to the troubleshooting operation can be displayed on the real device image corresponding to the faulty device, so that the user can intuitively see the feedback image and operation process corresponding to the troubleshooting operation, thereby improving the convenience of self-service troubleshooting, so that users can self-repair common usage obstacles. In one embodiment, the interactive details displayed as the foreground image can be used in a variety of ways such as pictures or text to display the troubleshooting effect.

[0082] In one embodiment, the troubleshooting visualization component is fused with the real device image corresponding to the faulty device to generate a mixed reality image. Specifically, the following steps may be included: performing correlation analysis and spatial analysis on the real device image and the troubleshooting visualization component to generate a mixed reality image.

[0083] As mentioned above, the troubleshooting visualization component is used to display the interaction details in response to the troubleshooting operation on the real device image corresponding to the faulty device. In this embodiment, one or more of association analysis, spatial analysis and multidimensional analysis are performed on the real device image and the visualization component to determine the positional relationship between each pixel on the real device image and the interaction details required to be displayed by the visualization component, so as to determine the real-time overlapping part, and output the image in the form of video. Among them, association analysis, also known as association mining, is used to find data correlation or causal structure in relational data or other information carriers. Spatial analysis is used to extract spatial data into different forms and extract its potential information. Multidimensional analysis is used to perform various analysis operations such as rolling up, drilling down, slicing, dicing, and rotating data organized in a multidimensional form in order to analyze the data, so that the analyst can observe the data in the database from multiple angles and multiple sides, so as to gain a deep understanding of the information and connotation contained in the data.

[0084] Step S270: displaying the mixed reality image on a preset terminal.

[0085] The preset terminal is a designated terminal device that the user can directly view. The terminal device can be hardware or software. When the terminal device is hardware, it can be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc., so as to directly present troubleshooting operations to the user and guide the user to conveniently perform troubleshooting operations.

[0086] Therefore, by generating a troubleshooting visualization component for the faulty device and fusing the troubleshooting visualization component with the real device image corresponding to the faulty device to generate a mixed reality image, the troubleshooting operation is displayed on the real device image corresponding to the faulty device in the mixed reality image to guide the user to visualize the troubleshooting, thereby improving the convenience of self-service troubleshooting.

[0087] In one embodiment, on-site scene information of the faulty equipment may also be added to the mixed reality image to enhance user participation and improve the guidance success rate.

[0088] Figure 3 The figure is a flowchart showing a visual troubleshooting method according to an example implementation.

[0089] In step S210, before obtaining the troubleshooting operation corresponding to the faulty device according to the device information and fault information of the faulty device, the method further includes the following steps S211 to S213:

[0090] Step S211, triggering the camera to take a picture of the faulty device;

[0091] Step S212, extracting features of the faulty device in the captured image;

[0092] Step S213: acquiring device information of the faulty device according to the characteristics of the faulty device.

[0093] Specifically, the faulty device can be photographed by a camera on the terminal device, and based on the photographed image, the characteristics of the faulty device in the image can be extracted to determine the device information corresponding to the faulty device according to the characteristics. In one embodiment, the device information of the faulty device is obtained according to the characteristic parameters, including: determining the device information of the faulty device from a device library, wherein the device library has a corresponding relationship between the device and the characteristics corresponding to the device.

[0094] Figure 4 It is a flowchart of a specific embodiment of the visual troubleshooting method according to the present application. In this embodiment, when a smart home terminal fails, the user can use a smart terminal device such as a mobile phone to log in to the self-service troubleshooting system for self-service troubleshooting. The self-service troubleshooting system has at least a device library, an instance library, an interactive component module, and a mixed reality computing module. Among them, the device library is used to store data of device information (device identification, device model, and device intelligent operation instructions, etc.) for identifying faulty devices. The instance library is used for the instance library to mainly store device manuals, fault cases, and repair manuals corresponding to fault codes for obtaining troubleshooting operations. The interactive component module is a visualization model component based on the conversion of interactive details such as device manuals and device operating instructions. The mixed reality computing module is used to generate visual troubleshooting videos. Specifically, such as Figure 4 As shown, the visual troubleshooting method specifically includes the following steps:

[0095] Step S401, the user uses a smart terminal such as a mobile phone or tablet to enter the self-service troubleshooting system;

[0096] Step S402: calling the camera of the smart terminal to collect a picture of the faulty device, and identifying device information of the faulty device from the picture of the faulty device based on the device library;

[0097] Step S403, triggering fault diagnosis to identify the cause of the fault, and generating a diagnosis report including the cause of the fault;

[0098] Step S404, according to the diagnosis report, obtaining troubleshooting operations based on the matching of device information and the instance library;

[0099] Step S405, obtaining troubleshooting videos and / or model information corresponding to the troubleshooting operation based on the interactive component module;

[0100] Step S406, the mixed reality calculation module performs mixed reality calculation based on the device library, the fault instance library, and the information provided by the interactive component module to generate visual video information;

[0101] Step S407, displaying the visual video information generated by the mixed reality computing module through the page of the self-service troubleshooting system to guide the user to perform self-service troubleshooting.

[0102] Specifically, after the user enters the self-service troubleshooting system using a smart terminal such as a mobile phone / tablet, the user can use the camera of the smart terminal to take a picture of the faulty device, and use information such as the device logo on the taken picture to obtain device information of the faulty device from the device library.

[0103] After obtaining the device information, the fault diagnosis corresponding to the device is triggered according to the device information to identify the cause of the fault, and a diagnostic report containing the cause of the fault is generated. Afterwards, based on the fault cases in the instance library and the repair manual corresponding to the fault code, the troubleshooting operation is obtained. The interactive component module is used to generate troubleshooting videos and / or model information corresponding to the troubleshooting operation for the faulty device according to the troubleshooting operation, that is, a troubleshooting visualization component. The mixed reality computing module performs mixed reality computing based on the information provided by the device library, the fault instance library and the interactive component module, thereby generating visual video information to display the video information on the page of the self-service troubleshooting system, so that users can conveniently solve various faults by themselves and improve the success rate of self-service troubleshooting.

[0104] Figure 5 FIG. 1 is a schematic diagram showing the structure of a visual troubleshooting system of the present application according to an exemplary embodiment. The visual troubleshooting system is used to run the aforementioned visual troubleshooting method. Figure 5 As shown, in this embodiment, the visual troubleshooting system includes at least the following modules.

[0105] The troubleshooting operation acquisition module 510 is used to acquire the troubleshooting operation corresponding to the faulty device according to the device information and fault information of the faulty device;

[0106] A visualization component generation module 520 is used to generate a troubleshooting visualization component for a faulty device according to the troubleshooting operation, and the troubleshooting visualization component is used to guide the user to troubleshoot;

[0107] A mixed reality image generation module 530 is used to fuse the troubleshooting visualization component with the real device image corresponding to the faulty device to generate a mixed reality image, and the mixed reality image is used to display the troubleshooting operation on the real device image corresponding to the faulty device;

[0108] The display module 540 is used to display the mixed reality image on a preset terminal.

[0109] In some other embodiments, the visual troubleshooting system may further include the following modules:

[0110] A photographing module 550 is used to photograph the faulty device to obtain a real image of the faulty device;

[0111] An image processing module 560 is used to process the real device image of the faulty device to obtain the characteristics of the faulty device;

[0112] The device information acquisition module 570 is used to acquire device information corresponding to the faulty device according to the characteristics.

[0113] Thus, the user can log in to the visual troubleshooting system through a web page or an application on a terminal device to implement the visual troubleshooting method described above in the system.

[0114] Figure 6 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. Various programs and data required for the operation of the system 600 are also stored in the RAM 603. The CPU 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0116] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0117] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 609, and / or installed from a removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the method defined in the present application is executed.

[0118] The above functions. It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above.

[0119] More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more conductors, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0120] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] As another aspect, the present application further provides a computer-readable medium, which may be included in the device corresponding to the system described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the device:

[0123] Obtain troubleshooting operations corresponding to the faulty device based on the device information and fault information of the faulty device;

[0124] Generate a troubleshooting visualization component for the faulty device based on the troubleshooting operation. The troubleshooting visualization component is used to guide the user in troubleshooting.

[0125] The troubleshooting visualization component is merged with the real device image corresponding to the faulty device to generate a mixed reality image, and the mixed reality image is used to display the troubleshooting operation on the real device image corresponding to the faulty device;

[0126] Display mixed reality images on the preset terminal.

[0127] The present application also provides a smart terminal, which includes at least a camera and a control component. The control component is configured to implement:

[0128] Obtain troubleshooting operations corresponding to the faulty device based on the device information and fault information of the faulty device;

[0129] Generate a troubleshooting visualization component for the faulty device based on the troubleshooting operation. The troubleshooting visualization component is used to guide the user in troubleshooting.

[0130] The troubleshooting visualization component is merged with the real device image corresponding to the faulty device to generate a mixed reality image, and the mixed reality image is used to display the troubleshooting operation on the real device image corresponding to the faulty device;

[0131] Displaying mixed reality images on a preset terminal;

[0132] The camera is configured to achieve: collecting images of the faulty device, so that the control component determines device information based on the images of the faulty device.

[0133] In another embodiment, the image of the faulty device may be collected using a smart terminal with a camera device, without being restricted by proprietary equipment.

[0134] The inventive concept of the above intelligent terminal is consistent with the inventive concept of the above troubleshooting visualization method, and will not be elaborated here.

[0135] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Claims

1. A visual troubleshooting method, characterized in that: include: According to the device information and fault information of the faulty device, a troubleshooting operation corresponding to the faulty device is obtained, where the faulty device is a smart home terminal; Generate a troubleshooting visualization component for the faulty device according to a feedback screen formed by the real-time interaction details of the faulty device when the troubleshooting operation is performed on the faulty device, wherein the troubleshooting visualization component includes a troubleshooting video for guiding a user to troubleshoot, and the troubleshooting visualization component is generated based on the relationship information and attribute information of the real-time interaction details; Using the real-time interaction details in the troubleshooting visualization component as a foreground image and the real device image corresponding to the faulty device as a background image, performing point-to-point fusion on the foreground image and the background image to generate a mixed reality image, wherein the mixed reality image is used to display the troubleshooting operation on the real device image corresponding to the faulty device, and the real device image is generated based on a device image model established for the faulty device; The mixed reality image is displayed in the form of video on a preset terminal.

2. The method according to claim 1, characterized in that Before obtaining a troubleshooting operation corresponding to the faulty device according to the device information and the fault information of the faulty device, the method further includes: Obtaining device information of the faulty device; Based on the device information, pre-diagnose the fault of the faulty device; According to the result of the fault pre-diagnosis, the fault information of the faulty device is determined.

3. The method according to claim 1, characterized in that The obtaining, according to the device information and fault information of the faulty device, a troubleshooting operation corresponding to the faulty device includes: According to the device information and fault information of the faulty device, a query is made in a device fault database to obtain troubleshooting operations corresponding to the faulty device, wherein the device fault database has troubleshooting operations corresponding to the device information, the fault information and a combination of the device information and the fault information.

4. The method according to claim 1, characterized in that: Before obtaining the troubleshooting operation corresponding to the faulty device according to the device information and fault information of the faulty device, the method further includes: Triggering a camera to take a photo of the faulty device; Extracting features of the faulty device in the captured image; According to the feature, device information of the faulty device is obtained.

5. The method according to claim 4, characterized in that The acquiring device information of the faulty device according to the feature includes: The device information of the faulty device is determined from a device library, wherein the device library contains a correspondence between the characteristics of the device and the device information.

6. The method according to claim 1, characterized in that The step of fusing the troubleshooting visualization component with a real device image corresponding to the faulty device to generate a mixed reality image includes: One or more of association analysis, spatial analysis and multi-dimensional analysis are performed on the real device image and the troubleshooting visualization component to generate a mixed reality image.

7. A visual troubleshooting system, characterized in that: include: A troubleshooting operation acquisition module, used to acquire a troubleshooting operation corresponding to the faulty device according to the device information and fault information of the faulty device, wherein the faulty device is a smart home terminal; A visualization component generation module, configured to generate a troubleshooting visualization component for the faulty device according to a feedback screen formed by real-time interaction details of the faulty device when the troubleshooting operation is performed on the faulty device, wherein the troubleshooting visualization component includes a troubleshooting video for guiding a user to troubleshoot, and the troubleshooting visualization component is generated based on relationship information and attribute information of the real-time interaction details; A mixed reality image generation module, used to use the real-time interaction details in the troubleshooting visualization component as a foreground image and the real device image corresponding to the faulty device as a background image, and to perform point-to-point fusion of the foreground image and the background image to generate a mixed reality image, wherein the mixed reality image is used to display the troubleshooting operation on the real device image corresponding to the faulty device, and the real device image is generated based on a device image model established for the faulty device; The display module is used to display the mixed reality image in the form of video on a preset terminal.

8. The system according to claim 7, characterized in that The system further comprises: A shooting module, used for shooting the faulty device to obtain a real image of the faulty device; An image processing module, used for processing a real device image of the faulty device to obtain features of the faulty device; The device information acquisition module is used to acquire device information corresponding to the faulty device according to the feature.

Images