MR-based image forming equipment maintenance control system and method, and server system

Abstract

According to the MR-based image forming equipment maintenance control system and method and the server system provided by the embodiment of the invention, the maintenance indication information associated with the identification information and the virtual three-dimensional model associated with the fault information are stored in the server; the mixed reality equipment can obtain corresponding maintenance indication information and a virtual three-dimensional model based on the identification information and the fault information, and displays a picture fused with the virtual three-dimensional model and a reality scene in a mixed reality space; afterwards, the mixed reality equipment controls each part model of the virtual three-dimensional model to execute disassembly demonstration based on maintenance guidance so as to guide a user to disassemble the image forming equipment and locate a fault area; abnormal information can be determined by comparing the part image with the corresponding part model. The virtual three-dimensional model which is the same as the image forming equipment is displayed in the mixed reality space, and disassembly demonstration is performed through maintenance guidance, so that a user can quickly know the structure of the image forming equipment, the user is assisted to quickly determine abnormal information, and the maintenance efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of image forming technology, and in particular to a maintenance control system, a maintenance system control method and a server system of an image forming device. Background Art

[0002] When after-sales personnel repair image forming equipment, they need to understand the repair methods of different models of image forming equipment. However, with the development of technology, the internal structure of image forming equipment has become more and more complicated. The internal structures of different models of image forming equipment are often inconsistent, and the parts are also different. It is difficult for after-sales personnel to be familiar with the structure of all models of image forming equipment, and it is difficult to accurately identify the failure or slight loss of parts with the naked eye. Therefore, when after-sales personnel repair, they may not be able to quickly disassemble and accurately locate the faulty parts in the image forming equipment, affecting the normal use of users. Even for experienced after-sales personnel, facing the relatively complex internal structure of current image forming equipment, if they only rely on personal experience for repair, the efficiency and accuracy of repair are difficult to guarantee. Summary of the invention

[0003] In view of this, the present application provides an MR-based image forming equipment maintenance control system, method and server system, so as to solve the problem in the prior art that the image forming equipment has a complex structure and is difficult for after-sales personnel to repair.

[0004] In a first aspect, an embodiment of the present application provides an image forming device maintenance control system based on MR, the maintenance control system comprising: A mixed reality device, and at least one server communicating with an image forming device and the mixed reality device, Wherein, the at least one server comprises a receiving module, configured to receive the fault information and identification information of the image forming device, a storage module, which stores the maintenance instruction information of the image forming device, the virtual three-dimensional model, the fault information and the identification information in association with each other, wherein the virtual three-dimensional model includes a plurality of component models, each component model corresponds to a component of the image forming device, a calling module, configured to call a virtual three-dimensional model and maintenance instruction information matching the image forming device based on the fault information and the identification information, and A determination module is used to compare the component image of the real scene with the corresponding component model to determine abnormal information; wherein display data is generated based on the virtual three-dimensional model and maintenance instruction information called by the calling module, and The mixed reality device is configured to provide maintenance guidance based on the generated display data and the determined abnormal information.

[0005] In an optional embodiment, the mixed reality device includes A display module is used to display a picture that combines the virtual three-dimensional model and the real scene in a mixed reality space. A control module is used to control the multiple component models of the virtual three-dimensional model to perform a disassembly demonstration based on the maintenance instruction information, wherein the disassembly demonstration is used to guide a user to disassemble the image forming device and locate a fault area.

[0006] In an optional embodiment, comparing the component image of the real scene with the corresponding component model to determine the abnormal information includes: Determine actual structural information of each component based on the component image of the fault area, and determine standard structural information of each component based on the corresponding component model; The abnormal information is determined according to the difference between the actual structure information and the corresponding standard structure information, and the abnormal information is used to guide the user to repair the image forming device.

[0007] In an optional embodiment, the controlling the multiple component models of the virtual three-dimensional model to perform a disassembly demonstration based on the maintenance instruction information includes: The mixed reality device acquires the user's operation image of the image forming device in real time through the camera device; The plurality of component models of the virtual three-dimensional model are controlled to perform a disassembly demonstration based on the maintenance instruction information and the operation image.

[0008] In a second aspect, an embodiment of the present application provides a server system, wherein the server system includes at least one server. Wherein, the at least one server comprises A receiving module, used for receiving fault information and identification information of an image forming device; a storage module, which stores the maintenance instruction information of the image forming device, the virtual three-dimensional model, the fault information and the identification information in association with each other, wherein the virtual three-dimensional model includes a plurality of component models, each component model corresponds to a component of the image forming device, a calling module, configured to call a virtual three-dimensional model and maintenance instruction information matching the image forming device based on the fault information and the identification information, and The determination module is used to compare the component image of the real scene with the corresponding component model to determine the abnormal information.

[0009] In an optional embodiment, comparing the component image of the real scene with the corresponding component model to determine the abnormal information includes: Determine the actual structural information of each component based on the component image of the real scene, and determine the standard structural information of each component based on the corresponding component model; The abnormal information is determined according to the difference between the actual structure information and the corresponding standard structure information, and the abnormal information is used to guide the user to repair the image forming device.

[0010] In a third aspect, an embodiment of the present application provides a method for controlling an image forming device maintenance system based on MR, the control method comprising: obtaining fault information and identification information of an image forming device; storing the maintenance instruction information and the virtual three-dimensional model of the image forming device in association with the fault information and the identification information, wherein the virtual three-dimensional model includes a plurality of component models, each component model corresponding to a component of the image forming device; Invoking a virtual three-dimensional model and maintenance instruction information matching the image forming device based on the fault information and the identification information; The component image of the real scene is obtained through the mixed reality device, and the component image is compared with the corresponding component model to determine the abnormal information.

[0011] In an optional embodiment, the control method further includes Displaying a picture that combines the virtual three-dimensional model and the real scene in the mixed reality space of the mixed reality device, Based on the maintenance instruction information, the multiple component models of the virtual three-dimensional model are controlled to perform a disassembly demonstration, where the disassembly demonstration is used to guide a user to disassemble the image forming device and locate a fault area.

[0012] In an optional embodiment, comparing the component image with the corresponding component model to determine the abnormal information includes: Determine actual structural information of each component based on the component image of the fault area, and determine standard structural information of each component based on the corresponding component model; The abnormal information is determined according to the difference between the actual structure information and the corresponding standard structure information, so as to guide the user to repair the image forming device.

[0013] In an optional embodiment, the controlling the multiple component models of the virtual three-dimensional model to perform a disassembly demonstration based on the maintenance instruction information includes: The mixed reality device acquires the user's operation image of the image forming device in real time through the camera device; The plurality of component models of the virtual three-dimensional model are controlled to perform a disassembly demonstration based on the maintenance instruction information and the operation image.

[0014] By adopting the solution provided in the embodiment of the present application, maintenance instruction information associated with identification information and a virtual three-dimensional model associated with fault information are stored in the server. During the maintenance process of the image forming device, the mixed reality device can obtain the corresponding maintenance instruction information and the virtual three-dimensional model based on the identification information and the fault information, and display the picture that integrates the virtual three-dimensional model and the real scene in the mixed reality space; thereafter, the mixed reality device controls the component models of the virtual three-dimensional model based on the maintenance instructions to perform a disassembly demonstration to guide the user to disassemble the image forming device and locate the fault area; the mixed reality device can obtain the component image of the fault area, compare the component image with the corresponding component model, determine the abnormal information, and guide the user to perform fault repair; in addition, the maintenance personnel can also identify the faulty components and replace or repair them based on their own maintenance experience with the help of the comparison results of the above-mentioned component images and the corresponding component models. The present application demonstrates disassembly through maintenance instructions in a mixed reality space, allowing users to quickly understand the condition of the image forming device and make up for the lack of maintenance experience and knowledge; in addition, the present application combines the machine vision, intelligent recognition comparison and maintenance experience of the mixed reality device to quickly determine the faulty components and locations, which can reduce the probability of missed detection and false detection, making fault detection more accurate and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] Figure 1 A structural schematic diagram of a maintenance control system provided in an embodiment of the present application; Figure 2 A flowchart of a maintenance system control method provided in an embodiment of the present application; Figure 3 A flowchart of another maintenance system control method provided in an embodiment of the present application; Figure 4 An exemplary schematic diagram of a maintenance system control method provided in an embodiment of the present application; Figure 5 An exemplary schematic diagram of another maintenance system control method provided in an embodiment of the present application; Figure 6 An exemplary schematic diagram of another maintenance system control method provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0018] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0019] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0020] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0021] The MR-based image forming device maintenance control system of the embodiment of the present application mainly includes a mixed reality (MR) device, an image forming device, and at least one server. The server can communicate with the image forming device and the MR device, and the image forming device and the MR device can also communicate with each other. Figure 1, image forming devices 1 to image forming devices N may represent image forming devices to be repaired of different manufacturers and different types, and MR devices 1 to MR devices N are used to assist users in repairing each image forming device. Printing management servers 1 to printing management servers N may represent servers provided by different printing manufacturers, and MR servers 1 to MR servers N may represent servers provided by different MR device manufacturers. Image forming device 1 may communicate with printing management server 1, MR device 1 may communicate with MR server 1, and printing management server 1 and MR server 1 may communicate with each other. The above is an exemplary description of the server. In one embodiment, image forming device 1 and MR device 1 may communicate with the same server 1 without distinguishing between printing management server 1 or MR server 1. Similarly, image forming device N and MR device N may communicate with the same server N without distinguishing between printing management server N or MR server N. In another embodiment, image forming devices 1 to image forming devices N and MR devices 1 to MR devices N may communicate with the same server without distinguishing between servers 1 to server N. The number of servers and the communication methods between each image forming device and MR device are not limited.

[0022] The server includes at least: a storage module for storing the maintenance instruction information, virtual three-dimensional model, identification information and fault information of the image forming device in association with each other. A receiving module for receiving the fault information and identification information of the image forming device, wherein the virtual three-dimensional model includes a plurality of component models, each of which corresponds to a component of the image forming device. A calling module for calling the virtual three-dimensional model and maintenance instruction information matching the image forming device based on the fault information and identification information. A determining module for comparing the component image of the real scene with the corresponding component model to determine abnormal information. The server can generate display data based on the virtual three-dimensional model and maintenance instruction information called by the calling module, and the MR device is configured to provide maintenance guidance based on the generated display data and the determined abnormal information.

[0023] When the user repairs the image forming device, the image forming device can be controlled to send identification information and fault information to the server. The identification information is used to determine the type or model of the image forming device, and the fault information may include a description of the fault phenomenon or a fault code. After receiving the identification information, the server can call the maintenance instruction information associated with the identification information, and after receiving the fault information, the virtual three-dimensional model associated with the fault information can be called. The MR device uploads the image of the components of the real scene to the server in real time, and the server compares the component image with the component model of the virtual three-dimensional model to determine the abnormal information. In addition, the server can control the virtual three-dimensional model to generate display data based on the content contained in the maintenance instruction information.

[0024] The MR device at least includes: a display module, where the user displays the screen that integrates the virtual three-dimensional model and the real scene in the mixed reality space. A control module, which is used to control multiple component models of the virtual three-dimensional model to perform a disassembly demonstration based on the maintenance instruction information, and the disassembly demonstration is used to guide the user to disassemble the image forming device and locate the fault area. During the maintenance process, the MR device can obtain the user's operation image of the image forming device in real time through the camera device, and can control multiple component models of the virtual three-dimensional model to perform a disassembly demonstration based on the maintenance instruction information and the operation image.

[0025] When the user disassembles the image forming device to display the fault area, the server can determine the actual structural information of each component based on the component image of the fault area, and can also determine the standard structural information of each component based on the corresponding component model. The server compares the actual structural information with the corresponding standard structural information, and determines abnormal information based on the difference between the two. The abnormal information is used to guide the user to repair the image forming device.

[0026] Figure 2 The following is a flow chart of a control method for an image forming device maintenance system based on MR provided in an embodiment of the present application. Figure 2 As shown, the method may include: Step 201: Acquire fault information and identification information of an image forming device.

[0027] The identification information may specifically refer to identification information such as manufacturer information, device model, serial number, device barcode or device ID of the image forming device. The server may determine the type of the current image forming device based on the identification information.

[0028] The method for the server to obtain identification information may include: (1) obtaining image information of the image forming device based on the camera device of the MR device, and obtaining identification information in a specific area of ​​the image information. Specifically, the user takes an image of the image forming device through the camera of the MR device, and then the MR device sends the image to the server. The server obtains the identification information based on the barcode area of ​​the image. Alternatively, the server may perform image recognition on the image of the image forming device to confirm the identification information.

[0029] (2) The user inputs identification information of the image forming device in the virtual interface of the MR device. Specifically, a control interface is provided in the mixed reality space constructed by the MR device, and the user can input information such as a barcode, model, or IP address of the image forming device in the control interface.

[0030] (3) The server establishes a connection with the image forming device and receives the identification information sent by the image forming device.

[0031] (4) The MR device establishes a connection with the image forming device, the image forming device sends the identification information to the MR device, and the MR device forwards the identification information to the server. Specifically, the MR device can be connected to the image forming device by wire or wirelessly via Bluetooth, wireless communication technology WiFi, Universal Serial Bus (USB), Near Field Communication (NFC), etc.

[0032] Through the above multiple optional methods, the server can quickly confirm the model of the image forming device.

[0033] The fault information may include: a fault code generated by the image forming device system, or text used to describe the fault phenomenon of the image forming device. The way in which the server obtains the fault information of the image forming device may include: (1) the user sends the fault information of the image forming device to the server through the MR device. Specifically, the MR device displays a control interface in the mixed reality space, and the user can enter the fault code or fault phenomenon in the control interface, and the MR device sends the fault code or fault phenomenon to the server.

[0034] (2) The server establishes a connection with the image forming device and receives fault information sent by the image forming device. The image forming device may send a fault code to the server. Optionally, the fault code may be used to indicate the area where the fault occurs as determined by the image forming device itself. For example, when the ink cartridge of the image forming device leaks or runs out, the control panel of the image forming device may display a message "Ink cartridge abnormality". After connecting to the server, the image forming device will send the fault code corresponding to the ink cartridge abnormality to the server.

[0035] (3) The MR device establishes a connection with the image forming device, the image forming device sends the fault information to the MR device, and the MR device forwards the fault information to the server.

[0036] Step 202 : storing the maintenance instruction information, the virtual three-dimensional model, the fault information and the identification information of the image forming device in association with each other.

[0037] For different types of image forming devices, the staff can pre-configure at least one associated virtual three-dimensional model for each type of image forming device. For different fault conditions of the image forming device, the staff can also pre-configure at least one associated maintenance instruction information for different fault conditions. The above association relationship is stored in the server. When maintenance is required, the user sends the identification information and fault information to the server, and the server sends the corresponding maintenance instruction information and virtual three-dimensional model to the MR device for maintenance instructions, so that the user can quickly locate the fault area and provide corresponding maintenance instructions.

[0038] The maintenance instruction information in the embodiment of the present application includes maintenance instructions for the image forming device, and the maintenance instructions are used to guide the user to disassemble the image forming device to determine the cause of the failure of the image forming device and the specific parts replacement and maintenance methods. The virtual three-dimensional model is displayed in the mixed reality space constructed by the MR device, and the internal structure of the virtual three-dimensional model is the same as that of the image forming device. The virtual three-dimensional model includes multiple component models, each component model corresponds to a component in the image forming device one by one, and the positional relationship between the component models is also the same as that in the image forming device.

[0039] In the embodiment of the present application, the maintenance instruction information and the virtual three-dimensional model are stored in the server to achieve a larger capacity of information storage. The user can set maintenance instruction information for more types of image forming devices, and the MR device only needs to obtain the corresponding maintenance instruction information when needed. In addition, when the image forming device is upgraded, the server can obtain the upgrade information of the image forming device and the latest maintenance instruction information in real time to achieve real-time data update.

[0040] In an optional embodiment, the MR device may also store a small amount of commonly used maintenance instruction information and virtual three-dimensional models in a local database. Specifically, when the MR device detects that the number of failures of a certain type of image forming device exceeds a preset threshold, the maintenance instruction information and the virtual three-dimensional model may be saved in the local database after the user completes the maintenance. The next time the image forming device of this type fails again, the MR device may directly obtain the maintenance instruction information and the virtual three-dimensional model from the local database to speed up the acquisition of the maintenance instruction information and the virtual three-dimensional model. For image forming devices with a lower frequency of failure, the MR device may directly delete the maintenance instruction information and the virtual three-dimensional model after the user completes the maintenance to save storage space in the local database.

[0041] Step 203: Calling a virtual three-dimensional model and maintenance instruction information matching the image forming device based on the fault information and the identification information.

[0042] For different fault phenomena, the MR device can display different virtual 3D models. For example, when a consumable device of an image forming device fails, the MR device can display a virtual 3D model of the consumable device in the mixed reality space, and the maintenance instructions are used to indicate the disassembly steps of the consumable device. For another example, when a paper transmission failure occurs in an image forming device, the MR device can display virtual 3D models related to the paper feed box, conveyor roller, and transfer roller in the mixed reality space, and the maintenance instructions are used to indicate the disassembly steps of the paper feed box, conveyor roller, and transfer roller.

[0043] In an optional embodiment of the present application, by extracting a virtual three-dimensional model and maintenance instructions that match the fault information, only the faulty part can be disassembled for demonstration, thereby improving the user's maintenance efficiency.

[0044] In step 204, a picture integrating the virtual three-dimensional model and the real scene is displayed in a mixed reality space, and each component model of the virtual three-dimensional model is controlled to perform a disassembly demonstration based on the maintenance instruction information. The disassembly demonstration is used to guide the user to disassemble the image forming device and locate the fault area.

[0045] During the maintenance process, maintenance personnel can view the virtual three-dimensional model of the image forming device through the MR device. The virtual three-dimensional model dynamically demonstrates the parts that need to be inspected and how to disassemble the parts. Users can disassemble the image forming device step by step following the dynamic demonstration of the virtual three-dimensional model.

[0046] In an optional embodiment, the MR device monitors the disassembly progress of the maintenance personnel in the real world through a camera device, and obtains the user's operation image of the image forming device in real time, and controls the various component models of the virtual three-dimensional model to perform a disassembly demonstration based on the maintenance instructions and the operation image. Specifically, if the MR device detects that the user has disassembled any component model, the MR device will dynamically demonstrate the disassembly process of the next component model according to the maintenance instructions. For example, if the disassembly step requires the removal of the front cover, the virtual three-dimensional model will prompt the maintenance personnel to remove the screw buckle corresponding to the front cover. When the maintenance personnel removes the front cover, the MR device performs image recognition through a camera or a laser radar, and detects that the front cover has been removed, then removes the front cover of the virtual three-dimensional model, displays the virtual three-dimensional model without the front cover, and further prompts the maintenance personnel to disassemble and inspect the components that need to be disassembled and inspected next. It can be understood that through the cooperation of the virtual three-dimensional model and the camera device, the MR device guides the user to complete the disassembly of the image forming device step by step. Even users who are inexperienced and unfamiliar with all the internal structural details of the current image forming device can use the MR device to perform fault troubleshooting and reduce the difficulty of maintenance.

[0047] Step 205: Acquire the component image of the real scene through the mixed reality device, compare the component image with the corresponding component model, and determine the abnormal information.

[0048] When maintenance personnel disassemble the parts of the faulty area, the images of the parts taken by the camera of the MR device can be sent to the server. The server determines the current actual structural information of each component, and the standard structural information of each component can be determined through the corresponding component model. After that, the server can calculate the difference between the actual structural information of each component and the corresponding standard structural information based on information such as size, shape, structure, installation position, and other physical properties. If the difference exceeds a pre-set threshold, the mixed reality device can determine that the component is abnormal. The server sends the determined abnormal information to the MR device, and guides the user to further replace or repair the parts based on the abnormal information.

[0049] On this basis, users can also combine their own maintenance experience to compare the difference between the actual parts and the corresponding parts models in the mixed reality space to determine abnormal information. Specifically, maintenance personnel control the MR device to enlarge and rotate the virtual three-dimensional model through gestures and other means. The virtual three-dimensional model can clearly display the status of the parts of the machine in a normal state. Maintenance personnel can also take out the part model separately from the virtual three-dimensional model displayed by the MR device, and then enlarge and rotate the details of the part model to view it. Maintenance personnel compare the part model with the actual parts of the image forming device with the naked eye to determine the location of the abnormal part.

[0050] In the embodiment of the present application, for some common faults or subtle faults that are difficult for maintenance personnel to notice with the naked eye, the server can quickly determine abnormal information based on the images taken in real time by the mixed reality device and the virtual three-dimensional model. For faults that are difficult for the server to identify, maintenance personnel can manually judge the abnormal information based on their own maintenance experience. By combining the intelligent recognition of the mixed reality device and the server with the manual inspection of the maintenance personnel, the recognition speed of abnormal information can be improved, and the recognition range of abnormal information can be expanded, effectively improving maintenance efficiency.

[0051] The following uses a printer as an example to further illustrate the maintenance system control method of the embodiment of the present application. It should be pointed out that the image forming device to be repaired is a printer is only an exemplary description and should not be used as a limitation on the protection scope of the present application. Other image forming devices include but are not limited to copiers, fax machines, multi-function image production and copying devices, electrostatic printing devices and any other similar devices are also applicable to the maintenance system control method of the embodiment of the present application. Figure 3 A flowchart of another maintenance system control method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method may include: Step 301: Maintenance personnel upload printer model and fault information via MR equipment.

[0052] When the printer begins to be repaired, the maintenance personnel first use the MR device camera to capture the current real world in real time into the MR device, and the MR device displays the real world image in the mixed reality space. The maintenance personnel can observe the printer nameplate through the real world image in the mixed reality space to obtain the model A1 of the current printer. The MR device displays the model information input interface in the mixed reality space, and the maintenance personnel inputs the obtained model A1 into the MR device. Alternatively, the MR camera scans the barcode of the current printer and determines that the printer model is A1 through the barcode. Alternatively, the MR device and the printer are connected via Bluetooth. After the connection is successful, the printer sends the model information A1 to the MR device. After the MR device confirms the printer model A1, it sends the information of the printer model A1 to the server.

[0053] The maintenance personnel observes through the MR device that the fault code displayed on the printer control panel is "Internal Fault 01". The MR device displays a fault information input interface in the mixed reality space, and the maintenance personnel inputs "Internal Fault 01" into the fault information input interface. Alternatively, the MR device is connected to the printer via Bluetooth. After the connection is successful, the printer sends the fault code "Internal Fault 01" to the MR device. If the printer does not accurately or promptly identify the device's fault code, the maintenance personnel can preliminarily determine the fault condition based on the observed printer working status or printing results, and directly select the corresponding fault option in the fault information input interface, such as "Printer paper jam", "Gray image on printed paper", "Print screen offset", etc. The MR device can identify the fault phenomenon to confirm the fault information. After the MR device determines the fault information, it sends the fault information to the server.

[0054] Step 302: The server calls corresponding maintenance instruction information based on the printer model, and calls corresponding virtual three-dimensional model based on the fault information.

[0055] After determining the printer model, the server calls the maintenance instruction information of the A1 model printer. Multiple types of maintenance instruction information can be pre-stored in the server, covering printers of different models. In order to facilitate retrieval, the server can also pre-store a maintenance instruction information association table, which includes multiple printer models, barcodes, and corresponding maintenance instruction information. Table 1 is an example of a maintenance instruction information association table, in which the first column is the printer model, including A1, B2, ... The second column is the printer barcode, including the barcode "12345678" of the A1 model, the barcode "00123456" of the B2 model, ... The third column is the maintenance instruction information, including A1 maintenance instruction information, B2 maintenance instruction information, ... After the server receives the printer model A1 sent by the MR device, or the barcode "12345678" corresponding to the A1 model, it calls the A1 maintenance instruction information.

[0056] Table 1 For different types of faults, the server can call the corresponding virtual three-dimensional model. Referring to Table 2, the server can set an association table between fault codes and virtual three-dimensional models for retrieval based on fault codes. When the fault code is "internal fault 01", the server calls "virtual three-dimensional model 01", and when the fault code is "internal fault 02", the server can call "virtual three-dimensional model 02". Different fault codes are used to indicate different fault types. For example, internal fault 01 is a drive system fault, and internal fault 02 is a photosensitive drum (OPC) component fault, consumables fault, etc. The correspondence between fault codes and fault types can be modified by relevant staff and is not limited here.

[0057] Table 2 Step 303: The MR device controls the virtual three-dimensional model to display the component model that needs to be inspected and the disassembly method of the component model based on the maintenance instruction information.

[0058] For the current fault being "internal fault 01", the MR device displays the virtual three-dimensional model 01 in the virtual three-dimensional model. Afterwards, the MR device can control the "virtual three-dimensional model 01" to perform a disassembly demonstration based on the steps recorded in the maintenance instruction information.

[0059] Assuming that "Internal Fault 01" is a gear rotation fault in the drive system, the virtual 3D model 01 will first highlight and prompt the maintenance personnel to remove the printer side cover. Figure 4 As shown in the figure, the side pointed by the arrow is the printer side cover, and the color of the printer side cover is changed to prompt the user to remove it. The MR device monitors whether the maintenance personnel have completed the operation of removing the printer side cover through the camera. When it is confirmed that the maintenance personnel have completed the removal of the side cover, the virtual 3D model 01 is updated. Figure 5 The virtual 3D model 01 will display the picture after the side cover is removed and highlight the "gear" to prompt the maintenance personnel to check the component gear.

[0060] Step 304 , determining abnormal parts by comparing the detailed states of the parts of the virtual three-dimensional model with the parts of the printer.

[0061] The mixed reality space displayed by the MR device includes the virtual 3D model 01 and the real image of the printer captured by the camera. When the maintenance personnel disassemble the printer following the disassembly demonstration, the MR device can compare the real image of the printer with the virtual 3D model 01 every time it detects that the maintenance personnel have completed a disassembly step, and intelligently identify abnormal information. Optionally, the MR device can also send the real image of the printer to the server in real time, and the server intelligently identifies abnormal information based on the real image of the printer and the virtual 3D model 01.

[0062] Optionally, the MR device can also wait until the maintenance personnel disassemble the printer to the faulty area before comparing the real image with the virtual three-dimensional model 01. Specifically, the MR device can obtain the image of the gears of the printer through the camera, and accurately compare the image with the virtual three-dimensional model 01, so as to automatically identify the difference between the gear model and the printer gears, and prompt the maintenance personnel to assist the user to complete the maintenance more efficiently. The printer image may include multi-angle images of the parts of the faulty area of ​​the printer recorded by the MR device through the camera when the printer is stationary or working. The MR device intelligently compares the status of each component in the image with each component in the virtual three-dimensional model 01 at multiple times, multiple angles, and multiple states, and comprehensively analyzes the comparison results to find possible abnormal components. If the fault involves multiple parts, it is necessary to guide the user in turn during the shooting and comparison process, and use maintenance experience to complete more detailed disassembly, inspection, repair, and replacement.

[0063] If the MR device or server does not identify abnormal information, the maintenance personnel can compare the gear model in the virtual 3D model 01 with the real gear in the printer, and select a single gear component model in the imaging screen of the MR device through gestures for detailed observation. Figure 6 Maintenance personnel can use gestures to enlarge or rotate the gear model to further accurately compare whether there is any abnormality in the printer gear.

[0064] Step 305: Maintenance personnel manually replace or repair abnormal parts to complete maintenance.

[0065] After confirming the cause of the fault, maintenance personnel can manually replace or repair the corresponding parts to complete the repair.

[0066] In the embodiment of the present application, the virtual 3D model dynamically demonstrates the maintenance steps, which can assist the maintenance personnel to complete the maintenance of various types of equipment and reduce the difficulty of maintenance. In addition, by combining the machine vision, intelligent recognition comparison of the MR equipment and the server and the maintenance experience of the maintenance personnel, the probability of missed detection and wrong detection can be reduced, the faulty components and locations can be quickly determined, and maintenance can be guided, making fault detection more accurate and efficient.

[0067] Corresponding to the above embodiments, the present application also provides an electronic device. Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device 700 may include: a processor 701, a memory 702 and a communication unit 703. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiment of the present application. It can be a bus structure or a star structure, and can also include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0068] The communication unit 703 is used to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0069] The processor 701 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs, instructions, and / or modules stored in the memory 702, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 701 can include only a central processing unit (CPU). In the embodiment of the present application, the CPU can be a single computing core or multiple computing cores.

[0070] The memory 702 is used to store the execution instructions of the processor 701. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0071] When the execution instructions in the memory 702 are executed by the processor 701 , the electronic device 700 is enabled to execute part or all of the steps in the above embodiments.

[0072] In a specific implementation, the present application also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment of the maintenance system control method provided in the present application. The storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0073] In a specific implementation, the present application also provides a computer program product, wherein the computer program product includes executable instructions, and when the executable instructions are executed on a computer, the computer executes part or all of the steps in each embodiment of the maintenance system control method provided in the present application.

[0074] The embodiment of the present application also provides a non-temporary computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the maintenance system control method provided by the embodiment of the present application.

[0075] The above-mentioned non-temporary computer-readable storage medium may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (Read Only Memory; hereinafter referred to as: ROM), an erasable programmable read-only memory (ErasableProgrammable Read Only Memory; hereinafter referred to as: EPROM) or flash memory, optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device.

[0076] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0077] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0078] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0079] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment and the terminal embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

Claims

1. An image forming equipment maintenance control system based on MR, characterized in that: The maintenance control system comprises: A mixed reality device, and at least one server communicating with an image forming device and the mixed reality device, Wherein, the at least one server comprises a receiving module, configured to receive the fault information and identification information of the image forming device, a storage module, which stores the maintenance instruction information of the image forming device, the virtual three-dimensional model, the fault information and the identification information in association with each other, wherein the virtual three-dimensional model includes a plurality of component models, each component model corresponds to a component of the image forming device, a calling module, configured to call a virtual three-dimensional model and maintenance instruction information matching the image forming device based on the fault information and the identification information, and A determination module is used to compare the component image of the real scene with the corresponding component model to determine abnormal information; wherein display data is generated based on the virtual three-dimensional model and maintenance instruction information called by the calling module, and The mixed reality device is configured to provide maintenance guidance based on the generated display data and the determined abnormal information.

2. The maintenance control system according to claim 1, characterized in that: The mixed reality device comprises A display module is used to display a picture that combines the virtual three-dimensional model and the real scene in a mixed reality space. A control module is used to control the multiple component models of the virtual three-dimensional model to perform a disassembly demonstration based on the maintenance instruction information, wherein the disassembly demonstration is used to guide a user to disassemble the image forming device and locate a fault area.

3. The maintenance control system according to claim 2, characterized in that: The comparing the component image of the real scene with the corresponding component model to determine the abnormal information includes: Determine actual structural information of each component based on the component image of the fault area, and determine standard structural information of each component based on the corresponding component model; The abnormal information is determined according to the difference between the actual structure information and the corresponding standard structure information, and the abnormal information is used to guide the user to repair the image forming device.

4. The maintenance control system according to claim 2, characterized in that: The step of controlling the plurality of component models of the virtual three-dimensional model to perform disassembly demonstration based on the maintenance instruction information includes: The mixed reality device acquires the user's operation image of the image forming device in real time through the camera device; The plurality of component models of the virtual three-dimensional model are controlled to perform disassembly demonstration based on the maintenance instruction information and the operation image.

5. A server system, comprising at least one server, in, The at least one server comprises A receiving module, used for receiving fault information and identification information of an image forming device; a storage module, which stores the maintenance instruction information of the image forming device, the virtual three-dimensional model, the fault information and the identification information in association with each other, wherein the virtual three-dimensional model includes a plurality of component models, each component model corresponds to a component of the image forming device, a calling module, configured to call a virtual three-dimensional model and maintenance instruction information matching the image forming device based on the fault information and the identification information, and The determination module is used to compare the component image of the real scene with the corresponding component model to determine the abnormal information.

6. The server system according to claim 5, characterized in that: The comparing the component image of the real scene with the corresponding component model to determine the abnormal information includes: Determine the actual structural information of each component based on the component image of the real scene, and determine the standard structural information of each component based on the corresponding component model; The abnormal information is determined according to the difference between the actual structure information and the corresponding standard structure information, and the abnormal information is used to guide the user to repair the image forming device.

7. A method for controlling a maintenance system of an image forming device based on MR, characterized in that: The control method comprises: obtaining fault information and identification information of an image forming device; storing the maintenance instruction information and the virtual three-dimensional model of the image forming device in association with the fault information and the identification information, wherein the virtual three-dimensional model includes a plurality of component models, each of which corresponds to a component of the image forming device; Invoking a virtual three-dimensional model and maintenance instruction information matching the image forming device based on the fault information and the identification information; The component image of the real scene is obtained through the mixed reality device, and the component image is compared with the corresponding component model to determine the abnormal information.

8. The maintenance system control method according to claim 7, characterized in that: The control method also includes Displaying a picture that combines the virtual three-dimensional model and the real scene in the mixed reality space of the mixed reality device, Based on the maintenance instruction information, the multiple component models of the virtual three-dimensional model are controlled to perform a disassembly demonstration, where the disassembly demonstration is used to guide a user to disassemble the image forming device and locate a fault area.

9. The maintenance system control method according to claim 8, characterized in that: The step of comparing the component image with the corresponding component model to determine abnormal information includes: Determine actual structural information of each component based on the component image of the fault area, and determine standard structural information of each component based on the corresponding component model; The abnormal information is determined according to the difference between the actual structure information and the corresponding standard structure information, and the abnormal information is used to guide the user to repair the image forming device.

10. The maintenance system control method according to claim 8, characterized in that: The step of controlling the plurality of component models of the virtual three-dimensional model to perform disassembly demonstration based on the maintenance instruction information includes: The mixed reality device acquires the user's operation image of the image forming device in real time through the camera device; The plurality of component models of the virtual three-dimensional model are controlled to perform disassembly demonstration based on the maintenance instruction information and the operation image.

Images