Fault identification and response method, fault identification and response system, server, and terminal device

Abstract

To provide a failure identification and handling method, a failure identification and handling system, a server, and a terminal device that identify information to confirm a failure.SOLUTION: The method includes: shooting a video including a failure code; identifying the failure code based on the video; correctly identifying a complete failure code without being restricted by a display type of the failure code and an image collection parameter; and acquiring information about a failure immediately and correctly based on the identified complete and correct failure code.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the field of machine maintenance, and in particular to a method and system for identifying and dealing with faults. [Background technology]

[0002] With the development of science and technology and the improvement of living standards, various electronic devices or electrical devices are widely applied in various fields. When a fault occurs in an appliance, a user or an after-sales service staff usually checks the fault code displayed on the appliance panel on-site, and then searches for fault information corresponding to the fault code in the maintenance manual to confirm and eliminate the fault. However, this method takes time, and if the fault is somewhat complicated, the user cannot confirm and eliminate the fault by himself, and must contact the after-sales service staff to request a visit service, which takes even more time and affects the normal use of the appliance. In addition, the fault information and troubleshooting method described in the maintenance manual are not intuitive, which affects the accuracy of fault confirmation and the efficiency of fault troubleshooting.

[0003] In recent years, a method has been proposed for identifying a fault by identifying information.

[0004] For example, the camera collection module uses a camera to capture an LED lamp group image and transmits it to the fault location, collection and identification module, which performs image recognition and analysis on the LED lamp group image to obtain specific status information of the hardware circuit, performs fault analysis based on the lamp group coding information base, obtains the analysis result of the hardware circuit, and then feeds it back.

[0005] In addition, for example, a digital camera can be used to take an image of a system error code displayed on a display unit of a user computer, and the image can be sent to a fault recovery process server via a mobile phone. The server then detects the system error code based on the image, and retrieves fault content information associated with the error code from a database and sends it to the mobile phone.

[0006] Furthermore, for example, if a two-dimensional code is displayed along with a fault code on the panel of an equipment, the user can scan the two-dimensional code and then quickly access a homepage containing maintenance information, thereby obtaining fault and maintenance information.

[0007] It should be noted that the above technical background introduction is merely provided for the convenience of providing a clearer and more complete description of the technical solutions of the present disclosure, and for the convenience of persons skilled in the art to easily understand them, and the mere fact that these solutions are described in the background section of the present disclosure does not constitute an indication that the above technical solutions are publicly known by persons skilled in the art. Summary of the Invention [Problem to be solved by the invention]

[0008] However, the inventor has found the following: In the above-mentioned conventional method, due to the diversification of fault code display methods or limitations due to parameters such as image collection speed and collection time, the display of fault codes in a single image is incomplete or inaccurate, and related fault and maintenance information cannot be obtained accurately and timely, which may affect the use of the equipment. In addition, in the method of obtaining fault and maintenance information by scanning a two-dimensional code and accessing a website, it is necessary to design two-dimensional codes for various faults in each equipment, which increases the burden on equipment manufacturing and makes it difficult to widely apply to existing general equipment. [Means for solving the problem]

[0009] In order to solve at least one of the above problems, the present disclosure proposes a fault identification and response method and system, which can accurately identify the complete fault code by taking a video including the fault code and identifying the fault code based on the video, without being limited by the fault code display method or image collection parameters, and can quickly and accurately obtain information about the fault based on the identified complete and accurate fault code.

[0010] According to a first aspect of an embodiment of the present disclosure, there is provided a fault identification and response method, the method including the steps of: taking a video including a fault code; identifying the fault code in the video and obtaining the identified fault code; determining information about the fault corresponding to the fault code based on the identified fault code, and generating display data based on the information about the fault; and displaying information about the fault based on the display data, wherein the fault code is displayed by a digital tube, and is displayed by flashing at a predetermined frequency according to the display principle of the digital tube; the step of identifying the fault code in the video and obtaining the identified fault code includes: superimposing adjacent or interval frames of the video based on a preset weight value, obtaining a plurality of superimposed frames, and identifying the fault code based on the plurality of superimposed frames.

[0011] According to a second aspect of the embodiment of the present disclosure, there is provided a fault identification and response system, comprising: a shooting unit for shooting a video including a fault code; an identification unit for identifying a fault code in the video and obtaining the identified fault code; a determination unit for determining information about a fault corresponding to the fault code according to the identified fault code, and generating display data according to the information about the fault; and a display unit for showing the information about the fault according to the display data, wherein the fault code is displayed by a digital tube, and is displayed by flashing at a predetermined frequency according to the display principle of the digital tube; the identification unit further comprises: a superposition module for superposing adjacent frames or interval frames of the video according to a preset weight value, and obtaining the superposed multiple frames; and a fourth identification module for identifying the fault code according to the superposed multiple frames.

[0012] According to a third aspect of an embodiment of the present disclosure, there is provided a server including: a second receiving unit, which receives a video including a fault code taken by a terminal device from the terminal device; an identification unit, which identifies a fault code in the video and obtains the identified fault code; a determination unit, which determines information about a fault corresponding to the fault code according to the identified fault code, and generates display data according to the information about the fault; and a second sending unit, which transmits the display data to the terminal device; wherein the fault code is displayed by a digital tube, and is displayed by flashing at a predetermined frequency according to the display principle of the digital tube; the identification unit includes: a superposition module, which superimposes adjacent frames or interval frames of the video based on a preset weight value, and obtains a plurality of superimposed frames; and a fourth identification module, which identifies the fault code based on the plurality of superimposed frames.

[0013] According to a fourth aspect of an embodiment of the present disclosure, there is provided a terminal device including: a shooting unit for shooting a video including a fault code; an identification unit for identifying a fault code in the video and obtaining the identified fault code; a determination unit for determining information about a fault corresponding to the fault code based on the identified fault code, and generating display data based on the information about the fault; and a display unit for showing the information about the fault based on the display data, wherein the fault code is displayed by a digital tube, and is displayed by flashing at a predetermined frequency according to the display principle of the digital tube; the identification unit further includes: a superposition module for superimposing adjacent frames or interval frames of the video based on a preset weight value, and obtaining the superimposed multiple frames; and a fourth identification module for identifying the fault code based on the superimposed multiple frames.

[0014] One of the beneficial effects of the embodiments of the present disclosure is that by taking a video containing fault codes and identifying the fault codes based on the video, the complete fault codes can be accurately identified without being limited by the fault code display manner or image acquisition parameters, so that information about the fault can be quickly and accurately obtained based on the identified complete and accurate fault codes.

[0015] With reference to the following description and drawings, particular embodiments of the present disclosure have been disclosed in detail and the principles of the present disclosure have been shown to be in use. It is to be understood that the embodiments of the present disclosure are not limited thereto in scope. The embodiments of the present disclosure encompass many alterations, modifications and equivalents within the spirit and scope of the appended claims.

[0016] Characteristic information described and shown in one embodiment may be used in the same or similar manner in one or more other embodiments, may be combined with features of the other embodiments, or may be used in place of characteristic information of the other embodiments.

[0017] It should be emphasized that the term "comprises" is used in the present text to indicate the presence of a feature, integral element, step or component, but does not exclude the presence or addition of one or more other features, integral elements, steps or components.

[0018] Many aspects of the present disclosure can be better understood by referring to the following attached drawings. The elements shown in the attached drawings are not drawn to scale, but are merely for illustrating the principles of the present disclosure. For convenience in expressing and describing some parts of the present disclosure, corresponding parts in the attached drawings may be enlarged or reduced. Elements or feature information shown in one attached drawing or one embodiment of the present disclosure may be combined with elements or feature information shown in one or more other attached drawings or embodiments. In addition, similar reference numerals in the attached drawings may indicate corresponding elements in several attached drawings and may also indicate corresponding elements used in one or more embodiments. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram of a fault identification and response method according to a first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram of one display scheme for fault codes according to the first embodiment of the present disclosure. [Diagram 3] FIG. 3 is a schematic diagram of one display scheme for fault codes according to the first embodiment of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram of another display method for fault codes according to the first embodiment of the present disclosure. [Diagram 5] FIG. 5 is a schematic diagram of another display method for fault codes according to the first embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram of yet another display method for fault codes according to the first embodiment of the present disclosure. [Figure 7] FIG. 7 is a schematic diagram of yet another display method for fault codes according to the first embodiment of the present disclosure. [Figure 8] FIG. 8 is a schematic diagram of one method for implementing step 102 according to the first embodiment of the present disclosure. [Figure 9] FIG. 9 is a schematic diagram of another method for implementing step 102 according to the first embodiment of the present disclosure. [Figure 10] FIG. 10 is a schematic diagram of yet another method for implementing step 102 according to the first embodiment of the present disclosure. [Figure 11] FIG. 11 is a schematic diagram of overlapping adjacent frames of the video according to the first embodiment of the present disclosure. [Figure 12] FIG. 12 is a schematic diagram of a fault identification and response system according to a second embodiment of the present disclosure. [Figure 13] FIG. 13 is a schematic diagram of a fault identification and response system according to a second embodiment of the present disclosure, which performs a corresponding method. [Figure 14] FIG. 14 is a schematic diagram of an embodiment of a recognition unit according to the second embodiment of the present disclosure. [Figure 15] FIG. 15 is a schematic diagram of another embodiment of the recognition unit according to the second embodiment of the present disclosure. [Figure 16] FIG. 16 is a schematic diagram of a second identification module according to a second embodiment of the present disclosure. [Figure 17] FIG. 17 is a schematic diagram of another embodiment of the recognition unit according to the second embodiment of the present disclosure. [Figure 18] FIG. 18 is a schematic diagram of a fourth identification module according to the second embodiment of the present disclosure. [Figure 19] FIG. 19 is a schematic diagram of a terminal device according to the third embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. (First embodiment) The first embodiment of the present disclosure provides a fault identification and response method. Figure 1 is a schematic diagram of the fault identification and response method according to the first embodiment of the present disclosure. As shown in Figure 1, the method includes: Step 101: Take a video containing the fault code; Step 102: identifying a fault code in the video and obtaining the identified fault code; Step 103: determining information about a fault corresponding to the identified fault code based on the identified fault code, and generating display data based on the information about the fault; Step 104 includes displaying information about the fault based on the display data.

[0021] Then, by taking a video containing the fault code and identifying the fault code based on the video, the complete fault code can be accurately identified without being limited by the fault code display manner or image collection parameters, so that information about the fault can be quickly and accurately obtained based on the identified complete and accurate fault code.

[0022] In the first embodiment of the present disclosure, the device to which the method is applied may be various devices capable of displaying a fault code, such as various industrial devices or various household devices. For example, the devices include an air conditioner, a washing machine, a refrigerator, or a water heater. When a fault occurs in the device, a fault code is displayed on the display panel of the device.

[0023] In the first embodiment of the present disclosure, the manufacturer of the device can indicate the fault code in various ways. For example, the fault code may be composed of at least one of numbers, letters, and symbols, and may have one or more digits. The specific display format and number of digits of the fault code may be set by the manufacturer of the device according to the actual needs.

[0024] In step 101, a video including a fault code is taken. In the first embodiment of the present disclosure, step 101 may be performed by a terminal device. That is, the video may be taken by a terminal device. For example, the terminal device may be a smartphone, an intelligent tablet, or intelligent glasses. The intelligent tablet may be a tablet terminal (a tablet-type information and communication terminal). The intelligent glasses may be smart glasses (a glasses-type information and communication terminal).

[0025] In the first embodiment of the present disclosure, for example, the captured video may be captured on a display panel area of ​​a device, and a fault code may be displayed on the display panel, which may be, for example, a control panel or other display panel of the device.

[0026] For example, the fault code is displayed in a fault display area of ​​the display panel.

[0027] In the first embodiment of the present disclosure, the display panel on which the fault code is displayed may be various kinds of display screens, for example, a liquid crystal display screen or a digital tube display screen.

[0028] In the first embodiment of the present disclosure, the method may be applied to various fault code display methods.

[0029] In addition, the duration of the video may be determined according to actual needs, for example, according to the manner of displaying the fault code.

[0030] Below, the display methods of the fault codes are described as examples, but the first embodiment of the present disclosure is not limited to these display methods.

[0031] For example, a fault code may be constantly cycled such that it is displayed during certain time periods and not during certain time periods, in which case the captured video may include the time periods during which the fault code was displayed.

[0032] 2 and 3 are schematic diagrams of a display method of a fault code according to the first embodiment of the present disclosure. As shown in Fig. 2, a fault code is displayed in "A9-01" during a certain time period, and as shown in Fig. 3, the fault code is not displayed during a certain time period.

[0033] Further, for example, a fault code may continually cycle such that part of it is displayed during one time period, and the remaining part is displayed during another time period, in which case the captured video may include a time period during which part of the fault code is displayed and another time period during which the remaining part of the fault code is displayed.

[0034] 4 and 5 are schematic diagrams of another display method of the fault code according to the first embodiment of the present disclosure. As shown in Fig. 4, a part of the fault code is displayed as "J5" in a certain time period, and as shown in Fig. 5, the remaining part of the fault code is displayed as "-01" in another time period.

[0035] In the first embodiment of the present application, the part of the fault code may be called an error code, indicating the component where the fault occurred, and the remaining part of the fault code may be called a detail code, indicating the specific fault content. The error code and the detail code are combined into one complete fault code. For example, "J5" in FIG. 4 indicates an error code, and "-01" in FIG. 5 indicates a detail code, and the combination of the two forms the complete error code "J5-01."

[0036] Further, for example, the fault code may be displayed by a digital tube and may be displayed by flashing at a predetermined frequency according to the display principle of the digital tube, in which case, the time length of the captured video may be any suitable time length, for example, several seconds to collect multiple frames for identifying the fault code.

[0037] 6 and 7 are schematic diagrams of another display manner of the fault code according to the first embodiment of the present disclosure. The digital tube flashes at a predetermined frequency to display the fault code, so that the fault code "J5" is displayed at the corresponding time as shown in FIG. 6, and the fault code "J5" is incompletely displayed at the corresponding time as shown in FIG. 7.

[0038] In step 102, a fault code in the video is identified, and the identified fault code is obtained. In the first embodiment of the present disclosure, step 102 may be performed by a local terminal device, or the terminal device may transmit data of the video to a server installed in a remote location, for example, a cloud server, and step 102 may be performed by the server.

[0039] The following provides specific examples of step 102 for different fault code display methods.

[0040] For example, for a display scheme in which fault codes are displayed during certain periods of time and not displayed during other periods of time, but are constantly cycling, i.e., for the examples shown in Figures 2 and 3, in step 102 it is possible to identify each frame of the video one by one.

[0041] 8 is a schematic diagram of one implementation of step 102 according to the first embodiment of the present disclosure. As shown in FIG. 8, in the step of identifying a frame of the video, Step 801: Average the pixel values ​​of all pixels in the fault code display area of ​​the frame; Step 802: if the average value satisfies a preset condition, determining that the fault code display area contains a fault code; and Step 803 includes obtaining an identified fault code by using a first discrimination model obtained by training in advance to discriminate the fault code display area including the fault code.

[0042] In step 802, the pre-defined condition may be determined by a visual difference between the fault code and the background of the display area. For example, if the background of the display area is dark but the displayed fault code is light, the pre-defined condition is that the average pixel value is greater than a pre-defined threshold. For further example, if the background of the display area is light but the displayed fault code is dark, the pre-defined condition is that the average pixel value is less than a pre-defined threshold. Here, the pre-defined threshold may be determined by the pixel value of the background of the display area.

[0043] In step 803, the first discriminant model used is obtained by training using the training data. For example, the first discriminant model is obtained by training a neural network using the training data. The neural network may have a conventional neural network structure.

[0044] In the first embodiment of the present disclosure, if the average value does not satisfy a preset condition, it is determined that the fault code display area does not contain a fault code, and no processing is performed on that frame of image.

[0045] In the first embodiment of the present disclosure, after identifying each frame, if the fault codes obtained by identifying each frame match, the fault codes are used as the identification result. However, if the fault codes obtained by identifying each frame do not match, the fault codes identified from more frames can be used as the identification result.

[0046] Further, for example, for a display manner in which the fault code is constantly cycled, with a part of the fault code being displayed in one time period and the remaining part being displayed in another time period, i.e., for the example of Fig. 4 and Fig. 5, Fig. 9 is a schematic diagram of another implementation method of step 102 according to the first embodiment of the present disclosure. As shown in Fig. 9, in the method, Step 901: obtaining an identified first partial fault code and a second partial fault code by identifying each frame of the video one by one; and Step 902 includes combining the first partial fault code and the second partial fault code into one complete fault code as an identified fault code.

[0047] In this way, it is possible to identify the complete fault codes of different parts that are displayed at different times, so that accurate fault information can be obtained.

[0048] For example, by identifying each frame of the video one by one, the first partial fault code “J5” in FIG. 4 and the second partial fault code “-01” in FIG. 5 are obtained, and the first partial fault code “J5” and the second partial fault code “-01” are combined into a complete fault code “J5-01”.

[0049] Here, the step of identifying a frame of the video is similar to steps 801-803, i.e. Calculating the average value of all pixel points in the fault code display area of ​​the frame; If the average value satisfies a preset condition, determining that the fault code display area contains a fault code; and The method includes obtaining an identified first partial fault code or a second partial fault code by using a first identification model obtained by prior training to identify the fault code display area including the fault code.

[0050] In the first embodiment of the present disclosure, it may be determined according to actual needs which part of the fault code to be prepended as the first partial fault code. For example, for some fault codes that contain the identified alphabet, it can be prepended as the first partial fault code. For further example, for some fault codes that do not contain the identified symbol, it can be prepended as the first partial fault code, but for some fault codes that contain the identified symbol, for example, some codes that contain the symbol "-", it can be prepended as the second partial fault code.

[0051] Further, for example, for the display manner in which the fault code is displayed by the digital tube flashing at a certain frequency, i.e., for the examples of Fig. 6 and Fig. 7, Fig. 10 is a schematic diagram of another implementation method of step 102 according to the first embodiment of the present disclosure. As shown in Fig. 10, in the method, Step 1001: overlapping adjacent or interval frames of the video according to a preset weight value to obtain overlapped frames; Step 1002 includes identifying a fault code based on the superimposed multiple frames.

[0052] This eliminates the problem that a single image cannot accurately display the fault code, making it possible to accurately determine the fault code and obtain accurate fault information.

[0053] In step 1001, every two adjacent or interval frames of the video are superimposed based on the weight value, and the interval frames may be two interval frames spaced apart by a preset number of frames.

[0054] In the first embodiment of the present disclosure, the weight value for superimposition may be set according to actual needs. For example, when the average of the weight value is 0.5, that is, the average value is taken for the pixel values ​​of the corresponding pixel points of two adjacent frames or two frames with an interval. For further example, when the average of the weight value is 1, that is, the pixel values ​​of the corresponding pixel points of two adjacent frames or two frames with an interval are added.

[0055] In the first embodiment of the present disclosure, for a display mode in which a fault code is constantly cycled so that a part of the fault code is displayed in one time period and the remaining part is displayed in another time period, step 1002 includes identifying each superimposed frame one by one to obtain an identified first partial fault code and a second partial fault code, and combining the first partial fault code and the second partial fault code into one complete fault code as an identified fault code.

[0056] FIG. 11 is a schematic diagram of superimposing adjacent frames of the video according to the first embodiment of the present disclosure. As shown in FIG. 11, in the 6-frame image included in the video, the images of the 2nd, 3rd, and 5th frames have incomplete fault codes captured due to the flashing of the digital tube. For two adjacent frames, superimposition is performed on the addition of pixel values, and after the 1st and 2nd frames are superimposed, the first partial fault code "LC" is identified; after the 3rd and 4th frames are superimposed, the normal fault code cannot be identified, so it is discarded; after the 5th and 6th frames are superimposed, the second partial fault code "-14" is identified. By combining the first partial fault code "LC" and the second partial fault code "-14", the complete fault code "LC-14" is obtained.

[0057] In the first embodiment of the present disclosure, step 102 may further include steps including, before identifying, segmenting the video to obtain each frame of the video, and determining a fault code display area for each frame of the video.

[0058] In this way, the fault code display area of ​​each frame can be determined, which makes it easy to identify the fault code for that fault code display area.

[0059] In the first embodiment of the present disclosure, for example, the position of the fault code display area can be determined based on a preset positional relationship between the fault code display area and a display panel displaying the fault code, or the position of the fault code display area can be determined based on a second identification model obtained by training in advance.

[0060] For example, the area captured in the video is approximately the area of ​​a display panel, and the position of the fault code display area is determined according to a preset positional relationship of the fault code display area with the display panel.

[0061] For example, the second discrimination model may be a neural network, and the neural network may have a conventional neural network structure.

[0062] In the first embodiment of the present disclosure, after a fault code is identified in step 102, information about a fault corresponding to the fault code is determined based on the identified fault code, and display data is generated based on the information about the fault in step 103. In the first embodiment of the present disclosure, step 103 may be performed by a local terminal device, or step 103 may be performed by a server.

[0063] In the first embodiment of the present disclosure, a fault information database may be constructed in advance, and the fault information database stores information regarding faults of various devices, and the information regarding the faults corresponds to the type, model number, and fault code of the device.

[0064] In the first embodiment of the present disclosure, the information about the fault corresponding to the fault code may include a fault content corresponding to the fault code and / or a first model of the equipment in which the fault occurs, The first model is a two-dimensional model or a three-dimensional model that can represent each component of the equipment.

[0065] In the first embodiment of the present disclosure, the fault content may include information on the location or component where the fault occurs and / or corresponding after-sales maintenance information. For example, the after-sales maintenance information may include a maintenance method and steps, and may further include information such as the cost of replacing the component.

[0066] In step 103, based on the identified fault code, searching in the fault information database, determining information about the fault corresponding to the fault code, and generating display data based on the information about the fault, where the display data is capable of showing information about the fault.

[0067] In step 104, information about the fault is displayed based on the display data. In the first embodiment of the present disclosure, if step 104 is performed by a local terminal device and step 103 is performed by a server, the server may send the generated display data to the terminal device.

[0068] In the first embodiment of the present disclosure, the information on the failure may be displayed in various ways. For example, the screen may display the failure content in text, or the screen may display the failure content in a combination of text and graphics, or a model of the device that can represent the failure content by combining fact and fiction may be displayed. In the first embodiment of the present disclosure, the method of displaying the information on the failure is not limited.

[0069] For example, in step 104, a second model of the device capable of expressing the fault content may be displayed based on the display data, and the second model may be formed based on the first model of the device. In this way, the structure of the device and the location of the fault can be intuitively shown, thereby improving the efficiency of fault determination and fault elimination.

[0070] For example, the second model is a two-dimensional or three-dimensional model that can represent each component of the equipment and can prominently show the location or component where the fault occurred.

[0071] For example, the second model can be displayed in the form of augmented reality, images, or videos, which allows the user to intuitively grasp the fault information, thereby further improving the efficiency of fault determination and troubleshooting.

[0072] For example, in the second model, the location or component where the fault occurred is displayed in a different color, which allows the fault information to be more intuitively understood, thereby further improving the efficiency of fault determination and fault rectification.

[0073] In the first embodiment of the present disclosure, after-sales maintenance information among the information regarding the failure may be displayed. For example, the maintenance method or steps may be displayed in the form of a moving image or video. This further improves the efficiency of troubleshooting.

[0074] As can be seen from the above first embodiment, by taking a video containing fault codes and identifying the fault codes based on the video, the complete fault codes can be accurately identified without being limited by the display manner of the fault codes or the image collection parameters, so that information about the fault can be quickly and accurately obtained based on the identified complete and accurate fault codes. Second embodiment The second embodiment of the present disclosure provides a fault identification and response system corresponding to the fault identification and response method described in the first embodiment, and for its specific implementation, reference may be made to the implementation of the method described in the first embodiment, and descriptions of the same or related content will be omitted.

[0075] FIG. 12 is a schematic diagram of a fault identification and response system according to a second embodiment of the present disclosure. As shown in FIG. 12, the fault identification and response system 1200 includes: a photographing unit 1201 for photographing a video including a fault code; an identification unit 1202 for identifying fault codes in the video and obtaining the identified fault codes; a determining unit 1203 for determining information about a fault corresponding to the identified fault code according to the identified fault code, and generating display data according to the information about the fault; and a display unit 1204 for displaying information about the fault based on the display data.

[0076] In the second embodiment of the present disclosure, as shown in FIG. 12 , the fault identification and response system 1200 may include a terminal device 1210 and a server 1220, where the terminal device 1210 may include a photographing unit 1201 and a display unit 1204, and the server 1220 may include an identification unit 1202 and a determination unit 1203.

[0077] Also, as shown in FIG. 12, the terminal device 1210 further includes: a first sending unit 1205 for sending the video to a server 1220; a first receiving unit 1206 for receiving display data from a server 1220; The server 1220 further A second receiving unit 1207 receives a video including a fault code taken by the terminal device 1210 from the terminal device; and a second sending unit 1208 for sending the display data to the terminal device 1210.

[0078] In the second embodiment of the present disclosure, the terminal device 1210 may be various kinds of terminal devices, for example, the terminal device may be a smart phone, an intelligent tablet, or an intelligent pair of glasses.

[0079] The server 1220 may be a variety of types of server, for example, a cloud server.

[0080] 13 is a schematic diagram of a fault identification and response system according to a second embodiment of the present disclosure performing a corresponding method. As shown in FIG. 13, the method includes: Step 1301: The terminal device 1210 takes a video including a fault code; Step 1302: The terminal device 1210 transmits the video to the server 1220; Step 1303: The server 1220 identifies a fault code in the video and obtains the identified fault code; Step 1304: The server 1220 determines information about a fault corresponding to the identified fault code based on the identified fault code, and generates display data based on the information about the fault; Step 1305: The server 1220 transmits the display data to the terminal device 1210; and Step 1306: The terminal device 1210 displays information about the fault based on the display data.

[0081] 14 is a schematic diagram of an embodiment of the identification unit according to the second embodiment of the present disclosure. As shown in FIG. 14, the identification unit 1202 includes: A first calculation module 1401 for taking an average value for pixel values ​​of all pixel points in a fault code display area of ​​the frame; a first determining module 1402 for determining that the fault code display area contains a fault code if the average value satisfies a preset condition; and a first identification module 1403 that acquires an identified fault code by identifying the fault code display area including the fault code using a first identification model obtained by pre-training.

[0082] 15 is a schematic diagram of another embodiment of the identification unit according to the second embodiment of the present disclosure. As shown in FIG. 15, the identification unit 1202 includes: A second identification module 1501, for obtaining an identified first partial fault code and a second partial fault code by identifying each frame of the video one by one; A first combination module 1502 combines the first partial fault code and the second partial fault code into a complete fault code as an identified fault code.

[0083] 16 is a schematic diagram of a second identification module according to a second embodiment of the present disclosure. As shown in FIG. 16, the second identification module 1501 includes: A second calculation module 1601 for taking an average value for pixel values ​​of all pixel points in the fault code display area of ​​the frame; a second determining module 1602 for determining that the fault code display area contains a fault code if the average value satisfies a preset condition; and a third identification module 1603 that obtains an identified first partial fault code or a second partial fault code by identifying the fault code display area including the fault code using a first identification model obtained by pre-training.

[0084] 17 is a schematic diagram of another embodiment of the identification unit according to the second embodiment of the present disclosure. As shown in FIG. 17, the identification unit 1202 includes: a superposition module 1701 for superposing adjacent or interval frames of the video based on a preset weight value to obtain a plurality of superposed frames; and a fourth identification module 1702 for identifying a fault code based on the superimposed multiple frames.

[0085] 18 is a schematic diagram of a fourth identification module according to the second embodiment of the present disclosure. As shown in FIG. 18, the fourth identification module 1702 includes: A fifth identification module 1801, for obtaining the first partial fault code and the second partial fault code by identifying each superimposed frame one by one; A second combination module 1802 combines the first partial fault code and the second partial fault code into a complete fault code as an identified fault code.

[0086] In the second embodiment of the present disclosure, the identification unit 1202 further comprises: a segmentation module for segmenting the video to obtain frames of the video; and a second determining module for determining a fault code display area for each frame of the video.

[0087] In the second embodiment of the present disclosure, the second determination module can determine the position of the fault code display area based on a preset positional relationship between the fault code display area and a display panel displaying the fault code, or can determine the position of the fault code display area based on a second identification model obtained by pre-training.

[0088] In a second embodiment of the present disclosure, the information regarding the fault corresponding to the fault code may include a fault content corresponding to the fault code and / or a first model of the equipment in which the fault occurred.

[0089] In a second embodiment of the present disclosure, the display unit 1204 displays a second model of the equipment that can represent the fault content based on the display data, and the second model may be formed based on the first model of the equipment.

[0090] For example, the first model of the equipment is a two-dimensional or three-dimensional model that can represent each component of the equipment, and the second model of the equipment is a two-dimensional or three-dimensional model that can represent each component of the equipment and can prominently represent the location or component where a fault has occurred.

[0091] Moreover, the display unit 1204 may display the second model in the manner of augmented reality, images or videos. In the second embodiment of the present disclosure, the fault content may include information on the location or components where the fault occurs and / or corresponding after-sales maintenance information.

[0092] In the second embodiment of the present disclosure, the functions of the above units and modules may be implemented by referring to the contents of the relevant steps in the first embodiment, and the description thereof will be omitted here.

[0093] As can be seen from the above embodiment, by taking a video containing fault codes and identifying the fault codes based on the video, the complete fault codes can be accurately identified without being limited by the display manner of the fault codes or the image acquisition parameters, so that information about the fault can be quickly and accurately obtained based on the identified complete and accurate fault codes. Third embodiment The third embodiment 3 of the present disclosure provides a terminal device corresponding to the fault identification and response method described in the first embodiment, and for its specific implementation, reference may be made to the implementation of the method described in the first embodiment, and descriptions of the same or related content will be omitted.

[0094] FIG. 19 is a schematic diagram of a terminal device according to the third embodiment of the present disclosure. As shown in FIG. 19, the terminal device 1900 includes: a camera unit 1901 for capturing a video including a fault code; an identification unit 1902 for identifying fault codes in the video and obtaining the identified fault codes; a determining unit 1903 for determining information about a fault corresponding to the identified fault code according to the identified fault code, and generating display data according to the information about the fault; and a display unit 1904 for displaying information relating to the fault based on the display data.

[0095] That is, all steps in the first embodiment may be executed by the terminal device, and the contents of the related steps in the first embodiment may be referred to for the implementation of the functions of each unit of the terminal device, and the description will be omitted here.

[0096] As can be seen from the above embodiment, by taking a video containing fault codes and identifying the fault codes based on the video, the complete fault codes can be accurately identified without being limited by the display manner of the fault codes or the image acquisition parameters, so that information about the fault can be quickly and accurately obtained based on the identified complete and accurate fault codes.

[0097] The above-mentioned devices and methods in the present disclosure may be realized by hardware or a combination of hardware and software. The present disclosure relates to a computer-readable program as follows, which, when executed by a logic unit, can cause the logic unit to realize the above-mentioned devices or components, or can cause the logic unit to realize the above-mentioned various methods or steps.

[0098] The present disclosure relates to a storage medium for storing the above-mentioned program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, and the like.

[0099] The present disclosure has been described above in conjunction with specific embodiments. However, those skilled in the art should understand that the descriptions are merely illustrative and do not limit the scope of protection of the present disclosure. Those skilled in the art can make various modifications and modifications to the present disclosure based on the spirit and principles of the present disclosure, and these modifications and modifications are also within the scope of the present disclosure. [Explanation of symbols]

[0100] 1200 Fault Identification and Response System 1210 Terminal Equipment 1201 Photography Unit 1202 Identification Unit 1203 Confirmed Units 1204 Display unit 1205 First transmitting unit 1206 First receiving unit 1207 Second receiving unit 1208 Second transmitting unit 1220 Server 1401 First Calculation Module 1402 First Confirmation Module 1403 First Identification Module 1501 Second Identification Module 1502 First combination module 1601 Second Calculation Module 1602 Second Confirmation Module 1603 Third Identification Module 1701 Superposition Module 1702 Fourth Identification Module 1801 5th Identification Module 1802 Second Combination Module 1901 Photography Unit 1902 Identification Unit 1903 Confirmed Units 1904 Display unit

Claims

1. A computer software implemented fault identification and response method, comprising: taking a video containing the fault code; identifying fault codes in the video and obtaining the identified fault codes; determining information about a fault corresponding to the fault code based on the identified fault code, and generating display data based on the information about the fault; and displaying information about the fault based on the display data; The fault code is It is displayed by a digital tube, and according to the display principle of the digital tube, it is displayed by flashing at a predetermined frequency; The step of identifying fault codes in the video and obtaining identified fault codes includes: overlapping adjacent or interval frames of the video based on a preset weight value to obtain a plurality of overlapped frames; and performing fault code identification based on the superimposed plurality of frames; A fault identification and response method comprising:

2. The step of identifying fault codes in the video and obtaining identified fault codes includes: and identifying each frame of the video in turn, wherein identifying a frame of the video comprises: taking an average value for pixel values ​​of all pixel points in a fault code display area of ​​said frame; determining that the fault code display area contains a fault code when the average value satisfies a preset condition; and The fault identification and response method according to claim 1, further comprising: obtaining an identified fault code by using a first identification model obtained by prior training to identify the fault code display area including the fault code.

3. The step of identifying fault codes in the video and obtaining identified fault codes includes: Obtaining a first partial fault code and a second partial fault code by identifying each frame of the video one by one; and 2. The method of claim 1, further comprising combining the first partial fault code and the second partial fault code into a complete fault code as an identified fault code.

4. The step of identifying a frame of the video comprises: taking an average value for pixel values ​​of all pixel points in a fault code display area of ​​said frame; determining that the fault code display area contains a fault code when the average value satisfies a preset condition; and The fault identification and response method according to claim 3, further comprising: obtaining an identified first partial fault code or a second partial fault code by identifying the fault code display area including the fault code using a first identification model obtained by prior training.

5. The step of identifying a fault code based on a plurality of superimposed frames comprises: Identifying each superimposed frame one by one to obtain an identified first partial fault code and a second partial fault code; and 2. The method of claim 1, further comprising combining the first partial fault code and the second partial fault code into a complete fault code as an identified fault code.

6. The step of identifying fault codes in the video and obtaining the identified fault codes may further include: Segmenting the video to obtain each frame of the video; and The method of any one of claims 1 to 5, further comprising: determining a fault code display area for each frame of the video.

7. said step of determining a fault code display area for each frame of said video comprises: determining the position of the fault code display area according to a preset positional relationship between the fault code display area and a display panel displaying the fault codes; or 7. The fault identification and response method according to claim 6, further comprising identifying the location of the fault code display area using a second identification model obtained by pre-training.

8. The fault identification and response method according to claim 1, wherein the information about the fault corresponding to the fault code includes a fault content corresponding to the fault code and / or a first model of the equipment in which the fault occurred.

9. The step of displaying information regarding the failure based on the display data includes:

9. The fault identification and response method of claim 8, further comprising displaying a second model of the equipment capable of representing the fault content based on the display data, the second model being formed based on the first model.

10. the first model of the device is a two-dimensional or three-dimensional model capable of representing each component of the device; the second model of the equipment is a two-dimensional or three-dimensional model capable of representing each component of the equipment and prominently representing the location or component where the fault has occurred; 10. The fault identification and response method of claim 9, wherein the step of displaying a second model of the equipment capable of representing the fault content includes displaying the second model in the manner of augmented reality, image or video.

11. The fault identification and response method according to any one of claims 8 to 10, characterized in that the fault details include information on the location or component where the fault occurs and / or corresponding after-sales maintenance information.

12. a camera unit for capturing a video including a fault code; an identification unit for identifying a fault code in the video and obtaining the identified fault code; a determining unit for determining information about a fault corresponding to the identified fault code based on the identified fault code, and generating display data based on the information about the fault; a display unit that displays information about the malfunction based on the display data, The fault code is It is displayed by a digital tube, and according to the display principle of the digital tube, it is displayed by flashing at a predetermined frequency; The identification unit is a superimposing module for superimposing adjacent or interval frames of the video based on a preset weight value to obtain a plurality of superimposed frames; and a fourth identification module for identifying a fault code based on the superimposed plurality of frames. A fault identification and response system comprising:

13. The identification unit identifies each frame of the video one by one; The identification unit is A first calculation module for taking an average value for pixel values ​​of all pixel points in a fault code display area of ​​the frame; a first determining module for determining that the fault code display area contains a fault code when the average value satisfies a preset condition; The fault identification and response system of claim 12, further comprising: a first identification module that obtains an identified fault code by identifying the fault code display area including the fault code using a first identification model obtained by pre-training.

14. The identification unit is A second identification module for obtaining an identified first partial fault code and a second partial fault code by identifying each frame of the video one by one; and a first combination module that combines the first partial fault code and the second partial fault code into a complete fault code, the identified fault code.

15. The second identification module includes: a second calculation module for taking an average value for pixel values ​​of all pixel points in a fault code display area of ​​the frame; a second determining module for determining that the fault code display area contains a fault code when the average value satisfies a preset condition; and a third identification module that obtains an identified first partial fault code or a second partial fault code by identifying the fault code display area including the fault code using a first identification model obtained by pre-training. The fault identification and response system of claim 14, further comprising:

16. The fourth identification module includes: A fifth identification module is configured to identify each superimposed frame one by one to obtain the first partial fault code and the second partial fault code that are identified; and a second combination module that combines the first partial fault code and the second partial fault code into a complete fault code, the identified fault code.

17. The identification unit is a segmentation module for segmenting the video to obtain each frame of the video; The fault identification and response system of any one of claims 12 to 16, further comprising: a second determination module for determining a fault code display area for each frame of the video.

18. The fault identification and response system of claim 17, characterized in that the second determination module determines the position of the fault code display area according to a preset positional relationship between the fault code display area and a display panel displaying the fault code, or determines the position of the fault code display area according to a second identification model obtained by pre-training.

19. The fault identification and response system of claim 12, wherein the information about the fault corresponding to the fault code includes a fault content corresponding to the fault code and / or a first model of an equipment in which the fault occurred.

20. The fault identification and response system of claim 19, wherein the display unit displays a second model of the equipment capable of representing the fault content based on the display data, the second model being formed based on the first model.

21. the first model of the device is a two-dimensional or three-dimensional model capable of representing each component of the device; the second model of the equipment is a two-dimensional or three-dimensional model capable of representing each component of the equipment and prominently representing the location or component where the fault has occurred; The fault identification and response system of claim 20 , wherein the display unit displays the second model in the manner of augmented reality, image or video.

22. The fault identification and response system according to any one of claims 19 to 21, characterized in that the fault details include information on the location or component where the fault occurs and / or corresponding after-sales maintenance information.

23. The fault identification and response system includes a terminal device and a server; the terminal device includes the photographing unit and the display unit; The fault identification and response system of claim 12 , wherein the server includes the identification unit and the determination unit.

24. The fault identification and response system according to claim 23 , wherein the terminal device is a smart phone, an intelligent tablet, or an intelligent pair of glasses.

25. a second receiving unit for receiving a video including a fault code captured by the terminal device from the terminal device; an identification unit for identifying a fault code in the video and obtaining the identified fault code; a determining unit for determining information about a fault corresponding to the identified fault code based on the identified fault code, and generating display data based on the information about the fault; a second transmission unit for transmitting the display data to the terminal device; The fault code is It is displayed by a digital tube, and according to the display principle of the digital tube, it is displayed by flashing at a predetermined frequency; The identification unit is a superimposing module for superimposing adjacent or interval frames of the video based on a preset weight value to obtain a plurality of superimposed frames; and a fourth identification module for identifying a fault code based on the superimposed plurality of frames. A server comprising:

26. a camera unit for capturing a video including a fault code; an identification unit for identifying a fault code in the video and obtaining the identified fault code; a determining unit for determining information about a fault corresponding to the identified fault code based on the identified fault code, and generating display data based on the information about the fault; a display unit that displays information about the malfunction based on the display data, The fault code is It is displayed by a digital tube, and according to the display principle of the digital tube, it is displayed by flashing at a predetermined frequency; The identification unit is a superimposing module for superimposing adjacent or interval frames of the video based on a preset weight value to obtain a plurality of superimposed frames; and a fourth identification module for identifying a fault code based on the superimposed plurality of frames. A terminal device characterized in that

Images