Parts identification method and identification device

Abstract

The present application provides a part identification method and device, the method comprising: acquiring an image including a part; detecting the image on the basis of a detection model, and identifying a type of the part; and determining part information, on the basis of the identified type of the part. The present application detects an image including a part on the basis of a detection model, identifies the type of the part, and determines the part information, thereby making it possible to shorten the time of checking part information and to improve the efficiency of maintaining equipment.

Description

[Technical Field]

[0001] The present application relates to the field of information technology, and more particularly to a method and apparatus for identifying parts. [Background technology]

[0002] When a device breaks down, it may be necessary to replace a damaged part in the device, for example, an engineer may check an inventory of the damaged part based on the technical name of the damaged part, obtain the part from the inventory, and then replace the damaged part.

[0003] When an engineer performs on-site maintenance on equipment, if he encounters an unusual or complicated part, he needs to contact the relevant person. The relevant person will provide the equipment model diagram (e.g., an assembly perspective diagram). The engineer will check the name of each part on-site based on the model diagram, and after confirming the name of the damaged part, he will check the inventory and quantity of the part according to the part name or number and other information.

[0004] It should be noted that the above description of the technical background is provided merely for the convenience of clearly and completely explaining the technical contents of the present application and for the purpose of facilitating understanding by those skilled in the art, and since these technical contents are described in the background section of the present application, they cannot be deemed to be publicly known by those skilled in the art. Summary of the Invention

[0005] For the same type of equipment, such as environmental equipment such as air conditioners, purifiers, or humidifiers, different models often have different structures, and the parts used vary, and the same parts are installed in different positions. In addition, the number of parts in the equipment is increasing, making the structure more and more complicated.

[0006] The inventors of the present application discovered that when engineers perform maintenance on equipment, they may not know the name of a certain part, so they must look up documents or inquire with relevant parties to obtain the name of the part and then check the inventory; this process generally takes a very long time and affects the efficiency of equipment maintenance.

[0007] In order to solve at least the above technical problem or a similar technical problem, an embodiment of the present application provides a part identification method and device that detects an image containing a part based on a detection model, identifies the type of the part, and specifies part information, thereby reducing the time required to check part information and improving the efficiency of equipment maintenance.

[0008] According to one aspect of the present embodiment, acquiring an image including the part; Detecting the image based on a detection model and identifying the type of the part; and identifying part information based on the identified part type; Identifying the part information based on the identified type of the part includes: Identifying a part within a predetermined area in the image based on the identified type of the part; and identifying the part information based on the identified part; The predetermined area includes a planar area or a spatial area, providing a method for identifying the part.

[0009] According to another aspect of the present invention, an acquisition means for acquiring an image including the component; an identification means for detecting the image based on a detection model and identifying the type of the part; and an identification means for identifying part information based on the identified type of the part, The identification means identifies the part information based on the type of the identified part, Identifying a part within a predetermined area in the image based on the identified type of the part; and identifying the part information based on the identified part; The predetermined area provides an identification device for the part, including a planar area or a spatial area.

[0010] A beneficial effect of the embodiments of the present application is that by detecting images containing parts based on a detection model, identifying the type of part, and specifying part information, the time required to check part information can be reduced and the efficiency of equipment maintenance can be improved.

[0011] With reference to the following description and drawings, particular embodiments of the present application are disclosed in detail, demonstrating the manner in which the principles of the present application may be employed. The embodiments of the present application should not be considered limiting in scope. Within the terms of the appended claims, the embodiments of the present application include many variations, modifications, and equivalents.

[0012] Features described and / or shown in one embodiment may be used in the same or similar manner in one or more other embodiments, may be combined with features in the other embodiments, or may substitute for features in the other embodiments.

[0013] The term "comprises" when used in this context refers to the presence of a feature, an entire element, a step or an element, but does not exclude the presence / addition of one or more other features, entire elements, steps or elements. [Brief explanation of the drawings]

[0014] Elements and features illustrated in one drawing or embodiment of the present application may be combined with elements and features shown in one or more other drawings or embodiments, and in the drawings, like reference numerals may be used to indicate corresponding parts in the several drawings and to indicate corresponding parts used in one or more embodiments.

[0015] The included drawings are provided for a better understanding of the embodiments of the present application, constitute a part of the specification, illustrate the embodiments of the present application, and together with the written description, explain the principles of the present application. Obviously, the following drawings are only a few examples of the present application, and those skilled in the art can derive other drawings based on these drawings without paying inventive work. Each drawing is as follows: [Figure 1] 1 is a schematic diagram of a component identification method according to a first embodiment of the present invention. [Figure 2] FIG. 1 is a schematic diagram of a method for training a detection model. [Figure 3] 1 is a schematic diagram of a method for identifying parts within a predetermined region in an image. [Figure 4] FIG. 10 is a schematic diagram of component positions within a first predetermined radius range. [Figure 5] FIG. 10 is a schematic diagram showing components within a first predetermined radius range. [Figure 6] FIG. 10 is another schematic diagram of a method for identifying parts within a predetermined region in an image. [Figure 7] FIG. 10 is a schematic diagram of component positions within a projection range of a second predetermined area. [Figure 8] FIG. 1 is a schematic diagram of recommending part information to an engineer. [Figure 9] 1 is a schematic diagram of how operation 104 is implemented. [Figure 10] 9 is a schematic diagram of extracting part features in operation 901. FIG. [Figure 11] 9 is a schematic diagram of extracting user features in operation 901. FIG. [Figure 12] FIG. 1 is a schematic diagram of a method for identifying a failed component based on operational data. [Figure 13] 2 is a schematic diagram of a vibration signal when a part is operating normally and when it is faulty. FIG. [Figure 14] FIG. 10 is a schematic diagram of a component identification device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] These and other features of the present application will become apparent from the following specification, which refers to the drawings. The specification and drawings specifically disclose certain embodiments of the present application and show some embodiments in which the principles of the present application can be employed. The present application is not limited to the described embodiments; on the contrary, it should be understood that the present application includes all amendments, modifications, and equivalents within the scope of the appended claims. Various embodiments of the present application will now be described with reference to the drawings. These embodiments are merely illustrative and do not limit the present application.

[0017] In the embodiments of the present application, terms such as "first" and "second" are used to distinguish different elements from one another, but do not indicate the spatial or chronological arrangement of these elements, and these elements should not be limited by these terms. The term "and / or" includes any and all combinations of one or more of the associated listed terms. Terms such as "include," "comprise," "have," and the like refer to the presence of stated features, elements, elements, or assemblies, but do not exclude the presence / addition of one or more other features, elements, elements, or assemblies.

[0018] In the examples of the present application, unless the context clearly indicates otherwise, the singular forms "a," "the," etc., should be understood to include the plural and not be limited to the meaning of "one," but broadly as "one kind" or "one type," and the term "the" should be understood to include both the singular and the plural. Also, unless the context clearly indicates otherwise, the term "based on" should be understood to mean "based at least in part on," and the term "based on" should be understood to mean "based at least in part on."

[0019] Example 1 Example 1 of the present application provides a method for identifying parts.

[0020] FIG. 1 is a schematic diagram of a component identification method according to a first embodiment of the present invention.

[0021] As shown in Figure 1, the part identification method is as follows: an operation 101 of obtaining an image including the part; an operation 102 of detecting the image based on a detection model and identifying the type of part; and an operation 103 of identifying part information based on the identified part type.

[0022] Here, the part information may include at least one of the part number, specifications, wiring code number, supplier, material, price, stock quantity, and exploded assembly diagram.

[0023] The component may be attached to a device, and the device may be, for example, an environmental device such as an air conditioner, a purifier, a humidifier, etc. However, the present application is not limited thereto, and the device may be any type of device other than an environmental device.

[0024] According to the first embodiment, an image including a part is detected based on a detection model, the type of the part is identified, and the part information is specified. This reduces the time required to check the part information, improves the efficiency of equipment maintenance, and also improves the accuracy of identifying the type of part.

[0025] In operation 101 of this embodiment, the image may include only one component or may include two or more components, and the two or more components may be spaced apart or at least partially overlapping in the image. The image may be a photograph taken by an engineer performing maintenance on the equipment using an imaging device, such as a camera, a mobile terminal such as a smartphone, or AR glasses. The image may also be obtained by cropping a video captured by the imaging device. For example, if an engineer wants to obtain component information for a certain component, he or she can use the imaging device to capture an image of the component and its surrounding components.

[0026] In operation 101, it is possible to further acquire model information of the equipment in which the part is located. This allows the image to be detected by referring to the model information in operation 102. Since information such as the name and number of parts corresponding to different models of equipment differs, the accuracy of identifying the type of part can be improved by detecting the image by referring to the model information.

[0027] The model information can be obtained by scanning the two-dimensional code or barcode of the equipment, reading the radio frequency identification (RFID) tag of the equipment, or the like, and input into the detection model in operation 102. Alternatively, the model information of the equipment can be manually entered by an engineer, or the model information of the equipment can be obtained from within a building information model (BIM).

[0028] In operation 102, the detection model may be a detection model based on a neural network. For example, it may be a detection model of a convolutional neural network (CNN) + YOLO (YOU ONLY LOOK ONCE) network. The detection model has high stability and high sensitivity to a large amount of data. In addition, this embodiment is not limited thereto, and the detection model may be of other types, such as a CNN + other types of networks, such as a CNN convolutional neural network + Faster R-CNN or a CNN + SSD network.

[0029] In operation 102, a detection model may be obtained by training, so that the trained detection model can be directly used to detect images when performing the component identification method of the present application.

[0030] Below, we will explain how to train the detection model, where the detection model is a CNN+YOLO network detection model.

[0031] 2 is a schematic diagram of a method for training a detection model. As shown in FIG. 2, the training method includes: An operation 201 of inputting a plurality of training images into the constructed neural network model; an operation 202 of outputting the detection result through a fully connected network of a neural network; an operation 203 of constructing a loss function based on the detection result and labeling information for the plurality of learning images; and an operation 204 of adjusting parameters in the neural network model so as to minimize the loss function and cause the neural network model to converge, and storing the adjusted neural network model as a detection model.

[0032] In this embodiment, before performing the learning process shown in FIG. 2 , multiple images of a specific device model within a specific range can be captured and components in each image can be labeled. For example, learning images can be created by enclosing components in the images with bounding boxes and labeling the component types. Each bounding box contains one component, and the bounding box is the smallest bounding box that encloses the continuous component area of ​​the component. The bounding box may be, for example, rectangular. Capturing multiple images of a specific range can prevent some components from being overlooked due to the imaging angle, thereby improving the accuracy of learning.

[0033] In operation 201, a plurality of training images are input to the constructed CNN+YOLO network detection model.

[0034] In operation 202, the detection model detects the training image and outputs the detection results from the fully connected network of the neural network, such as the coordinates of the bounding box in the training image and the type of part within the bounding box.

[0035] In operation 203, a loss function is constructed based on the detection result of operation 202 and information labeled in the training image (e.g., the coordinates of the bounding box labeled in the training image and the type of part in the bounding box). The loss function may be called, for example, YOLOv3LOSS.

[0036] The loss function is used to reflect the magnitude of the error between the detection result of operation 202 and the labeled information. For example, the loss function Loss can be expressed as the following equation (1). [Number 1] TIFF0007783509000001.tif42106(1)

[0037] where f is the sum of two input values ​​(e.g., two y i where y is the value of i indicates the detection result, and the other y i indicates labeled information), the dissimilarity calculation may be a mean square difference or may be in the form of cross entropy, etc.

[0038] In operation 204, the parameters in the neural network model are adjusted to minimize the loss function and cause the neural network model to converge, and the adjusted neural network model is saved as a detection model. For example, the process of adjusting the parameters in the neural network model in operations 201 to 203 and 204 can be repeated multiple times to minimize the loss function and obtain a final detection model.

[0039] The detection model trained based on the method of FIG. 2 can be used in operation 102 to detect the image acquired by operation 101.

[0040] For example, in operation 102, for the image acquired in operation 101, the detection model can output the bounding box coordinates of the parts in the image and the type of the parts within the bounding box. Here, for one image, the detection model can detect one or more bounding boxes.

[0041] For example, in operation 102, the image may be preprocessed by segmenting it, and then the bounding box coordinates and part type corresponding to each segment of the image may be output. The segmentation process may be implemented by classifying pixel points, and the image may be segmented based on the U-NET method, for example.

[0042] In operation 103, it is possible to identify which parts' part information needs to be output based on the identification result of operation 102, and output the corresponding part information. For example, based on the type of the identified part, parts within a predetermined area in the image are identified, and part information is identified based on the identified part. Here, the predetermined area can include a planar area (i.e., a two-dimensional area) or a spatial area (i.e., a three-dimensional area). This makes it possible to search for parts within a range corresponding to the predetermined area, thereby more accurately outputting part information required by the engineer.

[0043] 3 is a schematic diagram of a method for identifying a component within a predetermined region in an image, where the predetermined region is a planar region. As shown in FIG. 3, the method for identifying a component within the predetermined region includes the following steps: an operation 301 of determining location information of the identified part; an operation 302 of searching for parts within a first predetermined radius based on the distance between the parts; and an operation 303 of displaying parts within the first predetermined radius.

[0044] In operation 301, coordinates of the part identified in operation 102 can be looked up in a database corresponding to the part type, and the coordinates can include center coordinates and / or edge coordinates of the part, which can be expressed as (X, Y), for example.

[0045] In operation 302, the predetermined area may be a range corresponding to a first predetermined radius centered on the coordinates identified in operation 301. Within the predetermined area, parts are searched for based on the distance between the parts, where the distance between the parts can be searched, for example, from a database corresponding to the type of part.

[0046] In operation 303, the parts searched for in operation 302 can be displayed to the engineer. For example, related information about the searched parts can be displayed on the engineer's terminal device, such as a mobile phone. The related information can include at least one of the part number, part name, part coordinates, model number specifications, wiring code number, etc.

[0047] If the engineer finds the target part from among the displayed parts, he or she can identify the part by checking it through a selection operation (for example, by clicking on the screen of the terminal device).

[0048] Furthermore, if the engineer does not find the target part from the displayed parts, the range of the specified area can be expanded by adjusting the value of the first specified radius (for example, by zooming the screen), thereby allowing the engineer to search for the part within a wider range.

[0049] 4 is a schematic diagram of component locations within a first predetermined radius range. As shown in FIG. 4, target component O indicates a component identified by the detection model. When the first predetermined radius is r1, the component searched within the predetermined range (indicated by the solid circle) is possible component A. When the first predetermined radius is adjusted to r2, the components searched within the predetermined range (indicated by the dashed circle) are possible components A, B, C, and D. Here, possible component B is a required component (i.e., a required component).

[0050] In FIG. 4, the plane xy can represent a plane parallel to the image surface.

[0051] 5 is a schematic diagram showing parts within a first predetermined radius range. As shown in FIG. 5, in an image 500 acquired in operation 101, bounding boxes 501, 502, 503, and 504 of each part are displayed.

[0052] 6 is another schematic diagram of a method for identifying a component within a predetermined region in an image, where the predetermined region is a three-dimensional space region. As shown in FIG. 6, the method for identifying a component within the predetermined region includes the following steps: an operation 601 of determining location information of the identified part; an operation 602 of searching for a part within a projection range of a second predetermined area including the identified part based on a positional relationship between the identified part and other parts in the imaging direction of the image; and an operation 603 of displaying the parts within the projection range.

[0053] In operation 601, coordinates of the part identified in operation 102 can be looked up in a database corresponding to the part type, and the coordinates can include center coordinates and / or edge coordinates of the part, which can be expressed as (X, Y, Z), for example.

[0054] In operation 602, the second predetermined area may include an area enclosed by the edges of the identified part; for example, the second predetermined area may be equal to the area enclosed by the edges of the identified part, or the second predetermined area may be greater than the area enclosed by the edges of the identified part.

[0055] The range of the second predetermined region in the X direction is, for example, [X-Δx1, X+Δx2], and the range in the Y direction is, for example, [Y-Δy1, Y+Δy2]. The second predetermined region can be adjusted, for example, by adjusting the number of at least one of Δx1, Δx2, Δy1, and Δy2. The range of the projection range of the second region in the Z direction is, for example, [Z-Δz1, Z+Δz2].

[0056] In operation 602, the projection range of the second region is a predetermined region, and within the predetermined region, parts are searched for based on the distance between the parts, where the distance between the parts can be retrieved, for example, from a database corresponding to the type of part.

[0057] In operation 603, the parts found in operation 602 can be displayed to the engineer. For example, related information about the found parts can be displayed on the engineer's terminal device such as a mobile phone, and the related information can include at least one of the part number, part name, part coordinates, model number specifications, wiring code number, etc.

[0058] If the engineer finds the target part from among the displayed parts, he or she can identify the part by checking it through a selection operation (for example, by clicking on the screen of the terminal device).

[0059] Furthermore, if the engineer does not find the target part from the displayed parts, the range of the search can be expanded by adjusting the second predetermined area (for example, by zooming the screen), thereby enabling the part to be searched for within a wider range.

[0060] 7 is a schematic diagram of the location of components within the projection range of the second predetermined region. As shown in FIG. 7, the equipment is an air conditioner, and the target component O indicates the second predetermined region corresponding to the component identified by the detection model. The second predetermined region is within the projection range in the Z direction, and the searched components are possible components A, B, and C.

[0061] In FIG. 7, the plane xy can represent a plane parallel to the image surface.

[0062] In this embodiment, the method of identifying parts in the projection range can be used independently of operations 101 and 102 .

[0063] For example, as shown in Figure 1, the method for identifying parts is as follows: An operation 101a for displaying an image of the exterior of the device corresponding to the model of the device; an operation 102a of receiving a region setting operation for the appearance image; An operation 103a for searching for a part within a projection range of the set area; The method may further include an operation 104a of displaying the searched parts.

[0064] In operation 101a, a database can be searched to identify an image of the exterior of the equipment based on the model of the equipment, and the image of the exterior can be displayed on the engineer's terminal.

[0065] In operation 102a, the engineer can perform a region setting operation on the screen of the terminal, thereby setting a region in the external image, and the set region is, for example, on the xy plane.

[0066] In operation 103a, based on the area set in operation 102a, parts within the projection range of the set area are searched for, and the projection range of the set area is identified, for example, by increasing the coordinate value in the Z direction (the Z direction is perpendicular to the xy plane) of each point within the set area, and further, parts within the projection range are identified based on the data in the database.

[0067] In operation 104a, the parts found in operation 103a may be displayed in the form of an image, or these found parts may be displayed in the form of a list.

[0068] In this embodiment, operations 101a to 104a may be performed in parallel with operations 101 to 103, or may be performed after operation 103.

[0069] In this embodiment, if the type of part is not identified in operation 102, a model diagram of the equipment in which the part is located can be received and displayed, and the model diagram may be, for example, an assembly perspective view of the equipment, and the engineer can identify which part information needs to be obtained based on the model diagram.

[0070] In this embodiment, if part information cannot be determined in operation 103, for example, if part information cannot be determined based on the identified part or if the part cannot be identified, part information can be recommended to the engineer.

[0071] FIG. 8 is a schematic diagram of recommending part information to engineers. As shown in FIG. 8, the method of recommending part information includes: An operation 801 of acquiring actual operating information of the equipment and the usage and operation status of the parts; an operation 802 of predicting a loss state of each part based on the usage and operation state of each part, and estimating a part in which a failure has occurred; and an operation 803 of outputting part information of the estimated part, including a maintenance record of the equipment and / or information for maintaining the estimated part.

[0072] Operations 801 to 803 assist engineers in quickly finding the necessary parts information, thereby improving maintenance efficiency.

[0073] For example, in operations 801 to 803, the recommendation model can analyze the operating information of the equipment to determine the usage time and operating conditions of each general part, such as operating time, operating temperature, and / or operating time, and then sort the analysis results by the likelihood of a failure and push them to the on-site engineer. The engineer can then determine whether the recommended part is the faulty part based on the on-site situation. This allows the engineer to inspect the equipment based on the recommended part, improving inspection efficiency and quickly identifying the cause of the failure. At the same time, the recommended part that may have failed can be detected and replaced if necessary, improving maintenance quality and efficiency.

[0074] Here, the recommendation model may be an artificial intelligence (AI) model, for example, the AI ​​model may use a network model such as supervised learning. The recommendation model can be trained in the following manner: The usage time, operating status, operating parameters, etc. of each part are input into the neural network model, The neural network model outputs a ranking of the loss status of each part, recommends parts based on the ranking, and Learning is repeated until the neural network model converges, and the neural network model is saved as the recommendation model.

[0075] In this embodiment, when part information is acquired in operation 103, it can be further recommended using a recommended algorithm, thereby eliminating interfering part information and improving maintenance accuracy.

[0076] For example, as shown in FIG. 1, the method for identifying the part includes: The method further includes an operation 104 of identifying recommended parts based on the part information.

[0077] 9 is a schematic diagram of how operation 104 is implemented. As shown in FIG. 9, operation 104 includes: An operation 901 of extracting user features and part features from user information and part information; and an operation 902 of obtaining a score value for the user and the part based on the user characteristics and the part characteristics, the score value corresponding to a degree of recommendation, e.g., the higher the score value, the higher the degree of recommendation, or vice versa.

[0078] Operations 901 and 902 can help the engineer filter out noise component information, making it easier for the engineer to determine and more accurately identify the faulty component.

[0079] FIG. 10 is a schematic diagram of extracting part features in operation 901. As shown in FIG. 10, the method of extracting part features includes: An operation 1001 of extracting part information for each part from a parts database, wherein the extracted part information includes at least one of the part number, part name, model, release date, and replaceable part number; an operation 1002 of mapping the part number, release date, and replaceable part number in the part information and processing them using a fully connected neural network to obtain a first feature vector, which is, for example, a 256-dimensional feature vector, for each piece of information about each part; an operation 1003 of acquiring, for each part, a second feature vector, which is, for example, a 256-dimensional feature vector, by performing word vector processing such as embedding processing on the part names using a neural network; an operation 1004 of processing the model by a text convolutional neural network to obtain a third feature vector, which is, for example, a 256-dimensional feature vector; and an operation 1005 of performing a merge process on the first feature vector, the second feature vector, and the third feature vector to obtain a part feature vector corresponding to the part feature.

[0080] FIG. 11 is a schematic diagram of extracting user features in operation 901. As shown in FIG. 11, the method of extracting user features includes: An operation 1101 of extracting user information of a user from a database, where the user is, for example, an engineer who maintains equipment, and the user information includes at least one kind of information such as an identification (ID) number, age, years of service, level, area of ​​affiliation, for example, the organization where the engineer works, and city of location; operation 1102 of mapping the user's ID number, age, years of service, level, area of ​​affiliation, organization of affiliation, and city of location, respectively, and processing the mapping data using a fully connected neural network to obtain, for example, a fourth feature vector, which is a 256-dimensional feature vector; and an operation 1103 of performing a merge process on all the fourth feature vectors in the operation 1002 to obtain a user feature vector corresponding to the user feature.

[0081] In operation 902, the user feature vector and the part feature vector can be input to a fully connected neural network model, and the fully connected neural network model can output a score value for the user and the part. In operation 803, the score value can be made more accurate by also inputting the part usage record into the fully connected neural network.

[0082] If two or more parts are identified in operation 103, the score values ​​corresponding to the parts identified in operation 802 can be ranked in descending order based on the degree of recommendation and displayed to the engineer, for example, by being displayed on the engineer's terminal. If only one part is identified in operation 103, the score value corresponding to the one part identified in operation 802 can be displayed to the engineer, and the engineer can decide whether to select part information for that part based on the score value.

[0083] As shown in FIG. 1, the component identification method according to the first embodiment includes the following steps: The method may further include an operation 105 of identifying recommended parts based on at least one of the fault code, driving data, and image matching results.

[0084] Since information about the part where the failure occurred may be present in the information about the failure code, the recommended part can be identified based on the failure code.

[0085] Furthermore, since the results of comparing the driving data and / or images can also reflect information about the part where the failure occurred, it is possible to identify the part where the failure occurred and recommend the part where the failure occurred based on the results of comparing the driving data and / or images.

[0086] 12 is a schematic diagram of a method for identifying a faulty part based on operational data. As shown in FIG. 12, the method includes: an operation 1201 of obtaining operational data for a part in an image; and an operation 1202 of identifying a faulty component in the image based on the operational data.

[0087] In this embodiment, the image in operation 1201 may be an image acquired by operation 101. For example, the image may be a photograph taken by an engineer performing maintenance on a device using an imaging device such as a camera, a mobile terminal such as a smartphone, or AR glasses. The image may also be acquired by cropping a video captured by the imaging device. For example, if an engineer wants to acquire component information about a component, he or she can use the imaging device to capture an image of the component and its surrounding components.

[0088] In operation 1201, the part in the image may be a part in a particular area of ​​the image, which may be an area selected from the image by a user (e.g., an engineer) using a selection operation or the like, or the particular area may be a predetermined area according to FIG. 3 or FIG. 6.

[0089] In operation 1201, the driving data includes an audio signal generated when the component is driven and / or a vibration signal generated when the component is driven. For example, a user places an audio sensor (e.g., a microphone) at a position corresponding to a selected area in the image to collect an audio signal generated when the component is driven, or brings a vibration sensor into contact with the component to obtain a vibration signal generated when the component is driven. Here, the audio sensor or vibration sensor may be integrated into a mobile terminal such as a mobile phone or a camera, or the audio sensor or vibration sensor may be independent of the mobile terminal and transmit the acquired audio signal or vibration signal to the mobile terminal or the like via a wired or wireless method.

[0090] In operation 1202, the operating data of the part acquired in operation 1201 is compared with operating data during normal operation, and a determination is made as to whether a fault has occurred in the part based on the comparison result, where the fault can be determined based on information such as the amplitude and / or frequency of the waveform of the operating data. For example, if the difference between the amplitude of the waveform of the operating data and the amplitude of the waveform of the operating data during normal operation is greater than a first threshold value, and / or if the difference between the frequency of the waveform of the operating data and the frequency of the waveform of the operating data during normal operation is greater than a second threshold value, it is determined that a fault has occurred in the part. In operation 1202, the portable device can determine whether a fault has occurred in the part, or the portable device can transmit the acquired operating data to a server, and the server can determine whether a fault has occurred in the part.

[0091] Figure 13 is a schematic diagram of a vibration signal when a part is operating normally and when it has failed. Figure 13(A) shows the waveform of the vibration signal when a part is operating normally, Figure 13(B) shows an example of the waveform of the vibration signal when a part has failed, where the amplitude of the vibration signal waveform increases due to the part failure, and Figure 13(C) shows another example of the waveform of the vibration signal when a part has failed, where the frequency of the vibration signal waveform increases due to the part failure. In Figures 13(A), (B), and (C), the horizontal axis represents time and the vertical axis represents the signal amplitude.

[0092] Hereinafter, a method for determining a faulty part based on operation data will be described in one embodiment. An operation S1 of taking a photo of the air conditioner using a mobile phone; operation S2 of selecting an area of ​​the photo by circling it on the screen of the mobile phone; an operation S3 of bringing a sensor close to the selected area or in contact with the part in the area to acquire driving data of the part, the driving data may be a sound signal (acquired by a microphone) or a vibration signal (acquired by a vibration sensor), and the sensor may be a microphone or a vibration sensor built into the mobile phone or an external microphone or vibration sensor to the mobile phone; and an operation S4 of transmitting the acquired operating data to a server for analysis and determining whether or not a failure has occurred in the part.

[0093] In operation 105, identifying the part where the failure occurred based on the image matching result may include comparing position information based on the part in the image with reference position information, and identifying the part where the failure occurred based on the comparison result.

[0094] For example, if the deviation value of the part's position from the reference position is greater than a third threshold, or if the difference between the area of ​​the part blocking other parts at the current position and the area of ​​the part blocking other parts at the reference position is greater than a fourth threshold, it is determined that a failure has occurred in the part. This makes it possible to determine that some parts with small volumes or parts with insufficient operating data have failed, and for example, it can be determined that the part has failed due to a loose screw or other cause.

[0095] Below, two examples of a method for determining a faulty part based on the image matching results will be explained. An operation S11 of downloading a reference image of a specific part (e.g., a part that requires failure determination) from a database; an operation S12 of correcting the transparency of the reference image acquired in the operation S11 and superimposing and displaying it on the captured image screen; an operation S13 of adjusting the imaging angle and imaging range of the mobile phone based on the reference image superimposed and displayed on the imaging screen, and capturing an image of the part having the same imaging angle and imaging range as the reference image by capturing the image; The method includes an operation S14 of comparing the image of the part obtained in operation S13 with a reference image of the part, thereby comparing the position information of the part with the reference position information reflected in the reference image, and further determining whether or not the position of the part is misaligned based on the comparison result, and determining that a failure has occurred in the part if the position is misaligned.

[0096] Example 2 is For a specific component (e.g., a component requiring fault determination), an operation S21 is performed in which images of the component are taken from multiple angles, for example, from left to right and / or from top to bottom using a mobile phone; An operation S22 of uploading the images captured in operation step S21 to a server and analyzing the images of each frame by the server; an operation S23 of extracting an image having the same imaging angle as the reference image of the component from the database based on the analysis result; The method includes an operation S24 of comparing the image of the part obtained in operation S23 with a reference image of the part, thereby comparing position information of the part with reference position information reflected in the reference image, and further determining whether or not the position of the part is misaligned based on the comparison result, and determining that a failure has occurred in the part if the position is misaligned.

[0097] In this embodiment, for the part recommended in operation 104 or operation 105, the engineer can select the part finally confirmed from the recommended parts. Also, if operation 104 or operation 105 is not performed, the part corresponding to the part information specified in operation 103 is the part finally confirmed.

[0098] In this embodiment, the part information of the finally confirmed part can be displayed in a part information field, which can be displayed based on a user operation or a voice command. For example, a user (e.g., an engineer) can call up the part information field by clicking or by voice, and the part information field can be displayed on the screen of the engineer's terminal, and the part information can be displayed in the form of a table, a graph, or the like.

[0099] In this embodiment, a part information detail page related to the part information is displayed based on an operation on the displayed part information (for example, an operation in which an engineer clicks on the part information), and the part information detail page includes an order page that receives the order operation and can send an order to a parts warehouse requesting the supply of the part based on the order operation.

[0100] As shown in FIG. 1, the method for identifying the part includes: an operation 106 of locating a warehouse that supplies the part based on the equipment location and part information and calculating the shortest time to schedule the part; and / or The method further includes an operation 107 of outputting step information and / or a time plan for maintaining the component.

[0101] Here, the maintenance step information may include information such as a video or image of the maintenance and / or installation and / or disassembly of the part.

[0102] Operations 106 and 107 allow the engineer to be provided with a maintenance plan for the equipment based on the part information.

[0103] According to the first embodiment, an image including a part is detected based on a detection model, the type of the part is identified, and the part information is specified. This reduces the time required to check the part information, improves the efficiency of equipment maintenance, and also improves the accuracy of identifying the type of part. Example 2 The second embodiment provides a component identification device for executing the component identification method according to the first embodiment.

[0104] FIG. 14 is a schematic diagram of a component identification device according to a second embodiment. As shown in FIG. 14, the component identification device 1400 includes: an acquisition means 1401 for acquiring an image including a part; an identification means 1402 for detecting an image based on a detection model and identifying the type of part; and an identifying means 1403 for identifying part information based on the identified part type; Here, the operation of identifying part information based on the type of part identified by the identification means 1403 includes identifying a part within a predetermined area in the image based on the type of the identified part, and identifying part information based on the identified part, where the predetermined area includes a planar area or a spatial area.

[0105] For a detailed description of each means in the second embodiment, please refer to the description of the related operations in the first embodiment.

[0106] According to the second embodiment, an image including a part is detected based on a detection model, the type of the part is identified, and the part information is specified. This reduces the time required to check the part information, improves the efficiency of equipment maintenance, and also improves the accuracy of identifying the type of part.

[0107] The controllers described with reference to the embodiments of the present invention may be embodied directly in hardware, as software modules executed by a processor, or a combination of both. These hardware modules may be implemented, for example, using a Field Programmable Gate Array (FPGA) to harden these software modules.

[0108] The software module may be located in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The storage medium may be coupled to the processor, such that the processor can read information from and write information to the storage medium, or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in the memory of the mobile terminal or in a memory card insertable into the mobile terminal. For example, if the electronic device uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or a large-capacity flash memory device.

[0109] The controller described herein may be implemented as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any suitable combination thereof to perform the functions described herein, such as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors communicatively coupled to a DSP, or any other such configuration.

[0110] The embodiment of the present invention further relates to a storage medium, such as a hard disk, a magnetic disk, an optical disk, a DVD, or a flash memory, for storing the above program.

[0111] Furthermore, the limitations on each step in this solution are not deemed to limit the order of the steps, provided that they do not affect the implementation of the specific solution. The previous step may be performed first, later, or even simultaneously. As long as this solution can be implemented, any of these steps shall be deemed to fall within the scope of protection of this application.

[0112] Although the present application has been described above with reference to specific embodiments, those skilled in the art will understand that these descriptions are merely illustrative and do not limit the scope of the claims of the present application. Those skilled in the art may make various modifications and amendments to the present application based on the ideas and principles of the present application, and these modifications and amendments are also within the scope of the present application.

Claims

1. acquiring an image including the part; Detecting the image based on a detection model and identifying the type of the part; Identifying part information based on the identified type of the part and outputting the part information; Including, Identifying the part information based on the identified type of the part includes: Identifying position information of the identified part and identifying parts within a predetermined area based on distances between the identified part and other parts; Identifying the part information based on the identified part; Including, the predetermined region includes a planar region or a spatial region, Identifying the component within the predetermined area includes: determining location information of the identified component; Searching for a part within a first predetermined radius range based on a distance between the part and the other part; Displaying parts within the first predetermined radius; Including, The value of the first predetermined radius is adjustable. A component identification method performed by an identification device.

2. acquiring an image including the part; Detecting the image based on a detection model and identifying the type of the part; Identifying part information based on the identified type of the part and outputting the part information; Including, Identifying the part information based on the identified type of the part includes: specifying position information of the identified component, and specifying a component within a predetermined area based on a positional relationship between the component and other components in an imaging direction of the image; Identifying the part information based on the identified part; Including, the predetermined region includes a planar region or a spatial region, Identifying the component within the predetermined area includes: determining location information of the identified component; searching for a component within a projection range of a second predetermined area including the identified component based on a positional relationship between the component and the other component in an imaging direction of the image; displaying a part within the projection range; Including, the second predetermined area is adjustable; A component identification method performed by an identification device.

3. acquiring an image including the part; Detecting the image based on a detection model and identifying the type of the part; Identifying part information based on the identified type of the part and outputting the part information; Including, Identifying the part information based on the identified type of the part includes: specifying position information of the identified component, and specifying a component within a predetermined area based on a distance between the component and other components or a positional relationship between the component and other components in an imaging direction of the image; Identifying the part information based on the identified part; Including, the predetermined region includes a planar region or a spatial region, If the part information cannot be identified, Acquire actual operating information of the equipment in which the part is located and the usage and operating status of the part; Based on the usage and operation status of each part, the loss status of each part is predicted, and the part where the failure occurred is estimated, and Output the part information of the estimated part, The part information of the estimated part includes a maintenance record of the device and / or information on maintaining the estimated part. A component identification method performed by an identification device.

4. The part information is including at least one of the following: number, specification, wiring code number, supplier, material, price, inventory quantity, and exploded view diagram; 4. The part identification method according to claim 1, wherein the part identification method comprises:

5. The image is acquired by capturing an image using an imaging device or by cropping a video, the image includes two or more overlapping parts; 4. The part identification method according to claim 1, wherein the part identification method comprises:

6. inputting a plurality of training images into the constructed neural network model; outputting the detection result through a fully connected network of the neural network model; constructing a loss function based on the detection result and labeling information for the plurality of learning images; adjusting parameters in the neural network model so as to minimize the loss function and cause the neural network model to converge, and saving the adjusted neural network model as a detection model.

4. The part identification method according to claim 1, wherein the part identification method comprises:

7. Each of the training images has at least one bounding box and a type label corresponding to a part in the bounding box, and the bounding box is the smallest bounding box that encloses a continuous part region.

7. The part identification method according to claim 6.

8. Displaying an image of the exterior of the device corresponding to the model of the device; receiving a region setting operation for the appearance image; Searching for parts within a projection range of the set area; Displaying the found parts; further comprising:

4. The part identification method according to claim 1, wherein the part identification method comprises:

9. acquiring operational data for the component in the image; identifying a faulty component in the image based on the driving data; further comprising:

4. The part identification method according to claim 1, wherein the part identification method comprises:

10. The operation data includes an audio signal generated when the component is operating and / or a vibration signal generated when the component is operating.

10. The part identification method according to claim 9.

11. and further comprising: identifying a failed component in the image based on a comparison result between position information of the component in the image and reference position information.

4. The part identification method according to claim 1, wherein the part identification method comprises:

12. Locating a warehouse that supplies the part based on the equipment location and part information and calculating the shortest time to schedule the part; and / or Further comprising outputting step information and / or a time plan for maintaining the component.

4. The part identification method according to claim 1, wherein the part identification method comprises:

13. further comprising identifying a recommended part based on the part information; 4. The part identification method according to claim 1, wherein the part identification method comprises:

14. Identifying the recommended part based on the part information includes: extracting user features and part features from the user information and the part information; and obtaining a score value corresponding to a recommendation degree for the user and the part based on the user characteristics and the part characteristics. The method for identifying a part according to claim 13.

15. The user information includes at least one of information of an identification (ID) number, age, years of service, level, area of ​​affiliation, organization of affiliation, and city of location of an engineer who maintains the equipment where the part is located; The method for identifying a part according to claim 14.

16. extracting user features and part features based on a fully connected neural network model and obtaining the score values; The method for identifying a part according to claim 14.

17. further comprising identifying a recommended part based on the fault code; 4. The part identification method according to claim 1, wherein the part identification method comprises:

18. The part information is displayed in a part information field, The part information field is displayed based on a user operation or a voice command.

4. The part identification method according to claim 1, wherein the part identification method comprises:

19. further comprising displaying a part information detail page related to the part information based on an operation on the displayed part information; the part information detail page includes an order page, and the order page receives an order operation and transmits an order requesting the supply of the part to a parts warehouse based on the order operation; 4. The part identification method according to claim 1, wherein the part identification method comprises:

Images