Method and apparatus for identifying components

Abstract

The application provides a component identification method and device, the method comprising: acquiring an image containing a component; detecting the image based on a detection model, identifying the type of the component; and determining component information according to the identified type of the component. The application detects the image containing the component based on the detection model, identifies the type of the component, and determines the component information, thereby shortening the time for confirming the component information and improving the efficiency of repairing the equipment.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a method and device for identifying components. Background Technology

[0002] When equipment malfunctions, it is sometimes necessary to replace the damaged parts. For example, an engineer will check the inventory of the damaged part based on its technical name, obtain the part from the inventory, and then replace the damaged part.

[0003] When engineers are repairing equipment on-site, if they encounter a less common or complex component, they need to contact the relevant personnel. The personnel will provide the equipment model diagram (e.g., a photograph). The engineer will then confirm the name of each component on-site based on the model diagram. After confirming the name of the damaged component, the engineer will check the inventory and quantity of the component according to its name or serial number.

[0004] It should be noted that the above introduction to the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of this application and facilitating understanding by those skilled in the art. It should not be assumed that these technical solutions are known to those skilled in the art simply because they have been described in the background section of this application. Summary of the Invention

[0005] For the same type of equipment, such as air conditioners, air purifiers, or humidifiers, different models often have different structures, use different components, and the installation location of the same component may also differ. In addition, the number of components in the equipment is increasing, and the structure is becoming more and more complex.

[0006] The inventors of this application discovered that when engineers repair equipment, they sometimes do not know the names of certain parts, and therefore need to consult relevant personnel or look up information to obtain the part names, and then search for them in the inventory. This process often takes a long time and affects the efficiency of equipment repair.

[0007] To address at least the aforementioned technical problems or similar technical problems, embodiments of this application provide a method and apparatus for identifying components. Based on a detection model, the method detects images containing components, identifies the type of the component, and determines the component information. This can shorten the time required to confirm component information and improve the efficiency of equipment maintenance.

[0008] According to one aspect of the embodiments of this application, a method for identifying a component is provided, the method comprising:

[0009] Get an image containing the component;

[0010] The image is detected based on a detection model to identify the type of the component; and

[0011] Based on the identified component type, component information is determined.

[0012] The component information is determined based on the identified component type, including:

[0013] Based on the identified component type, determine the components within a preset area in the image; and

[0014] Based on the identified components, determine the component information.

[0015] The preset area includes a planar area or a spatial area.

[0016] According to another aspect of the embodiments of this application, a component identification device is provided, the device comprising:

[0017] The acquisition unit acquires an image containing the component;

[0018] The identification unit detects the image based on a detection model and identifies the type of the component; and

[0019] The determining unit determines component information based on the identified type of the component.

[0020] The determining unit determines component information based on the identified component type, including:

[0021] Based on the identified component type, determine the components within a preset area in the image; and

[0022] Based on the identified components, determine the component information.

[0023] The preset area includes a planar area or a spatial area.

[0024] The beneficial effects of the embodiments of this application are as follows: by detecting images containing components based on the detection model, identifying the type of components, and determining component information, the time for confirming component information can be shortened and the efficiency of equipment maintenance can be improved.

[0025] Specific embodiments of this application are disclosed in detail with reference to the following description and accompanying drawings, indicating how the principles of this application can be adopted. It should be understood that the embodiments of this application are not limited in scope. Within the scope of the appended claims, embodiments of this application include many changes, modifications, and equivalents.

[0026] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0027] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, whole, step, or component, but does not exclude the presence or addition of one or more other features, wholes, steps, or components. Attached Figure Description

[0028] The elements and features described in one drawing or embodiment of this application may be combined with elements and features shown in one or more other drawings or embodiments. Furthermore, in the drawings, similar reference numerals denote corresponding parts in several drawings and can be used to indicate corresponding parts used in more than one embodiment.

[0029] The accompanying drawings, which form part of the specification, are used to provide a further understanding of the embodiments of this application and illustrate the implementation methods of this application, together with the textual description, to explain the principles of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without creative effort. In the drawings:

[0030] Figure 1 This is a schematic diagram of the component identification method of Embodiment 1 of this application;

[0031] Figure 2 This is a schematic diagram of a method for training a detection model;

[0032] Figure 3 This is a schematic diagram of a method for determining components within a preset area in an image;

[0033] Figure 4 This is a schematic diagram showing the positions of components within a first preset radius.

[0034] Figure 5 This is a schematic diagram showing the components within a first preset radius range;

[0035] Figure 6 This is another schematic diagram of a method for determining components within a preset area in an image;

[0036] Figure 7 This is a schematic diagram showing the positions of components within the projection range of the second preset area;

[0037] Figure 8 This is a diagram illustrating the process of recommending component information to engineers;

[0038] Figure 9 This is a schematic diagram of one implementation method of operation 104;

[0039] Figure 10This is a schematic diagram of extracting component features in operation 801;

[0040] Figure 11 This is a schematic diagram of extracting user features in operation 801;

[0041] Figure 12 This is a schematic diagram of a method for identifying faulty components based on operational data;

[0042] Figure 13 This is a schematic diagram of the vibration signal of a component during normal operation and the vibration signal during a fault.

[0043] Figure 14 This is a schematic diagram of the component identification device of Embodiment 2. Detailed Implementation

[0044] Referring to the accompanying drawings, the foregoing and other features of this application will become apparent from the following description. Specific embodiments of this application are specifically disclosed in the description and drawings, illustrating partial implementations in which the principles of this application can be adopted. It should be understood that this application is not limited to the described embodiments; rather, it includes all modifications, variations, and equivalents falling within the scope of the appended claims. Various embodiments of this application are described below with reference to the accompanying drawings. These embodiments are merely exemplary and not intended to limit the scope of this application.

[0045] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish different elements by name, but do not indicate the spatial arrangement or chronological order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one or more of the terms listed in association and all combinations thereof. The terms "comprising," "including," "having," etc., refer to the presence of the stated features, elements, components, or assemblies, but do not exclude the presence or addition of one or more other features, elements, components, or assemblies.

[0046] In the embodiments of this application, the singular forms "a," "the," etc., including the plural forms, should be broadly understood as "a kind" or "a class" rather than limited to the meaning of "an"; furthermore, the term "the" should be understood to include both the singular and plural forms, unless the context clearly indicates otherwise. Additionally, the term "according to" should be understood as "at least partially based on…," and the term "based on" should be understood as "at least partially based on…," unless the context clearly indicates otherwise.

[0047] Example 1

[0048] Embodiment 1 of this application provides a method for identifying components.

[0049] Figure 1This is a schematic diagram of the component identification method of Embodiment 1 of this application.

[0050] like Figure 1 As shown, the component identification method includes:

[0051] Operation 101: Obtain an image containing the component;

[0052] Operation 102: Based on the detection model, detect the image and identify the type of component; and

[0053] Operation 103: Determine the component information based on the type of the identified component.

[0054] The information of a component may include at least one of the following: part number, specifications, wiring code, supplier, material, price, inventory quantity, disassembly and installation images, etc.

[0055] This component can be installed in a device, which can be an environmental device, such as an air conditioner, purifier, or humidifier. However, this application is not limited to this; the device can also be other types of devices besides environmental devices.

[0056] According to Example 1, the detection model is used to detect images containing components, identify the type of components, and determine component information. This can shorten the time for confirming component information and improve the efficiency of equipment maintenance. In addition, the accuracy of component type identification is also improved.

[0057] In operation 101 of this embodiment, the image may contain only one component or two or more components. These components may be separated from each other or at least partially overlap. The image may be a photograph taken by an engineer repairing the equipment using a camera, such as a camera, a smartphone, or AR glasses. Alternatively, the image may be a screenshot taken from a video captured by a camera. For example, if an engineer wants to obtain component information for a particular component, they can use a camera to photograph that component and its surrounding components to obtain an image.

[0058] In operation 101, the model information of the device to which the component belongs can be further obtained. Therefore, in operation 102, the image can be detected in conjunction with the model information. Since different device models correspond to different component names and serial numbers, detecting the image in conjunction with the model information can improve the accuracy of component type identification, thereby enhancing performance.

[0059] The device model information can be obtained by scanning the device's QR code or barcode, or by reading the device's RFID tag, and then input into the inspection model of Operation 102. Alternatively, engineers can manually input the device model information. Furthermore, the device model information can also be obtained from the Building Information Model (BIM).

[0060] In operation 102, the detection model can be a neural network-based detection model. For example, a detection model using a Convolutional Neural Network (CNN) + YOLO (YOU ONLY LOOK ONCE) network has good stability and good sensitivity to large amounts of data. Furthermore, this embodiment is not limited to this; the detection model can also be of other types, such as CNN + other types of networks, for example, CNN + Faster R-CNN or CNN + SSD network, etc.

[0061] In operation 102, the detection model can be trained, so that when implementing the component recognition method of this application, the trained detection model can be directly used to detect the image.

[0062] The training method of the detection model is explained below. The detection model is a CNN+YOLO network detection model.

[0063] Figure 2 This is a schematic diagram illustrating the method for training a detection model. For example... Figure 2 As shown, the training method includes:

[0064] Operation 201: Input multiple training images into the constructed neural network model;

[0065] Operation 202: The detection results are output through a fully connected neural network;

[0066] Operation 203: Construct a loss function based on the detection results and the annotation information of the plurality of training images; and

[0067] Operation 204: Adjust the parameters in the neural network model to minimize the loss function so that the neural network model converges. Use the adjusted neural network model as the detection model and save it.

[0068] In this embodiment, during the process Figure 2Before training, multiple images can be taken within a predetermined range for a specific device model, and the components in each image can be labeled. For example, components in the image can be enclosed by bounding boxes, and the type label of each component can be added, thus forming training images. Each bounding box contains one component, and the bounding box is the smallest bounding box surrounding that component's contiguous region. The bounding box is, for example, a rectangle. Furthermore, taking multiple images within the predetermined range avoids missing some components due to shooting angle issues, thus improving training accuracy.

[0069] In operation 201, multiple training images are input into the detection model of the constructed CNN+YOLO network.

[0070] In operation 202, the detection model performs detection on the training image and outputs the detection results from the fully connected network of the neural network. The detection results are, for example, the coordinates of the bounding box in the training image and the type of the part inside the bounding box.

[0071] In operation 203, based on the detection results of operation 202 and the information labeled on the training image (e.g., the coordinates of the labeled bounding boxes in the training image and the types of the parts inside the bounding boxes), a loss function is constructed, which may be called YOLOv3LOSS for example.

[0072] The loss function is used to reflect the magnitude of the error between the detection result of operation 202 and the labeled information. For example, the loss function Loss can be expressed as the following equation (1):

[0073]

[0074] Where f refers to the two input values ​​(e.g., two y values). i The value of y, where one y i This indicates the test result, another y i The difference calculation (representing the labeled information) can be in the form of mean squared error or cross entropy, etc.

[0075] In operation 204, the parameters in the neural network model are adjusted to minimize the loss function, thereby causing the neural network model to converge. The adjusted neural network model is then used as the detection model and saved. For example, the parameter adjustment process in operations 201-203 and operation 204 can be iterated multiple times to minimize the loss function and obtain the final detection model.

[0076] according to Figure 2 The detection model trained by this method can be used in operation 102 to detect the image obtained by operation 101.

[0077] For example, in operation 102, for the image obtained in operation 101, the detection model can output the bounding box coordinates of the components in the image, as well as the type of the components within the bounding boxes. Specifically, for a single image, the detection model can detect one or more bounding boxes.

[0078] For example, in operation 102, the image can be preprocessed, such as by segmentation. For each segmented part of the image, the bounding box coordinates and part type corresponding to that part can be output. The segmentation process can be achieved by classifying pixels, for example, by segmenting the image according to the U-NET method.

[0079] In operation 103, based on the recognition results of operation 102, it can be determined which component information needs to be output, and the corresponding component information can be output. For example, based on the type of the recognized component, the component within a preset area in the image can be determined, and the component information can be determined based on the determined component. The preset area can include a planar area (i.e., a two-dimensional area) or a spatial area (i.e., a three-dimensional area). Therefore, components can be queried within the range corresponding to the preset area, thereby enabling more accurate output of the component information required by the engineer.

[0080] Figure 3 This is a schematic diagram illustrating a method for determining components within a preset region in an image, specifically when the preset region is a planar region. For example... Figure 3 As shown, the method for determining components within a preset area includes:

[0081] Operation 301: Determine the location information of the identified components;

[0082] Operation 302: Based on the distance between components, search for components within a first preset radius range; and

[0083] Operation 303: Display the components within the first preset radius range.

[0084] In operation 301, the coordinates of the component identified in operation 102 can be retrieved from the database corresponding to the component type. These coordinates may include the center coordinates and / or edge coordinates of the component. For example, these coordinates may be represented as (X, Y).

[0085] In operation 302, the preset area can be a range corresponding to a first preset radius centered on the coordinates determined in operation 301. Within this preset area, components are searched based on the distances between them, where the distances between components can be found, for example, from a database corresponding to the component type.

[0086] In operation 303, the components searched in operation 302 can be displayed to the engineer. For example, the relevant information of the searched components can be displayed on the engineer's mobile phone or other terminal device. This relevant information may include at least one of the following: component number, component name, component coordinates, model specifications, wiring code, etc.

[0087] If an engineer finds the target component among the displayed components, they can confirm it by making a selection action (such as clicking on the screen of a terminal device).

[0088] Furthermore, if the engineer does not find the target part among the displayed components, they can adjust (e.g., by scaling the screen) the value of the first preset radius to expand the range of the preset area, thereby enabling the search for parts over a larger area.

[0089] Figure 4 This is a schematic diagram showing the positions of components within a first preset radius. For example... Figure 4 As shown, target part O represents the part identified by the detection model. When the first preset radius is r1, the part searched within the preset range (shown by the solid circle) is possible part A; when the first preset radius is adjusted to r2, the parts searched within the preset range (shown by the dashed circle) are possible parts A, B, C, and D. Among them, possible part B is the required part (i.e., the desired part).

[0090] exist Figure 4 In this context, the plane xy can represent a plane parallel to the image surface.

[0091] Figure 5 This is a schematic diagram showing components within a first preset radius range. For example... Figure 5 As shown, in the image 500 acquired by operation 101, the borders 501, 502, 503 and 504 of each component are displayed.

[0092] Figure 6 This is another schematic diagram illustrating a method for determining components within a preset region in an image, corresponding to the case where the preset region is a three-dimensional spatial region. For example... Figure 6 As shown, the method for determining components within a preset area includes:

[0093] Operation 601: Determine the location information of the identified component;

[0094] Operation 602: Based on the positional relationship between the component and other components in the image shooting direction, search for components within the projection range of a second preset area containing the identified component; and

[0095] Operation 603: Display the components within the projection range.

[0096] In operation 601, the coordinates of the component identified in operation 102 can be retrieved from the database corresponding to the component type. These coordinates may include the center coordinates and / or edge coordinates of the component. For example, these coordinates may be represented as (X, Y, Z).

[0097] In operation 602, the second preset area may include the area enclosed by the edge of the identified component. For example, the second preset area may be equal to the area enclosed by the edge of the identified component, or the second preset area may be larger than the area enclosed by the edge of the identified component.

[0098] The second preset region has a range in the X direction of, for example, [X-Δx1, X+Δx2], and a range in the Y direction of, for example, [Y-Δy1, Y+Δy2]. The second preset region can be adjusted; for example, at least one of Δx1, Δx2, Δy1, and Δy2 can be adjusted. The projection range of the second region in the Z direction is, for example, [Z-Δz1, Z+Δz2].

[0099] In operation 602, the projection range of the second region is the preset region. Within this preset region, components are searched based on the distance between them. The distance between components can be found, for example, from a database corresponding to the component type.

[0100] In operation 603, the components searched in operation 602 can be displayed to the engineer. For example, the relevant information of the searched components can be displayed on the engineer's mobile phone or other terminal device. This relevant information may include at least one of the following: component number, component name, component coordinates, model specifications, wiring code, etc.

[0101] If an engineer finds the target component among the displayed components, they can confirm it by making a selection action (such as clicking on the screen of a terminal device).

[0102] Furthermore, if the engineer does not find the target part among the displayed components, they can adjust (e.g., by scaling the screen) the second preset area to expand the search range, thereby enabling the search for parts in a larger area.

[0103] Figure 7 This is a schematic diagram showing the positions of components within the projection range of the second preset area. For example... Figure 7 As shown, the device is an air conditioner, and the target component O represents the second preset area corresponding to the component identified by the detection model. Within the projection range of the second preset area in the Z direction, the searched components are possible components A, B, and C.

[0104] exist Figure 7In this context, the plane xy can represent a plane parallel to the image surface.

[0105] In this embodiment, the method of identifying components using the projection range can be used independently of operations 101 and 102.

[0106] For example, such as Figure 1 As shown, the method for identifying components may also include:

[0107] Operation 101a displays an image of the appearance of the device corresponding to the model of the displayed device;

[0108] Operation 102a: Receive a region setting operation for the image of the appearance;

[0109] Operation 103a: Search for components within the projection range of the set area; and

[0110] Operation 104a: Display the searched components.

[0111] In Operation 101a, a search can be performed in the database to determine the image of the device's appearance based on the device model, and then the image of the appearance can be displayed on the engineer's terminal.

[0112] In operation 102a, the engineer can perform a region setting operation on the terminal's screen, thereby setting a region on the appearance image, such as on the xy plane.

[0113] In operation 103a, based on the area set in operation 102a, components within the projection range of the set area are searched. For example, the coordinate value of each point in the set area in the Z direction (the Z direction is perpendicular to the xy plane) is increased to determine the projection range of the set area; then, based on the data in the database, components within the projection range are determined.

[0114] In operation 104a, the components searched in operation 103a can be displayed as images or as a list.

[0115] In this embodiment, operations 101a to 104a can be parallel to operations 101 to 103, or they can be located after operation 103.

[0116] In this embodiment, if the type of component is not identified in operation 102, the model diagram of the device in which the component is located can be received and displayed. The model diagram is, for example, an exploded view of the device. The engineer can determine which component's information needs to be obtained based on the model diagram.

[0117] In this embodiment, if the component information cannot be determined in operation 103, for example, if the component information cannot be determined based on the identified component, or if the component cannot be identified, the component information can be recommended to the engineer.

[0118] Figure 8 This is a diagram illustrating the process of recommending component information to engineers. For example... Figure 7 As shown, the methods for recommending component information include:

[0119] Operation 801: Obtain actual operating information of the equipment and the operating conditions of its components;

[0120] Operation 802: Based on the operating conditions of each component, predict the wear and tear of each component and estimate the components that may fail; and

[0121] Operation 803: Output the component information of the estimated component, wherein the component information of the estimated component includes: the maintenance record of the equipment, and / or the maintenance information of the estimated component.

[0122] By operating 801 to 803, engineers can quickly find the required component information, improving maintenance efficiency.

[0123] For example, in operations 801-803, the recommendation model can analyze the equipment's operational information to determine the usage time and conditions of common components, such as operating duration and / or operating temperature and / or operating time. The analyzed data is then prioritized and pushed to on-site engineers according to potential faults. Engineers can then determine whether the recommended components are indeed the faulty ones based on the on-site situation. This allows engineers to perform repairs based on the recommended components, improving repair efficiency, quickly locating the cause of the fault, and simultaneously testing and replacing potentially faulty components if necessary, thus improving both repair quality and efficiency.

[0124] The recommendation model can be an artificial intelligence (AI) model, such as a supervised learning network model. The recommendation model can be trained using the following methods:

[0125] Input the usage time, operating conditions, and running parameters of each component into the neural network model;

[0126] The neural network model outputs a ranking of the losses of each component and recommends components based on this ranking; and

[0127] Repeatedly train the neural network model until it converges, then save the neural network model as the recommendation model.

[0128] In this embodiment, if the component information is obtained in operation 103, a recommendation algorithm can be used to make recommendations, thereby eliminating component information that may cause interference and improving the accuracy of maintenance.

[0129] For example, such as Figure 1 As shown, the method for identifying components further includes:

[0130] Operation 104: Determine the recommended components based on component information.

[0131] Figure 9 This is a schematic diagram of one implementation method for operation 104. For example... Figure 9 As shown, operation 104 may include the following operations:

[0132] Operation 901: Extract user features and component features from user information and component information;

[0133] Operation 902: Based on the user characteristics and the component characteristics, obtain a rating value for the user and the component. This rating value corresponds to the recommendation level. For example, the higher the rating value, the higher the recommendation level, or vice versa.

[0134] By operating 901 and 902, engineers can filter out component information that is considered noise, making it easier for them to make judgments and more accurately identify faulty components.

[0135] Figure 10 This is a schematic diagram of extracting component features in operation 901, such as... Figure 9 As shown, the methods for extracting component features include:

[0136] Operation 1001: Extract component information for each component from the component database. The extracted component information includes at least one of the following: component number, component name, model, release date, and replaceable component number.

[0137] Operation 1002: Map the component number, release time and replaceable component number in the component information and process them through a fully connected neural network. For each piece of information of each component, obtain the first feature vector. The first feature vector can be, for example, a 256-dimensional feature vector.

[0138] Operation 1003: Use a neural network to process the word vectors of the component names, for example, by embedding. Thus, for each component, a second feature vector is obtained, which may be, for example, a 256-dimensional feature vector.

[0139] Operation 1004: Process the text convolutional neural network on the corresponding model to obtain the third feature vector, which is, for example, a 256-dimensional feature vector;

[0140] Operation 1005: Concatenate the first feature vector, the second feature vector, and the third feature vector to obtain the component feature vector corresponding to the component feature.

[0141] Figure 11 This is a diagram illustrating the extraction of user features in Operation 901, such as... Figure 10 As shown, methods for extracting user features include:

[0142] Operation 1101: Extract user information from the database. The user is, for example, an engineer who repairs equipment. The user information includes at least one of the following: identification number, age, length of service, level, region, organization, and city. The organization may be, for example, the firm that the engineer works for.

[0143] Operation 1102: Map the user's ID number, age, length of service, level, region, organization, and city respectively, and process them through a fully connected neural network to obtain the fourth feature vector. For example, the fourth feature vector is a 256-dimensional feature vector.

[0144] Operation 1103: Concatenate all the fourth feature vectors from Operation 1002 to obtain the user feature vector corresponding to the user feature.

[0145] In operation 902, the user feature vector and the component feature vector can be input into a fully connected neural network model, which can output a rating value for the user and the component. Furthermore, in operation 803, the component's usage records can also be input into the fully connected neural network, thereby making the rating value more accurate.

[0146] If there are two or more components identified in operation 103, the rating values ​​corresponding to each component identified in operation 802 can be sorted from highest to lowest according to their recommendation level and displayed to the engineer, for example, on the engineer's terminal. If there is only one component identified in operation 103, the rating value corresponding to that single component identified in operation 802 can be displayed to the engineer, who can then decide whether to select that component based on the rating value.

[0147] like Figure 1 As shown, the component identification method of Embodiment 1 may further include:

[0148] Operation 105: Determine the recommended component based on at least one of the comparison results of fault codes, operating data, and images.

[0149] Since fault codes sometimes contain information about the faulty component, recommended components can be identified based on the fault codes.

[0150] Furthermore, the comparison results of operational data and / or images can also reflect information about the faulty components. Therefore, the faulty components can be identified based on the comparison results of operational data and / or images, and recommendations can be made for the faulty components.

[0151] Figure 12 This is a schematic diagram of a method for identifying faulty components based on operational data. For example... Figure 12 As shown, the method includes:

[0152] Operation 1201: Obtain the operational data of the components within the image;

[0153] Operation 1202: Identify the faulty component within the image based on the operational data.

[0154] In this embodiment, the image in operation 1201 can be the image obtained in operation 101. For example, the image can be a photograph taken by an engineer repairing the equipment using a camera, such as a camera, a smartphone, or AR glasses. Alternatively, the image can be obtained by taking a screenshot from a video captured by a camera. For example, if an engineer wants to obtain component information for a certain part, they can use a camera to photograph the part and its surrounding components to obtain an image.

[0155] In operation 1201, the component within the image can be a component within a specific region of the image. This specific region can be a region selected in the image by a user (e.g., an engineer) through a selection operation, or it can be... Figure 3 or Figure 6 The preset areas involved.

[0156] In operation 1201, the operational data includes: sound signals and / or vibration signals generated during component operation. For example, the user places a sound sensor (e.g., a microphone) at a location corresponding to a selected area in the image to collect sound signals generated during component operation; or, the user contacts a vibration sensor with the component to acquire vibration signals generated during component operation. The sound sensor or vibration sensor can be integrated into a portable terminal such as a mobile phone or camera; or, the sound sensor or vibration sensor can be independent of the portable terminal and transmit the acquired sound or vibration signals to the portable terminal via wired or wireless means.

[0157] In operation 1202, the operating data of the component acquired in operation 1201 can be compared with the operating data during normal operation. Based on the comparison result, it can be determined whether the component has malfunctioned. This malfunction can be determined based on information such as the amplitude and / or frequency of the waveform of the operating data. For example, if the difference between the amplitude of the operating data waveform and the amplitude of the waveform during normal operation is greater than a first threshold, and / or if the difference between the frequency of the operating data waveform and the frequency of the waveform during normal operation is greater than a second threshold, the component is determined to have malfunctioned. In operation 1202, a portable device can determine whether the component has malfunctioned, or the portable device can send the acquired operating data to a server for the server to determine whether the component has malfunctioned.

[0158] Figure 13 This is a schematic diagram of the vibration signal when a component is operating normally and the vibration signal when it is malfunctioning. Figure 13 Figure (A) shows the waveform of the vibration signal when the component is operating normally; Figure 13 (B) shows an example of the waveform of the vibration signal when a component fails, wherein the amplitude of the vibration signal waveform increases due to the component failure. Figure 13 Figure (C) shows another example of a vibration signal waveform when a component fails, where the frequency of the vibration signal waveform increases due to the component failure. Figure 13 In (A), (B), and (C), the horizontal axis represents time, and the vertical axis represents the amplitude of the signal.

[0159] The following example illustrates how to determine faulty components based on operational data. This example includes the following operations:

[0160] S1. Take a photo of the air conditioner using your mobile phone;

[0161] S2. Select an area of ​​the photo on the phone screen;

[0162] S3. Bring the sensor close to the selected area or to the component in the area to obtain the operating data of the component. The operating data can be an audio signal (obtained by a microphone) or a vibration signal (obtained by a vibration sensor). The sensor can be a built-in microphone or vibration sensor of the mobile phone, or an external microphone or vibration sensor of the mobile phone.

[0163] S4. Send the acquired operating data to the server for analysis to determine whether the component has malfunctioned.

[0164] In operation 105, determining the faulty component based on the comparison results of the image may include: comparing the position information of the component in the image with standard position information, and determining the faulty component based on the comparison results.

[0165] For example, if the deviation of a component's position from its standard position exceeds a third threshold, or if the difference between the area of ​​the component obstructing other components in its current position and the area of ​​the component obstructing other components under its standard position exceeds a fourth threshold, then the component is considered to be faulty. Therefore, for some small components or components lacking operational data, it is possible to determine the fault of that component; for example, it is possible to determine a component fault caused by loose screws.

[0166] Below, two examples illustrate the method for identifying faulty components based on image comparison results. Example 1 includes the following operations:

[0167] S11. For a specific component (e.g., a component that requires fault diagnosis), download the standard image of that component from the database;

[0168] S12. Modify the transparency of the standard image obtained in S11 so that it overlaps and is displayed on the shooting screen;

[0169] S13. Based on the standard image superimposed on the shooting screen, adjust the shooting angle and shooting range of the mobile phone and take a picture to obtain an image of the component. The image of the component has the same shooting angle and shooting range as the standard image.

[0170] S14. Compare the image of the component obtained in S13 with the standard image of the component to compare the position information of the component with the standard position information reflected in the standard image. Based on the comparison result, determine whether the position of the component has shifted. If the position has shifted, it is determined that the component has malfunctioned.

[0171] Example 2 includes the following operations:

[0172] S21. For a specific component (e.g., a component that needs to be fault-diagnosed), take pictures of the component from multiple angles, for example, use a mobile phone to take pictures of the component from left to right and / or from top to bottom.

[0173] S22. Upload the images captured in operation S21 to the server, and the server analyzes each frame of the image.

[0174] S23. Based on the analysis results, extract images that have the same shooting angle as the standard images of the component in the database, and use them as images of the component.

[0175] S24. Compare the image of the component obtained in S23 with the standard image of the component to compare the position information of the component with the standard position information reflected in the standard image. Based on the comparison result, determine whether the position of the component has shifted. If the position has shifted, it is determined that the component has malfunctioned.

[0176] In this embodiment, for the components recommended in operation 104 or operation 105, the engineer can select the final confirmed component from the recommended components. Furthermore, if there is no operation 104 or operation 105, then the component corresponding to the component information determined in operation 103 is the final confirmed component.

[0177] In this embodiment, the component information of the finally confirmed component can be displayed in a component information bar, which can be displayed based on user operation or voice command. For example, the user (e.g., an engineer) can activate the component information bar by clicking or by voice, and the component information bar can be displayed on the engineer's terminal screen. The component information bar can display component information in the form of tables or charts.

[0178] In this embodiment, based on an operation performed on the displayed component information (e.g., an engineer clicking on the component information), a component information details page associated with the component information is displayed. This component information details page includes an order page. The order page can receive order placement operations and, based on these operations, send an order to the component library requesting the provision of the component.

[0179] like Figure 1 As shown, the method for identifying components further includes:

[0180] Operation 106: Based on the location of the equipment and the information of the components, determine the warehouse location for providing the components, and calculate the shortest time to dispatch the components; and / or

[0181] Operation 107: Output the steps and / or time plan for repairing this component.

[0182] The maintenance procedure information may include video or image information such as the maintenance and / or installation and / or disassembly of the component.

[0183] By operating 106 and 107, maintenance solutions for the equipment can be provided to engineers based on component information.

[0184] According to Example 1, the detection model is used to detect images containing components, identify the type of components, and determine component information. This can shorten the time for confirming component information and improve the efficiency of equipment maintenance. In addition, the accuracy of component type identification is also improved.

[0185] Example 2

[0186] This embodiment 2 provides a component identification device, which is used to perform the component identification method of embodiment 1.

[0187] Figure 14This is a schematic diagram of the component identification device of Embodiment 2, as shown below. Figure 14 As shown, the identification device 1400 for this component includes:

[0188] Acquisition unit 1401 acquires an image containing the component;

[0189] The identification unit 1402 detects the image based on the detection model and identifies the type of the component; and the determination unit 1403 determines the component information based on the identified component type.

[0190] The operation of determining component information based on the type of the identified component by determining unit 1403 includes: determining the component in a preset area in the image based on the type of the identified component; and determining component information based on the determined component, wherein the preset area includes a planar area or a spatial area.

[0191] For a detailed description of each unit in Embodiment 2, please refer to the description of the relevant operations in Embodiment 1.

[0192] According to Embodiment 2, the detection model is used to detect images containing components, identify the type of components, and determine component information. This shortens the time required to confirm component information and improves the efficiency of equipment maintenance. In addition, the accuracy of component type identification is also improved.

[0193] The controller described in conjunction with embodiments of the present invention can be directly embodied in hardware, a software module executed by a processor, or a combination of both. These hardware modules can be implemented, for example, by embedding these software modules using a field-programmable gate array (FPGA).

[0194] The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. A storage medium can be coupled to the processor, enabling the processor to read information from and write information to the storage medium; or the storage medium can be an integral part of the processor. The processor and storage medium can reside in an ASIC. The software module can be stored in the memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the electronic device uses a high-capacity MEGA-SIM card or a high-capacity flash memory device, the software module can be stored in the MEGA-SIM card or the high-capacity flash memory device.

[0195] The controller described in this embodiment can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described herein. It can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors communicating with a DSP, or any other such configuration.

[0196] The embodiments of the present invention also relate to storage media for storing the above programs, such as hard disks, magnetic disks, optical disks, DVDs, flash memory, etc.

[0197] It should be noted that the limitations on each step involved in this solution are not considered as limiting the order of steps, provided that they do not affect the implementation of the specific solution. The steps listed first can be executed first, later, or even simultaneously. As long as this solution can be implemented, it should be considered to fall within the scope of protection of this application.

[0198] The present application has been described above with reference to specific embodiments. However, those skilled in the art should understand that these descriptions are exemplary and not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on its spirit and principles, and these modifications and variations are also within the scope of the present application.

Claims

1. A method for identifying a component, characterized in that, The method includes: Get an image containing the component; The image is detected based on a detection model to identify the type of the component; and Based on the identified component type, component information is determined and output. The component information is determined based on the identified component type, including: Determine the location information of the identified component; Based on the distance between components in the database corresponding to the type of the identified component, or the positional relationship between the component and other components in the database along the shooting direction of the image, other components different from the identified component within a preset area of ​​the image are determined; and Based on the identified components, determine the component information. The preset area includes a planar area or a spatial area. The database contains the location information of the components, as well as the distances between the components.

2. A method for identifying a component, characterized in that, The method includes: Get an image containing the component; The image is detected based on a detection model to identify the type of the component; and Based on the identified component type, component information is determined and output. The component information is determined based on the identified component type, including: Based on the identified component type, determine the components within a preset area in the image; and Based on the identified components, determine the component information. The preset area includes a planar area or a spatial area. When the component information cannot be determined, Obtain the actual operating information of the equipment in which the component is located, as well as the operating conditions of the component; Based on the operating conditions of each component, predict the wear and tear of each component and estimate which components will fail; and Output the estimated component information. The component information of the estimated component includes: the maintenance records of the equipment, and / or information on the maintenance of the estimated component.

3. The component identification method as described in claim 1 or 2, characterized in that, The information of the component includes at least one of the following: Item number, specifications, wiring code, supplier, material, price, inventory quantity, disassembly and installation images.

4. The component identification method as described in claim 1 or 2, characterized in that, The images were captured by a camera or obtained by extracting video footage. The image contains two or more overlapping components.

5. The component identification method as described in claim 1 or 2, characterized in that, The method further includes: A neural network model is built by inputting multiple training images; The detection result is output through the fully connected network of the neural network. A loss function is constructed based on the detection results and the annotation information of the plurality of training images; and The parameters in the neural network model are adjusted to minimize the loss function, thereby causing the neural network model to converge. The adjusted neural network model is then used as the detection model and saved.

6. The component identification method as described in claim 5, characterized in that, Each training image has: at least one border, and a type label corresponding to the component in the border, wherein the border is the smallest border that surrounds a region of consecutive components.

7. The method for identifying components as described in claim 1 or 2, characterized in that, The method further includes: The image shows the appearance of the device corresponding to the model of the display device; Receive a region setting operation for the image of the appearance; Search for components within the projection range of the defined area; and Displays the components found in the search.

8. The method for identifying components as described in claim 1 or 2, characterized in that, The method further includes: Obtain the operational data of the components within the image; Based on the operational data, faulty components within the image are identified.

9. The component identification method as described in claim 8, characterized in that, The operational data includes: sound signals and / or vibration signals generated by the component during operation.

10. The method for identifying components as described in claim 1 or 2, characterized in that, The method further includes: Based on the comparison between the location information of the components within the image and the standard location information, the faulty components within the image are identified.

11. The method for identifying components as described in claim 1 or 2, characterized in that, The method further includes: Based on the location of the equipment and the information of the components, determine the warehouse location for providing the components, and calculate the shortest time to dispatch the components; and / or Output the steps and / or time plan for repairing the component.

12. The method for identifying components as described in claim 1 or 2, characterized in that, The method further includes: The recommended components are determined based on component information.

13. The component identification method as described in claim 12, characterized in that, The recommended components are determined based on component information, including: Extract user features and component features from user information and component information; Based on the user characteristics and the component characteristics, a rating value is obtained for the user and the component, and the rating value corresponds to the recommendation level.

14. The component identification method as described in claim 13, characterized in that, The user information includes at least one of the following information about the engineer who repairs the equipment where the component is located: identification ID number, age, length of service, level, region, organization, and city.

15. The component identification method as described in claim 14, characterized in that, User features and component features are extracted based on a fully connected neural network model, and the score value is obtained.

16. The method for identifying components as described in claim 1 or 2, characterized in that, The method further includes: The recommended component is determined based on the fault code.

17. The method for identifying components as described in claim 1 or 2, characterized in that, The component information is displayed in the component information column. The component information bar is displayed based on user operation or voice command.

18. The method for identifying components as described in claim 1 or 2, characterized in that, The method further includes: Based on the operation performed on the displayed component information, a component information details page associated with the component information is displayed. The component information details page includes an order page, which receives order placement operations and, based on the order placement operations, sends an order to the component library requesting the provision of the component.

19. A component identification device, characterized in that, The device includes: The acquisition unit acquires an image containing the component; The identification unit detects the image based on a detection model and identifies the type of the component; and The determining unit determines component information based on the identified type of the component. The determining unit determines component information based on the identified component type, including: Determine the location information of the identified component; Based on the distance between components in the database corresponding to the type of the identified component, or the positional relationship between the component and other components in the database along the shooting direction of the image, other components different from the identified component within a preset area of ​​the image are determined; and Based on the identified components, determine the component information. The preset area includes a planar area or a spatial area. The database contains the location information of the components, as well as the distances between the components.

20. A component identification device, characterized in that, The device includes: The acquisition unit acquires an image containing the component; The identification unit detects the image based on a detection model and identifies the type of the component; and The determining unit determines component information based on the identified type of the component. The determining unit determines component information based on the identified component type, including: Based on the type of the identified component, other components in a preset area of ​​the image that are different from the identified component are determined; and Based on the identified components, determine the component information. The preset area includes a planar area or a spatial area. If the component information cannot be determined, the device: Obtain the actual operating information of the equipment in which the component is located, as well as the operating conditions of the component; Based on the operating conditions of each component, predict the wear and tear of each component and estimate which components will fail; and Output the estimated component information. The component information of the estimated component includes: the maintenance records of the equipment, and / or information on the maintenance of the estimated component.

Images