Sewing machine maintenance guidance method, system and equipment based on target detection and medium

Abstract

The invention provides a sewing machine maintenance guidance method, system and equipment based on target detection, and a medium. The method comprises the following steps: acquiring a to-be-detected image; the to-be-detected image is an image of a to-be-detected part of the sewing machine; based on the to-be-detected image, using a trained target detection model to obtain target part category information; the target part category information comprises a target part category and a target frame; judging whether the target part is a universal part or a non-universal part according to the type of the target part, and obtaining a judgment result; and obtaining a corresponding maintenance suggestion according to the judgment result, the target frame and a preset knowledge graph. According to the method, a user can quickly obtain the maintenance suggestion for the identified part only by shooting the image of the to-be-detected part of the sewing machine through the mobile phone APP, and does not need to wait for a professional or manually search maintenance data, so that the convenience and instantaneity of the maintenance service are greatly improved.

Description

Technical Field

[0001] This application belongs to the technical field of mechanical equipment fault diagnosis, and particularly relates to a sewing machine maintenance guidance method, system, device and medium based on object detection. Background Art

[0002] In the rapidly developing garment manufacturing industry, the sewing machine, as an indispensable device on the production line, its stable operation is crucial for ensuring production efficiency and product quality. However, the traditional sewing machine maintenance and repair process often relies on manual inspection and experience judgment, which is not only time-consuming and laborious, but also limited by the professional skills and experience level of maintenance personnel, and may lead to problems such as inaccurate diagnosis and untimely repair. With the rapid development of AI technology and the continuous increase in the popularity of smartphones, it has become possible to carry out fault diagnosis and maintenance guidance through object detection and knowledge graph technology in AI, bringing new solutions to sewing machine maintenance.

[0003] Although there are some mobile applications related to sewing machine maintenance on the current market, most of them still stay at the level of providing static information such as maintenance manuals, video tutorials or simple Q&A. When users encounter problems, they still need to judge the fault location of the sewing machine by themselves and manually find the corresponding repair methods. This process is not only cumbersome, but also prone to low repair efficiency or repair errors due to misjudgment. Summary of the Invention

[0004] This application provides a sewing machine maintenance guidance method, system, device and medium based on object detection, which is used to solve the problems that the fault diagnosis process of the existing sewing machine maintenance and diagnosis method is cumbersome and prone to misjudgment, resulting in low repair efficiency and repair errors.

[0005] In a first aspect, this application provides a sewing machine maintenance guidance method based on object detection, and the method includes: obtaining an image to be detected; the image to be detected is an image of the part to be detected of the sewing machine; obtaining the target part category information based on the image to be detected by using a trained object detection model; the target part category information includes the target part category and the target box; judging whether it is a general component or a non-general component according to the target part category, and obtaining a judgment result; obtaining corresponding maintenance suggestions according to the judgment result, the target box and a preset knowledge graph.

[0006] In an implementation manner of the first aspect, obtaining the target part category based on the image to be detected by using a trained object detection model includes: obtaining the target box and the target probability distribution based on the image to be detected by using a trained object detection model; obtaining the component category with the highest probability according to the target probability distribution as the target part category.

[0007] In an implementation of the first aspect, the training method of the trained object detection model includes: obtaining the original images of the components of the sewing machine based on sewing machines of various models; performing annotation on each of the original images respectively to obtain an annotated image set as the training data set; training a first neural network model according to the training data set to obtain the trained object detection model.

[0008] In an implementation of the first aspect, determining whether it is a general component and a non-general component according to the target part category, and obtaining the determination result includes: making a determination according to the target part category and a preset component attribute list; if the target part category is a general component in the preset component attribute list, the determination result is that the target part category is a general component; if the target part category is a non-general component in the preset component attribute list, the determination result is that the target part category is a non-general component.

[0009] In an implementation of the first aspect, the preset knowledge graph includes the model information of the general components and the corresponding maintenance suggestions, as well as the model information of the non-general components and the corresponding maintenance suggestions; obtaining the corresponding maintenance suggestions according to the determination result, the target box and the preset knowledge graph includes: if the determination result is that the target part category is a general component, obtaining the corresponding maintenance suggestions according to the target part category and the preset knowledge graph; if the determination result is that the target part category is a non-general component, obtaining the corresponding maintenance suggestions according to the target part category, the target box and the preset knowledge graph.

[0010] In an implementation of the first aspect, obtaining the corresponding maintenance suggestions according to the target part category, the target box and the preset knowledge graph includes: identifying the machine model according to the target box to obtain the identification result; if the identification result is that the machine model is not the machine panel, obtaining the corresponding maintenance suggestions according to the target part category and the preset knowledge graph; if the identification result is that the machine model is the machine panel, identifying the panel error characters on the machine panel, and obtaining the corresponding maintenance suggestions according to the identified panel error characters, the target part category and the preset knowledge graph.

[0011] In an implementation of the first aspect, identifying the panel error characters on the machine panel, and obtaining the corresponding maintenance suggestions according to the identified panel error characters, the target part category and the preset knowledge graph includes: performing binarization processing on the machine panel to obtain a binarized processing image; identifying the panel error characters according to the binarized processing image to obtain the corresponding panel error characters; obtaining the corresponding maintenance suggestions according to the panel error characters, the target part category and the preset knowledge graph.

[0012] In a second aspect, the present application provides a sewing machine maintenance guidance system based on object detection. The system includes: an image acquisition module configured to acquire an image to be detected, where the image to be detected is an image of a part of the sewing machine to be detected; an object recognition module configured to obtain object part category information based on the image to be detected by using a trained object detection model, where the object part category information includes an object part category and an object bounding box; a judgment module configured to judge whether it is a general component or a non-general component according to the object part category and obtain a judgment result; and a maintenance advice acquisition module configured to obtain corresponding maintenance advice according to the judgment result, the object bounding box, and a preset knowledge graph.

[0013] In a third aspect, the present application provides an electronic device, which includes: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned sewing machine maintenance guidance method based on object detection.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by an electronic device, it implements the above-mentioned sewing machine maintenance guidance method based on object detection.

[0015] As described above, the sewing machine maintenance guidance method, system, device, and medium based on object detection of the present application have the following beneficial effects:

[0016] The present application acquires an image to be detected, where the image to be detected is an image of a part of the sewing machine to be detected; obtains object part category information based on the image to be detected by using a trained object detection model, where the object part category information includes an object part category and an object bounding box; judges whether it is a general component or a non-general component according to the object part category and obtains a judgment result; and obtains corresponding maintenance advice according to the judgment result, the object bounding box, and a preset knowledge graph. It realizes that the user only needs to take a photo of the sewing machine through the mobile phone APP to quickly obtain maintenance advice for the identified part, without waiting for a professional to arrive or manually searching for maintenance materials, greatly improving the convenience and immediacy of the maintenance service, and solving the problems of cumbersome fault diagnosis process, easy misjudgment, low maintenance efficiency, and maintenance errors in the existing sewing machine maintenance diagnosis methods.

[0017] The technical solution of the sewing machine maintenance guidance based on object detection provided by the present application has good scalability. It is not limited to sewing machines and can also be extended to fault diagnosis and maintenance advice systems for other mechanical equipment or products, and only needs to adjust model training and the construction of the knowledge graph. Description of the Drawings

[0018] Figure 1 Schematic diagram of the hardware application scenario of the sewing machine maintenance guidance method based on object detection described in the embodiments of the present application.

[0019] Figure 2 Schematic diagram of the process of the sewing machine maintenance guidance method based on object detection described in the embodiments of the present application.

[0020] Figure 3 Schematic diagram of the process of the sewing machine maintenance guidance method based on object detection described in another embodiment of the present application.

[0021] Figure 4 Schematic diagram of the annotation of the original image described in the embodiments of the present application.

[0022] Figure 5 Schematic diagram of the JSON file containing two target categories of the sewing needle and the panel described in the embodiments of the present application.

[0023] Figure 6 Schematic diagram of the weight file after training the first neural network model described in the embodiments of the present application.

[0024] Figure 7A Schematic diagram of the detection effect of the sewing needle of the sewing machine described in the embodiments of the present application.

[0025] Figure 7B Schematic diagram of the detection effect of the sewing needle of the sewing machine described in another embodiment of the present application.

[0026] Figure 7C Schematic diagram of the detection effect of the machine panel of the sewing machine described in the embodiments of the present application.

[0027] Figure 7D Schematic diagram of the detection effect of the machine panel of the sewing machine described in another embodiment of the present application.

[0028] Figure 8A Schematic diagram of the detection effect of the machine panel described in the embodiments of the present application.

[0029] Figure 8B Shown as Figure 8A Schematic diagram of the image after binarization processing of the image in

[0030] Figure 9A Shown as Figure 8B Schematic diagram of the error message characters on the panel in

[0031] Figure 9B Shown as Figure 9A Schematic diagram of the recognition result of the error message characters on the panel in

[0032] Figure 10ASchematic diagram of the interface for identifying the sewing machine model QR code described in the embodiments of the present application.

[0033] Figure 10B Schematic diagram of the interface for manually selecting and identifying the sewing machine model described in the embodiments of the present application.

[0034] Figure 11A Schematic diagram of the knowledge graph of the non - general component panel described in the embodiments of the present application.

[0035] Figure 11B Schematic diagram of the knowledge graph of the general component sewing machine needle described in the embodiments of the present application.

[0036] Figure 12 Schematic diagram of the flow of the sewing machine maintenance guidance method based on object detection described in another embodiment of the present application.

[0037] Figure 13 Schematic diagram of the structure of the sewing machine maintenance guidance system based on object detection described in the embodiments of the present application.

[0038] Figure 14 Schematic diagram of the structure of the electronic device described in the embodiments of the present application.

[0039] Description of component labels

[0040] 1 Mobile terminal

[0041] 11 Processor

[0042] 12 Memory

[0043] 13 Input / output device

[0044] 14 Communication transmission device

[0045] 3 Sewing machine maintenance guidance system based on object detection

[0047] 31 Image acquisition module

[0048] 32 Object recognition module

[0049] 33 Judgment module

[0050] 34 Maintenance advice acquisition module

[0051] 4 Electronic device

[0052] 41 Processor

[0053] 42 Memory

[0054] 43 I / O interface

[0055] 44 Communication component

[0056] Steps S1 to S4 Specific implementation manners

[0057] The following uses specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0058] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner. Therefore, only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The types, quantities, and proportions of the components in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0059] Although there are some mobile applications related to sewing machine maintenance on the market currently, most of them still stay at the level of providing static information such as maintenance manuals, video tutorials, or simple Q&A, lacking the function of automatic recognition and intelligent diagnosis for specific fault parts. When users encounter problems, they still need to judge the fault parts of the sewing machine by themselves and manually search for corresponding maintenance methods. This process is not only cumbersome but also prone to misjudgment, resulting in low maintenance efficiency or maintenance errors. To overcome the above deficiencies of the prior art, the present application proposes a sewing machine maintenance guidance method, system, device, and medium based on object detection.

[0060] The following embodiments of the present application provide a sewing machine maintenance guidance method, system, device, and medium based on object detection, which solves the problems of cumbersome fault diagnosis process and easy misjudgment in the existing sewing machine maintenance diagnosis method, resulting in low maintenance efficiency and maintenance errors.

[0061] As Figure 1 shown, this embodiment provides a hardware application scenario of a sewing machine maintenance guidance method based on object detection. The sewing machine maintenance guidance method based on object detection provided by the embodiments of the present application can run on similar devices such as mobile terminals and computer terminals. Taking running on the mobile terminal as an example, Figure 1 is the hardware structure block diagram of the mobile terminal. As Figure 1 shown, the mobile terminal may include: a processor 11 and a memory 12. The processor 11 may be a central processing unit, and the memory 12 is used to store data. Figure 1 The mobile terminal in

[0062] Optionally, the mobile terminal may further include: an input / output device 13 and a communication transmission device 14.

[0063] Optionally, the memory may be used to store computer programs, such as software programs and modules of application software. The memory may include a high-speed random access memory, and may further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely disposed relative to the processor, and these remote memories may be connected to the mobile terminal through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0064] Optionally, the communication transmission device may be used to receive or send data via a network, and the network may include a wireless network provided by a communication provider of the mobile terminal. The communication transmission device may include a NIC (Network Interface Controller), which may be connected to other network devices through a base station so as to communicate with the Internet.

[0065] Optionally, the input / output device provides an interface between the processor and other interface modules, and the other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The input / output device may receive an image of a sewing machine component collected by the mobile phone, so that the mobile terminal executes a sewing machine repair guidance method based on object detection.

[0066] Next, the technical solutions in the embodiments of the present application will be described in detail with reference to the accompanying drawings in the embodiments of the present application.

[0067] As Figure 2 shown, this embodiment provides a sewing machine repair guidance method based on object detection, and the method includes the following steps S1 to S4.

[0068] Step S1, obtain an image to be detected; the image to be detected is an image of a part to be detected of a sewing machine.

[0069] Specifically, the present application uses the camera of a smart phone to take a photo of the faulty part of the sewing machine as the image to be detected.

[0070] Step S2, based on the image to be detected, use the trained object detection model to obtain target part category information; the target part category information includes a target part category and a target box.

[0071] Step S3: Determine whether it is a general component or a non-general component based on the target part category, and obtain the judgment result. Among them, the general component refers to a component with consistent maintenance suggestions, and the non-general component refers to a component with inconsistent maintenance suggestions.

[0072] Step S4: Obtain the corresponding maintenance suggestions according to the judgment result, the target box, and the preset knowledge graph. Among them, the preset knowledge graph is constructed by obtaining the maintenance information of each component according to the maintenance manuals of each model of sewing machine, and the knowledge graph constructed according to the maintenance information of each component. The maintenance information includes component category, maintenance relationship, and / or maintenance attribute.

[0073] Specifically, Figure 3 It shows a schematic flow chart of the sewing machine maintenance guidance method based on object detection described in another embodiment of the present application. As Figure 3 shown, the present application uses the smartphone camera to take a photo of the faulty part of the sewing machine and upload it to the cloud server. The cloud server uses the trained object detection model to automatically identify the sewing machine components in the photo, such as key parts like the needle plate, presser foot, and bobbin case. After identifying the specific part in the photo, it is determined whether the current specific part is a general part or a non-general part of the sewing machine, and corresponding maintenance suggestions, operation guides, or replacement part information are retrieved from the cloud database according to the judgment result, and presented to the user in the form of pictures and texts to help the user quickly locate the problem, understand the cause of the fault, and obtain an effective maintenance plan.

[0074] In an embodiment of the present application, the steps of obtaining the target part category information based on the image to be detected using the trained object detection model include the following steps S21 to S22.

[0075] Step S21: Use the trained object detection model based on the image to be detected to obtain the target box and the target probability distribution.

[0076] Step S22: Obtain the component category with the highest probability according to the target probability distribution as the target part category.

[0077] In an embodiment of the present application, the training method of the trained object detection model includes the following steps S211 to S213.

[0078] Step S211: Obtain the original images of each component of the sewing machine based on each model of sewing machine.

[0079] Step S212: Perform annotation on each of the original images respectively to obtain the annotated image set as the training data set.

[0080] Step S213: Train the first neural network model according to the training data set to obtain the trained object detection model.

[0081] Specifically, in this application, images of various components of sewing machines of different models and in different directions are collected as the original image set for training the first neural network model. The LabelImg annotation software is used to annotate the distribution of the collected original image set to obtain the annotated image set as the training data set. Finally, the first neural network model is trained according to the training data set to obtain the trained object detection model.

[0082] Figure 4 It shows the annotation schematic diagram of the original image described in the embodiment of this application. As Figure 4 shown, in this application, the LabelImg annotation software is used to annotate the distribution of the collected original image set, and an xml file corresponding to the annotated image is generated. The xml file includes information such as the number of detection targets and the coordinates of the target boxes as the training data set for training the first neural network model to obtain the trained object detection model. Among them, LabelImg is a free and open-source image annotation tool mainly used for image annotation work in computer vision tasks. Figure 4 The content of the xml file generated after the annotation of the original image shown is as follows:

[0083] <annotation>

[0084] <folder>Needle< / folder>

[0085] <filename>a-n1.jpg< / filename>

[0086] <path>D:\PycharmProjects\pythonProject18\faster_rcnn\Sewingmachine\Needle\a-n1.jpg< / path>

[0087] <source>

[0088]

[0089] In this application, the obtained labeled image dataset is randomly split into a training dataset and a test dataset, and the Faster R-CNN object detection algorithm is used to train with ResNet50 + FPN as the backbone. Taking the dataset for binary classification of sewing machine needles and panels as an example, as Figure 5 shown is a JSON file (used to determine the number of detection targets and assign index numbers) containing two target categories of needles and panels, Figure 5 and the content of the XML file formed after image annotation in

[0090]

[0091]

[0092] is as follows: Among them, "size" represents the image size, "object" represents the recognized object (i.e., the two targets defined in the JSON file), and "bndbox" is the coordinate of the target box.

[0093] Figure 6 Shown is a schematic diagram of the weight file after training the first neural network model described in the embodiment of this application. As Figure 6 shown, the weight parameter file of the model obtained by training with the training dataset and the Faster R-CNN algorithm in this application is as Figure 6 shown. By loading the model file and designing the corresponding prediction code, image detection can be performed. The detection effect of the sewing machine needle and panel dataset model is as Figures 7A - 7D shown. The model can recognize the categories defined in the JSON file, generate target boxes, and the probability that the image in the detected target box is a certain component category.

[0094] Among them, the full name of Faster R-CNN is Faster Region-based Convolutional Neural Networks, which is an object detection algorithm based on deep learning. It introduces a Region Proposal Network (RPN) on the basis of Fast R-CNN, achieving faster detection speed and higher detection accuracy.

[0095] A JSON (JavaScript Object Notation) file is a lightweight data interchange format, an open standard file format and data interchange format designed based on a subset of JavaScript syntax. It allows people to store and transmit structured data in text format and is easy to read, write, and parse.

[0096] In an embodiment of the present application, it is determined whether it is a general component and a non-general component according to the target part category, and obtaining the determination result includes the following steps S31 to S33.

[0097] Step S31: Make a judgment according to the target part category and the preset component attribute list;

[0098] Step S32: If the target part category is a general component in the preset component attribute list, the judgment result is that the target part category is a general component;

[0099] Step S33: If the target part category is a non-general component in the preset component attribute list, the judgment result is that the target part category is a non-general component.

[0100] Specifically, the preset component attribute list is the component attributes preset according to the maintenance suggestions of sewing machine components, including general components and non-general components. The general components are components with consistent maintenance suggestions, and the non-general components are components with inconsistent maintenance suggestions. For example, when a sewing needle is recognized according to the image to be detected, for all sewing machines, whether it is a flat machine or a split machine, there is only one maintenance suggestion, so the sewing needle is defined as a general component of the sewing machine because there is only one maintenance suggestion for the sewing needle regardless of the machine type. When a fault reporting panel is recognized according to the image to be detected, for sewing machines of different models, the maintenance suggestions for the fault reporting panel are different. At this time, it is necessary to further identify the sewing machine model to determine the final maintenance opinion, so the fault reporting panel is defined as a non-general component of the sewing machine because there are different maintenance suggestions for the fault reporting panel for different sewing machine models.

[0101] It should be noted that the general components include, but are not limited to, components such as sewing needles and rotary shuttles, and the non-general components include, but are not limited to, components such as electronic controls, threading structures, and error reporting panels. This application is not limited thereto, and users can set it by themselves according to needs.

[0102] In an embodiment of the present application, the preset knowledge graph includes the model information of the general components and corresponding maintenance suggestions, as well as the model information of the non-general components and corresponding maintenance suggestions; obtaining the corresponding maintenance suggestions according to the judgment result, the target box and the preset knowledge graph includes the following steps S41 to S42.

[0103] Step S41: If the judgment result is that the target part category is a general component, obtain the corresponding maintenance suggestion according to the target part category and the preset knowledge graph.

[0104] Step S42: If the judgment result is that the target part category is a non-general component, obtain the corresponding maintenance suggestion according to the target part category, the target box and the preset knowledge graph.

[0105] Specifically, for general components, the corresponding maintenance suggestions can be confirmed without the model of the sewing machine. For non-general components, the model of the sewing machine needs to be further confirmed to further confirm the corresponding maintenance suggestions. Therefore, when the target part category is a general component, obtain the corresponding maintenance suggestion according to the target part category and the preset knowledge graph; when the target part category is a non-general component, obtain the corresponding maintenance suggestion according to the target part category, the target box and the preset knowledge graph.

[0106] In an embodiment, the obtaining method of the preset knowledge graph includes: obtaining the maintenance information of each component according to the maintenance manuals of sewing machines of each model, where the maintenance information includes component categories, maintenance relationships, and / or maintenance attributes; constructing a knowledge graph according to the component categories, maintenance relationships, and / or maintenance attributes of each component as the preset knowledge graph.

[0107] In an embodiment of the present application, obtaining the corresponding maintenance suggestion according to the target part category, the target box and the preset knowledge graph includes the following steps S421 to S423.

[0108] Step S421: Identify the machine model according to the target box and obtain the identification result.

[0109] Step S422: If the identification result is that the machine model is a non-machine panel, obtain the corresponding maintenance suggestion according to the target part category and the preset knowledge graph.

[0110] Step S423: If the recognition result shows that the model type is the machine panel, recognize the panel error characters based on the machine panel, and obtain the corresponding maintenance suggestions according to the recognized panel error characters, the target part category, and the preset knowledge graph.

[0111] In an embodiment of the present application, recognizing the panel error characters based on the machine panel and obtaining the corresponding maintenance suggestions according to the recognized panel error characters, the target part category, and the preset knowledge graph includes the following steps S4231 to S4233.

[0112] Step S4231: Perform binarization processing on the machine panel to obtain a binarized image.

[0113] Step S4232: Recognize the panel error characters based on the binarized image to obtain the corresponding panel error characters.

[0114] Step S4233: Obtain the corresponding maintenance suggestions according to the panel error characters, the target part category, and the preset knowledge graph.

[0115] Specifically, for general components and non-general components, the methods for obtaining maintenance suggestions are different. For non-general components, it is necessary to further identify whether the machine model is the machine panel. When the machine model of the target part category is not the machine panel, obtain the corresponding maintenance suggestions according to the target part category and the preset knowledge graph. If the machine model of the target part category is the machine panel, perform binarization processing on the machine panel to obtain a binarized image, and recognize the panel error characters based on the binarized image to obtain the corresponding panel error characters. Finally, obtain the corresponding maintenance suggestions according to the panel error characters, the target part category, and the preset knowledge graph.

[0116] Figure 7A It shows a schematic diagram of the detection effect of the sewing machine needle described in the embodiment of the present application. Figure 7B It shows a schematic diagram of the detection effect of the sewing machine needle described in another embodiment of the present application. Figure 7C It shows a schematic diagram of the detection effect of the sewing machine machine panel described in the embodiment of the present application. Figure 7D It shows a schematic diagram of the detection effect of the sewing machine machine panel described in another embodiment of the present application. As Figures 7A - 7D shown, the trained object detection model can recognize the categories defined in the Figure 5 shown JSON file and generate a target box and the probability that the picture in the detected target box is a certain part category. In particular, for the recognized error panel parts, since the corresponding solutions for different panel error characters are also inconsistent (as shown in Table 1), it is also necessary to recognize the panel error characters on the panel.

[0117] Table 1

[0118]

[0119] Figure 8A It shows a schematic diagram of the detection effect of the machine panel described in the embodiment of the present application. Figure 8B Shown as Figure 8A a schematic diagram of the image after binary processing of the image in Figure 9A Shown as Figure 8B a schematic diagram of the error message characters on the panel in Figure 9B Shown as Figure 9A a schematic diagram of the recognition result of the error message characters on the panel in Figures 8A - 8B As shown, the present application performs binary processing on the machine panel image (as shown in Figure 8B ), cuts out the panel picture (as shown in Figure 9A ), for example, obtained by methods such as fixing the photographing position and then cutting the picture at a fixed ratio. At the same time, the open-source Tesseract-OCR is used to recognize the numbers in the cut picture, and the recognition result is as shown in Figure 9B . Finally, corresponding maintenance suggestions are obtained according to the recognition result, the category of the target part, and the preset knowledge graph. Among them, Tesseract-OCR is an open-source optical character recognition (OCR) engine for recognizing text in pictures and converting it into editable text.

[0120] In one embodiment, the present application determines general components and non-general components after identifying the sewing machine maintenance parts. When a non-general component of the sewing machine is identified, the model of the sewing machine needs to be recognized. Considering the difficulty of model recognition and the scalability of the solution, for example, the picture features of many sewing machines are similar, making it difficult to distinguish specific models. When a new model needs to be added, new model pictures need to be collected and the target detection model needs to be retrained again. The model recognition is difficult and cumbersome. Therefore, the present application creates models using a cloud platform, and recognizes the model of the sewing machine by scanning the QR code on the sewing machine body through a mobile app (for example, a mobile app such as WeChat with a QR code scanning function) or manually selecting the sewing machine model on the mobile app.

[0121] Figure 10A It shows a schematic diagram of the interface for recognizing the QR code of the sewing machine model described in the embodiment of the present application, Figure 10B It shows a schematic diagram of the interface for manually selecting and recognizing the sewing machine model described in the embodiment of the present application. As shown in Figures 10A - 10B , after completing the sewing machine target detection and the judgment of non-general parts, the APP automatically jumps to the interface for scanning the QR code on the machine body or manually selecting the current model. Considering the compatibility of old machines, both methods are provided by the APP for the user to choose (for example, old machines without QR codes are more suitable for manual selection, as shown in Figure 10A as shown, where the manually selected machine model and pictures can be created through Figure 10B the cloud platform shown in

[0122] After completing the above-mentioned process, the cloud platform will push maintenance guidance according to the results of picture target recognition. The maintenance guidance uses the technology of knowledge graph and is stored in the Neo4j database.

[0123] Figure 11A It shows a schematic diagram of the knowledge graph of the non-general component panel described in the embodiment of the present application. Figure 11B It shows a schematic diagram of the knowledge graph of the general component needle of the sewing machine described in the embodiment of the present application. As Figures 11A - 11B shown in the knowledge graph example, the present application extracts entities (such as "needle plate", "feed dog", "thread jamming", etc. in the fault solution statement), relationships ("belong to", "cause", "require", etc., for example, the "thread jamming" fault may "belong to" the feed system problem and "require" cleaning or replacing certain components) and attributes (adding specific attributes to each entity, such as specific descriptions of fault phenomena, recommended maintenance tools, etc.) from the collected maintenance manuals of various models of sewing machines to establish the corresponding knowledge graph, and searches for node content related to the results of picture target recognition in the knowledge graph and generates the corresponding reply (i.e., relevant maintenance suggestions for the identified part).

[0124] Figure 12 It shows a schematic diagram of the process of the sewing machine maintenance guidance method based on object detection described in another embodiment of the present application. As Figure 12 shown, the present application uses the smartphone camera to take pictures of the fault parts of the sewing machine, and uses the object detection model trained with the sewing machine data set to detect and recognize the taken pictures, and automatically recognizes the sewing machine components in the pictures, such as key parts like the needle plate, presser foot, bobbin case, etc. After identifying the specific part in the picture, it is judged whether the current specific part is a general part or a non-general part of the sewing machine. If it is a general part, the corresponding maintenance suggestions are obtained according to the target part category and the preset knowledge graph; if it is a non-general part, the machine model needs to be further identified. If this non-general part is not the machine panel, the corresponding maintenance suggestions are obtained according to the target part category and the preset knowledge graph. If this non-general part is the machine panel, the panel error characters are recognized according to the machine panel, and the corresponding maintenance suggestions are obtained according to the recognized panel error characters, the target part category and the preset knowledge graph. The maintenance suggestions can be presented to the user in the form of pictures, texts, etc., which can help the user quickly locate the problem, understand the cause of the fault and obtain an effective maintenance plan.

[0125] The protection scope of the sewing machine maintenance guidance method based on object detection described in the embodiments of the present application is not limited to the execution order of the steps listed in this embodiment. Any solution achieved by adding or subtracting steps of the prior art and replacing steps according to the principle of the present application is included in the protection scope of the present application.

[0126] The embodiments of the present application further provide a sewing machine maintenance guidance system based on object detection. The sewing machine maintenance guidance system based on object detection can implement the sewing machine maintenance guidance method described in the present application. However, the implementation devices of the sewing machine maintenance guidance method described in the present application include, but are not limited to, the structure of the sewing machine maintenance guidance system listed in this embodiment. Any structural deformation and replacement of the prior art made according to the principle of the present application are included in the protection scope of the present application.

[0127] As Figure 13 shown, this embodiment provides a sewing machine maintenance guidance system based on object detection. The system 3 includes: an image acquisition module 31, an object recognition module 32, a judgment module 33, and a maintenance advice acquisition module 34.

[0128] The image acquisition module 31 is configured to acquire an image to be detected; the image to be detected is an image of a part to be detected of a sewing machine;

[0129] The object recognition module 32 is configured to obtain object part category information based on the image to be detected by using a trained object detection model; the object part category information includes an object part category and a bounding box;

[0130] The judgment module 33 is configured to judge whether it is a general component or a non-general component according to the object part category, and obtain a judgment result;

[0131] The maintenance advice acquisition module 34 is configured to obtain corresponding maintenance advice according to the judgment result, the bounding box, and a preset knowledge graph.

[0132] It should be noted that the functions or operations of the image acquisition module 31, the object recognition module 32, the judgment module 33, and the maintenance advice acquisition module 34 described in the embodiments of the present disclosure correspond one by one to the steps in the vehicle orientation angle calculation method described above, so details are not described herein again.

[0133] In several embodiments provided by the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules / units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or modules or units can be in electrical, mechanical or other forms.

[0134] The modules / units described as separate components may or may not be physically separated. The components shown as modules / units may or may not be physical modules, that is, they can be located in one place or distributed to multiple network units. Some or all of the modules / units can be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, in each embodiment of the present application, the functional modules / units can be integrated in a processing module, or each module / unit can exist physically alone, or two or more modules / units can be integrated in one module / unit.

[0135] Those of ordinary skill in the art should also further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0136] As Figure 14 shown, this embodiment provides an electronic device, and the electronic device 4 includes: a processor 41 and a memory 42;

[0137] The memory 42 is used to store a computer program;

[0138] The processor 41 is used to execute the computer program stored in the memory 42 so that the electronic device 4 executes the sewing machine maintenance guidance method based on object detection as described above.

[0139] Preferably, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0140] This embodiment further includes one or more of an input / output (I / O) interface 43 and a communication component 44.

[0141] The I / O interface provides an interface between the processor and other interface modules, and the other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component is used for wired or wireless communication between the timer and other devices. Wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G, or 4G, or a combination of one or more of them. Accordingly, the communication component may include: a Wi-Fi module, a Bluetooth module, an NFC module.

[0142] The embodiments of the present application also provide a computer-readable storage medium. Those of ordinary skill in the art can understand that all or part of the steps in the method of implementing the above embodiments can be completed by instructing a processor through a program. The program can be stored in a computer-readable storage medium. The storage medium is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrating one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid-state disk (SSD)).

[0143] The embodiments of the present application can also provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, a computer, or a data center to another website, a computer, or a data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.).

[0144] When the computer program product is executed by a computer, the computer executes the method described in the foregoing method embodiments. The computer program product can be a software installation package. In the case where the foregoing method needs to be used, the computer program product can be downloaded and executed on the computer.

[0145] In summary, the method, system, device, and medium for sewing machine maintenance guidance based on object detection described in the present application have the following beneficial effects:

[0146] This application obtains an image to be detected; the image to be detected is an image of a part to be detected of a sewing machine; based on the image to be detected, the target part category information is obtained by using a trained target detection model; the target part category information includes the target part category and the target box; it is judged whether it is a general component or a non-general component according to the target part category, and a judgment result is obtained; according to the judgment result, the target box and a preset knowledge graph, corresponding maintenance suggestions are obtained. It realizes that the user only needs to take a photo of the sewing machine through the mobile phone APP to quickly obtain maintenance suggestions for the identified part, without waiting for a professional to arrive or manually searching for maintenance materials, greatly improving the convenience and immediacy of the maintenance service, and solving the problems of cumbersome fault diagnosis process, easy misjudgment, low maintenance efficiency and maintenance errors in the existing sewing machine maintenance diagnosis methods.

[0147] The technical solution of sewing machine maintenance guidance based on target detection provided by this application has good scalability. It is not limited to sewing machines and can also be extended to the fault diagnosis and maintenance advice systems of other mechanical equipment or products, only by adjusting the model training and the construction of the knowledge graph.

[0148] The descriptions of the processes or structures corresponding to the above respective drawings have their own emphases. For parts not detailed in a certain process or structure, reference can be made to the relevant descriptions of other processes or structures.

[0149] The above embodiments are only illustrative of the principles and effects of this application and are not used to limit this application. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by this application should still be covered by the claims of this application.< / annotation>

Claims

1. A sewing machine maintenance guidance method based on target detection, characterized in that: The method comprises: Acquire an image to be detected; the image to be detected is an image of a part to be detected of the sewing machine; Acquire target part category information based on the image to be detected using a trained target detection model; the target part category information includes a target part category and a target frame; Determine whether the target part is a universal component or a non-universal component according to the target part category, and obtain a determination result; Obtain corresponding maintenance suggestions according to the judgment result, the target box and the preset knowledge graph.

2. The sewing machine maintenance guidance method based on target detection according to claim 1, characterized in that: Acquiring target part category information based on the image to be detected using the trained target detection model includes: Based on the image to be detected, a target frame and a target probability distribution are obtained by using a trained target detection model; According to the target probability distribution, the component category with the largest probability is obtained as the target part category.

3. The sewing machine maintenance guidance method based on target detection according to claim 1, characterized in that: The training method of the trained target detection model includes: Acquire original images of various parts of the sewing machine based on various models of sewing machines; Annotate each of the original images to obtain an annotated image set as a training data set; The first neural network model is trained according to the training data set to obtain a trained target detection model.

4. The sewing machine maintenance guidance method based on target detection according to claim 1, characterized in that: Judging whether the target part category is a common component or a non-common component, and obtaining the judgment result includes: Making a judgment based on the target part category and the preset component attribute list; If the target part category is a common component in the preset component attribute list, the judgment result is that the target part category is a common component; If the target part category is a non-universal component in the preset component attribute list, the judgment result is that the target part category is a non-universal component.

5. The sewing machine maintenance guidance method based on target detection according to claim 1, characterized in that: The preset knowledge graph includes the model information of the common components and the corresponding maintenance suggestions, and the model information of the non-common components and the corresponding maintenance suggestions; Acquiring corresponding maintenance suggestions according to the judgment result, the target frame and the preset knowledge graph includes: If the judgment result is that the target part category is a common component, then obtaining corresponding maintenance suggestions according to the target part category and the preset knowledge graph; If the judgment result is that the target part category is a non-universal component, the corresponding maintenance suggestion is obtained according to the target part category, the target box and the preset knowledge graph.

6. The sewing machine maintenance guidance method based on target detection according to claim 5, characterized in that: Acquiring corresponding maintenance suggestions according to the target part category, the target frame and the preset knowledge graph includes: Perform machine model recognition according to the target frame and obtain recognition results; If the recognition result is that the machine model is not a machine panel, obtaining corresponding maintenance suggestions according to the target part category and the preset knowledge graph; If the recognition result is that the machine model is a machine panel, the panel error character is identified according to the machine panel, and the corresponding maintenance suggestion is obtained according to the identified panel error character, the target part category and the preset knowledge graph.

7. The sewing machine maintenance guidance method based on target detection according to claim 6, characterized in that: Identifying the panel error character according to the machine panel, and obtaining corresponding maintenance suggestions according to the identified panel error character, the target part category and the preset knowledge graph includes: Performing binarization processing on the machine panel to obtain a binarized image; Perform panel error reporting character recognition according to the binary processed image to obtain the corresponding panel error reporting character; The corresponding maintenance suggestion is obtained according to the panel error character, the target part category and the preset knowledge graph.

8. A sewing machine maintenance guidance system based on target detection, characterized in that: include: An image acquisition module is configured to acquire an image to be detected; the image to be detected is an image of a part to be detected of the sewing machine; A target recognition module is configured to obtain target part category information based on the image to be detected using a trained target detection model; The target part category information includes a target part category and a target frame; A judgment module is configured to judge whether the target part is a common component or a non-common component according to the category of the target part, and obtain a judgment result; The maintenance suggestion acquisition module is configured to acquire corresponding maintenance suggestions according to the judgment result, the target box and the preset knowledge graph.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the electronic device executes the sewing machine maintenance guidance method based on target detection as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by an electronic device, the sewing machine maintenance guidance method based on target detection as described in any one of claims 1 to 7 is implemented.

Images