A seam detection method and device, electronic equipment and storage medium

Abstract

The application provides a seam detection method and device, electronic equipment and storage medium. The method comprises: acquiring a to-be-processed image, the to-be-processed image being obtained by photographing a target seam, the target seam being obtained by sewing a plurality of to-be-sewn materials; using a neural network model to detect seam endpoint coordinates in the to-be-processed image to obtain endpoint coordinates of the target seam, the neural network model being obtained by training with seam endpoint coordinates as sample labels; and connecting the endpoint coordinates of the target seam to each other to obtain the target seam. The neural network model obtained by training with line segment endpoints as sample labels is used to detect seam endpoint coordinates in the to-be-processed image, and the endpoint coordinates of the detected target seam are connected to each other to obtain the target seam, thereby avoiding the condition that only the outer edge of the material can be detected by using an infrared edge detection sensor, and effectively identifying the seam of the to-be-sewn material during sewing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, image recognition and image processing, in particular to a seam detection method and device, an electronic device and a storage medium. BACKGROUND

[0002] Currently, the machine using infrared edge detection sensors can only detect the outer edge of the material to be sewn (for example, cloth), such as setting machine, tentering machine, coating machine, cutting bed, cloth laying machine and cloth inspection machine, etc. Specifically, two groups of infrared edge detection sensors are used to emit infrared rays, and the infrared rays are reflected to the receiving tube of the infrared edge detection sensor when encountering the material to be sewn. The two groups of infrared edge detection sensors are usually installed directly above the material to be sewn. If both groups of infrared edge detection sensors can receive the reflected infrared rays, it is confirmed that the material to be sewn has deviated. Similarly, if both groups of infrared edge detection sensors cannot receive the reflected infrared rays, it is confirmed that the material to be sewn has deviated. Only one group of infrared edge detection sensors can receive the reflected infrared rays, and the other group of infrared edge detection sensors cannot receive the reflected infrared rays, it is confirmed that there is no deviation. However, it is difficult to detect the inner edge of the material to be sewn in the edge rolling process, that is, it is difficult to detect the seam between the material to be sewn and another material (for example, trim). SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a seam detection method and device, an electronic device and a storage medium, which can improve the problem that it is difficult to detect the seam between the material to be sewn and another material.

[0004] The embodiments of the present application provide a seam detection method, which comprises: acquiring a to-be-processed image, the to-be-processed image being obtained by photographing a target seam, the target seam being obtained by sewing a plurality of materials to be sewn; detecting a seam endpoint coordinate in the to-be-processed image using a neural network model, the neural network model being obtained by training with a seam endpoint coordinate as a sample label, to obtain an endpoint coordinate of the target seam; and connecting the endpoint coordinates of the target seam to each other to obtain the target seam. In the implementation process of the above scheme, the neural network model obtained by training with a line segment endpoint as a sample label is used to detect the seam endpoint coordinate in the to-be-processed image, and the detected endpoint coordinates of the target seam are connected to each other, so as to identify the target seam of the material to be sewn during sewing, thereby avoiding the condition that the infrared edge detection sensor can only detect the outer edge of the material, and effectively identifying the seam of the material to be sewn during sewing.

[0005] Optionally, in the embodiment of the present application, the neural network model comprises: a plurality of feature extraction branches; and the detection of the seam end point coordinates in the to-be-processed image using the neural network model comprises: feature extraction of the to-be-processed image using the plurality of feature extraction branches respectively to obtain a plurality of image features; fusion of the plurality of image features to obtain image fusion features; and seam end point coordinate mapping of the image fusion features to obtain the seam end point coordinates. In the implementation process of the above scheme, the to-be-processed image is extracted in different feature extraction branches by using the plurality of feature extraction branches in the neural network model for feature extraction, feature fusion and coordinate mapping, so as to effectively increase the accuracy of identifying the seam of the to-be-sewn material during sewing.

[0006] Optionally, in the embodiment of the present application, before the detection of the seam end point coordinates in the to-be-processed image using the neural network model, the method further comprises: obtaining a plurality of sample images and a plurality of sample labels, wherein the sample labels are obtained by labeling the seam end point coordinates in the sample images; and training the neural network using the plurality of sample images as training data and the plurality of sample labels as training labels to obtain the neural network model. In the implementation process of the above scheme, the neural network is trained using the sample labels obtained by labeling the seam end point coordinates in the sample images, so that the obtained neural network model can detect the end point coordinates of the target seam, avoiding the case that the infrared edge detection sensor can only detect the outer edge of the material, and effectively identifying the seam of the to-be-sewn material during sewing.

[0007] Optionally, in the embodiment of the present application, the training of the neural network comprises: predicting the seam end point coordinates in the sample image using the neural network to obtain predicted seam end point coordinates; calculating a loss value between the predicted seam end point coordinates and the seam end point coordinates in the sample label; and training the neural network according to the loss value. In the implementation process of the above scheme, the neural network is trained according to the loss value between the predicted seam end point coordinates and the seam end point coordinates in the sample label, so that the obtained neural network model can detect the end point coordinates of the target seam, avoiding the case that the infrared edge detection sensor can only detect the outer edge of the material, and effectively identifying the seam of the to-be-sewn material during sewing.

[0008] Optionally, in the embodiment of the present application, after obtaining the end point coordinates of the target line seam, the method further comprises: determining whether the inclination angle of the straight line where the end point coordinates of the target line seam are located exceeds a preset angle range; if yes, adjusting the target line seam so that the inclination angle of the straight line where the end point coordinates of the target line seam are located is within the preset angle range. In the implementation process of the above scheme, by adjusting the target line seam when the inclination angle of the target line seam exceeds the preset angle range, the problem that the target line seam deviates and causes the unqualified processing of the material to be sewn is avoided, and the qualified rate of processing the material to be sewn is effectively improved.

[0009] Optionally, in the embodiment of the present application, after obtaining the target line seam, the method further comprises: determining whether the angle difference between the inclination angle of the target line seam and the inclination angle of the preset straight line is greater than a preset angle threshold; if yes, adjusting the target line seam so that the angle difference between the inclination angle of the target line seam and the inclination angle of the preset straight line is less than the preset angle threshold. In the implementation process of the above scheme, by adjusting the target line seam when the angle difference between the inclination angle of the target line seam and the inclination angle of the preset straight line is greater than the preset angle threshold, the problem that the target line seam deviates and causes the unqualified processing of the material to be sewn is avoided, and the qualified rate of processing the material to be sewn is effectively improved.

[0010] Optionally, in the embodiment of the present application, the material to be sewn comprises: cloth, leather and / or plastic.

[0011] The embodiment of the present application also provides a line seam detection device, comprising: a processing image acquisition module, configured to acquire a to-be-processed image, the to-be-processed image being obtained by photographing a target line seam, the target line seam being obtained by sewing a plurality of materials to be sewn; a line seam end point detection module, configured to detect line seam end point coordinates in the to-be-processed image by using a neural network model, and obtain end point coordinates of the target line seam, the neural network model being obtained by training with the line seam end point coordinates as sample labels; and a target line seam acquisition module, configured to connect the end point coordinates of the target line seam to each other, and obtain the target line seam.

[0012] Optionally, in the embodiment of the present application, the neural network model comprises: a plurality of feature extraction branches; and the line seam end point detection module comprises: an image feature acquisition submodule, configured to perform feature extraction on the to-be-processed image by using the plurality of feature extraction branches respectively, and obtain a plurality of image features; a fused feature acquisition submodule, configured to fuse the plurality of image features, and obtain image fused features; and an end point coordinate acquisition submodule, configured to perform line seam end point coordinate mapping on the image fused features, and obtain the line seam end point coordinates.

[0013] Optionally, in the embodiment of the present application, the line joint detection device further comprises: an image label obtaining module, configured to obtain a plurality of sample images and a plurality of sample labels, wherein the sample labels are obtained by labeling line joint endpoint coordinates in the sample images; and a neural network training module, configured to train the neural network by taking the plurality of sample images as training data and taking the plurality of sample labels as training labels, and obtain a neural network model.

[0014] Optionally, in the embodiment of the present application, the neural network training module comprises: an endpoint coordinate prediction submodule, configured to predict line joint endpoint coordinates in the sample images by using the neural network, and obtain predicted line joint endpoint coordinates; a loss value calculation submodule, configured to calculate a loss value between the predicted line joint endpoint coordinates and the line joint endpoint coordinates in the sample labels; and a loss training submodule, configured to train the neural network according to the loss value.

[0015] Optionally, in the embodiment of the present application, the line joint detection device further comprises: an inclination angle judgment module, configured to judge whether an inclination angle of a straight line where the endpoint coordinates of the target line joint are located exceeds a preset angle range; and a target line joint adjustment module, configured to adjust the target line joint if the inclination angle of the straight line where the endpoint coordinates of the target line joint are located exceeds the preset angle range, so that the inclination angle of the straight line where the endpoint coordinates of the target line joint are located is within the preset angle range.

[0016] Optionally, in the embodiment of the present application, the line joint detection device further comprises: a judgment module, configured to judge whether an angle difference between the inclination angle of the target line joint and the inclination angle of the preset straight line is greater than a preset angle threshold; and an adjustment module, configured to adjust the target line joint if the angle difference between the inclination angle of the target line joint and the inclination angle of the preset straight line is greater than the preset angle threshold, so that the angle difference between the inclination angle of the target line joint and the inclination angle of the preset straight line is less than the preset angle threshold.

[0017] Optionally, in the embodiment of the present application, the material to be sewn comprises: cloth, leather and / or plastic.

[0018] The embodiment of the present application also provides an electronic device, comprising a processor and a memory, wherein the memory stores machine readable instructions executable by the processor, and the machine readable instructions are executed by the processor to perform the method described above.

[0019] The embodiment of the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the method described above.

[0020] Other features and advantages of the embodiment of the present application will be described in the following description, and some will become apparent from the description, or will be understood by those skilled in the art through implementation of the embodiment of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced as follows. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0022] Figure 1 The flowchart of the seam detection method provided by the embodiments of the present application is shown.

[0023] Figure 2 The schematic diagram of the obtained image to be processed provided by the embodiments of the present application is shown.

[0024] Figure 3 The schematic diagram of the detection process of the seam end point provided by the embodiments of the present application is shown.

[0025] Figure 4 The structural schematic diagram of the neural network model provided by the embodiments of the present application is shown.

[0026] Figure 5 The detailed structural diagram of the neural network model provided by the embodiments of the present application is shown.

[0027] Figure 6 The flowchart of training the neural network model provided by the embodiments of the present application is shown.

[0028] Figure 7 The flowchart of adjusting the target seam of the material to be sewn provided by the embodiments of the present application is shown.

[0029] Figure 8 The structural schematic diagram of the seam detection device provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0030] The technical solutions of the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments in the present application. Based on the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0031] It can be understood that "first", "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art can understand that "first", "second" and the like do not limit the number and execution order, and "first", "second" and the like do not necessarily mean different.

[0032] Before introducing the line joint detection method provided by the embodiments of the present application, some concepts involved in the embodiments of the present application are introduced:

[0033] Machine vision (Machine Vision) is a detection machine equipped with sensing vision instrument, for example, an automatic focusing camera or a sensor, in which the optical detection instrument occupies a very high proportion, which can be used to detect defects of various products, or to judge and select objects, or to measure size, etc.

[0034] Image recognition (Image Recognition) is a technology that uses a computer to process, analyze and understand images to identify various different pattern target objects; common image recognition tasks include character recognition, target detection and semantic segmentation images, etc.

[0035] It should be noted that the line joint detection method provided by the embodiments of the present application can be executed by an electronic device, which refers to a device terminal or a server with the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant or a mobile Internet device, etc. The server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, including a mainframe, a minicomputer and a UNIX server.

[0036] The application scenarios of the line joint detection method are introduced as follows, which include but are not limited to: in various processes of the material to be sewn, a conveyor line is used to transport the material to be sewn, the line joint detection method can be used to detect the line joint between the material to be sewn and another material (such as an edge decoration), and the line joint can be used to correct the deviation of the material to be sewn in the conveying process, so as to avoid problems such as uneven edge rolling, and improve the processing and manufacturing quality of the material to be sewn. The above-mentioned material to be sewn includes but is not limited to cloth, leather and / or plastic, etc. For the convenience of understanding and description, the cloth is taken as an example for detailed description. The above-mentioned various processes include but are not limited to dyeing, drying, shaping, edge rolling and / or printing processes of the cloth, etc.

[0037] Of course, in the specific practice process, the thread seam detection method can also be used to enhance the function of the material processing machine, and the thread seam detection capability of the material processing machine is enhanced, including but not limited to: setting machine, tentering machine, coating machine, cutting bed, cloth laying machine and cloth inspection machine and the like.

[0038] Please refer to Figure 1 The flowchart of the thread seam detection method provided by the embodiment of the application is shown. The main idea of the thread seam detection method is to use a neural network model to detect the thread seam endpoint coordinates in the image to be processed. Since the neural network model is trained with line segment endpoint as sample label, the neural network model can detect the thread seam endpoint coordinates. Finally, the thread seam endpoint coordinates are connected, and the target thread seam is obtained. The implementation of the above thread seam detection method can include:

[0039] Step S110: obtaining an image to be processed, the image to be processed is obtained by photographing the target thread seam, and the target thread seam is obtained by sewing a plurality of materials to be sewn.

[0040] Please refer to Figure 2 The schematic diagram of the obtained image to be processed provided by the embodiment of the application is shown. The image to be processed refers to the image that needs to detect the target thread seam, which can be obtained by photographing the target thread seam. The image to be processed can include: the first material to be sewn (such as the edge decoration) and the second material to be sewn (such as the curtain), and sometimes some image backgrounds unrelated to the material to be sewn are also photographed. It can be understood that the first material to be sewn and the second material to be sewn can be different kinds of materials, or can be the same kind of materials with different patterns, for example: Figure 2 The curtain and the edge decoration in the above are the same kind of materials with different patterns, and the connecting line between the curtain and the edge decoration is the target thread seam (also called the inner edge) to be detected, and the other side of the edge decoration away from the curtain is the edge (also called the outer edge or the outer edge). Of course, in the specific implementation process, the first material to be sewn and the second material to be sewn can also be the same kind of materials with the same pattern, for example, the curtain and the edge decoration are the same kind of materials with the same pattern, but the thickness or manufacturing method of the edge decoration can be different from that of the curtain.

[0041] The obtaining manner of the image to be processed in step S110 is, for example: a first obtaining manner, using a terminal device such as a camera, a video recorder or a color camera to capture a target seam in the material to be sewn to obtain the image to be processed; then the terminal device sends the image to be processed to the electronic device, and then the electronic device receives the image to be processed sent by the terminal device, and the electronic device can store the image to be processed in a file system, a database or a mobile storage device; a second obtaining manner, obtaining a pre-stored image to be processed, for example: obtaining the image to be processed from a file system, a database or a mobile storage device; a third obtaining manner, using a browser or other application to obtain the image to be processed on the Internet.

[0042] Step S120: detecting the seam end point coordinates in the image to be processed using the neural network model to obtain the end point coordinates of the target seam, wherein the neural network model is trained using the seam end point coordinates as sample labels.

[0043] Referring to Figure 3 The detection process of the seam end point provided by the embodiment of the application is shown in the figure; the seam end point coordinates refer to the two end point coordinates of the target seam, for example Figure 3 The seam end point coordinates between the curtain and the trim detected in the embodiment can include the upper seam end point coordinates and the lower seam end point coordinates.

[0044] Step S130: connecting the end point coordinates of the target seam to each other to obtain the target seam.

[0045] The implementation of step S130 is, for example: connecting Figure 3 The seam end point coordinates between the curtain and the trim detected in the embodiment are connected to each other, that is, the upper seam end point coordinates and the lower seam end point coordinates are connected to each other, so that the target seam between the first material to be sewn (for example, the trim) and the second material to be sewn (for example, the curtain) is obtained.

[0046] In the implementation process of the above scheme, the neural network model trained using the line segment end point as the sample label is used to detect the seam end point coordinates in the image to be processed, and the end point coordinates of the target seam detected are connected to each other, so that the target seam of the material to be sewn during sewing is recognized, which avoids the case that the infrared edge detection sensor can only detect the outer edge of the material, and effectively recognizes the seam of the material to be sewn during sewing, thereby increasing the accuracy of recognizing the seam of the material to be sewn during sewing.

[0047] Referring to Figure 4The diagram shows a schematic of the neural network model provided in this application embodiment. The neural network model may include: multiple feature extraction branches, an attention fusion module, and a feature mapping module. The multiple feature extraction branches, attention fusion module, and feature mapping module are connected sequentially. The multiple feature extraction branches may include: a first feature extraction branch, a second feature extraction branch, and a third feature extraction branch. The first feature extraction branch, the second feature extraction branch, and the third feature extraction branch are connected in parallel with the attention fusion module.

[0048] Understandable Figure 4 The first and second feature extraction branches shown in the diagram both process the image to be processed. In practice, the second feature extraction branch may process only the binarized image, or it may process both the image to be processed and the binarized image simultaneously. Therefore, the target image processed by the second feature extraction branch should not be construed as a limitation on the embodiments of this application. As a first optional implementation of the above step S120, the process of detection using a neural network model may include:

[0049] Step S121: Use multiple feature extraction branches to extract features from the image to be processed to obtain multiple image features.

[0050] It is understandable that the aforementioned multiple feature extraction branches can include: a first feature extraction branch, a second feature extraction branch, and a third feature extraction branch. This can be achieved by first binarizing the image to be processed to obtain a binarized image, and then using the first, second, and / or third feature extraction branches to extract features from both the image to be processed and the binarized image, respectively, to obtain multiple image features. It is easy to see that feature extraction from the image to be processed and the binarized image includes many cases: The first case uses the first feature extraction branch to extract features from the image to be processed, and uses the second and third feature extraction branches to extract features from the binarized image. The second case (… Figure 4 In the case depicted in the diagram, the first and second feature extraction branches are used to extract features from the image to be processed, and the third feature extraction branch is used to extract features from the binarized image. In the third case, the first and third feature extraction branches are used to extract features from the image to be processed, and the second feature extraction branch is used to extract features from the binarized image. The above only lists a few cases where processing is not repeated. In practice, there are also cases where processing is repeated, many of which are too numerous to list, and therefore will not be elaborated upon further.

[0051] Step S122: Use the attention fusion module to fuse multiple image features to obtain image fusion features.

[0052] The attention fusion module refers to a neural network module for fusing multiple image features. The attention fusion module can specifically use a multi-head attention (Multi-Head Attentions) neural network.

[0053] Step S123: mapping the image fusion feature to a seam end point coordinate using a feature mapping module to obtain a seam end point coordinate.

[0054] The feature mapping module refers to mapping the image fusion feature to a seam end point coordinate. The feature mapping module can use various neural networks, such as a cascading pyramid network (CPN), a convolutional neural network (CNN), or a recurrent neural network (RNN).

[0055] Please refer to Figure 5 The detailed structure diagram of the neural network model provided by the embodiment of the application is shown. The plurality of feature extraction branches can include a first feature extraction branch, a second feature extraction branch, and a third feature extraction branch. Therefore, the image features include but are not limited to the first image features extracted by the first feature extraction branch, the second image features extracted by the second feature extraction branch, and the third image features extracted by the third feature extraction branch. The main function of the first feature extraction branch is to extract global feature information of the image to be processed. The main function of the second feature extraction branch is to extract local feature information of different scales of the image to be processed. The main function of the third feature extraction branch is to extract edge feature information (such as target seams and edges, etc.) in the binary image. Finally, the edge feature information, the global feature information, and the local feature information of different scales are mutually complementary and fused. These feature expressions can achieve the advantage of mutual complementation, thereby increasing the accuracy of seam detection.

[0056] The implementation of the above step S121 can include:

[0057] Step S121a: extracting features of the image to be processed using the first feature extraction branch to obtain first image features.

[0058] Referring to Figure 5The first feature extraction branch described above mainly extracts the global image features of the image to be processed, and can also be used to remove background information and enhance edge information. The first feature extraction branch can include a first encoder, a first convolutional layer, and a first self-attention layer; wherein the first encoder, the first convolutional layer, and the first self-attention layer are connected in sequence, and the first convolutional layer can be a neural network composed of multiple deconvolutions. The neural network composed of multiple deconvolutions is mainly used to decode the first convolutional features obtained by the first encoder, so the first convolutional layer can also be understood as a decoder.

[0059] The implementation of step S121a described above, for example, encodes the image to be processed using a first encoder to obtain first encoding features; then, the first encoding features are convoluted using a first convolutional layer to obtain first convolutional features; finally, the first convolutional features are processed using a first self-attention layer to obtain first image features.

[0060] Step S121b: using a second feature extraction branch to extract features from the image to be processed to obtain second image features.

[0061] Reference Figure 5 The second feature extraction branch described above can include multiple encoding branches, each of which includes a second encoder, a second convolutional layer, a cross-attention layer, and a feedforward neural network (FNN); wherein the second encoder, the second convolutional layer, the cross-attention layer, and the feedforward neural network are connected in sequence. The main function of the second feature extraction branch is to extract local detail features of different scales from the image to be processed, for example, the first encoding branch mainly extracts local detail features related to the target seam (or curtain), and the second encoding branch mainly extracts local detail features related to the edge (or trim). The number of cross-attention layers and feedforward neural networks described above can be set according to specific conditions (in order to save space, Figure 5 only 2 cross-attention layers and 2 FNNs are shown), for example, it can be set to Figure 5 3 times in , that is, 6 cross-attention layers and 6 FNNs are provided, or it can be set to Figure 5 5 times in , that is, 10 cross-attention layers and 10 FNNs are provided.

[0062] The implementation of the above step S121b is, for example: using the second encoder, the second convolutional layer, the cross-attention layer and the feedforward neural network in the first encoding branch (i.e., encoding branch 1) to sequentially perform operation processing on the to-be-processed image to obtain the first processing feature; then, using the second encoder, the second convolutional layer, the cross-attention layer and the feedforward neural network in the second encoding branch (i.e., encoding branch 2) to sequentially perform operation processing on the to-be-processed image to obtain the second processing feature; and finally, performing fusion processing on the first processing feature and the second processing feature to obtain the second image feature; wherein the fusion processing manner herein is various, including but not limited to: concatenate fusion, weighted fusion, mean fusion and / or direct addition fusion, etc.

[0063] The cross-attention layer has the function of cross-attention processing, that is, the input is the query matrix, the key matrix and the value matrix from top to bottom, and the cross-attention layer calculates the query matrix according to its own data mode, and calculates the key matrix and the value matrix from the opposite mode, thereby realizing the function of cross-attention. The above attention mechanism can be expressed by the formula as follows: Wherein, q represents the query matrix, k represents the key matrix, v represents the value matrix, and Attention(q, k, v) represents the attention calculation of the query matrix, the key matrix and the value matrix, that is, given a query matrix, the attention matrix is calculated with the key matrix, and the attention matrix is attached to the value matrix, thereby obtaining the final attention value (i.e., Attention value). That is, to prevent the result from being too large, a scale factor d k The vector dimensions of the key matrix and the value matrix are used, and the softmax function is used to normalize the result into a probability distribution, and finally multiplied by the value matrix, thereby obtaining the final attention value (i.e., Attention value).

[0064] Step S121c: first, the to-be-processed image is binarized to obtain a binarized image, and then the third feature extraction branch is used to extract features from the binarized image to obtain the third image feature.

[0065] Referring to Figure 5The third feature extraction branch described above mainly extracts edge information (such as target seams and edges, etc.) in the binarized image, and can also be used to remove background information and enhance edge information. The third feature extraction branch can include a third encoder, a third convolutional layer, and a third self-attention layer, wherein the third encoder, the third convolutional layer, and the third self-attention layer are connected in sequence. The main role of the third feature extraction branch is to extract edge information (such as target seams and edges, etc.) in the binarized image.

[0066] The implementation of step S121c described above, for example, first binarizes the image to be processed to obtain a binarized image, and then uses the third encoder, the third convolutional layer, and the third self-attention layer in the third feature extraction branch to sequentially perform feature extraction and operation processing on the binarized image to obtain third image features. It should be noted that the encoder (including the first encoder, the second encoder, and / or the third encoder) described above can use a neural network composed of multiple convolutional layers, or can use a backbone neural network, such as a VGG network, a ResNet network, or a Mobile network. The VGG network that can be used includes, for example, VGG16 or VGG19. The ResNet network that can be used includes, for example, ResNet12, ResNet18, ResNet50, or ResNet101.

[0067] It can be understood that the input of the third feature extraction branch described above is a binarized image, and the main role of the third feature extraction branch is to extract edge information (such as target seams and edges, etc.) in the binarized image, and the third self-attention layer in the third feature extraction branch can automatically focus on the target seam, thereby effectively increasing the accuracy of identifying the seam of the material to be sewn during sewing.

[0068] Referring to Figure 5 The attention fusion module described above can include a multi-head attention (Multi-Head Attentions) neural network. The implementation of step S122 described above, for example, uses the multi-head attention neural network in the attention fusion module to fuse multiple image features to obtain image fusion features.

[0069] Referring to Figure 5The feature mapping module can include a feedforward neural network layer and a feature mapper. The implementation of step S123 can include, for example, performing feedforward operation on the image fusion feature using the feedforward neural network layer in the feature mapping module to obtain a feedforward operation feature, and then performing seam end point coordinate mapping on the feedforward operation feature using the feature mapper to obtain the seam end point coordinates. The feature mapper can be a cascaded pyramid network (CPN), a convolutional neural network (CNN), or a recurrent neural network (RNN), etc.

[0070] In the implementation of the above scheme, the image features of the to-be-processed image in different layers (e.g., the second image feature after fusion of the first encoding branch and the second encoding branch, the image feature extracted by the self-attention of the first feature extraction branch, and the image feature extracted by the self-attention of the third feature extraction branch) are extracted, and the cross-attention mechanism (e.g., the cross-attention layer) and the multi-head attention mechanism (e.g., the multi-head attention neural network) are combined to obtain more rich features, thereby effectively increasing the accuracy of identifying the seam of the to-be-sewn material during sewing.

[0071] As a second optional implementation of step S120, the process of detecting the seam end point using a deep neural network (DNN) can include, for example, using a VGG network, a ResNet network, a Wide ResNet network, and an Inception network, etc. to detect the seam end point coordinates in the to-be-processed image to obtain the end point coordinates of the target seam. The Wide ResNet network can be, for example, a Wide ResNet-28-10 network, which is sometimes abbreviated as WRN-28-10. The Inception network can be, for example, an Inception v1, an Inception v2, or an Inception v3.

[0072] Please refer to Figure 6 The training process of the neural network model can include, for example, obtaining a plurality of sample images and a plurality of sample labels, the sample labels being obtained by labeling the seam end point coordinates in the sample images, and then training the neural network model using the sample images and the sample labels.

[0073] Step S210: Obtain a plurality of sample images and a plurality of sample labels, the sample labels being obtained by labeling the seam end point coordinates in the sample images.

[0074] The plurality of sample images and the plurality of sample labels can be obtained separately, for example, by manually collecting the plurality of sample images and manually identifying the plurality of sample labels of the plurality of sample images. Of course, the plurality of sample images and the plurality of sample labels can be packaged as a training data set and obtained together. Here, the training data set is obtained together as an example for description.

[0075] In the step S210, the training data set can be obtained in the following ways. In a first way, the training data set is received from other terminal devices and stored in a file system, a database or a mobile storage device. In a second way, the training data set is obtained from the file system, the database or the mobile storage device. In a third way, the training data set is obtained from the Internet using a browser or other software, or obtained from the Internet using other application programs.

[0076] In step S220, the neural network is trained using the plurality of sample images as training data and the plurality of sample labels as training labels to obtain a neural network model.

[0077] As an optional implementation of the step S220, the implementation of training the neural network can include:

[0078] In step S221, the neural network is used to predict the coordinates of the seam end points in the sample image to obtain predicted coordinates of the seam end points.

[0079] In step S222, the loss value between the predicted coordinates of the seam end points and the coordinates of the seam end points in the sample label is calculated.

[0080] The implementation of the step S222 can be, for example, using a softmax loss function, a focal loss function or an ArcFace loss function to calculate the loss value between the predicted coordinates of the seam end points and the coordinates of the seam end points in the sample label. The ArcFace loss function is represented by the following formula:

[0081]

[0082] where L ArcFace represents the loss value calculated by the ArcFace loss function, and N is the length of the output vector of the neural network. represents the angle between the output vector of the neural network and the i-th column of the parameter weight vector, and s and m are hyperparameters for adjusting the size of the ArcFace loss function, which reduces the loss value of the positive sample (i.e., the sample image containing the target seam) while increasing the loss value of the negative sample (i.e., the sample image not containing the target seam).

[0083] Step S223: training the neural network according to the loss value to obtain a neural network model.

[0084] For example, the implementation of step S223 is to update the network weight parameters of the neural network according to the loss value until the accuracy of the neural network no longer increases or the number of iterations (epochs) is greater than a preset threshold, and then the neural network model is obtained. The preset threshold can be set according to specific conditions, for example, 100 or 1000, etc.

[0085] Please refer to Figure 7 The flowchart of adjusting the target seam of the material to be sewn provided by the embodiment of the application is shown. As an optional implementation of the above seam detection method, after obtaining the end point coordinates of the target seam, the target seam of the material to be sewn can also be adjusted according to the end point coordinates. This implementation can include:

[0086] Step S310: determining whether the inclination angle of the straight line where the end point coordinates of the target seam are located exceeds a preset angle range.

[0087] For example, the implementation of step S310 is to use an executable program compiled or interpreted by a preset programming language to determine whether the inclination angle of the straight line where the end point coordinates of the target seam are located exceeds a preset angle range. The programming language that can be used includes C, C++, Java, BASIC, JavaScript, LISP, Shell, Perl, Ruby, Python, and PHP, etc. The preset angle range can be set according to specific conditions, for example, 5 degrees clockwise to -5 degrees counterclockwise, 10 degrees to -10 degrees, etc.

[0088] Step S320: if the inclination angle of the straight line where the end point coordinates of the target seam are located exceeds the preset angle range, adjusting the target seam so that the inclination angle of the straight line where the end point coordinates of the target seam are located is within the preset angle range.

[0089] For example, the implementation of step S320 is to adjust the target seam if the inclination angle of the straight line where the end point coordinates of the target seam are located exceeds the preset angle range, so that the material to be sewn (on the conveyor belt) no longer deviates, so that the inclination angle of the straight line where the end point coordinates of the target seam are located is always within the preset angle range. The preset angle range can be set according to specific conditions, for example, 5 degrees, 10 degrees, or 15 degrees, etc. After adjusting the target seam, the material to be sealed can be transported to the next process, which includes but is not limited to heat setting, printing and dyeing, cloth rolling, and / or cloth laying, etc.

[0090] As an optional implementation of the above seam detection method, after the target seam is obtained, the target seam can be adjusted according to the inclination angle of the target seam, which can include:

[0091] Step S330: determining whether the angle difference between the inclination angle of the target seam and the inclination angle of the preset straight line is greater than the preset angle threshold.

[0092] For example, the implementation of step S330 uses an executable program compiled or interpreted by a preset programming language to determine whether the angle difference between the inclination angle of the target seam and the inclination angle of the preset straight line is greater than the preset angle threshold. The programming language that can be used includes C, C++, Java, BASIC, JavaScript, LISP, Shell, Perl, Ruby, Python, PHP, etc. The preset angle threshold can be set according to specific circumstances, for example, 5 degrees, 10 degrees, or 15 degrees, etc.

[0093] Step S340: If the angle difference between the inclination angle of the target seam and the inclination angle of the preset straight line is greater than the preset angle threshold, the target seam is adjusted so that the angle difference between the inclination angle of the target seam and the inclination angle of the preset straight line is less than the preset angle threshold.

[0094] For example, if the angle difference between the inclination angle of the target seam and the inclination angle of the preset straight line is greater than the preset angle threshold, the target seam in the material to be sewn is adjusted so that the material to be sewn no longer deviates (on the conveyor belt), so that the angle difference between the inclination angle of the target seam and the inclination angle of the preset straight line is less than the preset angle threshold. The preset angle range can be set according to specific circumstances, for example, 5 degrees, 10 degrees, or 15 degrees, etc. After adjusting the target seam, the material to be sealed can be transported to the next process, which includes but is not limited to heat setting, printing, rolling, and / or laying, etc. As an optional implementation of the above seam detection method, the material to be sewn includes but is not limited to cloth, leather, and / or plastic, etc.

[0095] Please refer to Figure 8 The structure of the seam detection device provided by the embodiment of the application is shown. The embodiment of the application provides a seam detection device 400, which includes:

[0096] The processing image acquisition module 410 is configured to acquire a to-be-processed image, the to-be-processed image being obtained by photographing a target seam, and the target seam being obtained by sewing a plurality of materials to be sewn.

[0097] The line seam endpoint detection module 420 is configured to detect the line seam endpoint coordinates in the to-be-processed image by using the neural network model, and obtain the endpoint coordinates of the target line seam. The neural network model is obtained by training with the line seam endpoint coordinates as sample labels.

[0098] The target line seam obtaining module 430 is configured to connect the endpoint coordinates of the target line seam with each other, and obtain the target line seam.

[0099] Optionally, in the embodiment of the present application, the neural network model comprises a plurality of feature extraction branches; and the line seam endpoint detection module comprises:

[0100] The image feature obtaining sub-module is configured to extract features of the to-be-processed image by using the plurality of feature extraction branches respectively, and obtain a plurality of image features.

[0101] The fused feature obtaining sub-module is configured to fuse the plurality of image features, and obtain image fused features.

[0102] The endpoint coordinate obtaining sub-module is configured to perform line seam endpoint coordinate mapping on the image fused features, and obtain line seam endpoint coordinates.

[0103] Optionally, in the embodiment of the present application, the line seam detection device further comprises:

[0104] The image label obtaining module is configured to obtain a plurality of sample images and a plurality of sample labels. The sample labels are obtained by labeling line seam endpoint coordinates in the sample images.

[0105] The neural network training module is configured to train the neural network by taking the plurality of sample images as training data and taking the plurality of sample labels as training labels, and obtain the neural network model.

[0106] Optionally, in the embodiment of the present application, the neural network training module comprises:

[0107] The endpoint coordinate prediction sub-module is configured to predict the line seam endpoint coordinates in the sample images by using the neural network, and obtain predicted line seam endpoint coordinates.

[0108] The loss value calculation sub-module is configured to calculate a loss value between the predicted line seam endpoint coordinates and the line seam endpoint coordinates in the sample labels.

[0109] The loss training sub-module is configured to train the neural network according to the loss value.

[0110] Optionally, in the embodiment of the present application, the line seam detection device can further comprise:

[0111] The inclination angle judgment module is configured to judge whether an inclination angle of a straight line where the endpoint coordinates of the target line seam are located exceeds a preset angle range.

[0112] a target seam adjustment module configured to adjust the target seam if the angle of inclination of the straight line on which the end point coordinates of the target seam lie exceeds the preset angle range, so that the angle of inclination of the straight line on which the end point coordinates of the target seam lie is within the preset angle range.

[0113] Optionally, in the embodiment of the present application, the seam detection device can further include:

[0114] determine whether the angle difference between the angle of inclination of the target seam and the angle of inclination of the preset straight line is greater than a preset angle threshold.

[0115] if yes, adjust the target seam so that the angle difference between the angle of inclination of the target seam and the angle of inclination of the preset straight line is less than the preset angle threshold.

[0116] Optionally, in the embodiment of the present application, the material to be sewn includes cloth, leather and / or plastic.

[0117] It should be understood that the device corresponds to the seam detection method embodiments described above, and can perform each step involved in the above method embodiments. The specific functions of the device can be referred to the description above, and the detailed description is appropriately omitted here to avoid repetition. The device includes at least one software function module which can be stored in the memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0118] An electronic device provided in an embodiment of the present application includes a processor and a memory. The memory stores machine readable instructions executable by the processor. When the machine readable instructions are executed by the processor, the method described above is performed.

[0119] The embodiments of the present application further provide a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is run by a processor to execute the method described above. The computer readable storage medium can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0120] It should be noted that each embodiment in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant part can be referred to the part of the method embodiment.

[0121] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed apparatus and method can also be implemented by other ways. The apparatus embodiments described above are only schematic, and the flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can be performed in a different order from that noted in the flowcharts. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved.

[0122] In addition, each function module in each embodiment in the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. In addition, in the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0123] In this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations.

[0124] The above description is only an optional implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for detecting seams, characterized in that, include: Acquire an image to be processed, wherein the image to be processed is obtained by photographing the target seam, and the target seam is obtained by sewing multiple materials to be sewn; The neural network model is used to detect the endpoint coordinates of the seam in the image to be processed, and the endpoint coordinates of the target seam are obtained. The neural network model is trained using the seam endpoint coordinates as sample labels. Connect the endpoint coordinates of the target seam to obtain the target seam; Wherein, the target seam in the image to be processed is the connection line to be detected between the first material to be sewn and the second material to be sewn among the plurality of materials to be sewn. The plurality of materials to be sewn include: materials of different kinds, or materials of the same kind but different patterns, or materials of the same kind and the same pattern but different manufacturing methods. The neural network model includes: multiple feature extraction branches, an attention fusion module, and a feature mapping module. The multiple feature extraction branches, the attention fusion module, and the feature mapping module are sequentially connected. The multiple feature extraction branches include: a first feature extraction branch, a second feature extraction branch, and a third feature extraction branch. The step of using the neural network model to detect the seam endpoint coordinates in the image to be processed includes: using the first feature extraction branch to extract global features of the image to be processed; using the second feature extraction branch to extract local features of the image to be processed at different scales; performing binarization processing on the image to be processed to obtain a binarized image; using the third feature extraction branch to extract edge features in the binarized image; using the attention fusion module to fuse the edge features, the global features, and the local features at different scales to obtain image fusion features; and using the feature mapping module to map the image fusion features to the seam endpoint coordinates to obtain the seam endpoint coordinates. Before using a neural network model to detect the coordinates of the seam endpoints in the image to be processed, the method further includes: Multiple sample images and multiple sample labels are acquired, wherein the sample labels are obtained by annotating the coordinates of the seam endpoints in the sample images; The neural network model is obtained by training the neural network using the multiple sample images as training data and the multiple sample labels as training labels. The training of the neural network includes: The neural network is used to predict the coordinates of the seam endpoints in the sample image to obtain the predicted seam endpoint coordinates. Calculate the loss value between the predicted seam endpoint coordinates and the seam endpoint coordinates in the sample labels; calculate the loss value based on the ArcFace loss function; The ArcFace loss function expression is: Among them, L ArcFace represents the loss value calculated by the ArcFace loss function, N is the length of the neural network output vector, represents the angle between the neural network output vector and the parameter weight vector in the i-th column, and s and m are hyperparameters; The neural network is trained based on the loss value; After obtaining the endpoint coordinates of the target seam, the method further includes: Determine whether the inclination angle of the straight line containing the endpoint coordinates of the target seam exceeds a preset angle range; If so, adjust the target seam so that the inclination angle of the straight line containing the endpoint coordinates of the target seam is within the preset angle range; After obtaining the target seam, the method further includes: Determine whether the angle difference between the tilt angle of the target seam and the tilt angle of the preset straight line is greater than a preset angle threshold. If so, adjust the target seam so that the angle difference between the tilt angle of the target seam and the tilt angle of the preset straight line is less than the preset angle threshold. The materials to be sewn include: fabric, leather and / or plastic.

2. A seam detection device, characterized in that, include: The image acquisition module is used to acquire an image to be processed, wherein the image to be processed is obtained by photographing the target seam, and the target seam is obtained by sewing multiple materials to be sewn. The seam endpoint detection module is used to detect the seam endpoint coordinates in the image to be processed using a neural network model, and obtain the endpoint coordinates of the target seam. The neural network model is trained using the seam endpoint coordinates as sample labels. The target seam acquisition module is used to connect the endpoint coordinates of the target seam to obtain the target seam. Wherein, the target seam in the image to be processed is the connection line to be detected between the first material to be sewn and the second material to be sewn among the plurality of materials to be sewn. The plurality of materials to be sewn include: materials of different kinds, or materials of the same kind but different patterns, or materials of the same kind and the same pattern but different manufacturing methods. The neural network model includes: multiple feature extraction branches, an attention fusion module, and a feature mapping module, wherein the multiple feature extraction branches, the attention fusion module, and the feature mapping module are sequentially connected. The multiple feature extraction branches include: a first feature extraction branch, a second feature extraction branch, and a third feature extraction branch. The step of using the neural network model to detect the seam endpoint coordinates in the image to be processed includes: using the first feature extraction branch to extract global features of the image to be processed; using the second feature extraction branch to extract local features of the image to be processed at different scales; using the third feature extraction branch to extract edge features in the binarized image of the image to be processed; using the attention fusion module to fuse the edge features, the global features, and the local features at different scales to obtain image fusion features; and using the feature mapping module to map the image fusion features to seam endpoint coordinates to obtain the seam endpoint coordinates. Between the image processing acquisition module and the seam endpoint detection module, there is also a component: Multiple sample images and multiple sample labels are acquired, wherein the sample labels are obtained by annotating the coordinates of the seam endpoints in the sample images; The neural network model is obtained by training the neural network using the multiple sample images as training data and the multiple sample labels as training labels. The training of the neural network includes: The neural network is used to predict the coordinates of the seam endpoints in the sample image to obtain the predicted seam endpoint coordinates. Calculate the loss value between the predicted seam endpoint coordinates and the seam endpoint coordinates in the sample labels; calculate the loss value based on the ArcFace loss function; The ArcFace loss function expression is: Among them, L ArcFace represents the loss value calculated by the ArcFace loss function, N is the length of the neural network output vector, represents the angle between the neural network output vector and the parameter weight vector in the i-th column, and s and m are hyperparameters; The neural network is trained based on the loss value; After obtaining the endpoint coordinates of the target seam, the method further includes: Determine whether the inclination angle of the straight line containing the endpoint coordinates of the target seam exceeds a preset angle range; If so, adjust the target seam so that the inclination angle of the straight line containing the endpoint coordinates of the target seam is within the preset angle range; After obtaining the target seam, the method further includes: Determine whether the angle difference between the tilt angle of the target seam and the tilt angle of the preset straight line is greater than a preset angle threshold. If so, adjust the target seam so that the angle difference between the tilt angle of the target seam and the tilt angle of the preset straight line is less than the preset angle threshold. The materials to be sewn include: fabric, leather and / or plastic.

3. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method as described in claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in claim 1.

Images