Leather shoe product qualification rate evaluation system based on machine vision

Abstract

The invention discloses a leather shoe product qualification rate evaluation system based on machine vision, and relates to the technical field of leather shoe quality evaluation, and the leather shoe product qualification rate evaluation system comprises a camera module, a movable support module, a movable pressing module, a snatching module, a video analysis module, a camera control module, an image processing module and a quality evaluation module; the camera modules are used for shooting the outer sides and the inner sides of the leather shoes to obtain video data; according to the leather shoe product qualification rate evaluation system based on machine vision, by arranging a camera module, a movable support module, a movable pressing module, a snatching module, a video analysis module, a camera control module, an image processing module and a quality evaluation module, the surface of a leather shoe can be pressed, the leather shoe can be snapped, and the leather shoe can fall freely; meanwhile, the processes are shot, and whether the deformation resistance, the glossiness, the surface defect degree and the bottom flatness of the leather shoes are qualified or not is evaluated by comparing and analyzing leather shoe images obtained in the shooting process and before and after shooting.

Description

Technical Field

[0001] The invention relates to the technical field of leather shoe quality assessment, and in particular to a leather shoe product qualification rate assessment system based on machine vision. Background Art

[0002] Leather shoes refer to shoes with natural leather uppers and leather or rubber, plastic, PU foam, PVC, etc. as soles, which are processed by sewing, gluing or injection molding. In the production process of leather shoes, various defects may appear in leather shoes, so the quality of leather shoes needs to be controlled before they leave the factory.

[0003] The invention patent with publication number CN115330785B discloses a method for detecting the quality of textile uppers; the method comprises: obtaining a grayscale image of the current textile upper and obtaining multiple connected domains in the grayscale image, calculating the contour deviation index of each connected domain; obtaining the degree of deformity of the connected domain according to the contour deviation index combined with the kurtosis factor; further obtaining the radius deviation based on the LBP sequence of the connected domain, obtaining the punching distortion based on the radius deviation and the degree of deformity; obtaining the punching deviation according to the punching distortion and the degree of deviation of the connected domain; matching the template image with the connected domain in the grayscale image based on the punching deviation and obtaining the minimum weight, obtaining the upper quality index of the current textile upper based on the minimum weight, and judging the quality of the current textile upper according to the upper quality index; the accuracy of the quality judgment of the textile upper is increased.

[0004] However, the above-mentioned prior art mainly judges the quality of the textile upper by analyzing the through holes used to connect and fix the upper and the vamp. For leather shoes, because their material is harder than textile shoes, sports shoes, etc., and they are often used in more formal social occasions, their quality judgment focuses more on the appearance and comfort of the leather shoes. Therefore, the above-mentioned prior art applied to the quality judgment of leather shoes still has certain defects. Summary of the invention

[0005] The purpose of the present invention is to provide a leather shoe product qualification rate evaluation system based on machine vision to solve the above-mentioned deficiencies in the prior art.

[0006] In order to achieve the above-mentioned purpose, the present invention provides the following technical solutions: a leather shoe product qualification rate evaluation system based on machine vision, comprising a camera module, an active support module, an active pressing module, a snatching module, a video analysis module, a camera control module, an image processing module, and a quality evaluation module;

[0007] There are multiple camera modules, which are used to shoot the outside and inside of the leather shoes to obtain video data;

[0008] The movable bracket module is used to install the camera module and drive the camera module to move; wherein, part of the camera module can be installed on the movable bracket, and the movable bracket can be retracted and moved, and can drive the camera module thereon to capture the inside of the leather shoe and capture the situation inside the leather shoe. wherein, in order to ensure the clarity of the image captured by the camera module on the inside of the leather shoe, a lighting lamp, a fill light, etc. can be installed on the movable bracket, and can be extended into the inside of the leather shoe together with the camera module to improve the brightness of the inside of the leather shoe;

[0009] The movable pressing module is used to press the leather shoes and retract when the pressing force reaches the set force. The pressing makes the surface of the leather shoes appear concave or bend to a certain extent, so that the quality of the leather shoes can be evaluated by the concave, bend, and recovery after pressing. The pressing force can be monitored by installing a pressure sensor on the movable pressing module.

[0010] The grabbing module is used to grab the leather shoes, lift the leather shoes to a set height and then release the leather shoes, so that the leather shoes fall freely onto the detection platform, so that the leather shoes fall down, and it is convenient to test whether the leather shoes can fall quickly and steadily onto the detection platform to determine whether the bottom of the leather shoes is flat;

[0011] The video analysis module is used to identify the leather shoes, the movable support module, the movable pressing module and the grabbing module in the video data based on the target detection algorithm, and determine the positions of the leather shoes and the movable support module, so as to control the movement of the movable support module through the camera control module, drive the camera module thereon to extend into the inside of the leather shoes, and take videos and images of the inside of the leather shoes; wherein the target detection algorithm can use a pre-trained target detection algorithm to detect the leather shoes, and can also train the target detection algorithm through the image of the inside of the leather shoes, so that the target detection algorithm can identify the inside of the leather shoes;

[0012] The camera control module is used to control the movement of the movable support module based on the positions of the leather shoes and the movable support module obtained by the video analysis module, so that the camera module on the movable support module extends into the inner side of the leather shoes to shoot the inner side of the leather shoes, or controls the movable support module to reset after the camera module shoots the inner side of the leather shoes;

[0013] The image processing module is used to extract a plurality of key image frames from the video data according to a set rule;

[0014] The quality assessment module is used to judge whether the deformation resistance, glossiness, surface defects and bottom surface flatness of the leather shoes are qualified by comparing key image frames, and calculate the qualified rate of the leather shoes based on the judgment results.

[0015] Furthermore, the video analysis module is also used to train a target detection algorithm based on the image of the inside of the leather shoe, and is used to identify the inside of the leather shoe from the video data through the target detection algorithm.

[0016] Furthermore, the camera control module controls the movable bracket module to reset after the camera module shoots the inside of the leather shoe, including the following steps:

[0017] Obtaining a detection result of the target detection algorithm of the video analysis module;

[0018] If the target detection algorithm recognizes the inside of the leather shoe, the movable bracket module is controlled to return to the original path after the first set time period, otherwise it returns to the previous step.

[0019] Furthermore, the image processing module is used to extract a plurality of key image frames from the video data according to a set rule, specifically comprising the following steps:

[0020] When the target detection algorithm of the video analysis module detects that the active pressing module is in contact with the surface of the leather shoe, an image frame in which the active pressing module just contacts the surface of the leather shoe in the video data is marked to obtain a contact image frame;

[0021] When the target detection algorithm of the video analysis module detects that the active pressing module is separated from the surface of the leather shoe, an image frame in which the active pressing module is just separated from the surface of the leather shoe in the video data is marked to obtain a separated image frame;

[0022] Marking all image frames in the video data within a second set time period from the contact image frame to the separation image frame as first key image frames to obtain a first key image frame set;

[0023] The surface image of the leather shoe before pressing in the video data is obtained by using a target detection algorithm, and marked as a second key image frame one, to obtain a second key image frame set one; the image frame of the second set time length after the separated image frame is marked as a second key image frame two, to obtain a second key image frame set two;

[0024] The target detection algorithm of the video analysis module detects that the snatch module lets go of the leather shoes after snatching, and extracts the image frame when the leather shoes just touch the detection platform from the video data to obtain the landing image frame. The image frames within the third set time length from the landing image frame to the landing image frame are all marked as the third key image frame to obtain the third key image frame set.

[0025] Furthermore, the quality assessment module is used to judge whether the anti-deformation quality of the leather shoes is qualified by comparing the key image frames, including the following steps:

[0026] a1. Setting anti-deformation quality evaluation labels to mark the first key image frame set and the second key image frame set, wherein the anti-deformation quality evaluation labels include qualified anti-deformation quality and unqualified anti-deformation quality;

[0027] a2. Extract the first key image frame of each leather shoe in chronological order, and obtain the second key image frame of the corresponding leather shoe;

[0028] a3, calculating the similarity between the first key image frame and the second key image frame of each leather shoe in chronological order, and obtaining a similarity change curve;

[0029] a4. Based on the similarity change curve and its corresponding leather shoe evaluation label, a first deep neural network model is trained to obtain a leather shoe deformation quality evaluation model;

[0030] a5. Input the first key image frame into the leather shoe deformation quality evaluation model, calculate the similarity between each first key image frame and the second key image frame, obtain a similarity change curve, and output a predicted evaluation label based on the similarity change curve.

[0031] Furthermore, the quality assessment module is used to judge whether the glossiness of the leather shoes is qualified by comparing the key image frames, which specifically includes the following steps:

[0032] b1, obtaining brightness at each position of the image frames of the second key image frame set 1 and the second key image frame set 2;

[0033] b2. Calculate the brightness standard deviation of each position of the second key image frame 1 of each leather shoe to obtain the brightness standard deviation;

[0034] b3, calculating the square root of the arithmetic mean of the square difference between the brightness of each position of the second key image frame 2 and the average brightness of each position of the second key image frame 1 of each leather shoe, and obtaining the brightness error;

[0035] b4. Determine whether the brightness standard deviation and brightness error are less than the set brightness standard deviation threshold and brightness error threshold respectively. If so (i.e., the brightness standard deviation is less than the set brightness standard deviation threshold, and the brightness error is less than the brightness error threshold), the glossiness is judged to be qualified; if not, the glossiness is judged to be unqualified.

[0036] Furthermore, the quality assessment module is used to judge whether the surface defects of the leather shoes are qualified by comparing the key image frames, which specifically includes the following steps:

[0037] c1. Setting a defect evaluation label to mark the second key image frame set 1, wherein the defect evaluation label includes defect qualified and defect unqualified;

[0038] c2. Based on the second key image frame set 1 and the defect assessment label, training a second deep neural network model to obtain a defect assessment model;

[0039] c3. Input the second key image frame into the defect assessment model and output the predicted defect assessment label.

[0040] Furthermore, the quality assessment module is used to judge whether the flatness of the bottom surface of the leather shoes is qualified by comparing the key image frames, which specifically includes the following steps:

[0041] d1, extracting the third key image frame of each leather shoe in time sequence;

[0042] d2, starting from the second third key image frame, select the third key image frame in sequence and set it as the comparison frame;

[0043] d3, comparing the shoe deviation angle of the comparison frame with the third key image frame before the comparison frame, and determining whether the deviation angle is less than a set deviation threshold;

[0044] d4. If yes, the bottom of the leather shoes is judged to be flat;

[0045] d5. If not, determine whether the comparison frame is the last third key image frame;

[0046] d6. If yes, it is judged that the bottom of the leather shoe is uneven;

[0047] d7. If not, update the comparison frame to the third key image frame next to the comparison frame, and return to d3.

[0048] 1. Compared with the prior art, the leather shoe product qualification rate assessment system based on machine vision provided by the present invention can photograph leather shoes by setting a camera module, a movable bracket module, a video analysis module, a camera control module, an image processing module, and a quality assessment module, so as to achieve the effect of judging whether the glossiness and surface defect degree of the leather shoes are qualified by analyzing the photographed images.

[0049] 2. Compared with the prior art, the machine vision-based leather shoe product qualification rate assessment system provided by the present invention, by setting a camera module, a movable bracket module, a movable pressing module, a video analysis module, a camera control module, an image processing module, and a quality assessment module, can press the surface of the leather shoes during the shooting process, obtain the images of the leather shoes during and after pressing, and compare and analyze them with the images of the leather shoes before pressing, thereby achieving the effect of judging whether the deformation resistance of the leather shoes is qualified through the images of the leather shoes.

[0050] 3. Compared with the prior art, the machine vision-based leather shoe product qualification rate assessment system provided by the present invention, by setting a camera module, a movable bracket module, a grabbing module, a video analysis module, a camera control module, an image processing module, and a quality assessment module, can grab the leather shoes and make them fall freely, and obtain video images of the leather shoes after they fall freely and land on the ground, so as to achieve the effect of analyzing the video images of the leather shoes after they land to see whether the leather shoes stop shaking, so as to judge whether the flatness of the bottom of the leather shoes is qualified.

[0051] 4. Compared with the prior art, the machine vision-based leather shoe product qualification rate assessment system provided by the present invention, by setting a camera module, a movable bracket module, a movable pressing module, a grabbing module, a video analysis module, a camera control module, an image processing module, and a quality assessment module, can evaluate whether the leather shoes are qualified from four aspects: gloss, surface defect degree, deformation resistance, and bottom flatness, thereby ensuring the appearance and comfort of the leather shoes leaving the factory and better meeting the needs of the leather shoes usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0053] Figure 1 A system structure block diagram provided for an embodiment of the present invention;

[0054] Figure 2 A diagram of the steps for evaluating the anti-deformation quality provided by an embodiment of the present invention;

[0055] Figure 3 A glossiness evaluation step diagram provided by an embodiment of the present invention;

[0056] Figure 4 A defect evaluation step diagram provided by an embodiment of the present invention;

[0057] Figure 5 A diagram of the bottom surface flatness evaluation steps provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0059] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0060] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0062] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0063] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0064] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.

[0065] The embodiments described herein may be described with reference to plan views and / or cross-sectional views by means of idealized schematic diagrams of the present disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings have schematic properties, and the shapes of the regions shown in the figures illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.

[0066] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0067] See also Figure 1 , a leather shoe product qualification rate evaluation system based on machine vision, including a camera module, an active bracket module, an active pressing module, a snatch module, a video analysis module, a camera control module, an image processing module, and a quality evaluation module;

[0068] There are multiple camera modules, which are used to shoot the outside and inside of the leather shoes to obtain video data;

[0069] The movable bracket module is used to install the camera module and drive the camera module to move; wherein, part of the camera module can be installed on the movable bracket, and the movable bracket can be an electric telescopic rod that is retractable, and can drive the camera module thereon to capture the inside of the leather shoe and capture the situation inside the leather shoe. In order to ensure the clarity of the image captured by the camera module on the inside of the leather shoe, a lighting lamp, a fill light, etc. can be installed on the movable bracket, and can be extended into the inside of the leather shoe together with the camera module to improve the brightness of the inside of the leather shoe;

[0070] The active pressing module is used to press the leather shoes and retract when the pressing force reaches the set force. The pressing method causes a certain dent or bend to appear on the surface of the leather shoes, so that the quality of the leather shoes can be evaluated by the dent, bend, and recovery after pressing. The pressing force can be monitored by installing a pressure sensor on the active pressing module.

[0071] The snatch module is used to grab the leather shoes, lift them to a set height and then release them, so that they fall freely onto the testing platform. This makes it easy to test whether the leather shoes can fall quickly and steadily onto the testing platform, so as to determine whether the bottom of the leather shoes is flat.

[0072] The video analysis module is used to identify the leather shoes, movable bracket module, movable pressing module and grabbing module in the video data based on the target detection algorithm, and determine the positions of the leather shoes and the movable bracket module, so as to facilitate the subsequent control of the movement of the movable bracket module through the camera control module, drive the camera module thereon to extend into the inside of the leather shoes, and take videos and images of the inside of the leather shoes; wherein, the target detection algorithm can use a pre-trained target detection algorithm to detect the leather shoes, and can also train the target detection algorithm through the image of the inside of the leather shoes, so that the target detection algorithm can identify the inside of the leather shoes; the video analysis module is also used to train the target detection algorithm based on the image of the inside of the leather shoes, and is used to identify the inside of the leather shoes from the video data through the target detection algorithm.

[0073] Furthermore, the target detection algorithm can use the cross entropy loss function to train the target detector, the formula is as follows:

[0074]

[0075] Among them, t i is the ground-truth label, p i is the softmax probability of the i-th class, and n is the total number of classes;

[0076] Because a large number of proposals (candidate boxes generated by RPN for classification and regression) are generated during the training of the target detector, many of which are redundant. In order to remove dense repeated prediction results, the NMS post-processing step is required. The NMS module first sorts the predicted proposals according to the confidence score of each instance and selects the proposal with the highest confidence. The formula is as follows:

[0077]

[0078] Among them, bm is the proposal with the highest confidence, and bi represents other proposals generated for the true value; if this value is greater than the set NMS threshold (usually 0.5), the duplicates are deleted;

[0079] We can remove duplicates while avoiding complete suppression by calculating a confidence score and the product of the negative value of IoU and bm. The formula is as follows:

[0080]

[0081] The camera control module is used to control the movement of the movable support module based on the positions of the leather shoes and the movable support module obtained by the video analysis module, so that the camera module on the movable support module extends into the inner side of the leather shoes to shoot the inner side of the leather shoes; or control the movable support module to reset after the camera module shoots the inner side of the leather shoes, including the following steps:

[0082] (1) Obtain the detection results of the target detection algorithm of the video analysis module;

[0083] (2) If the target detection algorithm identifies the inside of the leather shoe, the movable bracket module is controlled to return to the original path after the first set time, otherwise it returns to the previous step.

[0084] The image processing module is used to extract multiple key image frames from the video data according to the set rules, which specifically includes the following steps:

[0085] (1) When the target detection algorithm of the video analysis module detects that the active pressing module is in contact with the surface of the leather shoe, an image frame in which the active pressing module just contacts the surface of the leather shoe in the video data is marked to obtain a contact image frame;

[0086] (2) when the target detection algorithm of the video analysis module detects that the active pressing module is separated from the surface of the leather shoe, an image frame in which the active pressing module is just separated from the surface of the leather shoe in the video data is marked to obtain a separated image frame;

[0087] (3) marking all image frames in the video data within a second set time period from the contact image frame to the separation image frame as first key image frames to obtain a first key image frame set;

[0088] (4) obtaining a surface image of the leather shoe before pressing in the video data by using a target detection algorithm, marking it as a second key image frame one, obtaining a second key image frame set one, and marking an image frame of a second set time length after separating the image frame as a second key image frame two, obtaining a second key image frame set two;

[0089] (5) The target detection algorithm of the video analysis module detects that the snatch module releases the leather shoes after snatching, and extracts the image frame when the leather shoes just come into contact with the detection platform from the video data to obtain the landing image frame. The image frames within the third set time length from the landing image frame to the landing image frame are all marked as the third key image frames to obtain the third key image frame set.

[0090] Please refer to Figure 2 The quality assessment module is used to judge whether the anti-deformation quality, glossiness, surface defects, and bottom surface flatness of the leather shoes are qualified by comparing the key image frames. The quality assessment module is used to judge whether the anti-deformation quality of the leather shoes is qualified by comparing the key image frames, and includes the following steps:

[0091] a1. Setting anti-deformation quality evaluation labels to mark the first key image frame set and the second key image frame set, wherein the anti-deformation quality evaluation labels include qualified anti-deformation quality and unqualified anti-deformation quality;

[0092] a2. Extract the first key image frame of each leather shoe in chronological order, and obtain the second key image frame 1 of the corresponding leather shoe;

[0093] a3, calculating the similarity between the first key image frame and the second key image frame of each leather shoe in chronological order, and obtaining a similarity change curve;

[0094] a4. Based on the similarity change curve and its corresponding leather shoe evaluation label, a first deep neural network model is trained to obtain a leather shoe deformation quality evaluation model;

[0095] When training the first deep neural network model, there is no limitation on the specific deep neural network model, and a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a generative adversarial network (GAN), etc. may be used, and a suitable loss function such as mean square error (MSE), cross-entropy loss (Cross-Entropy Loss), etc. may be selected;

[0096] During training, the similarity change curve and its corresponding leather shoe evaluation label are used as samples, and then the samples are randomly divided into a training set and a validation set in a ratio of 6:4;

[0097] The selected first deep neural network model is trained using the training set, wherein the similarity change curve is used as input during the training, and the evaluation label of the leather shoes corresponding to the similarity change curve is used as output, so as to obtain the parameter values ​​of the first deep neural network model;

[0098] The obtained parameter values ​​are then substituted into the first deep neural network model, and the accuracy of the output of the first deep neural network model is verified using the validation set to determine whether the set accuracy standard is reached. If not, the parameters, the selected deep neural network model or loss function are adjusted and retrained. If so, the parameter values ​​of the first deep neural network model are determined to obtain a leather shoe deformation quality evaluation model, which can output a predicted evaluation label based on the input similarity change curve.

[0099] a5. Input the first key image frame into the leather shoe deformation quality evaluation model, calculate the similarity between each first key image frame and the second key image frame, obtain the similarity change curve, and output the predicted evaluation label based on the similarity change curve.

[0100] Please refer to Figure 3 The quality assessment module is used to judge whether the glossiness of the leather shoes is qualified by comparing the key image frames, which specifically includes the following steps:

[0101] b1, obtaining brightness at each position of the image frames of the second key image frame set 1 and the second key image frame set 2;

[0102] b2. Calculate the brightness standard deviation of each position of the second key image frame 1 of each leather shoe to obtain the brightness standard deviation;

[0103] b3, calculating the square root of the arithmetic mean of the square difference between the brightness of each position of the second key image frame 2 and the average brightness of each position of the second key image frame 1 of each leather shoe, and obtaining the brightness error;

[0104] b4. Determine whether the brightness standard deviation and brightness error are less than the set brightness standard deviation threshold and brightness error threshold respectively. If so (i.e., the brightness standard deviation is less than the set brightness standard deviation threshold, and the brightness error is less than the brightness error threshold), the glossiness is judged to be qualified; if not, the glossiness is judged to be unqualified.

[0105] Please refer to Figure 4 The quality assessment module is used to judge whether the surface defects of leather shoes are qualified by comparing key image frames, which specifically includes the following steps:

[0106] c1. Setting defect evaluation labels to mark the second key image frame set 1, wherein the defect evaluation labels include defect qualified and defect unqualified;

[0107] c2. Based on the second key image frame set 1 and the defect assessment label, training a second deep neural network model to obtain a defect assessment model;

[0108] When training the second deep neural network model, the specific deep neural network model is not limited, and a suitable deep neural network model and loss function can be selected as needed;

[0109] During training, the second key image frame 1 in the second key image frame set 1 and the defect evaluation label corresponding to the second key image frame set 1 are used as samples, and then the samples are randomly divided into a training set and a validation set in a certain ratio, such as 7:3;

[0110] The selected second deep neural network model is trained using the training set, wherein during the training, each second key image frame in the second key image frame set is used as input, and the defect evaluation label corresponding to the second key image frame set is used as output, thereby obtaining the parameter values ​​of the second deep neural network model;

[0111] The obtained parameter values ​​are then substituted into the second deep neural network model, and the accuracy of the output of the second deep neural network model is verified using the validation set to determine whether the set accuracy standard is reached. If not, the parameters, the selected deep neural network model or loss function are adjusted and retrained. If so, the parameter values ​​of the second deep neural network model are determined to obtain a defect assessment model, which can output a predicted defect assessment label based on the second key image frame one of the input second key image frame set one.

[0112] c3. Input the second key image frame into the defect assessment model and output the predicted defect assessment label.

[0113] Please refer to Figure 5 The quality assessment module is used to judge whether the flatness of the bottom surface of the leather shoes is qualified by comparing the key image frames, and specifically includes the following steps:

[0114] d1, extracting the third key image frame of each leather shoe in time sequence;

[0115] d2, starting from the second third key image frame, select the third key image frame in sequence and set it as the comparison frame;

[0116] d3, comparing the shoe deviation angles of the comparison frame and the third key image frame before the comparison frame, and determining whether the deviation angle is less than a set deviation threshold;

[0117] d4. If yes, the bottom of the leather shoes is considered to be flat;

[0118] d5. If not, determine whether the comparison frame is the last third key image frame;

[0119] d6. If yes, it is judged that the bottom of the leather shoe is uneven;

[0120] d7. If not, update the comparison frame to the next third key image frame of the comparison frame, and return to d3.

[0121] Among them, during the evaluation of leather shoes by the quality assessment module, the leather shoes are considered qualified only if they meet the requirements of qualified anti-deformation quality, qualified glossiness, qualified defects, and flat bottom of the leather shoes. At the same time, the qualified rate of leather shoes is calculated based on the number of qualified leather shoes and the total amount of evaluated leather shoes.

[0122] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. The leather shoes product qualification rate evaluation system based on machine vision is characterized by: It includes a camera module, an active support module, an active pressing module, a snatch module, a video analysis module, a camera control module, an image processing module, and a quality assessment module; There are multiple camera modules, which are used to shoot the outside and inside of the leather shoes to obtain video data; The movable bracket module is used to install the camera module and drive the camera module to move; The movable pressing module is used to press the leather shoes and retract when the pressing force reaches the set force; The grabbing module is used to grab the leather shoes, lift the leather shoes to a set height, and then release the leather shoes, so that the leather shoes fall freely onto the detection platform; The video analysis module is used to identify the leather shoes, the movable support module, the movable pressing module and the snatching module in the video data based on the target detection algorithm, and determine the positions of the leather shoes and the movable support module; The camera control module is used to control the movement of the movable support module based on the positions of the leather shoes and the movable support module obtained by the video analysis module, so that the camera module on the movable support module extends into the inner side of the leather shoes to shoot the inner side of the leather shoes, or controls the movable support module to reset after the camera module shoots the inner side of the leather shoes; The image processing module is used to extract a plurality of key image frames from the video data according to a set rule; The quality assessment module is used to judge whether the deformation resistance, glossiness, surface defects and bottom surface flatness of the leather shoes are qualified by comparing key image frames, and calculate the qualified rate of the leather shoes based on the judgment results.

2. The machine vision-based leather shoe product qualification rate evaluation system according to claim 1 is characterized in that: The video analysis module is also used to train a target detection algorithm based on the image of the inside of the leather shoe, and is used to identify the inside of the leather shoe from the video data through the target detection algorithm.

3. The machine vision-based leather shoe product qualification rate evaluation system according to claim 2 is characterized in that: The camera control module controls the movable bracket module to reset after the camera module shoots the inside of the leather shoe, including the following steps: Obtaining a detection result of the target detection algorithm of the video analysis module; If the target detection algorithm recognizes the inside of the leather shoe, the movable bracket module is controlled to return to the original path after the first set time period, otherwise it returns to the previous step.

4. The machine vision-based leather shoe product qualification rate evaluation system according to claim 1 is characterized in that: The image processing module is used to extract a plurality of key image frames from the video data according to a set rule, and specifically includes the following steps: When the target detection algorithm of the video analysis module detects that the active pressing module is in contact with the surface of the leather shoe, an image frame in which the active pressing module just contacts the surface of the leather shoe in the video data is marked to obtain a contact image frame; When the target detection algorithm of the video analysis module detects that the active pressing module is separated from the surface of the leather shoe, an image frame in which the active pressing module is just separated from the surface of the leather shoe in the video data is marked to obtain a separated image frame; Marking all image frames in the video data within a second set time period from the contact image frame to the separation image frame as first key image frames to obtain a first key image frame set; The surface image of the leather shoe before pressing in the video data is obtained by using a target detection algorithm, and marked as a second key image frame one, to obtain a second key image frame set one; the image frame of the second set time length after the separated image frame is marked as a second key image frame two, to obtain a second key image frame set two; The target detection algorithm of the video analysis module detects that the snatch module lets go of the leather shoes after snatching, and extracts the image frame when the leather shoes just touch the detection platform from the video data to obtain the landing image frame. The image frames within the third set time length from the landing image frame to the landing image frame are all marked as the third key image frame to obtain the third key image frame set.

5. The machine vision-based leather shoe product qualification rate evaluation system according to claim 4 is characterized in that: The quality assessment module is used to judge whether the anti-deformation quality of the leather shoes is qualified by comparing the key image frames, and includes the following steps: a1. Setting anti-deformation quality evaluation labels to mark the first key image frame set and the second key image frame set, wherein the anti-deformation quality evaluation labels include qualified anti-deformation quality and unqualified anti-deformation quality; a2. Extract the first key image frame of each leather shoe in chronological order, and obtain the second key image frame 1 of the corresponding leather shoe; a3, calculating the similarity between the first key image frame and the second key image frame of each leather shoe in chronological order, and obtaining a similarity change curve; a4. Based on the similarity change curve and its corresponding leather shoe evaluation label, a first deep neural network model is trained to obtain a leather shoe deformation quality evaluation model; a5. Input the first key image frame into the leather shoe deformation quality evaluation model, calculate the similarity between each first key image frame and the second key image frame, obtain a similarity change curve, and output a predicted evaluation label based on the similarity change curve.

6. The machine vision-based leather shoe product qualification rate evaluation system according to claim 4 is characterized in that: The quality assessment module is used to judge whether the glossiness of the leather shoes is qualified by comparing the key image frames, and specifically includes the following steps: b1, obtaining brightness of each position of the image frames of the second key image frame set 1 and the second key image frame set 2; b2. Calculate the brightness standard deviation of each position of the second key image frame 1 of each leather shoe to obtain the brightness standard deviation; b3, calculating the square root of the arithmetic mean of the square difference between the brightness of each position of the second key image frame 2 and the average brightness of each position of the second key image frame 1 of each leather shoe, and obtaining the brightness error; b4. Determine whether the brightness standard deviation and the brightness error are less than the set brightness standard deviation threshold and brightness error threshold respectively. If so, the glossiness is determined to be qualified; if not, the glossiness is determined to be unqualified.

7. The machine vision-based leather shoe product qualification rate evaluation system according to claim 4 is characterized in that: The quality assessment module is used to judge whether the surface defects of leather shoes are qualified by comparing key image frames, and specifically includes the following steps: c1. Setting a defect evaluation label to mark the second key image frame set 1, wherein the defect evaluation label includes defect qualified and defect unqualified; c2. Based on the second key image frame set 1 and the defect assessment label, training a second deep neural network model to obtain a defect assessment model; c3. Input the second key image frame into the defect assessment model and output the predicted defect assessment label.

8. The machine vision-based leather shoe product qualification rate evaluation system according to claim 4 is characterized in that: The quality assessment module is used to determine whether the flatness of the bottom surface of the leather shoe is qualified by comparing the key image frames, and specifically includes the following steps: d1, extracting the third key image frame of each leather shoe in time sequence; d2, starting from the second third key image frame, select the third key image frame in sequence and set it as the comparison frame; d3, comparing the shoe deviation angle of the comparison frame with the third key image frame before the comparison frame, and determining whether the deviation angle is less than a set deviation threshold; d4. If yes, the bottom of the leather shoes is considered to be flat; d5. If not, determine whether the comparison frame is the last third key image frame; d6. If yes, it is judged that the bottom of the leather shoe is uneven; d7. If not, update the comparison frame to the third key image frame next to the comparison frame, and return to d3.

Images