Shoe appearance inspection system, shoe appearance inspection method, and computer-readable storage medium

Abstract

Provided are a shoe appearance inspection system, a shoe appearance inspection method, and a computer-readable storage medium that can improve inspection efficiency and inspection accuracy. In the shoe appearance inspection system, a virtual line extraction unit (128) extracts a plurality of virtual lines that respectively connect a reference point and a plurality of feature points from an image of a shoe as an inspection object. A model storage unit (136) stores a learning model generated by machine learning, and the machine learning uses the virtual lines and contours extracted from images of a plurality of shoes as qualified products as teacher data. A pass / fail determination unit (134) inputs the plurality of virtual lines and contours extracted from an image of a shoe as an inspection object into the learning model, thereby determining whether the shoe as an inspection object is a qualified product and outputting the result.

Description

Technical Field

[0001] This invention relates to a technique for inspecting the appearance of shoes, and more particularly to a shoe appearance inspection system, a shoe appearance inspection method, and a computer-readable storage medium. Background Technology

[0002] In the past, for fixed-shape articles such as metal products, visual inspection was sometimes performed using sensors or image processing to improve inspection efficiency and accuracy (see, for example, Patent Document 1). Even in the case of amorphous articles whose shapes can vary considerably, techniques are known to determine the correspondence and consistency between parts of articles with opposite and completely different shapes using image processing (see, for example, Non-Patent Document 1). Both the techniques in Patent Document 1 and Non-Patent Document 1 are characterized by determining the identity of articles, and are common in this respect. As image processing, whether they are substantially 100% identical is taken as the determination content.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-076819

[0006] Non-patent literature

[0007] Non-Patent Document 1: Object Retrieval Technology for Deformed Objects at Arbitrary Angles, [Online], November 26, 2018, NTT Co., Ltd., Internet <URL: https: / / www.ntt.co.jp / news2018 / 1811 / 181126b.html#b1> Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] To maintain product quality, the manufacturing process of footwear includes an inspection step. While footwear products are generally fixed in shape, the upper, in particular, is made of mesh fiber or leather, making it susceptible to deformation and prone to slight variations in shape depending on the individual or the specific circumstances. Furthermore, manufacturing steps such as attaching the upper to the sole or applying adhesives are performed manually, which can introduce minor deviations in the attachment or coating locations. Due to this nature of footwear, visual inspection has traditionally been conducted by hand. However, visual inspection cannot eliminate the possibility of operational errors or missed inspections, and the inspection workload is heavy. Therefore, there is a need to establish a technology that can improve inspection efficiency and accuracy.

[0010] This invention was made in view of this problem, and its purpose is to provide a shoe appearance inspection technology that can improve inspection efficiency and inspection accuracy.

[0011] Technical means to solve the problem

[0012] To address the aforementioned problem, a shoe appearance inspection system according to an embodiment of the present invention includes: an image acquisition unit for acquiring an image of the inspection object, i.e., a shoe; a reference point extraction unit for extracting reference points using a predetermined reference point extraction method, wherein the reference points serve as appearance feature points in the image of the inspection object, i.e., the shoe; an element point extraction unit for extracting multiple element points using a predetermined element point extraction method, wherein the multiple element points serve as appearance feature points in the image of the inspection object, i.e., the shoe; an imaginary line extraction unit for extracting multiple imaginary lines from the image of the inspection object, i.e., the shoe, connecting the reference points and the multiple element points respectively; a model storage unit for storing a learning model generated by machine learning, wherein the machine learning uses the imaginary lines extracted from the images of multiple shoes (which are considered qualified products) as teacher data; and a pass / fail determination unit for inputting the multiple imaginary lines extracted from the image of the inspection object, i.e., the shoe, into the learning model, thereby determining whether the inspection object, i.e., the shoe, is a qualified product.

[0013] The image of the shoe obtained is an image of the shoe in a lasted state using a shoe mold. The reference point extraction unit extracts feature points on the appearance of the shoe mold that protrudes from the shoe in the image as reference points.

[0014] The process can be as follows: the acquired shoe images include multiple images taken from multiple angles; the model storage unit stores the learning model generated by machine learning; the machine learning uses imaginary lines extracted from multiple images for a shoe as teacher data; the pass / fail determination unit inputs the imaginary lines extracted from multiple images for the inspection object, i.e., the shoe, into the learning model, thereby determining whether the inspection object, i.e., the shoe, is a qualified product.

[0015] It may also include a contour extraction unit, which extracts the contour of the shoe from the image of the object to be inspected, i.e., the shoe. This may be: a model storage unit stores a learning model generated through machine learning, whereby the machine learning uses imaginary lines and contours extracted from multiple images of shoes considered as qualified products as teacher data; and a qualification / disqualification unit inputs multiple imaginary lines and contours extracted from the image of the object to be inspected, i.e., the shoe, into the learning model to determine whether the object to be inspected, i.e., the shoe, is a qualified product.

[0016] Another embodiment of the present invention is a method for inspecting the appearance of shoes. The method includes: acquiring an image of the shoe being inspected using a prescribed image acquisition unit; extracting reference points using a prescribed reference point extraction method performed by a computer, the reference points serving as feature points in the image of the shoe being inspected; extracting multiple feature points using a prescribed feature point extraction method performed by a computer, the multiple feature points serving as feature points in the image of the shoe being inspected; extracting multiple imaginary lines connecting the reference points and the multiple feature points from the image of the shoe being inspected using a computer; reading a learning model generated by machine learning from a prescribed storage unit, the machine learning using the imaginary lines extracted from the images of multiple shoes (considered qualified products) as teacher data; and determining whether the shoe being inspected is a qualified product by inputting the multiple imaginary lines extracted from the image of the shoe being inspected into the learning model using a computer.

[0017] Furthermore, any combination of the above-mentioned constituent elements, or any substitution of the constituent elements or statements of the present invention among methods, apparatus, programs, temporary or non-temporary storage media storing programs, systems, etc., are also valid embodiments of the present invention.

[0018] The effects of the invention

[0019] This invention provides a shoe appearance inspection technology that can improve inspection efficiency and accuracy. Attached Figure Description

[0020] Figure 1 This is a structural diagram of the shoe appearance inspection system of this embodiment.

[0021] Figure 2 It is an image obtained by taking a picture of the outer instep of the footwear from the side, and comparing the shape of the upper of the qualified product with that of the unqualified product.

[0022] Figure 3 It is a graph that compares the slope of the axis in the acceptable product with the slope of the axis in the unacceptable product using the back image.

[0023] Figure 4 This is a diagram showing the positional offset of the high-hardness raw material in the midfoot area of ​​the midsole.

[0024] Figure 5 This is a functional block diagram representing the basic structure of a shoe appearance inspection device.

[0025] Figure 6 This diagram schematically illustrates the method for extracting reference points, feature points, and imaginary lines from an image obtained by taking a picture of the outer instep of a shoe with a shoe last from the side.

[0026] Figure 7 This diagram schematically illustrates the method for extracting reference points, feature points, and imaginary lines from an image obtained by taking a picture of the outer instep of a shoe without a shoe last from the side.

[0027] Figure 8 This diagram schematically illustrates the method for extracting reference points, feature points, and imaginary lines from a rear image of a shoe product with a shoe last.

[0028] Figure 9 This diagram schematically illustrates a method for extracting contours from an image obtained by photographing the outer instep of a shoe with a shoe last from the side.

[0029] Figure 10 This is a functional block diagram representing the basic structure of a shoe appearance inspection learning device.

[0030] Figure 11 This is a flowchart illustrating the process of extracting multiple imaginary lines and contours from an image of a footwear product and using a learning model to infer whether it is a qualified product.

[0031] Explanation of symbols

[0032] R1: First reference point

[0033] P1: First element point

[0034] L1: First Imaginary Line

[0035] R2: Second reference point

[0036] P2: Second element point

[0037] L2: Second Imaginary Line

[0038] R3: Third reference point

[0039] P3: Third element point

[0040] L3: Third Imaginary Line

[0041] P4: Fourth Element Point

[0042] L4: Fourth Imaginary Line

[0043] P5: Fifth Element Point

[0044] L5: Fifth Imaginary Line

[0045] L6: Sixth Imaginary Line

[0046] P7: Seventh Element Point

[0047] L8: The Eighth Imaginary Line

[0048] 12: Outsole

[0049] 14: Lower layer midsole

[0050] 16: Upper midsole

[0051] 20: Shoe upper

[0052] 30: Shoe last

[0053] SL: Outline

[0054] 100: Shoe Appearance Inspection System

[0055] 110: Shoe Appearance Inspection Device

[0056] 112: Shoe Appearance Inspection Learning Device

[0057] 120: Image Acquisition Department

[0058] 124: Reference Point Extraction Unit

[0059] 126: Feature Point Extraction Department

[0060] 128: Imaginary Line Extraction Section

[0061] 130: Contour Extraction Section

[0062] 134: Qualification / Failure Determination Department

[0063] 136: Model Storage Department

[0064] 220: Image Acquisition Department

[0065] 224: Benchmark Point Extraction Unit

[0066] 226: Feature Point Extraction Department

[0067] 228: Imaginary Line Extraction Section

[0068] 230: Contour Extraction Section

[0069] 236: Model Storage Department Detailed Implementation

[0070] Figure 1This is a structural diagram of the shoe appearance inspection system 100 according to this embodiment. The shoe appearance inspection system 100 includes a shoe appearance inspection device 110 and a shoe appearance inspection learning device 112. The shoe appearance inspection device 110 and the shoe appearance inspection learning device 112 can be configured as a computer including a central processing unit (CPU), a graphics processing unit (GPU), random access memory (RAM), read-only memory (ROM), auxiliary storage devices, communication devices, etc. The shoe appearance inspection device 110 and the shoe appearance inspection learning device 112 can be configured as separate computers, or they can be implemented by a single computer that combines the functions of both. In this embodiment, an example of implementation by a separate computer will be described.

[0071] The shoe appearance inspection device 110 is communicatively connected to multiple photographic devices that capture images of multiple shoe products 10. Since shoe products 10 are easily deformed when held by an operator, making accurate inspection difficult, they are photographed, for example, when placed on a table as shown in the figure; however, the method of fixing the shoe products 10 is not limited to this. The multiple photographic devices include a left-side photographic device 50 that photographs the shoe products 10 from the left side, a right-side photographic device 52 that photographs from the right side, an upper photographic device 54 that photographs from directly above, a front photographic device 56 that photographs from the front, a rear photographic device 58 that photographs from the rear, and a lower photographic device 59 that photographs the bottom surface from below. The left-side photographic device 50 photographs the outer instep of the left shoe product 10 and the inner instep of the right shoe product 10. The right-side photographic device 52 photographs the inner instep of the left shoe product 10 and the outer instep of the right shoe product 10. Images captured by the left-side camera 50, right-side camera 52, top camera 54, front camera 56, rear camera 58, and bottom camera 59 are sent to the shoe appearance inspection device 110. The shoe appearance inspection device 110 inspects the appearance of the shoe product 10 based on the received images. A shoe appearance inspection learning device 112 is communicatively connected to the shoe appearance inspection device 110 and performs machine learning on the images of the shoe product 10 to generate a learning model. The learning model is used for inspection by the shoe appearance inspection device 110.

[0072] Figure 2 It is a diagram that compares the upper shape of qualified products with that of unqualified products by taking an image of the outer instep of the footwear from the side. Figure 2 (a) is an example of a qualified product. Figure 2(b) is an example of a defective product. The illustrated shoe product 10 has a sole formed by sequentially attaching the outsole 12, the lower midsole 14, and the upper midsole 16 from bottom to top. Regarding the shoe product 10, it is formed in a so-called last state where the upper 20 is glued to the instep periphery of the shoe last (mold) 30 placed on the upper midsole 16 and then glued to the sole. The lower midsole 14, upper midsole 16, and outsole 12, which contain resins such as ethylene-vinyl acetate (EVA), are not easily deformed after manufacturing and their shapes are generally fixed, except for manufacturing deviations. In contrast, the upper 20 contains materials such as mesh fiber raw materials or leather, whose shapes may not be fixed after manufacturing. When the shoe last 30 is pulled out of the shoe product 10, the height of the toe drops slightly due to the rebound force of the sole, making the upper 20 prone to deformation and causing shape deviations for each individual. In contrast, when the shoe last 30 is used for lasting, the shape of the upper 20 is more easily maintained. Therefore, the state with the shoe last 30 is more suitable for inspection, but even in the state without the shoe last 30, the inspection accuracy is improved through machine learning.

[0073] The upper is attached to the sole after being fitted with a last; however, because this attachment process is done manually, deviations can occur, sometimes resulting in shape discrepancies. Figure 2 In the qualified products of (a), the upper shape 22a has a slightly curved shape from the instep to the toe, in contrast, Figure 2 In (b) of the non-conforming products, the shoe upper shape 22b has a shape that is approximately a straight line from the instep to the toe. These differences are minute and negligible to the naked eye, and are difficult to distinguish without observation from the appropriate direction. Within the shoe as a whole, the positional relationship or length balance of feature points shows clear differences between conforming and non-conforming products. Therefore, by using image processing and machine learning, the error range of the ratio of the length to that of conforming products is modeled. If the error is determined by this learning model to be within the acceptable range, the product is presumed to be conforming; if it exceeds the acceptable range, it is presumed to be non-conforming.

[0074] Figure 3 It is a graph that compares the slope of the axis in the acceptable product with the slope of the axis in the unacceptable product using the back image. Figure 3 (a) is an example of a qualified product. Figure 3 (b) is an example of a defective product. Figure 3 In the qualified products of (a), the horizontal axis 18a is approximately horizontal and the vertical axis 24a is approximately vertical. Conversely, in Figure 3In (b) of the non-conforming products, the horizontal axis 18b is tilted slightly to the left of the horizontal, and the vertical axis 24b is also tilted slightly to the left of the vertical. These differences in slope are so small as to be negligible to the human eye, and difficult to distinguish unless the product is properly placed on a horizontal platform. However, the difference in slope is clearly distinguishable between conforming and non-conforming products. Therefore, by modeling the permissible range of the slope ratio relative to conforming products using image processing and machine learning, if the error is determined by this learning model to be within the permissible range, the product is presumed to be conforming; if it exceeds the permissible range, it is presumed to be non-conforming.

[0075] In this embodiment, running shoes are exemplified as footwear product 10. This product can be used to inspect footwear including various sports shoes or leather shoes, especially various footwear products made by attaching uppers to the soles, as well as sandals or slippers without uppers.

[0076] Errors can occur not only in the uppers but also in the soles, sometimes during the resin molding process. Figure 4 This indicates a positional offset of the high-hardness raw material in the midfoot area of ​​the midsole. In the lower midsole 14, high-hardness raw material is sometimes used locally to ensure rigidity at a position corresponding to the midfoot area of ​​the foot. However, errors may occur in the placement of this high-hardness raw material during the sole forming process. Since the portion using the high-hardness raw material is not visible in the lower midsole 14, the operator cannot easily detect the error visually. The figure shows the outline of the sole. The midfoot area M1, indicated by a diagonal line from the upper left to the lower right, represents the high-hardness raw material portion in a acceptable product. The midfoot area M2, indicated by a diagonal line from the upper right to the lower left, represents the high-hardness raw material portion in a non-acceptable product. As shown in the figure, the positional difference is clear when comparing the two, but when viewed from the inside of the shoe (left side of the figure), there is no positional difference between midfoot area M1 and midfoot area M2. The difference only exists when viewed from the outside of the shoe (right side of the figure). However, when the operator performs a visual inspection, it is difficult to detect positional shifts such as the difference in position between the midfoot M1 and midfoot M2 by simply observing a single shoe.

[0077] On the inner side of the footwear 10, the imaginary lines La and La', from the reference point Ra at the apex of the toe, to the starting points of the midfoot M1 and midfoot M2, have the same length. On the outer side of the footwear 10, the imaginary line Lb, from the reference point Ra to the starting point of the midfoot M1, and the imaginary line Lb', from the reference point Ra to the starting point of the midfoot M2, have different lengths. The width Lc of the midfoot M1 and the width Lc' of the midfoot M2 are approximately the same length in this example. In this case, the magnitude of the error can be represented by the ratio (Ln' / Ln) of the lengths of the imaginary lines of the qualified and unqualified products. Therefore, based on acquiring multiple images of the inspection object taken from multiple photographic directions, the ratio of the lengths of the imaginary lines extracted from each image to the lengths of the imaginary lines extracted from the images of qualified products is calculated and compared with its learning result. If so, it is theoretically possible to determine whether the magnitude of the positional deviation error is within the allowable range.

[0078] Figure 5 This is a functional block diagram showing the basic structure of the shoe appearance inspection device 110. This diagram depicts functional blocks focused on functionality, which can be implemented in various forms by hardware, software, or combinations thereof. The shoe appearance inspection device 110 includes an image acquisition unit 120, an image storage unit 122, a reference point extraction unit 124, a feature point extraction unit 126, an imaginary line extraction unit 128, a contour extraction unit 130, an extraction data storage unit 132, a pass / fail determination unit 134, and a model storage unit 136.

[0079] The image acquisition unit 120 acquires images of the inspection object, i.e., the shoe product 10, from the left-side imaging device 50, the right-side imaging device 52, the top imaging device 54, the front imaging device 56, and the rear imaging device 58, and saves them in the image storage unit 122. In the image storage unit 122, the images are classified and saved together with attribute information such as the product model name, size, and whether it is for the left or right foot of the inspection object, i.e., the shoe product 10.

[0080] The reference point extraction unit 124 extracts reference points using a prescribed reference point extraction method. These reference points serve as feature points on the appearance of the image of the inspection object, i.e., the shoe product 10. The feature point extraction unit 126 extracts multiple feature points using a prescribed feature point extraction method. These multiple feature points serve as feature points on the appearance of the image of the inspection object, i.e., the shoe product 10. The imaginary line extraction unit 128 extracts multiple imaginary lines from the image of the inspection object, i.e., the shoe product 10, connecting the reference points to the multiple feature points. The contour extraction unit 130 extracts the contour of the shoe product 10 from the image of the inspection object, i.e., the shoe product 10.

[0081] Figure 6This diagram schematically illustrates a method for extracting reference points, feature points, and imaginary lines from an image of the outer instep of a shoe with a shoe last, taken from the side. In this embodiment, reference points and feature points, serving as feature points, are extracted from the image of the shoe 10, and imaginary lines connecting these reference points and feature points are extracted. Multiple such imaginary lines are extracted, and the data of these multiple imaginary lines are input into a prescribed machine learning model. The model determines whether the position, slope, ratio of the slope relative to a qualified product, length, and ratio of the length relative to a qualified product of each imaginary line are within an acceptable error range, thereby inferring whether the shoe 10 is a qualified product.

[0082] Furthermore, by inspecting multiple images of a shoe product 10 taken from multiple photographic directions, the determination of whether it is a qualified product is not based solely on images from one direction, but rather on a comprehensive assessment of the entire shoe. This is because even if the positions, slopes, ratios of slopes relative to qualified products, lengths, and ratios of lengths relative to qualified products of multiple imaginary lines extracted from images taken from one side are within acceptable limits, errors exceeding these limits can sometimes occur in the positional relationships between these imaginary lines extracted from multiple images taken from multiple directions.

[0083] Furthermore, a contour is extracted from the image of the footwear 10, and the contour is input into a prescribed machine learning model to determine whether the overall balance of the shape or position of the contour is within the allowable error range, thereby presuming whether the footwear 10 is a qualified product. In contour-based inspection, this can also be determined if errors exceeding the allowable range occur in the positional relationship or balance of the contours among multiple images taken from multiple directions.

[0084] Reference points and feature points are shape-stable feature points on the appearance that can be extracted using a prescribed extraction method based on image processing. The reference point extraction unit 124 extracts a portion of the shoe last 30 protruding from the shoe opening of the shoe product 10, and uses the endpoint closest to the heel on the upper edge detected during edge detection in image processing of the shoe last 30 as the first reference point R1. The upper edge of the shoe last 30 exhibits minimal shape change during the shoe-making process, and the shape of the shoe last 30 used when inspecting other product models is similar; therefore, it is preferable as an easily extractable reference point. Furthermore, by setting a single reference point extracted from an image and connecting multiple feature points one-to-many from a common reference point to extract multiple imaginary lines, the number of feature points to be extracted is not excessively increased, thus avoiding an increase in processing load. In this respect, it is more advantageous in terms of processing load compared to the method of separately connecting multiple feature points one-to-many from multiple different feature points to extract multiple imaginary lines. In addition, as a variation, feature points that can be extracted by taking a pattern or text attached to a part of the shoe last 30 as a marker can also be used as the first reference point R1.

[0085] The feature point extraction unit 126 extracts the foremost point of the toe side of the outsole 12, detected by edge detection during image processing of the outsole 12, as the first feature point P1. The first feature point P1 is the vertex that protrudes to the foremost point in the arc-shaped contour of the toe side of the outsole 12. The imaginary line extraction unit 128 extracts the first imaginary line L1 connecting the first reference point R1 and the first feature point P1.

[0086] The feature point extraction unit 126 extracts the point where the curvature of the raised portion on the toe side changes, detected by edge detection during image processing of the outsole 12, i.e., the starting point of the outsole 12 rising from the ground plane towards the toe (also called the "toe spring starting point"), as the second feature point P2. The imaginary line extraction unit 128 extracts the second imaginary line L2 connecting the first reference point R1 and the second feature point P2.

[0087] The feature point extraction unit 126 extracts the point where the curvature of the raised portion on the heel side changes, detected by edge detection during image processing of the outsole 12, i.e., the starting point of the outsole 12 rising from the ground plane towards the heel (also called the "heel cut-off starting point"), as the third feature point P3. The imaginary line extraction unit 128 extracts the third imaginary line L3 connecting the first reference point R1 and the third feature point P3.

[0088] Feature point extraction unit 126 extracts the last point of the heel side detected by edge detection in the image processing of the shoe sole as the fourth feature point P4. The fourth feature point P4 is the apex protruding to the very end of the arc-shaped contour on the heel side of the lower midsole 14. Imaginary line extraction unit 128 extracts the fourth imaginary line L4 connecting the first reference point R1 and the fourth feature point P4.

[0089] The feature point extraction unit 126 extracts the uppermost point detected by edge detection during image processing of the shoe opening as the fifth feature point P5. The outer and inner sides of the shoe opening of the shoe product 10 are both wavy, and the feature point extraction unit 126 extracts the apex or highest point of the arc of the wavy shape as the fifth feature point P5. The imaginary line extraction unit 128 extracts the fifth imaginary line L5 connecting the first reference point R1 and the fifth feature point P5. Furthermore, if the object of inspection, i.e., the shoe product 10, is a boot or other shoe with a non-wavy shape at the opening, for example, the frontmost or rearmost point of the shoe opening can be extracted as the fifth feature point P5.

[0090] exist Figure 6 The example illustrates the extraction of five feature points and five imaginary lines, but the number of feature points or imaginary lines is not limited to this. Other parts can also be used as feature points depending on the stability of the shoe's shape.

[0091] Figure 7 This diagram schematically illustrates a method for extracting reference points, feature points, and imaginary lines from an image taken from the side of the instep of a shoe without a last. The shoe 10 in this diagram is an example of an inspection performed after the last 30 has been removed. Unlike the case with the last 30, reference points are extracted from a portion of the shoe 10. The reference point extraction unit 124 extracts the lowest point detected during edge detection in the image processing of the shoe opening as a second reference point R2. Since both the outer and inner sides of the shoe opening of the shoe 10 are wavy, the reference point extraction unit 124 extracts the lowest point or lowest point of the arc of the wavy shape as the second reference point R2. Furthermore, if the object of inspection, i.e., the shoe 10, is a boot or similar shoe with a non-wavy opening shape, the rearmost or frontmost point of the shoe opening can be extracted as the reference point, for example. As a variation, any one of multiple feature points can also be set as the reference point.

[0092] The imaginary line extraction unit 128 extracts a first imaginary line L1 connecting the second reference point R2 and the first feature point P1. The imaginary line extraction unit 128 extracts a second imaginary line L2 connecting the second reference point R2 and the second feature point P2. The imaginary line extraction unit 128 extracts a third imaginary line L3 connecting the second reference point R2 and the third feature point P3. The imaginary line extraction unit 128 extracts a fourth imaginary line L4 connecting the second reference point R2 and the fourth feature point P4. The imaginary line extraction unit 128 extracts a fifth imaginary line L5 connecting the second reference point R2 and the fifth feature point P5.

[0093] Figure 8 This schematically illustrates a method for extracting reference points, feature points, and imaginary lines from a rear image of a shoe with a shoe last. The reference point extraction unit 124 extracts the vertex or highest point of the upper edge detected by edge detection during image processing of the shoe last 30 as a third reference point R3. Furthermore, as the third reference point R3 in the rear image, points different from the first reference point R1 in the side image are designated as reference points. However, for example, in the side image, the vertex or highest point of the upper edge detected by edge detection during image processing of the shoe last 30 can also be designated as a reference point, thereby setting common feature points as reference points respectively.

[0094] The feature point extraction unit 126 extracts the leftmost point detected by edge detection in the image processing of the lower mid-base 14 as the sixth feature point P6. The sixth feature point P6 is the leftmost vertex protruding from the arc-shaped contour on the left side of the lower mid-base 14. The imaginary line extraction unit 128 extracts the sixth imaginary line L6 connecting the third reference point R3 and the sixth feature point P6.

[0095] The feature point extraction unit 126 extracts the lowest point detected by edge detection during image processing of the outsole 12 as the seventh feature point P7. The seventh feature point P7 is the lowest point or lowest part of the arc-shaped contour of the outsole 12. The imaginary line extraction unit 128 extracts the seventh imaginary line L7 connecting the third reference point R3 and the seventh feature point P7.

[0096] The feature point extraction unit 126 extracts the rightmost point detected by edge detection in the image processing of the lower mid-base 14 as the eighth feature point P8. The eighth feature point P8 is the rightmost vertex protruding from the arc-shaped contour on the right side of the lower mid-base 14. The imaginary line extraction unit 128 extracts the eighth imaginary line L8 connecting the third reference point R3 and the eighth feature point P8.

[0097] like Figures 6 to 8 As shown, reference points and feature points are also extracted from the images of the right side, front, and top of the footwear 10, and imaginary lines are extracted as well.

[0098] Figure 9Schematically shows a method for extracting a contour in an image taken from the outer dorsal side of a shoe product with a shoe mold. In the image processing of the shoe product 10, the contour extraction unit 130 extracts the overall contour SL of the shoe product 10 through edge detection. Since only one contour SL can be obtained from one image of the shoe product 10, there are the following aspects: As the items that can be inspected, it is less than the inspection using imaginary lines, and in terms of the object of machine learning, it is also less than imaginary lines. Correspondingly, compared with imaginary lines that can learn multiple teacher data, it is difficult to improve the detection accuracy based on learning. On the other hand, different from the extraction of reference points or feature points, the edge detection of the contour SL does not require the features of the shape, so it can be extracted more easily. In addition, the inspection based on the contour can be used not only for the inspection of finished products but also for the inspection of each of the multiple steps included in the manufacturing process.

[0099] As Figure 9 shown, the contour is also extracted from the images on the right side, front side, rear side, upper side, lower side of the shoe product 10 or the image of the shoe product 10 without a shoe mold.

[0100] Refer again to Figure 1 . The model storage unit 136 stores a learning model that has been pre-generated and learned through machine learning. The machine learning uses multiple imaginary lines and contours extracted from the images of multiple shoe products as qualified products as teacher data. The model storage unit 136 stores a learning model generated through machine learning. The machine learning uses multiple imaginary lines and contours respectively extracted from multiple images for one shoe product 10 as teacher data. In the learning model, since the allowable ranges of errors in data such as the position, slope, ratio of the slope relative to the qualified product, length, ratio of the length relative to the qualified product, position of the contour, and balance of multiple qualified products have been formed, by comparing with the position, slope, ratio of the slope relative to the qualified product, length, ratio of the length relative to the qualified product, position of the contour, and balance of the imaginary line extracted from the image of the inspection object, it can be inferred whether the error converges within the allowable range, that is, whether it is a qualified product. As will be described later, the learning model is pre-generated by the shoe appearance inspection learning device 112 and stored in the model storage unit 136.

[0101] The pass / fail determination unit 134 inputs multiple imaginary lines and contours extracted from the image of the inspection object, i.e., the shoe product 10, into the learning model. It compares the position, slope, ratio of the slope to that of a qualified product, length, ratio of the length to that of a qualified product, and the position and balance of the contours. Based on this, it can determine whether the error converges within the acceptable range, i.e., whether the inspection object, i.e., the shoe product 10, is a qualified product. The pass / fail determination unit 134 inputs multiple imaginary lines and contours extracted from multiple images of the inspection object, i.e., the shoe product 10, into the learning model. It comprehensively compares the position, slope, ratio of the slope to that of a qualified product, length, ratio of the length to that of a qualified product, and the position and balance of the contours of the imaginary lines in relation to the shoe as a whole, and determines whether the inspection object, i.e., the shoe, is a qualified product. The pass / fail determination unit 134 outputs the determined result through methods such as screen display and feeds it back to the learning model stored in the model storage unit 136 as data indicating whether the product is qualified or unqualified.

[0102] Figure 10 This is a functional block diagram showing the basic structure of the shoe appearance inspection learning device 112. The image acquisition unit 220, image storage unit 222, reference point extraction unit 224, feature point extraction unit 226, imaginary line extraction unit 228, contour extraction unit 230, and extraction data storage unit 232 correspond to the image acquisition unit 120, image storage unit 122, reference point extraction unit 124, feature point extraction unit 126, imaginary line extraction unit 128, contour extraction unit 130, and extraction data storage unit 132, respectively, and each has the same function.

[0103] The machine learning unit 234 uses data from multiple imaginary lines and contours stored in the extraction data storage unit 232 as teacher data, and generates a learning model through machine learning to determine whether the error between the imaginary lines and contours converges within an acceptable range, and stores it in the model storage unit 236. The learning model is sent to the shoe appearance inspection device 110 for appearance inspection of the shoe product 10.

[0104] Teacher data includes, for example Figures 6 to 9The image shows information about multiple imaginary lines and contours extracted from the shoe. The imaginary line data includes the position, slope, slope ratio relative to the qualified product, length, and length ratio relative to the qualified product, obtained from multiple images of multiple qualified products. The contour data includes the position and balance of the contour, obtained from multiple images of multiple qualified products. Since the imaginary lines and contours exhibit deviations in position, slope, length, and balance, machine learning is used to model the acceptable range of error. The shoe product 10 has multiple types of product models, and each product model has multiple sizes, categorized as left-foot and right-foot. The machine learning unit 234 performs machine learning separately on the multiple imaginary lines and contours according to attributes such as product model, size, and whether it is for left or right feet. By performing machine learning separately according to attributes, the judgment accuracy can be further improved.

[0105] In this embodiment, an example is described of generating a learning model by performing machine learning only on images of footwear 10 that are considered acceptable products. In a variation, the model may be configured to learn and attach acceptable labels to imaginary lines and contours extracted from images of acceptable products, and to learn and attach unacceptable labels to imaginary lines and contours extracted from images of unacceptable products. Compared to learning only on acceptable products, this requires more teacher data, but correspondingly, it improves the accuracy of classifying products as acceptable or unacceptable. Furthermore, in another variation, the learning model may be generated solely through machine learning of imaginary lines, without using contour machine learning.

[0106] Figure 11 This is a flowchart illustrating the process of extracting multiple imaginary lines and contours from an image of a footwear product and using a learning model to infer whether it is a qualified product.

[0107] Image acquisition unit 120 captures images of the footwear 10 from multiple photographic directions using multiple photographic devices such as left-side photographic device 50 or right-side photographic device 52 (S10), thereby acquiring multiple images (S11). Reference point extraction unit 124 and feature point extraction unit 126 extract reference points and multiple feature points from the images (S12). Imaginary line extraction unit 128 extracts multiple imaginary lines from the images based on the reference points and multiple feature points and saves them in extraction data storage unit 132 (S14). Pass / fail determination unit 134 inputs the imaginary line data into the learning model stored in model storage unit 136 and determines whether the error of the imaginary lines is within the allowable range (S16). Contour extraction unit 130 extracts the contour from the images of the footwear 10 and saves it in extraction data storage unit 132 (S18). Pass / fail determination unit 134 inputs the contour data into the learning model stored in model storage unit 136 and determines whether the contour error is within the allowable range (S19). The pass / fail determination unit 134 makes a comprehensive judgment on whether the error of the imaginary line and the error of the outline are within the allowable range, and then presumes whether the footwear product 10 is a pass / fail product (S20).

[0108] The present invention has been described above based on embodiments. Those skilled in the art will understand that the embodiments are illustrative, and various modifications can exist in the combination of their constituent elements or processing techniques; furthermore, such modifications are also within the scope of the present invention.

[0109] Industrial availability

[0110] This invention provides a shoe appearance inspection technology that can improve inspection efficiency and accuracy.

Claims

1. A shoe appearance inspection system, characterized in that, include: The image acquisition unit acquires images of the object being inspected, namely the shoe. The reference point extraction unit extracts reference points using a prescribed reference point extraction method. These reference points serve as feature points on the appearance of the image of the inspected object, i.e., the shoe. The feature point extraction unit extracts multiple feature points using a prescribed feature point extraction method. These multiple feature points serve as appearance feature points in the image of the inspected object, i.e., the shoe. The imaginary line extraction unit extracts multiple imaginary lines from the image of the inspected object, i.e., the shoe, that connect the reference point to the multiple feature points respectively. The model storage unit stores the learning model generated by machine learning, which uses the imaginary lines extracted from images of multiple shoes as qualified products as teacher data. as well as The pass / fail determination unit inputs multiple imaginary lines extracted from the image of the inspection object, i.e., the shoe, into the learning model to determine whether the inspection object, i.e., the shoe, is a qualified product.

2. The shoe appearance inspection system according to claim 1, characterized in that, The obtained image of the shoe is an image of the shoe after it has been fitted with a shoe last using a shoe mold. The reference point extraction unit extracts feature points from the appearance of the shoe model protruding from the shoe in the image as the reference points.

3. The shoe appearance inspection system according to claim 1 or 2, characterized in that, The acquired images of the shoe include multiple images taken from various angles. The model storage unit stores the learning model generated through machine learning, which uses imaginary lines extracted from multiple images for a single shoe as teacher data. The pass / fail determination unit inputs the imaginary lines extracted from multiple images for the inspection object, i.e., the shoe, into the learning model, thereby determining whether the inspection object, i.e., the shoe, is a qualified product.

4. The shoe appearance inspection system according to claim 1 or 2, characterized in that, It also includes a contour extraction unit that extracts the contour of the shoe from the image of the object being inspected, i.e., the shoe. The model storage unit stores the learning model generated through machine learning, which uses the imaginary lines and contours extracted from multiple images of shoes that are considered qualified products as teacher data. The pass / fail determination unit will input multiple imaginary lines and contours extracted from the image of the inspection object, i.e. the shoe, into the learning model, thereby determining whether the inspection object, i.e. the shoe, is a qualified product.

5. A method for inspecting the appearance of shoes, characterized in that, include: The process of acquiring an image of the object to be inspected, namely the shoe, using a specified image acquisition unit; The process of extracting reference points using a prescribed reference point extraction method performed by a computer, wherein the reference points serve as feature points on the appearance of the image of the inspected object, i.e., the shoe. The process of extracting multiple feature points using a prescribed feature point extraction method performed by a computer, wherein the multiple feature points are used as appearance feature points in the image of the inspected object, i.e., the shoe. The process of using a computer to extract multiple imaginary lines from the image of the object under inspection, i.e., the shoe, that connect the reference point to the multiple feature points respectively; The process of reading a learning model generated by machine learning from a specified storage unit, wherein the machine learning extracts the imaginary lines from images of multiple shoes as qualified products as teacher data; as well as The process of using a computer to determine whether the shoe is a qualified product by inputting multiple imaginary lines extracted from the image of the object to be inspected into the learning model.

6. A computer-readable storage medium storing a shoe appearance inspection program, the shoe appearance inspection program being characterized in that it enables a computer to: The function to obtain an image of the object being inspected, namely the shoe; The function of extracting reference points through a specified reference point extraction method, wherein the reference points serve as feature points on the appearance of the image of the inspected object, i.e., the shoe. The function of extracting multiple feature points through a specified feature point extraction method, wherein the multiple feature points are used as appearance feature points in the image of the inspected object, namely the shoe. The function of extracting multiple imaginary lines from the image of the inspected object, i.e., the shoe, to connect the reference point with the multiple element points respectively; The ability to store learning models generated through machine learning, which uses the imaginary lines extracted from images of multiple shoes as qualified products as teacher data; as well as Multiple imaginary lines extracted from the image of the inspection object, i.e., the shoe, are input into the learning model to determine whether the inspection object, i.e., the shoe, is a qualified product.

Images