Image processing method and training data generation method

Abstract

The purpose of the present invention is to contribute to the generation of training data using features of an image other than an object. The image processing method includes: a transformation step of performing a first affine transformation on a first image occupying a first region to obtain a second image occupying a second region that overlaps and does not coincide with the first region; and a moving step of moving or copying pixels of the second image to the first region to obtain a third image. The second area is divided into a third area and a fourth area, the third area is located in the first area, the fourth area is located outside the first area, the first area is divided into a third area and a fifth area, and the fifth area is located outside the second area. The movement step includes: a first step of setting a movement source pixel group, which is a group of pixels connected to a first number included in the fourth region; and a second step of performing a second affine transformation on the movement source pixel group to obtain a movement destination pixel group disposed in the fifth region.

Description

Technical Field

[0001] The present disclosure relates to an image processing method and a training data generation method. Background Art

[0002] Machine learning, which uses training data to identify desired objects in images, is well known. Using data augmentation to increase training data is one method for improving the accuracy of machine learning. Data augmentation involves, for example, adjusting image brightness, rotating the image, or magnifying the image to generate new training data.

[0003] For example, Patent Document 1 listed below discloses a technique of generating a cropped image obtained by cutting out a predetermined area including the object from an image of the object, rotating the cropped image, and attaching it to a background image.

[0004] Prior art literature

[0005] Patent document: Japanese Patent Application Laid-Open No. 2022-137611. Summary of the Invention

[0006] Technical problem to be solved by the invention

[0007] Patent Document 1 does not disclose how to set the background image. For example, when generating second training data from first training data, it is unclear whether information about the image of the area after removing the cropped image from the image of the object is reflected in the second training data. In this case, for example, the brightness distribution in the second training data may differ from that in the first training data.

[0008] For example, assuming that the brightness value of the background image is fixed, not only the brightness distribution but also the ratio of images in the second training data will differ from the ratio of images in the first training data. Regarding the data expansion method for obtaining the second training data, for example, it is believed that there is room for improvement in removing image information in the area after the cropped image.

[0009] The present disclosure has been made in view of the above-mentioned problems, and an object thereof is to contribute to the generation of training data that utilizes features of images other than objects.

[0010] Technical means to solve the problem

[0011] The first mode of the image processing method disclosed herein includes: a transformation process, performing a first affine transformation on a first image having a plurality of pixels and occupying a first area to obtain a second image occupying a second area that overlaps with and is inconsistent with the first area and has the plurality of pixels; and a movement process, moving or copying the plurality of pixels of the second image to the first area to obtain a third image.

[0012] The second area is divided into a third area and a fourth area. The third area is located within the first area, and the fourth area is located outside the first area.

[0013] The first area is divided into the third area and the fifth area, and the fifth area is located outside the second area.

[0014] The moving process includes: a first process of setting a first moving source pixel group, which is a pixel group connected to the multiple pixels contained in the fourth area, that is, a specified first number of pixels greater than 1; and a second process of performing a second affine transformation on the first moving source pixel group to obtain a first moving destination pixel group arranged in the fifth area.

[0015] The first affine transformation is either or both of a rotation and a parallel translation. The second affine transformation is either or both of a rotation and a parallel translation, or a mirror transformation.

[0016] A second aspect of the image processing method disclosed herein is the first aspect, wherein the first movement source pixel group is inscribed in the fourth area, and the first movement destination pixel group is inscribed in the fifth area.

[0017] A third aspect of the image processing method disclosed herein is the first or second aspect, wherein the movement step includes: a third step of setting a second movement source pixel group, the second movement source pixel group being a pixel group connected to the plurality of pixels included in the fourth region, i.e., a predetermined second number of pixels; and a fourth step of performing a third affine transformation on the second movement source pixel group to obtain a second movement destination pixel group arranged within the fifth region. The second number is equal to or less than the first number. The third affine transformation is either or both of a rotation and a parallel translation, or a mirror transformation.

[0018] A fourth aspect of the image processing method disclosed herein is the third aspect, wherein the second number is 1. The third step and the fourth step are repeatedly performed until all of the plurality of pixels in the second image are moved or copied to the first area.

[0019] A fifth aspect of the image processing method disclosed herein is the third aspect, wherein the first step and the second step are performed a plurality of times before the third step.

[0020] For example, a partitioning pattern that starts at a position away from the third area, connects pixels in the fourth area, and is enlarged is set as the first movement source pixel group.

[0021] For example, the first movement destination pixel group is arranged in a partitioned area that is enlarged by connecting pixels in the fifth area starting from a position away from the third area.

[0022] For example, the fourth region and the fifth region are in a mirror image relationship, and the second affine transformation is a mirror image transformation.

[0023] For example, the fourth region and the fifth region are in a mirror image relationship, and the third affine transformation is a mirror image transformation.

[0024] The training data generation method disclosed herein is a method for generating training data for machine learning used to identify objects in images. The method includes the following steps: using a plurality of third images obtained by any of the image processing methods disclosed herein as the training data; photographing the object and setting the first image before the image processing method; and, when each of the third images is obtained, regionally extracting a group of pixels corresponding to the object in either the first or second image from a group of pixels located in the third region in both the first and second images.

[0025] Effects of the Invention

[0026] According to the first aspect of the image processing method of the present disclosure, training data for an object labeled in the third area can be easily obtained by extracting features other than just the brightness distribution of images other than the object.

[0027] According to the second aspect, the third aspect, and the fifth aspect of the image processing method of the present disclosure, it is possible to obtain training data that further reflects features of images other than the object.

[0028] According to the fourth aspect of the image processing method of the present disclosure, it is easy to reflect information on brightness distribution among features of images other than the target in training data.

[0029] For example, the training data obtained by the training data generation method disclosed in the present invention is provided to machine learning for recognizing the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a diagram illustrating a first image.

[0031] Figure 2 A diagram showing the relationship between the second image and the first area.

[0032] Figure 3 This is a diagram illustrating an example of the third image.

[0033] Figure 4is a diagram illustrating another example of the third image.

[0034] Figure 5 This is a flowchart illustrating the process of creating a training image.

[0035] Figure 6 is a flowchart illustrating the content of the first pixel group movement process.

[0036] Figure 7 This is a flowchart illustrating the content of the division process.

[0037] Figure 8 This is a diagram illustrating the first area, the second area, the third area, and the fourth area.

[0038] Figure 9 This is a diagram illustrating the first area, the second area, the third area, and the fourth area.

[0039] Figure 10 This is a diagram illustrating the first area, the second area, the third area, and the fourth area.

[0040] Figure 11 This is a diagram illustrating the first area, the second area, the third area, and the fourth area.

[0041] Figure 12 This is a diagram illustrating the first area, the second area, the third area, and the fourth area.

[0042] Figure 13 This is a diagram illustrating the first area, the second area, the third area, and the fourth area.

[0043] Figure 14 This is a diagram illustrating the first area, the second area, the third area, and the fourth area.

[0044] Figure 15 This is a diagram illustrating the first area, the second area, the third area, and the fourth area.

[0045] Figure 16 This is a diagram illustrating the first area, the second area, the third area, and the fourth area.

[0046] Figure 17 This is a diagram illustrating the first area, the second area, the third area, and the fourth area.

[0047] Figure 18 is a flowchart illustrating the content of the second pixel group movement process.

[0048] Figure 19 This is a flowchart showing a first alternative example of the content of the first pixel group movement process.

[0049] Figure 2035 is a flowchart illustrating the content of the division process performed in step S35c.

[0050] Figure 21 is a flowchart illustrating the content of the first configuration process.

[0051] Figure 22 305 is a flowchart illustrating the content of the division process performed in step S305.

[0052] Figure 23 It is a conceptual diagram explaining the division process performed in step S35c.

[0053] Figure 24 It is a conceptual diagram explaining the division process performed in step S35c.

[0054] Figure 25 It is a conceptual diagram explaining the division process performed in step S35c.

[0055] Figure 26 It is a conceptual diagram explaining the division process performed in step S35c.

[0056] Figure 27 It is a conceptual diagram explaining the division process performed in step S35c.

[0057] Figure 28 It is a conceptual diagram explaining the division process performed in step S35c.

[0058] Figure 29 It is a conceptual diagram explaining the division process performed in step S35c.

[0059] Figure 30 It is a conceptual diagram explaining the division process performed in step S305.

[0060] Figure 31 It is a conceptual diagram explaining the division process performed in step S305.

[0061] Figure 32 It is a conceptual diagram explaining the division process performed in step S305.

[0062] Figure 33 It is a conceptual diagram explaining the division process performed in step S305.

[0063] Figure 34 It is a conceptual diagram explaining the division process performed in step S305.

[0064] Figure 35 It is a conceptual diagram explaining the division process performed in step S305.

[0065] Figure 36This is a flowchart showing a second alternative example of the content of the first pixel group movement process.

[0066] Figure 37 35 is a flowchart illustrating the content of the division process executed in step S35d.

[0067] Figure 38 This is a block diagram illustrating the generation and use of training images.

[0068] Figure 39 A diagram illustrating an example display image. DETAILED DESCRIPTION

[0069] Below, with reference to the attached Figure 1 The following describes various embodiments of the present disclosure. The components described in each embodiment are merely illustrative, and the scope of the present disclosure is not intended to be limited to these illustrative examples. In the accompanying drawings, the dimensions and quantities of various parts are sometimes exaggerated or simplified as needed to facilitate understanding. In the accompanying drawings, parts with the same structure and function are marked with the same reference numerals, and repeated descriptions are omitted as appropriate.

[0070] In this specification, expressions indicating relative or absolute positional relationships (e.g., "rotation"), unless otherwise specified, not only strictly indicate the positional relationship but also include tolerances and indicate relative displacements with respect to angles or distances within a range that achieves the same degree of functionality. Expressions indicating the equality of two or more entities (e.g., "superposition"), unless otherwise specified, not only strictly indicate quantitative equality but also include tolerances or differences that allow for the same degree of functionality.

[0071] Unless otherwise specified, expressions indicating shapes (e.g., “square”) are intended to indicate not only a strictly geometrically defined shape but also a range within which the same degree of effect is achieved.

[0072] The expression “equipped with,” “having,” “equipped with,” “including,” or “containing” one component is not intended to be exclusive so as to exclude the presence of other components.

[0073] The expression "connection", unless otherwise specified, refers to the state where two elements are connected.

[0074] <1.Outline>

[0075] Figure 1 This is a diagram illustrating an image F1 having a plurality of pixels obtained by photographing a predetermined object. For example, the predetermined object is a chuck pin. The chuck pins are provided in plurality on a base for holding a semiconductor wafer (not shown) to support the periphery of the semiconductor wafer. Figure 1In FIG, the chuck pins are schematically represented as a set of three cylinders, presented as a pixel group 100 forming part of the first image F1.

[0076] The first image F1 occupies the first region R1. For example, the first region R1 has an outline Q. In the first image F1, pixels are arranged in a matrix along two non-parallel directions. Here, the two directions are orthogonal, and the outline Q is roughly rectangular. For example, if each pixel in the first image F1 is square, the outline Q is rectangular. For example, if each pixel in the first image F1 is circular, the outline Q is a shape with semicircles arranged on each side of the rectangle, which is included in the above-mentioned "roughly rectangular" shape.

[0077] To avoid complication in the diagram, Figure 1 In FIG, the reference numerals representing the first region R1 and the reference numerals representing the first image F1 are collectively labeled “F1 ( R1 )”. This labeling does not mean that the first region R1 and the first image F1 are identical.

[0078] Figure 2 is a diagram showing the relationship between the second image F2 and the first region R1. Figure 2 In the figure, the reference numerals representing the first region R1 and the reference numerals representing the outline Q are collectively labeled "Q(R1)". This label does not imply the identity of the first region R1 and the outline Q. The second image F2 is an image having a plurality of pixels obtained by applying the first affine transformation to the first image F1. The second image F2 occupies the second region R2. To avoid complication in the figure, Figure 2 In FIG. 1 , the reference numerals representing the second region R2 and the reference numerals representing the second image F2 are collectively labeled “F2 ( R2 ).” This labeling does not indicate that the second region R2 and the second image F2 are identical.

[0079] The first affine transformation is either a rotation or a parallel translation, or both, in the affine transformation. The second region R2 overlaps and does not coincide with the first region R1. When the first affine transformation uses a rotation angle of 2nπ radians (n is an integer), it is accompanied by a parallel translation of a non-zero distance. When the first affine transformation uses a parallel translation distance of zero, it is accompanied by a rotation of an angle other than 2nπ radians (n is an integer).

[0080] The second region R2 is divided into a third region R3 and a fourth region R4. The fourth region R4 is a collective term for one or more and four or fewer fourth regions, here four fourth regions R41, R42, R43, and R44. The third region R3 is located within the first region R1, while the fourth region R4 is located outside the first region R1. The third region R3 overlaps the first region R1 and the second region R2. However, in the third region R3, pixels in the first image F1 and pixels in the second image F2 do not necessarily overlap.

[0081] The first region R1 is divided into a third region R3 and a fifth region R5. The fifth region R5 is a general term for one or more and four or fewer fifth regions R51, R52, R53, and R54. The fifth region R5 is located outside the second region R2.

[0082] The pixel group 100 is located in the third region R3 in both the first image F1 and the second image F2. The pixel group 100 is labeled by region extraction from the first image F1 or the second image F2.

[0083] Assume machine learning for recognizing an object (a chuck pin in the above example) in an image. In this machine learning, images are provided as training data (hereinafter referred to as "training images"). As one of the training images, the first image F1 can be used.

[0084] Figure 3 The third image 3A is an example of a third image 3A. The third image 3A is obtained by moving or copying a plurality of pixels of the second image F2 to the first region R1 (also referred to as a "moving process"). It can also be said that the third image 3A is obtained by data expansion of the first image F1. To avoid complication in the diagram, Figure 3 In FIG. 1 , the reference numerals representing the first region R1 and the reference numerals representing the outer contour Q are collectively labeled “Q(R1)”. This labeling does not imply that the first region R1 and the outer contour Q are identical.

[0085] The movement process, for example, includes a first process and a second process. The first process sets the movement source pixel group Gk (k is a positive integer) contained in the fourth region R4. The movement source pixel group Gk is a pixel group having a number Mk (k is the same as k in the movement source pixel group Gk: Mk is an integer greater than 1) of pixels connected thereto. For example, the number Mk is the square of the square of an integer greater than 2 (i.e., Mk ≥ 4), and the pixels in the movement source pixel group Gk are arranged in a square. In the present disclosure, "arranged in a square" means arranged in a matrix along two mutually orthogonal directions.

[0086] The second step applies a second affine transformation to the source pixel group Gk to obtain the destination pixel group Hk (k is the same as k in the source pixel group Gk). The second affine transformation is either a rotation or a parallel translation, or both, or a mirror transformation. When the second affine transformation uses a rotation angle of 2nπ radians (n is an integer), the second affine transformation is accompanied by a parallel translation of a non-zero distance. When the second affine transformation uses a parallel translation of zero distance, the second affine transformation is accompanied by a rotation of an angle other than 2nπ radians (n is an integer).

[0087] The following example illustrates a case where the sum of the rotation angles of the first affine transformation and the second affine transformation is mπ / 2 (where m is an integer) radians. In this case, if the pixels in the source pixel group Gk are arranged in a square, the pixels in the destination pixel group Hk are also arranged in a square.

[0088] The destination pixel group Hk is arranged in the fifth region R5. In the above example, the destination pixel group Hk is arranged in any one of the fifth regions R51, R52, R53, and R54.

[0089] Images 31a, 32a, 33a, 34a occupy fifth regions R51, R52, R53, R54, respectively. Figure 3 In FIG, the reference numerals representing the fifth region R51 and the reference numerals representing the image 31a are combined and labeled as "31a (R51)". This label does not mean that the fifth region R51 is identical to the image 31a. Figure 3 In FIG, the reference numerals representing the fifth region R52 and the reference numerals representing the image 32a are combined and labeled as "32a (R52)". This label does not mean that the fifth region R52 is identical to the image 32a. Figure 3 In FIG, the reference numerals representing the fifth region R53 and the reference numerals representing the image 33a are combined and labeled as "33a (R53)". This label does not mean that the fifth region R53 is identical to the image 33a. Figure 3 In FIG. 3 , the reference numerals representing the fifth region R54 and the reference numerals representing the image 34 a are collectively labeled “ 34 a ( R54 ).” This labeling does not indicate that the fifth region R54 is identical to the image 34 a .

[0090] Image 30a occupies the third region R3, which is consistent between the second image F2 and the third image F3. Figure 3 In FIG. 1 , the reference numerals representing the third region R3 and the reference numerals representing the image 30 a are collectively labeled “ 30 a ( R3 ).” This labeling does not mean that the third region R3 is identical to the image 30 a.

[0091] The information contained in the source pixel group Gk is reflected in the destination pixel group Hk. Both the first and second affine transformations involve either or both rotation and parallel translation, with the number Mk being greater than one. Not only the brightness or hue of each pixel constituting the source pixel group Gk, but also the positional relationships between those pixels are reflected in the destination pixel group Hk. The third image 3A differs from the first image F1 and largely reflects the information contained in the first image F1. The third image 3A is suitable as a training image for machine learning used to identify objects in images. For example, the third image 3A can be used together with the first image F1 as a training image.

[0092] <2. Example of Arrangement of Destination Pixel Group Hk>

[0093] For example, the outer contour of the movement source pixel group Gk overlaps with the outer contour of the movement destination pixel group Hk. Figure 3 , exemplified is a case where pixels in the movement source pixel group Gk are arranged in a square, pixels in the movement destination pixel group Hk are also arranged in a square, and the integer k is greater than or equal to 1 and less than or equal to 4.

[0094] Images 31a, 32a, 33a, and 34a include pixel group sets 310a, 320a, 330a, and 340a, respectively. Pixel group set 310a is a set of pixel groups 311a, 312a, 313a, 314a, 315a, 316a, and 317a, which are examples of any of the movement destination pixel groups H1, H2, H3, and H4. Pixel group set 320a is a set of pixel groups 321a, 322a, 323a, and 324a, which are examples of any of the movement destination pixel groups H1, H2, H3, and H4. Pixel group set 330a is a set of pixel groups 331a, 332a, 333a, and 334a, which are examples of any of the movement destination pixel groups H1, H2, H3, and H4. The pixel group set 340 a is a set of pixel groups 341 a , 342 a , 343 a , 344 a , 345 a , and 346 a , which are examples of any of the movement destination pixel groups H1 , H2 , H3 , and H4 .

[0095] By updating the number Mk and repeatedly executing the first and second steps in pairs, the movement destination pixel groups Hk of different sizes are obtained. Figure 3Pixel groups 311a and 331a have the same number of pixels, M1, which is greater than the number of pixels in each of the other pixel groups 312a, 313a, 314a, 315a, 316a, 317a, 321a, 322a, 323a, 324a, 332a, 333a, 334a, 341a, 342a, 343a, 344a, 345a, and 346a. By performing the paired first and second steps, pixel group 311a is arranged in fifth region R51, and pixel group 331a is arranged in fifth region R53.

[0096] Pixel groups 312a, 321a, and 332a have the same number of pixels, M2. This number M2 is less than M1 and greater than the number of pixels in each of pixel groups 313a, 314a, 315a, 316a, 317a, 322a, 323a, 324a, 333a, 334a, 341a, 342a, 343a, 344a, 345a, and 346a. Pixel group 312a is located in the fifth region R51, pixel group 321a is located in the fifth region R52, and pixel group 332a is located in the fifth region R53.

[0097] Pixel groups 313a, 314a, 322a, 333a, 341a, and 342a have an equal number of pixels, M3. This number M3 is less than M2 and greater than the number of pixels in each of pixel groups 315a, 316a, 317a, 323a, 324a, 334a, 343a, 344a, 345a, and 346a. Pixel groups 313a and 314a are located in the fifth region R51, pixel group 322a is located in the fifth region R52, pixel group 333a is located in the fifth region R53, and pixel groups 341a and 342a are located in the fifth region R54.

[0098] Each of pixel groups 315a, 316a, 317a, 323a, 324a, 334a, 343a, 344a, 345a, and 346a has the same number of pixels, M4, which is less than M3. Pixel groups 315a, 316a, and 317a are arranged in a fifth region R51, pixel groups 323a and 324a are arranged in a fifth region R52, pixel group 334a is arranged in a fifth region R53, and pixel groups 343a, 344a, 345a, and 346a are arranged in a fifth region R54.

[0099] The first and second steps are repeated in this way to obtain pixel groups 310a, 320a, 330a, and 340a. For example, a minimum value is set for the number Mk of pixels when the first and second steps are repeated. Figure 3For example, the number M4 is set to the minimum value of the number Mk.

[0100] The larger the number Mk, the more the third image 3A reflects the information of the first image F1, but the number of the obtained movement destination pixel groups Hk is small. Since Mk≥2, the entire fifth region R5 is not necessarily completely occupied by the movement destination pixel groups Hk. Figure 3 For example, the case where the images 31a, 32a, 33a, and 34a respectively have pixels other than the pixel group sets 310a, 320a, 330a, and 340a is illustrated.

[0101] Within the area of the fifth region R5 not occupied by the destination pixel groups H1, H2, H3, and H4, a second affine transformation is performed on each pixel in the fourth region R4 other than the source pixel groups G1, G2, G3, and G4, thereby repositioning the pixels. This repositioning results in the fifth region R5 being completely occupied by the pixels located in the fourth region R4.

[0102] Among the pixels other than the pixel group set 310a in image 31a, the pixels other than the pixel group set 320a in image 32a, the pixels other than the pixel group set 330a in image 33a, and the pixels other than the pixel group set 340a in image 34a, the second affine transformation is applied to each of the pixels other than the moving source pixel groups G1, G2, G3, and G4 in the fourth region R41, R42, R43, and R44.

[0103] For example, among the pixels outside the pixel group set 310a in image 31a, pixels in which the second affine transformation is applied to each pixel other than the moving source pixel group G1, G2, G3, and G4 in the fourth region R41 are used. For example, among the pixels outside the pixel group set 320a in image 32a, pixels in which the second affine transformation is applied to each pixel other than the moving source pixel group G1, G2, G3, and G4 in the fourth region R42 are used. For example, among the pixels outside the pixel group set 330a in image 33a, pixels in which the second affine transformation is applied to each pixel other than the moving source pixel group G1, G2, G3, and G4 in the fourth region R43 are used. For example, among the pixels outside the pixel group set 340a in image 34a, pixels in which the second affine transformation is applied to each pixel other than the moving source pixel group G1, G2, G3, and G4 in the fourth region R44 are used.

[0104] The number Mk can be a fixed value greater than 2. Figure 3For example, the number M1 is set to the minimum value of the number Mk, the pixel group set 310a includes only the pixel group 311a as an example of the movement destination pixel group H1, the pixel group set 330a includes only the pixel group 331a as an example of the movement destination pixel group H1, and the pixel group sets 320a and 340a are empty sets that do not include the movement destination pixel group H1. In this case, pixels other than the pixel group 311a in image 31a, pixels in image 32a, pixels other than the pixel group 331a in image 33a, and pixels in image 34a are pixels that have been subjected to the second affine transformation for each pixel other than the movement source pixel group G1 in the fourth regions R41, R42, R43, and R44.

[0105] The fact that the number Mk is not fixed but updated and the first and second steps are repeatedly executed contributes to the third image F3 largely reflecting the information of the first image F1 .

[0106] Alternatively, pixels obtained by performing the second affine transformation on all pixels in the fourth region R4 may be arranged in the fifth region R5. This is the case where the source pixel group Gk has only a single number of pixels, and Mk is fixed to 1. This case is less desirable than a case where the number Mk is 2 or greater because it is difficult to reflect the positional relationship between the pixels constituting the source pixel group Gk on the destination pixel group Hk.

[0107] Figure 4 The third image 3B is a diagram illustrating an example of a third image 3B. The third image 3B is obtained by converting the first affine transformation and the second affine transformation into the same image in terms of rotation angle and movement distance. Figure 1 、 Figure 2 、 Figure 3 The situations illustrated in the example are not obtained at the same time.

[0108] Images 31b, 32b, 33b, 34b occupy fifth regions R51, R52, R53, R54, respectively. Figure 4 In FIG, the reference numerals representing the fifth region R51 and the reference numerals representing the image 31b are combined and labeled as "31b (R51)". This label does not mean that the fifth region R51 is identical to the image 31b. Figure 4 In the figure, the reference numerals representing the fifth region R52 and the reference numerals representing the image 32b are combined and labeled as "32b (R52)". This label does not mean that the fifth region R52 is identical to the image 32b. Figure 4 In FIG, the reference numerals representing the fifth region R53 and the reference numerals representing the image 33b are combined and labeled as "33b (R53)". This label does not mean that the fifth region R53 is identical to the image 33b. Figure 4In FIG. 3 , the reference numerals representing the fifth region R54 and the reference numerals representing the image 34 b are collectively labeled “ 34 b ( R54 ).” This labeling does not indicate that the fifth region R54 is identical to the image 34 b .

[0109] Image 30b occupies the third region R3, which is consistent between the second image F2 and the third image F3. Figure 4 In FIG. 1 , the reference numerals representing the third region R3 and the image 30 b are combined and labeled as “ 30 b ( R3 )”. This labeling does not mean that the third region R3 and the image 30 b are identical.

[0110] Images 31b, 32b, 33b, and 34b have pixel group sets 310b, 320b, 330b, and 340b, respectively. Pixel group set 310b is a set of pixel groups 311b, 312b, 313b, 314b, 315b, and 316b, which are examples of any of the movement destination pixel groups Hk. Pixel group set 320b is a set of pixel groups 321b and 322b, which are examples of any of the movement destination pixel groups Hk. Pixel group set 330b is a set of pixel groups 331b, 332b, 333b, 334b, and 335b, which are examples of any of the movement destination pixel groups Hk. Pixel group set 340b is a set of pixel groups 341b and 342b, which are examples of any of the movement destination pixel groups Hk.

[0111] By updating the number Mk and repeating the first and second steps, the pixel group set 310b is arranged in the fifth region R51, the pixel group set 320b is arranged in the fifth region R52, the pixel group set 330b is arranged in the fifth region R53, and the pixel group set 340b is arranged in the fifth region R54.

[0112] Among the pixels other than the pixel group set 310b in image 31b, the pixels other than the pixel group set 320b in image 32b, the pixels other than the pixel group set 330b in image 33b, and the pixels other than the pixel group set 340b in image 34b, a second affine transformation is applied to each of the pixels other than the moving source pixel group Gk in the fourth region R41, R42, R43, and R44.

[0113] For example, for pixels outside the pixel group set 310b in image 31b, pixels in which the second affine transformation is applied to each pixel other than the moving source pixel group Gk in the fourth region R41 are used. For example, for pixels outside the pixel group set 320b in image 32b, pixels in which the second affine transformation is applied to each pixel other than the moving source pixel group Gk in the fourth region R42 are used. For example, for pixels outside the pixel group set 330b in image 33b, pixels in which the second affine transformation is applied to each pixel other than the moving source pixel group Gk in the fourth region R43 are used. For example, for pixels outside the pixel group set 340b in image 34b, pixels in which the second affine transformation is applied to each pixel other than the moving source pixel group Gk in the fourth region R44 are used.

[0114] For example, the third images 3A and 3B are used as training images. In this case, the first image F1 may or may not be used as the training image.

[0115] Figure 5 This is a flowchart illustrating the process of creating training images. Figure 5 This process is labeled as “creation of training image” in FIG. This process includes steps S1 , S2 , S3 , S4 , and S5 , which are performed in sequence.

[0116] Step S1 sets the first image F1. Specifically, an image obtained by capturing an object, such as a chuck pin, is set as the first image F1. For example, the chuck pin appears as a pixel group 100 in the third region R3 of the first image F1.

[0117] Step S2 generates the second image F2. Specifically, step S2 generates the second image F2 from the first image F1 by the first affine transformation (see Figure 2 ).

[0118] Step S3 executes the first pixel group movement process. The first pixel group movement process is a process of setting a movement source pixel group Gk and obtaining a movement destination pixel group Hk by performing a second affine transformation on it. Figure 3 For example, by executing step S3, pixel group sets 310a, 320a, 330a, and 340a are obtained. Figure 4 Specifically, by executing step S3, pixel group sets 310b, 320b, 330b, and 340b are obtained.

[0119] Step S4 executes the second pixel group movement process. This second pixel group movement process applies a second affine transformation to each pixel in the fourth region R4, other than the source pixel group Gk, within the range of the fifth region R5 not occupied by the destination pixel group Hk in the first pixel group movement process, thereby repositioning the pixels. In the above example, after executing step S4, third images 3A and 3B are obtained.

[0120] Step S5 executes saving of the third image. The saved third image can be used as a training image.

[0121] The annotation process for obtaining pixel group 100 is performed in either step S1 or step S2 for each third image. However, this annotation process extracts the pixel group corresponding to the object from the pixel group located in the third region R3 in both the first image F1 and the second image F2. Because pixels in the first image F1 and the second image F2 do not necessarily overlap in the third region R3, regional extraction is performed from the overlapping pixel groups.

[0122] For example, when a plurality of first affine transformations are predefined, a third region R3 is predefined for each third image. Among images of the object (described later as photographic image F0), an image of the object captured at a position corresponding to the determined third region R3 is adopted as the first image F1.

[0123] <3. Example of Setting the Movement Source Pixel Group Gk>

[0124] Figure 6 2 is a flowchart illustrating the content of the first pixel group movement process executed in step S3. Step S3 includes steps S31, S32, S33, S34, S35a, S35b, S36, S37, S38, S39, and S30.

[0125] Figure 7 This is a flowchart illustrating the content of the division process executed in steps S35a and S35b. Figure 8 、 Figure 9 、 Figure 10 It is a diagram illustrating the case where step S35a is executed sequentially. Figures 11 to 17 Detailed description will be given later, but both steps S35a and S35b can be said to be steps of obtaining a set of segmentation patterns that are candidates for the movement source pixel group Gk from the fourth region R4.

[0126] In step S2 (refer to Figure 5), step S31 is executed. Step S31 identifies the third region R3, the fourth region R4, and the fifth region R5. Such identification is performed by comparing the first image F1 with the second image F2. For example, the first region R1 is identified as the region occupied by the first image F1, the outline Q is extracted from the first region R1, and the third region R3 and the fourth region R4 are identified based on the comparison of the outline Q with the second region R2. The fifth region R5 is identified by comparing the first region R1 with the third region R3. Figure 6 In the embodiment, step S31 is simply labeled as "identification of the third to fifth areas".

[0127] After executing step S31, execute step S32. Step S32 specifies one of the fourth regions R4. Figure 2 In the example of the fourth region R4, fourth regions R41, R42, R43, and R44 are shown. Step S32 selects and specifies these multiple regions, for example, without duplication. Hereinafter, the region specified in step S32 in the fourth region R4 may be referred to as "fourth region R4z."

[0128] After executing steps S33 and S35a, or steps S33, S34, S35b, S36, S37, and S38, described later, a determination is made in step S39 as to whether all fourth regions R4 have been designated. If the determination in step S39 is negative, step S32 is executed again to select and designate undesignated regions within the fourth region R4. This can also be considered an update of the fourth region R4z.

[0129] In step S33 , the number of convex corners whose vertices are not positioned on the outer contour Q (hereinafter and in the drawings, simply referred to as “convex corners”) among the convex corners in the fourth region R4z is determined. Figure 2 The fourth region R41, R42, R43, and R44 shown in the example are all one (single) convex corner. Figures 8 to 17 , based on the relationship between the case where the convex angle is singular and the case where the convex angle is plural, an example is given with Figures 1 to 3 The first region R1, the second region R2, the third region R3, and the fourth region R4 are arranged in different situations. In the following, for simplicity of explanation, the case where the pixel shape is square and the first region R1 and the second region R2 are both rectangular is exemplified.

[0130] exist Figures 8 to 17 In the embodiment, the first region R1 is divided into a third region R3 and fifth regions R55 and R56, and the second region R2 is divided into a third region R3 and fourth regions R45 and R46.

[0131] As described above, the first region R1 is a rectangle having vertices P51, P52, P53, and P54 located in the counterclockwise direction. The rectangle has a side L512 connecting points P51 and P52, a side L523 connecting points P52 and P53, a side L534 connecting points P53 and P54, and a side L541 connecting points P54 and P51.

[0132] As described above, the second region R2 is a rectangle having vertices P41, P42, P43, and P44 located in the counterclockwise direction. The rectangle has a side L412 connecting points P41 and P42, a side L423 connecting points P42 and P43, a side L434 connecting points P43 and P44, and a side L441 connecting points P44 and P41.

[0133] The side L441 intersects the side L541 at a point P61, the side L412 intersects the side L523 at a point P62, the side L423 intersects the side L523 at a point P63, and the side L423 intersects the side L534 at a point P64.

[0134] The third region R3 forms a hexagonal shape bounded by points P41, P62, P63, P64, P54, and P61. The fourth region 45 has only one convex corner, the one with point P42 as its vertex. The fourth region 46 has only one convex corner, the one with point P43 as its vertex, and the one with point P44 as its vertex. The corner with point P54 as its vertex is a concave corner, and the corners with point P61 and point P64 as their vertices are both located on the outer contour Q. These corners do not qualify as "convex corners" in the aforementioned sense.

[0135] When the fourth region R45 is specified in step S32, the judgment result in step S33 is “single.” When the fourth region R46 is specified in step S32, the judgment result in step S33 is “plural.”

[0136] If the result of the judgment in step S33 is "odd", step S35a is executed. Step S35a is a segmentation process, which includes steps S351, S352, S353, S354, S355, and S356 (see Figure 7 ). This segmentation process is adopted in both steps S35a and S35b.

[0137] The specific content of step S35a is as follows: the fourth region R45 is specified in step S32. Figures 7 to 11 And explain.

[0138] In step S35a, step S351 is executed. Step S351 specifies the vertex of the convex corner as the starting point in the process for obtaining the distinguishing figure. Figure 8 For example, point P42 is the vertex, which becomes the starting point.

[0139] After executing step S351, step S352 is executed. Step S352 enlarges the partitioning pattern. For example, a square is used as the partitioning pattern. For example, the initial value of the partitioning pattern used in step S352 is a square with four pixels. The amount by which the partitioning pattern is enlarged in step S352 is, for example, the number of pixels set to one in each of two directions parallel to the adjacent sides of the square serving as the partitioning pattern.

[0140] Figure 8 This illustrates a state where the square is enlarged with point P42 as the starting point in step S352 to obtain a segmentation pattern T40.

[0141] After executing step S352, step S353 is executed. Step S353 determines whether the distinguishing pattern contacts the third region R3. If this determination is negative, step S354 is executed. In step S354, it is determined whether the distinguishing pattern contacts the vertices of other convex corners. If this determination is negative, step S352 is executed again to further enlarge the distinguishing pattern.

[0142] Figure 8 The illustrated segmentation pattern T40 does not touch the edge that forms the boundary of the third region R3, so the judgment result in step S353 is negative. The fourth region R45 does not have any other convex corners, so the judgment result in step S354 is negative. Step S352 is executed again, and the segmentation pattern is enlarged from the segmentation pattern T40.

[0143] Figure 9 This example illustrates a situation where steps S353, S354, and S352 are repeatedly executed to obtain a magnified segmentation pattern T41 starting at point P42. Between points P62 and P63, segmentation pattern T41 contacts edge L523 and, consequently, third region R3. Therefore, if step S353 is executed first after segmentation pattern T41 is obtained, the result of step S353 is affirmative.

[0144] If the result of step S353 is affirmative, step S354 and step S352 are not executed, and the segmentation pattern T41 is not further enlarged. This is because the segmentation pattern obtained by further enlarging the segmentation pattern T41 will exceed the fourth region R45, which is not preferable.

[0145] The case where the determination in step S353 is negative and the determination in step S354 is positive will be exemplified later.

[0146] If the result of step S353 or step S354 is positive, step S355 is executed. Step S355 determines whether the size of the obtained segmentation pattern is equal to or smaller than a predetermined value. The predetermined value corresponds to the minimum value of the number Mk.

[0147] The distinguishing pattern is a candidate for the movement source pixel group Gk. When step S35a is executed, the fourth region R4z has a single convex corner, and the distinguishing pattern is adopted as the movement source pixel group Gk. The smaller the specified value, the more movement source pixel groups Gk are obtained, which helps the third image F3 more closely reflect the features of the first image F1.

[0148] The determination in step S354 of whether to further execute step S352 helps to increase the size of the distinguishing pattern and thus move the pixel groups included in the source pixel group Gk, thereby making the third image F3 more reflect the features of the first image F1.

[0149] The following description uses the case where the size of the partitioning figure T41 is not less than the specified value as an example. In this case, the judgment in step S355 is negative, and step S356 is executed. Step S356 updates the starting point. For example, either point P412 or point P423 is used as the updated starting point. Point P412 is located at the position farthest from the original starting point, point P42, in the portion where the outline of the partitioning figure T41 contacts the edge L412. Point P423 is located at the position farthest from the original starting point, point P42, in the portion where the outline of the partitioning figure T41 contacts the edge L423.

[0150] Figure 10 The following shows how segmentation patterns T42, T43, and T44 are obtained. The segmentation patterns T42, T43, and T44 are obtained, for example, by the following process.

[0151] If the starting point is updated to point P412 in step S356, steps S352 and S353 are executed to enlarge the segmentation pattern, resulting in segmentation pattern T42. Segmentation pattern T42 is a square enlarged from point P412, and contacts edge L523 between points P62 and P63, thereby contacting third region R3.

[0152] Assume that the segmentation pattern T42 is obtained and the result of the determination in step S353 is affirmative, indicating that the size of the segmentation pattern T42 is not less than the predetermined value mentioned in step S355. In this case, step S356 is executed again and the starting point is updated to point P423.

[0153] Steps S352 and S353 are executed to enlarge the segmentation pattern starting at point P423 to obtain segmentation pattern T43. Segmentation pattern T43 is a square enlarged from point P423, and contacts edge L523 between points P62 and P63, and contacts third region R3.

[0154] Suppose that segmentation pattern T43 is obtained, and the result of the determination in step S353 is affirmative, indicating that the size of segmentation pattern T43 has not fallen below the predetermined value mentioned in step S355. In this case, segmentation pattern T44 is obtained based on segmentation pattern T43 in the same manner as the process of obtaining segmentation pattern T43 based on segmentation pattern T41. Segmentation pattern T44 contacts edge L523 between points P62 and P63, and thus contacts third region R3.

[0155] When the size of the segmentation pattern T44 is equal to or smaller than the predetermined value mentioned in step S355 , there is no need to obtain a segmentation pattern further, and the process returns to step S39 .

[0156] If the fourth region R45 is specified instead of the fourth region R46 in step S32, and steps S35a and S39 are executed, the determination in step S39 becomes negative, and step S32 is executed again. In this case, the fourth region R46 is specified by executing step S32. The determination in the subsequent step S33 is "plural," and step S39 is executed through steps S34, S35b, S36, S37, and S38.

[0157] Figures 11 to 16 This example illustrates the case where the fourth region R46 is specified in step S32. After executing steps S34, S35b, and S36, step S37 determines whether all salient corners in the fourth region R4z specified in step S32 have been selected. If the determination in step S37 is negative, step S34 is executed again to select and specify the remaining salient corners in the fourth region R4z. This can also be considered an update of the salient corners in the fourth region R4z.

[0158] Step S35b is a segmentation process. The specific content of step S35b is as follows: the case where the convex corner with point P44 as the vertex is specified in step S34, using Figure 7 and Figures 11 to 13 The specific content of step S35b is as follows: the case where the convex corner with point P43 as the vertex is specified in step S34, and the Figure 7 、 Figures 14 to 16 And explain.

[0159] Reference Figure 7 , in step S35b, step S351 is executed. Step S351 specifies the vertex of the convex corner as the starting point in the process for obtaining the distinguishing figure. Figure 11For example, point P44 is the vertex, which becomes the starting point. Figure 11 This illustrates a state where the square is enlarged with point P44 as the starting point in step S352 to obtain a segmentation figure V41.

[0160] For the distinguishing pattern V41, the judgment results in steps S353 and S354 are both negative, and step S352 is executed again.

[0161] Figure 12 This example illustrates a case where steps S353, S354, and S352 are repeatedly executed to obtain a distinguishing pattern V42 that is enlarged starting at point P44. Distinguishing pattern V42 contacts edge L541 between points P54 and P61, and thus contacts third region R3. Therefore, if step S353 is executed first after obtaining distinguishing pattern V42, the result of step S353 is affirmative.

[0162] Similar to step S35a, steps S355 and S356 are executed, and points P441 and P443 become new starting points. Point P441 is located at the position farthest from the original starting point, point P44, in the portion where the outline of the partitioning figure V42 contacts the side L441. Point P443 is located at the position farthest from the original starting point, point P44, in the portion where the outline of the partitioning figure V42 contacts the side L434.

[0163] Figure 13 This shows how segmentation patterns V43 and V46 are obtained. For example, segmentation patterns V43 and V46 are obtained through the following processing. If the starting point is updated to point P441 in step S356, steps S352, S353, and S354 are executed to enlarge the segmentation pattern, obtaining segmentation pattern V43. If the starting point is updated to point P443 in step S356, steps S352, S353, and S354 are executed to enlarge the segmentation pattern, obtaining segmentation pattern V46.

[0164] For distinguishing pattern V43, the judgment result in step S353 is affirmative, and step S355 is executed. In contrast, for distinguishing pattern V46, the judgment result in step S353 is negative, and the judgment in step S354 is affirmative, and step S355 is executed. This is because distinguishing pattern V46 does not contact point P54 but contacts point P43. Therefore, further enlarging distinguishing pattern V46 would result in a distinguishing pattern that exceeds fourth region R45, which is not desirable.

[0165] For the ratio Figure 13 The generation of the smaller distinction patterns V43 and V46 shown in FIG. 3 is omitted. After executing step S355 and obtaining a positive judgment result, the process returns to the first pixel group movement process (refer to FIG. Figure 6 ), execute step S36.

[0166] Step S36 stores the set of differentiation patterns obtained in step S35b (referred to as "differentiation pattern group") for each convex corner specified in step S34. The stored differentiation pattern group for each convex corner becomes a candidate for the movement source pixel group Gk.

[0167] In step S37, it is determined whether all salient corners have been selected. For example, if the salient corner with point P43 as the vertex is not specified in step S34, but the salient corner with point P44 as the vertex is specified, the salient corner with point P43 as the vertex is specified in step S34 via step S37.

[0168] Figure 14 In step S35b, after the salient corner with point P43 as the vertex is specified, steps S352, S353, and S354 are executed via step S351 to obtain a segmentation pattern V49. The segmentation pattern V49 is further enlarged with point P43 as the starting point.

[0169] Figure 15 This example illustrates the state where segmentation figure V49 is enlarged to obtain segmentation figure V47. Segmentation figure V47 contacts point P54, and the result of step S353 is affirmative, leading to step S356 via step S355. Point P443 is located at the point farthest from the original starting point, point P43, in the portion where the outline of segmentation figure V47 contacts edge L434. After point P443 is designated as the starting point in step S356, steps S352, S353, and S354 are executed.

[0170] Figure 16 This illustrates the case where a segmentation pattern V48 starting at point P443 is obtained. This segmentation pattern V48 does not contact the third region R3 but contacts vertex P44. When segmentation pattern V48 is obtained, the judgment result in step S353 is negative, while the judgment in step S354 is positive, and step S355 is executed. This is because segmentation pattern V48 does not contact the third region R3 but contacts point P44. Therefore, further enlarging segmentation pattern V48 would result in the segmentation pattern extending beyond fourth region R46, which is not desirable.

[0171] For the ratio Figure 16 The generation of the small distinction graphics V47 and V48 shown in FIG is omitted. After executing step S355 and obtaining a positive judgment result, the processing returns to the first pixel group movement processing (refer to Figure 6 ), execute step S36. In step S36, the segmentation graph group obtained based on the convex corner including the vertex P43 is stored.

[0172] exist Figures 12 to 16In the example of , since the salient corner including the vertex P44 and the salient corner including the vertex P43 are both selected as the starting point, the judgment result of step S37 is affirmative, and step S38 is executed.

[0173] In step S38 , the segmentation pattern group obtained based on the convex corner including the vertex P44 and the segmentation pattern group obtained based on the convex corner including the vertex P43 are compared, and one of them is selected as the movement source pixel group Gk.

[0174] The comparison is based on either or both of the size and number of the distinguishing graphics contained in the distinguishing graphic group ( Figure 6 (The use of "AND / OR" in the preceding sentence will omit the description.) A larger segmentation pattern size helps the third image F3 more clearly reflect the relationships between pixels in the first image F1. If the segmentation patterns are of the same size, a larger number of segmentation patterns of that size helps the third image F3 more clearly reflect the relationships between pixels in the first image F1. The following example illustrates a case in which, in step S38, a segmentation pattern group derived from a convex corner including vertex P44 is used as the movement source pixel group Gk.

[0175] By Figures 8 to 16 The processing described in step S39 is affirmative, and step S30 is executed. Step S30 executes the second affine transformation. This processing can be regarded as rotating and moving the distinguishing graphics obtained in steps S35a and S38 and arranging them in the fifth region R5. Based on this viewpoint, Figure 6 In the embodiment, step S30 is marked as "first configuration process".

[0176] Figure 17 1 shows the situation after executing step S30. Pixel groups D41, D42, D43, D44, E42, E43, and E46 are shown as the destination pixel group Hk. Pixel groups D41, D42, D43, and D44 are located in the fifth region R56, while pixel groups E42, E43, and E46 are located in the fifth region R55.

[0177] The pixel groups D41, D42, D43, and D44 are respectively corresponding to the segmentation patterns T41, T42, T43, and T44 (refer to Figure 10 ) is obtained by performing a second affine transformation on the pixel groups E42, E43, and E46, which are respectively obtained by performing a second affine transformation on the pixel groups E42, E43, and E46. Figure 13 ) is obtained by performing a second affine transformation on .

[0178] For example, through the first affine transformation, the fifth region R55 overlaps with the fourth region 46, and the fifth region R56 overlaps with the fourth region 45. The pixel groups D41, D42, D43, and D44 obtained by performing the second affine transformation on the segmentation patterns T41, T42, T43, and T44 contained in the fourth region R45 are contained in the fifth region R56. The pixel groups E42, E43, and E46 obtained by performing the second affine transformation on the segmentation patterns V42, V43, and V46 contained in the fourth region R46 are contained in the fifth region R55.

[0179] The situation where the source pixel group Gk is inscribed in the fourth region R4 and the destination pixel group Hk is inscribed in the fifth region R5 helps to increase the size of the segmentation pattern, thereby helping to better reflect the relationship between the pixels of the first image F1 in the third image F3.

[0180] In the above example, segmentation figure T41, which is enlarged with vertex P42 as its starting point and has a positive result in step S353, is in contact with side L523 and inscribed in fourth region R45. Pixel group D41, obtained by performing the second affine transformation on segmentation figure T41, is located at a position including vertex P53, in contact with side L423, and inscribed in fifth region R56.

[0181] Partitioning figure V42, which has been enlarged with vertex P44 as its starting point and for which the judgment result in step S353 is affirmative, is in contact with side L541 and inscribed in fourth region R46. Pixel group E42, obtained by performing a second affine transformation on partitioning figure V42, is located at a position including vertex P51, in contact with side L441, and inscribed in fifth region R55.

[0182] Figure 18 1 is a flowchart illustrating the content of the second pixel group movement process performed in step S4. Step S4 includes steps S41, S42, and S43, which are performed in sequence. Step S41 selects a pixel group located in the fourth region R4 and not used in the first arrangement process performed in step S30. For example, in the fourth region R45, this pixel group is divided into the following: T41, T42, T43, and T44 (see FIG. 1 ). Figure 10 ) and exists in the fourth region R45, the pixel group is divided into the fourth region R46 into the distinguishing patterns V42, V43, V46 (refer to Figure 13 ) and exists in the fourth region R46.

[0183] Step S42 divides the pixel group selected in step S41 into each pixel. In the first process, this is equivalent to the case where the value 1 is used as the number Mk. Step S43 performs a second affine transformation on each pixel divided in step S42, and arranges the pixel in the area of the fifth region R5 where no pixel group has been found. Figure 18 Step S43 is referred to as "second arrangement processing." This arrangement of pixels corresponds to obtaining pixels in a region of the fifth region R5 where pixels have not yet been arranged.

[0184] The pixel obtained in step S43 is, for example, a pixel arranged in the fifth region R56 outside the pixel group D41, D42, D43, and D44, and is arranged in the fifth region R56. For the fifth region R55, the pixel is a pixel arranged in the fifth region R55 outside the pixel group E42, E43, and E46 (see Figure 17 ).

[0185] After step S43 is completed, step S4 is also completed, and the process returns to step S5.

[0186] Since the second image F2 is not used as a training image, once the destination pixel group Hk is obtained, the source pixel group Gk can be deleted or retained. By following the second affine transformation, the source pixel group Gk can be moved to obtain the destination pixel group Hk, or the source pixel group Gk can be copied to obtain the destination pixel group Hk.

[0187] <4. Other Examples of First Pixel Group Movement Processing>

[0188] Figure 19 In step S3 (refer to Figure 5 ) is a flowchart of a first alternative example of the content of the first pixel group movement process performed in FIG. Step S3 of the first alternative example has Figure 6 Steps S31, S32, S33, and S39 shown in FIG, steps S30 and S34 which are limited as described later, and steps S35c and S360.

[0189] Figure 19 The contents of the processing performed in steps S31, S32, S33, and S39 and the order of the processing between them are the same as Figure 6 The same ones in . Figure 19 Step S35c in Figure 6 Similarly, step S35b in step S34 is executed after step S360 and before step S39. Step S360 is executed when the result of the judgment in step S39 is positive. After step S360, step S30 is executed. After step S30, Figure 6 Similarly, the process returns to step S4 (refer to Figure 5 ).

[0190] In the first alternative example, in step S34, the convex angle that is most likely to obtain the largest segmentation pattern among the convex angles of the fourth region R4 specified in step S32 is specified. By obtaining the segmentation pattern based on the specified convex angle, steps S35b, S36, S37, and S38 (see Figure 6 ).

[0191] exist Figure 19 In step S34, the distance from the vertex of the convex corner along the outer contour of the fourth region R4 is used as a reference for specifying a convex corner of the fourth region R4 specified in step S32. For a convex corner, the distance is the shortest distance from the vertex of the convex corner to the outer contour Q along the side intersecting the vertex of the convex corner and the outer contour Q.

[0192] The convex corner having the longest distance is designated in step S34. When the distances for different convex corners are equal, any convex corner may be designated in step S34.

[0193] At once Figures 8 to 17 In the example shown in FIG, in the fourth region R46, sides L441 and L434 sandwich a convex corner having vertex P44. Side L434 does not intersect outline Q, while side L441 intersects outline Q at point P61. The aforementioned distance for the convex corner having vertex P44 is the distance from vertex P44 to point P61 along side L441. This can also be considered the distance from vertex P44 along side L441 to the third region R3.

[0194] In the fourth region R46, sides L434 and L423 sandwich a convex corner with vertex P43. Side L434 does not intersect outline Q, while side L423 intersects outline Q at points P63 and P64. The distance from vertex P43 to point P64 along side L423 is shorter than the distance from vertex P43 to point P63 along side L423. The aforementioned distance for the convex corner with vertex P43 is the distance from vertex P43 to point P64 along side L423. This can also be considered the distance from vertex P43 along side L423 to the third region R3.

[0195] The distance from vertex P44 to point P61 along side L441 is longer than the distance from vertex P43 to point P64 along side L423. Therefore, in step S34, the salient corner having vertex P44 is specified.

[0196] Step S35c and steps S35a and S35b (see Figure 6 、 Figure 7) corresponds. Step S35c performs segmentation processing based on the single convex corner of the fourth region R4 specified in step S32. If the result of the judgment in step S33 is "single," this single convex corner is the single convex corner. If the result of the judgment in step S33 is "plural," this single convex corner is the convex corner specified in step S34 as described above. Figure 20 35 is a flowchart illustrating the content of the division process performed in step S35c.

[0197] Figure 21 is exemplified in Figure 19 Flowchart of the contents of the first configuration process executed in step S30 used in FIG. Step S30 includes steps S302, S303, S304, S305, and S309. Figure 22 This is a flowchart illustrating the contents of step S305.

[0198] <4-1. Example of Contents of Division Processing (Step S35c)>

[0199] Figures 23 to 29 It is a conceptual diagram explaining the division process performed in step S35c. Figures 23 to 29 The fourth region R4 and a portion of the third region R3 are shown. Figures 23 to 29 The fourth region R4 shown in FIG Figures 8 to 17 Each pixel in the fourth region R4 is drawn as a square. Figures 23 to 29 Description of step S35c.

[0200] Step S35c includes steps S351a, S354a, S352a, S353a, S353b, S355a, S36a, S356a, S357a, S358a, and S359a.

[0201] Step S351a sets the initial values for the start and end points of the segmentation pattern determined in step S35c. Specifically, if the result of step S33 is "single," the vertices of the singular convex corner are designated as the start and end points of the segmentation pattern. If the result of step S33 is "plural," the vertices of the convex corners designated as described above in step S34 are designated as the start and end points of the segmentation pattern.

[0202] The fourth region R4 has vertices P4a and P4b. Figures 8 to 17 The vertices P44 and P43 shown in the example correspond to each other.

[0203] Since the fourth region R4 has a plurality of convex corners with different vertices, step S34 is executed after step S33 is executed in step S3. The fourth region R4 contacts the third region R3 at points P4c and P4d, and the outer contour Q (not shown) intersects the edge of the fourth region R4 at points P4c and P4d. Points P4c and P4d intersect with Figures 8 to 17 The vertices P61 and P64 shown in the example correspond to each other.

[0204] The distance between vertices P4a and P4c (in Figures 23 to 29 10 pixels in the figure) is greater than the distance between vertices P4b and P4d (in Figures 23 to 29 Therefore, the vertex P4a is specified in step S34, and the vertex P4a is specified as the starting point P4as and the end point P4ae in step S351a.

[0205] After executing step S351a, execute step S354a. In step S354a, determine whether the end point can move one unit farther from the starting point. The movement is limited to the area where the pixel exists. Figure 7 Corresponding to step S354 in Figures 23 to 29 In , one unit is equivalent to one pixel in each of the two directions where the pixels are adjacent, and the moving direction is the diagonal direction of the square in which the pixels appear. Figure 23 In the example shown in , since the end point P4ae moves from the starting point P4as to the lower right direction in the figure, the end point P4ae is also located in the fourth region R4, so the judgment result of step S354a is affirmative.

[0206] If the result of the determination in step S354a is positive, the process executed in step S352a moves the end point away from the starting point by one unit. If the result of the determination in step S354a is negative, step S355a is executed.

[0207] Figure 24 Indicates from Figure 23 The state after step S352a is executed. The shaded square in the figure represents a partition pattern defined by the starting point P4as and the end point P4ae.

[0208] The segmentation pattern connects and amplifies the pixels in the fourth region R4. The amplification starts from a position away from the third region R3. The segmentation pattern can be set as the moving source pixel group Gk.

[0209] After executing step S352a, execute step S353a. In step S353a, determine whether the segmentation pattern defined by the starting point and the end point includes at least one pixel in the third region R3 (in Figure 20If the result of this judgment is negative, step S354a is executed again. Figure 24 In the state shown in the example, the judgment result of step S353a is negative, and the end point P4ae moves by executing steps S354a and S352a again. The size of the divided graphics is expanded to 4 pixels by this movement (refer to Figure 25 ).

[0210] Figure 26 This indicates that the end point P4ae reaches the boundary between the third region R3 and the fourth region R4 after the steps S354a and S352a are repeatedly executed. After this state, the end point P4ae moves to the third region R3 by executing steps S353a, S354a, and S352a. Figure 27 The status shown.

[0211] exist Figure 27 In the illustrated state, the partitioning pattern defined by starting point P4as and end point P4ae includes a single pixel from the third region R3. Therefore, the judgment result in step S353a is affirmative, and step S353b is executed. The partitioning pattern should consist solely of pixels from the fourth region R4. Therefore, it is necessary to perform a process that cancels the processing performed in the previous step S352a, which yielded a positive judgment result in step S353a. This cancellation process is performed in step S353b.

[0212] Specifically, the processing of step S353b causes the end point to move closer to the starting point by one unit. Figure 27 The state shown in FIG353b is executed to obtain Figure 26 The status shown.

[0213] After executing step S353b, execute step S355a. In step S355a, Figure 7 Similarly to step S355 illustrated in FIG, it is determined whether the size of the segmentation pattern obtained in step S353b is equal to or smaller than a predetermined value.

[0214] If the result of the judgment in step S355a is negative, the distinguishing pattern provided for the judgment should be used for the first configuration process (refer to Figure 19 In step S30), the distinguishing pattern is stored in a distinguishing pattern list (not shown). If the determination result of step S355a is positive, since the distinguishing pattern provided for the determination is not used in the first configuration process, step S36a is not executed.

[0215] After a positive determination result is obtained in step S355a, or after a negative determination result in step S355a and step S36a is executed, step S356a is executed. The execution of step S356a sets a false starting point. The false starting point is a candidate for a starting point that can serve as the starting point for segmenting a pattern under specified conditions. The false starting point is located farther from the starting point toward the end point in each of the two adjacent pixel components of the distance from the starting point to the end point.

[0216] Figure 28 Indicates the state after executing step S356a. Figure 28 In step S36a, the segmentation pattern V4a stored in the segmentation pattern list is represented by a cross-hatched pixel group. For example, the segmentation pattern list is represented as Lv={V4a}.

[0217] exist Figure 28 2 also illustrates pseudo starting points P4a1s and P4a2s. These pseudo starting points P4a1s and P4a2s are determined based on starting point P4as and end point P4ae. Specifically, pseudo starting points P4a1s and P4a2s are set at positions that are separated from starting point P4as by a distance (here, five pixels) from end point P4ae in each of two directions adjacent to the pixel.

[0218] The false starting point is adopted as the starting point if it satisfies any one of the following conditions (α) and (β) and both of the conditions (γ): (α) the false starting point is located in the fourth region R4, (β) the false starting point is located outside the outer contour of the fourth region R4 and outside the outer contour of the third region R3, and (γ) the false starting point is different from the previous starting point. Figure 28 As far as is concerned, both the false starting points P4a1s and P4a2s satisfy the conditions (β) and (γ).

[0219] After executing step S356a, step S357a is executed. By executing step S357a, the pseudo starting points that satisfy the specified conditions (corresponding to the aforementioned conditions (α), (β), and (γ)) are sequentially registered as starting points in a starting point list (not shown). In the above example, both pseudo starting points P4a1s and P4a2s are registered as starting points in the starting point list. For example, the starting point list is represented as Ls = {P4a1s, P4a2s}.

[0220] After executing step S357a, step S358a is executed. In step S358a, it is determined whether a starting point is registered in the starting point list. If this determination is positive, step S359a is executed to reset the starting point and end point. The starting point at the top of the starting point list is set as the new starting point and end point. Executing step S359a resets the starting point and end point, which can also be considered an update.

[0221] After executing step S359a, step S354a is executed. When executing step S359a, the starting point used for the reset of the starting point and the end point is deleted from the starting point list.

[0222] In the above example, when step S358a is executed, the starting point list is Ls = {P4a1s, P4a2s}. When step S359a is executed, the starting point P4a1s at the top of the starting point list is reset as the starting point and end point, and the starting point list is updated to Ls = {P4a2s}. In this way, starting points not used for setting the segmented pattern are retained in the starting point list.

[0223] Based on the starting point P4a1s, steps S354a, S352a, S353a, and S353b are executed to obtain the segmentation pattern V4a1 (refer to Figure 29 The segmentation pattern V4a1 has a size of nine pixels. For example, the predetermined value in step S355a is set to four pixels, and step S36a is executed. By executing step S36a, the segmentation pattern list is represented as Lv = {V4a, V4a1}.

[0224] Furthermore, by executing step S356a, the pseudo starting points P4a3s and P4a4s are set. The pseudo starting point P4a3s satisfies conditions (β) and (γ). The pseudo starting point P4a4s satisfies conditions (α) and (γ). By executing step S357a, the pseudo starting points P4a3s and P4a4s are then added to the starting point list. Thus, the starting point list becomes Ls = {P4a2s, P4a3s, P4a4s}.

[0225] The result of the judgment in the subsequent step S358a is affirmative, and step S359a is executed. In step S359a, the starting point P4a2s registered at the beginning of the starting point list is adopted as the new starting point and end point, and the starting point list becomes Ls = {P4a3s, P4a4s}.

[0226] Based on the starting point P4a2s, steps S354a, S352a, and S353a are executed to obtain the distinguishing pattern V4a2 (refer to Figure 29 ). For the end point P4a2e obtained before the positive judgment result is obtained in step S353a, the judgment result of step S354a is negative. This situation is equivalent to Figure 7 The case where a positive judgment result is obtained in step S354 is illustrated.

[0227] After obtaining the segmentation pattern V4a2, step S36a is executed, and the segmentation pattern list is represented as Lv = {V4a, V4a1, V4a2}. Furthermore, step S356a is executed to set the temporary starting points P4a5s and P4a6s (the temporary starting point P4a6s coincides with the vertex P4b). The temporary starting point P4a5s satisfies conditions (α) and (γ), and the temporary starting point P4a6s satisfies conditions (β) and (γ). By executing step S357a, the starting point list becomes Ls = {P4a3s, P4a4s, P4a5s, P4a6s}.

[0228] Afterwards, step S359a is executed in the order logged in the starting point list, but in the above example, the judgment result in step S355a of the distinguishing figure obtained based on starting points P4a3s and P4a4s is affirmative, and the false starting point obtained thereafter does not meet any of the conditions (α) and (β).

[0229] Since no distinguishing pattern is actually obtained based on the starting point P4a6s, its size is processed as 0 for convenience. Therefore, the judgment result of step S355a is affirmative, and since the false starting point set in step S356a coincides with the starting point P4a6s, the condition (γ) is not satisfied.

[0230] The determination result in step S355a of the segmentation pattern obtained based on the starting point P4a5s is affirmative, and the subsequent obtained false starting point also does not satisfy the prescribed condition in step S357a.

[0231] In this way, Figure 29 In the example of , even if the starting point registered in the starting point list does not exist and the judgment result of step S358a is negative, the classification graphic list still maintains Lv = {V4a, V4a1, V4a2}. The judgment result of step S358a is negative, and the processing returns to step S39 (refer to Figure 19 ).

[0232] Figure 29 The distinguishing graphics V4a, V4a1, and V4a2 shown in the example are respectively Figure 13 These correspond to the segmentation graphics V42, V43, and V46 illustrated in FIG. Figure 16 The distinguishing graphics V47 and V48 based on the convex corner including the vertex P43 are shown in FIG. Figure 19 It is not obtained in the first pixel group movement processing illustrated in .

[0233] Reference Figure 19 After the determination in step S39, steps S32, S33, and S35c (and step S34 if the designated fourth region R4 has multiple convex corners) are executed until all fourth regions R4 are designated. After all fourth regions R4 are designated through the determination in step S39, step S360 is executed.

[0234] The process executed in step S360 collects and sorts the segmentation patterns registered in the segmentation pattern list obtained for each of the designated fourth regions R4 based on their sizes. Figures 8 to 17 For example, the segmentation pattern list is Lv={V42, V46, V43, T41, T43, T42, T44}.

[0235] After executing step S360, execute Figure 21 Step S30 is shown. Step S30 performs the first configuration process. In step S30, steps S302, S303, S305, and S309 are performed in sequence.

[0236] In step S302, the fifth region R5 is designated. Figure 19 Steps S33 and S34 described above are executed in the same manner as steps S303 and S304.

[0237] Among the convex corners of the fifth region R5 specified in step S302, the convex corner that results in the largest partitioning region is determined. Within the partitioning region, the partitioning patterns registered in the partitioning pattern list undergo a second affine transformation and are arranged as the destination pixel group Hk. The partitioning region is formed by connecting and enlarging the pixels within the fifth region R5. This enlargement begins at a location farther from the third region R3. Determining the partitioning region based on the convex corners determined in this manner facilitates the placement of larger partitioning patterns within the partitioning region.

[0238] In step S304, the distance along the outer contour of the fifth region R5 from the vertex of the convex corner is used as a reference for specifying a convex corner of the fifth region R5 specified in step S302. For a convex corner, this distance is the shortest distance from the vertex to the second region R2 along a side intersecting the vertex of the convex corner with the second region R2.

[0239] In step S304, the salient angle with the longest distance is specified. When the distances for different salient angles are equal, any salient angle can be specified in step S304.

[0240] At once Figures 8 to 17 In the example shown in FIG, in the fifth region R55, sides L541 and L512 sandwich a salient corner having vertex P51. Side L512 does not intersect the second region R2, while side L541 intersects the second region R2 at point P61. The aforementioned distance for the salient corner having vertex P51 is the distance from vertex P51 to point P61 along side L541. This can also be considered the distance from vertex P51 along side L541 to the third region R3.

[0241] In the fifth region R55, sides L512 and L523 sandwich a salient corner with vertex P52. Side L512 does not intersect the second region R2, while side L523 intersects the second region between points P62 and P63. The distance from vertex P52 to point P62 along side L523 is shorter than the distance from vertex P52 to any point where side L523 intersects the second region R2. The aforementioned distance for the salient corner with vertex P52 is the distance from vertex P52 to point P62 along side L523. This can also be considered the distance from vertex P52 along side L523 to the third region R3.

[0242] The distance from vertex P51 to point P61 along edge L541 is longer than the distance from vertex P52 to point P62 along edge L523. Therefore, step S304 specifies the salient corner having vertex P51.

[0243] Step S305 performs segmentation based on the single convex corner of the fifth region R5 specified in step S302. This single convex corner is the single convex corner if the result of step S303 is "single," or the convex corner specified in step S304 as described above if the result of step S303 is "plural."

[0244] After executing step S305, it is determined in step S309 whether all fifth regions R5 have been designated. If this determination is negative, the process returns to step S302. If there is no fifth region R5 that has not been segmented in step S305, the determination result in step S309 is positive, and the process returns to step S4 (refer to step S4). Figure 5 ).

[0245] At once Figures 8 to 17 For example, the fifth region R55 and R56 are designated in steps S302 and S309, respectively. For example, the fifth region R55 is designated before the fifth region R56 is designated.

[0246] <4-2. Example of Contents of Division Processing (Step S305)>

[0247] Figures 30 to 35 This is to explain the step S305 (refer to Figure 22 ) is a conceptual diagram of the segmentation processing performed in . Figures 30 to 35 The fifth region R5 and a portion of the third region R3 are shown. Figures 30 to 35 The fifth region R5 shown in FIG. Figures 8 to 17 Each pixel in the fifth region R5 is drawn as a square. Figures 30 to 35 Description of step S305.

[0248] Step S305 includes steps S351b, S354b, S352b, S353c, S353d, S355b, S36b, S356b, S356c, S357b, S358b, and S359b.

[0249] Step S351b sets the initial values for the start and end points of the partitioning region determined in step S305. Specifically, if the result of step S303 is "single," the vertices of the singular salient corner are designated as the start and end points of the partitioning region. If the result of step S303 is "plural," the vertices of the salient corner determined as described above in step S304 are designated as the start and end points of the partitioning region.

[0250] The fifth region R5 has vertices P5a and P5b. Figures 8 to 17 The vertices P51 and P52 shown in the example correspond to each other.

[0251] Since the fifth region R5 has a plurality of convex corners with different vertices, step S304 is executed after step S303 is executed in step S30. The fifth region R5 contacts the third region R3 at points P5c and P5d, and the edge of the second region R2 (not shown) intersects the edge of the fifth region R5 at points P5c and P5d. Points P5c and P5d are respectively Figures 8 to 17 The vertices P61 and P62 shown in the example correspond to each other.

[0252] The distance between vertices P5a and P5c (in Figures 30 to 35 10 pixels in the figure) is greater than the distance between vertices P5b and P5d (in Figures 30 to 35 Therefore, the vertex P5a is specified in step S304, and the vertex P5a is specified as the starting point P5as and the end point P5ae in step S351b.

[0253] After executing step S351b, execute step S354b. In step S354b, determine whether the end point can move one unit farther from the starting point. The movement is limited to the area where the pixel exists. Figures 30 to 35 In , one unit is equivalent to one pixel in each of the two directions where the pixels are adjacent, and the moving direction is the diagonal direction of the square in which the pixels appear. Figure 30 In the example shown in , since the end point P5ae moves from the starting point P5as to the lower left in the figure, the end point P5ae is also located in the fifth region R5, so the judgment result of step S354b is affirmative.

[0254] If the result of the determination in step S354b is positive, the process executed in step S352b moves the end point away from the starting point by one unit. If the result of the determination in step S354b is negative, step S355b is executed.

[0255] Figure 31 Indicates from Figure 30 The state after step S352b is executed. The shaded square in the figure represents the partition area defined by the starting point P5as and the end point P5ae.

[0256] After executing step S352b, execute step S353c. In step S353c, determine whether the partition area defined by the starting point and the end point includes at least one pixel in the third area R3 (in Figure 22 If the result of this judgment is negative, step S354b is executed again. Figure 31 In the state shown in the example, the determination result of step S353c is negative, and the end point P5ae moves by executing steps S354b and S352b again. This movement expands the partition area to a size of 4 pixels.

[0257] Figure 32 Indicates a state where steps S354b and S352b are repeatedly executed and the end point P5ae reaches the boundary between the third region R3 and the fifth region R5. After this state, by executing steps S353c, S354b, and S352b, the end point P5ae moves to the third region R3, and the result is Figure 33 The status shown.

[0258] exist Figure 33 In the illustrated state, the partitioning pattern defined by starting point P5as and end point P5ae includes a pixel in the third region R3. Therefore, the judgment result in step S353c is affirmative, and step S353d is executed. The partitioning region is the area where the partitioning pattern is arranged and should consist only of pixels within the fifth region R5. Therefore, it is necessary to perform a process to offset the processing performed in the previous step S352b, which yielded a positive judgment result in step S353c. This offsetting process is performed in step S353d.

[0259] Specifically, the processing of step S353d causes the end point to move closer to the starting point by one unit. Figure 33 The state shown in FIG353a is executed in step S353d, and the Figure 32 The status shown. Figure 34 , the distinction area V5a obtained by executing step S353d is represented by a group of pixels to which cross hatching is applied.

[0260] After executing step S353d, step S355b is executed. In step S355b, it is determined whether a segmentation pattern smaller than the size of the segmentation area exists. The segmentation area referred to in step S355b is the segmentation area obtained in step S353d executed immediately before. The segmentation pattern referred to in step S355b is the segmentation pattern registered in the segmentation pattern list immediately before executing step S353d. In the above example, step S355b determines whether a segmentation pattern smaller than the size of segmentation area V5a exists in the segmentation pattern list Lv = {V4a, V4a1, V4a2}.

[0261] If the result of step S355b is affirmative, step S36b is executed. In step S36b, a second affine transformation is applied to the segmentation region used for the determination to create segmentation patterns. In this case, the largest segmentation pattern among the segmentation patterns that resulted in a positive result in step S355b is placed in the segmentation region. In this arrangement, for example, the vertex of the segmentation pattern is placed at the starting point that defines the segmentation region.

[0262] In the above example, the partition area V5a has a size of 25 pixels. The partition patterns V4a, V4a1, and V4a2 stored in the partition pattern list are all smaller than 25 pixels, and the result of the determination in step S355b is affirmative.

[0263] The largest segmentation figure V4a among the segmentation figures V4a, V4a1 and V4a2 is placed in the segmentation area V5a. Figure 19 ), register the partition patterns in descending order. For example, the partition pattern configured in the partition area is selected from the first one in the partition pattern list.

[0264] In step S36b, the segmentation pattern placed in the segmentation area is deleted from the segmentation pattern list. In the above example, segmentation pattern V4a is placed in segmentation area V5a using the second affine transformation, and the segmentation pattern list is updated to Lv={V4a1, V4a2}.

[0265] After executing step S36b, execute step S356c. If the judgment result of step S355b is negative, execute step S356b. By executing steps S356b and S356c, a false starting point is set. The false starting point is a candidate starting point that can serve as the starting point for the segmentation area under specified conditions. The setting of the false starting point is different in steps S356b and S356c.

[0266] The provisional starting point set in step S356b is located at a position away from the starting point toward the end point in each of the components in two directions in which pixels of the distance from the starting point to the end point are adjacent.

[0267] Figure 34 Indicates the state after executing step S356b. Figure 34 , the pseudo starting points P5a1s and P5a2s are illustrated. These pseudo starting points P5a1s and P5a2s are determined based on the starting point P5as and the end point P5ae. Specifically, the pseudo starting points P5a1s and P5a2s are set at positions that are separated from the starting point P5as by a distance (here, five pixels) from the end point P5ae in each of two directions adjacent to the pixel.

[0268] The false starting point is adopted as the starting point if it satisfies any one of the following conditions (δ) and (ε) and both of the conditions (ζ): (δ) the false starting point is located in the fifth region R5, (ε) the false starting point is located outside the outer contour of the fifth region R5 and outside the outer contour of the third region R3, and (ζ) the false starting point is different from the previous starting point. Figure 34 As far as is concerned, both the false starting points P5a1s and P5a2s satisfy the conditions (ε) and (ζ).

[0269] After executing step S356b, step S357b is executed. Through the execution of step S357b, the pseudo starting points that satisfy the specified conditions (corresponding to the aforementioned conditions (δ), (ε), and (ζ)) are sequentially registered as starting points in a starting point list (not shown). In the above example, the pseudo starting points P5a1s and P5a2s are both registered as starting points in the starting point list. For example, the starting point list is represented as Ls = {P5a1s, P5a2s}.

[0270] After executing step S357b, step S358b is executed. In step S358b, it is determined whether a starting point is registered in the starting point list. If so, step S359b is executed to reset the starting point and end point. The starting point at the top of the starting point list is set as the new starting point and end point. Execution of step S359b resets the starting point and end point, or rather, updates them.

[0271] After executing step S359b, step S354b is executed. When executing step S359b, the starting point used for the reset of the starting point and the end point is deleted from the starting point list.

[0272] In the above example, when step S358b is executed, the starting point list is Ls = {P5a1s, P5a2s}. When step S359b is executed, the starting point P5a1s at the top of the starting point list is reset as the starting point and end point, and the starting point list is updated to Ls = {P5a2s}. In this way, starting points not used for zone division are retained in the starting point list.

[0273] The temporary starting point set in step S356c is located away from the starting point in each of the two adjacent directions of the pixels of the arranged segmentation pattern. In the above example, the size of segmentation area V5a matches the size of segmentation pattern V4a. In step S356c, temporary starting points P5a1s and P5a2s are set in the same manner as in step S356b.

[0274] For example, assume that step S353d results in a partition region V5a, and the partition pattern list is Lv = {V4a1, V4a2}. In this case, step S36b is executed to place partition pattern V4a1 within partition region V5a using the second affine transformation. The size of partition pattern V4a1 is four pixels in either direction of pixel adjacency. In this case, assume that the vertex of partition pattern V4a1 is aligned with starting point P5as, and partition pattern V4a1 is placed within partition region V5a using the second affine transformation. In this scenario, the virtual starting point P5a1s is positioned one pixel closer to starting point P5as than the virtual starting point P5a1s set in step S356b, and the virtual starting point P5a2s is positioned one pixel closer to starting point P5as than the virtual starting point P5a2s set in step S356b.

[0275] Furthermore, in this case, consider a scenario where the vertex of the segmentation pattern V4a1 is aligned with the end point P5ae, and segmentation pattern V4a1 is placed in segmentation region V5a through a second affine transformation. In this scenario, the pseudo starting points P5a1s and P5a2s coincide with the pseudo starting points P5a1s and P5a2s set in S356b, respectively. In this case, a pixel group with a width of one pixel is left in an L-shaped arrangement between the segmentation pattern V4a1 and the start point P5as in segmentation region V5a. It is difficult to place a segmentation pattern in such a position.

[0276] The vertex of the partitioning figure arranged in the partitioning area coincides with the starting point defining the partitioning area, which facilitates the arrangement of a larger partitioning figure in the partitioning area.

[0277] In steps S357b, S358b, and S359b executed after step S356c, the same processing as steps S357b, S358b, and S359b executed after step S356b is performed.

[0278] After resetting the starting point P5a1s as the starting point and the end point and executing step S354b, execute steps S352b, S353c, and S353d to obtain the end point P5a1e and the partition area V5a1 (see Figure 35Similarly, after resetting the starting point P5a2s to the starting point and the end point and executing step S354b, execute steps S352b, S353c, and S353d to obtain the end point P5a2e and the partition area V5a2 (refer to Figure 35 ).

[0279] Each time step S357b is executed, a starting point is added to the starting point list, and each time step S359b is executed, a starting point is deleted from the starting point list. As with the segmentation process executed in step S35c, the starting point registered in the starting point list is updated, and soon the judgment result of step S358b is negative. If the judgment result of step S358b is negative, the process returns to step S309 (refer to Figure 21 ), and the process further returns to step S4 (refer to Figure 5 ), and perform the second pixel group movement processing.

[0280] <4-3. Using Mirror Image Transformation>

[0281] When the fourth region R4 and the fifth region R5 overlap and have a mirror image relationship, the mirror image transformation is used in the second affine transformation. The specific processing is described below using the first pixel group movement processing performed in step S3. Figures 8 to 17 For example, let's assume that the fourth region R45 and the fifth region R56 have a mirror image relationship, and the fourth region R46 and the fifth region R55 have a mirror image relationship. These mirror images are also considered to be line symmetric, and the axis of this line symmetry corresponds to the straight line connecting points P61 and P63.

[0282] Figure 36 is a flowchart showing another example of the content of the first pixel group movement process. Step S3 has Figure 6 Steps S31, S32, S33, S39, Figure 19 Steps S34, S35d, and S301 described in .

[0283] Figure 36 The contents of the processing performed in steps S31, S32, S33, and S39 and the order of the processing between them are the same as Figure 6 The same ones in . Figure 36 The content of the processing executed in step S34 and its relationship with the processing executed in step S33 and Figure 19 The same ones in .

[0284] Step S301 is performed after step S31 is performed to identify the third region R3 , the fourth region R4 , and the fifth region R5 , and before the fourth region R4 is specified in step S32 .

[0285] The process executed in step S301 generates a coordinate transformation matrix from the fourth region R4 to the fifth region R5 . Mirror transformation is used in this coordinate transformation matrix.

[0286] After executing step S301, steps S32, S33, and S34 are executed as described above. Step S35d performs segmentation processing based on the single convex corner of the fourth region R4 specified in step S32. If the result of step S33 is "single," this single convex corner is the single convex corner; if the result of step S33 is "plural," this single convex corner is the convex corner specified in step S34 as described above.

[0287] Figure 37 This is a flowchart illustrating the contents of the segmentation process executed in step S35d. This segmentation process has the following characteristics: Figure 20 ), replace step S36a with the structure of step S302.

[0288] The processing executed in step S302 uses the processing performed in step S301 (see Figure 36 ) generates a coordinate transformation matrix to move the segmented graphic to the fifth region. Copying may be performed instead of moving.

[0289] At once Figures 8 to 17 In the example shown, pixel groups D41, D42, D43, D44, E42, E43, and E46 are obtained by movement generated by mirror transformation of the distinguishing figures T41, T42, T43, T44, V42, V43, and V46, respectively.

[0290] However, the arrangement relationship between pixels in pixel group D41 is also a mirror image of the arrangement relationship between pixels in segmentation pattern T41. The same applies to segmentation patterns T42, T43, T44, V42, V43, and V46.

[0291] <5. General Explanation>

[0292] Taking the above-mentioned pixel groups D41, D42, D43, D44, E42, E43, and E46 as an example, the process until the movement destination pixel group Hk is obtained is re-understood as follows.

[0293] The transformation step is a step of performing a first affine transformation on the first image F1 occupying the first region R1 to obtain the second image F2 occupying the second region R2.

[0294] The movement process includes a first step and a second step. In the first step, the segmentation pattern T41 is set to include a group of pixels Ma (greater than 1) within the fourth region R45, which is the source pixel group Ga. In the second step, a second affine transformation is performed on the segmentation pattern T41, determining the pixel group D41 within the fifth region R56 as the destination pixel group Ha.

[0295] In the first step, the segmentation pattern V42 is set as the pixel group included in the fourth region R46 and connected to a number Mb greater than 1, i.e., the movement source pixel group Gb. In the second step, a second affine transformation is performed on the segmentation pattern V42 to determine the pixel group E42 arranged in the fifth region R55 as the movement destination pixel group Hb.

[0296] By executing the first step and the second step, the objects marked in the third region R3 ( Figures 1 to 4 A training image of a rotating chuck in the example can be easily obtained by extracting features other than just the brightness distribution of the image other than the object.

[0297] The movement process includes a third step and a fourth step. In the fourth region R45, the third step sets the partitioning patterns T42, T43, and T44 to connect to a pixel group Mc, which has a number less than Ma, namely, the movement source pixel group Gc. When the number Mc is equal to the number Ma, the third step can be understood as a repetition of the first step. When the number Mc is smaller than the number Ma, the third step can be understood as a repetition of the first step after the number Mk has been updated. In the above example, Ma>Mc.

[0298] In the fourth region R46, the third step sets the partitioning patterns V43 and V46 to connect to a pixel group Md, which is smaller than the number Mb, i.e., the movement source pixel group Gd. If the number Md is equal to the number Mb, the third step can be considered a repetition of the first step. If the number Md is smaller than the number Mb, the third step can be considered a repetition of the first step after the number Mk has been updated. In the above example, Mb > Md.

[0299] The fourth step applies a third affine transformation to the source pixel group obtained in the third step to obtain the destination pixel group. If the third step is understood as a repetition of the first step, the fourth step is understood as a repetition of the second step. Like the second affine transformation, the third affine transformation is an affine transformation and is either a rotation, a parallel translation, or both, or a mirror transformation.

[0300] In the fifth region R56, the fourth step is to perform the division pattern T43 (see Figure 10) Perform the third affine transformation and find the pixel group D43 as the destination pixel group Hk. Reduce the number Mc and update and repeat the third and fourth steps to set the segmentation patterns T42 and T44 to find the pixel groups D42 and D44.

[0301] By executing the third and fourth steps, a training image is obtained that further reflects features of images other than the labeled object.

[0302] The process of setting partitioning patterns T41 and T43 and obtaining pixel groups D41 and D43 can be considered a repetition of the first and second steps, which reduce and update the number Ma. The process of setting partitioning patterns T42 and T44 and obtaining pixel groups D42 and D44 can be considered a repetition of the third and fourth steps, which further reduce and update the number Mc. For example, from this perspective, the first and second steps are performed multiple times before the third step.

[0303] Regarding the fifth region R55, the fourth step is to perform the division pattern V46 (see Figure 13 ) Perform a third affine transformation and determine pixel group E46 as the destination pixel group Hk. Reduce and update the number Md, and repeat the third and fourth steps to set a segmentation pattern V43 to determine pixel group E43.

[0304] The steps of setting partitioning patterns V42 and V46 and obtaining pixel groups E42 and E46 can be considered a repetition of the first and second steps, which reduce and update the number Mb. The steps of setting partitioning pattern V43 and obtaining pixel group E43 can be considered the third and fourth steps, which further reduce the number Md. For example, from this perspective, the first and second steps are performed multiple times before the third step.

[0305] By repeating the first and second steps, or repeating the third and fourth steps, a training image that further reflects the features of images other than the labeled object is obtained.

[0306] The second configuration process (see Figure 18 Step S43 corresponds to the case where the number of pixel groups set in the third step is one. The third affine transformation performed in the second placement process facilitates reflecting information about the brightness distribution of image features other than the labeled object into the training image. For example, the third and fourth steps are repeated until all pixels in the second image F2 are moved or copied to the first region R1.

[0307] In the first alternative example, segmentation patterns V4a, V4a1, and V4a2 connect and amplify pixels within the fourth region R4, starting from a position away from the third region R3. This segmentation pattern is set as the movement source pixel group Gk. Segmentation regions V5a, V5a1, and V5a2 connect and amplify pixels within the fifth region R5, starting from a position away from the third region R3. Within these segmentation regions, the movement destination pixel group Hk is arranged.

[0308] In the second alternative example, the fourth region R4 and the fifth region R5 are mirror images. In this case, the second affine transformation is, for example, a mirror image transformation. Similarly, when the fourth region R4 and the fifth region R5 are mirror images, a mirror image transformation can be used in the third affine transformation.

[0309] <6. Generation and Utilization of Training Images>

[0310] Figure 38 This is a block diagram illustrating the generation and use of training images. A camera 600 captures a structure 90 and obtains a captured image F0. The structure 90 includes, for example, a rotary chuck 91. The captured image F0 is output from the camera 600. For example, a known camera is used as the camera 600.

[0311] The photographic image F0 is input to the image generator 610. The image generator 610 generates at least one training image Ji (i is an integer greater than or equal to 1). The training image Ji is output from the image generator 610.

[0312] The image generator 610 has a function schematically illustrated by an image input interface 611 as hardware. This function is equivalent to Figure 5 The setting of the first image is exemplified as step S1 in FIG.

[0313] Image input interface 611 selects and outputs a first image F1 from photographic image F0. For example, first image F1 is selected from photographic image F0 based on whether at least one rotating chuck 91 can be labeled as pixel group 100 in third region R3. This condition is set by predetermining at least one first affine transformation simultaneously with or before input of photographic image F0.

[0314] The image generator 610 has a function schematically illustrated by a first affine transformation processing unit 612 as hardware. This function corresponds to the transformation process, specifically, the function of performing a first affine transformation on the first image F1 to obtain a second image F2. This function corresponds to the first affine transformation processing unit 612. Figure 5 The first affine transformation processing unit 612 outputs the second image F2.

[0315] The image generator 610 has a function schematically illustrated by a region recognition unit 613 as hardware. Specifically, this function is the recognition of the third region R3, the fourth region R4, and the fifth region R5. Figure 6 This function is executed by using the first image F1 and the second image F2 (refer to Figure 2 、 Figures 8 to 17 ). The region identification unit 613 outputs the third region R3, the fourth region R4, and the fifth region R5.

[0316] The image generator 610 has a function schematically illustrated by a segmentation processing unit 614 as hardware. This function corresponds to either or both of the first and third steps, and corresponds to the Figure 6 The process after executing step S31 and before executing step S30. Figure 18 The fourth region R4 is input to the segmentation processing unit 614, which generates a movement source pixel group Gk. The movement source pixel group Gk is output from the segmentation processing unit 614.

[0317] The image generator 610 has a function schematically illustrated by the second affine transformation unit 615 as hardware. This function corresponds to either or both of the second and fourth steps, and corresponds to the second step. Figure 6 The first configuration process is illustrated as step S30 in FIG. This function may include Figure 18 The second affine transformation unit 615 performs the function of the second configuration process illustrated in step S43 of FIG. The execution of the function of the second affine transformation unit 615 is to use the fifth region R5 and the movement source pixel group Gk (refer to FIG. Figure 3 、 Figure 4 、 Figure 17 ). The second affine transformation unit 615 generates a movement destination pixel group Hk and outputs it.

[0318] The image generator 610 has a function schematically illustrated by a synthesizing unit 616 as hardware. This function is to synthesize the movement destination pixel group Hk with the third region R3 to generate a third image F3. This function is included in one or both of the second and fourth steps, or is included in one or both of them. Figure 6 The first configuration process illustrated as step S30 in the embodiment of the present invention, or accompanying the first configuration process. This function may also be included in the Figure 18 The second configuration process illustrated in step S43 of , or accompanying the second configuration process.

[0319] The image generator 610 has a function schematically illustrated by a storage unit 617 as hardware. Specifically, this function is the storage of the third image F3. Figure 5The third image F3 stored in the storage unit 617 is read out from the storage unit 617 as the training image Ji.

[0320] The assisting device 620 has a function of adding an auxiliary image to the photographic image F0. For example, the assisting device 620 generates a display image F9, which is an image obtained by adding a marker identifying the pixel group 100 to the photographic image F0. The display image F9 is output from the assisting device 620.

[0321] The auxiliary device 620 has a function schematically illustrated by a learning unit 621 as hardware. This function is a function of generating a learned model W using the training image Ji. As described above, in addition to the training image Ji, the first image F1 can be used as a training image as described above. Figure 38 The dashed arrow in FIG. 3 illustrates a method of using the first image F1 as a training image.

[0322] The learned model W is output from the learning unit 621. The learned model W includes a program and parameters having a function of selecting the pixel group 100 from image data.

[0323] Support device 620 includes a function schematically illustrated by an element identification unit 622 as hardware. This function uses the learned model W to select pixel groups 100 from photographic image F0 and assigns a label identifying these pixels within photographic image F0. This label is added to photographic image F0 to generate display image F9. Display image F9 is output from element identification unit 622 and, in turn, from support device 620.

[0324] Both the image generator 610 and the auxiliary device 620 can be implemented using, for example, a processor and a memory. A processor can be composed of, for example, one or more central processing units (CPUs). The memory can be composed of a volatile storage medium such as, for example, RAM (Random Access Memory), or a non-volatile storage medium such as a hard disk drive (HDD) or a solid state drive (SSD). The functions of the storage unit 617 are, for example, performed by this memory.

[0325] This memory stores, for example, programs and various information. The processor implements the various functions and processes described above by, for example, reading and executing programs stored in the memory. For example, the RAM is used as a workspace to temporarily store information generated or acquired. At least some of the functions schematically illustrated as hardware by the image generator 610 and the auxiliary device 620 can be implemented using hardware such as dedicated circuits.

[0326] The display image F9 is input to the display 630. The display 630 displays the display image F9. Figure 38 The identification 631 of the pixel group 100 is determined as an example.

[0327] The function of the image generator 610 can also be described as a method for generating training images Ji, which serve as training data for machine learning used to identify an object (in the above example, a chuck pin) in the photographic image F0. This machine learning uses the training images Ji (or further uses the first image F1) to create a learned model W, and is exemplified as a function of the assist device 620.

[0328] As described above, the function of the storage unit 617 uses the third image F3 as training data, that is, the training image Ji. Figure 5 It is obtained by the image processing method illustrated in steps S2, S3, and S4.

[0329] As previously described with respect to the function of the image input interface 611 and as previously described with respect to the image processing method Figure 5 As illustrated in step S1 of FIG. 1 , the first image F1 is set from the photographic image F0. The photographic image F0 is obtained by photographing the object (in Figure 38 In the example, it is obtained by rotating the chuck 91).

[0330] Pixel group 100 is labeled by region extraction. Pixel group 100 is extracted from the pixel group located in third region R3 in both first image F1 and second image F2. This region extraction is performed when each third image F3 is obtained.

[0331] Based on this situation, the method for generating training data, i.e., training images Ji, may include the following steps: using the third image F3 as the training image Ji; photographing the object and setting the first image F1 before generating the second image F2 and the third image F3; and extracting the pixel group 100 from the area of either the first image F1 or the second image F2 each time each third image F3 is obtained.

[0332] Figure 39 This figure shows another example of a display image F9 displayed on the display 630. For example, the camera 600 and the display 630 are mounted on the same device, and the marked object is photographed while the mark is visually recognized. An example of such a device is smart glasses.

[0333] Figure 39Image 8 is shown as an example of display image F9. Marker 80 is shown in image 8. A marked object is captured in image 8, and frame 81 surrounds the object. Label 83 indicates that the object is a spin chuck, as indicated by "SpinChuck." Label 83 is connected to frame 81 via arrow 82. Frame 81, arrow 82, and label 83 are included in marker 80.

[0334] <7. Changes>

[0335] <7-1. Case where the fourth region R4 and the fifth region R5 do not overlap>

[0336] For example, Figure 17 Specifically, the fifth region R55 overlaps with the fourth region 46, and the fifth region R56 overlaps with the fourth region 45. However, the present disclosure is not limited to the case where the fourth region R4 overlaps with the fifth region R5. For example, it is feasible that if the fifth region R55 does not include an area the size of the partitioning figure V42, the second affine transformation is not performed on it, but the second affine transformation is performed on a smaller partitioning figure, such as the partitioning figure V46, to arrange the pixel group E46 along the vertex P51 and the edges L512 and L541.

[0337] <7-2. Case where the fourth region R4 and the fifth region R5 overlap>

[0338] The first affine transformation is a rotation without parallel translation, and when the center of the rotation is the center of the first region R1, the fourth region R4 and the fifth region R5 overlap. When such a first affine transformation is used, the second affine transformation is also a rotation without parallel translation.

[0339] assumed Figures 8 to 17 The second region R2 shown is obtained by a first affine transformation that rotates only about the center of the first region R1. In this case, the pixel groups D41, D42, D43, D44, E42, E43, and E46 can be obtained by performing a second affine transformation on the distinguishing patterns T41, T42, T43, T44, V42, V43, and V46, respectively, that rotates only about the center of the first region R1.

[0340] <7-3. Others>

[0341] In the movement source pixel group Gk and the movement destination pixel group Hk, for example, a circular shape or a polygonal shape may be adopted.

[0342] Furthermore, it is of course possible to appropriately combine all or part of the above-described embodiments and various modifications within a range that does not conflict with each other.

[0343] Description of Reference Numerals

[0344] F1: First image

[0345] F2: Second picture

[0346] F3, 3A, 3B: Third image

[0347] G1, G2, G3, G4, Ga, Gb, Gc, Gd, Gk: moving source pixel group

[0348] H1, H2, H3, H4, Ha, Hb, Hk: Movement destination pixel group

[0349] Ji: training images

[0350] M1, M2, M3, M4, Ma, Mb, Mc, Md, Mk: number

[0351] R1: First Area

[0352] R2: Second Area

[0353] R3: The third area

[0354] R4, R41, R42, R43, R44, R45, R46, R4z: The fourth area

[0355] R5, R51, R52, R53, R54, R55, R56: The fifth area

[0356] S1, S2, S3, S4, S5, S30, S31, S32, S33, S34, S35a, S35b, S35c, S35d, S36, S36a, S36b, S3 7. S38, S39, S41, S42, S43, S301, S302, S303, S304, S305, S309, S351, S351a, S351b, S35 2. S352a, S352b, S353, S353a, S353b, S353c, S353d, S354, S354a, S354b, S355, S355a, S355b, S356, S356a, S356b, S356c, S357a, S357b, S358a, S358b, S359a, S359b, S360: Steps

[0357] T40, T41, T42, T43, T44, V41, V42, V43, V46, V47, V48, V49, V4a, V4a1, V4a2: Differentiate graphics

[0358] V5a, V5a1, V5a2: Differentiate regions

Claims

1. An image processing method, wherein: include: a transformation step of performing a first affine transformation on a first image having a plurality of pixels and occupying a first area, to obtain a second image having the plurality of pixels and occupying a second area overlapping and inconsistent with the first area; as well as a moving step of moving or copying the plurality of pixels of the second image to the first area to obtain a third image, The second area is divided into a third area and a fourth area, the third area is located within the first area, and the fourth area is located outside the first area. The first area is divided into the third area and the fifth area, and the fifth area is located outside the second area. The moving process includes: A first step is to set a first movement source pixel group, wherein the first movement source pixel group is a pixel group connected to the plurality of pixels included in the fourth area, that is, a predetermined first number of pixels greater than 1; as well as In the second step, a second affine transformation is performed on the first movement source pixel group to obtain a first movement destination pixel group arranged in the fifth area. The first affine transformation is either or both of a rotation and a parallel translation, and the second affine transformation is either or both of a rotation and a parallel translation, or a mirror transformation.

2. The image processing method according to claim 1, wherein: The first movement source pixel group is inscribed in the fourth area, and the first movement destination pixel group is inscribed in the fifth area.

3. The image processing method according to claim 1 or 2, wherein: The moving process includes: The third step is to set a second movement source pixel group, wherein the second movement source pixel group is a pixel group connected to the plurality of pixels included in the fourth area, that is, a predetermined second number of pixels; and In a fourth step, a third affine transformation is performed on the second movement source pixel group to obtain a second movement destination pixel group arranged in the fifth area. The second number is less than the first number, The third affine transformation is either or both of a rotation and a parallel translation, or a mirror transformation.

4. The image processing method according to claim 3, wherein: The second number is 1, and the third step and the fourth step are repeatedly performed until all the plurality of pixels of the second image are moved or copied to the first area.

5. The image processing method according to claim 3, wherein: The first step and the second step are performed a plurality of times before the third step.

6. The image processing method according to any one of claims 1 to 5, wherein: A partitioning pattern that starts from a position away from the third area, connects pixels in the fourth area, and is enlarged is set as the first movement source pixel group.

7. The image processing method according to any one of claims 1 to 6, wherein: The first movement destination pixel group is arranged in a partitioned area that is enlarged by connecting pixels in the fifth area starting from a position away from the third area.

8. The image processing method according to claim 1 or 2, wherein: The fourth region and the fifth region are in a mirror image relationship, and the second affine transformation is a mirror image transformation.

9. The image processing method according to any one of claims 3 to 5, wherein: The fourth region and the fifth region are in a mirror image relationship, and the third affine transformation is a mirror image transformation.

10. A method for generating training data, which is a method for generating training data for machine learning for recognizing objects in images, wherein: The method for generating training data includes the following steps: using the plurality of third images obtained by the image processing method according to any one of claims 1 to 9 as the training data; Before the image processing method, the object is photographed and the first image is set; as well as When each of the third images is obtained, a pixel group corresponding to the object is regionally extracted in either the first image or the second image from the pixel group located in the third area in both the first image and the second image.

Images