Quality control system using three-dimensional image data and quality control program using three-dimensional image data

Abstract

To provide a novel device that generates three-dimensional image data or a novel program that generates three-dimensional image data, or provide a novel quality control system using new three-dimensional image data and machine learning, or a novel quality control program using new three-dimensional image data and machine learning.SOLUTION: A three-dimensional image data generation device according to an embodiment of the present invention includes: a sub-pixel data extraction unit that extracts first, second, and third sub-pixel data from captured first image data; a height information extraction unit that extracts height information from the first image data; and an image data integration unit that generates second image data where the first, the second, and the third sub-pixel data and the height information are stored in one row for each pixel.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to a three-dimensional image data generation device. Alternatively, this invention relates to a quality control system using three-dimensional image data, particularly a quality control system using machine learning. Alternatively, this invention relates to a quality control program using machine learning. [Background technology]

[0002] Numerous image recognition technologies using machine learning have been reported. For example, Patent Document 1 describes an image processing device comprising: an image input means for inputting an image; a pixel combination acquisition means for acquiring a combination of two pixels from the input image; a co-occurrence acquisition means for acquiring the co-occurrence of the gradient direction of the brightness of the two pixels related to the acquired combination; an occurrence frequency acquisition means for acquiring the distribution of the occurrence frequency of acquired co-occurrences while changing the combination of pixels to be acquired; a probability density function generation means for generating a probability density function corresponding to the acquired distribution of occurrence frequencies; and a probability density function output means for outputting the generated probability density function using parameters that define the probability density function.

[0003] Furthermore, as a method for determining whether or not a subject has an abnormality using machine learning-based image recognition technology, Patent Document 2 describes an abnormality discrimination system comprising: a feature extraction unit that extracts feature quantities of a subject from multiple image data acquired sequentially for the subject; and a determination unit that determines whether or not a subject has an abnormality based on the feature quantities extracted by the feature extraction unit and a predetermined threshold. The feature extraction unit corresponds to an encoder that, together with a decoder and a classifier, constitutes an adversarial autoencoder. The system is adapted by sequentially repeating a first learning process to enable the decoder to reconstruct the input image data from the feature quantities extracted by the encoder; a second learning process to enable the classifier to distinguish between a sample from a predetermined distribution and the feature quantities extracted by the encoder; and a third learning process to ensure that the distribution of feature quantities extracted by the encoder approaches a predetermined distribution based on the classifier's identification result. The threshold is determined by clustering multiple feature quantities extracted from multiple normal subjects by the adapted feature extraction unit.

[0004] The image data obtained by imaging a product is generally two-dimensional image data composed of three parameters defined using the RGB color model or the HSV color space. In recent years, technologies for generating three-dimensional image data of a product by acquiring the height information of the product have been reported. For example, Patent Document 3 discloses a measurement microscope capable of performing measurement with three-dimensional profile information and observation of an optical image, and capable of reducing the influence of measurement light on other imaging. The measurement microscope device includes an observation imaging of an object image and acquisition of a three-dimensional shape. It has an observation illumination light source for generating illumination light at the time of imaging the observation image, a measurement light projection means for performing structured illumination by projecting measurement light from an oblique direction onto the object, an imaging means for acquiring a plurality of fringe images by obtaining the measurement light projected by the measurement light projection means and reflected by the object, and for imaging an observation image having texture information using the observation illumination light source, a height image acquisition means for acquiring a height image having height information based on the plurality of fringe images, a measurement control unit for controlling the measurement light projection means, and a main body case for housing the measurement light projection means and the imaging means. The measurement light projection means includes a measurement light source that is a light source of the measurement light, and a pattern generation unit that is disposed on the optical path of the measurement light from the measurement light source to the object, generates a periodic projection pattern of the measurement light emitted by the measurement light source, and projects it onto the object. The pattern generation unit can be switched between a light projection state in which the measurement light is projected onto the object and a non-light projection state in which the measurement light is not projected onto the object by the measurement control unit. The measurement control unit is characterized in that when imaging an observation image by irradiating the object with the observation illumination light source, the measurement light source is kept lit and the pattern generation unit is in the non-light projection state. A measurement microscope device is described.

[0005] In Patent Document 3, it is necessary to obtain an observation image having texture information in the RGB color model and a height image respectively, and then synthesize and display the two types of image data. Therefore, when applying the technology of generating three-dimensional image data of conventional products to the image recognition technology using machine learning, the number of processes increases, and each time, it is necessary to synthesize two types of image data to generate three-dimensional image data, which compresses the system resources and makes it difficult to realize a high-speed quality management system.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0007] One object of an embodiment of the present invention is to provide an apparatus for generating new three-dimensional image data. Or, one object of an embodiment of the present invention is to provide a program for generating new three-dimensional image data. Or, one object of an embodiment of the present invention is to provide a quality management system using new three-dimensional image data and machine learning. Or, one object of an embodiment of the present invention is to provide a quality management program using new three-dimensional image data and machine learning.

Means for Solving the Problems

[0008] A three-dimensional image data generation apparatus according to one embodiment of the present invention comprises: a sub-pixel data extraction unit that extracts first, second, and third sub-pixel data from captured first image data; a height information extraction unit that extracts height information from the first image data; and an image data integration unit that generates second image data in which the first, second, and third sub-pixel data and height information are stored in one row for each pixel.

[0009] When the first sub-pixel data is S1, the second sub-pixel data is S2, the third sub-pixel data is S3, and the height information is Hi, the second image data may have the data arrangement (S1, S2, S3, Hi).

[0010] The sub-pixel data extraction unit generates third image data having a data arrangement of (R,G,B,a), where the first sub-pixel data is R, the second sub-pixel data is G, the third sub-pixel data is B, and transparency is a. The image data integration unit may then rewrite a in the third image data to Hi to generate second image data.

[0011] A three-dimensional image data generation program according to one embodiment of the present invention causes a computer to extract first, second, and third sub-pixel data from captured first image data, to extract height information from the first image data, and to generate second image data in which the first, second, and third sub-pixel data and height information are stored in one row for each pixel.

[0012] When the first sub-pixel data is S1, the second sub-pixel data is S2, the third sub-pixel data is S3, and the height information is Hi, the second image data may have the data arrangement (S1, S2, S3, Hi).

[0013] The computer may generate a third image data having the data arrangement (R,G,B,a), where the first sub-pixel data is R, the second sub-pixel data is G, the third sub-pixel data is B, and transparency is a. The computer may then generate a second image data by overwriting a in the third image data with Hi.

[0014] A quality control system according to one embodiment of the present invention includes: a sub-pixel data extraction unit that extracts first, second, and third sub-pixel data from one or more captured first image data; a height information extraction unit that extracts height information from one or more first image data; an image data integration unit that generates one or more second image data in which the first, second, and third sub-pixel data and height information are stored in one row for each pixel; an autoencoder that generates difference data including the difference between training image data composed of one or more second image data and reconstructed image data generated from the training image data; and a file generation unit that calculates the average value and variance of the difference data for each predetermined interval and generates a file.

[0015] The autoencoder extracts features from multiple training image data pixels by pixels, generates reconstructed image data based on the extracted features, and the file generation unit may apply a Gaussian mixture model to calculate the mean and variance of the difference data.

[0016] The system further includes a determination unit that determines whether or not there are abnormal values ​​in the differential data, and the determination unit may determine whether or not there are abnormal values ​​in the differential data generated from the third image data of the object being examined.

[0017] The determination unit may set a second interval having a different size from the first interval based on the range of the first interval containing the difference data determined to be an outlier, and calculate the mean and variance of the difference data for each of the second intervals.

[0018] The system further includes a determination unit that determines whether or not there are abnormal values ​​in the differential data, and the determination unit may compare the design data with the differential data to determine whether or not there are abnormal values ​​in the differential data.

[0019] The design data has a predetermined range of tolerance values ​​for each section, and the determination unit may compare the range of tolerance values ​​with the difference data to determine whether there are any abnormal values ​​in the difference data, and weight the sections containing abnormal values.

[0020] When the first sub-pixel data is S1, the second sub-pixel data is S2, the third sub-pixel data is S3, and the height information is Hi, the second image data may have the data arrangement (S1, S2, S3, Hi).

[0021] The sub-pixel data extraction unit generates fourth image data having a data arrangement of (R,G,B,a), where the first sub-pixel data is R, the second sub-pixel data is G, the third sub-pixel data is B, and transparency is a. The image data integration unit may then rewrite a in the fourth image data to Hi to generate second image data.

[0022] A quality control program according to one embodiment of the present invention includes: causing a computer to extract first, second, and third sub-pixel data from one or more captured first image data; causing a computer to extract height information from one or more first image data; causing a computer to generate second image data in which the first, second, and third sub-pixel data and height information are stored in one row for each pixel; causing a computer to generate difference data including the difference between training image data composed of one or more second image data and reconstructed image data generated from the training image data; and causing the computer to calculate the mean and variance values ​​of the difference data for each predetermined interval and generate a file.

[0023] Alternatively, a computer may be used to extract features from multiple training image data pixels by pixel, generate reconstructed image data based on the extracted features, and then apply a Gaussian mixture model to calculate the mean and variance of the difference data.

[0024] The computer may be instructed to determine whether there are any abnormal values ​​in the difference data generated from the third image data of the object being examined, and then the computer may be instructed to generate a second image containing the abnormal values ​​based on the first image data of the object being examined and the determination result.

[0025] Based on the range of a first interval containing the difference data determined to be an outlier, the computer may be instructed to set a second interval having a different size from the first interval, and the computer may be instructed to calculate the mean and variance of the difference data for each of the second intervals.

[0026] The design data and the difference data can be compared, and the computer can be used to determine whether or not there are any outliers in the difference data.

[0027] The design data may have a predetermined tolerance range for each section, and the computer may compare the tolerance range with the difference data to determine whether there are any abnormal values ​​in the difference data, and then weight the sections that contain abnormal values.

[0028] When the first sub-pixel data is S1, the second sub-pixel data is S2, the third sub-pixel data is S3, and the height information is Hi, the second image data may have the data arrangement (S1, S2, S3, Hi).

[0029] The computer may generate a third image data having the data arrangement (R,G,B,a), where the first sub-pixel data is R, the second sub-pixel data is G, the third sub-pixel data is B, and transparency is a. The computer may then generate a second image data by overwriting a in the third image data with Hi. [Effects of the Invention]

[0030] One embodiment of the present invention provides an apparatus for generating novel three-dimensional image data. Alternatively, one embodiment of the present invention provides a program for generating novel three-dimensional image data. Alternatively, one embodiment of the present invention provides a quality control system using novel three-dimensional image data and machine learning. Alternatively, one embodiment of the present invention provides a quality control program using novel three-dimensional image data and machine learning. [Brief explanation of the drawing]

[0031] [Figure 1]This is a block diagram showing a three-dimensional image data generation device 100 according to one embodiment of the invention. [Figure 2] This is a flowchart illustrating the general process related to one embodiment of the present invention. [Figure 3] This is a flowchart illustrating step S10 for extracting sub-pixel data according to one embodiment of the present invention. [Figure 4] This is a flowchart illustrating step S20 for extracting height information Hi from first image data according to one embodiment of the present invention. [Figure 5] This is a flowchart illustrating step S30 for integrating sub-pixel data and height information Hi according to one embodiment of the present invention. [Figure 6] (A) shows an RGB image created by combining R subpixel data, G subpixel data, and B subpixel data; (B) shows an image based on transparency a; and (C) shows an RGBa image created by combining R subpixel data, G subpixel data, B subpixel data, and transparency a. [Figure 7] (A) shows an RGB image created by combining R subpixel data, G subpixel data, and B subpixel data; (B) shows an image based on height information Hi; and (C) shows an RGBHi image created by combining R subpixel data, G subpixel data, B subpixel data, and height information Hi. [Figure 8] This is a block diagram showing a quality control system 200 according to one embodiment of the present invention. [Figure 9] This is a flowchart illustrating the general process related to one embodiment of the present invention. [Figure 10] (A) is a flowchart illustrating step S10 for generating differential data according to one embodiment of the present invention, and (B) is a conceptual diagram illustrating step S10 for generating differential data according to one embodiment of the present invention. [Figure 11] Figures (A) to (D) show the relationship between one pixel (px) and a predetermined section A used by the file generation unit 230 for image data. [Figure 12] This flowchart illustrates step S30, which involves calculating the mean and variance values ​​of differential data and generating a file according to one embodiment of the present invention. [Figure 13] This is a flowchart illustrating step S40 for visualizing the features of one embodiment of the present invention. [Figure 14] (A) is a schematic diagram showing the RGBHi image generated by the visualization processing unit 250, and (B) is a schematic diagram (RGBa) of the top view of the product. [Figure 15] This is a flowchart illustrating a product quality control method for one embodiment of the present invention. [Figure 16] (A) is a schematic diagram showing the RGBHi image generated by the visualization processing unit 250, and (B) is a schematic diagram (RGBa) of the top view of the object with damage. [Modes for carrying out the invention]

[0032] Hereinafter, one embodiment of the present invention will be described with reference to the drawings. The embodiments shown below are examples of embodiments of the present invention, and the present invention is not limited to these embodiments.

[0033] In the drawings referenced in this embodiment, identical or similar reference numerals (simply numbers followed by A, B, etc.) are denoted by the same or similar reference numerals. Furthermore, the dimensional ratios in the drawings may differ from the actual ratios for illustrative purposes, and some components may be omitted from the drawings. In addition, ordinal numbers such as "First," "Second," and "Third" in this specification are used solely for the purpose of simplifying the explanation and should not be interpreted restrictively.

[0034] [Three-dimensional image data generation device] Figure 1 is a block diagram showing a three-dimensional image data generation device 100 according to one embodiment of the present invention. The three-dimensional image data generation device 100 includes, for example, a sub-pixel data extraction unit 120, a height information extraction unit 130, and an image data integration unit 140. In one embodiment, the three-dimensional image data generation device 100 further includes, for example, an input unit 110, a control unit 160, and an output unit 170. In one embodiment, the three-dimensional image data generation device 100 further includes a storage unit 180 and a communication unit 190, but is not limited to these.

[0035] The sub-pixel data extraction unit 120 is a program or module that extracts first sub-pixel data S1, second sub-pixel data S2, and third sub-pixel data S3 from the captured first image data. In one embodiment, the sub-pixel data extraction unit 120 is stored in the storage unit 180 and executed by the control unit 160. The first sub-pixel data S1, second sub-pixel data S2, and third sub-pixel data S3 extracted from the first image data may correspond to, for example, red (R), green (G), and blue (B) that define the RGB color model. Alternatively, the first sub-pixel data S1, second sub-pixel data S2, and third sub-pixel data S3 may correspond to hue (H), saturation (S), and lightness (V) that define the HSV color space. The sub-pixel data extraction unit 120 receives the captured first image data as input and generates these first, second, and third sub-pixel data.

[0036] The height information extraction unit 130 is a program or module that extracts height information Hi from the first image data. In one embodiment, the subpixel data extraction unit 120 is stored in the storage unit 180 and executed by the control unit 160. The method for extracting height information Hi can be set according to the configuration of the input first image data. Details of the method for extracting height information Hi will be described later.

[0037] The image data integration unit 140 is a program or module that generates second image data in which the first sub-pixel data S1, the second sub-pixel data S2, the third sub-pixel data S3, and height information Hi are stored in one row for each pixel. In one embodiment, the sub-pixel data extraction unit 120 is stored in the storage unit 180 and executed by the control unit 160. In this embodiment, the second image data has, for example, the following data structure. (S1, S2, S3, Hi)

[0038] Conventionally, image data having a structure that defines the RGB color model (R, G, B) or the HSV color space (H, S, V) and data containing height information were generated as separate files for image processing or saving. The present invention differs from the conventional technology in that it generates a data file in which the first sub-pixel data S1, the second sub-pixel data S2, the third sub-pixel data S3, and height information Hi are stored in one row for each pixel. Conventional technology, which always requires processing two files, increases the size of the data to be processed and the number of steps involved in image processing, thus reducing processing speed.

[0039] Another conventional technique involves a data format in which one of the three subpixel data (R, G, B) is replaced with height information (Hi). For example, a data file is known in which (G, B, Hi) is stored in one row for each pixel. However, in such a data format, one subpixel data is lost, resulting in a decrease in the accuracy of the displayed image.

[0040] In this invention, since a data file containing the first sub-pixel data S1, the second sub-pixel data S2, the third sub-pixel data S3, and height information Hi stored in one row for each pixel can be used for image processing, the size of the data to be processed is smaller than in conventional methods, and the number of steps required to perform image processing is also reduced, resulting in improved processing speed. Furthermore, in this invention, since three sub-pixel data sets—the first sub-pixel data S1, the second sub-pixel data S2, and the third sub-pixel data S3—can be used, the accuracy of the displayed image is improved.

[0041] The input unit 110 is an input device for operating the three-dimensional image data generation device 100, and can use known input devices such as a keyboard, mouse, and a display device (for example, a liquid crystal display or an organic EL display). The input unit 110 may also include an imaging unit 111 that acquires first image data to be provided to the sub-pixel data extraction unit 120 and the height information extraction unit 130. The imaging unit 111 is a camera that acquires the first image data, and can use known image sensors such as CCD or CMOS. In this embodiment, the sub-pixel data extraction unit 120 is a device capable of capturing a color image because it extracts first sub-pixel data S1, second sub-pixel data S2, and third sub-pixel data S3 that define an RGB color model or HSV color space. An example of such an imaging unit 111 is a device equipped with image sensors for the right eye and left eye, capable of stereo imaging using a line scan method.

[0042] The control unit 160 consists of a known central processing unit (CPU), an operating system (OS), and a control program or module for controlling the three-dimensional image data generation device 100. Alternatively, the control unit 160 may be provided as a single program including the OS and the control program or module. The control program or module constituting the control unit 160 is stored in the storage unit 180 and executed by the CPU.

[0043] The output unit 170 is a known output device for outputting the second image data generated by the image data integration unit 140. The output unit 170 also includes a display unit 171. The display unit 171 may be a known display device (e.g., a liquid crystal display or an organic EL display), and is not particularly limited. In one embodiment, the output unit 170 may be a display device equipped with a touch panel that can also function as an input unit 110.

[0044] The storage unit 180 is a storage device that stores the operating system (OS) and control program or module constituting the control unit 160, the program or module constituting the sub-pixel data extraction unit 120, the program or module constituting the height information extraction unit 130, the program or module constituting the image data integration unit 140, a file containing the first sub-pixel data S1, the second sub-pixel data S2, and the third sub-pixel data S3 generated by the sub-pixel data extraction unit 120, a file containing the height information Hi generated by the height information extraction unit 130, or the second image data generated by the image data integration unit 140. The storage unit 180 is composed of a known storage device, such as a memory, hard disk, or solid-state drive (SSD). The storage unit 180 may be built into a computer having the control unit 160, or it may be located on a server or network drive that can communicate via the communication unit 190.

[0045] The communication unit 190 is a known wired or wireless communication device controllable by the control unit 160. The communication unit 190 can be connected to communication networks such as local area networks (LANs), wide area networks (WANs), and the Internet. The communication unit 190 may also be a communication device conforming to wireless communication standards such as Wi-Fi® (a communication method using the IEEE 802.11 standard) or Bluetooth®. The communication unit 190 may also perform data communication with servers or network drives located outside the computer having the communication unit 190.

[0046] [Processing by a three-dimensional image data generation device] The processing performed by the three-dimensional image data generation device 100 will be explained with reference to Figures 2 to 5. Figure 2 is a flowchart illustrating the general outline of the processing according to one embodiment of the present invention. The three-dimensional image data generation device 100 extracts first sub-pixel data S1, second sub-pixel data S2, and third sub-pixel data S3 from the captured first image data (step S10). The three-dimensional image data generation device 100 extracts height information Hi from the first image data (step S20). Then, the three-dimensional image data generation device 100 generates second image data in which the first sub-pixel data S1, second sub-pixel data S2, third sub-pixel data S3, and height information Hi are stored in one row for each pixel (step S30).

[0047] Refer to Figure 3. Figure 3 is a flowchart illustrating step S10 for extracting sub-pixel data according to one embodiment of the present invention. The sub-pixel data extraction unit 120 acquires the first image data (step S11). The first image data can be acquired by the imaging unit 111 imaging the product. Alternatively, the sub-pixel data extraction unit 120 may acquire the first image data that has been acquired in advance and stored in the storage unit 180.

[0048] The sub-pixel data extraction unit 120 extracts the first sub-pixel data S1, the second sub-pixel data S2, and the third sub-pixel data S3 from the first image data (step S13). The method for extracting the first sub-pixel data S1, the second sub-pixel data S2, and the third sub-pixel data S3 from the first image data is not particularly limited, and known image processing techniques can be used, so a detailed explanation is omitted. In this embodiment, the sub-pixel data extraction unit 120 preferably processes a 48-bit image, but it may also process a 24-bit image.

[0049] The sub-pixel data extraction unit 120 generates a file containing the extracted first sub-pixel data S1, second sub-pixel data S2, and third sub-pixel data S3 (step S15). The file generated in step S15 may be, for example, image data having a (R,G,B) structure that defines the RGB color model or image data having a (H,S,V) structure that defines the HSV color space. Alternatively, for the purpose of generating a file with a (S1,S2,S3,Hi) data structure by the image data integration unit 140, a file with a (S1,S2,S3,0) data structure with the Hi component set to 0 may be generated.

[0050] Refer to Figure 4. Figure 4 is a flowchart illustrating step S20 for extracting height information Hi from first image data according to one embodiment of the present invention. The height information extraction unit 130 acquires the first image data (step S21). The height information extraction unit 130 extracts height information Hi from the first image data (step S23). The method for extracting height information Hi is not particularly limited, and known image processing techniques can be used. For example, if the imaging unit 111 is a device that measures the three-dimensional shape of an object by a fringe projection method using fringe-shaped measurement light, the height information extraction unit 130 can extract height information Hi by performing calculation processing using the fringe projection method. Also, if the imaging unit 111 is a stereo camera, the height information extraction unit 130 can calculate the distance Z from the imaging unit 111 to the object (i.e., height information Hi) using the following formula, based on the distance B between the left and right image sensors arranged in the imaging unit 111, the focal length F between the two image sensors, and the parallax D between the two image sensors. Z = B*F / D

[0051] The height information extraction unit 130 generates a file containing the extracted height information Hi (step S25). For the purpose of generating a file with the data structure (S1, S2, S3, Hi) by the image data integration unit 140, the pixel coordinate data and the Hi component data at those coordinates may be transmitted to the image data integration unit 140. In this embodiment, the height information extraction unit 130 preferably processes the height information Hi with 16 bits, but it may also process it with 8 bits.

[0052] Refer to Figure 5. Figure 5 is a flowchart illustrating step S30 for integrating sub-pixel data and height information Hi according to one embodiment of the present invention. The image data integration unit 140 reads the sub-pixel data extracted by the sub-pixel data extraction unit 120 and the height information Hi extracted by the height information extraction unit 130 (step S31).

[0053] The image data integration unit 140 integrates the read subpixel data and height information Hi to generate a file having the data structure (S1, S2, S3, Hi) (step S33). Alternatively, if the image data integration unit 140 generates a file having the data structure (S1, S2, S3, 0), it replaces the 0s in (S1, S2, S3, 0) with height information Hi to generate a file having the data structure (S1, S2, S3, Hi). For example, for image data having the structure (R, G, B) that defines the RGB color model, a second image data having the structure (R, G, B, Hi) is generated, and for image data having the structure (H, S, V) that defines the HSV color space, a second image data having the structure (H, S, V, Hi) is generated.

[0054] In one embodiment, if the sub-pixel data extraction unit 120 generates image data having a data arrangement of (R, G, B, a) where the first sub-pixel data is R, the second sub-pixel data is G, the third sub-pixel data is B, and transparency is a, the image data integration unit 140 can rewrite a in the image data to Hi to generate second image data.

[0055] In this embodiment, the second image data generated stores the first sub-pixel data S1, the second sub-pixel data S2, the third sub-pixel data S3, and height information Hi in one row for each pixel. As a result, the size of the data used for image processing is smaller than in conventional methods, and the number of steps required to perform image processing is also reduced, thus improving processing speed. Furthermore, since the second image data generated in this embodiment can utilize three sub-pixel data sets—the first sub-pixel data S1, the second sub-pixel data S2, and the third sub-pixel data S3—the accuracy of the displayed image is improved.

[0056] [3D Image Data Generation Program] In one embodiment, a program for executing the above-described processes can be provided. Alternatively, in one embodiment, the program can be provided as a recording medium storing the program. This will be explained with reference to Figures 2 to 5. The program causes the three-dimensional image data generation device 100 to extract a first sub-pixel data S1, a second sub-pixel data S2, and a third sub-pixel data S3 from the captured first image data (step S10). The program causes the three-dimensional image data generation device 100 to extract height information Hi from the first image data (step S20). Then, the program causes the three-dimensional image data generation device 100 to generate second image data in which the first sub-pixel data S1, the second sub-pixel data S2, the third sub-pixel data S3, and the height information Hi are stored in one row for each pixel (step S30).

[0057] This program causes the sub-pixel data extraction unit 120 to take in the first image data (step S11). This program causes the sub-pixel data extraction unit 120 to extract the first sub-pixel data S1, the second sub-pixel data S2, and the third sub-pixel data S3 from the first image data (step S13). This program causes the sub-pixel data extraction unit 120 to generate a file containing the extracted first sub-pixel data S1, the second sub-pixel data S2, and the third sub-pixel data S3 (step S15).

[0058] This program causes the height information extraction unit 130 to take in the first image data (step S21). This program causes the height information extraction unit 130 to extract height information Hi from the first image data (step S23). This program causes the height information extraction unit 130 to generate a file containing the extracted height information Hi (step S25). For the purpose of generating a file having the data structure (S1, S2, S3, Hi) by the image data integration unit 140, the program may also transmit the pixel coordinate data and the Hi component data at those coordinates to the image data integration unit 140. In this embodiment, it is preferable for this program to cause the height information extraction unit 130 to process the height information Hi in 16 bits, but it may also process it in 8 bits.

[0059] This program causes the image data integration unit 140 to read the sub-pixel data extracted by the sub-pixel data extraction unit 120 and the height information Hi extracted by the height information extraction unit 130 (step S31). This program causes the image data integration unit 140 to integrate the read sub-pixel data and height information Hi to generate a file having the data structure (S1, S2, S3, Hi) (step S33). Alternatively, if the image data integration unit 140 generates a file having the data structure (S1, S2, S3, 0), this program replaces the 0s in (S1, S2, S3, 0) with the height information Hi to generate a file having the data structure (S1, S2, S3, Hi).

[0060] In one embodiment, if the program causes the sub-pixel data extraction unit 120 to generate image data having a data arrangement of (R,G,B,a) where the first sub-pixel data is R, the second sub-pixel data is G, the third sub-pixel data is B, and transparency is a, the program can cause the image data integration unit 140 to rewrite a in the image data to Hi and generate second image data.

[0061] In this embodiment, the second image data generated stores the first sub-pixel data S1, the second sub-pixel data S2, the third sub-pixel data S3, and height information Hi in one row for each pixel. As a result, the size of the data used for image processing is smaller than in conventional methods, and the number of steps required to perform image processing is also reduced, thus improving processing speed. Furthermore, since the second image data generated in this embodiment can utilize three sub-pixel data sets—the first sub-pixel data S1, the second sub-pixel data S2, and the third sub-pixel data S3—the accuracy of the displayed image is improved.

[0062] As an example, Figure 6(A) shows an RGB image created by combining R subpixel data, G subpixel data, and B subpixel data; Figure 6(B) shows an image based on transparency a; and Figure 6(C) shows an RGBa image created by combining R subpixel data, G subpixel data, B subpixel data, and transparency a. Furthermore, Figure 7(A) shows an RGB image created by combining R subpixel data, G subpixel data, and B subpixel data; Figure 7(B) shows an image based on height information Hi; and Figure 7(C) shows an RGBHi image created by combining R subpixel data, G subpixel data, B subpixel data, and height information Hi. Comparing the general RGBa image shown in Figure 6(C) with the RGBHi image according to this embodiment shown in Figure 7(C), it can be seen that the display of the RGBHi image with height information Hi added differs when the same object (or product) is imaged. In this embodiment, it is possible to generate an RGBHi image that includes height information Hi.

[0063] [Quality Management System] In one embodiment, a quality control system comprising a three-dimensional image data generation device 100 can be provided. Figure 8 is a block diagram showing a quality control system 200 according to one embodiment. In addition to the sub-pixel data extraction unit 120, height information extraction unit 130, and image data integration unit 140 described above, the quality control system 200 includes, for example, an autoencoder 220 and a file generation unit 230. In one embodiment, the quality control system 200 further includes, for example, an input unit 110, a determination unit 240, a control unit 260, and an output unit 270. In one embodiment, the quality control system 200 further includes a visualization processing unit 250, a storage unit 280, and a communication unit 290, but is not limited to these.

[0064] Figure 10(B) shows a diagram illustrating the autoencoder 220. Figure 11 shows the relationship between one pixel (px) and a predetermined section A used by the file generation unit 230 for image data of a product undergoing quality control. Figures 11(A) to 11(D) show image data of e × f pixels. The autoencoder 220 is one of the known algorithms for dimensionality reduction using a neural network, and comprises an encoder 127 that extracts features by compressing the input layer 121 into an intermediate layer 123, and a decoder 129 that restores the features from the intermediate layer 123 to the output layer 125. In one embodiment, the autoencoder 220 may be a program or module stored in the storage unit 280 and executed by the control unit 260. In this embodiment, the autoencoder 220 receives input of training image data 10 composed of a plurality of second image data, and performs encoding and decoding for each image of the training image data 10. The autoencoder 220 generates reconstructed image data 30 for each of the multiple training image data 10 by encoding and decoding each pixel (first interval) of the training image data 10. At this time, the autoencoder 220 extracts features contained in the training image data 10 for each pixel and generates difference data between the training image data 10 and the reconstructed image data 30 for each of the training image data 10. In one embodiment, the difference data is generated as an integer value for each pixel and may include values ​​of 0 or less. Note that because the autoencoder 220 places a heavy computational load on the quality control system 200, it may be configured as a processing unit that is located outside the quality control system 200 and can communicate with the communication unit 190 via a network.

[0065] The file generation unit 230 is a program or module that applies a Gaussian mixture model to calculate the mean and variance values ​​of the difference data for each predetermined interval (second interval). In one embodiment, the file generation unit 230 is stored in the storage unit 280 and executed by the control unit 260. The file generation unit 230 generates a file containing the mean and variance values ​​of the difference data calculated for each predetermined interval. In this specification, the "predetermined interval" can be set considering the quality required for the product subject to quality control, or the size of defects that occur in the product. In Figure 4, an example is shown in which 3 x 4 pixels are set as the predetermined interval, but it is not limited to this. The variance values ​​calculated by the file generation unit 230 are unimodal or multimodal normal distributions. Note that the autoencoder 220 and the file generation unit 230 may be a single programmer or module that performs a series of processes.

[0066] The input unit 110 is an input device for operating the quality control system 200, and since the above-described configuration can be used, a detailed explanation is omitted.

[0067] The determination unit 240 is a program or module that determines whether or not there are abnormal values ​​in the differential data generated by the file generation unit 230. In one embodiment, the determination unit 240 is stored in the storage unit 280 and executed by the control unit 260. In one embodiment, the determination unit 240 can determine whether or not there are abnormal values ​​in the differential data generated from the image data of the object being examined. Furthermore, in order to make it easier for the user to recognize abnormal values, the determination unit may generate a weighted image of the interval containing the abnormal value when it detects an abnormal value that falls outside a preset tolerance range.

[0068] The visualization processing unit 250 is a program or module that generates an image based on the mean variability or variance variability contained in the file generated by the file generation unit 230. In one embodiment, the file generation unit 230 is stored in the storage unit 280 and executed by the control unit 260. In the quality control system 200, the visualization processing unit 250 converts the mean variability or variance variability contained in the file generated by the file generation unit 230 into a human-readable image and outputs it. This makes it possible to visually recognize manufacturing errors and defects in products during product quality control, and allows people to easily understand information that is important for reviewing the manufacturing process.

[0069] The control unit 260 consists of a known central processing unit (CPU), operating system (OS), and control program or module for controlling the quality control system 200, and has the same configuration as the control unit 160 described above, so a detailed explanation is omitted.

[0070] The output unit 270 is a known output device for outputting differential data generated by the autoencoder 220, files generated by the file generation unit 230, or images generated by the visualization processing unit 250. Furthermore, since the output unit 270 has the same configuration as the output unit 170 described above, a detailed explanation is omitted.

[0071] The storage unit 280 is a storage device that stores the operating system (OS) and control program or module constituting the control unit 260, the program or module constituting the sub-pixel data extraction unit 120, the program or module constituting the height information extraction unit 130, the program or module constituting the image data integration unit 140, a file containing the first sub-pixel data S1, the second sub-pixel data S2, and the third sub-pixel data S3 generated by the sub-pixel data extraction unit 120, a file containing height information Hi generated by the height information extraction unit 130, or the second image data generated by the image data integration unit 140, the program or module constituting the autoencoder 220, the program or module constituting the file generation unit 230, the program or module constituting the visualization processing unit 250, the difference data generated by the autoencoder 220, the file generated by the file generation unit 230, or the image generated by the visualization processing unit 250. Since the storage unit 280 has the same configuration as the storage unit 180 described above, a detailed explanation is omitted.

[0072] Since the communication unit 290 has the same configuration as the communication unit 190 described above, a detailed explanation will be omitted.

[0073] [Processing by the quality control system] In the processing by the quality control system 200, the training image data 10, which is composed of multiple second image data, is generated by the processing by the three-dimensional image data generation device described above. A detailed explanation of the processing by the three-dimensional image data generation device will be omitted. The processing by the quality control system 200 will be explained with reference to Figures 9 to 16. Figure 9 is a flowchart illustrating the outline of the processing according to one embodiment of the present invention. The quality control system 200 generates difference data including the difference between the training image data composed of multiple second image data and the reconstructed image data generated from the training image data (step S110). The quality control system 200 calculates the average value and variance of the difference data for each predetermined interval and generates a file (step S120). Then, the quality control system 200 generates an image based on the data of the generated file (step S130). In this embodiment, processing using the first sub-pixel data S1, the second sub-pixel data S2, the third sub-pixel data S3 and height information Hi can be realized.

[0074] Refer to Figure 10. Figure 10(A) is a flowchart illustrating step S110 for generating differential data according to one embodiment of the present invention. Figure 10(B) is a conceptual diagram illustrating step S110 for generating differential data according to one embodiment of the present invention. The autoencoder 220 acquires a plurality of training image data 10 (step S111). The plurality of training image data 10 can be acquired by the imaging unit 111 imaging the product. Alternatively, the autoencoder 220 may acquire a plurality of training image data 10 that have been acquired in advance and stored in the storage unit 280.

[0075] The autoencoder 220 compresses the training image data 10 input to the input layer 221 using the encoder 227 to generate the intermediate layer 223. The autoencoder 220 then reconstructs the image data using the decoder 229 and outputs it to the output layer 225 to generate the reconstructed image data 30 (step S115). Through this process, the autoencoder 220 extracts features from the training image data 10 pixel by pixel (step S113).

[0076] More specifically, for example, the learning image data 10 of e×f pixels includes sub-pixel data for m pixels. Each pixel includes four components: the values (luminance values) of red (R), green (G), and blue (B) (sub-pixels) and the height information Hi. When the autoencoder 220 performs learning using n sheets of the learning image data 10, for the first sheet of the learning image data 10 to the nth sheet of the learning image data 10, compression by the encoder 227 and restoration by the decoder 229 are performed for each color component. The autoencoder 220 generates difference data from the restored image data 30 for a plurality of learning image data 10. For example, for the first pixel of the first sheet of the learning image data (R n1 , , n1 , 1m , nm , nm , , nm , n1 , n1 ,

[0077] , nm ,G 11 ,B 11 ,Hi 11 ), for the mth pixel (R 1m ,G 1m ,B 1m ,Hi 1m ), and for the first pixel of the nth sheet of the learning image data (R n1 ,G[[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​In this embodiment, by performing machine learning by repeating step S110 using multiple training image data 10, the difference data value between the training image data 10 and the reconstructed image data 30 approaches 0. The obtained data may be processed in the next step S120 or stored in the storage unit 280. In one embodiment, the above processing may be performed using (H,S,V) which defines the HSV color space.

[0078] In one embodiment, a person may use image processing software or the like to process the training image data 10 and correct the difference data generated by the autoencoder 220. Alternatively, if the difference data contains abnormal values ​​exceeding a specified value, the quality control system 200 may recognize the abnormal values ​​and display that the data is abnormal.

[0079] Refer to Figure 12. Figure 12 is a flowchart illustrating step S120, which calculates the mean and variance of difference data and generates a file according to one embodiment of the present invention. The file generation unit 230 calculates the mean and variance of the difference data generated by the autoencoder 220 for each pixel, for each predetermined interval. Based on the multiple difference data for each pixel, the file generation unit 230 generates a file to which a Gaussian mixture model is applied for each interval. For example, the difference data for the first interval is (μ R1 μ G1 μ B1 μ Hi1 σ R1 σ G1 σ B1 σ Hi1 σ RG1 σ GB1 σ BR1 σ RHi1 σ GHi1 σ BHi1 ), the difference data for the second interval is (μ R2 μ G2 μ B2 σ R2 σ G2 σ B2 σ RG2 σ GB2 σ BR2 σ RHi2 σ GHi2 σ BHi2), the difference data for the kth interval is (μ Rk μ Gk μ Bk σ Rk σ Gk σ Bk σ RGk σ GBk σ BRk σ RHik σ GHik σ BHik ) is defined as follows.

[0080] For example, the file generation unit 230 calculates the average μ for each component in each interval. For example, the average μ of the red (R) component in interval 1 / k. R1 This is calculated using the following formula (1).

number

[0081] Furthermore, the file generation unit 230 calculates the variance ρ for each component in each interval. For example, the variance σ of the red (R) component in interval 1 / k. R1 This is calculated using the following formula (2).

number

[0082] Furthermore, the file generation unit 230 calculates the covariance σ for each component in each interval (step S121). For example, the covariance σ of the red (R) and green (G) components in interval 1 / k. RG1 This is calculated using the following formula (3).

number

[0083] The file generation unit 230 outputs the calculation results as a normal distribution (step S123).

[0084] The obtained file may be processed in the next step S130, or it may be stored in the storage unit 280.

[0085] The setting of a predetermined interval (second interval) for generating a file in process S120 may be set by a person via the input unit 210, taking into consideration the quality required for the product subject to quality control and the size of defects that occur in the product. Reducing the size of the predetermined interval increases the accuracy of the generated file, but increases the amount of computation and lengthens the processing time in the file generation unit 230. In one embodiment, if a second interval is given as an initial value, the determination unit 240 may reset a third interval having a different size from the second interval based on the range of the second interval that includes the difference data determined to be an abnormal value, and the file generation unit 230 may perform calculation processing using the difference data for each more appropriate third interval to generate a file that includes the mean and variance values ​​of the difference data. The optimal interval may be set by repeating such a resetting process (refinement).

[0086] Furthermore, since (A) the maximum interval size in which the expected defect can be detected ≤ (X) the optimal interval size ≤ (B) the minimum interval size that can absorb the variability of the training image data 10, the size of (A) is calculated from the product specifications, and an interval setting (X) greater than or equal to the size of (A) is tentatively set considering the computation time. Model data is generated from the training image data 10 by the file generation unit 230. Images are generated from the generated model data by the visualization processing unit 150 and analyzed. If the model data shows a distribution with little variability, training is completed. On the other hand, if the model data shows a distribution with a lot of variability, it is determined that the size of (X) is greater than or equal to the size of (B), and the setting value is automatically changed numerically so that the size of (X) is less than or equal to the size of (B), and the model data is generated from the training image data 10 by the file generation unit 230. In this way, the quality control system 200 can automatically set the optimal interval size.

[0087] In one embodiment, when the file generation unit 230 performs the above calculations on the training image data 10 for each predetermined interval, it is preferable that the predetermined interval is moved relative to the training image data 10 such that a part of the second interval overlaps with the first interval. For example, in Figure 11(A), interval A including the first pixel is set. The file generation unit 230 moves interval A in the X-axis direction in order to calculate the mean and variance values. For example, as shown in Figure 11(B), the file generation unit 230 sets interval A' such that the two pixels on the right of interval A and the two pixels on the left of the moved interval A' overlap. Also, as shown in Figure 11(C), the file generation unit 230 sets interval B such that the two pixels on the bottom of interval A and the two pixels on the top of the moved interval B overlap. Furthermore, as shown in Figure 11(D), the file generation unit 230 sets interval B' such that the two pixels on the right of interval B and the two pixels on the left of the moved interval B' overlap. In other words, the file generation unit 230 sets section B such that the two pixels on the right and the two pixels below section A overlap with the two pixels on the left and the two pixels above section B after movement in the Y-axis direction.

[0088] Here, if the size of the training image data 10 is vertical (Him) × horizontal (Wim), the size of the predetermined interval is vertical (Hsp) × horizontal (Wsp), and the amount of movement of the interval is vertical (Hsd) × horizontal (Wsd), H = [(Him - Hsp) / Hsd] + 1 W = [(Wim - Wsp) / Wsd] + 1 As a result, the number of files generated is H × W. By setting the pre- and post-movement intervals to overlap in this way, the file generation unit 230 can generate more accurate average and variance values, preventing product defects from being overlooked. On the other hand, overlapping the pre- and post-movement intervals results in a larger number of files being generated than when they are not overlapped, which increases the processing time and load on the file generation unit 230. For this reason, the ratio of overlapping the pre- and post-movement intervals, i.e., the amount of interval movement, may be set considering the calculation speed, etc. Furthermore, the file generation unit 230 may be located outside the quality control system 200 and may consist of a processing unit that can communicate with the communication unit 290 via a network.

[0089] The visualization processing unit 250 aggregates the files output by the file generation unit 230 as images. Figure 13 is a flowchart illustrating step S130 for visualizing features according to one embodiment of the present invention. The visualization processing unit 250 aggregates the data obtained for each interval as a single image (step S131). The visualization processing unit 250 combines the mean or variance values ​​output by the file generation unit 230 based on the pixel position information of the difference data. As a result, the visualization processing unit 250 outputs the mean or variance value as an image for each color component (step S133). The output image of the mean value will be the number of color components, and the output image of the variance value will be the square of the number of color components.

[0090] The display unit 271 displays the image output by the visualization processing unit 250, allowing the user to visually perceive the variability in the characteristics of the object being examined. Alternatively, the visualization processing unit 250 may store the image in the storage unit 280, or output it to a printer via the output unit 270. Furthermore, the communication unit 290 may transmit the image to a server or network drive.

[0091] The images aggregated by the visualization processing unit 250 may differ from human intuition. In one embodiment, images processed by a human may be overwritten as visualization information for a new file. This allows the file output by the file generation unit 230 to be corrected to an ideal state from the visualization information, correcting unexpected variations in the images generated from the training image data 10 and enabling correct judgment.

[0092] [Evaluation of manufacturing tolerances] In one embodiment, when machine learning is performed in the quality control system 200 using non-defective products (good products or acceptable products) as training image data 10, users can visually recognize the variability in the products. Specifically, the visualization processing unit 250 generates an RGBHi image by combining the height information Hi contained in the file. Figure 14(A) is a schematic diagram showing the RGBHi image generated by the visualization processing unit 250, and Figure 14(B) is a schematic diagram (RGBa) of the top view of the product. In one embodiment, the visualization processing unit 250 can display the RGBHi image as an image in which the height information Hi is represented by data values ​​from 0 to 255 (256 gradations). Alternatively, the visualization processing unit 250 can also display the RGBHi image as a mask image converted to 0 and 1 depending on whether or not a reference height is met. As an example, Figure 14(A) shows an RGBHi image in which the visualization processing unit 250 evaluates the height variability in three stages and displays the parts of the product with large variability in a darker color. From the image in Figure 14(A), it can be seen that in the product shown in Figure 14(B), large variations, i.e., large manufacturing errors, occur in the area enclosed by the dashed line.

[0093] RGBHi images, which use 256 gradations for height information (Hi), improve judgment accuracy when there is a correlation between the RGB image and the height information (Hi). Furthermore, combining the RGB image and height information (Hi) into a single image speeds up the judgment process. On the other hand, in RGBHi images combined with a mask image converted to 0 or 1 depending on whether the reference height is met, abnormal height areas blend in with the background color and are displayed differently from normal RGB images, making visual judgment easier for operators.

[0094] [Product Review] In one embodiment, after performing the machine learning described above, the product, which is the object under inspection, can be evaluated by the quality control system 200. Refer to Figure 15. Figure 15 is a flowchart showing a product quality control method according to one embodiment of the present invention. The imaging unit 111 captures an image of the product, which is the object under inspection, and the autoencoder 220 takes in the image data of the object under inspection as the input layer 221 (step S150). The autoencoder 220 compresses and restores the image data of the object under inspection. At this time, even if the object under inspection has a defect, the machine learning-trained autoencoder 220 generates restored image data 30 that does not have a defect. The file generation unit 230 compares the image data of the object under inspection input to the autoencoder 220 with the restored image data 30 and generates difference data that includes abnormal values ​​indicating a defect in the object under inspection (step S160). The determination unit 240 determines that the object under inspection has a defect from the abnormal values ​​included in the difference data. Based on the determination result by the determination unit 240, the visualization processing unit 250 generates an image that shows the presence of a defect in the image data of the object under inspection (step S170).

[0095] In one embodiment, the quality control system 200 may generate difference data by comparing the image data of the object under test with the reconstructed image data 30 generated during training, without using the autoencoder 220. In this process, the calculation processing by the autoencoder 220 is omitted, so although the accuracy of the difference data may decrease slightly, the overall processing time can be reduced.

[0096] In this embodiment, the autoencoder 220 generates difference data between the image data of the object under test and the reconstructed image data, and determines damage to the object from the abnormal values ​​contained in the difference data. However, as a modified example, the autoencoder 220 can also determine damage to the object when it is unable to reconstruct the image data generated from the image data of the object under test.

[0097] Figure 16(A) is a schematic diagram showing an RGBHi image generated by the visualization processing unit 250, and Figure 16(B) is a schematic diagram (RGBa) of a top view of the object with a defect. For example, in Figure 16(A), in order to make it easier for the user to see that there is a defect in the object, the visualization processing unit 250 colors the area where an abnormal value was detected (a diagonal line is added in Figure 16(A)) and generates an image by combining it with the RGBHi image data of the object. From the image in Figure 16(A), it can be recognized that the object shown in Figure 16(B) has a defect in the area enclosed by the dashed line. Alternatively, the visualization processing unit 250 may draw lines in the area adjacent to the area where an abnormal value was detected, displaying a circular or elliptical shape surrounding the defect in the object on the image and displaying it on the display unit 271.

[0098] [Comparison of design data and product] In one embodiment, the manufacturing error between the design data and the product can be evaluated by performing machine learning using image data of a good product and then evaluating the product's design data. Specifically, the file generation unit 230 compares the design data input to the autoencoder 220 with the reconstructed image data 30 and generates difference data that includes abnormal values ​​indicating the difference between the design data and the reconstructed image data 30. The determination unit 240 determines from the abnormal values ​​included in the difference data that the product deviates (or differs) from the design data. Based on the determination result by the determination unit 240, the visualization processing unit 250 can generate an image that shows the product deviates from the design data.

[0099] To make the aforementioned outliers and variations easier for users to see, the determination unit 240 may weight the portion containing the outlier when it detects an outlier in the differential data, and the visualization processing unit 250 may generate an image that includes the weighted portion. In other words, the visualization processing unit 250 can generate an image that emphasizes the weighted portion.

[0100] [Resetting the interval process] As described above, the size of the interval used by the file generation unit 230 can be reset. If a second interval is provided as an initial value, the determination unit 240 resets a third interval having a different size from the second interval based on the range of the second interval containing the difference data that was determined to be an abnormal value, or the range of the second interval containing the difference data that was determined to have a large variation. The file generation unit 230 then processes the difference data for each more appropriate third interval and generates a file containing the mean and variance values ​​of the difference data.

[0101] [Quality Control Program] In one embodiment, a program for performing the above-described processes can be provided. Alternatively, in one embodiment, it can be provided as a recording medium storing the program. This will be explained with reference to Figures 2 to 16.

[0102] This program causes the quality control system to generate difference data that includes the difference between multiple training image data and the reconstructed image data generated from the training image data (step S110). This program causes the autoencoder 220 to input multiple training image data 10 (step S11).

[0103] This program instructs the autoencoder 220 to compress the training image data 10 input to the input layer 221 using the encoder 227, thereby generating an intermediate layer 223. The program also instructs the decoder 229 to decompress the image data and output it to the output layer 225, generating reconstructed image data 30 (step S115). This program then instructs the autoencoder 220 to extract features from the training image data 10 pixel by pixel through this process (step S113).

[0104] More specifically, as described above, this program compresses and decompresses the four components of each pixel of the training image data 10: red (R), green (G), blue (B), and blue (B) plus height information Hi, to generate difference data. This program then processes the obtained difference data in the next step S120 or stores it in the storage unit 280.

[0105] In this embodiment, by performing machine learning by repeating step S110 using training image data, the program can bring the value of the difference data between the training image data and the reconstructed image data closer to zero.

[0106] This program causes the quality control system 200 to calculate the mean and variance of the difference data for each predetermined interval and generate a file (step S120). This program causes the file generation unit 230 to calculate the mean and variance for each interval from the difference data generated by the autoencoder 220. As described above, this program causes the file generation unit 230 to calculate the mean μ for each color component for each interval from the obtained difference data. This program also causes the file generation unit 230 to calculate the variance ρ for each color component for each interval. Furthermore, this program causes the file generation unit 230 to calculate the covariance σ for each component for each interval (step S121).

[0107] This program causes the file generation unit 230 to output the calculation results as a normal distribution (step S123). The obtained file may be processed in the next step S130, or it may be stored in the storage unit 280.

[0108] This program causes the quality control system 200 to generate an image based on the data of the generated file (process S130). Since the program calculates the mean and variance of the difference data for each predetermined interval and generates a file, the program may set the intervals considering the balance between accuracy and processing time. The intervals can also be set by a person via the input unit 110. Alternatively, if a second interval is provided as an initial value, the program causes the judgment unit 240 to reset a third interval with a different size from the second interval based on the range of the second interval containing the difference data that was determined to be an abnormal value, and causes the file generation unit 230 to perform calculation processing using the difference data for each more appropriate third interval, thereby generating a file containing the mean and variance of the difference data. By repeating this resetting process (refinement), the optimal intervals can be set.

[0109] This program aggregates the data obtained for each interval into a single image in the visualization processing unit 250 (step S131). This program combines the mean or variance values ​​output by the file generation unit 230 into the visualization processing unit 250 based on the pixel position information of the difference data. This program outputs the mean or variance values ​​for each component as an image in the visualization processing unit 250 (step S133).

[0110] This program allows the user to visually perceive variations in the characteristics of the object being examined by displaying the image output by the visualization processing unit 250 on the display unit 271. Alternatively, the visualization processing unit 250 may store the image in the storage unit 280, or output it to a printer via the output unit 270. Furthermore, this program may have the communication unit 290 transmit the image to a server or network drive.

[0111] [Manufacturing Error Evaluation Program] In one embodiment, when machine learning is performed in the quality control system 200 using non-defective products (good products or acceptable products) as training image data 10, users can visually recognize the variability in the products. Specifically, the manufacturing error evaluation program can cause the visualization processing unit 250 to generate an RGBHi image by combining it with the height information Hi contained in the file. Figure 14(A) is a schematic diagram showing the RGBHi image generated by the visualization processing unit 250, and Figure 14(B) is a schematic diagram (RGBa) of the top view of the product. In one embodiment, the manufacturing error evaluation program can display the RGBHi image as an image in which the height information Hi is represented by data values ​​from 0 to 255 (256 gradations). Alternatively, the manufacturing error evaluation program can display the RGBHi image as a mask image converted to 0 and 1 depending on whether or not a standard height is met. As an example, Figure 14(A) shows an RGBHi image in which the manufacturing error evaluation program evaluates the height variability in three stages, and parts of the product with large variability are displayed in darker colors. From the image in Figure 14(A), it can be seen that in the product shown in Figure 14(B), large variations, i.e., large manufacturing errors, occur in the area enclosed by the dashed line.

[0112] [Product Evaluation Program] In one embodiment, the product evaluation program performs the machine learning described above and then has the product, which is the object under test, evaluated by the quality control system 200. The product evaluation program has the imaging unit 111 capture an image of the product under test and has the autoencoder 220 take in the image data of the object under test as the input layer 121 (step S150). The product evaluation program has the autoencoder 220 compress and then decompress the image data of the object under test. At this time, even if the object under test has a defect, the machine learning-trained autoencoder 220 generates decompressed image data 30 that does not have a defect. The product evaluation program has the file generation unit 230 compare the image data of the object under test input to the autoencoder 220 with the decompressed image data 30 and generate difference data that includes abnormal values ​​indicating defects in the object under test (step S160). The product evaluation program has the determination unit 240 determine that the object under test has a defect based on the abnormal values ​​included in the difference data. Based on the determination result from the determination unit 240, the product evaluation program causes the visualization processing unit 250 to generate an image indicating the presence of a defect in the image data of the object being examined (step S170).

[0113] In one embodiment, the product evaluation program may be configured to generate difference data by comparing the image data of the object under test with the reconstructed image data 30 generated during training, without using the autoencoder 220. In this process, since the calculation processing by the autoencoder 220 is omitted, the accuracy of the difference data may decrease slightly, but the overall processing time can be reduced.

[0114] In this embodiment, the product evaluation program causes the autoencoder 220 to generate difference data between the image data of the object under test and the reconstructed image data, and then determines whether there are any defects in the object under test from the abnormal values ​​contained in the difference data. However, as a modification, the autoencoder 220 can also determine whether there are any defects in the object under test if it is not possible to reconstruct the image data when generating the reconstructed image data from the image data of the object under test.

[0115] As an example, in Figure 16(A), the product evaluation program instructs the visualization processing unit 250 to color-code (add diagonal lines in Figure 16(A)) the area where an abnormal value was detected, in order to make it easier for the user to see that there is a defect in the object being tested. This color-codes the area and combines it with the RGBHi image data of the object being tested to generate an image. From the image in Figure 16(A), it can be recognized that the object being tested, as shown in Figure 16(B), has a defect in the area enclosed by the dashed line. The product evaluation program can also instruct the visualization processing unit 250 to draw lines in the area adjacent to the area where an abnormal value was detected, displaying a circular or elliptical shape surrounding the defect in the object being tested in the image, and displaying it on the display unit 271.

[0116] [Design data and product comparison program] In one embodiment, the manufacturing error between the design data and the product can be evaluated by performing machine learning using image data of a good product and then evaluating the product's design data. The comparison program causes the file generation unit 230 to compare the design data input to the autoencoder 220 with the reconstructed image data 30 and generate difference data that includes abnormal values ​​indicating the difference between the design data and the reconstructed image data 30. The comparison program causes the determination unit 240 to determine from the abnormal values ​​included in the difference data that the product deviates (or differs) from the design data. Based on the determination result by the determination unit 240, the comparison program causes the visualization processing unit 250 to generate an image indicating that the product deviates from the design data.

[0117] To make the aforementioned abnormal values ​​and variations easier for users to see, the program according to one embodiment can, when the determination unit 240 detects an abnormal value in the differential data, weight the portion containing the abnormal value, and cause the visualization processing unit 250 to generate an image including the weighted portion. In other words, the comparison program can cause the visualization processing unit 250 to generate an image that emphasizes the weighted portion.

[0118] [Program for resetting the interval] As described above, the size of the interval used by the autoencoder 220 can be reset. When a second interval is provided as the initial value, the reset processing program causes the determination unit 240 to reset a third interval having a different size from the second interval, based on the range of the second interval containing the difference data determined to be an abnormal value, or the range of the second interval containing the difference data determined to have a large variation. The program then causes the autoencoder 220 to calculate difference data for each more appropriate third interval, and causes the file generation unit 230 to generate a file containing the mean and variance values ​​of the difference data.

[0119] Although one embodiment of the present invention has been described above with reference to the drawings, the present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the invention. For example, a quality control system or quality control program of this embodiment, to which a person skilled in the art has added, deleted, or modified components as appropriate, is also included in the scope of the present invention as long as it retains the gist of the present invention. Furthermore, the above-described embodiments can be combined as appropriate as long as they do not contradict each other, and technical matters common to each embodiment are included in each embodiment even if not explicitly described.

[0120] Any effects or benefits other than those brought about by the embodiments described above, if they are clear from the description herein or easily predictable to a person skilled in the art, are naturally considered to be brought about by the present invention. [Explanation of Symbols]

[0121] 100 Three-dimensional image data generation device, 110 Input unit, 111 Imaging unit, 120 Sub-pixel data extraction unit, 130 Height information extraction unit, 140 Image data integration unit, 160 Control unit, 170 Output unit, 171 Display unit, 180 Storage unit, 190 Communication unit, 200 Quality control system, 220 Autoencoder, 230 File generation unit, 240 Judgment unit, 250 Visualization processing unit, 260 Control unit, 270 Output unit, 271 Display unit, 280 Storage unit, 290 Communication unit

Claims

1. A sub-pixel data extraction unit that extracts first, second, and third sub-pixel data from one or more captured first image data, A height information extraction unit that extracts height information from one or more first image data, An image data integration unit that generates one or more second image data in which the first, second and third sub-pixel data and the height information are stored in one row for each pixel, An autoencoder that generates difference data including the difference between training image data composed of one or more of the second image data and reconstructed image data generated from the training image data, A quality control system comprising a file generation unit that calculates the average value and variance of the difference data for each predetermined interval and generates a file.

2. The autoencoder extracts features from each of the multiple training image data for each pixel, and generates reconstructed image data based on the extracted features. The quality control system according to claim 1, wherein the file generation unit applies a Gaussian mixture model to calculate the mean and variance values ​​of the difference data.

3. The system further comprises a determination unit that determines whether or not there are abnormal values ​​in the differential data, The quality control system according to claim 1, wherein the determination unit determines whether or not there are abnormal values ​​in the difference data generated from the third image data of the object being examined.

4. The quality management system according to claim 3, wherein the determination unit sets a second interval having a different size from the first interval based on the range of the first interval including the difference data determined to be an abnormal value, and calculates the mean value and variance value of the difference data for each of the second intervals.

5. The system further comprises a determination unit that determines whether or not there are abnormal values ​​in the differential data, The quality management system according to claim 1, wherein the determination unit compares the design data with the difference data to determine whether or not there are any abnormal values ​​in the difference data.

6. The design data has a predetermined tolerance range for each section. The quality control system according to claim 5, wherein the determination unit compares the range of the tolerance value with the difference data, determines whether there are any abnormal values ​​in the difference data, and weights the interval containing the abnormal values.

7. The first sub-pixel data is S １ , the second sub-pixel data S ２ , the third sub-pixel data S ３ , and when the height information is set to Hi, the second image data is (S １ , S ２ , S ３ A quality management system according to any one of claims 1 to 6, having a data arrangement of ,Hi.

8. The sub-pixel data extraction unit generates a fourth image data having a data arrangement of (R, G, B, a), where the first sub-pixel data is R, the second sub-pixel data is G, the third sub-pixel data is B, and transparency is a. The quality control system according to any one of claims 1 to 6, wherein the image data integration unit, when the height information is set to Hi, rewrites the a of the fourth image data to Hi to generate the second image data.

9. The computer is made to extract first, second, and third sub-pixel data from one or more captured first image data. The computer is made to extract height information from one or more of the first image data. The computer is made to generate one or more second image data sets, each containing the first, second, and third sub-pixel data and the height information, stored in one row for each pixel. The computer is made to generate difference data including the difference between training image data composed of one or more of the second image data and reconstructed image data generated from the training image data. A quality control program comprising causing the computer to calculate the average value and variance of the difference data for each predetermined interval and generating a file.

10. The computer extracts features from each pixel of the multiple training image data, and generates reconstructed image data based on the extracted features. The quality control program according to claim 9, wherein a Gaussian mixture model is applied to cause the computer to calculate the mean and variance values ​​of the difference data.

11. The computer is instructed to determine whether or not there are any abnormal values ​​in the differential data generated from the third image data of the object being examined. The quality control program according to claim 9, which causes the computer to generate a second image including the abnormal value based on the first image data of the object to be inspected and the judgment result.

12. The quality control program according to claim 11, wherein the computer is instructed to set a second interval having a different size from the first interval based on the range of a first interval containing the difference data determined to be an abnormal value, and the computer is instructed to calculate the mean and variance values ​​of the difference data for each of the second intervals.

13. A quality control program according to claim 9, which compares design data with the difference data and causes the computer to determine whether or not there are any abnormal values ​​in the difference data.

14. The design data has a predetermined tolerance range for each section. The quality control program according to claim 13, wherein the computer compares the range of the tolerance value with the difference data, determines whether there are any abnormal values ​​in the difference data, and weights the interval containing the abnormal values.

15. The first sub-pixel data is S １ The second sub-pixel data is S ２ The third sub-pixel data is S ３ When the height information is Hi, the second image data has a data arrangement of (S １ , S ２ , S ３ , Hi). The quality management program according to any one of claims 9 to 14

16. The computer is made to generate a third image data having a data arrangement of (R, G, B, a), where the first sub-pixel data is R, the second sub-pixel data is G, the third sub-pixel data is B, and the transparency is a. A quality control program according to any one of claims 9 to 14, wherein when the height information is set to Hi, the a of the third image data is rewritten to Hi, and the computer is instructed to generate the second image data.

Images