Information processing method, information processing device, information processing system, and program

Abstract

An information processing method according to one aspect of the present disclosure comprises: calculating a first color vector on the basis of two or more pixel values indicating the color of a pixel in a first image, and calculating a second color vector on the basis of two or more pixel values indicating the color of a pixel in a second image different from the first image (S10); calculating the angle formed by the first color vector and the second color vector (S20); and searching for a corresponding point in the first image and the second image on the basis of the calculated angle (S30).

Description

Information processing method, information processing device, information processing system, and program

[0001] The present disclosure relates to an information processing method, an information processing device, an information processing system, and a program.

[0002] Conventionally, there is a technique for measuring the three-dimensional shape of an object (subject) based on images generated by two image capture devices capturing the object from different positions. The three-dimensional shape is measured, for example, by triangulation using the imaging position shift (parallax) between the two images. For example, corresponding points that indicate the same position in the two images are searched for, and the parallax of the two image capture devices is calculated based on the searched corresponding points.

[0003] Patent Document 1 discloses a three-dimensional shape measurement device that uses triangulation to calculate three-dimensional shape information based on hue data for each pixel of an image obtained by capturing the reflected light of a pattern light whose hue changes continuously from an object to be measured.

[0004] Patent No. 5743433

[0005] If the object contains multiple components with different reflectances, the light irradiated onto the object and the light returning from the object may have the same hue even if they have different colors. In such cases, corresponding points may not be found correctly, and the shape of the object may not be accurately determined.

[0006] The present disclosure provides an information processing method and the like that can accurately search for corresponding points in multiple images.

[0007] An information processing method according to one aspect of the present disclosure calculates a first color vector based on two or more pixel values ​​indicating the color of a pixel in a first image, calculates a second color vector based on two or more pixel values ​​indicating the color of a pixel in a second image different from the first image, calculates the angle between the first color vector and the second color vector, and searches for corresponding points in the first image and the second image based on the calculated angle.

[0008] In addition, an information processing device according to one aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to calculate a first color vector based on two or more pixel values ​​indicating the color of a pixel in a first image, and calculates a second color vector based on two or more pixel values ​​indicating the color of a pixel in a second image different from the first image, calculates the angle between the first color vector and the second color vector, and searches for corresponding points in the first image and the second image based on the calculated angle.

[0009] In addition, an information processing system according to one aspect of the present disclosure includes the information processing device described above, a first imaging device that generates the first image by imaging an object in two or more wavelength bands, and a second imaging device that generates the second image by imaging the object in the two or more wavelength bands from a position different from that of the first imaging device.

[0010] Furthermore, a program according to one aspect of the present disclosure is a program for causing a computer to execute the information processing method.

[0011] According to an information processing method and the like according to an aspect of the present disclosure, corresponding points in a plurality of images can be searched for with high accuracy.

[0012] Fig. 1 is a block diagram showing the configuration of a shape measurement system according to an embodiment. Fig. 2 is a diagram for explaining a specific example of a process for calculating an angle formed by two color vectors according to an embodiment. Fig. 3 is a diagram for explaining cost values ​​according to an embodiment. Fig. 4 is a flowchart showing a shape measurement method according to an embodiment. Fig. 5 is a block diagram showing an information processing device according to an embodiment. Fig. 6 is a flowchart showing an information processing method according to an embodiment.

[0013] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0014] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in the independent claims are described as optional components.

[0015] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not refer to the number or order of components unless otherwise specified, but are used for the purpose of avoiding confusion and distinguishing between components of the same type.

[0016] (Embodiment) [Configuration] FIG. 1 is a block diagram showing the configuration of a shape measurement system 10 according to an embodiment.

[0017] The shape measurement system 10 is a system for measuring the shape of an object such as a part. Specifically, the shape measurement system 10 captures images of the object using multiple imaging devices, such as stereo cameras, that capture images of the object from different positions, and measures the three-dimensional shape of the object based on images of the object captured (generated) by each imaging device. The shape measurement system 10 is an example of an information processing system.

[0018] The shape measurement system 10 includes a shape measurement device 100 , a first imaging device 200 , a second imaging device 210 , and an irradiation device 220 .

[0019] The shape measurement device 100 is a computer that measures the shape of an object based on multiple images generated by capturing images of the object using the first imaging device 200 and the second imaging device 210. For example, the shape measurement device 100 calculates the three-dimensional shape of the object (information indicating the three-dimensional shape) using triangulation from the multiple images acquired from the first imaging device 200 and the second imaging device 210. The shape measurement device 100 is an example of an information processing device.

[0020] The shape measurement device 100 is realized by a computer including, for example, a communication interface for communicating with the first imaging device 200, the second imaging device 210, the irradiation device 220, etc., provided in the shape measurement system 10, a non-volatile memory for storing programs executed by each processing unit, a volatile memory which is a temporary storage area for executing the programs, an input / output port for transmitting and receiving signals, and a processor such as a CPU (Central Processing Unit) that executes the programs. The communication interface may be realized by a connector to which a communication line is connected for wired communication, or by a wireless communication circuit for wireless communication.

[0021] The first imaging device 200 is a camera that generates a first image by capturing an image of an object in two or more wavelength bands. That is, the first imaging device 200 captures an image of the object by detecting light reflected from the object in two or more wavelength bands. The first image is an image generated by the first imaging device 200 capturing an image of the object.

[0022] The second imaging device 210 is a camera that generates a second image by capturing an image of an object in two or more wavelength bands. That is, the second imaging device 210 captures an image of the object by detecting light reflected from the object in two or more wavelength bands. The second image is an image generated by the second imaging device 210 capturing an image of the object.

[0023] For example, the first imaging device 200 and the second imaging device 210 capture images of the same object from different positions. As a result, parallax occurs between the first image and the second image. The shape measurement device 100 calculates this parallax and measures the shape of the object using the calculated parallax. Specifically, the shape measurement device 100 measures the three-dimensional shape of the object using the first image, the second image, the parallax, and the distance between the first imaging device 200 and the second imaging device 210.

[0024] The first imaging device 200 and the second imaging device 210 capture an image of an object by detecting light in the same wavelength band. Specifically, the first imaging device 200 and the second imaging device 210 each capture an image of an object by detecting light in two or more wavelength bands different from each other.

[0025] The first imaging device 200 and the second imaging device 210 function as, for example, a stereo camera. The first imaging device 200 and the second imaging device 210 are arranged side by side so that their imaging directions are parallel to each other, for example.

[0026] The two or more wavelength bands may be arbitrarily determined and are not particularly limited. For example, the two or more wavelength bands may be wavelength bands corresponding to the three colors RGB (Red, Green, Blue). For example, the red wavelength band is 640 nm to 770 nm, the green wavelength band is 490 nm to 550 nm, and the blue wavelength band is 430 nm to 490 nm.

[0027] The two or more wavelength bands may be wavelength bands corresponding to RGB, or may be wavelength bands corresponding to any color, such as Y (Yellow), M (Magenta), C (Cyan), UV (Ultraviolet), or IR (Infrared). The bandwidth of the wavelength bands may be determined arbitrarily. For example, the first imaging device 200 and the second imaging device 210 may detect two or more light beams having different wavelength bands (e.g., peak wavelengths) and a bandwidth of 50 nm to 150 nm. For example, the first imaging device 200 and the second imaging device 210 may detect two or more light beams having different wavelengths and a single wavelength. The two or more wavelength bands may be wavelength bands corresponding to R and B, UV and IR, or UV, RGB, and IR, or may be realized by any combination of the wavelength bands.

[0028] The illumination device 220 is a light source that illuminates the object. Specifically, the illumination device 220 illuminates the object with light of two or more wavelength bands. More specifically, the illumination device 220 illuminates the object with light including two or more wavelength bands that are detected by the first imaging device 200 and the second imaging device 210. The first imaging device 200 and the second imaging device 210 capture an image of the object by, for example, detecting light that is illuminated onto the object by the illumination device 220 and reflected by the object.

[0029] The light irradiated by the irradiation device 220 may be light including two or more wavelength bands detected by the first imaging device 200 and the second imaging device 210. For example, the light irradiated by the irradiation device 220 may be light including only two or more wavelength bands detected by the first imaging device 200 and the second imaging device 210. Alternatively, for example, the light irradiated by the irradiation device 220 may be white light when the first imaging device 200 and the second imaging device 210 detect light corresponding to RGB.

[0030] Furthermore, for example, the irradiation device 220 may irradiate the object with pattern light. For example, the pattern light is dot pattern light.

[0031] The pattern of the pattern light does not have to be a dot pattern, but may be, for example, a random pattern or a stripe pattern.

[0032] The irradiation device 220 is realized by, for example, an LED (Light Emitting Diode). The irradiation device 220 may be realized by any type of light source, such as an LD (Laser Diode), a halogen lamp, or a fluorescent lamp.

[0033] The number of imaging devices included in the shape measurement system 10 may be more than one, and may be three or more.

[0034] Next, a specific configuration of the shape measurement device 100 will be described.

[0035] The shape measurement device 100 includes an acquisition unit 110 , a first calculation unit 120 , a search unit 130 , a second calculation unit 140 , a measurement unit 150 , an output unit 160 , and a storage unit 170 .

[0036] The acquisition unit 110 is a processing unit that acquires a first image generated by the first imaging device 200 capturing an image of an object in two or more wavelength bands, and a second image generated by the second imaging device 210 capturing an image of the object in two or more wavelength bands from a position different from that of the first imaging device 200. The acquisition unit 110, for example, causes the illumination device 220 to irradiate light onto the object, and then causes the first imaging device 200 and the second imaging device 210 to capture images of the object, and then acquires the first image and the second image from the first imaging device 200 and the second imaging device 210.

[0037] The first calculation unit 120 is a processing unit that calculates a color vector based on two or more pixel values ​​that indicate the colors of pixels in the image acquired by the acquisition unit 110. Specifically, the first calculation unit 120 calculates a first color vector based on two or more pixel values ​​that indicate the colors of pixels in the first image. The first calculation unit 120 also calculates a second color vector based on two or more pixel values ​​that indicate the colors of pixels in the second image. The first color vector is a color vector calculated from the pixel values ​​of pixels included in the first image. The second color vector is a color vector calculated from the pixel values ​​of pixels included in the second image.

[0038] A pixel value is a value that indicates the color of a pixel. The two or more pixel values ​​are, for example, values ​​that indicate colors corresponding to two or more wavelength bands of light detected by each of the first imaging device 200 and the second imaging device 210. The two or more pixel values ​​are, for example, values ​​that indicate each of the three colors of RGB. In other words, the value that indicates the R color, the value that indicates the G color, and the value that indicates the B color of a pixel are each an example of a pixel value.

[0039] A color vector is a vector that represents the color of a pixel. For example, if two or more pixel values ​​are the values ​​of each of the three colors RGB, the color vector is represented as a three-dimensional vector such as (R pixel value, G pixel value, B pixel value). In this way, the color vector calculated by the first calculation unit 120 corresponds to two or more pixel values ​​and is a vector with two or more dimensions.

[0040] The two or more pixel values ​​may be any color value such as Y, M, C, UV, or IR, in addition to RGB, so as to correspond to two or more wavelength bands.

[0041] For example, the first calculation section 120 calculates a color vector corresponding to each of a plurality of pixels included in the first image and the second image based on the pixel value of the pixel.

[0042] The first calculation unit 120 also calculates the angle between the first color vector and the second color vector. For example, the first calculation unit 120 combines each of the multiple first color vectors calculated from the first image with multiple second color vectors calculated from the second image, and calculates the angle between the combined first and second color vectors.

[0043] The combination of the first color vectors and the second color vectors may be arbitrary. For example, the first color vectors and the second color vectors may be combined in a round-robin manner. Alternatively, the combination of the first color vectors and the second color vectors whose angles are calculated may be determined based on the position of the pixel in the image.

[0044] The search unit 130 is a processing unit that searches for corresponding points in the first image and the second image based on the calculated angle. Specifically, the search unit 130 searches for corresponding points in the first image and the second image based on a cost value that is based on the calculated angle. More specifically, the search unit 130 calculates a cost value based on the angle between the first color vector and the second color vector, and determines corresponding points based on the calculated cost value.

[0045] A corresponding point is a point (e.g., a pixel) that indicates the same position (e.g., the same part of the object) in the first image and the second image. The search unit 130 identifies the pixel in the first image and the pixel in the second image that are the corresponding point, for example, based on the multiple angles calculated by the first calculation unit 120.

[0046] For example, in searching for corresponding points, the search unit 130 searches for corresponding points using a similarity based on the angle calculated by the first calculation unit 120. For example, the first calculation unit 120 further calculates a cosine similarity based on the calculated angle. In addition, for example, in searching for corresponding points, the search unit 130 searches for corresponding points based on the cosine similarity calculated by the first calculation unit 120. Specifically, in calculating the cosine similarity, the first calculation unit 120 calculates a plurality of cosine similarities based on a plurality of first color vectors calculated from two or more pixel values ​​of a plurality of pixels located in a first range in the first image and a plurality of second color vectors calculated from two or more pixel values ​​of a plurality of pixels located in a second range in the second image.

[0047] For example, the first calculation section 120 combines a plurality of first color vectors and a plurality of second color vectors in a round-robin manner, calculates an angle for each combination, and calculates the cosine similarity based on the calculated angles.

[0048] Furthermore, for example, the first calculation unit 120 further calculates a cost value based on the calculated multiple cosine similarities. For example, the first calculation unit 120 calculates the cost value by adding up the calculated multiple cosine similarities. Furthermore, for example, in searching for corresponding points, the search unit 130 searches for corresponding points based on the cost value calculated by the first calculation unit 120.

[0049] For example, in calculating the cost values, the first calculation unit 120 repeatedly moves at least one of the position of the first range in the first image and the position of the second range in the second image to repeatedly calculate the cost values. In other words, the first calculation unit 120 repeatedly changes at least one of the position of the first range and the position of the second range to calculate the cost value for each combination of the position of the first range and the position of the second range.

[0050] For example, in calculating the cost value, the first calculator 120 may repeatedly move one of the position of the first range in the first image and the position of the second range in the second image along the epipolar line to repeatedly calculate the cost value. For example, in calculating the cost value, the first calculator 120 may repeatedly move the position of the second range along one of two orthogonal axes in the image coordinate system to repeatedly calculate the cost value. For example, in calculating the cost value, the first calculator 120 may repeatedly move the position of the second range along the horizontal axis in the image coordinate system (e.g., the horizontal axis in the second image shown in FIG. 2B, which will be described later) to repeatedly calculate the cost value.

[0051] Furthermore, for example, in the search for corresponding points, the search unit 130 identifies a first range and a second range that result in the highest cost value calculated by the first calculation unit 120. That is, for example, in the search for corresponding points, the search unit 130 identifies a combination of the first range and the second range that results in the highest cost value among the multiple cost values ​​calculated by the first calculation unit 120. In this manner, for example, the first calculation unit 120 determines, as in block matching, a rectangular range (also referred to as a kernel) that includes a pixel in the first image and pixels surrounding that pixel as a first region, and identifies a second range from the second image that results in the highest cost value when combined with the first range. Furthermore, for example, the search unit 130 further determines that multiple pixels located in the identified first range and multiple pixels located in the identified second range contain corresponding points.

[0052] The first range and the second range include the same number of pixels, for example, four in each row direction of the image and four in each column direction of the image, etc. The search unit 130 determines that, for example, of the plurality of pixels located in the specified first range and the plurality of pixels located in the specified second range, pixels that correspond to the same positions in each range are corresponding points.

[0053] The size of the first range and the second range (the number of pixels included in these ranges) may be determined arbitrarily and is not particularly limited. For example, the number of pixels included in the first range and the second range may be 3×3, 4×4, or 5×5, where (the number of pixels in the row direction in the image)×(the number of pixels in the column direction in the image).

[0054] The second calculation unit 140 is a processing unit that calculates the disparity between the first image and the second image based on the corresponding points searched for by the search unit 130. For example, the second calculation unit 140 calculates the disparity by matching (stereo matching) each part (each pixel) of the first image and the second image. For example, the second calculation unit 140 calculates the difference (amount of positional deviation) between corresponding points, which are the positions of corresponding parts between the first image and the second image, as the disparity.

[0055] The measurement unit 150 is a processing unit that measures the shape of the object based on the calculated parallax. For example, the measurement unit 150 measures the three-dimensional shape of the object using triangulation based on the first image, the second image, the parallax calculated by the second calculation unit 140, and the distance between the first image capture device 200 and the second image capture device 210. For example, the measurement unit 150 generates a depth image in which the object is reflected, which shows the measurement result of the shape of the object.

[0056] The output unit 160 is a processing unit that outputs the measurement results of the measurement unit 150. For example, the output unit 160 outputs the measurement results of the shape of the object to a display (not shown) or the like, thereby displaying a depth image showing the shape of the object on the display.

[0057] The shape measurement device 100 may be equipped with the display.

[0058] Each processing unit, such as the acquisition unit 110, the first calculation unit 120, the search unit 130, the second calculation unit 140, the measurement unit 150, and the output unit 160, is realized, for example, by a processor and a memory in which a control program executed by the processor is stored.

[0059] The storage unit 170 is a storage device that stores various types of information. The storage unit 170 stores, for example, information indicating the distance between the first imaging device 200 and the second imaging device 210, information indicating the kernel size, etc. The storage unit 170 is realized by, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0060] [Specific Example] Next, a specific process for calculating a color vector and searching for corresponding points will be described. In the specific example described below, the first image capturing device 200 and the second image capturing device 210 capture an image of the object 300 by detecting light in wavelength bands corresponding to RGB, respectively.

[0061] FIG. 2 is a diagram for explaining a specific example of the process of calculating the angle between two color vectors according to the embodiment.

[0062] As shown in FIG. 2A, for example, the first imaging device 200 and the second imaging device 210 are arranged side by side above the object 300, and capture images of the object 300 with their imaging directions facing downward.

[0063] First, the first calculation unit 120 sets a first region in the first image and a second region in the second image. In this example, the first calculation unit 120 sets the first region and the second region to include 16 (4×4) pixels.

[0064] The i-th kernel shown in (b) and (c) of Figure 2 is a specific example of a set region (first region or second region). i is, for example, an integer greater than or equal to 1. For example, the first region set in the upper left corner of the first image is defined as the first kernel, and the first region shifted one pixel to the right from there is defined as the second kernel. The first region shifted one pixel further to the right from there is defined as the third kernel. Furthermore, for example, if the first region is shifted to the right edge of the first image and defined as the k-th kernel, the first region is then returned to the left edge and further shifted one pixel downward from there to define the k+1-th kernel. The second region is also positioned like the i-th kernel, similar to the first region. For example, if the first image and the second image have the same number of pixels and both the first region and the second region are located in the i-th kernel, the positions of the first region and the second region in the image coordinate system are the same (coordinates).

[0065] In the above, the kernel is moved by one pixel at a time, but it may be moved by multiple pixels at a time. Also, the kernel may be moved so that the same pixel is not included.

[0066] For example, the first calculation unit 120 calculates 16 first color vectors corresponding to 16 pixels included in the first kernel in the first image and 16 second color vectors corresponding to 16 pixels included in the first kernel in the second image. Next, the first calculation unit 120 combines pixels at the same positions in the first and second images and calculates the angle between the first color vector and the second color vector corresponding to each combined pixel. The first calculation unit 120 also calculates cosine similarities corresponding to each of the 16 angles calculated in this manner. The first calculation unit 120 also calculates a cost value by adding up the 16 cosine similarities calculated in this manner.

[0067] Next, the first calculation unit 120 calculates 16 first color vectors corresponding to the 16 pixels included in the first kernel in the first image and 16 second color vectors corresponding to the 16 pixels included in the second kernel in the second image. The first calculation unit 120 also calculates the angles, cosine similarities, and cost values ​​as described above.

[0068] In this manner, the first calculation unit 120 repeatedly moves the position of the kernel (second region) in the second image to repeatedly calculate the cost value. For example, the first calculation unit 120 repeatedly moves the position of the kernel in the second image from one end of the second image to the other end located opposite the one end on the horizontal axis so that an arbitrary position in the kernel is located on the horizontal axis of the image coordinate system. In this manner, the first calculation unit 120 calculates multiple cost values. The arbitrary position may be, for example, the position of an arbitrary pixel in the kernel, the center of the kernel, or any other arbitrary position. Furthermore, for example, the first calculation unit 120 may repeatedly move the position of the kernel in the second image rightward from the left end to the right end of the second image in FIG. 2C. Of course, for example, the first calculation unit 120 may repeatedly move the position of the kernel in the second image leftward from the right edge to the left edge of the second image in (c) of Figure 2, or may repeatedly move it downward from the top edge to the bottom edge of the second image, or may repeatedly move it upward from the bottom edge to the top edge of the second image, or may repeatedly move it in any direction.

[0069] Next, the search unit 130 identifies the ordinal number of the kernel in the second image that has the largest cost value among the multiple cost values ​​calculated by the first calculation unit 120. If the number of the kernel in the second image identified here is m, for example, the search unit 130 determines that multiple pixels included in the first kernel in the first image and multiple pixels included in the m-th kernel in the second image are corresponding points.

[0070] Furthermore, for example, after the first calculation unit 120 moves the kernel in the second image to the right end of the second image (the right end in the row direction in the image) and calculates the cost value, it calculates 16 first color vectors corresponding to the 16 pixels included in the second kernel in the first image and 16 second color vectors corresponding to the 16 pixels included in the first kernel in the second image.

[0071] In this way, the first calculation unit 120 repeatedly moves the position of the kernel (first region) in the first image and repeatedly calculates the cost value. The direction in which the kernel in the first image is moved is, for example, the same as that of the kernel in the second image, but may be different or may be arbitrarily determined.

[0072] Similarly to the above, for example, the first calculation unit 120 fixes the position of the second kernel in the first image and repeatedly moves the position of the kernel in the second image to repeatedly calculate cost values. Similarly to the above, the search unit 130 also searches for corresponding points between multiple pixels included in the second kernel in the first image and pixels in the second image based on the calculated multiple cost values.

[0073] For example, after moving the kernel in each of the first and second images to the right end of each image (the right end in the row direction of the image) and searching for corresponding points as described above, the kernel in each image is returned to the left end of each image and then moved one pixel downward from there, and the search for corresponding points is performed in the same manner as described above.

[0074] By repeating this process, corresponding points for each pixel in the first image and the second image are searched for.

[0075] Next, the above process will be explained using a general formula. For example, a search is made for a corresponding point in the second image with the vth pixel (v is an integer equal to or greater than 1) included in the i-th kernel in the first image. The vth pixel is a number assigned to each of the multiple pixels included in the kernel according to the pixel order. In addition, if the pixel value of the vth pixel is (pixel value of R, pixel value of G, pixel value of B) = (r1 ｉ，ｖ , g1 ｉ，ｖ , b1ｉ，ｖ In this case, the first color vector c1 of the v-th pixel included in the i-th kernel in the first image is ｖ is expressed as the following equation (1) in the coordinate system of the RGB color space shown in FIG.

[0076]

[0077] The coordinate system of the RGB color space is a three-axis Cartesian coordinate system including, for example, an axis corresponding to the R pixel value, an axis corresponding to the G pixel value, and an axis corresponding to the B pixel value.

[0078] The unit vector in the R-axis direction, the unit vector in the G-axis direction, and the unit vector in the B-axis direction are respectively Let's say.

[0079] For example, the pixel value of the v-th pixel included in the i+d (d: i+d is an integer greater than or equal to 1)-th kernel in the second image is (pixel value of R, pixel value of G, pixel value of B) = (r ｉ＋ｄ，ｖ , g2 ｉ＋ｄ，ｖ , b2 ｉ＋ｄ，ｖ In this case, the second color vector c2 of the v-th pixel included in the i+d-th kernel in the second image is ｖ is expressed as the following equation (2) in the coordinate system of the RGB color space shown in FIG.

[0080]

[0081] c1 ｖ and C2 ｖ The angle between ｉ，ｖ Then, the cosine similarity cosθ ｉ，ｖ is expressed as the following equation (3).

[0082]

[0083] Furthermore, a cost value t calculated from the pixel values ​​of a plurality of pixels included in the i-th kernel in the first image and the pixel values ​​of a plurality of pixels included in the (i+d)-th kernel in the second image is ｉ，ｄ is expressed as the following equation (4).

[0084]

[0085] 3 is a diagram for explaining the cost value according to the embodiment. Specifically, FIG. 3 shows the cost value t when the value of d is changed. ｉ，ｄ 1 is a graph showing a schematic diagram of the

[0086] From the graph shown in FIG. 3, the value of d (d shown in FIG. 3) at which the cost value is maximized is ｏｐｔｉ The search unit 130 searches for, for example, the v-th pixel included in the i-th kernel in the first image and the i+d ｏｐｔｉ The v-th pixel included in the v-th kernel is determined to be a corresponding point.

[0087] [Processing Procedure] Fig. 4 is a flowchart showing a shape measuring method according to an embodiment. For example, the shape measuring device 100 performs the processing shown in Fig. 4.

[0088] First, the shape measurement device 100 acquires (S110) a first image generated by the first imaging device 200 capturing an image of the object in two or more wavelength bands, and a second image generated by the second imaging device 210 capturing an image of the object in two or more wavelength bands from a position different from that of the first imaging device 200. For example, the shape measurement device 100 controls the first imaging device 200 and the second imaging device 210 to cause each of the first imaging device 200 and the second imaging device 210 to capture an image of the object, and acquires images of the object from each of the first imaging device 200 and the second imaging device 210.

[0089] In addition, when executing step S110, specifically when the first imaging device 200 and the second imaging device 210 are caused to capture images of the object, the shape measurement device 100 may control the irradiation device 220 so that the irradiation device 220 irradiates the object with light (specifically, light of two or more wavelength bands).

[0090] Next, the shape measurement device 100 calculates a first color vector based on two or more pixel values ​​indicating the color of a pixel in the first image, and calculates a second color vector based on two or more pixel values ​​indicating the color of a pixel in the second image (S120). Specifically, the shape measurement device 100 calculates a color vector for each pixel included in the first image and the second image based on the pixel value indicating the color of the pixel. For example, the shape measurement device 100 calculates multiple color vectors corresponding to multiple pixels included in a first range in the first image and multiple pixels included in a second range in the second image.

[0091] Next, the shape measurement device 100 calculates the angle between the first color vector and the second color vector (S130). For example, the shape measurement device 100 calculates a plurality of angles while changing the combination of a plurality of first color vectors and a plurality of second color vectors.

[0092] Next, the shape measurement device 100 determines whether or not calculation has been completed for all positions on the same horizontal axis (S140). Specifically, the shape measurement device 100 determines whether or not step S120 and step S130 have been executed for all positions on the horizontal axis of the image coordinate system to which the second range can be moved.

[0093] When the shape measurement device 100 determines that the calculation has not been performed on the entire horizontal axis (No in S140), it moves the second range on the same horizontal axis and executes the processing from step S120 onwards again.

[0094] On the other hand, if the shape measurement device 100 determines that all points on the same horizontal axis have been calculated (Yes in S140), it searches for corresponding points in the first image and the second image based on the calculated angles (S150).

[0095] For example, after step S130, the shape measurement device 100 calculates cosine similarities based on the calculated angles. Furthermore, for example, in calculating the cosine similarities, the shape measurement device 100 calculates multiple cosine similarities based on multiple first color vectors calculated from two or more pixel values ​​of multiple pixels located in a first range in the first image and multiple second color vectors calculated from two or more pixel values ​​of multiple pixels located in a second range in the second image. Furthermore, the shape measurement device 100 calculates a cost value by, for example, adding up the multiple calculated cosine similarities. For example, in calculating the cost value, the shape measurement device 100 repeatedly moves the position of the second range in the second image on the same horizontal axis and repeatedly calculates the cost value. In this case, for example, in searching for corresponding points, the shape measurement device 100 identifies the first range and the second range that have the highest calculated cost value, and determines that corresponding points are included between multiple pixels located in the identified first range and multiple pixels located in the identified second range.

[0096] Next, the shape measurement device 100 calculates the parallax between the first image and the second image based on the searched corresponding points (S160).

[0097] Next, the shape measurement device 100 measures (calculates) the distance between the first image capture device 200 and the object and the distance between the second image capture device 210 and the object based on the calculated parallax (S170). For example, the shape measurement device 100 further measures the shape of the object (e.g., three-dimensional shape) based on the distance between the first image capture device 200 and the object and the distance between the second image capture device 210 and the object.

[0098] Next, the shape measurement device 100 outputs the measurement result (S180). For example, the shape measurement device 100 outputs shape information indicating the shape of the measured object as the measurement result. For example, the shape measurement device 100 outputs the shape information to a display (not shown) or the like, thereby displaying a depth image indicating the shape of the object on the display.

[0099] The measurement result may be distance information indicating the distance between the first imaging device 200 and the object and the distance between the second imaging device 210 and the object. The measurement result may also be position information indicating the positions of corresponding points. The measurement result includes, for example, any one or more of shape information, distance information, and position information.

[0100] [Representative Example] Fig. 5 is a block diagram showing an information processing device 400 according to an embodiment. Fig. 6 is a flowchart showing an information processing method according to an embodiment.

[0101] The information processing device 400 includes a processor 410 and a memory 420 connected to the processor 410. The processor 410 uses the memory 420 to execute the information processing method shown in Fig. 6. The shape measurement device 100 is a specific example of the information processing device 400.

[0102] First, the information processing device 400 calculates a first color vector based on two or more pixel values ​​indicating the color of a pixel in a first image, and calculates a second color vector based on two or more pixel values ​​indicating the color of a pixel in a second image different from the first image (S10).

[0103] Next, the information processing device 400 calculates the angle between the first color vector and the second color vector (S20).

[0104] Next, the information processing device 400 searches for corresponding points (e.g., positions of corresponding points) in the first image and the second image based on the calculated angles (S30). The information processing device 400 outputs information indicating the searched corresponding points (e.g., positions of corresponding points) to a terminal used by the user, for example.

[0105] In this way, the information processing device 400 searches for corresponding points in the first image and the second image based on the length of the first color vector based on the first image, the length of the second color vector based on the second image, and the angle between the first color vector and the second color vector.

[0106] This reduces calculation costs and improves the matching rate (the probability that correct corresponding points are found). In other words, the shape measurement device 100 is advantageous in color stereo matching. Furthermore, because the index used to search for corresponding points is the angle, corresponding points can be calculated with high accuracy even when there is a large difference in brightness between the first and second images (i.e., when there is a large difference in the lengths of the color vectors).

[0107] The information processing device 400 may measure the shape of an object, similar to the shape measurement device 100. In this case, the information processing device 400 may output shape information as a measurement result without outputting position information. Furthermore, the information processing device 400 only needs to search for corresponding points, and does not need to measure the shape of the object if it does not output shape information as a measurement result.

[0108] [Effects, etc.] Below, examples of techniques that can be obtained from the disclosure of this specification will be given, and effects, etc. that can be obtained from these techniques will be described.

[0109] Technique 1 is an information processing method that calculates a first color vector based on two or more pixel values ​​indicating the color of a pixel in a first image, and calculates a second color vector based on two or more pixel values ​​indicating the color of a pixel in a second image different from the first image (S10), calculates the angle between the first color vector and the second color vector (S20), and searches for corresponding points in the first image and the second image based on the calculated angle (S30).

[0110] According to this method, pixel values ​​are used to search for corresponding points. In other words, information indicating the color of pixels is used to search for corresponding points. Therefore, even if there are positions on an object that are the same color but have different brightness, the corresponding points can be searched for by appropriately distinguishing them.

[0111] Depending on the shape and material of the object, the degree of reflected light from the object may differ between the first image, the second image, and the left and right images. Therefore, conventional corresponding point search methods such as the NCC (Normalized Cross Correlation) method have the problem of matching errors. Furthermore, in methods such as the NCC method, normalization is performed on multiple pixels within the kernel, so matching errors are likely to occur if there are differences in the brightness of some of the pixels within the kernel. Furthermore, if the object includes multiple components whose reflectances are different, the hues of the reflected light from the object may appear to be the same even if they are different (e.g., RGB values). In such cases, conventional methods such as the NCC method cannot solve the problem of incorrect matching.

[0112] Therefore, the information processing method according to Technology 1 enables, for example, when performing stereoscopic imaging of an image (color image), to search for corresponding points without losing the high-density color information inherent in the color image and without increasing the burden of computational processing. In the information processing method according to Technology 1, for example, pixel values ​​such as the RGB values ​​of each pixel are expressed as a three-dimensional vector (color vector) in RGB color space. This can improve the matching accuracy (in other words, the accuracy of searching for corresponding points) of occlusion parts, such as the side parts of an object shown in the image. Furthermore, since the color information dimension in the image is not reduced (i.e., color information is not reduced) compared to conventional stereo matching techniques that gray out images, matching can be performed with higher accuracy. In other words, false matching can be reduced. Furthermore, the information processing method according to Technology 1 can shorten the calculation time compared to processes that simply extend colors, such as the NCC method.

[0113] From the above, the information processing method according to Technology 1 can shorten the calculation time compared to stereo matching methods such as the NCC method. In other words, the information processing method according to Technology 1 can reduce the amount of processing.

[0114] Technology 2 is the information processing method described in Technology 1, further comprising acquiring a first image generated by a first imaging device imaging an object in two or more wavelength bands and a second image generated by a second imaging device imaging the object in two or more wavelength bands from a position different from that of the first imaging device, calculating a disparity between the first image and the second image based on the searched corresponding points, and measuring the shape of the object based on the calculated disparity.

[0115] The first imaging device is, for example, the above-described first imaging device 200. The second imaging device is, for example, the above-described second imaging device 210. The object is, for example, the above-described object 300.

[0116] This method uses the angle between color vectors to measure the shape, allowing for accurate measurement of the shape of an object even if there are differences in local brightness. Furthermore, because shape is measured using color vectors, the number of calculation steps is fewer than with conventional color stereo matching technology, and the number of dimensions in the calculation process is also fewer (i.e., the amount of data / calculation is smaller), resulting in faster calculation times.

[0117] As described above, according to the information processing method of Technique 2, the shape of an object can be measured with high accuracy.

[0118] The first image and the second image may be generated by different imaging devices or the same imaging device. For example, the imaging device may capture an image of an object to generate the first image, and then the imaging device may move and capture an image of the object to generate the second image. In other words, the first imaging device and the second imaging device may be the same imaging device or different imaging devices.

[0119] Technique 3 is the information processing method according to Technique 2, further comprising irradiating the object with light of two or more wavelength bands.

[0120] This makes it easier to accurately reflect colors in two or more wavelength bands in the first image and the second image, thereby enabling the shape of the object to be measured with high accuracy.

[0121] Technique 4 is the information processing method according to Technique 2 or 3, further comprising irradiating the object with patterned light.

[0122] This allows the shape of the object to be measured with even greater accuracy.

[0123] Technique 5 is the information processing method according to Technique 4, in which the pattern light is dot pattern light.

[0124] This allows the shape of the object to be measured with even greater accuracy.

[0125] Technique 6 is an information processing method according to any one of Techniques 1 to 5, further calculating a cosine similarity based on the calculated angle, and searching for corresponding points based on the calculated cosine similarity.

[0126] This allows for accurate searching of corresponding points.

[0127] Technique 7 is an information processing method according to Technique 6, in which, in calculating the cosine similarities, multiple cosine similarities are calculated based on multiple first color vectors calculated from two or more pixel values ​​of multiple pixels located in a first range in the first image, and multiple second color vectors calculated from two or more pixel values ​​of multiple pixels located in a second range in the second image; this information processing method further calculates a cost value by adding together the multiple calculated cosine similarities; and in searching for corresponding points, corresponding points are searched for based on the calculated cost value.

[0128] This allows for accurate searching of corresponding points.

[0129] Technique 8 is an information processing method according to Technique 7, in which, in calculating the cost value, the position of the first range in the first image and / or the position of the second range in the second image are repeatedly moved to repeatedly calculate the cost value, and in searching for corresponding points, the first range and the second range that have the highest calculated cost value are identified, and it is determined that multiple pixels located in the identified first range and multiple pixels located in the identified second range contain corresponding points.

[0130] This allows for accurate searching of corresponding points.

[0131] Technique 9 is an information processing device 400 that includes a processor 410 and a memory 420. The processor 410 uses the memory 420 to calculate a first color vector based on two or more pixel values ​​that indicate the color of a pixel in a first image, and calculates a second color vector based on two or more pixel values ​​that indicate the color of a pixel in a second image that is different from the first image, calculates the angle between the first color vector and the second color vector, and searches for corresponding points in the first image and the second image based on the calculated angle.

[0132] This provides the same effects as the information processing method according to Technique 1.

[0133] Technology 10 is an information processing system including the information processing device 400 described in Technology 9, a first imaging device that generates a first image by capturing an image of an object in two or more wavelength bands, and a second imaging device that generates a second image by capturing an image of the object in the two or more wavelength bands from a position different from that of the first imaging device.

[0134] The shape measurement system 10 is a specific example of an information processing system. The information processing system includes, for example, an information processing device 400, a first imaging device 200, and a second imaging device 210. The information processing device 400 controls, for example, the first imaging device 200 and the second imaging device 210 to acquire a first image and a second image.

[0135] According to this, the object can be imaged by the first imaging device and the second imaging device, and the shape of the object can be measured with high accuracy.

[0136] Technique 11 is a program for causing a computer to execute the information processing method according to any one of techniques 1 to 8.

[0137] According to this, the information processing method according to any one of techniques 1 to 8 can be realized by a computer.

[0138] (Other Embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0139] For example, in the above embodiment, the shape measurement system 10 includes two imaging devices (the first imaging device 200 and the second imaging device 210). The number of imaging devices included in the shape measurement system 10 may be two or more, as long as it is plural. For example, the shape measurement system 10 may include three or more imaging devices arranged at different positions. Furthermore, for example, the shape measurement device 100 may acquire images of an object from each of the three or more imaging devices, calculate color vectors for each of the acquired three or more images as described above, and search for corresponding points in each of the three or more images based on the calculated color vectors to measure the shape of the object. The information processing device 400 and the information processing system are also similar to the shape measurement device 100 and the shape measurement system 10.

[0140] Furthermore, for example, in the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit, the order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0141] Furthermore, for example, in the above-described embodiments, each component of the processing unit may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0142] Furthermore, each component may be realized by hardware. Each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0143] Furthermore, the method of communication between the devices in the above-described embodiment is not particularly limited. Furthermore, a relay device (such as a broadband router, not shown) may be involved in the communication between the devices.

[0144] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM. Also, the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. For example, the present disclosure may be realized as a shape measurement method, as a program for causing a computer to execute the shape measurement method, or as a non-transitory recording medium on which such a program is recorded and which is readable by a computer.

[0145] In addition, this disclosure also includes forms obtained by applying various modifications to the embodiments that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of the embodiments within the scope that does not deviate from the intent of this disclosure.

[0146] The present disclosure is applicable to an apparatus for measuring the three-dimensional shape of an object.

[0147] REFERENCE SIGNS LIST 10 Shape measurement system 100 Shape measurement device 110 Acquisition unit 120 First calculation unit 130 Search unit 140 Second calculation unit 150 Measurement unit 160 Output unit 170 Storage unit 200 First imaging device 210 Second imaging device 220 Irradiation device 300 Object 400 Information processing device 410 Processor 420 Memory

Claims

1. An information processing method comprising: calculating a first color vector based on two or more pixel values indicating the color of a pixel in a first image; calculating a second color vector based on two or more pixel values indicating the color of a pixel in a second image different from the first image; calculating the angle between the first color vector and the second color vector; and searching for corresponding points in the first image and the second image based on the calculated angle.

2. The information processing method according to claim 1, further comprising: acquiring the first image generated by a first imaging device imaging an object in two or more wavelength bands; and acquiring the second image generated by a second imaging device imaging the object in the two or more wavelength bands from a position different from that of the first imaging device; calculating a parallax between the first image and the second image based on the searched corresponding points; and measuring the shape of the object based on the calculated parallax.

3. The information processing method according to claim 2, further comprising irradiating the object with light of the two or more wavelength bands.

4. The information processing method according to claim 3, further comprising irradiating the object with patterned light.

5. The information processing method according to claim 4, wherein the pattern light is a dot pattern light.

6. The information processing method according to claim 1, further comprising: calculating a cosine similarity based on the calculated angle; and searching for the corresponding points based on the calculated cosine similarity.

7. The information processing method according to claim 6, wherein in calculating the cosine similarities, a plurality of cosine similarities are calculated based on a plurality of first color vectors calculated from two or more pixel values of a plurality of pixels located in a first range in the first image and a plurality of second color vectors calculated from two or more pixel values of a plurality of pixels located in a second range in the second image; the information processing method further calculates a cost value by adding together the calculated plurality of cosine similarities; and in searching for corresponding points, the corresponding points are searched for based on the calculated cost value.

8. The information processing method according to claim 7, wherein in calculating the cost value, the position of the first range in the first image and / or the position of the second range in the second image are repeatedly moved to repeatedly calculate the cost value; and in searching for the corresponding point, the first range and the second range for which the calculated cost value is the highest are identified, and it is determined that the corresponding point is included in a plurality of pixels located in the identified first range and a plurality of pixels located in the identified second range.

9. An information processing device comprising: a processor; and a memory, wherein the processor uses the memory to calculate a first color vector based on two or more pixel values indicating the color of a pixel in a first image, and calculate a second color vector based on two or more pixel values indicating the color of a pixel in a second image different from the first image, calculate an angle between the first color vector and the second color vector, and search for corresponding points in the first image and the second image based on the calculated angle.

10. An information processing system comprising: the information processing device according to claim 9; a first imaging device that generates the first image by imaging an object in two or more wavelength bands; and a second imaging device that generates the second image by imaging the object in the two or more wavelength bands from a position different from that of the first imaging device.

11. A program for causing a computer to execute the information processing method according to any one of claims 1 to 8.

Images