Information processing method, information processing device, information processing system, and program

Abstract

An information processing method according to one embodiment of the present disclosure comprises: calculating a first color vector on the basis of two or more pixel values representing the color of a pixel in a first image, and calculating a second color vector on the basis of two or more pixel values representing the color of a pixel in a second image different from the first image (S10); an angle formed by the first color vector and the second color vector is calculated (S20), and a corresponding point is searched for in the first image and the second image on the basis of the calculated angle (S30).

Description

Technical Field

[0001] This disclosure relates to information processing methods, information processing apparatus, information processing systems, and programs. Background Technology

[0002] Conventionally, there is a technique for measuring the three-dimensional shape of an object (the subject) by generating images from two cameras taken from different positions. The measurement of the three-dimensional shape is performed, for example, using the imaging position offset (parallax) between the two images and through triangulation. For instance, corresponding points representing the same position are searched in the two images, and the parallax between the two cameras is calculated based on the found corresponding points.

[0003] Patent document 1 discloses a three-dimensional shape measuring device that uses triangulation to calculate three-dimensional shape information based on the hue data of each pixel in a photographic image obtained by photographing the reflected light of a pattern of continuously changing hue reflected from the object being measured.

[0004] (Existing technical documents) (Patent Documents) Patent Document 1: Japanese Patent No. 5743433 Summary of the Invention

[0005] The problem that the invention aims to solve In this case, when an object comprises multiple parts with varying reflectivities, even if the hues of the reflected light returning from the object differ, the same hue may appear. In such situations, it may be impossible to accurately locate corresponding points, thus hindering the precise measurement of the object's shape.

[0006] This disclosure provides an information processing method capable of searching for corresponding points in multiple images with high precision.

[0007] Methods for solving problems In one aspect of the information processing method disclosed herein, a first color vector is calculated based on two or more pixel values ​​representing the color of a pixel in a first image, and a second color vector is calculated based on two or more pixel values ​​representing the color of a pixel in a second image, wherein the second image is different from the first image. The angle between the first color vector and the second color vector is calculated, and a corresponding point is searched in the first image and the second image based on the calculated angle.

[0008] Furthermore, one aspect of the information processing apparatus disclosed herein includes a processor and a memory. The processor uses the memory to calculate a first color vector based on two or more pixel values ​​representing the color of a pixel in a first image, and calculates a second color vector based on two or more pixel values ​​representing the color of a pixel in a second image, which is different from the first image. The processor calculates the angle between the first color vector and the second color vector, and searches for a corresponding point in the first image and the second image based on the calculated angle.

[0009] Furthermore, one aspect of the information processing system disclosed herein includes: the information processing apparatus described above; a first camera device for generating the first image by photographing an object in two or more wavelengths; and a second camera device for generating the second image by photographing the object from a position different from the first camera device in the two or more wavelengths.

[0010] Furthermore, one aspect of this disclosure relates to a program for causing a computer to perform the aforementioned information processing method.

[0011] The effects of the invention According to one aspect of the information processing method disclosed herein, corresponding points can be searched in multiple images with high precision. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating the configuration of the shape measurement system according to the embodiment.

[0013] Figure 2 This is a diagram illustrating a specific example of the calculation process for the angle formed by the two color vectors involved in the implementation method.

[0014] Figure 3 This is a diagram used to illustrate the cost values ​​involved in the implementation method.

[0015] Figure 4 This is a flowchart illustrating the shape measurement method involved in the embodiment.

[0016] Figure 5 This is a block diagram illustrating an information processing apparatus according to an embodiment.

[0017] Figure 6 This is a flowchart illustrating the information processing method involved in the implementation method. Detailed Implementation

[0018] The embodiments will now be described in detail with reference to the accompanying drawings.

[0019] Furthermore, the embodiments described below are all general or specific examples. The numerical values, shapes, materials, constituent elements, the arrangement and connection methods of constituent elements, steps, and the order of steps shown in the following embodiments are all examples and are not intended to limit this disclosure. In addition, constituent elements not described in the independent technical solution in the following embodiments will be described as arbitrary constituent elements.

[0020] Furthermore, in this specification, ordinal numbers such as "the first" and "the second" do not indicate the quantity or order of constituent elements unless otherwise specified, and are only used to avoid confusion among similar constituent elements and to distinguish them.

[0021] (Implementation Method) [constitute] Figure 1 This is a block diagram showing the configuration of the shape measurement system 10 according to the embodiment.

[0022] The shape measurement system 10 is a system for measuring the shape of objects such as parts. Specifically, the shape measurement system 10 photographs the object from multiple camera devices, such as a stereo camera, at different positions, and measures the three-dimensional shape of the object based on the images containing the object captured (generated) by each camera device. The shape measurement system 10 is an example of an information processing system.

[0023] The shape measurement system 10 includes a shape measurement device 100, a first camera device 200, a second camera device 210, and an illumination device 220.

[0024] The shape measuring device 100 is a computer that measures the shape of an object based on multiple images generated by the first camera device 200 and the second camera device 210 by capturing images of the object. For example, the shape measuring device 100 calculates the three-dimensional shape (information representing the three-dimensional shape) of the object using triangulation based on the multiple images acquired by the first camera device 200 and the second camera device 210. The shape measuring device 100 is an example of an information processing device.

[0025] The shape measuring device 100 is implemented, for example, by a computer, which includes: a communication interface for communicating with the first camera device 200, the second camera device 210, and the illumination device 220 of the shape measuring system 10; non-volatile memory for storing programs executed by each processing unit; volatile memory serving as a temporary storage area for executing programs; input / output ports for transmitting and receiving signals; and a processor such as a CPU (Central Processing Unit) for executing programs. The communication interface can be implemented by a connector for connecting communication lines for wired communication, or by a wireless communication circuit for wireless communication.

[0026] The first imaging device 200 is a camera that generates a first image by capturing images of an object in two or more wavelengths. That is, the first imaging device 200 captures images of the object by detecting reflected light from the object in two or more wavelengths. The first image is an image generated by the first imaging device 200 capturing images of the object.

[0027] The second imaging device 210 is a camera that generates a second image by capturing images of an object in two or more wavelengths. That is, the second imaging device 210 captures images of the object by detecting reflected light from the object in two or more wavelengths. The second image is an image generated by the second imaging device 210 capturing images of the object.

[0028] The first camera device 200 and the second camera device 210 photograph the same object from different positions. Therefore, a parallax occurs between the first image and the second image. The shape measuring device 100 calculates this parallax and uses the calculated parallax to measure the shape of the object. Specifically, the shape measuring device 100 uses the first image, the second image, the parallax, and the distance between the first camera device 200 and the second camera device 210 to measure the three-dimensional shape of the object.

[0029] The first camera device 200 and the second camera device 210 photograph the object by detecting light of the same wavelength. Specifically, the first camera device 200 and the second camera device 210 each photograph the object by detecting light of two or more different wavelengths.

[0030] For example, the first camera device 200 and the second camera device 210 function as stereo cameras. The first camera device 200 and the second camera device 210 are arranged side by side, such that their respective shooting directions are parallel.

[0031] Two or more wavelengths can be arbitrarily determined without special restrictions. For example, two or more wavelengths can correspond to the three colors of RGB (Red, Green, Blue). For example, the wavelength of red light is 640nm to 770nm, the wavelength of green light is 490nm to 550nm, and the wavelength of blue light is 430nm to 490nm.

[0032] Furthermore, the two or more bands can correspond to RGB bands, or bands corresponding to any color such as Y (Yellow), M (Magenta), C (Cyan), UV (Ultraviolet), or IR (Infrared). The bandwidth of each band can be arbitrarily determined. For example, the first camera device 200 and the second camera device 210 can detect light with two or more bands (e.g., peak wavelengths) that are different and have bandwidths from 50nm to 150nm. Also, for example, the first camera device 200 and the second camera device 210 can detect light with two or more wavelengths that are different and are of a single wavelength. Furthermore, the two or more bands can correspond to R and B bands, UV and IR bands, or UV, RGB, and IR bands; the bands can be arbitrarily combined to achieve this.

[0033] The illumination device 220 is a light source that illuminates an object. Specifically, the illumination device 220 illuminates the object with light of two or more wavelengths. More specifically, the illumination device 220 illuminates the object with light of two or more wavelengths detected by the first imaging device 200 and the second imaging device 210. The first imaging device 200 and the second imaging device 210, for example, capture images of the object by detecting reflected light that is illuminated by the illumination device 220 and reflected by the object.

[0034] Furthermore, the light irradiated by the irradiation device 220 can be any light containing two or more wavelengths detectable by the first camera device 200 and the second camera device 210. For example, the light irradiated by the irradiation device 220 can also be any light containing only two or more wavelengths detectable by the first camera device 200 and the second camera device 210. Alternatively, for example, if the first camera device 200 and the second camera device 210 detect light corresponding to RGB, the light irradiated by the irradiation device 220 can also be white light.

[0035] Furthermore, for example, the illumination device 220 can also illuminate the object with patterned light. For example, the patterned light is dot matrix patterned light.

[0036] In addition, the pattern of patterned light does not have to be a dot matrix pattern. For example, the pattern of patterned light can be a random pattern or a striped pattern.

[0037] The illumination device 220 is implemented, for example, by an LED (Light Emitting Diode). The illumination device 220 can be implemented by any type of light source, such as an LD (Laser Diode), a halogen lamp, or a fluorescent lamp.

[0038] In addition, the shape measurement system 10 can have multiple camera devices, or even more than three.

[0039] Next, the specific configuration of the shape measuring device 100 will be described.

[0040] The shape measuring device 100 includes an acquisition unit 110, a first calculation unit 120, a search unit 130, a second calculation unit 140, a measurement unit 150, an output unit 160, and a storage unit 170.

[0041] The acquisition unit 110 is a processing unit that acquires a first image and a second image. The first image is an image generated by the first imaging device 200 by capturing an object at two or more wavelengths, and the second image is an image generated by the second imaging device 210 by capturing an object at two or more wavelengths from a position different from the first imaging device 200. For example, the acquisition unit 110 causes the illumination device 220 to illuminate the object, then causes the first imaging device 200 and the second imaging device 210 to capture images of the object, and then acquires the first image and the second image from the first imaging device 200 and the second imaging device 210.

[0042] The first calculation unit 120 is a processing unit that calculates a color vector based on two or more pixel values ​​representing the colors of pixels in the image acquired by the acquisition unit 110. Specifically, the first calculation unit 120 calculates a first color vector based on two or more pixel values ​​representing the colors of pixels in the first image. Furthermore, the first calculation unit 120 calculates a second color vector based on two or more pixel values ​​representing the colors of pixels in the second image. The first color vector is a color vector calculated based on the pixel values ​​of pixels included in the first image. The second color vector is a color vector calculated based on the pixel values ​​of pixels included in the second image.

[0043] A pixel value is a value that represents the color of a pixel. Two or more pixel values ​​can be, for example, values ​​representing the colors of two or more wavelengths of light detected by the first imaging device 200 and the second imaging device 210, respectively. Two or more pixel values ​​can also be, for example, values ​​representing the individual colors of the three primary colors (RGB). In other words, the values ​​representing the R color, the G color, and the B color in a pixel are examples of pixel values.

[0044] A color vector is a vector used to represent the color of a pixel. For example, when two or more pixel values ​​are the individual values ​​of the three primary colors RGB, the color vector is represented as a three-dimensional vector such as (pixel value of R, pixel value of G, pixel value of B). Thus, the color vector calculated by the first calculation unit 120 corresponds to two or more pixel values, becoming a vector of two dimensions or more.

[0045] In addition, for two or more pixel values, in order to correspond to two or more bands, not only RGB, but also arbitrary color values ​​such as Y, M, C, UV or IR can be used.

[0046] For example, the first calculation unit 120 calculates a color vector corresponding to each of the plurality of pixels contained in the first image and the second image, based on the pixel value of the pixel.

[0047] Furthermore, the first calculation unit 120 calculates the angle between the first color vector and the second color vector. For example, the first calculation unit 120 combines each of the plurality of first color vectors calculated based on the first image with the plurality of second color vectors calculated based on the second image, and calculates the angle between the combined first color vector and the second color vector.

[0048] Furthermore, the combination of multiple first color vectors and multiple second color vectors can be arbitrary. For example, multiple first color vectors and multiple second color vectors can be fully combined and paired. Alternatively, the combination of first color vectors and second color vectors to be calculated can be determined based on the position of pixels in the image.

[0049] The search unit 130 is a processing unit that searches for corresponding points in the first and second images based on calculated angles. Specifically, the search unit 130 searches for corresponding points in the first and second images based on the cost value of the calculated angle. More specifically, the search unit 130 calculates the cost value based on the angle formed by the first color vector and the second color vector, and determines the corresponding point based on the calculated cost value.

[0050] A corresponding point refers to a point (e.g., a pixel) that represents the same location (e.g., the same spot in an object) in both the first and second images. The search unit 130 determines, for example, pixels in the first image and pixels in the second image that become corresponding points based on multiple angles calculated by the first calculation unit 120.

[0051] For example, in the search for corresponding points, the search unit 130 uses the similarity based on the angle calculated by the first calculation unit 120 to search for corresponding points. For example, the first calculation unit 120 also calculates cosine similarity based on the calculated angle. Furthermore, for example, in the search for corresponding points, the search unit 130 searches for corresponding points based on the cosine similarity calculated by the first calculation unit 120. Specifically, in the calculation of cosine similarity, the first calculation unit 120 calculates multiple cosine similarities based on multiple first color vectors and multiple second color vectors, wherein the multiple first color vectors are calculated from two or more pixel values ​​of multiple pixels located in a first range in the first image, and the multiple second color vectors are calculated from two or more pixel values ​​of multiple pixels located in a second range in the second image.

[0052] For example, the first calculation unit 120 performs full combination pairing of multiple first color vectors with multiple second color vectors, calculates the angle for each combination, and calculates the cosine similarity based on the calculated angle.

[0053] Furthermore, for example, the first calculation unit 120 also calculates the cost value based on multiple calculated cosine similarities. For example, the first calculation unit 120 calculates the cost value by adding the multiple calculated cosine similarities. Furthermore, for example, in the search for corresponding points, the search unit 130 searches for corresponding points based on the cost value calculated by the first calculation unit 120.

[0054] For example, in calculating the cost value, the first calculation unit 120 repeatedly calculates the cost value by repeatedly moving at least one of the position of the first range in the first image and the position of the second range in the second image. That is, the first calculation unit 120 repeatedly changes at least one of the position of the first range and the position of the second range, and calculates the cost value according to various combinations of the positions of the first range and the second range.

[0055] For example, in calculating the cost value, the first calculation unit 120 can repeatedly calculate the cost value by repeatedly moving either the position of the first region in the first image or the position of the second region in the second image along the epipolar line. For example, in calculating the cost value, the first calculation unit 120 can also repeatedly calculate the cost value by repeatedly moving the position of the second region along one of two mutually orthogonal axes in the image coordinate system. For example, in calculating the cost value, the first calculation unit 120 can also repeatedly calculate the cost value by moving the position of the second region along the horizontal axis of the image coordinate system (e.g., described later). Figure 2 The horizontal axis in the second image shown in (b) is moved repeatedly to repeatedly calculate the cost value.

[0056] Furthermore, for example, in the search for corresponding points, the search unit 130 determines a first range and a second range where the cost value calculated by the first calculation unit 120 is the highest. That is, for example, in the search for corresponding points, the search unit 130 determines a combination of a first range and a second range where the cost value is the highest among multiple cost values ​​calculated by the first calculation unit 120. Thus, for example, like block matching, the first calculation unit 120 defines a pixel in the first image and a rectangular area (also called a kernel) containing the pixels surrounding that pixel as a first region, and determines a second range in the second image where the cost value becomes the highest when combined with the first range. Furthermore, for example, the search unit 130 also determines that multiple pixels located in the determined first range and multiple pixels located in the determined second range contain corresponding points.

[0057] The first and second ranges contain the same number of pixels, for example, four in each row direction and four in each column direction in the image. The search unit 130 determines, for example, that among the multiple pixels located in the determined first range and the multiple pixels located in the determined second range, pixels corresponding to the same position within their respective ranges are corresponding points.

[0058] Furthermore, the size of the first and second ranges (the number of pixels contained in these ranges) can be arbitrarily determined without any particular limitation. For example, when the number of pixels contained in the first and second ranges is (the number of pixels in the row direction of the image) × (the number of pixels in the column direction of the image), it can be 3×3, 4×4, or 5×5, etc.

[0059] The second calculation unit 140 is a processing unit that calculates the disparity between the first image and the second image based on the corresponding points searched by the search unit 130. For example, the second calculation unit 140 calculates the disparity by matching (stereo matching) each part (pixel) of the first image and the second image. For example, the second calculation unit 140 calculates the disparity as the difference (position offset) between the positions of corresponding parts, i.e., the corresponding points, between the first image and the second image.

[0060] The measurement unit 150 is a processing unit that measures the shape of an object based on calculated parallax. For example, the measurement unit 150 measures the three-dimensional shape of the object using triangulation based on the first image, the second image, the parallax calculated by the second calculation unit 140, and the distance between the first imaging device 200 and the second imaging device 210. For example, the measurement unit 150 generates a depth image of the object that represents the measurement results of the object's shape.

[0061] The output unit 160 is a processing unit that outputs the measurement results of the measurement unit 150. For example, the output unit 160 outputs the measurement results of the shape of the object to a display (not shown), thereby displaying a depth image or the like representing the shape of the object on the display.

[0062] Alternatively, the shape measuring device 100 may also include the display.

[0063] Each processing unit, such as the acquisition unit 110, the first calculation unit 120, the search unit 130, the second calculation unit 140, the measurement unit 150, and the output unit 160, is implemented, for example, by a processor and a memory storing a control program executed by the processor.

[0064] Storage unit 170 is a storage device that stores various types of information. Storage unit 170 may store, for example, information indicating the distance between the first camera device 200 and the second camera device 210, and information indicating the size of the core. Storage unit 170 may be implemented using, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0065] [Specific example] Next, the specific processing of color vector calculation and corresponding point search will be explained. Furthermore, in the specific examples described below, we will explain how the first camera device 200 and the second camera device 210 each capture images of the object 300 by detecting light corresponding to each RGB band.

[0066] Figure 2 This is a diagram illustrating a specific example of the calculation process for the angle formed by the two color vectors involved in the implementation method.

[0067] like Figure 2 As shown in (a), for example, the first camera device 200 and the second camera device 210 are arranged side by side above the object 300 and take pictures of the object 300 with their respective shooting directions downward.

[0068] First, the first calculation unit 120 sets a first region in the first image and a second region in the second image. In this example, the first calculation unit 120 sets a first region and a second region containing 16 (4×4) pixels.

[0069] Figure 2The i-th kernel shown in (b) and (c) refers to a specific example of the defined region (region 1 or region 2). i is, for example, an integer greater than or equal to 1. For example, the region 1 in the first image, defined as the upper left, is designated as the first kernel. The region 1, moved one pixel to the right from that position, is designated as the second kernel. The region 1, moved one pixel further to the right from that position, is designated as the third kernel. Furthermore, for example, if the region 1 is moved to the right end of the first image and designated as the k-th kernel, then the region 1 is moved back to the left end and further moved one pixel down from that position. The region 1 designated in this way is designated as the (k+1)-th kernel. Similarly, the position of the second region is designated as the i-th kernel. For example, if the first and second images have the same number of pixels, and both the first and second regions are located in the i-th kernel, the position (coordinates) of the first region and the second region in the image coordinate system are the same.

[0070] Additionally, while the kernel is moved one pixel at a time as described above, it can also be moved multiple pixels at a time. Furthermore, the kernel can be moved in a manner that does not contain duplicate pixels.

[0071] For example, the first calculation unit 120 calculates 16 first color vectors corresponding to the 16 pixels contained in the first kernel of the first image, and 16 second color vectors corresponding to the 16 pixels contained in the first kernel of the second image. Next, the first calculation unit 120 combines pixels at the same position in the first and second images and calculates the angle between the first and second color vectors corresponding to the combined pixels. Furthermore, the first calculation unit 120 calculates the cosine similarity to each of the 16 angles calculated in this way. Finally, the first calculation unit 120 calculates the cost value by adding the 16 cosine similarities calculated in this way.

[0072] Next, the first calculation unit 120 calculates 16 first color vectors corresponding to the 16 pixels contained in the first kernel in the first image, and 16 second color vectors corresponding to the 16 pixels contained in the second kernel in the second image. Furthermore, the first calculation unit 120 calculates the angle, cosine similarity, and cost value as described above.

[0073] In this way, the first calculation unit 120 repeatedly moves the position of the kernel (second region) in the second image and repeatedly calculates the cost value. For example, the first calculation unit 120 repeatedly moves the position of the kernel in the second image from one end of the second image to the other end on the horizontal axis of the image coordinate system opposite to that end, so that an arbitrary position in the kernel is located on that horizontal axis. In this way, the first calculation unit 120 calculates multiple cost values. The arbitrary position can be, for example, the position of any pixel in the kernel, the center of the kernel, or any other arbitrary position. Furthermore, for example, the first calculation unit 120 can also move the position of the kernel in the second image from... Figure 2 The second image in (c) is repeatedly moved to the right from its left end to its right end in the right direction. Of course, the first calculation unit 120 can also, for example, make the position of the kernel in the second image change from... Figure 2 The second image in (c) can be moved repeatedly from the right end to the left end along the left direction, or repeatedly from the top end to the bottom end along the down direction, or repeatedly from the bottom end to the top end along the up direction, or repeatedly in any direction.

[0074] Next, the search unit 130 determines which kernel in the second image has the largest cost value among the multiple cost values ​​calculated by the first calculation unit 120. Here, if the number of the determined kernel in the second image is the m-th, the search unit 130 determines, for example, that the multiple pixels contained in the first kernel in the first image correspond to the multiple pixels contained in the m-th kernel in the second image.

[0075] Furthermore, for example, after moving the kernel in the second image to the right end of the second image (the right end of the row direction of the image) and calculating the cost value, the first calculation unit 120 calculates 16 first color vectors corresponding to the 16 pixels contained in the second kernel in the first image, and 16 second color vectors corresponding to the 16 pixels contained in the first kernel in the second image.

[0076] In this way, the first calculation unit 120 repeatedly moves the position of the kernel (first region) in the first image and repeatedly calculates the cost value. The direction of moving the kernel in the first image may be the same as that of the kernel in the second image, but it may also be different and can be arbitrarily specified.

[0077] Similarly, for example, the first calculation unit 120 fixes the position of the second kernel in the first image, repeatedly moves the position of the kernel in the second image, and repeatedly calculates the cost value. The search unit 130 also searches for corresponding points of multiple pixels contained in the second kernel in the first image and pixels in the second image based on the calculated multiple cost values, as described above.

[0078] For example, after moving the kernels in each of the first and second images to the right end of each image (the right end of the row direction in the image) and searching for the corresponding points as described above, the kernels in each image are returned to the left end of each image, and then moved down one pixel from that position to search for the corresponding points in the same way as described above.

[0079] By repeatedly performing this process, the corresponding points of each pixel in the first and second images can be searched out.

[0080] Next, the above processing will be explained using a general formula. For example, search for the corresponding point in the second image of the v-th (v: an integer greater than 1) pixel contained in the i-th kernel of the first image. The v-th pixel refers to the number assigned to each of the multiple pixels contained in the kernel according to the pixel order. Furthermore, let the pixel value of the v-th pixel be (pixel value of R, pixel value of G, pixel value of B) = (r1... i，v g1 i，v b1 i，v In this case, the first color vector c1 of the v-th pixel contained in the i-th kernel of the first image. v ,exist Figure 2 The coordinate system of the RGB color space shown in (d) is as shown in equation (1) below.

[0081] [Mathematical Expression 1] The coordinate system of the RGB color space is, for example, a three-axis Cartesian coordinate system containing the axis corresponding to the pixel value of R, the axis corresponding to the pixel value of G, and the axis corresponding to the pixel value of B.

[0082] In addition, the unit vectors in the R-axis direction, G-axis direction, and B-axis direction are set as follows.

[0083] [Mathematical Expression 2] Furthermore, for example, let the pixel value of the v-th pixel contained in the i+d-th kernel (d: i+d is an integer greater than 1) in the second image be (pixel value of R, pixel value of G, pixel value of B) = (r2) i+d，v g2 i+d，v b2 i+d，v In this case, the second color vector c2 of the v-th pixel contained in the (i+d)-th kernel of the second image. v exist Figure 2 The coordinate system of the RGB color space shown in (d) is as shown in equation (2) below.

[0084] [Mathematical Expression 3] c1 v With c2 v Let the angle be θ i，v Then the cosine similarity cosθ i，v As shown in equation (3) below.

[0085] [Mathematical Expression 4] Furthermore, the cost value t is calculated based on the pixel values ​​of multiple pixels contained in the i-th kernel of the first image and the pixel values ​​of multiple pixels contained in the (i+d)-th kernel of the second image. i，d As shown in equation (4) below.

[0086] [Mathematical Expression 5] Figure 3 This is a diagram used to illustrate the cost values ​​involved in the implementation method. Specifically, Figure 3 This schematically illustrates the cost value t when the value of d mentioned above is changed. i，d The image.

[0087] from Figure 3 The graph shown illustrates the value of d when the cost is maximized. Figure 3 The d shown opti The search unit 130, for example, determines that the v-th pixel contained in the i-th kernel of the first image is related to the (i+d)-th pixel in the second image. opti The v-th pixel contained in each kernel is the corresponding point.

[0088] [Processing order] Figure 4 This is a flowchart illustrating the shape measurement method according to the embodiment. For example, the shape measuring device 100 performs... Figure 4 The processing shown.

[0089] First, the shape measuring device 100 acquires a first image and a second image (S110). The first image is generated by the first imaging device 200 by photographing the object at two or more wavelengths, and the second image is generated by the second imaging device 210 by photographing the object from a different position than the first imaging device 200 at two or more wavelengths. For example, the shape measuring device 100 controls the first imaging device 200 and the second imaging device 210 to photograph the object respectively, and acquires images containing the object from the first imaging device 200 and the second imaging device 210 respectively.

[0090] In addition, when the shape measuring device 100 performs step S110, specifically when the first camera device 200 and the second camera device 210 respectively take pictures of the object, the irradiation device 220 can also be controlled to irradiate light (specifically, light of two or more wavelengths) onto the object.

[0091] Next, the shape measuring device 100 calculates a first color vector based on two or more pixel values ​​representing the color of a pixel in the first image, and calculates a second color vector based on two or more pixel values ​​representing the color of a pixel in the second image (S120). Specifically, the shape measuring device 100 calculates a color vector for each pixel contained in the first and second images based on the pixel value representing the color of that pixel. For example, the shape measuring device 100 calculates multiple color vectors corresponding to multiple pixels located in a first range in the first image and multiple pixels located in a second range in the second image, respectively.

[0092] Next, the shape measuring device 100 calculates the angle between the first color vector and the second color vector (S130). For example, the shape measuring device 100 calculates multiple angles while changing multiple combinations of the first color vector and multiple second color vectors.

[0093] Next, the shape measuring device 100 determines whether calculations have been completed for all positions on the same horizontal axis (S140). Specifically, the shape measuring device 100 determines whether steps S120 and S130 have been performed for all positions on the second movable horizontal axis in the image coordinate system.

[0094] If it is determined that the calculation for all positions on the same horizontal axis has not been completed ("No" in S140), the shape measuring device 100 moves the second range on the same horizontal axis and performs the processing after step S120 again.

[0095] On the other hand, if it is determined that calculations have been completed for all positions on the same horizontal axis ("Yes" in S140), the shape measuring device 100 searches for corresponding points in the first and second images based on the calculated angles (S150).

[0096] For example, after step S130, the shape measuring device 100 calculates cosine similarity based on the calculated angle. Furthermore, for example, in the calculation of cosine similarity, the shape measuring device 100 calculates multiple cosine similarities based on multiple first color vectors and multiple second color vectors. The multiple first color vectors are calculated based on two or more pixel values ​​of multiple pixels in a first range located in the first image, and the multiple second color vectors are calculated based on two or more pixel values ​​of multiple pixels in a second range located in the second image. Further, the shape measuring device 100 calculates the cost value, for example, by adding the calculated multiple cosine similarities. For example, in the calculation of the cost value, the shape measuring device 100 repeatedly moves the position of the second range in the second image along the same horizontal axis, thereby repeatedly calculating the cost value. In this case, for example, in the search for corresponding points, the shape measuring device 100 determines the first range and the second range where the calculated cost value is the highest, and determines that the corresponding point is contained within the multiple pixels in the determined first range and the multiple pixels in the determined second range.

[0097] Next, the shape measuring device 100 calculates the disparity between the first image and the second image based on the searched corresponding points (S160).

[0098] Next, the shape measuring device 100 measures (calculates) the distance between the first camera device 200 and the object, and the distance between the second camera device 210 and the object, based on the calculated parallax (S170). For example, the shape measuring device 100 also measures the shape (e.g., three-dimensional shape) of the object based on the distance between the first camera device 200 and the object, and the distance between the second camera device 210 and the object.

[0099] Next, the shape measuring device 100 outputs the measurement result (S180). For example, the shape measuring device 100 outputs shape information representing the shape of the measured object as the measurement result. For example, the shape measuring device 100 outputs the shape information to a display (not shown) and displays a depth image or the like representing the shape of the object on that display.

[0100] Alternatively, the measurement results can be distance information indicating the distance between the first camera device 200 and the object, and the distance between the second camera device 210 and the object. Furthermore, the measurement results can also be location information indicating the position of a corresponding point. The measurement results may include, for example, any one or more of the following: shape information, distance information, and location information.

[0101] [Representative Example] Figure 5 This is a block diagram illustrating the information processing apparatus 400 according to the embodiment. Figure 6This is a flowchart illustrating the information processing method involved in the implementation method.

[0102] The information processing device 400 includes a processor 410 and a memory 420 connected to the processor 410. The processor 410 uses the memory 420 to perform operations. Figure 6 The information processing method shown is illustrated. The shape measuring device 100 is a specific example of the information processing device 400.

[0103] First, the information processing device 400 calculates a first color vector based on two or more pixel values ​​representing the color of a pixel in the first image, and calculates a second color vector based on two or more pixel values ​​representing the color of a pixel in a second image that is different from the first image (S10).

[0104] Next, the information processing device 400 calculates the angle between the first color vector and the second color vector (S20).

[0105] Next, the information processing device 400 searches for a corresponding point (e.g., the position of the corresponding point) in the first and second images based on the calculated angle (S30). The information processing device 400 outputs information indicating the searched corresponding point (e.g., the position of the corresponding point) to a terminal used by the user, for example.

[0106] In this way, the information processing device 400 searches for corresponding points in the first image and the second image based on the length of the first color vector based on the first image, the length of the second color vector based on the second image, and the angle between the first color vector and the second color vector.

[0107] Accordingly, computational costs can be reduced and the matching rate (the probability that the correct corresponding point is found) can be improved. That is, it is more advantageous in color stereo matching according to the shape measuring device 100. In addition, since the index used to search for corresponding points is angle, corresponding points can be calculated with high accuracy even when there is a large difference in brightness between the first image and the second image (i.e., a large difference in the length of the color vector).

[0108] Furthermore, like the shape measuring device 100, the information processing device 400 can also measure the shape of an object. In this case, the information processing device 400 may output shape information as the measurement result without outputting position information. Additionally, the information processing device 400 can simply search for the corresponding point, and may not need to measure the shape of the object if it does not output shape information as the measurement result.

[0109] [Effects, etc.] The following examples illustrate techniques obtained from the disclosure of this specification and explain the effects obtained from such techniques.

[0110] Technique 1 is an information processing method in which a first color vector is calculated based on two or more pixel values ​​representing the color of a pixel in a first image, and a second color vector is calculated based on two or more pixel values ​​representing the color of a pixel in a second image different from the first image (S10). The angle between the first color vector and the second color vector is calculated (S20), and a corresponding point is searched in the first image and the second image based on the calculated angle (S30).

[0111] Therefore, pixel values ​​are used in the search for corresponding points. That is, information representing the color of a pixel is used in the search for corresponding points. Thus, for example, even locations within an object that have the same color but different brightness can be appropriately distinguished and searched for corresponding points.

[0112] Depending on the shape and material of the object, the intensity of reflected light from the object may differ between left and right images, such as the first image and the second image. Therefore, traditional corresponding point search methods, such as the NCC (Normalized Cross Correlation) method, suffer from matching errors. Furthermore, in methods like NCC, since multiple pixels within the kernel are normalized, matching errors are easily generated when some of these pixels have brightness differences. Additionally, when the object comprises multiple parts with different reflectivities, even if the hue (e.g., RGB values) of the reflected light from the object differs, cases where the hue is the same may occur. In such cases, traditional methods like NCC cannot resolve the mismatch problem.

[0113] Therefore, in the information processing method of Technique 1, for example, when using an image (color image) for stereoscopic photography, the search for corresponding points can be achieved without losing the original high-density color information of the color image and without increasing the computational burden. In the information processing method of Technique 1, for example, pixel values ​​such as the RGB values ​​of each pixel are represented as three-dimensional vectors (color vectors) in the RGB color space. Accordingly, for example, the matching accuracy for occlusions such as the side views of objects reflected in the image can be improved (in other words, the search accuracy for corresponding points). Furthermore, compared to traditional stereo matching techniques that grayscale the image, high-precision matching can be achieved because the dimensionality of color information in the image is not reduced (i.e., color-related information is not reduced). In other words, false matching can be reduced. Moreover, according to the information processing method of Technique 1, the computation time can be shortened compared to simple color expansion processes such as the NCC method.

[0114] In summary, the information processing method described in Technique 1 can shorten the computation time compared to stereo matching methods such as the NCC method. In other words, the information processing method described in Technique 1 can reduce the amount of processing required.

[0115] Technique 2 is the information processing method described in Technique 1. In this information processing method, a first image and a second image are acquired. The first image is an image generated by a first camera device by taking pictures of an object in two or more bands. The second image is an image generated by a second camera device by taking pictures of an object from a different position than the first camera device in two or more bands. The parallax between the first image and the second image is calculated based on the searched corresponding points. The shape of the object is measured based on the calculated parallax.

[0116] The first camera device is, for example, the first camera device 200 described above. The second camera device is, for example, the second camera device 210 described above. The object is, for example, the object 300 described above.

[0117] Therefore, by using the angle formed by color vectors to measure shape, the shape of an object can be measured with high accuracy even if there are differences in brightness. Furthermore, because shape measurement uses color vectors, compared to traditional color stereo matching techniques, it involves fewer computational steps and has a lower dimensionality (i.e., less data / computation), thus accelerating computation time.

[0118] In summary, based on the information processing method involved in Technique 2, the shape of an object can be measured with high precision.

[0119] Furthermore, the first image and the second image can be generated by different camera devices or by the same camera device. For example, the camera device generates the first image by photographing the object, and then the camera device moves and photographs the object to generate the second image. That is, the first camera device and the second camera device can be the same camera device or different camera devices.

[0120] Technology 3 is the information processing method described in Technology 2, which further includes irradiating the object with light of two or more wavelengths.

[0121] Therefore, in the first and second images, colors of more than two bands can be easily and accurately represented. This allows for the high-precision measurement of the shape of an object.

[0122] Technique 4 is the information processing method described in Technique 2 or 3, which further includes irradiating the object with patterned light.

[0123] Therefore, it is possible to measure the shape of an object with even higher precision.

[0124] Technology 5 is the information processing method described in Technology 4, where the patterned light is a dot matrix patterned light.

[0125] Therefore, it is possible to measure the shape of an object with even higher precision.

[0126] Technique 6 is the information processing method described in any one of Techniques 1 to 5. In this information processing method, cosine similarity is calculated based on the calculated angle, and corresponding points are searched based on the calculated cosine similarity during the search for corresponding points.

[0127] Therefore, it is possible to search for the corresponding point with high precision.

[0128] Technique 7 is the information processing method described in Technique 6. In the calculation of cosine similarity, multiple cosine similarities are calculated based on multiple first color vectors and multiple second color vectors. The multiple first color vectors are calculated based on two or more pixel values ​​of multiple pixels in a first range in a first image, and the multiple second color vectors are calculated based on two or more pixel values ​​of multiple pixels in a second range in a second image. In this information processing method, a cost value is calculated by adding the calculated multiple cosine similarities. In the search for corresponding points, the corresponding points are searched based on the calculated cost value.

[0129] Therefore, it is possible to search for the corresponding point with high precision.

[0130] Technique 8 is the information processing method described in Technique 7. In calculating the cost value, the cost value is repeatedly calculated by repeatedly moving at least one of the positions of the first range in the first image and the second range in the second image. In the search for corresponding points, the first range and the second range where the calculated cost value is the highest are determined, and it is determined that multiple pixels located in the determined first range and multiple pixels located in the determined second range contain corresponding points.

[0131] Therefore, it is possible to search for the corresponding point with high precision.

[0132] Technology 9 is an information processing device 400, which includes a processor 410 and a memory 420. The processor 410 uses the memory 420 to calculate a first color vector based on two or more pixel values ​​representing the color of a pixel in a first image, and calculates a second color vector based on two or more pixel values ​​representing the color of a pixel in a second image different from the first image. The processor 410 calculates the angle between the first color vector and the second color vector, and searches for a corresponding point in the first image and the second image based on the calculated angle.

[0133] Accordingly, the same effect as the information processing method involved in Technique 1 can be achieved.

[0134] Technology 10 is an information processing system comprising: the information processing apparatus 400 described in Technology 9; a first camera device for generating a first image by photographing an object in two or more wavelengths; and a second camera device for generating a second image by photographing the object from a position different from the first camera device in two or more wavelengths.

[0135] The shape measurement system 10 is a specific example of an information processing system. The information processing system, for example, includes an information processing device 400, a first imaging device 200, and a second imaging device 210. The information processing device 400, for example, acquires a first image and a second image by controlling the first imaging device 200 and the second imaging device 210.

[0136] Accordingly, the object can be photographed using the first and second camera devices, and the shape of the object can be measured with high precision.

[0137] Technique 11 is a program for causing a computer to perform any one of Techniques 1 to 8.

[0138] Accordingly, the information processing method involved in any of the techniques 1 to 8 can be implemented by a computer.

[0139] (Other implementation methods) While the implementation methods have been described above, this disclosure is not limited to the above-described implementation methods.

[0140] For example, in the above embodiment, the shape measurement system 10 includes two camera devices (a first camera device 200 and a second camera device 210). The shape measurement system 10 can include multiple camera devices, such as two or more. For example, the shape measurement system 10 may also include three or more camera devices positioned at different locations. Furthermore, for example, the shape measurement device 100 may acquire an image containing the object from each of the three or more camera devices, calculate a color vector for each of the acquired three or more images as described above, search for corresponding points in each of the three or more images based on the calculated color vectors, and measure the shape of the object. The information processing device 400 and the information processing system are also the same as those for the shape measurement device 100 and the shape measurement system 10.

[0141] Furthermore, as in the above embodiments, the processing performed by a specific processing unit can also be performed by other processing units. Additionally, the order of multiple processes can be changed, or multiple processes can be executed in parallel.

[0142] Furthermore, for example, in the embodiments described above, each component of the processing unit can also be implemented by executing software programs suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or processor reading and executing software programs recorded on a recording medium such as a hard disk or semiconductor memory.

[0143] Furthermore, each component can be implemented in hardware. Each component can also be a circuit (or integrated circuit). These circuits can be used as a whole to form a single circuit, or they can be separate circuits. Moreover, these circuits can be general-purpose circuits or special-purpose circuits.

[0144] Furthermore, the communication method between devices in the above embodiments is not particularly limited. In device-to-device communication, relay devices (such as broadband routers, not shown) may also be present.

[0145] Furthermore, the general or specific embodiments of this disclosure can be implemented by a system, apparatus, method, integrated circuit, computer program, or a non-transitory recording medium such as a computer-readable CD-ROM. Additionally, it can be implemented by any combination of systems, apparatus, methods, integrated circuits, computer programs, and recording media. For example, this disclosure can also be implemented as a shape measurement method, as a program for causing a computer to execute a shape measurement method, or as a computer-readable non-transitory recording medium on which the program is recorded.

[0146] Furthermore, this disclosure includes any modifications to the implementation methods that can be conceived by those skilled in the art, or any combination of the constituent elements and functions in the implementation methods without departing from the spirit of this disclosure.

[0147] Industrial availability This disclosure relates to an apparatus for measuring the three-dimensional shape of an object.

[0148] Symbol Explanation 10 Shape Measurement System 100 Shape measuring device 110 Acquisition Department 120 First Computing Department 130 Search Department 140 Second Computing Department 150 Measurement Department 160 Output Section 170 Storage Department 200 First camera device 210 Second camera device 220 Irradiation Device 300 objects 400 Information Processing Device 410 processor 420 Memory.

Claims

1. An information processing method, wherein in the information processing method, A first color vector is calculated based on two or more pixel values ​​representing the color of pixels in a first image, and a second color vector is calculated based on two or more pixel values ​​representing the color of pixels in a second image, wherein the second image is different from the first image. Calculate the angle between the first color vector and the second color vector. Based on the calculated angle, a corresponding point is searched in the first image and the second image.

2. The information processing method as described in claim 1, Further in the information processing method, The first image and the second image are acquired. The first image is generated by a first camera device capturing images of an object at two or more wavelengths. The second image is generated by a second camera device capturing images of the object from a different position than the first camera device at the two or more wavelengths. Based on the searched corresponding points, the disparity between the first image and the second image is calculated. The shape of the object is measured based on the calculated parallax.

3. The information processing method as described in claim 2, In the information processing method, light of two or more wavelengths is further irradiated onto the object.

4. The information processing method as described in claim 3, In the information processing method, patterned light is further irradiated onto the object.

5. The information processing method as described in claim 4, The patterned light is a dot matrix patterned light.

6. The information processing method as described in claim 1, The information processing method further includes calculating cosine similarity based on the calculated angle. In the search for the corresponding points, the corresponding points are searched based on the calculated cosine similarity.

7. The information processing method as described in claim 6, In the calculation of the cosine similarity, multiple cosine similarities are calculated based on multiple first color vectors and multiple second color vectors. The multiple first color vectors are calculated based on two or more pixel values ​​of multiple pixels located in a first range in the first image, and the multiple second color vectors are calculated based on two or more pixel values ​​of multiple pixels located in a second range in the second image. The information processing method further includes calculating the cost value by summing the calculated cosine similarities. In the search for the corresponding point, the corresponding point is searched based on the calculated cost value.

8. The information processing method as described in claim 7, In calculating the cost value, the cost value is repeatedly calculated by repeatedly shifting at least one of the positions of the first range in the first image and the second range in the second image. In the search for the corresponding point, the first range and the second range where the calculated cost value is the highest are determined, and it is determined that the corresponding point is contained in a plurality of pixels located in the determined first range and a plurality of pixels located in the determined second range.

9. An information processing device, The information processing device includes: Processor; and memory, The processor uses the memory. A first color vector is calculated based on two or more pixel values ​​representing the color of pixels in a first image, and a second color vector is calculated based on two or more pixel values ​​representing the color of pixels in a second image, which is different from the first image. Calculate the angle between the first color vector and the second color vector. Based on the calculated angle, a corresponding point is searched in the first image and the second image.

10. An information processing system, comprising: The information processing apparatus according to claim 9; The first camera device generates the first image by photographing the object in two or more wavelengths; and The second camera device generates the second image by capturing images of the object from a position different from that of the first camera device in two or more wavelengths.

11. A program for causing a computer to perform the information processing method according to any one of claims 1 to 8.

Images