Image processing method

Abstract

This invention provides an image processing method that allows for the simple and rapid execution of depth image alignment without the use of jigs or other fixtures. [Solution] The control unit acquires a first depth image and a second depth image (S1), and sets one of the first element images as the first reference image (S3). The control unit calculates the first distance difference, which is the difference between the distance indicated by the distance information of the first reference image and the distance indicated by the distance information of the first surrounding images arranged around the first reference image (S5). The control unit sets one of the second element images as the second reference image (S6). The control unit calculates the second distance difference, which is the difference between the distance indicated by the distance information of the second reference image and the distance indicated by the distance information of the second surrounding images arranged around the second reference image (S8), and calculates the relative distance difference between them (S9). The control unit repeats steps S6, S8, and S9 to identify the second reference image that has the smallest relative distance difference (S12).

Description

[Technical Field]

[0001] The present invention relates to an image processing method. [Background technology]

[0002] Conventionally, the method described in Patent Document 1 is known as an example of an image processing method for aligning depth images acquired from different imaging points. Here, a depth image is an image of an object to be imaged from a predetermined position, and includes distance information from the predetermined position to each element included in the object to be imaged. The method described in Patent Document 1 uses a 3D laser scanner to acquire 3D point cloud data (depth image) of a structure that includes a jig placed at the same position within its measurement range, from each of several measurement points. This jig is equipped with three or more connected spheres. The center coordinates of the three or more spheres equipped with the jig are calculated from the acquired 3D point cloud data, and initial alignment data is generated between the multiple 3D point cloud data so that these center coordinates match. From the generated initial alignment data, final alignment data is generated between the multiple 3D point cloud data using a predetermined alignment algorithm. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. 2023 / 223437 [Overview of the project] [Problems that the invention aims to solve]

[0004] The above technology is intended for measuring civil engineering structures such as underground tunnels. In such cases, the measurement range is very wide, allowing the jig to be placed within that range. However, when the measurement range is relatively narrow, it may not be possible to place the jig within that range. Furthermore, since the above technology requires obtaining depth images with the jig included within the measurement range, there may be limitations on the depth images that can be obtained, potentially making its operation complicated. In addition, obtaining the final alignment data requires the use of a predetermined alignment algorithm, and this process takes a certain amount of time, making it unsuitable for in-line measurement in production lines, etc.

[0005] The purpose of this disclosure is to provide an image processing method that can easily and quickly perform processing for aligning depth images without using jigs or the like. [Means for solving the problem]

[0006] The image processing method according to this disclosure includes an acquisition step of acquiring an image of an object to be imaged, which includes multiple elements, taken from a first predetermined position, and comprising a first depth image including distance information from the first predetermined position to each element for each first element image representing each element, and a second depth image including distance information from the second predetermined position to each element for each second element image representing each element, and a first selection step of selecting one of the multiple first element images in the first depth image as a first reference image, and the distance indicated by the distance information corresponding to the first reference image and the first surrounding image which is the first element image arranged around the first reference image. The method includes: a first calculation step of calculating a first distance difference, which is the difference between the distance indicated by distance information and the second depth image; a second selection step of selecting one of the multiple second element images in the second depth image as the second reference image; a second calculation step of calculating a second distance difference, which is the difference between the distance indicated by distance information corresponding to the second reference image and the distance indicated by distance information corresponding to the second surrounding image, which is the second element image arranged around the second reference image; a third calculation step of calculating a relative distance difference, which is the difference between the first distance difference and the second distance difference; and a selection step of identifying the second reference image in which the relative distance difference is smallest by repeatedly executing the second selection step, the second calculation step and the third calculation step.

[0007] According to this disclosure, the process for aligning depth images can be performed simply and quickly without using jigs or other fixtures. [Brief explanation of the drawing]

[0008] [Figure 1] This is an overall structural diagram of the image processing system 100. [Figure 2] This is a block diagram showing the configuration of the control device 10. [Figure 3] This is a flowchart showing the image synthesis process. [Figure 4] This is an explanatory diagram showing examples of the first depth image D1 and the second depth image D2. [Figure 5] This is an explanatory diagram showing examples of the first depth image D1, in which each of the first element images D1E has been identified by the element identification step, and the second depth image D2, in which each of the second element images D2E has been identified. [Figure 6] This is an explanatory diagram showing an example of a state in which one of several first element images D1E in the first depth image D1 is selected as the first reference image D1S, and a first surrounding image D1M arranged around the first reference image D1S. [Figure 7] This is an explanatory diagram illustrating an example of a state in which one of several second element images D2E in the second depth image D2 is selected as the second reference image D2S, and a second surrounding image D2M arranged around the second reference image D2S. [Figure 8] This is an explanatory diagram showing examples of the first depth image D1, where C1 6 was used as the alignment reference; the second depth image D2, where C2 5 was used as the alignment reference; and the third depth image D3, which was generated by aligning the first depth image D1 and the second depth image D2 according to their respective references. [Figure 9] This is a conceptual diagram showing the first distance difference table. [Figure 10] This is a conceptual diagram showing the second distance difference table. [Modes for carrying out the invention]

[0009] First, embodiments of this disclosure will be listed and described. (1) The image processing method of the present disclosure includes an acquisition step of acquiring an image of an object to be imaged, which includes a plurality of elements, taken from a first predetermined position, and comprising a first depth image including distance information from the first predetermined position to each element for each first element image representing each of the elements, and an acquisition step of acquiring an image of the object to be imaged, which includes a second depth image including distance information from the second predetermined position to each of the elements for each second element image representing each of the elements, and a first selection step of selecting one of the plurality of first element images in the first depth image as a first reference image, and the distance indicated by the distance information corresponding to the first reference image and the first surrounding image which is the first element image arranged around the first reference image. The method includes: a first calculation step of calculating a first distance difference, which is the difference between the distance indicated by the distance information; a second selection step of selecting one of the multiple second element images in the second depth image as the second reference image; a second calculation step of calculating a second distance difference, which is the difference between the distance indicated by the distance information corresponding to the second reference image and the distance indicated by the distance information corresponding to the second surrounding image, which is the second element image arranged around the second reference image; a third calculation step of calculating a relative distance difference, which is the difference between the first distance difference and the second distance difference; and a selection step of identifying the second reference image in which the relative distance difference is smallest by repeatedly executing the second selection step, the second calculation step and the third calculation step.

[0010] When an object containing multiple elements is imaged from a single location, the captured image may not include images corresponding to all elements due to factors such as some elements being hidden from view by others. To resolve this problem, the object is imaged from multiple locations, and the captured images are combined to image all elements included in the object. In the image processing method according to this disclosure, a first depth image is obtained from a first predetermined position, and a second depth image is obtained from a second predetermined position. One of the multiple first element images in the first depth image is selected as the first reference image, and the first distance difference is calculated based on the distance information corresponding to the first reference image and the distance information corresponding to the first surrounding images arranged around the first reference image. Similarly, a second reference image is selected from the second element images in the second depth image, and the second distance difference is calculated based on the distance information corresponding to the second reference image and the distance information corresponding to the second surrounding images arranged around the second reference image. When the first reference image and the second reference image correspond to the same element, the relative distance difference, which is the difference between the first distance difference and the second distance difference, approximates 0. The image processing method according to this disclosure identifies a second reference image in which the relative distance difference is smallest. In this way, the image processing method according to this disclosure can quickly perform processing for aligning depth images without using jigs or the like.

[0011] (2) In the image processing method described in (1), the first depth image and the second depth image may be images captured such that their imaging areas overlap each other. The first selection step may select one of the multiple first element images in the first depth image that is located in the overlapping area where the second depth image and the imaging area overlap, as the first reference image. The second selection step may select one of the multiple second element images in the second depth image that is located in the overlapping area, as the second reference image.

[0012] In this case, it is sufficient that the first depth image and the second depth image are captured so that their imaging areas overlap, and it is not necessary for the first and second depth images to capture the entirety of the object being imaged. Since the constraints on the arrangement of imaging devices such as TOF cameras that capture the first and second depth images are relaxed, the range of applicability of image processing methods is expanded.

[0013] (3) In the image processing method described in (1) or (2), the first calculation step may calculate the difference between the distance indicated by the distance information corresponding to the first reference image and the distance indicated by the distance information corresponding to the first surrounding image positioned in a first direction relative to the first reference image as the first distance difference. The second calculation step may calculate the difference between the distance indicated by the distance information corresponding to the second reference image and the distance indicated by the distance information corresponding to the second surrounding image positioned in a first direction relative to the second reference image as the second distance difference.

[0014] In this case, the first distance difference is calculated based on the distance information corresponding to the first reference image and the distance information corresponding to the first surrounding image positioned in the first direction relative to the first reference image. Furthermore, the second distance difference is calculated based on the distance information corresponding to the second reference image and the distance information corresponding to the second surrounding image positioned in the first direction relative to the second reference image. This increases the likelihood that the element corresponding to the second reference image, which has the smallest relative distance difference, coincides with the element corresponding to the first reference image.

[0015] (4) In the image processing method described in any of (1) to (3), the first calculation step may further calculate the difference between the distance indicated by the distance information corresponding to the first reference image and the distance indicated by the distance information corresponding to the first surrounding image positioned in a second direction relative to the first reference image as the first distance difference. The second calculation step may further calculate the difference between the distance indicated by the distance information corresponding to the second reference image and the distance indicated by the distance information corresponding to the second surrounding image positioned in a second direction relative to the second reference image as the second distance difference. The third calculation step may calculate the relative distance difference relating to the difference between the first distance difference and the second distance difference relating to the first direction, and the relative distance difference relating to the difference between the first distance difference and the second distance difference relating to the second direction. The identification step may calculate the sum of the multiple relative distance differences calculated by the third calculation step and identify the second reference image for which the sum is smallest.

[0016] In this case, the likelihood that the first reference image and the second reference image correspond to the same elements is further increased.

[0017] (5) In the image processing method described in any of (1) to (4), the object to be imaged may be a plurality of rods bound together. Each of the elements included in the object to be imaged may be each end face of the plurality of rods.

[0018] One possible method for counting the number of rods in a bundle is to image the bundled rods and count the end faces of the rods included in the image. However, if the end faces of the multiple rods are unevenly aligned, the number of rods may not be accurately counted by imaging from a single location. The image processing method involves using the bundled rods as the imaging target, and performing processing on the images corresponding to each end face of the multiple rods included in the first and second depth images, respectively, as the first and second element images. This identifies the end face of the rod corresponding to the second reference image, where the relative distance difference is smallest, as the end face of the rod corresponding to the first reference image, thus enabling rapid alignment of the first and second depth images.

[0019] (6) In the image processing method described in any of (1) to (5), the first selection step may select one of the plurality of first element images corresponding to each end face of the plurality of rods as the first reference image. The first calculation step may calculate the first distance difference using the first element image corresponding to the end face adjacent to the end face corresponding to the first reference image as the first surrounding image. The second selection step may select one of the plurality of second element images corresponding to each end face as the second reference image. The second calculation step may calculate the second distance difference using the second element image corresponding to the end face adjacent to the end face corresponding to the second reference image as the second surrounding image.

[0020] The end faces of multiple bundled rods are often arranged adjacently such that one end face is surrounded by another. This positional relationship between end faces is similar to the positional relationship between the first and second element images in the first and second depth images. The image processing method selects one of the multiple first element images included in the first depth image as the first reference image, and calculates the first distance using the first element image adjacent to the first reference image as the first surrounding image. The image processing method also selects one of the multiple second element images included in the second depth image as the second reference image, and calculates the second distance using the second element image adjacent to the second reference image as the second surrounding image. This identifies the end face of the rod corresponding to the second reference image with the smallest relative distance difference as the end face of the rod corresponding to the first reference image, thus enabling accurate and rapid alignment of the first and second depth images.

[0021] Embodiments of this disclosure will be described below with reference to the drawings. The drawings are used to illustrate the technical features that this disclosure may adopt, and the configurations of the devices described are not intended to be limiting, but are merely illustrative examples. First, with reference to Figure 1, the system configuration of the image processing system 100 that executes the image processing method according to this disclosure will be described. The image processing method according to this disclosure is a method for extracting a reference for aligning images from images showing the elements contained in each image when combining an image of an imaging target F containing multiple elements, an image taken from a first predetermined position P1 and an image taken from a second predetermined position P2 into a single image. In this specification, the image processing system 100 and the imaging target F will be described with the positive Y-axis direction as the upward direction, the negative Y-axis direction as the downward direction, the positive X-axis direction as the rightward direction, the negative X-axis direction as the leftward direction, and the positive Z-axis direction as the depth direction, as shown in Figure 1. However, the actual configuration of the image processing system 100 and the imaging target F may differ from this. Furthermore, in this specification, "up and down," "left and right," and "depth" include arrangements that are substantially perceived as up and down, left and right, and depth. Also, in the following descriptions, for multiple identical configurations, reference numerals may be assigned to only some of the configurations, while the reference numerals for other configurations may be omitted.

[0022] The image processing system 100 includes a control device 10 and TOF (Time-of-Flight) cameras T1 and T2. The control device 10 may be a dedicated device for the image processing method, or it may be a general-purpose device such as a personal computer (PC) with a dedicated application for performing the image processing method installed. In this embodiment, a control device 10 using a general-purpose PC is provided as an example.

[0023] TOF cameras T1 and T2 are cameras that can measure the distance to the image target F using the time of flight of light. TOF camera T1 images the image target F from a first predetermined position P1. TOF camera T2 images the image target F from a second predetermined position P2, which is different from the first predetermined position P1. In this embodiment, each of TOF cameras T1 and T2 is equipped with a light-emitting unit, a light-receiving unit, and a depth image generation circuit (not shown). TOF cameras T1 and T2 emit light from the light-emitting unit. The light-receiving units of TOF cameras T1 and T2 receive the light that has been reflected back from each element included in the image target F after being emitted from the light-emitting unit. The light-receiving unit has, for example, 50,000 to 300,000 light-receiving elements (for example, 600 horizontally x 400 vertically).

[0024] In TOF cameras T1 and T2, based on the timing of light emission by the light-emitting unit and the timing of light reception by the light-receiving unit, the distance to the point of light reflection and the direction from which the light came are determined for each light-receiving element. Therefore, TOF cameras T1 and T2 can acquire information indicating the distance from TOF cameras T1 and T2 to numerous reflection points on the image target F, and information indicating the position relative to TOF cameras T1 and T2. The information indicating the distance from TOF cameras T1 and T2 to numerous reflection points on the image target F will be referred to as "distance information" below. The information indicating the position of numerous reflection points on the image target F relative to TOF cameras T1 and T2 will be referred to as "position information" below.

[0025] In other words, the TOF cameras T1 and T2 can acquire distance information indicating the distance in the Z-axis direction (distance in the depth direction) for numerous reflection points in the imaging target F, in addition to position information indicating the position in the X-axis direction (left-right direction) and the Y-axis direction (up-down direction) as shown in Figure 1. The depth image generation circuit performs preprocessing such as distance thresholding and noise reduction on the measurement results for each photodetector to generate a depth image, which is a two-dimensional image containing information including distance information from each TOF camera T1 and T2 to each element included in the imaging target F. Note that some or all of the processing performed by the depth image generation circuit may be performed by the control device 10. In the image processing system 100, the TOF cameras T1 and T2 are examples of means by which the image processing system 100 acquires the first depth image D1 and the second depth image D2, which will be described later, as the targets of processing.

[0026] In this embodiment, as shown in Figure 1, the imaging target F by the TOF cameras T1 and T2 is a bar material B consisting of multiple steel bars bound together by binding wire U. Such an imaging target F includes the end faces Ba of the multiple bar materials B as elements constituting the imaging target F. In Figure 1, the end faces Ba are circular. The shape of the end faces Ba may also be elliptical, polygonal, or the like. Conventionally, the number of multiple bar materials B bound together by binding wire U has been counted by performing pattern matching using the shape of the end faces Ba of the bar materials B on an image obtained by imaging such an imaging target F with a camera.

[0027] A depth image can be obtained by capturing the end faces Ba of multiple rods B using a TOF camera, and the number of rods B can be counted by counting the number of images showing the end faces Ba included in the acquired depth image. On the other hand, as shown in Figure 1, the end faces Ba of multiple rods B may be bundled together in an uneven state. Because the end faces Ba of multiple rods B are uneven, in a depth image captured from a specific location, one end face Ba may be hidden from view by other end faces Ba. In such cases, not all images showing the end faces Ba may appear in the depth image, and the number of rods B may not be accurately counted. To avoid this situation, depth images obtained from multiple locations are combined so that all end faces Ba appear in the depth image. The image processing method according to this disclosure is intended to simplify and quickly perform the process of aligning each depth image when combining multiple depth images obtained from multiple locations into a single depth image.

[0028] Next, the configuration of the control device 10 will be described with reference to Figure 2. The control device 10 comprises a control unit 20, a storage unit 21, and an external device connection unit 28. The control unit 20 includes a CPU that controls the entire control device 10. The control unit 20 is electrically connected to the storage unit 21 via a data bus (not shown). The storage unit 21 includes a non-volatile storage device such as an HDD. In this embodiment, the HDD is an example of a non-temporary storage medium. The non-temporary storage medium can be any storage medium capable of storing information regardless of the period for which the information is stored. The non-temporary storage medium does not have to include a temporary storage medium (e.g., transmitted signals). In this embodiment, the storage device is an HDD, but the storage device may be composed of other non-temporary storage media that can retain data regardless of the length of time the information is stored, such as flash memory or RAM. The storage unit 21 stores a program that causes the control unit 20 to execute the image synthesis process, which will be described later in Figure 3. The control unit 20 functions as an example of a processor that executes the image synthesis process by deploying the program stored in the storage unit 21. The memory unit 21 may be located outside the control device 10.

[0029] The program executed by the control unit 20 is provided by recording it on, for example, a magnetic disk, optical disk, semiconductor memory, or similar recording medium, and storing it in the storage unit 21, etc. The recording medium on which the program is recorded may be any storage format as long as it is a recording medium that the control unit 20 can read. Alternatively, the program executed by the control unit 20 may be provided by being electrically connected to an external network (not shown) via an external device connection unit 28 and downloaded from an external server device or the like connected to the network. Furthermore, the program may be pre-installed in the control device 10.

[0030] The control unit 20 is electrically connected to an operation unit 22 such as a keyboard, touch panel, or mouse, and a display unit 23 such as a display that shows images, via an input / output interface (not shown). The external device connection unit 28 is a connector for electrically connecting to external information devices such as TOF cameras T1 and T2. The control unit 20 is electrically connected to each of the TOF cameras T1 and T2 via the external device connection unit 28 and can receive image information corresponding to the first depth image D1 and the second depth image D2 generated by the TOF cameras T1 and T2. Alternatively, the control unit 20 may be electrically connected via the external device connection unit 28 to an information device that stores image information showing the first depth image D1 and the second depth image D2 generated by the TOF cameras T1 and T2, and receive image information from that information device. Furthermore, the control unit 20 may be electrically connected to an external network (not shown) via an external device connection unit 28 to receive image information corresponding to the first depth image D1 and the second depth image D2 from an external device connected to the network. The external device connection unit 28 is not limited to a wired communication connector and may be configured to allow external devices to connect wirelessly.

[0031] The image processing method according to this embodiment will be described with reference to Figure 3. The control unit 20 of the control device 10 executes the image processing method by performing the image synthesis process shown in Figure 3. The image synthesis process is executed when the control unit 20 detects an instruction to start the image synthesis process. In this embodiment, the instruction to start the image synthesis process is executed by the user of the control device 10 via the operation unit 22. In the following description, each step of the process will be abbreviated as "S".

[0032] When the image synthesis process is started, the control unit 20 executes an acquisition step (S1). In the acquisition step, the control unit 20 acquires a first depth image D1 from the TOF camera T1, which is an image of the imaging target F taken from a first predetermined position P1, and includes distance information from the first predetermined position P1 to each of the multiple end faces Ba, which are elements included in the imaging target F, for each of the first element images D1E. The control unit 20 also acquires a second depth image D2 from the TOF camera T2, which is an image of the imaging target F taken from a second predetermined position P2, and includes distance information from the second predetermined position P2 to each of the multiple end faces Ba, which are elements included in the imaging target F, for each of the second element images D2E. Specifically, the control unit 20 sends an instruction to the TOF camera T1 via the external device connection unit 28 to transmit image information corresponding to the first depth image D1 generated by the TOF camera T1. Upon receiving this instruction, the TOF camera T1 transmits the image information corresponding to the generated first depth image D1 to the control device 10. Furthermore, the control unit 20 sends an instruction to the TOF camera T2 via the external device connection unit 28 to transmit image information corresponding to the second depth image D2 generated by the TOF camera T2. Upon receiving this instruction, the TOF camera T2 transmits the image information corresponding to the generated second depth image D2 to the control device 10.

[0033] Figure 4(A-1) shows an example of the first depth image D1. As shown in Figure 4(A-1), the first depth image D1 is composed of multiple first element images D1E, each corresponding to an end face Ba included in the imaging target F. The first depth image D1 is a grayscale image in which the distance in the Z-axis direction (distance in the depth direction) indicated by the distance information is converted to a predetermined gradation in order to visualize the distance information corresponding to each pixel. Specifically, each pixel of the first depth image D1 is represented such that the shorter the distance in the Z-axis direction indicated by the distance information corresponding to each pixel, the closer it approaches black, and the longer the distance in the Z-axis direction indicated by the distance information, the closer it approaches white. Since the end face Ba of the rod B is almost flat, the distance from the TOF camera T1 to each end face Ba is almost equal. Therefore, the distance information corresponding to multiple pixels that make up one first element image D1E corresponding to one end face Ba will indicate an almost equal distance in the Z-axis direction. Consequently, one first element image D1E exhibits a circular shape, which is the shape of the corresponding end face Ba, and is represented by a gray of roughly the same gradation overall. In Figure 4(A-1), the outlines of each first element image D1E are shown in black; this is to clearly distinguish each of the first element images D1E. In this way, the first depth image D1 is composed of multiple circular first element images D1E, each shown in a different shade of gray, as shown in Figure 4(A-1).

[0034] Figure 4(A-2) shows an example of the second depth image D2. Similar to the first depth image D1, the second depth image D2 is composed of multiple second element images D2E, which are images corresponding to each of the end faces Ba included in the imaging target F, as shown in Figure 4(A-2). In the second depth image D2, the second element images D2E are represented as circular images shown in different shades of gray.

[0035] The first depth image D1 and the second depth image D2 are captured such that their imaging areas overlap. The overlapping region DA in each of the first depth image D1 and the second depth image D2 is defined as the overlapping region DA, as shown in Figures 4(A-1) and (A-2). When the first depth image D1 and the second depth image D2 are captured in such a way that they each have an overlapping region DA, the entire target object F does not necessarily have to be captured in each of the first depth image D1 and the second depth image D2. Alternatively, the entire target object F may be captured in each of the first depth image D1 and the second depth image D2. In this case, the entire imaging area of ​​each of the first depth image D1 and the second depth image D2 becomes the overlapping region DA.

[0036] For the sake of simplicity, in Figures 4 to 8, the number of first element images D1E included in the first depth image D1 and the number of second element images D2E included in the second depth image D2 are set to approximately 10. The actual object being imaged F is approximately 100 rods B, each having a circular end face Ba with a diameter of 20 mm to 30 mm, bound together with binding wire U. Therefore, the actual first depth image D1 and second depth image D2 will contain approximately 100 first element images D1E and second element images D2E, each showing the end face Ba of multiple rods B.

[0037] Returning to the explanation of Figure 3, the control unit 20 then performs an element identification step (S2). In the element identification step, the control unit 20 performs a process to identify the image corresponding to the end face Ba of the rod B from among the first element image D1E and the second element image D2E included in the first depth image D1 and the second depth image D2.

[0038] Specifically, the control unit 20 acquires the value of the size (diameter) of the end face Ba of the known bar B. Then, it determines whether the first element image D1E arranged at the uppermost left in the first depth image D1 acquired in the acquisition step corresponds to the size of the acquired end face Ba. When it is determined that this first element image D1E corresponds to the size of the end face Ba, it is specified that this first element image D1E corresponds to the end face Ba, and an identification symbol is attached to the specified first element image D1E. In the present embodiment, let the number of the depth image be T, and the Nth first element image D1E included in the Tth depth image is given an identification symbol of "C T N ". In this case, as shown in FIG. 5(B-1), for the first element image D1E arranged at the uppermost left in the first depth image D1, the first element image D1E is given an identification symbol of "C 1 1".

[0039] Next, the control unit 20 determines whether the first element image D1E to the immediate right of the first element image D1E of "C 1 1" corresponds to the size of the end face Ba. When it is determined that this first element image D1E corresponds to the size of the end face Ba, it is specified that this first element image D1E corresponds to the end face Ba, and an identification symbol is attached to the specified first element image D1E. In this case, as shown in FIG. 5(B-1), for the first element image D1E arranged to the immediate right of the first element image D1E to which "C 1 1" is attached, the second element image D1E is given an identification symbol of "C 1 2". By repeatedly executing such processing, the control unit 20 attaches an identification symbol to each of the first element images D1E included in the first depth image D1 that is specified as corresponding to the end face Ba. In the present embodiment, as shown in FIG. 5(B-1), it is assumed that each of the first element images D1E included in the first depth image D1 is given an identification symbol from "C 1 1" to "C 1 10 ".

[0040] The control unit 20 performs the same processing on the second depth image D2 and assigns an identification symbol to each of the second element images D2E included in the second depth image D2 that are identified as corresponding to the end face Ba. In this embodiment, as shown in Figure 5(B-2), each of the second element images D2E included in the second depth image D2 is assigned the "C 2 1 to C 2 10 Let's assume that the identification code " is assigned to it.

[0041] The above is not an exhaustive list of examples of the element identification step. For example, by applying a predetermined pattern matching technique, processing may be performed to identify the image corresponding to the end face Ba of the rod B from among the first element image D1E and the second element image D2E included in the first depth image D1 and the second depth image D2.

[0042] Returning to the explanation of Figure 3, the control unit 20 then performs the first selection step (S3). In the first selection step, the control unit 20 selects one of the multiple first element images D1E identified as corresponding to the end face Ba of the rod B in the first depth image D1 as the first reference image D1S. In this embodiment, the control unit 20 selects one first element image D1E from among the multiple first element images D1E located in the overlapping region DA of the first depth image D1 as the first reference image D1S.

[0043] The selection of the first reference image D1S is performed automatically by the control unit 20. In this embodiment, the control unit 20 selects the first element image D1E that is positioned closest to the center of the overlapping region DA. Alternatively, the control unit 20 may select the first element image D1E positioned furthest to the right in the X-axis direction among the first element images D1E placed in the overlapping region DA. Furthermore, the selection of the first reference image D1S may be performed manually by the operator, who selects one of the first element images D1E placed in the overlapping region DA. As shown in Figure 6(C), in the first selection step, C 1Assume that the first element image D1E, which is assigned the identification code 6, is selected as the first reference image D1S. In Figure 6(C), C 1 To explain that the first element image D1E, which is assigned the identification code 6, was selected as the first reference image D1S, C 1 The contour of the first element image D1E, which is assigned the identification symbol 6, is shown thicker than the contours of the other first element images D1E. Below, C 1 The first reference image D1S, which is the first element image D1E to which the identification code 6 is assigned, is simply C 1 It may be indicated as 6. Figure 9 shows the first distance difference table provided in the storage unit 21. The control unit 20 selects C as the first reference image D1S in S3. 1 By storing 6 in the first reference image column of the first distance difference table, C is set as the first reference image D1S. 1 Remember that you selected 6.

[0044] Next, the control unit 20 performs the first calculation step (S5). In the first calculation step, the control unit 20 calculates the first distance difference, which is the difference between the distance indicated by the distance information corresponding to the first reference image D1S and the distance indicated by the distance information corresponding to the first surrounding image D1M, which is the first element image D1E arranged around the first reference image D1S. The multiple bar materials B included in the imaging target F are round steel bars, and their end faces Ba are circular. Therefore, around a certain end face Ba, there are often six other end faces Ba arranged so as to surround that end face Ba. In the first depth image D1 shown in Figure 6(C), C 1 Six first element images D1E are arranged around 6. "C" is arranged around the first reference image D1S. 1 7", C 1 3", C 1 2", C 1 5", C 1 9", C 1 10 Each of the first element images D1E corresponding to " is defined as the first surrounding image D1M. In this embodiment, the first surrounding image D1M is defined as the first element image D1E located in the overlapping region DA of the first depth image D1. Hereafter, C 1 7,C1 3,C 1 2,C 1 5,C 1 9,C 1 10 Each of the first surrounding image D1M, which is the first element image D1E to which the identification symbol is assigned, is simply C 1 7,C 1 3,C 1 2,C 1 5,C 1 9,C 1 10 This is sometimes indicated.

[0045] In the first calculation step, the control unit 20 first calculates the first reference image D1S(C 1 6) The distance indicated by the distance information corresponding to C 1 Obtain the distance in the Z-axis direction from the TOF camera T1 to the end face Ba corresponding to 6. 1 The distance indicated by the distance information corresponding to 6 is "Z 1 It is represented as "6". 1 6 is the first reference image D1S(C 1 6) may be the average value of the distances obtained from the distance information corresponding to each of the pixels that make up the component. Also, Z 1 6 is the first reference image D1S(C 1 6) The distance may also be obtained from the distance information corresponding to the pixels that make up the central part of the image.

[0046] The control unit 20 selects the first reference image D1S from the first surrounding image D1M, C 1 C is placed to the right of 6. 1 Obtain the distance indicated by the distance information corresponding to 7. C 1 The distance indicated by the distance information corresponding to 7 is "Z 1 This is represented as "7". In this embodiment, the control unit 20 is C 1 The first surrounding image D1M, which is assigned the identification symbol 7, is the first reference image D1S, C 1 It is defined as being at a 0° position relative to 6. Note that depending on the binding state of multiple rods B, C 1 C is directly to the right of 6. 1 It is not guaranteed that 7 will be provided. In this embodiment, the control unit 20 is C1 The center point of 6 and C 1 If the line connecting the center point of 7 has an inclination of 0° ± α° with respect to the X-axis, it is defined as being at the 0° position. In this embodiment, α = 15. The value of α may be changed as appropriate, and may be less than 15 or greater than 15.

[0047] The control unit 20 is Z 1 From the value of 7 to Z 1 The value obtained by subtracting 6 is derived. The control unit 20 stores this value in the storage unit 21 as the first distance difference at 0°. As shown in Figure 9, the first distance difference table is C, which is the first reference image D1S. 1 C is the first peripheral image D1M, located at 0° relative to 6. 1 It has a field for storing the first distance difference value for 7. The control unit 20 puts C in this field. 1 C is at a 0° angle to 6. 1 The first distance difference for 7 is "Z 1 7-Z 1 By storing the value "6", "Z 1 7-Z 1 The value "6" is stored in the storage unit 21.

[0048] Next, the control unit 20 selects the first reference image D1S from the first surrounding image D1M, C 1 C is positioned diagonally to the upper right of 6. 1 Retrieve the distance indicated by the distance information corresponding to 3. C 1 The distance indicated by the distance information corresponding to 3 is "Z 1 It is represented as "3". 1 3 is defined as C at the 0° position. 1 For 7, the first reference image D1S is C 1 It is positioned at a location that approximates a position obtained by rotating 60° counterclockwise from 6 as the reference point. In this embodiment, the control unit 20 is located at C 1 The first surrounding image D1M, which is assigned the identification symbol 3, is the first reference image D1S, C 1 It is defined as being at a 60° position relative to 6. In this embodiment, the control unit 20 is C 1 The center point of 6 and C 1When the straight line connecting the center point of 3 has an inclination of 60° ± α° with respect to the X-axis, it is defined as being at the 60° position.

[0049] The control unit 20 is Z 1 Derives a value obtained by subtracting the value of Z from the value of 3. The control unit 20 is C 1 6. The control unit 20 stores the value of the first distance difference "Z 1 3 - Z 1 6" for C at the 60° position with respect to 6 in the first distance difference table, thereby causing the value of "Z 1 3 - Z 1 6" to be stored in the storage unit 21. 1 3 - Z 1 6" to be stored in the storage unit 21.

[0050] Subsequently, similarly, the control unit 20 defines each of the other first surrounding images D1M, namely C 1 2, C 1 5, C 1 9, C 1 10 to be at positions of 120°, 180°, 240°, and 300° respectively with respect to C, which is the first reference image D1S 1 6. The control unit 20 determines that when the straight line connecting the center point of C 1 6 and the center points of C 1 2, C 1 5, C 1 9, C 1 10 has inclinations of 120° ± α°, 180° ± α°, 240° ± α°, and 300° ± α° with respect to the X-axis, they are respectively at positions of 120°, 180°, 240°, and 300°.

[0051] Also, the distances indicated by the distance information corresponding to each of C 1 2, C 1 5, C 1 9, C 1 10 are represented as "Z 1 2", "Z 1 5", "Z 1 9", "Z 1 10 ". The control unit 20 is C 1 2, C 1 5, C1 9,C 1 10 Obtain the distance indicated by the distance information corresponding to each, and Z 1 2,Z 1 5,Z 1 9,Z 1 10 The control unit 20 will be "Z 1 2-Z 1 6" Z 1 5-Z 1 6" Z 1 9-Z 1 6" Z 1 10 -Z 1 Each of the values ​​of "6" is C 1 The values ​​of the first distance difference corresponding to the first surrounding image D1M at positions of 120°, 180°, 240°, and 300° relative to 6 are stored in the first distance difference table and then stored in the storage unit 21.

[0052] Next, the control unit 20 performs the second selection step (S6). In the second selection step, the control unit 20 selects one of the multiple second element images D2E in the second depth image D2 as the second reference image D2S. In this embodiment, the control unit 20 selects one second element image D2E from among the multiple second element images D2E arranged in the overlapping region DA of the second depth image D2 as the second reference image D2S. The selection of the second reference image D2S is performed automatically by the control unit 20. In this embodiment, the control unit 20 selects the second element image D2E located in the leftmost position among the second element images D2E arranged in the overlapping region DA.

[0053] As shown in Figure 7(D-1), the second selection step is C 2 Assume that the second element image D2E, which is assigned the identification code 4, is selected as the second reference image D2S. In Figure 7(D-1), C 2 To explain that the second element image D2E, which is assigned the identification symbol 4, was selected as the second reference image D2S, C 2 The contour of the second element image D2E with the identification mark 4 is shown thicker than the contours of other second element images D2E. Below, C2 The second element image D2E, which has been assigned the identification code 4, is simply C 2 It may be indicated as 4. Figure 10 shows the second distance difference table provided in the storage unit 21. The control unit 20 selects C as the second reference image D2S in S6. 2 By storing 4 in the second reference image column of the second distance difference table, C is set as the second reference image D2S. 2 Remember that you selected 4.

[0054] Next, the control unit 20 performs a second calculation step (S8). In the second calculation step, the control unit 20 calculates a second distance difference, which is the difference between the distance indicated by the distance information corresponding to the second reference image D2S and the distance indicated by the distance information corresponding to the second surrounding image D2M, which is the second element image D2E arranged around the second reference image D2S. In the second depth image D2 shown in Figure 7(D-1), the second reference image D2S is C 2 Multiple second-element images D2E are arranged around 4. These are second-element images D2E arranged around the second reference image D2S, and are located in the overlapping region DA, "C 2 5", C 2 1", C 2 Let each of the second element images D2E corresponding to "8" be the second surrounding image D2M. Hereafter, C 2 5,C 2 1,C 2 Each of the second-element images D2E, to which an identification symbol of 8 is assigned, is simply C 2 5,C 2 1,C 2 It may be indicated by an identification code as 8th grade.

[0055] In the second calculation step, the control unit 20 first calculates the second reference image D2S(C 2 4) The distance indicated by the distance information corresponding to C 2 Obtain the distance in the Z-axis direction from the TOF camera T2 to end face Ba corresponding to 4. 2 The distance indicated by the distance information corresponding to 4 is "Z 2 It is represented as "4". 2 4 is the second reference image D2S(C 24) may be the average value of the distances obtained from the distance information corresponding to each of the pixels that make up the component. Also, Z 2 4 is the second reference image D2S(C 2 4) The distance may also be obtained from the distance information corresponding to the pixels that make up the central part of the image.

[0056] The control unit 20 controls the second reference image D2S, which is C, from the second ambient image D2M. 2 C is positioned at a 0° angle to 4. 2 Obtain the distance indicated by the distance information corresponding to 5. Regarding "placed at position x°", as in the first calculation step, the second surrounding image D2M, which is placed to the right of the second reference image D2S, is considered to be "placed at position 0°". In addition, the other second surrounding images D2M, which are placed to surround the second reference image D2S, are defined to be at positions 60°, 120°, 180°, 240°, and 300° in counterclockwise order. If they are within ±α° of 0°, 60°, 120°, 180°, 240°, and 300°, they are considered to be at positions 0°, 60°, 120°, 180°, 240°, and 300° respectively.

[0057] The second reference image is D2S, which is C 2 C is positioned at a 0° angle to 4. 2 The distance indicated by the distance information corresponding to 5 is "Z 2 It is represented as "5". The control unit 20 is Z 2 From the value of 5 to Z 2 The value obtained by subtracting 4 is derived. The control unit 20 stores this value in the storage unit 21 as the second distance difference at 0°. Figure 10 shows the second reference image D2S, C 2 C is the second peripheral image D2M, located at 0° relative to 4. 2 The second distance difference table is shown, which has a column for storing the second distance difference value for 5. The storage unit 21 includes the second distance difference table, C 2 C is at a 0° angle to 4. 2 The second distance difference for 5 is "Z 2 5-Z 2 Store the value "4".

[0058] The control unit 20 controls the second reference image D2S, C 2 C is located at ±α° for 60° and 300° relative to 4. 2 1,C 2 The same applies to 8 as well. 2 1,C 2 Obtain the distance indicated by the distance information corresponding to each of the 8, and set that distance to Z 2 1,Z 2 Let it be 8. The control unit 20 is C 2 C is the second peripheral image D2M, positioned at a 60° angle to 4. 2 The second distance difference for 1 is "Z 2 1-Z 2 The value of "4" is stored in the second distance difference table. The control unit 20 then... 2 C is the second surrounding image D2M, positioned at a 300° angle to 4. 2 The second distance difference for 8 is "Z 2 8-Z 2 The value "4" is stored in the second distance difference table. Note that the second reference image D2S is C 2 Since there are no second surrounding images D2M positioned at 120°, 180°, and 240° relative to 4, the corresponding second distance difference values ​​are not stored in the second distance difference table.

[0059] Next, the control unit 20 performs a third calculation step (S9). In the third calculation step, the control unit 20 calculates the relative distance difference, which is the difference between the first distance difference and the second distance difference.

[0060] In the first calculation step (S5) described above, the control unit 20 calculates the difference between the distance indicated by the distance information corresponding to the first reference image D1S and the distance indicated by the distance information corresponding to the first surrounding image D1M, which is located at positions 0°, 60°, 120°, 180°, 240°, and 300° relative to the first reference image D1S, as the first distance difference at each position of 0°, 60°, 120°, 180°, 240°, and 300°. Based on this, as the processing in the third calculation step, the control unit 20 calculates the difference between the distance indicated by the distance information corresponding to the second reference image D2S and the distance indicated by the distance information corresponding to the second surrounding image D2M, which is located at positions 0°, 60°, 120°, 180°, 240°, and 300° relative to the second reference image D2S, as the second distance difference at each position of 0°, 60°, 120°, 180°, 240°, and 300°.

[0061] Specifically, the control unit 20 controls the first reference image D1S, C 1 C is the first periphery image D1M, positioned at 0° relative to 6. 1 The first distance difference derived for 7 is "Z 1 7-Z 1 From the value of "6", the second reference image D2S is C 2 C is the second peripheral image D2M, positioned at 0° relative to 4. 2 The second distance difference derived for 5 is "Z 2 5-Z 2 The relative distance difference is derived by subtracting the value of "4". Specifically, the calculation shown in the following equation (1) is performed, which gives the first reference image D1S(C 1 6) and the second reference image D2S(C 2 4) The first surrounding image D1M(C) is positioned at 0° relative to each of the four images. 1 7) and second peripheral image D2M(C 2 The absolute values ​​of the first distance difference and the second distance difference related to 5) are derived. The control unit 20 stores the derived values ​​in the storage unit 21.

[0062] |(Z 1 7-Z 1 6)-(Z 2 5-Z 2 4) | ···(1)

[0063] Similarly, the control unit 20 performs the calculation shown in the following equation (2) to obtain the first reference image D1S(C 1 6) and the second reference image D2S(C 2 4) The first surrounding image D1M(C) is positioned at a 60° angle to each other. 1 3) and second peripheral image D2M(C 2 1) The absolute values ​​of the first distance difference and the second distance difference related to 1) are derived. Furthermore, the control unit 20 performs the calculation shown in the following equation (3) to obtain the first reference image D1S(C 1 6) and the second reference image D2S(C 2 4) The first surrounding image D1M(C) is positioned at a 300° angle to each of the other images. 1 10 ) and second peripheral image D2M(C 2 The absolute values ​​of the first distance difference and the second distance difference related to 8) are derived. The control unit 20 stores each of the calculated relative distance difference values ​​in the storage unit 21.

[0064] |(Z 1 3-Z 1 6)-(Z 2 1-Z 2 4) | ...(2) |(Z 1 10 -Z 1 6)-(Z 2 8-Z 2 4) | ...(3)

[0065] Next, the control unit 20 selects all second element images D2E located in the overlapping region DA of the second depth image D2, calculates the second distance difference, and determines whether it has calculated the relative distance difference based on the calculated second distance difference (S11). The control unit 20 makes this determination by checking whether the identification symbols corresponding to all second element images D2E located in the overlapping region DA of the second depth image D2 are stored in the second reference image column of the second distance difference table. If there are second element images D2E that are not stored in the second reference image column of the second distance difference table (S11: NO), the control unit 20 executes the second selection step (S6). In this S6 process, the control unit 20 selects the second element images D2E located in the overlapping region DA of the second depth image D2 that have not yet been selected as the second reference image D2S. Then, based on the selected second reference image D2S, it sequentially executes the second calculation step (S8) and the third calculation step (S9).

[0066] Specifically, as part of the processing in the second selection step, the control unit 20 selects the second reference image D2S, which is one of the second element images D2E placed in the overlapping region DA of the second depth image D2, that was not selected in the previous second selection step. 2 Select 5. In Figure 7 (D-2), C 2 To explain that the second element image D2E, which is assigned the identification code 5, was selected as the second reference image D2S, C 2 The contour of the second element image D2E assigned the identification code 5 is shown thicker than the contours of other second element images D2E.

[0067] Next, the control unit 20 performs the second calculation step, C 2 The second element image D2E is positioned to surround 5, and C is positioned in the overlapping region DA. 2 6,C 2 2,C 2 1,C 2 4,C 2 8,C 2 Each of the second element images D2E corresponding to 9 is made into a second surrounding image D2M. The control unit 20 controls C 2 6,C 2 2,C 2 1,C2 4,C 2 8,C 2 Each of the second element images D2E corresponding to 9 is the second reference image D2S, C 2 It is defined that 5 is positioned at 0°, 60°, 120°, 180°, 240°, and 300°, respectively. The control unit 20 controls the second reference image D2S, C 2 5 and the second peripheral image D2M is C 2 6,C 2 2,C 2 1,C 2 4,C 2 8,C 2 Z is the distance indicated by the distance information corresponding to 9. 2 5,Z 2 6,Z 2 2,Z 2 1,Z 2 4,Z 2 8,Z 2 The control unit 20 obtains each of the 9 values. The control unit 20 determines the second reference image D2S of the second distance difference table C 2 In the columns corresponding to 5, for 0°, 60°, 120°, 180°, 240°, and 300°, the second distance difference is "Z 2 6-Z 2 5", Z 2 2-Z 2 5", Z 2 1-Z 2 5", Z 2 4-Z 2 5", Z 2 8-Z 2 5", Z 2 9-Z 2 Calculate and store each value of "5".

[0068] Next, the control unit 20 processes the first reference image D1S, C, as part of the processing in the third calculation step. 1 C is the first periphery image D1M, positioned at 0° relative to 6. 1 The first distance difference derived for 7 is "Z 1 7-Z 1 From the value of "6", the second reference image D2S is C 2 C is the second peripheral image D2M, positioned at 0° relative to 5. 2The second distance difference derived for 6 is "Z 2 6-Z 2 The relative distance difference is derived by subtracting the value of "5". Similarly, the control unit 20 also derives the relative distance difference for the first surrounding image D1M and the second surrounding image D2M, which are positioned at 60°, 120°, 180°, 240°, and 300° from the first reference image D1S and the second reference image D2S. Specifically, by performing the calculations shown in equations (4) to (9) below, the control unit 20 similarly derives the relative distance difference for the first surrounding image D1M and the second surrounding image D2M, which are positioned at 0°, 60°, 120°, 180°, 240°, and 300° from the first reference image D1S and the second reference image D2S. The control unit 20 stores each of the calculated relative distance difference values ​​in the storage unit 21.

[0069] |(Z 1 7-Z 1 6)-(Z 2 6-Z 2 5) | ...(4) |(Z 1 3-Z 1 6)-(Z 2 2-Z 2 5) | ···(5) |(Z 1 2-Z 1 6)-(Z 2 1-Z 2 5) | ...(6) |(Z 1 5-Z 1 6)-(Z 2 4-Z 2 5) | ...(7) |(Z 1 9-Z 1 6)-(Z 2 8-Z 2 5) | ...(8) |(Z 1 10 -Z 1 6)-(Z 2 9-Z 2 5) | ···(9)

[0070] Next, the control unit 20 has not selected all the second element images D2E located in the overlapping region DA of the second depth image D2 and calculated the second distance difference (S11:NO). Therefore, as the processing for the second selection step, it selects the second reference image D2S, which is C, from among the second element images D2E located in the overlapping region DA of the second depth image D2 that were not selected in the previous second selection steps. 2 Select 6. In Figure 7 (D-3), C 2 To explain that the second element image D2E, which is assigned the identification code 6, was selected as the second reference image D2S, C 2 The contour of the second element image D2E assigned the identification code 6 is shown thicker than the contours of other second element images D2E.

[0071] Next, the control unit 20 performs the second calculation step, C 2 The second element image D2E is arranged to surround 6, and C is located in the overlapping region DA. 2 7,C 2 3,C 2 2,C 2 5,C 2 9,C 2 10 Each of the corresponding items is designated as the second surrounding image D2M. The control unit 20 controls C 2 7,C 2 3,C 2 2,C 2 5,C 2 9,C 2 10 Each of the corresponding second peripheral images D2M is the second reference image D2S, C 2 It is defined that 6 is positioned at 0°, 60°, 120°, 180°, 240°, and 300° respectively. The control unit 20 controls the second reference image D2S, C 2 6 and the second peripheral image D2M is C 2 7,C 2 3,C 2 2,C 2 5,C 2 9,C 2 10 The distance Z is the distance indicated by the distance information corresponding to the distance. 2 6,Z 2 7,Z 23,Z 2 2,Z 2 5,Z 2 9,Z 2 10 The control unit 20 obtains each value of the second reference image D2S of the second distance difference table C 2 In the columns corresponding to 6, for 0°, 60°, 120°, 180°, 240°, and 300°, the second distance difference is "Z 2 7-Z 2 6", Z 2 3-Z 2 6", Z 2 2-Z 2 6", Z 2 5-Z 2 6", Z 2 9-Z 2 6", Z 2 10 -Z 2 Calculate and store each value of "6".

[0072] Note C 2 7,C 2 3,C 2 10 Regarding these, a portion of them is a second element image D2E located outside the overlapping region DA. As in this embodiment, such a second element image D2E is also a second reference image D2S C 2 The second peripheral image D2M of 6 may or may not be added.

[0073] Next, the control unit 20 processes the first reference image D1S, C, as part of the processing in the third calculation step. 1 C is the first periphery image D1M, positioned at 0° relative to 6. 1 The first distance difference derived for 7 is "Z 1 7-Z 1 From the value of "6", the second reference image D2S is C 2 C is the second peripheral image D2M, positioned at 0° relative to 6. 2 The second distance difference derived for 7 is "Z 2 7-Z 2The relative distance difference is derived by subtracting the value of "6". Similarly, the control unit 20 also derives the relative distance difference for the first surrounding image D1M and the second surrounding image D2M, which are positioned at 60°, 120°, 180°, 240°, and 300° from the first reference image D1S and the second reference image D2S. Specifically, by performing the calculations shown in equations (10) to (15) below, the control unit 20 similarly derives the relative distance difference for the first surrounding image D1M and the second surrounding image D2M, which are positioned at 0°, 60°, 120°, 180°, 240°, and 300° from the first reference image D1S and the second reference image D2S. The control unit 20 stores each of the calculated relative distance difference values ​​in the storage unit 21.

[0074] |(Z 1 7-Z 1 6)-(Z 2 7-Z 2 6) | ...(10) |(Z 1 3-Z 1 6)-(Z 2 3-Z 2 6) | ...(11) |(Z 1 2-Z 1 6)-(Z 2 2-Z 2 6) | ...(12) |(Z 1 5-Z 1 6)-(Z 2 5-Z 2 6) | ...(13) |(Z 1 9-Z 1 6)-(Z 2 9-Z 2 6) | ...(14) |(Z 1 10 -Z 1 6)-(Z 2 10 -Z 2 6) | ...(15)

[0075] From this point onward, the control unit 20 selects all second element images D2E located in the overlapping region DA of the second depth image D2, calculates the second distance difference, and repeats the processes of S6, S8, and S9 until it calculates the relative distance difference based on the calculated second distance difference. If the control unit 20 stores identification symbols corresponding to all second element images D2E located in the overlapping region DA of the second depth image D2 in the second reference image column of the second distance difference table, it determines that it has selected all second element images D2E located in the overlapping region DA of the second depth image D2, calculated the second distance difference, and calculated the relative distance difference based on the calculated second distance difference (S11: YES).

[0076] In this case, the control unit 20 performs a specific step (S12). In the specific step, the control unit 20 identifies a second reference image D2S in which the relative distance difference is smallest.

[0077] If the end face Ba corresponding to the first reference image D1S in the first depth image D1 and the end face Ba corresponding to the second reference image D2S in the second depth image D2 correspond to the same end face Ba, then the first surrounding image D1M and the second surrounding image D2M, which are positioned in the same direction with respect to the first reference image D1S and the second reference image D2S respectively, will also correspond to the same end face Ba. In such a case, the first distance difference related to the first surrounding image D1M, which is positioned in the same direction with respect to the first reference image D1S, and the second distance difference related to the second surrounding image D2M, which is positioned in the same direction with respect to the second reference image D2S, will be equal to or very close to each other. The inventors of this disclosure have discovered such a law. Based on this law, in order to determine which of the second reference images D2S in the second depth image D2 corresponds to the end face Ba corresponding to the first reference image D1S in the first depth image, the control unit 20 performs a process in a specific step to identify the second reference image D2S that has the smallest relative distance difference.

[0078] The control unit 20 may identify the one with the smallest relative distance difference among those calculated in the third calculation step (S9) described above, and determine that the second reference image D2S corresponding to the identified value corresponds to the end face Ba corresponding to the first reference image D1S. On the other hand, since the arrangement of the end faces Ba of the rod B in the imaging target F varies, there may be multiple combinations of end faces Ba that are arranged adjacently in similar positional relationships in the imaging target F. Even in such a case, if the end face Ba corresponding to the first reference image D1S in the first depth image D1 and the end face Ba corresponding to the second reference image D2S in the second depth image D2 correspond to the same end face Ba, the relative distance corresponding to the second surrounding image D2M, which is arranged in any direction with respect to the second reference image D2S, will be the same value or a very similar value. In order to accurately determine whether the end face Ba corresponding to the first reference image D1S in the first depth image D1 and the end face Ba corresponding to the second reference image D2S in the second depth image D2 correspond to the same end face Ba, the control unit 20 calculates the sum of multiple relative distance differences calculated for one second reference image D2S in the third calculation step, and identifies the second reference image D2S that has the smallest calculated sum as the end face Ba corresponding to the first reference image D1S.

[0079] Specifically, the control unit 20 controls the second reference image D2S, which is stored in column No. 1 of the second distance difference table. 2 For 4, calculate the sum of the values ​​derived from the aforementioned equations (1), (2), and (3), and determine the result C 2 The sum of the relative distance differences corresponding to 4 is stored in the storage unit 21. The control unit 20 also stores C, which is the second reference image D2S stored in column No. 2 of the second distance difference table. 2 For 5, calculate the sum of the values ​​derived from the aforementioned equations (4), (5), (6), (7), (8) and (9), and set the result C 2 The sum of the relative distance differences corresponding to 5 is stored in the storage unit 21. The control unit 20 also stores C, which is the second reference image D2S stored in column No. 3 of the second distance difference table. 2For 6, calculate the sum of the values ​​derived by the aforementioned equations (10), (11), (12), (13), (14), and (15), and set the result C 2 The sum of the relative distance differences corresponding to 6 is stored in the memory unit 21.

[0080] The control unit 20 performs this process for all second reference images D2S stored in the second distance difference table and compares the sum of relative distance differences corresponding to each second reference image D2S. Based on the comparison results, the control unit 20 identifies which second reference image D2S corresponds to the one with the smallest sum of relative distance differences. In this embodiment, C 2 Assume that the sum of relative distance differences corresponding to 5 is the smallest value compared to the sum of other relative distance differences. The control unit 20 determines that C is the first reference image D1S in the first depth image D1. 1 End face Ba corresponding to 6, and C in the second depth image D2 2 Identify that end face Ba corresponding to 5 is the same end face Ba.

[0081] Next, the control unit 20 performs a synthesis step (S13). In the synthesis step, the control unit 20 generates a third depth image D3, which is a single depth image obtained by combining the first depth image D1 and the second depth image D2. Specifically, the control unit 20 uses the first reference image D1S in the first depth image D1 as a reference, and aligns the second depth image D2 with the second element image D2E identified in a specific step as a reference, thereby synthesizing the first depth image D1 and the second depth image D2 to generate the third depth image D3.

[0082] Specifically, as shown in Figure 8(E-1), the control unit 20 controls the first depth image D1, which is the first reference image D1S. 1 6 is used as the reference for alignment. Also, as shown in Figure 8(E-2), the control unit 20 uses the second element image D2E, which is identified in a specific step, as the second depth image D2. 2 5 is used as the reference for alignment. Then, the control unit 20 is C 1 6 and C 2The first depth image D1 and the second depth image D2 are superimposed so that they overlap with 5. When the first depth image D1 and the second depth image D2 are superimposed, the first element image D1E and the second element image D2E contained in their respective overlapping regions DA will overlap. For the overlapping parts, the third depth image D3 may be generated by deleting either the overlapping first element image D1E or the second element image D2E. In this embodiment, the third depth image D3 shown in Figure 8(E-3) is generated by deleting the second element image D2E from the overlapping first element image D1E and the second element image D2E. Alternatively, the third depth image D3 may be generated by deleting the first element image D1E from the overlapping first element image D1E and the second element image D2E. The control unit 20 terminates the image synthesis process.

[0083] The control unit 20 can quickly and accurately count the number of end faces Ba of the bar B included in the imaging target F by counting the number of images corresponding to the end faces Ba of the bar B that appear in the third depth image D3 obtained in this way. The calculation process performed in the image synthesis process according to this disclosure mainly consists of repeated addition and subtraction. For this reason, the control unit 20 can easily and quickly align the first depth image D1 and the second depth image D2 compared to when an alignment algorithm for a 3D point cloud such as ICP (Iterctive Closest Point) is used. Therefore, for example, when counting the number of bundled bar B in a steel bar manufacturing line in a factory, the image processing system 100 can be installed in the line and the control device 10 can execute the image synthesis process of this embodiment, thereby quickly and accurately counting the objects to be counted.

[0084] As described above, the control unit 20 of the control device 10 executes the image processing method according to the present disclosure by performing image synthesis processing. The control unit 20 executes an acquisition step (S1) to acquire an image of the imaging target F, which includes the end faces Ba of a plurality of rods B as a plurality of elements, from a first predetermined position P1, and a first depth image D1 which includes distance information from the first predetermined position P1 to each element for each first element image D1E representing each element, and a second depth image D2 which is an image of the imaging target F, which is taken from a second predetermined position P2, and a second depth image D2 which includes distance information from the second predetermined position P2 to each element for each second element image D2E representing each element. The control unit 20 executes a first selection step (S3) to select one of the plurality of first element images D1E in the first depth image D1 as the first reference image D1S. The control unit 20 performs a first calculation step (S5) to calculate a first distance difference, which is the difference between the distance indicated by the distance information corresponding to the first reference image D1S and the distance indicated by the distance information corresponding to the first surrounding image D1M, which is the first element image D1E arranged around the first reference image D1S. The control unit 20 performs a second selection step (S6) to select one of the multiple second element images D2E in the second depth image D2 as the second reference image D2S. The control unit 20 performs a second calculation step (S8) to calculate a second distance difference, which is the difference between the distance indicated by the distance information corresponding to the second reference image D2S and the distance indicated by the distance information corresponding to the second surrounding image D2M, which is the second element image D2E arranged around the second reference image D2S. The control unit 20 performs a third calculation step (S9) to calculate a relative distance difference, which is the difference between the first distance difference and the second distance difference. The control unit 20 performs a selection step (S12) to identify the second element image D2E that has the smallest relative distance difference by repeatedly performing the second selection step, the second calculation step and the third calculation step.

[0085] When an image target F, which includes the end faces Ba of multiple rods B as multiple elements, is imaged from a single location, the captured image may not include images corresponding to all elements due to some elements being in the blind spots of other elements. To resolve this problem, the image target F is imaged from multiple locations, and the captured images are combined to image all elements included in the image target F. In the image synthesis process of this embodiment, a first depth image D1 is obtained from a first predetermined position P1, and a second depth image D2 is obtained from a second predetermined position P2. Then, one of the multiple first element images D1E in the first depth image D1 is selected as the first reference image D1S, and a first distance difference is calculated based on the distance information corresponding to the first reference image D1S and the distance information corresponding to the first surrounding image D1M arranged around the first reference image D1S. For the second depth image D2, a second reference image D2S is selected from the second element images D2E, and a second distance difference is calculated based on the distance information corresponding to the second reference image D2S and the distance information corresponding to the second surrounding image D2M arranged around the second reference image D2S. If the first reference image D1S and the second reference image D2S correspond to the same element, the relative distance difference, which is the difference between the first distance difference and the second distance difference, approximates to 0. In the image synthesis process, the control unit 20 identifies the second reference image D2S that has the smallest relative distance difference. In this way, the control unit 20 can quickly perform the process for aligning the depth image without using jigs or the like by executing the image synthesis process according to this disclosure.

[0086] The first depth image D1 and the second depth image D2 are images captured so that their imaging areas overlap with each other. In the first selection step, the control unit 20 selects one of the multiple first element images D1E in the first depth image D1 that is located within the overlapping region DA where the imaging area of ​​the second depth image D2 overlaps with that of the first depth image D1 as the first reference image D1S. In the second selection step, the control unit 20 selects one of the multiple second element images D2E in the second depth image D2 that is located within the overlapping region DA as the second reference image D2S.

[0087] In this case, it is sufficient that the first depth image D1 and the second depth image D2 are captured so that their imaging areas overlap, and it is not necessary for the first depth image D1 and the second depth image D2 to capture the entirety of the target F. Since the constraints on the arrangement of imaging devices such as TOF cameras T1 and T2 that capture the first depth image D1 and the second depth image D2 are relaxed, the scope of application of the image synthesis process according to this embodiment is expanded.

[0088] In the first calculation step, the control unit 20 calculates the difference between the distance indicated by the distance information corresponding to the first reference image D1S and the distance indicated by the distance information corresponding to the first surrounding image D1M, which is positioned in a first direction relative to the first reference image D1S, as the first distance difference. In the second calculation step, the control unit 20 calculates the difference between the distance indicated by the distance information corresponding to the second reference image D2S and the distance indicated by the distance information corresponding to the second surrounding image D2M, which is positioned in a first direction relative to the second reference image D2S, as the second distance difference.

[0089] In this case, the first distance difference is calculated based on the distance information corresponding to the first reference image D1S and the distance information corresponding to the first surrounding image D1M, which is positioned in the first direction relative to the first reference image D1S. Furthermore, the second distance difference is calculated based on the distance information corresponding to the second reference image D2S and the distance information corresponding to the second surrounding image D2M, which is positioned in the first direction relative to the second reference image D2S. This increases the likelihood that the element corresponding to the second reference image D2S, which has the smallest relative distance difference, coincides with the element corresponding to the first reference image D1S.

[0090] In the first calculation step, the control unit 20 calculates the difference between the distance indicated by the distance information corresponding to the first reference image D1S and the distance indicated by the distance information corresponding to the first surrounding image D1M, which is positioned in a second direction relative to the first reference image D1S, as the first distance difference. In the second calculation step, the control unit 20 calculates the difference between the distance indicated by the distance information corresponding to the second reference image D2S and the distance indicated by the distance information corresponding to the second surrounding image D2M, which is positioned in a second direction relative to the second reference image D2S, as the second distance difference. In the third calculation step, the control unit 20 calculates the relative distance difference relating to the difference between the first distance difference and the second distance difference relating to the first direction, and the relative distance difference relating to the difference between the first distance difference and the second distance difference relating to the second direction. In the identification step, the control unit 20 calculates the sum of the multiple relative distance differences calculated in the third calculation step and identifies the second reference image D2S for which the sum is smallest.

[0091] In this case, the likelihood that the first reference image D1S and the second reference image D2S correspond to the same elements is further increased.

[0092] The imaging target F consists of multiple bundled rods B, and each element included in the imaging target F is the end face Ba of each of the multiple rods B.

[0093] One possible method for counting the number of bundled rods B is to image the bundled rods B and count the end faces Ba of the bundled rods B included in the imaged images. However, if the end faces Ba of the bundled rods B are uneven, the number of rods B may not be accurately counted by imaging from a single location. The control unit 20 that performs the image synthesis processing according to this embodiment uses the bundled rods B as the imaging target F, and uses the images corresponding to each end face Ba of the bundled rods B included in the first depth image D1 and the second depth image D2 as the first element image D1E and the second element image D2E, respectively, and performs the respective processing. As a result, the end face Ba of the rod B corresponding to the second reference image D2S, where the relative distance difference is smallest, is identified as the end face Ba of the rod B corresponding to the first reference image D1S, so that the alignment of the first depth image D1 and the second depth image D2 can be performed quickly.

[0094] In the first selection step, the control unit 20 selects one of a plurality of first element images D1E corresponding to each end face Ba of the plurality of bar materials B as the first reference image D1S. In the first calculation step, the control unit 20 calculates the first distance difference using the first element image D1E corresponding to the end face Ba adjacent to the end face Ba corresponding to the first reference image D1S as the first surrounding image D1M. In the second selection step, the control unit 20 selects one of a plurality of second element images D2E corresponding to each end face Ba as the second reference image D2S. In the second calculation step, the control unit 20 calculates the second distance difference using the second element image D2E corresponding to the end face Ba adjacent to the end face Ba corresponding to the second reference image D2S as the second surrounding image D2M.

[0095] The end faces Ba of multiple bundled rods B are often arranged adjacently such that one end face Ba is surrounded by another. The positional relationship between these end faces Ba is the same as the positional relationship between the first element image D1E and the second element image D2E in the first depth image D1 and the second depth image D2. The control unit 20, which performs the image synthesis processing according to this embodiment, selects one of the multiple first element images D1E included in the first depth image D1 as the first reference image D1S, and calculates the first distance using the first element image D1E arranged adjacent to the first reference image D1S as the first surrounding image D1M. The control unit 20 also selects one of the multiple second element images D2E included in the second depth image D2 as the second reference image D2S, and calculates the second distance using the second element image D2E arranged adjacent to the second reference image D2S as the second surrounding image D2M. This identifies the end face Ba of rod B corresponding to the second reference image D2S, where the relative distance difference is smallest, as the same end face Ba of rod B corresponding to the first reference image D1S. Thus, the alignment of the first depth image D1 and the second depth image D2 can be performed accurately and quickly.

[0096] This disclosure is not limited to the embodiments described above and in the drawings. For example, the following embodiments are also included in the technical scope of this disclosure, and various modifications can be made without departing from the spirit of the disclosure.

[0097] (1) The acquisition step, first selection step (S1), first calculation step (S3), second selection step (S5), second calculation step (S6), third calculation step (S8), and identification step (S9) are not limited to being executed in this order. For example, the second selection step and the second calculation step may be executed in this order, and then the first selection step and the first calculation step may be executed in this order. Also, the synthesis step (S13) may be a process executed by another program, and the image synthesis process may not include the synthesis step (S13). Furthermore, the first depth image D1 and the second depth image D2, after the processing corresponding to the element identification step (S2) has been performed, may be acquired in the acquisition step.

[0098] (2) The image processing system 100 is configured to acquire a first depth image D1 taken from a first predetermined position P1 and a second depth image D2 taken from a second predetermined position P2 of the imaging target F. For example, the first depth image D1 and the second depth image D2 may be stored in a server (not shown), and the image processing system 100 may be configured so that the control device 10 acquires the first depth image D1 and the second depth image D2 from the server. In such a case, the image processing system 100 does not need to include TOF cameras T1 and T2.

[0099] (3) The image processing system 100 may include only one TOF camera, TOF camera T1, and may be configured such that TOF camera T1 captures a first depth image D1 from a first predetermined position P1, and then TOF camera T1 moves to a second predetermined position P2 to capture a second depth image D2.

[0100] (4) The control device 10 does not need to acquire the first depth image D1 and the second depth image D2. For example, the image processing method according to this disclosure may be executed by the control unit 20 performing the processing S1 or below on the first depth image D1 and the second depth image D2 that exist on the network.

[0101] (5) In the above embodiment, in S11, the control unit 20 selects all second element images D2E located in the overlapping region DA of the second depth image D2, calculates the second distance difference, and determines whether it has calculated the relative distance difference based on the calculated second distance difference. In this regard, in S11, it is not necessary to select "all" second element images D2E to calculate the second distance difference and determine whether it has calculated the relative distance difference based on the calculated second distance difference. For example, if the control unit 20 selects the first reference image D1S from among the first element images D1E located in the central part of the overlapping region DA of the first depth image D1 in the first selection step, it may determine "YES" in S11 when it repeatedly selects multiple second element images D2E located in the central part of the Y-axis direction from among the second element images D2E located in the overlapping region DA of the second depth image D2 as the second reference image D2S.

[0102] (6) The CPU in the control unit 20 functions as a processor that performs image synthesis processing by loading the program stored in the HDD included in the storage unit 21 onto the RAM included in the storage unit 21. A general-purpose processor may be used as the CPU. A microcomputer, ASIC, FPGA, etc. may be used as a processor instead of the CPU.

[0103] (7) The program for performing the image synthesis process may be downloaded, for example, from a server connected to a network (not shown) via the external device connection unit 28 and stored in the storage unit 21. In this case, the program for performing the crystal grain size evaluation process only needs to be stored on a non-temporary storage medium such as an HDD provided on the server.

[0104] (8) The image synthesis process may be distributed and performed by multiple electronic devices (i.e., multiple CPUs). For example, part of the image synthesis process may be performed by the control units of the TOF cameras T1 and T2, or by other servers connected to the control device 10 via a network (not shown). [Explanation of Symbols]

[0105] 10 Control device 20 Control Unit 21 Memory section 100 Image Processing Systems B Bar material Ba end face D1 First Depth Image D1E First Element Image D1M First Surround Image D1S First Reference Image D2 Second Depth Image D2E (Dual-to-Effect) Second Element Image D2M Second Surround Image D2S Second Reference Image DA overlap area F: Target of imaging P1 First predetermined position P2 Second predetermined position T1, T2 TOF cameras

Claims

1. An acquisition step to acquire an image of an object to be imaged, which includes multiple elements, taken from a first predetermined position, and comprising a first depth image including distance information from the first predetermined position to each element for each first element image representing each element, and an acquisition step to acquire an image of the object to be imaged, which includes a second depth image including distance information from the second predetermined position to each element for each second element image representing each element, A first selection step of selecting one of the multiple first element images in the first depth image as the first reference image, A first calculation step of calculating a first distance difference, which is the difference between the distance indicated by distance information corresponding to the first reference image and the distance indicated by distance information corresponding to the first surrounding image, which is the first element image arranged around the first reference image; A second selection step involves selecting one of the multiple second element images in the second depth image as the second reference image, A second calculation step involves calculating a second distance difference, which is the difference between the distance indicated by the distance information corresponding to the second reference image and the distance indicated by the distance information corresponding to the second surrounding image, which is the second element image arranged around the second reference image. A third calculation step involves calculating the relative distance difference, which is the difference between the first distance difference and the second distance difference. The second selection step, the second calculation step, and the third calculation step are repeated to identify the second reference image in which the relative distance difference is smallest. Image processing methods including [specific details omitted].

2. The first depth image and the second depth image are images captured such that their imaging areas overlap. The first selection step involves selecting one of the multiple first element images in the first depth image that lies within the overlapping region where the second depth image and the imaging region overlap, as the first reference image. The image processing method according to claim 1, wherein the second selection step is to select one of the multiple second element images in the second depth image that is located within the overlapping region as the second reference image.

3. The first calculation step calculates the difference between the distance indicated by the distance information corresponding to the first reference image and the distance indicated by the distance information corresponding to the first surrounding image positioned in a first direction relative to the first reference image as the first distance difference. The image processing method according to claim 1, wherein the second calculation step calculates the difference between the distance indicated by distance information corresponding to the second reference image and the distance indicated by distance information corresponding to the second surrounding image positioned in the first direction relative to the second reference image as the second distance difference.

4. The first calculation step further calculates the difference between the distance indicated by the distance information corresponding to the first reference image and the distance indicated by the distance information corresponding to the first surrounding image positioned in a second direction relative to the first reference image, as the first distance difference. The second calculation step further calculates the difference between the distance indicated by the distance information corresponding to the second reference image and the distance indicated by the distance information corresponding to the second surrounding image positioned in the second direction relative to the second reference image, as the second distance difference. The third calculation step calculates the relative distance difference relating to the difference between the first distance difference and the second distance difference relating to the first direction, and the relative distance difference relating to the difference between the first distance difference and the second distance difference relating to the second direction, The image processing method according to claim 3, wherein the identifying step involves calculating the sum of a plurality of relative distance differences calculated by the third calculation step, and identifying the second reference image in which the sum is smallest.

5. The image processing method according to any one of claims 1 to 4, wherein the object to be imaged is a plurality of bundled rods, and each element included in the object to be imaged is each end face of the plurality of rods.

6. The first selection step involves selecting one of the multiple first element images corresponding to each end face of the multiple rods as the first reference image. The first calculation step calculates the first distance difference using the first element image corresponding to the end face adjacent to the end face corresponding to the first reference image as the first surrounding image, The second selection step involves selecting one of the plurality of second element images corresponding to each end face as the second reference image. The image processing method according to claim 5, wherein the second calculation step calculates the second distance difference using the second element image corresponding to an end face arranged adjacent to the end face corresponding to the second reference image as the second surrounding image.

Images