Image processing device, image processing method and program

Abstract

To provide an image processing device, an image processing method, and a program which are capable of highly accurately performing position estimation of an object based on image data.SOLUTION: An image processing device 10 estimates a position of an object 101 in an imaging area 105 on the basis of image data of the imaging area 105 which is set in an upstream side of a belt conveyor 100 by which the object 101 is moved. The image processing device 10 estimates a speed of the object 101 on the belt conveyor 100. When the object 101 satisfies a preset condition, the image processing device 10 estimates a position of the object 101 in a processing area 106 which is set in a downstream side of the belt conveyor 100 at a prescribed time on the basis of the detected position of the object 101 and the estimated speed of the object 101.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] At a production site, products are transported on a belt conveyor. Various systems have been proposed that track products on the belt conveyor and mark the tracked products with a projector or perform tasks on the tracked products with a robot. For example, Patent Document 1 describes a robot system that detects the position of an object transported by a conveyor and has a robot perform tasks on the object based on the detected position. The robot system disclosed in Patent Document 1 captures an image of an item transported by the conveyor and detects the position of the item through image processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-47511 Summary of the Invention [Problem to be solved by the invention]

[0004] Using the technology disclosed in Patent Document 1, the position of a target object can be detected and a robot can work on that target object. However, when detecting the position of an object using image processing, there is a risk that part of the robot may appear in the image. This may cause noise, which may reduce the accuracy of detecting the position of the target object using the image.

[0005] An example of an object of the present disclosure is to provide an image processing device, an image processing method, and a program that can accurately estimate the position of an object based on image data. [Means for solving the problem]

[0006] In order to achieve the above object, an image processing device according to one aspect of the present disclosure includes: a first position estimation unit that estimates a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves; a speed estimation unit that estimates a speed of the object on the path; a second position estimation unit that, when the object satisfies a set condition, estimates a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; The present invention is characterized in that it is provided with:

[0007] In order to achieve the above object, an image processing method according to one aspect of the present disclosure includes: estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves; estimating the velocity of the object along the path; If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; The present invention is characterized in that it is provided with:

[0008] Furthermore, in order to achieve the above object, a program according to one aspect of the present disclosure includes: On the computer, estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves; estimating the velocity of the object along the path; If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; The present invention is characterized in that the following is executed. [Effects of the Invention]

[0009] As described above, according to the technology of the present disclosure, it is possible to accurately estimate the position of an object based on image data. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a system in which an image processing apparatus according to an embodiment is used. [Figure 2] FIG. 2 is a block diagram showing the configuration of the image processing apparatus according to the embodiment. [Figure 3] FIG. 3 is a diagram for explaining a system for marking an object with a light-projecting marker, in which the image processing device according to the embodiment is used. [Figure 4] FIG. 4 is a block diagram showing a specific configuration of the image processing device. [Figure 5] FIG. 5 is a diagram for explaining the conversion from the coordinates in the imaging coordinate system to the coordinates in the three-dimensional coordinate system. [Figure 6] FIG. 6 is a diagram for explaining the conversion from the coordinates of the three-dimensional coordinate system to the coordinates of the projection imaging coordinate system. [Figure 7] FIG. 7 is a flow diagram showing the operation of the image processing device when performing calibration. [Figure 8] FIG. 8 is a flowchart showing the operation of the image processing device during position estimation. [Figure 9] FIG. 9 is a block diagram illustrating an example of a computer that realizes the image processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] First, an overview will be given to facilitate understanding of the embodiments to be described below.

[0012] 1 is a diagram illustrating a system that uses an image processing device 10 according to this embodiment. The image processing device 10 is used, for example, at a product production site to estimate the position of an object 101, such as a product being carried on a belt conveyor 100 and transported.

[0013] An imaging area 105 is set on the upstream side of the belt conveyor 100, and a processing area 106 is set on the downstream side. The imaging area 105 is an area that is imaged by the imaging device 21. The processing area 106 is an area where optical or physical processing is performed on the object 101.

[0014] When the object 101 is imaged in the imaging area 105, it is inspected based on the image data to determine whether it is a defective product. If the object 101 is determined to be a defective product, optical or physical processing is performed in the processing area 106. The optical processing is, for example, marking the object 101 with a light-projecting marker. The physical processing is, for example, removing the object 101 with a robot arm. The position of the object 101 in the processing area 106 is estimated by the image processing device 10.

[0015] The image processing device 10 is a device that detects the position of the object 101 in an imaging area 105 and estimates its position in a processing area 106 from that position. The image processing device 10 acquires image data of the imaging area 105 in which the object 101 is located, using an imaging device 21. The image processing device 10 estimates the position of the object 101 in the imaging area 105 and the movement speed of the object 101 from the image data. The image processing device 10 estimates the position of the object 101 in the processing area 106 at a predetermined time from the estimated position and movement speed.

[0016] As described above, in this embodiment, the imaging area 105 and the processing area 106 are set at different positions on the belt conveyor 100. When the imaging area 105 and the processing area 106 overlap, a light projecting marker or a robot arm may appear in the image data captured by the imaging area 105. If a light projecting marker or the like appears in the image data, this may become noise and affect the position detection of the object 101 and the results of the inspection for defective products, which are performed based on the image data.

[0017] Therefore, the image processing device 10 according to this embodiment enables separation of the imaging area 105 and the processing area 106, and prevents unnecessary objects from being captured when capturing an image of the imaging area 105. As a result, the image processing device 10 realizes highly accurate position detection of the object 101 and inspection of defective products.

[0018] [Device configuration] FIG. 2 is a block diagram showing the configuration of the image processing device 10 according to this embodiment.

[0019] The image processing device 10 includes a first position estimation unit 1, a speed estimation unit 2, and a second position estimation unit 3.

[0020] The first position estimation unit 1 estimates the position of the object 101 in a first area set on the upstream side of the belt conveyor 100, which is the path along which the object 101 moves, from image data of the first area. The first area is the imaging area 105 described in FIG. 1. The image data is data obtained by capturing an image using the imaging device 21.

[0021] The speed estimation unit 2 estimates the speed of the object 101 on the belt conveyor 100.

[0022] When the object 101 satisfies the set conditions, the second position estimation unit 3 estimates the position of the object 101 in a second area set downstream of the belt conveyor 100 at a predetermined time based on the detected position of the object 101 and the estimated speed of the object 101. The second area is the processing area 106 described in FIG. 1. An example of a case where the object 101 satisfies the set conditions is when the object 101 is inspected based on image data and determined to be a defective product. The inspection of the object 101 may be performed by the image processing device 10 or by another device.

[0023] The image processing device 10 configured as described above estimates the position of the object 101 on the downstream side of the belt conveyor 100 from the position of the object 101 on the upstream side of the belt conveyor 100 and the speed of the object 101. As a result, by using the image processing device 10 of this embodiment at a product production site, it is possible to detect the position of the object 101 and inspect it on the upstream side of the belt conveyor 100, and to perform optical or physical processing on the object 101 on the downstream side of the belt conveyor 100.

[0024] 1, the image processing device 10 enables separation of the imaging area 105 and the processing area 106. As a result, no unnecessary objects are captured in the image data obtained by capturing an image of the imaging area 105. Therefore, the image processing device 10 enables accurate position detection of the object 101 and inspection of defective products using image data.

[0025] The specific contents of the image processing device 10 will be described in more detail below. In the following, the image processing device 10 will be described as being used in a system that inspects an object 101 on the upstream side of a belt conveyor 100 and marks the object 101 with a light-projecting marker on the downstream side depending on the inspection results. First, the system that marks the object 101 with a light-projecting marker will be described.

[0026] FIG. 3 is a diagram for explaining a system for marking an object 101 with a light-projecting marker, in which the image processing device 10 according to this embodiment is used.

[0027] The belt conveyor 100 transports an object 101 in the direction of the arrow in the figure. A plurality of markers 100A are provided on each side of the width of the belt conveyor 100. The markers 100A face each other in the width direction and are arranged at equal intervals along the belt extension direction (conveyance direction). The positional relationship between the plurality of markers 100A, for example, the distance between the markers, is known. The markers 100A are fixed and do not move even if the belt portion of the belt conveyor 100 moves, but they move when the belt conveyor 100 itself moves.

[0028] An imaging device 21 is provided on the upstream side of the belt conveyor 100. A projection device 22 and an imaging device 23 are provided on the downstream side of the belt conveyor 100. The imaging device 21, the projection device 22, and the imaging device 23 are connected to the image processing device 10.

[0029] The imaging device 21 captures an image of the upstream side of the belt conveyor 100. The area captured by the imaging device 21 is the imaging area 105 described in FIG. 1. The imaging area 105 is set to include the marker 100A. The imaging device 21 is, for example, a camera, an infrared camera, an ultrasonic camera, an X-ray camera, or the like.

[0030] The projection device 22 acquires the position of the object 101 on the downstream side of the belt conveyor 100, which is estimated by the image processing device 10, and marks the object 101 at that position with a light-projecting marker 102. The projection area 107 onto which the projection device 22 projects the light-projecting marker 102 is the processing area 106 described in FIG. 1. The projection device 22 may be, for example, a projector.

[0031] The imaging device 23 is set to capture an imaging area 108 set on the downstream side of the belt conveyor 100 so as to overlap with a projection area 107 of the projection device 22. The imaging device 23 captures an image of a projection pattern projected by the projection device 22 when calibrating the projection device 22. The imaging device 23 is, for example, a camera, an infrared camera, an ultrasonic camera, an X-ray camera, or the like.

[0032] The imaging region 108 is set to include the marker 100A. As described above, the positional relationship between the markers 100A is known. Therefore, the positional relationship between the imaging region 105 set to include the marker 100A and the imaging region 108 is clear.

[0033] FIG. 4 is a block diagram showing a specific configuration of the image processing device 10. As shown in FIG.

[0034] The image processing device 10 further includes a processing execution unit 4 and a calibration unit 5 in addition to the first position estimation unit 1, the speed estimation unit 2, and the second position estimation unit 3 described above.

[0035] Before the image processing device 10 estimates the position of the object 101, the calibration unit 5 performs calibration to estimate the parameters (internal parameters and external parameters) of the image capturing device 21 and the image capturing device 23. Since the calibration methods for the image capturing device 21 and the image capturing device 23 are the same, only the calibration method for the image capturing device 21 will be described below.

[0036] The internal parameters are parameters indicating, for example, the focal length of the lens of the imaging device 21 and the distortion coefficient of the lens. The calibration unit 5 estimates the internal parameters using, for example, the method of Z. Zhang described in "A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000." This method involves capturing images of a board on which a grid pattern is printed from multiple directions using the imaging device 21, and estimating the parameters based on feature points in the images. The estimated parameters can be used during imaging.

[0037] The extrinsic parameters are parameters that indicate the position and angle (extrinsic parameters) of the imaging device 21 relative to the belt conveyor 100. These extrinsic parameters are parameters that convert a three-dimensional coordinate system (second coordinate system) set on the belt conveyor 100 into a coordinate system (first coordinate system) of image data obtained by the imaging device 21. More specifically, the calibration unit 5 obtains the extrinsic parameters by solving a PnP (Perspective-n-Point) problem using image data obtained by imaging the imaging area 105 with the imaging device 21. An algorithm for solving the PnP problem is described, for example, in "V. Lepetit, F. Moreno-Noguer, and P. Fua, "EPnP: An Accurate O(n) Solution to the PnP problem," International Journal of Computer Vision, vol. 81, no. 2, pp. 155-166, 2009."

[0038] Furthermore, the calibration unit 5 performs calibration between the projection device 22 and the imaging device 23. The coordinate system of the projection area 107 of the projection device 22 (hereinafter referred to as the projection coordinate system) and the coordinate system of the imaging area 108 of the imaging device 23 (hereinafter referred to as the projection imaging coordinate system) are different coordinate systems. This is because the projection device 22 and the imaging device 23 cannot be installed in the same position. The calibration unit 5 estimates the geometric transformation between the projection imaging coordinate system and the projection coordinate system from image data obtained by capturing the projection pattern projected by the projection device 22 with the imaging device 23. The projection pattern is a checkerboard pattern in which multiple markers are arranged in a grid pattern.

[0039] For example, a perspective projection transformation is used for this transformation. If the transformation parameter for transforming the projection imaging coordinate system into the projection coordinate system is H, the coordinates of the projection coordinate system are (x', y'), and the coordinates of the projection imaging coordinate system are (x, y), then (x', y') = H(x, y).

[0040] The calibration unit 5 calculates the coordinates (x m ,y m ) and the coordinates (x') of the corners of each marker in the projection pattern projected by the projection device 22. m ,y' m ) and the corresponding relationship is calculated. Then, the calibration unit 5 estimates a transformation parameter H that minimizes the positional deviation of the coordinates of the two corners by the least squares method using the following equation (1).

[0041] Σ i ||x' m i -H(x m i )|| …(1) where x m =(x m ,y m ), x' m =(x' m ,y' m ), where i is the number of each marker in the projection pattern.

[0042] The first position estimation unit 1 estimates the position of the object 101 in the imaging area 105 based on image data acquired from the imaging device 21. More specifically, the first position estimation unit 1 estimates the coordinates of the object 101 in a coordinate system in the image data (hereinafter referred to as an imaging coordinate system). Then, the first position estimation unit 1 converts the coordinates into coordinates in a three-dimensional coordinate system set on the belt conveyor 100, and estimates the position of the object 101 on the belt conveyor 100. When estimating the position of the object 101 from the image data, the first position estimation unit 1 corrects lens distortion in the image data using a lens distortion coefficient, which is an internal parameter.

[0043] 5 is a diagram for explaining the conversion from the coordinates in the imaging coordinate system to the coordinates in the three-dimensional coordinate system. In this example, the three-dimensional coordinate system has the belt extension direction as the X axis, the belt width direction as the Y axis, and the belt height direction as the Z axis.

[0044] The first position estimation unit 1 detects the coordinates of the object 101 in the imaging coordinate system. The first position estimation unit 1 converts the detected coordinates in the imaging coordinate system into coordinates in a three-dimensional coordinate system. Note that the coordinates of the object 101 in the converted three-dimensional coordinate system exist on the XY plane where Z=0.

[0045] The relationship between the coordinates (u, v) of the object 101 in the two-dimensional imaging coordinate system and the coordinates (X, Y, Z) in the three-dimensional coordinate system can be expressed, for example, as the following equation (2) using a pinhole camera model.

[0046] s[uv 1] T =A[R t][XY 0 1] T =A[r0r1t][XY 1] T [XY 1] T =s(A[r0r1t]) -1 [uv 1] T …(2) Here, R is the rotation matrix [r0, r1, r2], t is the translation vector, A is the camera matrix (external parameter), and s is a constant, which can be obtained from the constraint that the third dimension on the right-hand side of the above equation (2) is 1.

[0047] The first position estimation unit 1 converts the estimated coordinates of the object 101 in the imaging coordinate system into coordinates in a three-dimensional coordinate system using the above conversion formula, thereby estimating the position of the object 101 on the upstream side of the belt conveyor 100.

[0048] The speed estimation unit 2 estimates the moving speed of the object 101 from image data acquired from the imaging device 21. The moving speed of the object 101 estimated by the speed estimation unit 2 means a speed (hereinafter referred to as the speed for calculation) used in calculation when the second position estimation unit 3 estimates the position of the object 101, and is not the actual moving speed of the object 101. The speed estimation unit 2 may acquire the speed of the belt conveyor 100 from, for example, a speed sensor, and estimate the speed for calculation from the acquired speed, or may use the acquired speed as the speed for calculation. Furthermore, the speed estimation unit 2 may use the average value of the speeds of the multiple objects 101 as the speed for calculation.

[0049] The second position estimation unit 3 estimates the coordinates of the object 101 in the three-dimensional coordinate system at a predetermined time (the time of transmission to the linked device, i.e., the time of executing the marking process) based on the coordinates of the object 101 in the three-dimensional coordinate system estimated by the first position estimation unit 1 and the calculation speed estimated by the speed estimation unit 2. As described above, the positional relationship (distance) between the imaging area 105 and the imaging area 108 is known. The second position estimation unit 3 estimates the coordinates of the object 101 in the imaging area 108 in the three-dimensional coordinate system from the distance and the position and speed of the object 101.

[0050] It is assumed that the object 101 on the belt conveyor 100 moves in a straight line.

[0051] FIG. 6 is a diagram for explaining the conversion from the coordinates of the three-dimensional coordinate system to the coordinates of the projection imaging coordinate system.

[0052] The second position estimation unit 3 converts the coordinates of the estimated three-dimensional coordinate system into the projection imaging coordinate system using the following equation (4): s is a constant as above, and is determined from the constraint that the third dimension on the right side of the following equation (4) is 1. s[uv 1] T =(A[r0r1t])[XY 1] T …(4)

[0053] The second position estimation unit 3 uses the lens distortion coefficient, which is an internal parameter, to add the lens distortion of the imaging device 23 to the coordinates in the projection imaging coordinate system estimated by the above equation (4). A specific method for adding the lens distortion is described in, for example, URL: https: / / docs.opencv.org / 4.5.2 / d9 / d0c / group__calib3d.html.

[0054] The second position estimation unit 3 converts the coordinates (u, v) of the projection imaging coordinate system into coordinates (x, y) of the projection coordinate system based on the calibration results between the projection device 22 and the imaging device 23 performed by the calibration unit 5.

[0055] The processing execution unit 4 executes marking with the light-projecting marker 102 (see FIG. 3) by the projection device 22 on the object 101 located at the position in the imaging area 108 estimated by the second position estimation unit 3 at a predetermined time.

[0056] Next, the operation of the image processing device in this embodiment will be described with reference to FIG. 7. FIGS. 7 and 8 are flow diagrams showing the operation of the image processing device 10. In the following description, FIGS. 1 to 6 will be referred to as appropriate. In this embodiment, an image processing method is implemented by operating the image processing device. Therefore, the description of the image processing method in this embodiment will be replaced by the following description of the operation of the image processing device 10.

[0057] FIG. 7 is a flow diagram showing the operation of the image processing device 10 when performing calibration.

[0058] The calibration unit 5 performs calibration to estimate internal parameters of the image capturing device 21 and the image capturing device 23 (S1). Next, the calibration unit 5 performs calibration to estimate external parameters of the image capturing device 21 and the image capturing device 23 (S2). The calibration unit 5 performs calibration between the projection device 22 and the image capturing device 23 (S3).

[0059] The execution order of S1 to S3 in FIG. 7 can be changed as appropriate.

[0060] FIG. 8 is a flowchart showing the operation of the image processing device 10 during position estimation.

[0061] The image processing device 10 acquires image data from the imaging device 21 (S1). The first position estimation unit 1 estimates the position of the object 101 in the imaging area 105, that is, the coordinates of the object 101 in the imaging coordinate system, based on the acquired image data (S12). At this time, the first position estimation unit 1 corrects lens distortion in the image data and then estimates the coordinates of the object 101 in the imaging coordinate system.

[0062] Next, the first position estimation unit 1 converts the estimated coordinates in the imaging coordinate system into coordinates in the three-dimensional coordinate system (S13). This conversion is performed using the above equation (2).

[0063] The speed estimation unit 2 estimates the moving speed of the object 101 from the acquired image data (S14). Here, the speed estimated by the speed estimation unit 2 is the above-mentioned speed for calculation. That is, the speed estimation unit 2 calculates the conveyor speed of the belt conveyor 100 from the speeds of the multiple objects 101, and estimates the conveyor speed as the speed for calculation.

[0064] The second position estimation unit 3 estimates the coordinates of the object 101 in the imaging area 108 in three-dimensional coordinates from the coordinates of the three-dimensional coordinate system converted in S13 and the velocity estimated in S14 (S15). The second position estimation unit 3 converts the estimated three-dimensional coordinates into coordinates of the projection imaging coordinate system (S16).

[0065] The second position estimation unit 3 converts the coordinates of the projection imaging coordinate system into coordinates of the projection coordinate system (S17). The second position estimation unit 3 performs the conversion of S17 based on the calibration result between the projection device 22 and the imaging device 23 performed by the calibration unit 5.

[0066] The processing execution unit 4 executes marking with the light-projecting marker 102 (see FIG. 3) by the projection device 22 on the object 101 located at the coordinates of the projection coordinate system obtained in S17 (S18).

[0067] As described above, the image processing device 10 according to this embodiment enables separation of the imaging area 105 and the processing area 106, and prevents unnecessary objects from appearing in the image when capturing an image of the imaging area 105. As a result, the image processing device 10 realizes highly accurate position detection of the object 101 and inspection of defective products.

[0068] In this embodiment, optical processing is performed by the projection device 22 in the processing area 106, but physical processing may also be performed, such as removing the target object 101 using a robot arm.

[0069] The program in the embodiment may be any program that causes a computer to execute steps S1 to S3 shown in Fig. 7 and steps S11 to S18 shown in Fig. 8. By installing and executing this program in a computer, the image processing device and image processing method in the present embodiment can be realized. In this case, the processor of the computer functions as a first position estimation unit 1, a speed estimation unit 2, a second position estimation unit 3, a processing execution unit 4, and a calibration unit 5, and performs processing.

[0070] The program in this embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the first position estimation unit 1, the speed estimation unit 2, the second position estimation unit 3, the process execution unit 4, and the calibration unit 5.

[0071] A computer that realizes the image processing device by executing the program of this embodiment will now be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of a computer that realizes the image processing device 10 of this embodiment.

[0072] 9, the computer 110 includes a CPU 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. Note that the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111.

[0073] The CPU 111 loads the program (code) according to this embodiment stored in the storage device 113 into the main memory 112 and executes it in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). The program according to this embodiment is provided in a state stored in a computer-readable recording medium 120. The program according to this embodiment may be distributed over the Internet connected via the communication interface 117.

[0074] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0075] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0076] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0077] The image processing device 10 in this embodiment can be realized by using hardware corresponding to each part, rather than a computer on which a program is installed. Furthermore, the image processing device 10 may be realized in part by a program and in the remaining part by hardware.

[0078] A part or all of the above-described embodiment can be expressed by (Supplementary Note 1) to (Supplementary Note 30) described below, but is not limited to the following description.

[0079] (Appendix 1) a first position estimation unit that estimates a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves; a speed estimation unit that estimates a speed of the object on the path; a second position estimation unit that, when the object satisfies a set condition, estimates a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; An image processing device comprising:

[0080] (Appendix 2) 10. The image processing device according to claim 1, The first position estimation unit Estimating coordinates of the object in a first coordinate system in the image data, and converting the coordinates of the object in the first coordinate system into coordinates in a second coordinate system set on the path; The second position estimation unit estimating the coordinates of the object in the second coordinate system at the predetermined time based on the transformed coordinates in the second coordinate system and the velocity of the object; Image processing device.

[0081] (Appendix 3) 3. The image processing device according to claim 2, The speed estimation unit estimating a velocity of the object based on coordinates in the second coordinate system obtained by converting coordinates of the object in the first coordinate system; Image processing device.

[0082] (Appendix 4) 10. The image processing device according to claim 2, wherein: a first calibration unit that estimates parameters of an imaging device that images the first region; An image processing device comprising:

[0083] (Appendix 5) 10. The image processing device according to claim 2, wherein: The first calibration unit Estimating parameters for converting coordinates in the coordinate system of the imaging device and coordinates in the second coordinate system; Image processing device.

[0084] (Appendix 6) 6. The image processing device according to claim 1, a processing execution unit that executes optical or physical processing on the object located at the estimated position in the second region at the predetermined time; The image processing device further comprises:

[0085] (Appendix 7) 7. The image processing device according to claim 6, the processing execution unit executes optical marking on the object located at the estimated position in the second area at the predetermined time using a projection device. Image processing device.

[0086] (Appendix 8) 10. The image processing device according to claim 6 or 7, a second calibration unit that estimates parameters of an imaging device that images the second region; An image processing device comprising:

[0087] (Appendix 9) 9. The image processing device according to claim 8, The second calibration unit Estimating parameters for converting coordinates in the second coordinate system and coordinates in the coordinate system of the imaging device; Image processing device.

[0088] (Appendix 10) 10. The image processing device according to claim 9, The second calibration unit Estimating parameters for converting coordinates in the coordinate system of the imaging device and coordinates in the coordinate system of the projection device; Image processing device.

[0089] (Appendix 11) a step of estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves; estimating a velocity of the object along the path; If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; An image processing method comprising:

[0090] (Appendix 12) 12. The image processing method according to claim 11, further comprising: In the step of estimating the position of the object in the first region, Estimating coordinates of the object in a first coordinate system in the image data, and converting the coordinates of the object in the first coordinate system into coordinates in a second coordinate system set on the path; In the step of estimating a position in the second region, estimating the coordinates of the object in the second coordinate system at the predetermined time based on the transformed coordinates in the second coordinate system and the velocity of the object; Image processing methods.

[0091] (Appendix 13) 13. The image processing method according to claim 12, further comprising: In the step of estimating the velocity of the object, estimating a velocity of the object based on coordinates in the second coordinate system obtained by converting coordinates of the object in the first coordinate system; Image processing methods.

[0092] (Appendix 14) 14. The image processing method according to claim 12 or 13, estimating parameters of an imaging device that images the first region; An image processing method comprising:

[0093] (Appendix 15) 14. The image processing method according to claim 12 or 13, In the step of estimating parameters of an imaging device that captures an image of the first region, Estimating parameters for converting coordinates in the coordinate system of the imaging device and coordinates in the second coordinate system; Image processing methods.

[0094] (Appendix 16) 16. The image processing method according to any one of Supplementary Note 11 to Supplementary Note 15, performing optical or physical processing on the object at the estimated position in the second region at the predetermined time; The image processing method further comprises:

[0095] (Appendix 17) 17. The image processing method according to claim 16, further comprising: In the step of performing the processing, optical marking is performed on the object located at the estimated position in the second area at the predetermined time by a projection device. Image processing methods.

[0096] (Appendix 18) 18. The image processing method according to claim 16 or 17, estimating parameters of an imaging device that images the second region; An image processing method comprising:

[0097] (Appendix 19) 19. The image processing method of claim 18, further comprising: In the step of estimating parameters of an imaging device that images the second region, parameters for converting coordinates in the second coordinate system and coordinates in a coordinate system of the imaging device are estimated. Image processing methods.

[0098] (Appendix 20) 20. The image processing method of claim 19, further comprising: In the step of estimating parameters of an imaging device that captures the second region, parameters for converting coordinates in a coordinate system of the imaging device and coordinates in a coordinate system of the projection device are estimated. Image processing methods.

[0099] (Appendix 21) On the computer, a step of estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves; estimating a velocity of the object along the path; If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; A program that executes.

[0100] (Appendix 22) 22. The program of claim 21, In the step of estimating the position of the object in the first region, Estimating coordinates of the object in a first coordinate system in the image data, and converting the coordinates of the object in the first coordinate system into coordinates in a second coordinate system set on the path; In the step of estimating a position in the second region, estimating the coordinates of the object in the second coordinate system at the predetermined time based on the transformed coordinates in the second coordinate system and the velocity of the object; program.

[0101] (Appendix 23) 23. The program of claim 22, In the step of estimating the velocity of the object, estimating a velocity of the object based on coordinates in the second coordinate system obtained by converting coordinates of the object in the first coordinate system; program.

[0102] (Appendix 24) 24. The program according to claim 22 or 23, To the computer estimating parameters of an imaging device that images the first region; A program that executes.

[0103] (Appendix 25) 24. The program according to claim 22 or 23, In the step of estimating parameters of an imaging device that captures an image of the first region, Estimating parameters for converting coordinates in the coordinate system of the imaging device and coordinates in the second coordinate system; program.

[0104] (Appendix 26) 26. The program of any one of Supplementary Note 21 to Supplementary Note 25, The computer, performing optical or physical processing on the object at the estimated position in the second region at the predetermined time; A program that executes.

[0105] (Appendix 27) 27. The program of claim 26, In the step of performing the processing, optical marking is performed on the object located at the estimated position in the second area at the predetermined time by a projection device. program.

[0106] (Appendix 28) 28. The program according to claim 26 or 27, The computer, estimating parameters of an imaging device that images the second region; A program that executes.

[0107] (Appendix 29) 39. The program of claim 38, In the step of estimating parameters of an imaging device that images the second region, parameters for converting coordinates in the second coordinate system and coordinates in a coordinate system of the imaging device are estimated. program.

[0108] (Appendix 30) 29. The program of claim 29, In the step of estimating parameters of an imaging device that captures the second region, parameters for converting coordinates in a coordinate system of the imaging device and coordinates in a coordinate system of the projection device are estimated. program. [Industrial Applicability]

[0109] The image processing device of the present disclosure can be used in a system that estimates the position of a transported object on a path and performs optical or physical processing on the object. [Explanation of symbols]

[0110] 1:First position estimation part 2: Speed ​​estimation part 3:Second position estimation section 4: Processing execution unit 5: Calibration section 10: Image processing device 21: Imaging device 22: Projection device 23: Imaging device 100: conveyor belt 100A: Marker 101 :Object 102: Light projecting marker 105: Imaging area 106: Processing area 107: Projection area 108: Imaging area 110: Computer 111:CPU 112: Main memory 113: Storage device 114: Input interface 115: Display Controller 116: Writer 117: Communication interface 118: Input device 119: Display device 120: Recording medium 121: Bus

Claims

1. a first position estimation unit that estimates a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves; a speed estimation unit that estimates a speed of the object on the path; a second position estimation unit that, when the object satisfies a set condition, estimates a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; and a processing execution unit that executes marking with a light-projecting marker on the object located at the estimated position in the second area at the predetermined time using a projection device whose second area overlaps with a projection area; a calibration unit that estimates a geometric transformation between a coordinate system of the imaging device and a projection coordinate system of the projection device based on image data obtained by capturing an image of the second area using an imaging device that captures the projection pattern projected by the projection device; Equipped with the second position estimation unit transforms the position of the object in the second region into the projected coordinate system using the estimated geometric transformation; the processing execution unit executes marking on the object using the transformed position of the object in the second area. Image processing device.

2. 2. The image processing device according to claim 1, The first position estimation unit Estimating coordinates of the object in a first coordinate system in the image data, and converting the coordinates of the object in the first coordinate system into coordinates in a second coordinate system set on the path; The second position estimation unit estimating the coordinates of the object in the second coordinate system at the predetermined time based on the transformed coordinates in the second coordinate system and the velocity of the object; Image processing device.

3. 3. The image processing device according to claim 2, The speed estimation unit estimating a velocity of the object based on coordinates in the second coordinate system obtained by converting coordinates of the object in the first coordinate system; Image processing device.

4. 2. The image processing device according to claim 1, a processing execution unit that executes optical or physical processing on the object located at the estimated position in the second area at the predetermined time; The image processing device further comprises:

5. a step of estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves; estimating a velocity of the object along the path; If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; At the predetermined time, marking the object located at the estimated position in the second area with a light-projecting marker using a projection device whose second area overlaps with a projection area; a step of estimating a geometric transformation between a coordinate system of the imaging device and a projection coordinate system of the projection device based on image data obtained by capturing an image of the second area using an imaging device that captures an image of the projection pattern projected by the projection device; Equipped with In estimating the position in the second region, the position of the object in the second region is transformed into the projected coordinate system using the estimated geometric transformation; performing marking on the object using the transformed position of the object in the second region; Image processing methods.

6. On the computer, a step of estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves; estimating a velocity of the object along the path; If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; At the predetermined time, marking the object located at the estimated position in the second area with a light-projecting marker using a projection device whose second area overlaps with a projection area; a step of estimating a geometric transformation between a coordinate system of the imaging device and a projection coordinate system of the projection device based on image data obtained by capturing an image of the second area using an imaging device that captures an image of the projection pattern projected by the projection device; Execute In estimating the position in the second region, the position of the object in the second region is transformed into the projected coordinate system using the estimated geometric transformation; performing marking on the object using the transformed position of the object in the second region; program.

Images