Image processing device, image processing method, and image processing program

Abstract

The Ethernet processing unit (310) receives a plurality of payload packets each storing a plurality of divided images obtained by dividing a captured image. The Ethernet processing unit (310) and the packet discarding unit (311) select, from the plurality of payload packets, a payload packet storing the divided image to be subjected to image processing.

Description

Technical Field

[0001] The present disclosure relates to image processing.

Background Art

[0002] In factory automation (FA), various positioning techniques for positioning objects such as workpieces to match the position (control target position) of the object to be positioned to the target position using a servo motor have been put into practical use. As a method for measuring the control target position, there is a method for measuring the control target position by image processing of a captured image captured by a camera. In the method for measuring the control target position by image processing, for example, as disclosed in Patent Document 1, the camera images the object to be positioned so that the control target position is included. Then, the image processing device calculates the distance between the control target position and the target position by analyzing the captured image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional positioning technique by image processing represented by Patent Document 1, the function for performing image processing in the image processing device acquires the entire captured image. For this reason, in the conventional technique, when the image processing device receives the captured image, there is a problem that it takes a long time from the reception of the captured image until the function for performing image processing acquires the captured image.

[0005] The main object of the present disclosure is to solve such problems. More specifically, the object of the present disclosure is to shorten the time from the reception of the captured image until the function for performing image processing acquires the captured image.

Means for Solving the Problems

[0006] The image processing device relating to this disclosure is A packet receiving unit that receives multiple payload packets, each containing multiple segmented images obtained by dividing an captured image, The system includes a packet selection unit that selects a payload packet containing a segmented image to be processed from among the plurality of payload packets. [Effects of the Invention]

[0007] According to this disclosure, the time from receiving an captured image to the function that performs image processing can be shortened. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example configuration of the positioning system according to Embodiment 1. [Figure 2] The method for transmitting captured images according to Embodiment 1 is shown. [Figure 3] A diagram showing an example of the hardware configuration of an image processing device according to Embodiment 1. [Figure 4] A diagram showing an example of the hardware configuration of a motion controller according to Embodiment 1. [Figure 5] A diagram showing an example of the functional configuration of an image processing device and a motion controller according to Embodiment 1. [Figure 6] A diagram showing an example of a filter table according to Embodiment 1. [Figure 7] A diagram illustrating the method for determining the positioning utilization area according to Embodiment 1. [Figure 8] A diagram illustrating the method for determining which packets to be discarded according to Embodiment 1. [Figure 9] A flowchart showing an example of the operation of the positioning system according to Embodiment 1. [Figure 10] A diagram showing an example of operation of the image processing apparatus according to Embodiment 1. [Figure 11] A flowchart showing an example of the operation of the pattern detection unit according to Embodiment 1. [Figure 12] Flowchart showing an operation example of the motion controller according to Embodiment 1.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described with reference to the drawings. In the following description of the embodiments and the drawings, those denoted by the same reference numerals indicate the same or corresponding parts.

[0010] Embodiment 1. ***Description of the Configuration*** FIG. 1 shows a configuration example of the positioning system 1 according to the present embodiment.

[0011] In FIG. 1, in the positioning system 1, the positioning of the work 2 is performed by image processing. The positioning system 1 includes a moving mechanism 100, a camera 200, an image processing device 300, a motion controller 400, and a servo driver 500.

[0012] The work 2 is, for example, a printed circuit board. The work 2 has a feature portion 3 used for positioning. The feature portion 3 is, for example, a mounted chip, an unmounted pattern, a recognition mark, etc. Also, the feature portion 3 may be composed of a combination of these. The work 2 corresponds to the object to be positioned because it is the object to be positioned by the positioning system 1. Also, the work 2 corresponds to the object to be imaged because it is the object to be imaged by the camera 200.

[0013] The moving mechanism 100 moves the work 2. The moving mechanism 100 includes an XY stage 110, a servo motor 120a, and a servo motor 120b. The servo motor 120a drives the XY stage 110 along the X-axis direction. The servo motor 120b drives the XY stage 110 along the Y-axis direction. The moving mechanism 100 may include an XYθ stage instead of the XY stage 110.

[0014] The camera 200 images the workpiece 2. Then, the camera 200 transmits the captured image obtained by imaging to the image processing device 300. In the present embodiment, it is assumed that the camera 200 transmits the captured image to the image processing device 300 via Ethernet (registered trademark). Also, in the present embodiment, the camera 200 communicates with the image processing device 300 using the GigE Vision standard. GigE Vision is an interface standard for industrial cameras.

[0015] The image processing device 300 receives the captured image from the camera 200. Also, the image processing device 300 performs image processing on the captured image and identifies the position of the feature portion 3 on the workpiece 2. Then, the image processing device 300 positions the workpiece 2 by identifying the position of the feature portion 3. Note that the operation procedure of the image processing device 300 corresponds to an image processing method. Also, a program for realizing the operation of the image processing device 300 corresponds to an image processing program.

[0016] The motion controller 400 generates a movement command for bringing the position of the workpiece 2 closer to the target position based on the position of the feature portion 3 identified by the image processing device 300. Then, the motion controller 400 outputs the generated movement command to the servo driver 500.

[0017] The servo driver 500 performs feedback control of the servo motor 120a and the servo motor 120b based on the movement command received from the motion controller 400.

[0018] FIG. 2 shows a method in which the camera 200 divides the captured image into a plurality of packets and transmits them.

[0019] The captured image 4 of workpiece 2, captured by camera 200, is divided into multiple segmented images in raster scan format. The size of each segmented image is constant. These segmented images are then stored in multiple payload packets P(1) to P(N). The data is transmitted in the following order: leader packet P(0), one or more payload packets P(1) to P(N), and trailer packet P(N+1). Leader packet P(0) is the first packet for captured image 4. Each of the payload packets P(1) to P(N) contains a segmented image. Trailer packet P(N+1) is the last packet for captured image 4. No segmented images are stored in leader packet P(0) or trailer packet P(N+1). The leader packet P(0), each payload packet P(1) to P(N), and the trailer packet P(N+1) are assigned a block ID, which is the sequential number of the captured image 4, and a packet ID, which is the sequential number of the packet within the captured image 4. The block ID is an integer value that is incremented for each captured image 4. In Figure 2, camera 200 assigns "1" as the block ID for captured image 4. The packet ID is an integer value that is incremented for each packet. The packet ID is initialized to "0" for each captured image 4. In Figure 2, camera 200 assigns packet IDs from "0" to "N+1" sequentially to multiple packets.

[0020] Figure 3 shows an example of the hardware configuration of the image processing device 300. The image processing device 300 is a computer. As shown in Figure 3, the image processing device 300 includes a processor 301, memory 302, storage device 303, network interface 304, and motion controller interface 305.

[0021] Figure 4 shows an example of the hardware configuration of the motion controller 400. The motion controller 400 is a computer. As shown in Figure 4, the motion controller 400 includes a processor 401, memory 402, storage device 403, image processing device interface 404, and servo driver interface 405.

[0022] Figure 5 shows an example of the functional configuration of the image processing device 300 and the motion controller 400.

[0023] The Ethernet processing unit 310 receives the segmented images transmitted by the camera 200 in the form of Ethernet® frames. The Ethernet processing unit 310 then inputs the IP (Internet Protocol) packets, which are the payload, to the UDP / IP processing unit 320. More specifically, the Ethernet processing unit 310 selects from the payload packets P(1) to P(N) shown in Figure 2 that contain the segmented images to be processed for positioning workpiece 2. The Ethernet processing unit 310 then forwards the selected payload packet to the UDP / IP processing unit 320. The Ethernet processing unit 310 discards all payload packets other than the selected one. The Ethernet processing unit 310 corresponds to the packet receiving unit. Furthermore, the Ethernet processing unit 310, together with the packet discarding unit 311 (described later), corresponds to the packet selection unit. The processing performed by the Ethernet processing unit 310 corresponds to the packet receiving process. Also, the processing performed by the Ethernet processing unit 310, together with the processing performed by the packet discarding unit 311, corresponds to the packet selection process.

[0024] The UDP / IP processing unit 320 extracts UDP (User Datagram Protocol) packets from the incoming IP packets. Then, the UDP / IP processing unit 320 inputs the GigE Vision packets, which are the payload of the UDP packets, to the GigE Vision processing unit 330.

[0025] The GigE Vision processing unit 330 constructs a partial image, which is a part of the captured image 4, from one or more GigE Vision packets. The partial image constructed by the GigE Vision processing unit 330 is an image showing the positioning area, which will be described later. The GigE Vision processing unit 330 inputs the partial image to the pattern detection unit 340. The GigE Vision processing unit 330 determines which image 4 each acquired GigE Vision packet constitutes based on the block ID shown in Figure 2. The GigE Vision processing unit 330 then constructs a partial image from the payload packets in the order indicated by the packet IDs shown in Figure 2.

[0026] Furthermore, the size of a single GigE Vision packet should be such that it fits within a single UDP packet. The size of a single UDP packet should be such that it fits within a single IP packet. The size of a single IP packet should be such that it fits within a single Ethernet frame. Jumbo frames may be used to satisfy these conditions.

[0027] The pattern detection unit 340 performs image processing on the partial image input from the GigE Vision processing unit 330. The pattern detection unit 340 then identifies the location of the feature portion 3 on the workpiece 2. The pattern detection unit 340 identifies the position of the feature portion 3, for example, by a pixel-level detection process using template matching. The pattern detection unit 340 may perform sub-pixel position estimation processing instead of, or in addition to, the pixel-level detection process. The pattern detection unit 340 then outputs the result of the image processing (position of the feature portion) to the motion controller 400. The pattern detection unit 340, together with the pixel range determination unit 341 and the discard packet determination unit 331 described later, corresponds to the packet designation unit. Furthermore, the processing performed by the pattern detection unit 340, together with the processing performed by the pixel range determination unit 341 and the discard packet determination unit 331 described later, corresponds to the packet designation processing.

[0028] The packet discarding unit 311 refers to the filter table 312 and decides whether to discard the payload packet received by the Ethernet processing unit 310 or to input it to the UDP / IP processing unit 320. As mentioned above, the packet discarding unit 311, together with the Ethernet processing unit 310, corresponds to the packet selection unit. Furthermore, the processing performed by the packet discarding unit 311, together with the processing performed by the Ethernet processing unit 310, corresponds to the packet selection process.

[0029] The pixel range determination unit 341 determines the image region (hereinafter referred to as the positioning utilization region) to be used for positioning in the future captured image 4, which will be captured at a future imaging timing. The pixel range determination unit 341 determines the positioning utilization region based on the position of the feature portion 3 at the future imaging timing, which is predicted by the position prediction unit 420, which will be described later. Each of the multiple segmented images that constitute the positioning utilization area is called a positioning segmented image. The GigE Vision processing unit 330, as described above, combines multiple positioning segmented images to construct a partial image in which the positioning utilization area is captured. Details of the positioning utilization area will be described later. As mentioned above, the pixel range determination unit 341, together with the pattern detection unit 340 and the discard packet determination unit 331, corresponds to the packet designation unit. Furthermore, the processing performed by the pixel range determination unit 341, together with the processing performed by the pattern detection unit 340 and the discard packet determination unit 331, corresponds to the packet designation processing.

[0030] The discard packet determination unit 331 determines the payload packets to be discarded. Specifically, the discard packet determination unit 331 calculates the block ID corresponding to the future captured image 4. Furthermore, based on the positioning utilization area in the future captured image 4, the discard packet determination unit 331 calculates the packet ID of the payload packets for the future captured image 4 that do not contain positioning segmented images. The discard packet determination unit 331 then registers the calculation results in the filter table 312. In other words, the discard packet determination unit 331 designates the payload packet containing the positioning segmentation image prior to the selection of the payload packet for the future captured image 4 by the Ethernet processing unit 310. As mentioned above, the discard packet determination unit 331, together with the pattern detection unit 340 and the pixel range determination unit 341, corresponds to the packet designation unit. Furthermore, the processing performed by the discard packet determination unit 331, together with the processing performed by the pattern detection unit 340 and the pixel range determination unit 341, corresponds to the packet designation processing.

[0031] The functions of the Ethernet processing unit 310, UDP / IP processing unit 320, etc., shown in Figure 5, are implemented, for example, by a program. The storage device 303 shown in Figure 3 stores the program that implements the functions of the Ethernet processing unit 310, UDP / IP processing unit 320, etc. These programs are loaded from the storage device 303 into memory 302. Then, the processor 301 executes these programs to perform operations on the Ethernet processing unit 310, UDP / IP processing unit 320, etc.

[0032] In the motion controller 400, the movement control unit 410 generates a movement command to bring the workpiece 2 closer to the target position. More specifically, the movement control unit 410 generates a movement command based on the position of the feature portion 3 identified by the image processing device 300. The movement control unit 410 then outputs the generated movement command to the servo driver 500.

[0033] The position prediction unit 420 predicts the position of feature portion 3 at a future imaging timing. More specifically, the position prediction unit 420 predicts the position of feature portion 3 based on the past and present positions of feature portion 3 identified by the pattern detection unit 340. The position prediction unit 420 then notifies the pixel range determination unit 341 of the predicted position of the feature portion 3.

[0034] The functions of the movement control unit 410 and the position prediction unit 420 shown in Figure 5 are implemented, for example, by a program. The storage device 403 shown in Figure 4 stores the program that implements the functions of the movement control unit 410 and the position prediction unit 420. These programs are loaded from the storage device 403 into memory 402. Then, the processor 401 executes these programs to control the operation of the movement control unit 410 and the position prediction unit 420.

[0035] Figure 6 shows an example of a filter table 312.

[0036] The filter table 312 is a table generated in memory 302. The filter table 312 holds information for the Ethernet processing unit 310 to identify payload packets that should be discarded in the future. The filter table 312 in Figure 6 contains pairs of block IDs and packet IDs as information for identifying payload packets that should be discarded in the future. In Figure 6, the packet IDs of the payload packets to be discarded are shown, which store the segmented images of the captured image 4, where block ID is "3". Specifically, payload packets with packet ID "1" are to be discarded. Also, payload packets with packet ID "2" are to be discarded. Furthermore, payload packets with packet ID "33" are to be discarded. Also, payload packets with packet ID "34" are to be discarded. Also, payload packets with packet ID "35" are to be discarded. Figure 6 shows an example where the filter table 312 is configured using a ring buffer. The filter table 312 may also be configured using a hash set or hash map with a block ID and packet ID pair as keys.

[0037] Figure 7 shows how the position prediction unit 420 and the pixel range determination unit 341 determine the position utilization area 5.

[0038] First, the position prediction unit 420 predicts the position of feature portion 3 at time t1 (t1=t0+Δt), which is a future imaging timing, at time t0. More specifically, the position prediction unit 420 predicts the position of the feature portion 3 at time t1 based on the position of the feature portion 3 at time t0 and the predicted amount of movement of the feature portion 3 (the amount of movement of the workpiece 2 between time t0 and time t1) obtained from the movement command at time t0 generated by the movement control unit 410. In addition to these, the position prediction unit 420 may also use a damped oscillation model generated from the past position information of the feature portion 3 to predict the position of the feature portion 3.

[0039] Next, the pixel range determination unit 341 determines the entire view of the feature portion 3 and the rectangular area surrounding the feature portion 3 in the future captured image 4, which will be captured at time t1, as the positioning utilization area 5. A segmented image in which at least a portion of feature portion 3 is captured is called a feature segmented image. In this embodiment, an image showing the entire view of feature portion 3 is constructed by combining one or more feature segmented images. A segmented image in which at least a portion of the rectangular region surrounding feature portion 3 is captured is called a perimeter segmented image. In this embodiment, an image showing the entire view of the surrounding rectangular region is constructed by combining one or more perimeter segmented images. If a single segmented image contains at least a portion of feature segment 3 and at least a portion of the surrounding rectangular region, then that segmented image is both a feature segmented image and a surrounding segmented image. As mentioned above, each of the multiple segmented images that constitute the positioning region is called a positioning segmented image. For the sake of simplicity, in this embodiment, each positioning segmented image is either a feature segmented image or a perimeter segmented image. In actual operation, the positioning segmented image may be a segmented image that is both a feature segmented image and a perimeter segmented image, as described above.

[0040] The size of the positioning area 5 may be fixed or variable, as long as it is large enough to completely include the entire view of the feature portion 3. As an example of variable size, if the prediction accuracy of the position of feature portion 3 may decrease, the size of the positioning utilization area 5 can be increased. For example, if the movement distance of workpiece 2 is predicted to be large from time t0 to time t1, the size of the positioning utilization area 5 can be increased. Also, if the vibration amplitude of workpiece 2 is predicted to be large from time t0 to time t1, the size of the positioning utilization area 5 can be increased.

[0041] Figure 8 illustrates how the discard packet determination unit 331 determines which packets to discard. In other words, Figure 8 shows how the discard packet determination unit 331 calculates the packet ID of a payload packet containing a segmented image that does not correspond to a positioning segmented image.

[0042] Here, the current time is t0 [seconds], and the future imaging timing is t1 (t1 = t0 + Δt) [seconds]. Also, the frame rate is f [fps], and the block ID of the image 4 captured at the current time t0 is b0. The block ID of the future captured image 4, which will be captured at a future time t1, is calculated as b1 = b0 + f [fps] × Δt [seconds]. In positioning using this future captured image 4, the leader packet, the trailer packet, and all payload packets containing segmented images (positioning segmented images) corresponding to the positioning utilization area 5 are subject to image processing by the pattern detection unit 340. In other words, payload packets containing segmented images that do not correspond to the positioning utilization area 5 are to be discarded. Here, the starting coordinates of the raster scan of the positioning area 5 are (xi, yi), and the ending coordinates are (xj, yj). The packet IDs of the payload packets that will store the segmented images of the future captured image 4 are P(1) to P(N). The packet ID of the payload packet that will store the segmented image containing the starting coordinates of the positioning area 5 is P(i). The packet ID of the payload packet that will store the segmented image containing the ending coordinates of the positioning area 5 is P(j). In this case, payload packets with packet IDs P(1) to P(i-1) and payload packets with packet IDs P(j+1) to P(N) are discarded.

[0043] ***Explanation of operation*** Next, an example of the operation of the positioning system 1 according to this embodiment will be described. Figure 9 shows an example in which the positioning system 1 positions the workpiece 2 using image processing.

[0044] First, if a new workpiece 2 is placed in step S100, the process proceeds to step S200.

[0045] In step S200, camera 200 images workpiece 2 placed on the XY stage 110 and records the imaging time. Camera 200 performs imaging of workpiece 2 in step S200 for each imaging cycle.

[0046] Next, in step S300, the camera 200 transmits the captured image 4 and the acquisition time obtained in step S200 to the image processing device 300. The camera 200 communicates with the image processing device 300 using, for example, the GigE Vision standard. The acquisition time is stored in the leader packet. Multiple divided images obtained from the captured image 4 are stored in one or more payload packets. Then, as shown in Figure 2, the leader packet, payload packet, and trailer packet are transmitted from the camera 200 to the image processing device 300. The camera 200 completes the transmission in step S300 by the start time of the next imaging cycle.

[0047] In step S400, the image processing device 300 receives the leader packet, payload packet, and trailer packet transmitted from the camera 200 and acquires the segmented image and its capture time. The image processing device 300 then performs image processing to determine the current position of the feature portion 3 on the workpiece 2.

[0048] In step S500, the image processing device 300 notifies the motion control unit 410 and the position prediction unit 420 of the imaging time and the current position of the feature portion 3 identified in step S400.

[0049] Further details of the processing performed by the image processing device 300, as described in steps S400 and S500, will be discussed later.

[0050] In step S600, the movement control unit 410 generates a movement command to bring the workpiece 2 closer to the target position based on the current position of the feature portion 3 identified by the image processing device 300. In addition, the position prediction unit 420 predicts the position of the feature portion 3 at a future imaging timing.

[0051] In step S700, the motion control unit 410 of the motion controller 400 notifies the servo driver 500 of the movement command generated in step S600. In addition, the position prediction unit 420 of the motion controller 400 notifies the pixel range determination unit 341 of the image processing device 300 of the predicted position of the feature portion 3 calculated in step S600.

[0052] Further details of the processing of the motion controller 400, as described in steps S600 and S700, will be discussed later.

[0053] In step S800, the servo driver 500 performs feedback control on servo motors 120a and 120b so that the amount of movement of the XY stage 110 approaches the movement command. Then, the position of the workpiece 2 is changed as servo motors 120a and 120b drive the XY stage 110.

[0054] As shown in step S900, steps S200 to S800 are repeated until workpiece 2 moves to the target position.

[0055] Figure 10 illustrates the detailed processing of the image processing apparatus 300 in steps S400 and S500.

[0056] The image processing device 300 executes the processes from steps S410 to S440 in Figure 10 for each image processing cycle. The length of the image processing cycle is the same as the length of the imaging cycle. Note that the start time of the image processing cycle does not need to be simultaneous with the start time of the imaging cycle.

[0057] In Figure 10, the Ethernet processing unit 310 and UDP / IP processing unit 320, and the GigE Vision processing unit 330 and pattern detection unit 340 operate on separate CPU (Central Processing Unit) cores. Therefore, the Ethernet processing unit 310 and UDP / IP processing unit 320, and the GigE Vision processing unit 330 and pattern detection unit 340 operate in parallel. More specifically, while the GigE Vision processing unit 330 and the pattern detection unit 340 are processing the captured image 4 with block ID=n, the Ethernet processing unit 310 and the UDP / IP processing unit 320 are processing the packet corresponding to the captured image 4 with block ID=n+1.

[0058] First, in step S410, the Ethernet processing unit 310 selects from among the Ethernet frames that have arrived at the network interface 304 that should be used to input IP packets to the UDP / IP processing unit 320. More specifically, the Ethernet processing unit 310 uses the packet discarding unit 311 to determine whether or not to discard the received Ethernet frame. The packet discard unit 311 refers to the GigE Vision protocol header of the received Ethernet frame. The packet discard unit 311 then determines whether the block ID and packet ID pair in the Ethernet frame matches any of the block ID and packet ID pairs registered in the filter table 312. If the block ID and packet ID pair of an Ethernet frame matches the block ID and packet ID pair in the filter table 312, the packet discard unit 311 determines that the Ethernet frame should be discarded. The Ethernet processing unit 310 discards the Ethernet frame that the packet discard unit 311 has determined should be discarded without inputting it to the UDP / IP processing unit 320. After the Ethernet frame is discarded, the packet discard unit 311 removes the block ID and packet ID pair of the Ethernet frame from the filter table 312. On the other hand, if the packet discarding unit 311 does not determine that the Ethernet frame should be discarded, the Ethernet processing unit 310 inputs the IP packet of the Ethernet frame's payload to the UDP / IP processing unit 320. In other words, the Ethernet processing unit 310 and the packet discarding unit 311 select the IP packet to be processed for image processing from among multiple IP packets.

[0059] Next, in step S420, the UDP / IP processing unit 320 extracts UDP packets from the incoming IP packets. Furthermore, the UDP / IP processing unit 320 extracts GigE Vision packets from the payload of the extracted UDP packets. The UDP / IP processing unit 320 then inputs the extracted GigE Vision packets to the GigE Vision processing unit 330. The GigE Vision packets input to the GigE Vision processing unit 330 are either leader packets, payload packets, or trailer packets.

[0060] In step S430, the GigE Vision processing unit 330 constructs a partial image from the leader packet, one or more payload packets, and the trailer packet. The partial image constructed by the GigE Vision processing unit 330 is an image showing the positioning utilization area 5. The GigE Vision processing unit 330 constructs a partial image by combining the segmented images (positioning segmented images) stored in all payload packets having a common block ID, in the order indicated by the packet IDs. If a portion of the payload packet was discarded in step S410 during the construction of the partial image, the GigE Vision processing unit 330 skips storing the segmented image in the memory area corresponding to the discarded payload packet. In this case, no segmented image is stored in that memory area. Therefore, the values ​​originally stored in that memory area (initial values, undefined values, etc.) are maintained. When the GigE Vision processing unit 330 receives a trailer packet and completes the construction of a partial image, it inputs the partial image, the imaging time obtained from the leader packet, and the starting and ending coordinates obtained from the raster scan of the partial image to the pattern detection unit 340. The GigE Vision processing unit 330 inputs the coordinates of the first pixel of the segmented image at the beginning of the partial image to the pattern detection unit 340 as the starting coordinates of the partial image. The GigE Vision processing unit 330 also inputs the coordinates of the last pixel of the segmented image at the end of the partial image to the pattern detection unit 340 as the ending coordinates of the partial image. The starting coordinates of a sub-image are, as explained using Figure 8, the coordinates (xk, yk) of the first pixel in the sub-image stored in the payload packet with packet ID P(i). The ending coordinates of a sub-image are, as explained using Figure 8, the coordinates (xl, yl) of the last pixel in the sub-image stored in the payload packet with packet ID P(j).

[0061] Figure 11 shows an example of the operation of the pattern detection unit 340 in step S440.

[0062] In step S441, the pattern detection unit 340 receives a partial image, the time the partial image was captured, and the starting and ending coordinates of the partial image from the GigE Vision processing unit 330.

[0063] Next, in step S442, the pattern detection unit 340 performs image processing on the partial image to identify the current position of the feature portion 3 on the workpiece 2. Note that, in order to perform image processing efficiently, the pattern detection unit 340 may include a portion of the area outside the partial image in its image processing. Furthermore, the pattern detection unit 340 may perform image processing after adding padding around the partial image.

[0064] Next, in step S443, the pattern detection unit 340 notifies the movement control unit 410 and the position prediction unit 420 of the motion controller 400 of the position of the feature portion 3 and the time at which the partial image used to identify the position of the feature portion 3 was captured.

[0065] Then, in step S444, the pixel range determination unit 341, executed by the pattern detection unit 340, obtains the predicted position of the feature portion 3 in the future captured image 4 from the position prediction unit 420. At this time, the pixel range determination unit 341 may also obtain information to identify the future captured image 4. For example, the pixel range determination unit 341 may obtain information indicating the difference Δt between the imaging time of the current captured image 4 and the imaging time of the future captured image 4 as information to identify the future captured image 4. Alternatively, the pixel range determination unit 341 may obtain information indicating the difference N between the block ID of the current captured image 4 and the block ID of the future captured image 4 as information to identify the future captured image 4. In the example shown in Figure 10, while the GigE Vision processing unit 330 and the pattern detection unit 340 are processing a partial image of the captured image 4 with block ID=n, the Ethernet processing unit 310 and the UDP / IP processing unit 320 have already started receiving the packet containing the captured image 4 with block ID=n+1. Therefore, in the example shown in Figure 10, the position prediction unit 420 calculates the predicted position of feature portion 3 in image 4 with block ID = n+2 from a partial image of image 4 with block ID = n. In other words, the difference between the block ID of the current image 4 and the block ID of the future image 4 is 2.

[0066] Next, in step S445, the pixel range determination unit 341 determines the positioning utilization area 5 in the future captured image 4 based on the predicted position of the feature portion 3 acquired in step S444. The method for determining the positioning utilization area 5 is as explained with reference to Figure 7. The aforementioned image 4 with block ID=n corresponds to image 4 at time t0 in Figure 7, and the image 4 with block ID=n+2 corresponds to image 4 at time t1 in Figure 7.

[0067] In step S446, the discard packet determination unit 331, executed by the pattern detection unit 340, determines the packets to be discarded. Specifically, the discard packet determination unit 331 calculates the block ID corresponding to the future captured image 4. The discard packet determination unit 331 also calculates the packet ID of the payload packets for the future captured image 4 that do not contain positioning division images, based on the positioning utilization area 5 in the future captured image 4. The discard packet determination unit 331 then registers the calculation results in the filter table 312. The method for calculating the block ID and packet ID is as explained using Figure 8. Furthermore, the contents registered in the filter table 312 are as explained using Figure 6.

[0068] As shown in step S447, steps S441 to S446 are repeated until workpiece 2 moves to the target position.

[0069] Figure 12 shows an example of the operation of the motion controller 400 in steps S600 and S700. The motion controller 400 executes steps S601 to S604 in Figure 12 for each calculation cycle.

[0070] In step S601, when the start time of the calculation cycle arrives, in step S602, the movement control unit 410 and the position prediction unit 420 acquire the position of the feature portion 3 identified by the image processing device 300, and the acquisition time of the captured image 4 used to identify the position of the feature portion 3.

[0071] In step S603, the movement control unit 410 generates a movement command to bring the workpiece 2 closer to the target position based on the position of the feature portion 3 identified by the image processing device 300. Furthermore, the position prediction unit 420 predicts the position of the feature portion 3 at a future imaging timing. The prediction method is as explained using Figure 7. Here, the current imaging time is denoted as t0 [seconds], the future imaging timing as t1(t0 + Δt) [seconds], and the frame rate as f [fps]. When calculating the predicted position in the Nth image after the current image 4, Δt is expressed as Δt [seconds] = 1 / f × N. Here, the value of N is determined, for example, between the image processing device 300 and the motion controller 400 before the start of positioning. Alternatively, the image processing device 300 may notify the motion controller 400 of the value of N each time during positioning.

[0072] In step S604, the movement control unit 410 of the motion controller 400 notifies the servo driver 500 of the movement command generated in step S603. Furthermore, the position prediction unit 420 of the motion controller 400 notifies the pixel range determination unit 341 of the image processing device 300 of the predicted position of the feature portion 3 calculated in step S603. When the position prediction unit 420 notifies the predicted position of the feature portion 3, it may also notify the pixel range determination unit 341 of information indicating which future captured image 4 the predicted position corresponds to. As such information, the position prediction unit 420 may notify, for example, one of the values ​​of Δt, t0+Δt, or N.

[0073] As shown in step S605, steps S601 to S604 are repeated until workpiece 2 moves to the target position.

[0074] ***Explanation of the effects of the embodiment*** In this embodiment, the position prediction unit 420 predicts the position of the feature portion 3 at a future imaging timing. The pixel range determination unit 341 then determines the positioning utilization area 5 in the captured image 4 to be captured at that future imaging timing. In addition, the discard packet determination unit 331 identifies future payload packets that are not needed for positioning. Furthermore, the packet discard unit 311 discards these unnecessary payload packets. Therefore, according to this embodiment, the time from when the Ethernet processing unit 310 receives the captured image 4 from the network-connected camera 200 until when the pattern detection unit 340 acquires the captured image 4 can be shortened. This enables high-speed positioning of the object to be positioned. In addition, since the amount of data input to the UDP / IP processing unit 320 is reduced, the memory usage by socket buffers, which are commonly used in network reception processing, can be reduced. Furthermore, in this embodiment, the camera 200 captures and transmits the entire workpiece 2, and the image processing device 300 discards unnecessary areas of the captured image. Therefore, it is possible to start image processing targeting areas other than the feature portion 3 without delay as needed. This is an effect that cannot be obtained with conventional methods that reduce the amount of transmitted data by having the camera 200 capture only a part of the workpiece 2. Also, this is an effect that cannot be obtained with conventional methods that reduce the amount of transmitted data by having the camera 200 transmit only a part of the captured image (ROI: Region of Interest).

[0075] In this embodiment, the image processing device 300 and the motion controller 400 are separate devices, but they may be combined into a single device. This allows the movement control unit 410 and the position prediction unit 420 to operate on the same memory as the pixel range determination unit 341. As a result, the positioning utilization area 5 in the captured image 4, which will be captured at a future imaging timing, can be determined at a faster speed.

[0076] Furthermore, in this embodiment, the Ethernet processing unit 310 and UDP / IP processing unit 320 and the GigE Vision processing unit 330 and pattern detection unit 340 operate in parallel on separate CPU cores. Alternatively, they may operate sequentially on a single CPU core. In this case, the Ethernet processing unit 310 and UDP / IP processing unit 320 process packets for the captured image 4 with block ID=n+1 only after the GigE Vision processing unit 330 and pattern detection unit 340 have completed processing the partial image with block ID=n. Therefore, the position prediction unit 420 calculates the predicted position of the feature portion 3 in the captured image with block ID=n+1 from the partial image with block ID=n. In other words, the difference between the block ID of the current captured image 4 and the block ID of the future captured image 4 will be 1.

[0077] Furthermore, in this embodiment, the size of the positioning area 5 may be variable or fixed. If the size of the positioning area 5 is variable, the average time it takes from when the Ethernet processing unit 310 receives the captured image 4 until the pattern detection unit 340 acquires the captured image 4 is shortened. On the other hand, fixing the size of the positioning area 5 means setting an upper limit on the number of packets that the image processing device 300 processes without discarding. Therefore, fixing the size of the positioning area 5 reduces the maximum time it takes from when the Ethernet processing unit 310 receives the captured image 4 until the pattern detection unit 340 acquires the captured image 4. This has the effect of shortening the imaging cycle and achieving more accurate positioning.

[0078] Furthermore, in this embodiment, an example was described in which the Ethernet processing unit 310 selects a payload packet containing segmented images used for image processing for positioning. The Ethernet processing unit 310 can be used not only for positioning but also for extracting segmented images of feature portions 3 from a series of segmented images. In other words, the Ethernet processing unit 310 can receive multiple payload packets, each containing multiple segmented images obtained by segmenting an image of an object that is not a positioning target but contains the feature portion 3. The Ethernet processing unit 310 can then select from the multiple payload packets the payload packet containing the segmented image in which the feature portion 3 is captured, as the payload packet containing the segmented image to be processed.

[0079] Note that the procedure described in this embodiment is just one example. Therefore, it is acceptable to perform only a portion of the procedure described in this embodiment. Furthermore, at least some of the procedures described in this embodiment may be combined with procedures not described in this embodiment. Furthermore, the configuration and procedures described in this embodiment may be modified as necessary.

[0080] ***Supplementary explanation of hardware configuration*** Finally, we will provide supplementary explanations regarding the hardware configuration of the image processing device 300 and the motion controller 400.

[0081] Processor 301 is an integrated circuit (IC) that performs processing. Processor 301 includes CPU, DSP (Digital Signal Processor), etc. Memory 302 is RAM (Random Access Memory). The storage device 303 can be ROM (Read Only Memory), flash memory, HDD (Hard Disk Drive), etc. The network interface 304 and the motion controller interface 305 are electronic circuits that perform data communication processing, respectively. The network interface 304 and the motion controller interface 305 are, for example, a communication chip or a NIC (Network Interface Card), respectively.

[0082] Furthermore, the operating system (OS) is also stored in the memory device 303. Then, at least a portion of the OS is executed by processor 301. The processor 301 executes programs that implement functions such as the Ethernet processing unit 310 and the UDP / IP processing unit 320, while executing at least a part of the OS. Processor 301 executes the OS, which handles task management, memory management, file management, communication control, and other functions. Furthermore, at least one of the information, data, signal values, and variable values ​​indicating the processing results of the Ethernet processing unit 310, UDP / IP processing unit 320, etc., is stored in at least one of the memory 302, storage device 303, registers in the processor 301, and cache memory. Furthermore, the programs that implement functions such as the Ethernet processing unit 310 and the UDP / IP processing unit 320 may be stored on portable recording media such as magnetic disks, flexible disks, optical disks, compact disks, Blu-ray® discs, and DVDs. Portable recording media containing the programs that implement functions such as the Ethernet processing unit 310 and the UDP / IP processing unit 320 may also be distributed.

[0083] Processor 401 is also an IC that performs processing. Processor 401 also includes CPU, DSP, etc. Memory 402 is also RAM. The storage device 403 can also be ROM, flash memory, HDD, etc. The image processing device interface 404 and the servo driver interface 405 are also electronic circuits that perform data communication processing, respectively. The image processing device interface 404 and the servo driver interface 405 are, for example, communication chips or NICs.

[0084] Furthermore, the operating system is also stored in the memory device 403. Then, at least a portion of the OS is executed by processor 401. The processor 401 executes a program that implements the functions of the movement control unit 410 and the position prediction unit 420 while executing at least a part of the OS. Processor 401 executes the OS, handling task management, memory management, file management, communication control, and other functions. Furthermore, at least one of the information, data, signal values, and variable values ​​indicating the processing results of the movement control unit 410 and the position prediction unit 420 is stored in at least one of the memory 402, storage device 403, registers in the processor 401, and cache memory. Furthermore, the programs that implement the functions of the movement control unit 410 and the position prediction unit 420 may be stored in a portable recording medium such as a magnetic disk, flexible disk, optical disk, compact disk, Blu-ray® disk, or DVD. The portable recording medium containing the programs that implement the functions of the movement control unit 410 and the position prediction unit 420 may then be distributed.

[0085] Furthermore, at least one of the "parts" in the Ethernet processing unit 310, UDP / IP processing unit 320, etc., may be read as "circuit," "process," "procedure," "processing," or "circuitry." Similarly, at least one of the "parts" in the movement control unit 410 and position prediction unit 420 may be read as "circuit," "process," "procedure," "processing," or "circuitry." Furthermore, the image processing device 300 and the motion controller 400 may be implemented by processing circuits. Examples of processing circuits include logic ICs (Integrated Circuits), GAs (Gate Arrays), ASICs (Application Specific Integrated Circuits), and FPGAs (Field-Programmable Gate Arrays). In this case, the Ethernet processing unit 310, UDP / IP processing unit 320, packet discarding unit 311, etc., are each implemented as part of the processing circuit. The movement control unit 410 and position prediction unit 420 are also each implemented as part of the processing circuit. In this specification, the higher-level concept encompassing both the processor and the processing circuit is referred to as "processing circuitry." In other words, a processor and a processing circuit are specific examples of "processing circuits," respectively. [Explanation of symbols]

[0086] 1 Positioning system, 2 Workpiece, 3 Feature area, 4 Captured image, 5 Positioning utilization area, 100 Moving mechanism, 110 XY stage, 120a Servo motor, 120b Servo motor, 200 Camera, 300 Image processing unit, 301 Processor, 302 Memory, 303 Storage device, 304 Network interface, 305 Motion controller interface, 310 Ethernet processing unit, 311 Packet discarding unit, 312 Filter table, 320 UDP / IP processing unit, 330 GigE Vision processing unit, 331 Discard packet determination unit, 340 Pattern detection unit, 341 Pixel range determination unit, 400 Motion controller, 401 Processor, 402 Memory, 403 Storage device, 404 Image processing unit interface, 405 Servo driver interface, 410 Movement control unit, 420 Position prediction unit, 500 Servo driver.

Claims

1. A packet receiving unit that receives a plurality of payload packets in which a plurality of divided images obtained by dividing a captured image of a positioning object that is a positioning target are stored; and a packet selection unit for selecting, from the plurality of payload packets, a payload packet storing a positioning divided image that is a divided image to be subjected to image processing for positioning the positioning object.

2. The packet selection unit The image processing device according to claim 1 , wherein payload packets other than a selected payload packet are discarded from among the plurality of payload packets.

3. The object to be positioned has a feature portion, The plurality of payload packets includes a payload packet storing a feature division image, which is a division image in which the feature portion is captured, The packet selection unit The image processing device according to claim 1 , wherein a payload packet in which the feature division image is stored is selected as the payload packet in which the positioning division image is stored.

4. The plurality of payload packets includes a payload packet storing a surrounding divided image, which is a divided image capturing an area surrounding the characteristic portion; The packet selection unit The image processing device according to claim 3 , wherein a payload packet in which the surrounding divided image is stored is selected as the payload packet in which the positioning divided image is stored.

5. The image processing device further comprises: a packet designation unit that designates a payload packet in which the positioning divided image is stored prior to the selection of a payload packet by the packet selection unit; The packet selection unit The image processing device according to claim 1 , wherein a payload packet designated by said packet designation unit is selected.

6. The object to be positioned has a feature portion, The plurality of payload packets include a payload packet storing a feature divided image which is a divided image showing the feature portion, and a payload packet storing a surrounding divided image which is a divided image showing an area surrounding the feature portion, The packet designation unit is The image processing device according to claim 5 , wherein a payload packet in which the feature divided image is stored and a payload packet in which the surrounding divided image is stored are designated as the payload packet in which the positioning divided image is stored.

7. The packet designation unit is The time t of the characteristic portion ０ and the time t ０ From the time t ０ Time t after １ The time t of the characteristic portion predicted based on the amount of movement of the object to be positioned between １ Get the predicted position at The time t of the acquired characteristic portion １ Using the predicted position at time t １ The image captured at time t １ The plurality of divided images at the time t １ Among the multiple payload packets at time t １ 7. The image processing device according to claim 6, wherein the image processing device specifies a payload packet in which a divided image for positioning is stored.

8. The packet designation unit is The image processing apparatus of claim 5, which uses fixed size registration segment images.

9. The packet receiving unit The image processing device according to claim 1 , wherein the image processing device receives a plurality of payload packets each storing a plurality of divided images obtained by dividing the captured image in a raster scan format.

10. A computer receives a plurality of payload packets each storing a plurality of divided images obtained by dividing a captured image of a positioning object that is a positioning target, An image processing method in which the computer selects, from the plurality of payload packets, a payload packet storing a positioning divided image that is a divided image to be subjected to image processing for positioning the positioning object.

11. A packet reception process for receiving a plurality of payload packets in which a plurality of divided images obtained by dividing an image of an object to be positioned are stored; and a packet selection process for selecting, from the plurality of payload packets, a payload packet storing a positioning divided image, which is a divided image that is to be subjected to image processing for positioning the positioning object.

Images