Image processing device, image processing method, and image processing program

Abstract

The Ethernet processing unit (310) receives a multitude of payload packets containing a multitude of split images obtained by splitting a captured image. The Ethernet processing unit (310) and a packet drop unit (311) select from the multitude of payload packets one payload packet containing a split image to be processed.

Description

Technical field

[0001] The present disclosure relates to image processing. Background on the state of the art

[0002] In factory automation (FA), various technologies (positioning technologies) are used that employ a servo motor to align the position (the position of the control object) of a position setting object, for example a workpiece, to a target position.

[0003] One technique for measuring the position of a controlled object involves image processing of an image captured by a camera. In the method disclosed in patent literature 1 for measuring the position of the controlled object using image processing, the camera, for example, captures the position setting object in such a way that the position of the controlled object is included within it. Subsequently, an image processing device analyzes the captured image and then calculates the distance between the position of the controlled object and the target position. Reference list of patent literature

[0004] Patent Literature 1: JP 2019-003388 A Summary of the invention: Technical problem

[0005] In conventional position setting technology using image processing, as described in patent literature 1, the function performing the image processing within the image processing device captures the entire captured image. Therefore, the problem with conventional technology is that, after receiving the captured image, it takes a long time for the image processing function to process the image.

[0006] The present disclosure is primarily aimed at solving such a problem. In particular, the present disclosure aims to reduce the time required by the image processing function to process the captured image after it has been received. Solution to the problem

[0007] An image processing device according to the present disclosure comprises: a packet receiving unit for receiving a multitude of user data packets, which store a multitude of shared images obtained by sharing a captured image; and a package selection unit for selecting from the multitude of user data packages; a user data package that stores a split image to be subjected to image processing. Advantageous effects of the invention

[0008] According to the present disclosure, it is possible to shorten the time required by the function that performs image processing to acquire the captured image after receiving the captured image. Brief description of the drawings Fig. Figure 1 is a representation showing an example of a configuration of a position setting system according to embodiment 1. Fig. 2 represents a method for sending a recorded image according to embodiment 1. Fig. Figure 3 is a representation that shows an example of a hardware configuration of an image processing device according to embodiment 1. Fig. Figure 4 is a representation showing an example of a hardware configuration of a motion control unit according to embodiment 1. Fig. Figure 5 is a representation that shows an example of a functional configuration of the image processing device and the motion control unit according to embodiment 1. Fig. Figure 6 is a representation showing an example of a filter table according to embodiment 1. Fig. Figure 7 is a representation describing a method for determining a position setting application range according to embodiment 1. Fig. Figure 8 is a representation describing a method for determining whether to discard a package according to embodiment 1. Fig. Figure 9 is a flowchart that illustrates an example of the operation of the position setting system according to embodiment 1. Fig. Figure 10 is a representation showing an example of the operation of an image processing device according to embodiment 1. Fig. Figure 11 is a flowchart showing an example of the operation of a pattern acquisition unit according to embodiment 1. Fig. Figure 12 is a flowchart showing an example of the operation of a motion control unit according to embodiment 1. Description of embodiments

[0009] The embodiments are described below with reference to the drawings. In the following description of the embodiments and in the drawings, identical reference numerals denote identical or corresponding parts. Design 1.*** Configuration Description ***

[0010] Fig. Figure 1 illustrates an example of a configuration of a position setting system 1 according to the present embodiment.

[0011] In Fig. In position setting system 1, the position setting of a workpiece 2 is carried out using image processing.

[0012] The position setting system 1 includes a motion mechanism 100, a camera 200, an image processing device 300, a motion control unit 400 and a servo driver 500.

[0013] Workpiece 2 is, for example, a printed circuit board. Workpiece 2 has a feature part 3, which serves for position setting. Feature part 3 is, for example, an installed chip, an unimplemented pattern, an identification mark, or the like. Furthermore, feature part 3 can be configured with any combination of these features.

[0014] Since workpiece 2 is subject to position adjustment by the position adjustment system 1, workpiece 2 corresponds to a position adjustment object. Since workpiece 2 is also recorded by camera 200, workpiece 2 also corresponds to a recording object. The movement mechanism 100 moves the workpiece 2.

[0015] The motion mechanism 100 includes an XY table 110, a servo motor 120a and a servo motor 120b.

[0016] Servomotor 120a drives XY phase 110 in the X-axis direction. Servomotor 120b drives XY phase 110 in the Y-axis direction.

[0017] The movement mechanism 100 can include an XYθ stage instead of the XY phase 110.

[0018] Camera 200 captures the workpiece 2. Camera 200 then transmits the captured image to the image processing device 300. In the present embodiment, camera 200 transmits the captured image to the image processing device 300 via Ethernet (registered trademark). Furthermore, in this embodiment, camera 200 communicates with the image processing device 300 using the GigE Vision standard. GigE Vision is an interface standard for industrial cameras.

[0019] The image processing device 300 receives the image captured by the camera 200. Furthermore, the image processing device 300 performs image processing on the captured image and determines the position of the feature part 3 on the workpiece 2. Subsequently, the image processing device 300 adjusts the position of the workpiece 2 by determining the position of the feature part 3.

[0020] An operating sequence of the image processing device 300 corresponds to an image processing method. Furthermore, a program that implements the operation of the image processing device 300 is synonymous with an image processing program.

[0021] The motion control unit 400 generates a motion command to move the position of the workpiece 2 closer to a target position, based on the position of the feature part 3 specified by the image processing device 300. The motion control unit 400 then outputs the generated motion command to the servo driver 500.

[0022] The servo driver 500 performs feedback control of the servo motor 120a and the servo motor 120b based on the motion command received from the motion control unit 400.

[0023] Fig. Figure 2 illustrates a method by which the camera 200 divides the captured image into a large number of packets and sends them.

[0024] An image 4 of workpiece 2, captured by camera 200, is divided into a multitude of split images in raster scanning format. The size of each split image is uniform. These multiple split images are then stored in multiple data packets P(1) to P(N). Subsequently, a leader packet P(0), one or more data packets P(1) - P(N), and a trailer packet P(N+1) are sent sequentially. The leader packet P(0) is the first packet for the captured image 4. Each of the data packets P(1) - P(N) stores one split image. The trailer packet P(N+1) is the last packet for the captured image 4. Neither the leader packet P(0) nor the trailer packet P(N+1) stores a split image.

[0025] The leader packet P(0), each of the payload packets P(1) to P(N), and the trailer packet P(N+1) are assigned a block ID, which is a sequential number of the captured frame 4, and a packet ID, which is a sequential number of a packet within the captured frame 4. The block ID is an integer value that is incremented for each captured frame 4.

[0026] In Fig. In step 2, camera 200 assigns the block ID "1" to the captured image 4. The packet ID is an integer value that is incremented for each packet. The packet ID is initialized to "0" for each captured image 4. Fig. 2 assigns the package IDs “0” to “N+1” to the multitude of packages in sequence to the camera 200.

[0027] Fig. Figure 3 shows an example of a hardware configuration of the image processing device 300.

[0028] The image processing device 300 is a computer.

[0029] As in Fig. As shown in Figure 3, the image processing device 300 comprises a processor 301, a memory 302, a storage device 303, a network interface 304 and a motion control interface 305.

[0030] Fig. Figure 4 shows an example of a hardware configuration of the 400 motion control unit.

[0031] The motion control unit 400 is a computer.

[0032] As in Fig. As shown in Figure 4, the motion control unit 400 comprises a processor 401, a memory 402, a storage device 403, an image processing device interface 404 and a servo driver interface 405.

[0033] Fig. Figure 5 shows an example of functional configurations of the image processing device 300 and the motion control unit 400.

[0034] An Ethernet processing unit 310 receives a split image in the form of an Ethernet frame sent by camera 200. The Ethernet processing unit 310 then sends an Internet Protocol (IP) packet, which is a payload, to the UDP / IP processing unit 320.

[0035] More precisely, the Ethernet processing unit 310 selects from the in Fig. From the two displayed data packets P(1) - P(N), the Ethernet processing unit 310 selects a data packet that stores a split image, which is then subjected to image processing for the position setting of workpiece 2. Subsequently, the Ethernet processing unit 310 transmits the selected data packet to the UDP / IP processing unit 320. The Ethernet processing unit 310 discards all data packets except the selected one.

[0036] The Ethernet processing unit 310 corresponds to a packet receiving unit. Furthermore, the Ethernet processing unit 310, together with a packet rejection unit 311 described below, corresponds to a packet selection unit. Moreover, a process performed by the Ethernet processing unit 310 corresponds to a packet receiving process. Furthermore, a process performed by the Ethernet processing unit 310, together with a process performed by the packet rejection unit 311, corresponds to a packet selection process.

[0037] The UDP / IP processing unit 320 extracts a User Datagram Protocol (UDP) packet from the incoming IP packet. The UDP / IP processing unit 320 then forwards a GigE Vision packet, which is a payload of the UDP packet, to a GigE Vision processing unit 330.

[0038] The GigE Vision Processing Unit 330 creates a partial image from one or more GigE Vision packages, representing a portion of the captured image 4. The partial image created by the GigE Vision Processing Unit 330 is an image of a position setting usage area, which is described below. The GigE Vision Processing Unit 330 then passes the partial image to a Pattern Capture Unit 340.

[0039] The GigE Vision processing unit 330 determines, based on the in Fig. The block ID shown in Figure 2 indicates which captured image 4 the received GigE Vision packet belongs to. The GigE Vision processing unit 330 creates the partial image from the payload packets according to the sequence specified in Figure 2. Fig. The package ID shown in section 2 is specified.

[0040] It is assumed that the size of a single GigE Vision packet is such that it fits into a single UDP packet. It is assumed that the size of the single UDP packet is such that it fits into a single IP packet. It is assumed that the size of the single IP packet is such that it fits into a single Ethernet frame. To meet these conditions, a jumbo frame can be used.

[0041] The pattern capture unit 340 performs image processing on the partial image input by the GigE Vision processing unit 330. The pattern capture unit 340 then determines the position of feature part 3 on the workpiece 2.

[0042] The pattern capture unit 340 determines the position of feature part 3, for example, by means of pixel unit capture processing via a template comparison. Instead of or in addition to pixel unit capture processing, the pattern capture unit 340 can perform subpixel position determination processing. The pattern capture unit 340 then outputs a result (the position of a feature part) from the image processing to the motion control unit 400.

[0043] The pattern capture unit 340 corresponds to a packet specification unit together with a pixel area specification unit 341 and a packet rejection specification unit 331, which are described below. Furthermore, a process performed by the pattern capture unit 340 corresponds to a packet specification process together with a process performed by the pixel area specification unit 341 and the packet rejection specification unit 331, which are described below.

[0044] The packet rejection unit 311 uses a filter table 312 to determine whether the payload packet received by the Ethernet processing unit 310 is discarded or forwarded to the UDP / IP processing unit 320.

[0045] As described above, the packet rejection unit 311, together with the Ethernet processing unit 310, corresponds to the packet selection unit. Furthermore, the process performed by the packet rejection unit 311 corresponds to the packet selection process together with the process performed by the Ethernet processing unit 310.

[0046] The pixel area setting unit 341 defines an image area (hereinafter referred to as the position setting usage area) that will be used for position setting in a future captured image 4 to be captured at a future capture time. The pixel area setting unit 341 defines the position setting usage area based on the position of the feature part 3 at the future capture time, which is predicted by a position prediction unit 420, described below.

[0047] Each of the multiple split images that make up the position setting usage area is referred to as a split image position setting. The GigE Vision 330 processing unit described above creates the partial image that displays the position setting usage area by combining the multiple split images of the position setting. Details of the position setting usage area are described below.

[0048] As described above, the pixel area specification unit 341, together with the pattern capture unit 340 and the packet rejection specification unit 331, corresponds to the packet specification unit. Furthermore, the process performed by the pixel area specification unit 341 corresponds to the packet specification process together with the process performed by the pattern capture unit 340 and the packet rejection specification unit 331.

[0049] Packet Discard Unit 331 determines which payload packet is discarded. More precisely, Packet Discard Unit 331 calculates the block ID corresponding to the future captured image 4. Furthermore, Packet Discard Unit 331 calculates the packet ID of any payload packet that does not store a split image in the position setting, among the payload packets for the future captured image 4, based on the position setting usage range in the future captured image 4. Packet Discard Unit 331 then registers a calculation result in filter table 312.

[0050] In other words, the packet rejection setting unit 331 sets a payload packet that stores a split image of the position setting before the Ethernet processing unit 310 selects a payload packet for the future captured image 4.

[0051] As described above, the packet rejection specification unit 331 for packets to be rejected, together with the pattern capture unit 340 and the pixel area specification unit 341, corresponds to the packet specification unit. Furthermore, the process performed by the packet rejection specification unit 331 corresponds to the packet specification process, together with the process performed by the pattern capture unit 340 and the pixel area specification unit 341.

[0052] The functions of the Ethernet Processing Unit 310, the UDP / IP Processing Unit 320, and the like, which are in Fig. The elements shown in section 5 are implemented, for example, by programs. The elements shown in Fig. 3 Storage device 303 shown stores the programs that implement the functions of the Ethernet processing unit 310, the UDP / IP processing unit 320 and the like.

[0053] These programs are loaded from storage device 303 into memory 302. Processor 301 then executes these programs to operate Ethernet processing unit 310, UDP / IP processing unit 320, and the like.

[0054] In the motion control unit 400, a motion control unit 410 generates a motion command to move the position of the workpiece 2 closer to a target position. More precisely, the motion control unit 410 generates the motion command based on the position of the feature part 3, which is specified by the image processing device 300.

[0055] The motion control unit 410 then sends the generated motion command to the servo driver 500.

[0056] The position prediction unit 420 predicts the position of feature part 3 at the future recording time. More precisely, the position prediction unit 420 predicts the position of feature part 3 based on the past and current positions of feature part 3, which were specified by the pattern acquisition unit 340.

[0057] The position prediction unit 420 then informs the pixel area definition unit 341 of the predicted position of the feature part 3.

[0058] The functions of the motion control unit 410 and the position prediction unit 420, which are in Fig. The elements shown in section 5 are implemented, for example, by programs. The elements shown in Fig. The storage device 403 shown in Figure 4 stores the programs that implement the functions of the motion control unit 410 and the position prediction unit 420.

[0059] These programs are loaded from the storage device 403 into the memory 402. The processor 401 then executes these programs to operate the motion control unit 410 and the position prediction unit 420.

[0060] Fig. Figure 6 shows an example of filter table 312.

[0061] Filter table 312 is a table created in memory 302. Filter table 312 contains information for determining a user data packet that should be discarded in the future by the Ethernet processing unit 310.

[0062] In filter table 312 in Fig. 6 are pairs of the block ID and the packet ID registered as information to specify a payload packet that is to be discarded in the future. Fig. Figure 6 shows the packet ID of the payload packet to be discarded, which stores the split image of the captured image 4 with block ID "3". Specifically, the payload packet with packet ID "1" can be discarded. Furthermore, the payload packet with packet ID "2" can be discarded. Furthermore, the payload packet with packet ID "33" can be discarded. Furthermore, the payload packet with packet ID "34" can be discarded. Furthermore, the payload packet with packet ID "35" can be discarded.

[0063] Fig. Figure 6 shows an example of filter table 312 configured using a ring buffer. Filter table 312 can be configured using a hash set or hash mapping, where the key is a pair consisting of the block ID and the packet ID.

[0064] Fig. Figure 7 illustrates a procedure by which the position prediction unit 420 and the pixel area setting unit 341 define a position setting usage area 5.

[0065] First, the position prediction unit 420 predicts the position of feature part 3 at a time t1 (t1=t0+Δt), which corresponds to the future recording time, at a time t0.

[0066] More precisely, the position prediction unit 420 predicts the position of feature part 3 at time t1 based on the position of feature part 3 at time t0 and the predicted movement amount (the movement amount of workpiece 2 from time t0 to time t1) of feature part 3, which was calculated from the motion command generated by the motion control unit 410 at time t0.

[0067] In addition, the position prediction unit 420 can use a damped vibration model generated from previous position information about feature part 3 to predict the position of feature part 3.

[0068] Next, the pixel area setting unit 341 determines the entire view of feature part 3 and the rectangular area surrounding feature part 3 in the future image 4 to be captured at time t1 as the position setting usage area 5.

[0069] A split image showing at least part of the feature part 3 is referred to as a feature-split image. In the present embodiment, an image showing the entire view of the feature part 3 is configured by a combination of one or more feature-split images. Furthermore, an image showing at least part of the rectangular area around the feature part 3 is referred to as an environment-split image. In the present embodiment, an image showing the entire view of the surrounding rectangular area is configured by a combination of one or more environment-split images.

[0070] If a split image shows at least part of feature part 3 and at least part of the surrounding rectangular area, the split image is both a feature-split image and a surrounding-split image.

[0071] As described above, each of the plurality of split images that constitute the position setting area is referred to as a split image of the position setting or a split image by the position setting. For the sake of simplicity, in the present embodiment, each of the split images of the position setting is either a feature-split image or an environment-split image. In actual operation, the split image of the position setting may be a split image as described above that is both a feature-split image and an environment-split image.

[0072] The size of the position setting usage area 5 can be either fixed or variable, as long as it is large enough to fully encompass the entire view of the feature part 3.

[0073] As an example of a variable, it is conceivable that the size of the position setting application range 5 is increased if there is a possibility that the prediction accuracy of the position of the feature part 3 will decrease. If it is predicted that the movement path of the workpiece 2 from time t0 to time t1 is large, it is conceivable, for example, to increase the size of the position setting application range 5. Furthermore, if it is predicted that the vibration amplitude of the workpiece 2 from time t0 to time t1 is large, it is conceivable to increase the size of the position setting application range 5.

[0074] Fig. Section 8 describes a procedure by which the packet drop determination unit 331 determines which packet should be dropped. That is, Fig. Figure 8 illustrates a procedure by which the packet rejection specification unit 331 calculates the packet ID of a payload packet that stores a split image that does not correspond to a split image of the position setting. That is, Fig. Figure 8 illustrates a procedure by which the packet rejection setting unit 331 calculates the packet ID of a payload packet that stores a split image of the position setting.

[0075] Here, the current time is given as t0 [seconds] and the future recording time as t1 (t1=t0+Δt) [seconds]. The frame rate is also given as f [fps] and the block ID of the image 4 recorded at the current time t0 is given as b0.

[0076] The block ID of a future captured image 4, to be captured at future time t1, is calculated by b1 = b0 + f [fps] × Δt [seconds]. In the position setting using the future captured image 4, the leader package, the trailer package, and any payload packages containing split images (split images of the position setting) that correspond to the position setting usage range 5 are subjected to image processing in the pattern acquisition unit 340. In other words, a payload package containing a split image that does not correspond to the position setting usage range 5 can be discarded.

[0077] Here, the head and track coordinates obtained by raster scanning of the position setting usage area 5 are designated (xi, yi) and (xj, yj), respectively. Furthermore, the packet IDs of the payload packets storing the split images of the future captured image 4 are designated P(1) to P(N). Then, the packet ID of the payload packet storing the split image containing the head coordinates of the position setting usage area 5 is designated P(i), and the packet ID of the payload packet storing the split image containing the tail coordinates of the position setting usage area 5 is designated P(j). In this case, the payload packets with packet IDs P(1) to P(i-1) and the payload packets with packet IDs P(j+1) to P(N) are discarded. *** Description of a business ***

[0078] Next, an example of the operation of the position setting system 1 according to the present embodiment will be described.

[0079] Fig. Figure 9 shows an example of the position setting system 1 for performing the position setting of the workpiece 2 using image processing.

[0080] First, in step S100, when workpiece 2 is repositioned, the process continues with step S200.

[0081] In step S200, camera 200 captures workpiece 2, which is placed on the XY stage 110, and records a recording time. Camera 200 performs the capture of workpiece 2 in step S200 for each recording cycle.

[0082] Next, in step S300, camera 200 sends image 4, captured in step S200, and the capture time to image processing device 300. Camera 200 communicates with image processing device 300, for example, via the GigE Vision standard. The capture time is stored in the leader packet. The multiple images split from the captured image 4 are stored in one or more payload packets. Subsequently, as described in Fig. Figure 2 shows the leader packet, the payload packets, and the trailer packet being sent from camera 200 to image processing device 300. Camera 200 completes the transmission of step S300 until the start time of the next acquisition cycle.

[0083] In step S400, the image processing device 300 receives the leader packet, the payload packets, and the trailer packet sent by the camera 200 and acquires the split images and their acquisition times. The image processing device 300 then performs the image processing and determines the current position of the feature part 3 on the workpiece 2.

[0084] In step S500, the image processing device 300 notifies the motion control unit 410 and the position prediction unit 420 of the motion control unit 400 about the recording time and the current position of the feature part 3 specified in step S400.

[0085] The details of the processing by the image processing device 300, which is described in step S400 and step S500, are explained below.

[0086] In step S600, the motion control unit 410 generates a motion command to move the position of the workpiece 2 closer to a target position, based on the current position of the feature part 3 determined by the image processing device 300. In addition, the position prediction unit 420 predicts the position of the feature part 3 at the future acquisition time.

[0087] In step S700, the motion control unit 410 of the motion control unit 400 notifies the servo driver 500 of the motion command generated in step S600. Furthermore, the position prediction unit 420 of the motion control unit 400 informs the pixel area definition unit 341 of the image processing device 300 of the predicted position of the feature part 3, calculated in step S600.

[0088] The details of the processing of the motion control unit 400, as described in steps S600 and S700, are explained below.

[0089] In step S800, the servo driver 500 performs feedback control of the servo motors 120a and 120b to approximate the movement of the XY stage 110 to the movement command. Subsequently, the servo motors 120a and 120b drive the XY table 110 to change the position of the workpiece 2.

[0090] As specified in step S900, steps S200 to S800 are repeated until workpiece 2 moves into the target position.

[0091] Fig. Section 10 describes the detailed processing of the image processing device 300 in steps S400 and S500.

[0092] The image processing device 300 performs the processes of steps S410 to S440 in Fig. 10 for each image processing cycle. The length of the image processing cycle corresponds to the length of the acquisition cycle. The start time of the image processing cycle does not have to match the start time of the acquisition cycle.

[0093] In Fig. In version 10, the Ethernet Processing Unit 310 and the UDP / IP Processing Unit 320, as well as the GigE Vision Processing Unit 330 and the Pattern Capture Unit 340, operate on separate central processing unit (CPU) cores. Therefore, the Ethernet Processing Unit 310 and the UDP / IP Processing Unit 320, as well as the GigE Vision Processing Unit 330 and the Pattern Capture Unit 340, operate in parallel.

[0094] More precisely, while the GigE Vision processing unit 330 and the pattern capture unit 340 process the captured image 4 with block ID = n, the Ethernet processing unit 310 and the UDP / IP processing unit 320 perform the receiving process of a packet corresponding to the captured image 4 with block ID = n+1.

[0095] First, in step S410, the Ethernet processing unit 310 selects an Ethernet frame from the Ethernet frames that have arrived at the network interface 304, from which an IP packet must be entered into the UDP / IP processing unit 320.

[0096] More specifically, the Ethernet processing unit 310 uses the packet rejection unit 311 to determine whether a received Ethernet frame needs to be discarded or not.

[0097] Packet drop unit 311 refers to the header of the GigE Vision protocol of the received Ethernet frame. Packet drop unit 311 then determines whether the block ID and packet ID set held by the Ethernet frame matches the block ID and packet ID set registered in filter table 312.

[0098] If the group consisting of the block ID and packet ID of the Ethernet frame matches the group consisting of the block ID and packet ID in filter table 312, the packet drop unit 311 determines that the Ethernet frame must be dropped. The Ethernet processing unit 310 drops the Ethernet frame identified as to be dropped by the packet drop unit 311 without forwarding it to the UDP / IP processing unit 320. After the Ethernet frame is dropped, the packet drop unit 311 deletes the group consisting of the block ID and packet ID of the Ethernet frame from filter table 312.

[0099] If, however, the packet drop unit 311 does not determine that the Ethernet frame must be dropped, the Ethernet processing unit 310 forwards an IP packet of the Ethernet frame's payload to the UDP / IP processing unit 320. That is, the Ethernet processing unit 310 and the packet drop unit 311 select one IP packet from the multitude of IP packets to be processed.

[0100] Next, in step S420, the UDP / IP processing unit 320 extracts a UDP packet from the incoming IP packet. Additionally, the UDP / IP processing unit 320 extracts a GigE Vision packet from the payload of the extracted UDP packet.

[0101] The UDP / IP processing unit 320 then forwards the extracted GigE Vision packet to the GigE Vision processing unit 330. The GigE Vision packet that is fed into the GigE Vision processing unit 330 is either a leader packet, a payload packet, or a trailer packet.

[0102] In step S430, the GigE Vision Processing Unit 330 creates a partial image from the Leader package, one or more payload packages, and the Trailer package. The partial image created by the GigE Vision Processing Unit 330 is an image that shows the Position Setting Usage Area 5.

[0103] The GigE Vision Processing Unit 330 creates the sub-image by combining the split images (split images of the position setting) stored in all payload packets with the same block ID, in the order specified by the packet ID. If some of the payload packets were discarded in step S410, the GigE Vision Processing Unit 330 skips storing the split image in a memory area corresponding to the discarded payload packets during sub-image creation. In this case, the split image is not stored in the memory area. Therefore, a value (an initial value, an indefinite value, or the like) that was originally stored in the memory area is retained.

[0104] Once the GigE Vision processing unit 330 has completed the construction of the sub-image after receiving the trailer package, the GigE Vision processing unit 330 passes the sub-image, the capture time obtained from the leader package, and the head and tail coordinates of the sub-image obtained by raster scanning to the pattern capture unit 340.

[0105] The GigE Vision processing unit 330 transmits the coordinates of the first pixel of the split head image of the sub-image to the pattern capture unit 340 as the head coordinates of the sub-image. Furthermore, the GigE Vision processing unit 330 transmits the coordinates of the last pixel of the split image of the sub-image to the pattern capture unit 340 as the end coordinates of the sub-image.

[0106] The head coordinates of the sub-image are determined using Fig. 8 and are the coordinates (xk, yk) of the first pixel of the split image stored in the payload packet whose packet ID is P(i). The final coordinates of the sub-image are determined based on Fig. 8 described and are the coordinates (xl, yl) of the last pixel of the split image that is stored in the payload packet with packet ID P (j).

[0107] Fig. Figure 11 illustrates an example of the operation of the sample acquisition unit 340 in step S440.

[0108] In step S441, the GigE Vision processing unit 330 transmits a partial image, the acquisition time of the partial image, and the head and end coordinates of the partial image to the pattern capture unit 340.

[0109] Next, in step S442, the pattern capture unit 340 performs image processing on the partial image to determine the current position of feature part 3 on workpiece 2. To perform image processing efficiently, the pattern capture unit 340 can also process the image on a portion of the area outside the partial image. Furthermore, the pattern capture unit 340 can perform image processing with fill around the partial image.

[0110] Next, in step S443, the pattern capture unit 340 notifies the motion control unit 410 and the position prediction unit 420 of the motion control unit 400 about the position of a feature part 3 and the acquisition time of the partial image that was used to determine the position of the feature part 3.

[0111] Subsequently, in step S444, the pixel area setting unit 341, executed by the pattern acquisition unit 340, acquires the predicted position of feature part 3 in the future captured image 4 from the position prediction unit 420. At this point, the pixel area setting unit 341 can acquire information for setting the future captured image 4. The pixel area setting unit 341 receives information specifying a difference Δt between the capture time of the currently captured image 4 and the capture time of the future captured image 4, for example, as information for setting the future captured image 4. Furthermore, the pixel area setting unit 341 can receive information specifying a difference N between the block ID of the currently captured image 4 and the block ID of the future captured image 4, as information for setting the future captured image 4.

[0112] In the example of Fig. 10. The GigE Vision processing unit 330 and the pattern capture unit 340 process the sub-image of the captured image 4 with block ID = n, while the Ethernet processing unit 310 and the UDP / IP processing unit 320 have already started the receiving process of the packets that store the captured image 4 with block ID = n+1.

[0113] Therefore, the position forecast unit calculates 420 in the example of Fig. 10. From the partial image of the captured image 4 with block ID = n, the predicted position of feature part 3 in the captured image 4 with block ID = n+2 is obtained. That is, the difference between the block ID of the currently captured image 4 and the block ID of the future captured image 4 is 2.

[0114] Next, in step S445, the pixel area setting unit 341 uses the predicted position of feature part 3 determined in step S444 to define the position setting usage area 5 in the future captured image 4. The procedure for defining the position setting usage area 5 is based on Fig. 7 described.

[0115] The image 4 described above, with block ID = n, corresponds to the image 4 captured at time t0 in Fig. 7, and the above-described recorded image 4 with block ID = n+2 corresponds to the recorded image 4 at time t1 in Fig. 7.

[0116] In step S446, the packet rejection specification unit 331, executed by the pattern acquisition unit 340, determines which packet is to be rejected. Specifically, packet rejection specification unit 331 calculates the block ID corresponding to the future captured image 4. Furthermore, packet rejection specification unit 331 calculates the packet ID of a payload packet that does not store the split image of the position setting among the payload packets for the future captured image 4, based on the position setting usage area 5 in the future captured image 4. Subsequently, packet rejection specification unit 331 registers a calculation result in filter table 312.

[0117] The procedure for calculating the block ID and the packet ID is based on: Fig. 8 described. Furthermore, the registration contents in filter table 312 correspond to those in Fig. 6 described.

[0118] As specified in step S447, steps S441 to S446 are repeated until workpiece 2 moves into the target position.

[0119] Fig. Figure 12 shows an example of the operation of the motion control unit 400 in steps S600 and S700.

[0120] The motion control unit 400 performs steps S601 to S604 in Fig. 12 for each calculation cycle.

[0121] When the start time of the calculation cycle is reached in step S601, the motion control unit 410 and the position prediction unit 420 determine in step S602 the position of a feature part 3 determined by the image processing device 300 and the acquisition time of the acquired image 4 that was used to determine the position of the feature part 3.

[0122] In step S603, the motion control unit 410 generates the motion command to move the position of the workpiece 2 closer to the target position, based on the position of the feature part 3 specified by the image processing device 300.

[0123] Furthermore, the position prediction unit 420 predicts the position of feature part 3 at the future recording time.

[0124] The prediction method is based on Fig. 7 described.

[0125] Here, the current acquisition time is denoted as t0 [seconds], the future acquisition time as t1 (t0+Δt) [seconds], and the frame rate as f [fps]. When calculating the predicted position in the captured image 4 N pages later, starting from the currently captured image 4, Δt is represented as Δt [seconds] = 1 / f×N. Here, the value of N is determined, for example, before the start of the position setting between the image processing device 300 and the motion control unit 400. Alternatively, the image processing device 300 can inform the motion control unit 400 of the value of N during the position setting, if necessary.

[0126] In step S604, the motion control unit 410 notifies the servo driver 500 of the motion control unit 400 about the motion command generated in step S603.

[0127] Furthermore, the position prediction unit 420 communicates the predicted position of the feature part 3 calculated in step S603 to the motion control unit 400, the pixel area setting unit 341, and the image processing device 300.

[0128] When the position prediction unit 420 informs the pixel area definition unit 341 about the predicted position of the feature part 3, the position prediction unit 420 can also provide the pixel area definition unit 341 with information about which future captured image 4 corresponds to the predicted position. For example, the position prediction unit 420 provides the pixel area definition unit 341 with one of the values ​​Δt, t0+Δt, and N as such information.

[0129] As specified in step S605, steps S601 to S604 are repeated until workpiece 2 moves into the target position. *** Description of the effect of the embodiment ***

[0130] In the present embodiment, the position prediction unit 420 predicts the position of the feature part 3 at a future acquisition time. Subsequently, the pixel area setting unit 341 determines which position setting application area 5 of the acquired image 4 is to be captured at the future acquisition time. Furthermore, the packet rejection setting unit 331 specifies a future payload packet that is not required for position setting. Additionally, the packet rejection unit 311 rejects the unnecessary payload packet. Therefore, according to the present embodiment, it is possible to shorten the time from when the Ethernet processing unit 310 receives the captured image 4 from the network-connected camera 200 until the pattern acquisition unit 340 acquires the acquired image 4.This allows for high-speed position adjustment of a position adjustment object.

[0131] Furthermore, since the amount of data input into the UDP / IP processing unit 320 is reduced, it is possible to reduce the amount of memory used by a socket buffer, which is commonly used in a network receive process.

[0132] Furthermore, in the present embodiment, the camera 200 captures and transmits the entire workpiece 2, and the image processing device 300 discards an unnecessary area of ​​the captured image. Therefore, image processing for the area outside the feature part 3 can be started without delay, if required. This effect cannot be achieved by the conventional method, in which the amount of data transmitted by the camera 200 is reduced by capturing only a portion of the workpiece 2. This effect cannot be achieved by the conventional method, in which the amount of data transmitted is reduced by the camera 200 sending only a part (ROI: Region of Interest) of the captured image.

[0133] In the present embodiment, the image processing device 300 and the motion control unit 400 are separate devices; however, the image processing device 300 and the motion control unit 400 can be a single device. This allows the motion control unit 410 and the position prediction unit 420 to access the same memory as the pixel area setting unit 341. As a result, the position setting application area 5 in the captured image 4, which is acquired at a later, yet to be determined, acquisition time, can be captured at a higher speed.

[0134] Furthermore, in the present embodiment, the Ethernet processing unit 310 and the UDP / IP processing unit 320, as well as the GigE Vision processing unit 330 and the pattern capture unit 340, operate in parallel on separate CPU cores. Alternatively, they can be executed sequentially on a single CPU core. In this case, after the GigE Vision processing unit 330 and the pattern capture unit 340 have completed processing the sub-image with block ID = n, the Ethernet processing unit 310 and the UDP / IP processing unit 320 perform the packet reception process for the captured image 4 with block ID = n+1. Therefore, the position prediction unit 420 calculates the predicted position of feature part 3 in the captured image with block ID = n+1 from the sub-image with block ID = n.This means that the difference between the block ID of the currently captured image 4 and the block ID of the future captured image 4 is 1.

[0135] Furthermore, in the present embodiment, the size of the position setting application range 5 can be either variable or fixed. If the size of the position setting application range 5 is made variable, the average time from when the Ethernet processing unit 310 receives the captured image 4 until the pattern acquisition unit 340 acquires the captured image 4 is reduced. On the other hand, fixing the size of the position setting application range 5 means that an upper limit is set for the number of packets that are not discarded but processed by the image processing device 300.Therefore, if the size of the position setting usage area 5 is set, the maximum time from when the Ethernet processing unit 310 receives the captured image 4 until the pattern acquisition unit 340 acquires the captured image 4 is reduced. This allows the acquisition cycle to be shortened and more precise position setting to be achieved.

[0136] Furthermore, the present embodiment describes an example in which the Ethernet processing unit 310 selects a payload data packet that stores a split image to be used for image processing for position setting. The Ethernet processing unit 310 can be used not only for position setting but also for extracting a split image of the feature part 3 from a plurality of split images.

[0137] This means that the Ethernet processing unit 310 can receive a multitude of payload packets containing a multitude of split images obtained by splitting a captured image of a recording object that has feature part 3, which is not a position setting object. Subsequently, the Ethernet processing unit 310 can select from the multitude of payload packets one payload packet containing a split image with feature part 3 as the payload packet to be subjected to image processing.

[0138] The methods described in the present embodiment are examples.

[0139] Therefore, it is acceptable to carry out only a part of the procedures described in the present embodiment.

[0140] Alternatively, it is also permissible to carry out a combination of at least some of the methods described in the present embodiment and the methods not described in the present embodiment.

[0141] Alternatively, the configurations and procedures described in the present embodiment can be modified as required. ***Supplementary description of the hardware configuration***

[0142] Finally, a supplementary description of the hardware configurations of the image processing device 300 and the motion control unit 400 is given.

[0143] The 301 processor is an integrated circuit (IC) that performs processing.

[0144] The 301 processor is a (CPU), a digital signal processor (DSP), or the like.

[0145] Memory 302 is a random access memory (RAM).

[0146] The storage device 303 is a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or similar.

[0147] The network interface 304 and the motion control unit interface 305 are each electronic circuits that perform a communication process for data.

[0148] For example, the network interface 304 and the motion control unit interface 305 are each a communication chip or a network interface card (NIC).

[0149] In addition, the 303 storage device also stores an operating system (OS).

[0150] Then at least part of the OS will be executed by the 301 processor.

[0151] While at least part of the operating system is running, the processor 301 executes programs that implement the functions of the Ethernet processing unit 310, the UDP / IP processing unit 320, and the like.

[0152] The 301 processor, which runs the operating system, handles tasks such as task management, memory management, file management, communication control, and the like.

[0153] In addition, at least one of the information, data, signal values ​​and variable values ​​indicating the results of processes of the Ethernet processing unit 310, the UDP / IP processing unit 320 and the like is stored in at least one of the memory 302, the storage device 303 and a register and a cache memory in the processor 301.

[0154] Furthermore, programs implementing the functions of the Ethernet Processing Unit 310, the UDP / IP Processing Unit 320, and the like can be stored on a portable recording medium such as a magnetic disk, a flexible disk, an optical disk, a compact disc, a Blu-ray disc (registered trademark), or a DVD. Subsequently, the portable recording medium containing the programs implementing the functions of the Ethernet Processing Unit 310, the UDP / IP Processing Unit 320, and the like can be distributed.

[0155] The 401 processor is an integrated circuit (IC) that performs computational processing.

[0156] The processor 401 is a CPU, a DSP, or something similar.

[0157] The 402 RAM is also a type of memory.

[0158] The 403 storage device is also a ROM, a flash memory, a hard drive, or something similar.

[0159] Each of the image processing device interface 404 and servo driver interface 405 is also an electronic circuit that performs a communication process for data.

[0160] Each of the image processing device interface 404 and servo driver interfaces 405, for example, is also a communication chip or a network card. Furthermore, the 403 storage device also stores an operating system.

[0161] Then at least part of the OS will be executed by processor 401.

[0162] While at least part of the operating system is running, the processor 401 executes programs that implement the functions of the motion control unit 410 and the position prediction unit 420.

[0163] The 401 processor, which runs the operating system, handles tasks such as task management, memory management, file management, communication control, and the like.

[0164] In addition, at least one of the information, data, signal values ​​and variable values ​​that indicate the results of the processes of the motion control unit 410 and the position prediction unit 420 will be stored in the memory 402, the storage device 403 and a register and a cache memory in the processor 401.

[0165] Furthermore, the programs implementing the functions of the Motion Control Unit 410 and the Position Prediction Unit 420 can be stored on a portable storage medium such as a magnetic disk, a flexible disk, an optical disk, a compact disc, a Blu-ray disc (registered trademark), or a DVD. The portable storage medium containing the programs implementing the functions of the Motion Control Unit 410 and the Position Prediction Unit 420 can then be distributed.

[0166] Furthermore, the “unit” of at least one of the Ethernet processing units 310, the UDP / IP processing units 320, and the like can be read as “circuit,” “step,” “procedure,” “process,” or “circuit arrangement.” The “unit” of at least one of the motion control unit 410 and the position prediction unit 420 can also be read as “circuit,” “step,” “procedure,” “process,” or “circuit arrangement.”

[0167] Furthermore, the image processing device 300 and the motion control unit 400 can be implemented using a processing circuit. The processing circuit is, for example, a logic integrated circuit (IC), a gate array (GA), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0168] In this case, each of the Ethernet processing units 310, the UDP / IP processing units 320, the packet drop units 311, and the like is implemented as part of the processing circuit. The motion control unit 410 and the position prediction unit 420 are also implemented as part of the processing circuit.

[0169] In this description, an overarching concept of the processor and the processing circuit is referred to as the "processing circuit".

[0170] That is, the processor and the processing circuit are each a specific example of the "processing circuit". Reference symbol list

[0171] 1: Position setting system; 2: Workpiece; 3: Feature part; 4: Captured image; 5: Position setting application area; 100: Motion mechanism; 110: XY phase; 120a: Servo motor; 120b: Servo motor; 200: Camera; 300: Image processing device; 301: Processor; 302: RAM; 303: Storage device; 304: Network interface; 305: Motion control unit interface; 310: Ethernet processing unit; 311: Packet discard unit; 312: Filter table; 320: UDP / IP processing unit; 330: GigE vision processing unit; 331: Packet discard setter; 340: Pattern capture unit; 341: Pixel area setter; 400: Motion control unit; 401: Processor; 402: Memory; 403: Storage device; 404: Image processing device interface; 405: Servo driver interface; 410: Motion control unit; 420: Position prediction unit; 500: Servo driver. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2019-003388 A

[0004]

Claims

[1] Image processing device comprising: a packet receiving unit for receiving a multitude of user data packets, which store a multitude of shared images obtained by sharing a captured image; and a packet selection unit to select from the multitude of payload packets a payload packet that stores a split image to be subjected to image processing. [2] Image processing device according to claim 1, wherein the packet selection unit discards a user data packet that is not the selected user data packet from the plurality of user data packets. [3] Image processing device according to claim 1, wherein The packet receiving unit receives a multitude of user data packets, which store a multitude of shared images obtained by sharing a captured image of a position setting object, and The packet selection unit selects a payload packet that stores an image split by a position setting, which is a split image that is to be subjected to image processing for the position setting of the position setting object, as the payload packet that stores the split image that is to be subjected to image processing. [4] Image processing device according to claim 3, wherein the position setting object has a feature part, the multitude of payload data packages includes a payload data package that stores a feature-split image, which is a split image that shows the feature part, and The packet selection unit selects the payload packet that stores the feature-split image rather than the payload packet that stores the image split by a position setting. [5] Image processing device according to claim 4, wherein the multitude of payload data packages includes a payload data package that stores an environment-split image, which is a split image that shows an area around the feature part, and The packet selection unit selects a payload packet that stores the environment-split image rather than the payload packet that stores the image split by the position setting. [6] Image processing device according to claim 3, further comprising: a packet specification unit for specifying a payload packet that stores the image split by the position setting, prior to the selection of a payload packet by the packet selection unit, wherein The packet selection unit selects the payload packet specified by the packet specification unit. [7] Image processing device according to claim 6, wherein the position setting object has a feature part, the multitude of payload packages include a payload package that stores a feature-split image, which is a split image showing the feature part, and a payload package that stores an environment-split image, which is a split image showing an area around the feature part, and The package specification unit specifies the payload packet that stores the feature-split image and the payload packet that stores the environment-split image as payload packets that store the image split by the position setting. [8] Image processing device according to claim 7, wherein The package specification unit receives a predicted position of the feature part at time t1, which is predicted based on the position of the feature part at time t0 and the movement amount of the position setting object between time t0 and time t1, which is a time after time t0. a payload data package that stores an image split by the position setting at time t1, among a multitude of payload data packages at time t1 that store a multitude of split images at time t1, which were obtained by splitting a captured image of the position setting object at time t1 using the obtained predicted position of the feature part at time t1. [9] Image processing device according to claim 6, wherein the packet specification unit uses a fixed-size image split by the position setting. [10] Image processing device according to claim 1, wherein the packet receiving unit receives a plurality of user data packets which store a plurality of split images obtained by splitting the captured image into raster scanning format. [11] Image processing device according to claim 1, wherein The packet receiving unit receives a multitude of user data packets, which store a multitude of split images obtained by splitting a captured image of a recording object with a feature part, and The package selection unit selects a payload package that stores a split image showing the feature part, rather than the payload package that stores the split image that has undergone image processing. [12] Image processing techniques, including: through a computer, receiving a multitude of payload data packets that store a multitude of shared images obtained by sharing a captured image; and by the computer, selecting a payload package from the multitude of payload packages that stores a split image to be subjected to image processing. [13] Image processing program that causes a computer to perform the following: a packet reception process for receiving a multitude of payload packets that store a multitude of shared images obtained by sharing a captured image; and a package selection procedure for selecting from the multitude of payload packages; a payload package that stores a split image to be subjected to image processing.

Images