Program and data processing device

Abstract

Reduce the possibility of selecting the wrong workpiece. [Solution] A specific image is obtained, which is an image of the placement area for processing multiple workpieces. The specific image is an image taken with multiple workpieces, either unprocessed or processed, placed in the placement area. The multiple workpieces include a first workpiece and a second workpiece, each differing from the others in one or more of the following items: material, shape, and processing content. An output process is performed to output reference information, which is used to pick up the processed multiple workpieces, to the picker using the specific image. The reference information includes first information, which associates the first identifier and first position information of the first workpiece, and second information, which associates the second identifier and second position information of the second workpiece.

Description

Technical Field

[0001] This specification relates to a technique for picking up a plurality of workpieces.

Background Art

[0002] Various processes are performed on various workpieces. Patent Document 1 discloses a technique for printing an image on a printing medium. This technique fixes an ink image on the printing medium by ejecting an ultraviolet-curable ink onto the printing medium and irradiating the ink image with ultraviolet rays from an ultraviolet lamp.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Processing (for example, printing, laser engraving, etc.) can be performed on a plurality of workpieces. The plurality of workpieces can be processed together, and each workpiece can be picked up after the processing. The picking up can be performed, for example, by a person or a robot. Here, when the plurality of processed workpieces include a plurality of different types of workpieces, there may be a case of misselection of the workpieces.

[0005] This specification discloses a technique for reducing the possibility of misselection of workpieces.

Means for Solving the Problems

[0006] The technique disclosed in this specification can be realized as the following application examples.

[0007] [Application Example 1] A program that causes a computer to implement the following functions: a function to acquire a specific image which is an image of a placement area for processing multiple workpieces, wherein the specific image is an image taken with multiple unprocessed or processed workpieces placed in the placement area, and the multiple workpieces include a first workpiece and a second workpiece in which one or more items among material, shape, and processing content are different from each other; and a function to output to a picker using the specific image which references reference information to pick up the multiple processed workpieces, wherein the reference information includes first information which is information that associates a first identifier and first position information of the first workpiece, and second information which is information that associates a second identifier and second position information of the second workpiece.

[0008] With this configuration, reference information is output to the picker using a specific image taken with multiple workpieces placed in the placement area. The reference information includes first information, which associates the first identifier and first position information of the first workpiece, and second information, which associates the second identifier and second position information of the second workpiece. Therefore, the possibility of incorrect selection between the first and second workpieces is reduced.

[0009] Furthermore, the technologies disclosed herein can be implemented in various forms, for example, as data processing methods and data processing devices, computer programs for realizing the functions of such methods or devices, and recording media (e.g., non-temporary recording media) on which such computer programs are recorded. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram illustrating an example of a manufacturing system. [Figure 2] This is a flowchart illustrating an example of a partial manufacturing process. [Figure 3] This is a flowchart illustrating an example of a partial manufacturing process. [Figure 4]This is a diagram illustrating an example of batch information. [Figure 5] Figures (A)-(G) show examples of images processed by partial manufacturing processes. [Figure 6] This diagram illustrates an example of layout. [Figure 7] (A) is a diagram showing an example of a label sheet. (B)-(D) are diagrams showing trays. [Figure 8] Figures (A)-(G) show examples of images processed by partial manufacturing processes. [Figure 9] This is a block diagram representing another embodiment of the manufacturing system. [Figure 10] This is a flowchart illustrating an example of a partial manufacturing process. [Figure 11] This is a flowchart illustrating an example of a partial manufacturing process. [Figure 12] This is a flowchart illustrating an example of a partial manufacturing process. [Figure 13] (A) and (B) are diagrams representing another example of the reference information. [Modes for carrying out the invention]

[0011] A. First Example: A1. System configuration: Figure 1 is a block diagram representing an example of a manufacturing system. Manufacturing system 1000 performs a partial manufacturing process, which is part of the product manufacturing process. In this embodiment, the product is an acrylic ornament, specifically an acrylic sheet with an image printed on it. The partial manufacturing process includes processing an acrylic sheet, which is an example of a workpiece. In this embodiment, processing the workpiece includes printing an image onto the workpiece. An acrylic sheet with a printed image is also called an acrylic charm.

[0012] In this embodiment, the manufacturing system 1000 includes a data processing device 200, a first printing device 400, a camera 500, and a second printing device 700. These devices 200, 400, 500, and 700 are connected to a local area network NT.

[0013] The data processing device 200 is, for example, a personal computer. The data processing device 200 executes a process for assisting in the processing of work (details will be described later). The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus not shown in the figure. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0014] The processor 210 is a device configured to perform data processing, and is, for example, a Central Processing Unit (CPU) or a System on a chip (SoC). The volatile storage device 220 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores the data of the program 231 and the object detection model 232, respectively. In this embodiment, the object detection model 232 is a program module that forms a trained machine learning model. Details of the program 231 and the object detection model 232 will be described later.

[0015] The display unit 240 is a device configured to display an image, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive an operation by a user, such as a button, a lever, or a touch panel disposed on top of the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. The user can input various requests and instructions to the data processing device 200 by operating the operation unit 250. The display unit 240 may display operation elements (for example, buttons, sliders, etc.), and the displayed elements may be operated through the operation of the operation unit 250.

[0016] The communication interface 270 is an interface for communicating with other devices (including, for example, one or more of a USB interface, a wired LAN interface, and a wireless interface of IEEE 802.11). The communication interface 270 is connected to the local area network NT.

[0017] The first printing device 400 includes a printing execution unit 410 and a printing control device 490 that controls the printing execution unit 410. In this embodiment, the printing execution unit 410 prints an image on a workpiece. The configuration of the printing execution unit 410 may be various configurations. In this embodiment, the printing execution unit 410 includes a head 412, a light source 413, a platen 416 that supports the workpiece from below, and a moving device 418 that moves the platen 416. The head 412, the light source 413, and the moving device 418 are arranged inside the printing execution unit 410. The printing control device 490 may be an external device connected to the printing execution unit 410, or alternatively, may be a device arranged inside the printing execution unit 410.

[0018] The moving device 418 includes a rail that slidably supports the platen 416 and an electric motor that slides the platen 416 along the rail (not shown). The moving device 418 moves the platen 416 between an outer position where a predetermined placement area AA on the upper surface of the platen 416 is located outside the printing execution unit 410 and an inner position where the placement area AA is located inside the printing execution unit 410. With the platen 416 in the outer position, a workpiece is placed on the placement area AA of the platen 416 (for example, workpieces Wa1, Wa2, Wb1, Wb2). After this, the platen 416 moves toward the inside of the printing execution unit 410. The head 412 forms an ink image on the workpiece by spraying ultraviolet-curable ink onto the workpiece. By adjusting the position of the platen 416 relative to the head 412, the head 412 can form an ink image on workpieces at various positions on the placement area AA. The light source 413 fixes the ink image on the workpiece by irradiating it with ultraviolet light. After the ink image is fixed, the platen 416 moves back to the outer position. In this state, the workpiece with the printed image is picked up from the platen 416.

[0019] Figure 1 shows four workpieces Wa1, Wa2, Wb1, and Wb2. Workpieces Wa1 and Wa2 are elliptical acrylic plates, while workpieces Wb1 and Wb2 are rectangular acrylic plates. These workpieces Wa1, Wa2, Wb1, and Wb2 are placed on a platen 416. The printing execution unit 410 prints images onto workpieces Wa1, Wa2, Wb1, and Wb2. Workpieces Wa1x, Wa2x, Wb1x, and Wb2x, each bearing the printed image, are formed from workpieces Wa1, Wa2, Wb1, and Wb2. Workpieces Wa1x, Wa2x, Wb1x, and Wb2x are then placed in trays TR1, TR2, and TR3, prepared according to the type of workpiece, for the next printing process (details will be described later).

[0020] The print control device 490 is a computer similar to the data processing device 200. Although not shown in the figures, the print control device 490 includes a processor, volatile memory, non-volatile memory, and a communication interface. The print control device 490 is connected to the local area network NT. The print control device 490 controls the print execution unit 410 according to instructions received through the local area network NT.

[0021] Camera 500 is a digital camera that photographs a subject and generates image data representing the photographed subject. In this embodiment, the position and orientation of camera 500 are pre-adjusted so that the entire placement area AA of the platen 416, which is located at an outer position, is included in the shooting range of camera 500. With a workpiece placed on the placement area AA of the platen 416, camera 500 generates image data representing the workpiece on the placement area AA of the platen 416 (details will be described later). Camera 500 is connected to a local area network NT. Camera 500 takes pictures according to instructions received through the local area network NT. Camera 500 transmits the image data to another device (e.g., data processing device 200) through the local area network NT.

[0022] The second printing apparatus 700 includes a printing execution unit 710 and a printing control device 790 that controls the printing execution unit 710. In this embodiment, the printing execution unit 710 prints labels on roll paper. The configuration of the printing execution unit 710 can vary. Although not shown in the figures, in this embodiment, the printing execution unit 710 includes a holder for supporting the roll paper, a transport device for transporting the roll paper, and a print head for printing an image on the roll paper. The portion of the roll paper on which the label image has been printed is separated from the roll paper and used as a label sheet LS. The printing control device 790 may be an external device connected to the printing execution unit 710, or it may be a device located inside the printing execution unit 710.

[0023] Figure 1 shows label sheets LS1, LS2, and LS3 placed in trays TR1, TR2, and TR3, respectively. Although not shown in the illustration, each label sheet LS1, LS2, and LS3 has an image printed on it that represents the workpiece identifier. Label sheets LS1, LS2, and LS3 are used to represent the identifier associated with trays TR1, TR2, and TR3.

[0024] The print control device 790 is a computer similar to the data processing device 200. Although not shown in the figures, the print control device 790 includes a processor, a volatile memory device, a non-volatile memory device, and a communication interface. The print control device 790 is connected to the local area network NT. The print control device 790 controls the print execution unit 710 according to instructions received through the local area network NT.

[0025] A2. Partial manufacturing process: Figures 2 and 3 are flowcharts illustrating an example of partial manufacturing. Figure 3 shows the process following Figure 2. In this embodiment, the workpiece processed by the partial manufacturing process is an acrylic sheet cut into a predetermined shape. The partial manufacturing process includes printing an image onto the acrylic sheet. In this embodiment, multiple types of workpieces for multiple types of products are processed together. The processor 210 of the data processing device 200 (Figure 2) proceeds with the partial manufacturing process according to the program 231. In this embodiment, the operator inputs a start instruction for the partial manufacturing process to the data processing device 200 by operating the operation unit 250 of the data processing device 200. The processor 210 starts the partial manufacturing process according to the start instruction. Note that the start instruction may be input to the data processing device 200 via the communication interface 270 by a device other than the data processing device 200.

[0026] In S210, the processor 210 acquires batch information data. Batch information represents the processing details of multiple types of workpieces that are processed together. Figure 4 is a diagram showing an example of batch information. Batch information 233 shows the correspondence between identifier ID, workpiece shape WS, total number Nw, and image data IMd. Workpiece shape WS indicates the shape of the workpiece. In this embodiment, workpiece shape WS is set to an elliptical shape WSa or a rectangular shape WSb. Total number Nw indicates the total number of workpieces. Image data IMd indicates image data for printing.

[0027] Batch information 233 represents the correspondence between multiple types of workpieces. In the example in Figure 4, batch information 233 represents three types of workpieces associated with three identifiers ID1-ID3. The workpiece shapes WS for identifiers ID1 and ID2 are elliptical WSa, and the workpiece shape WS for the third identifier ID3 is rectangular WSb. The total number Nw for identifiers ID1 and ID2 is 1, and the total number Nw for the third identifier ID3 is 2. Identifiers ID1-ID3 are associated with mutually different image data I1d, I2d, and I3d (Figure 4 shows an overview of images I1, I2, and I3 represented by image data I1d, I2d, and I3d). Since the total number Nw for the third identifier ID3 is 2, the total number of workpieces to be processed is 4.

[0028] Works Wa1 and Wa2 in Figure 1 are raw workpieces associated with identifiers ID1 and ID2. Works Wb1 and Wb2 are raw workpieces associated with a third identifier ID3. Multiple workpieces represented by batch information are prepared in advance. For example, as shown in Figure 1, raw workpieces Wa1, Wa2, Wb1, and Wb2 are placed in box BX. Workpieces Wa1, Wa2, Wb1, and Wb2 may be placed in box BX by a robot or by a person.

[0029] In this embodiment, multiple sets are processed sequentially based on multiple batch information (each set is a set of multiple workpieces to be processed together). In S210 (Figure 2), the processor 210 acquires unprocessed batch information as target batch information, which is the batch information to be processed. Batch data representing multiple batch information is prepared in advance. For example, a start instruction may include data information that specifies the batch data. The data information may specify batch data stored in various storage devices. The storage device may be selected from, for example, storage device 215 (e.g., non-volatile storage device 230), a storage device (not shown) connected to the communication interface 270 (e.g., USB flash drive), or a storage device of a server that can communicate with the data processing device 200. The processor 210 acquires the batch data specified by the start instruction and acquires the target batch information data by referring to the batch data. Hereinafter, batch information 233 (Figure 4) will be assumed to be the target of processing. The identifier ID indicated by the target batch information is also called the target identifier.

[0030] In S215 (Figure 2), the processor 210 starts capturing images of the platen 416 (Figure 1) in placement area AA with the camera 500. In this embodiment, the processor 210 sends a command to the camera 500 to start capturing images. The camera 500 starts capturing images according to the command. In this embodiment, the camera 500 periodically repeats capturing still images (for example, the camera 500 takes a picture every second). The camera 500 then sends the captured image data to the data processing device 200. The camera 500 may also start capturing moving images. The camera 500 then sends the moving image data to the data processing device 200 (multiple frame images included in a moving image are an example of captured images).

[0031] In S220, the workpieces are placed on the placement area AA of the platen 416. In this embodiment, the operator manually places the workpieces Wa1, Wa2, Wb1, and Wb2 from the box BX (Figure 1) onto the placement area AA. Each workpiece Wa1, Wa2, Wb1, and Wb2 is placed so that they do not overlap each other. The position and orientation of each workpiece Wa1, Wa2, Wb1, and Wb2 on the placement area AA may be set arbitrarily. Alternatively, the position and orientation of each workpiece Wa1, Wa2, Wb1, and Wb2 on the placement area AA may be set to a predetermined position and orientation for each workpiece. Furthermore, the placement of workpieces on the placement area AA may be performed by a robot (not shown). In any case, after the start of shooting by the camera 500 (S215), the processor 210 proceeds to S225 without waiting for the placement of multiple workpieces to be completed.

[0032] In S225 (Figure 2), the processor 210 obtains the latest captured image data from the captured image data provided by the camera 500. Figures 5(A)-5(G) show examples of images processed by the partial manufacturing process. Figure 5(A) shows an example of a captured image. The captured image IM01 is a rectangular image having two sides parallel to the first direction Dx and two sides parallel to the second direction Dy which is perpendicular to the first direction Dx. The data of the captured image IM01 is bitmap data representing the color values ​​of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. The color values ​​are represented, for example, by the gradation values ​​of red R, green G, and blue B (for example, values ​​between zero and 255).

[0033] Image IM01 in Figure 5(A) shows the entire placement area AA of the platen 416. Image IM01 shows the state before the workpiece is placed on the placement area AA.

[0034] In S230 (Figure 2), the processor 210 detects the workpiece from the captured image. There are various methods for detecting the workpiece. In this embodiment, the processor 210 detects the workpiece from the captured image using a trained object detection model 232 (Figure 1). The object detection model 232 may be any machine learning model capable of detecting workpieces. In this embodiment, the object detection model 232 is a model called "RTMDet" disclosed in the following paper. Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang and Kai Chen. "Rtmdet: An Empirical Study of Designing Real-Time Object Detectors", arXiv.2212.07784, December 16, 2022, https: / / doi.org / 10.48550 / arXiv.2212.07784.

[0035] RTMDet is a model that detects the bounding box and category (i.e., the type of object) of an object, and performs pixel-level region segmentation called instance segmentation. In this embodiment, the object detection model 232 is pre-trained to detect the bounding boxes, types, and regions of multiple types of objects to be detected, including multiple types of workpieces before the image is printed and multiple types of workpieces after the image is printed. In this embodiment, the object detection model 232 is trained to detect elliptical WSa (Figure 4) workpieces (e.g., workpieces Wa1 and Wa2 in Figure 1) and rectangular WSb workpieces (e.g., workpieces Wb1 and Wb2 in Figure 1) as different types of workpieces. The training method for the object detection model 232 may be various, for example, the training method described in the above-mentioned RTMDet paper.

[0036] When the captured image IM01 in Figure 5(A) is processed, the workpiece is not detected. As will be described later, if the captured image represents the workpiece, the workpiece is detected from the captured image.

[0037] In S235 (Figure 2), the processor 210 uses the work detection results to determine the placement status of each of the multiple types of work. The processor 210 determines the placement status of each of the multiple target identifiers (here, identifiers ID1-ID3 (Figure 4)). In this embodiment, the placement status is determined to be either "placed" or "not placed". If Nw workpieces having workpiece shape WS are detected, the processor 210 determines the placement status to be "placed". If the number of detected workpieces having workpiece shape WS is less than Nw, the processor 210 determines the placement status to be "not placed".

[0038] For example, if the total number of detected workpieces having a rectangular WSb (Figure 4) is less than 2, the processor 210 determines the placement status of the third identifier ID3 to "Not Placed". If two workpieces having a rectangular WSb are detected, the processor 210 determines the placement status of the third identifier ID3 to "Placed". If the total number of detected workpieces having the workpiece shape WS is greater than the total number Nw, the processor 210 may determine the placement status to "Placed".

[0039] As shown in identifiers ID1 and ID2 (Figure 4), the same raw workpiece (in this case, the same workpiece shape WS) may be common to multiple identifier IDs. Here, we assume that the same raw workpiece (for example, the same workpiece shape WS) is associated with K identifier IDs (where K is an integer greater than or equal to 2). The processor 210 may determine the placement status of each of the K identifier IDs by comparing the sum of the K numbers Nw for the K identifier IDs with the number of workpieces detected. Specifically, if the number of workpieces detected is less than the sum of the K numbers Nw, the processor 210 may determine the placement status of each of the K identifier IDs to "not placed". For example, if the sum of the K numbers Nw is 2 and the number of workpieces detected is 1, the placement status of the K identifier IDs may be determined to be "not placed". Also, the number of detected K items Nd may be set to the value obtained by dividing the number of detected workpieces by K. For example, if the number of detected workpieces is 1, the number of detected items Nd may be set to 1 / K. If the number of detected workpieces is greater than or equal to the sum of the number Nw of K workpieces, the processor 210 may determine the placement status of K workpieces to "placed". In the example in Figure 4, if the total number of detected workpieces having an elliptical shape WSa (Figure 4) is less than 2, the processor 210 determines the placement status of identifiers ID1 and ID2 to "not placed". If two or more workpieces having an elliptical shape WSa are detected, the processor 210 determines the placement status of identifiers ID1 and ID2 to "placed".

[0040] In S240 (Figure 2), the processor 210 displays the placement status on the display unit 240. Figure 5(B) shows an example of the displayed screen. The status screen D01 shows the superimposed image IM01o and the status table TSa. The superimposed image IM01o is an image in which the work detection results are superimposed on the captured image IM01 used for work detection (in the example of Figure 5(B), since no work was detected from the captured image IM01, the superimposed image IM01o is the same as the captured image IM01). The status table TSa shows the correspondence between identifier ID and placement status STa. The placement status STa indicates either "OK" or "NG". "OK" indicates "placed", and "NG" indicates "not placed". In the example of Figure 5(B), the placement status STa for all identifier IDs is "NG". By observing the status screen D01, the operator can easily recognize the identifiers of unplaced work.

[0041] In this embodiment, the status table TSa further represents the workpiece shape WS, the number of detected workpieces Nd, the total number Nw, and the target image IMt, which is an image represented by the image data IMd (Figure 4). By observing the status screen D01, the operator can easily recognize the type of unplaced workpiece.

[0042] In S240 (Figure 2), the processor 210 generates data for the status screen D01 and uses the generated data to display the status screen D01 on the display unit 240.

[0043] In S245, the processor 210 determines whether all workpieces can be detected from the captured image. In this embodiment, the processor 210 determines that all workpieces have been detected if Nw workpieces for each of the multiple target identifiers (here, identifiers ID1-ID3 (Figure 4)) are detected from the captured image. If the number of detected workpieces for one or more target identifiers is less than the total number Nw, the processor 210 determines that one or more workpieces have not been detected. Note that the method for calculating the number of detected workpieces when the same workpiece is common to multiple identifier IDs is the same as the method for calculating the number of detections Nd in S235.

[0044] If it is determined that one or more workpieces are not detected (S245: No), the processor 210 repeats the process from S225 to S245 until all workpieces are detected from the captured image. The processor 210 displays the superimposed image and the placement status STa in near real time. During the repetition of the S225-S245 process, the placement of the workpieces (S220) proceeds.

[0045] Figure 5(C) shows another example of a captured image. Captured image IM02 is an image taken with three workpieces Wa1, Wa2, and Wb1 placed in placement area AA. When captured image IM02 is processed, in S230 (Figure 2), the processor 210 detects workpieces Wa1, Wa2, and Wb1 from captured image IM02. Object regions OB01-OB03 in the figure show examples of the regions of the detected workpieces. Object regions OB01-OB03 represent workpieces Wa1, Wa2, and Wb1, respectively. As described above, the object detection model 232 detects the type of object. The types of object regions OB01 and OB02 are elliptical workpieces WSa, and the type of object region OB03 is a rectangular workpiece WSb.

[0046] Figure 5(D) shows another example of the status screen. Status screen D02 shows an example of the screen displayed in S240 (Figure 2) when the captured image IM02 (Figure 5(C)) is processed. Status screen D02 shows the superimposed image IM02o and the status table TSa. The superimposed image IM02o is an image in which the contour lines of the detected object regions OB01-OB03 are superimposed on the captured image IM02. The status table TSa shows the number of detections Nd and the placement status STa updated from the placement status STa in Figure 5(B). In the example in Figure 5(D), two elliptical workpieces WSa are detected. Therefore, for identifiers ID1 and ID2 of "103" and "104", the number of detections Nd is set to 1 and the placement status STa is set to "OK". Also, the number of detected rectangular workpieces WSb is 1. Therefore, for identifier ID3 of "105", the detection count Nd is set to 1, and the placement status STa is set to "NG". By observing the status screen D02, the operator can easily recognize that there are not enough rectangular workpieces.

[0047] Figure 5(E) shows another example of a captured image. Captured image IM03 is an image taken with workpieces Wa1, Wa2, and Wb1, as described in Figure 5(C), plus workpiece Wb2, placed in placement area AA. In S230 (Figure 2), the processor 210 detects workpieces Wa1, Wa2, and Wb1, as well as workpiece Wb2. The processor 210 acquires object areas OB01-OB03, as described in Figure 5(C), plus object area OB04, which represents workpiece Wb2. The type of object area OB04 is a rectangular workpiece WSb.

[0048] Figure 5(F) shows another example of the status screen. Status screen D03 shows an example of the screen displayed in S240 (Figure 2) when the captured image IM03 (Figure 5(E)) is being processed. Status screen D03 shows the superimposed image IM03o and the status table TSa. The superimposed image IM03o is an image in which the contour lines of the detected object regions OB01-OB04 are superimposed on the captured image IM03. The status table TSa shows the number of detections Nd and the placement status STa updated from the placement status STa in Figure 5(D). In the example in Figure 5(F), two rectangular workpieces WSb are detected. Therefore, for identifier ID3 of "105", the number of detections Nd is set to 2 and the placement status STa is set to "OK". By observing status screen D03, the operator can easily recognize that all workpieces have been placed.

[0049] If it is determined that all workpieces have been detected (Figure 2: S245: Yes), the imposition process is executed in S250. In this embodiment, the imposition process positions the target image for each workpiece according to the position and orientation of each workpiece on the placement area AA, and generates printable image data representing each positioned target image. In this embodiment, the data processing device 200 performs the imposition process without using instructions from the operator.

[0050] Figure 6 is a diagram illustrating an example of panelization. The figure shows the captured image IM03 and the resulting image 250r. The processor 210 determines the target image to be associated with each object region OB01-OB04 according to the type of object region OB01-OB04 (here, work shape WS) detected from the captured image IM03 representing all workpieces, and the target batch information (here, batch information 233 (Figure 4)). The resulting image 250r represents the captured image IM03 with the panelized images superimposed. The processor 210 assigns the target image I1 to the object region OB01 associated with the elliptical shape WSa. Furthermore, the processor 210 assigns the target image I2 to the object region OB02 associated with the elliptical shape WSa, the target image I3 to the object region OB03 associated with the rectangular shape WSb, and the target image I3 to the object region OB04 associated with the rectangular shape WSb.

[0051] Furthermore, the processor 210 adjusts the position and orientation of the target image to match the position and orientation of the corresponding object region. The rotation angle in the captured image may be used as the orientation. As shown in the resulting image 250r, the position and rotation angle of the target images I1, I2, I3, and I3 are set to the position and rotation angle of the corresponding object region OB01-OB04.

[0052] Furthermore, various positions within the object region may be used as the position of the object region (for example, the position of the centroid of the object region). Also, there may be various methods for obtaining the rotation angle of the object region. For example, the processor 210 may determine the rotation angle of the object region by matching the object region with multiple template images of workpieces having different rotation angles.

[0053] The processor 210 generates print image data representing the imposition result. For example, the processor 210 generates print image IMP data representing the portion of the resulting image 250r corresponding to the placement area AA.

[0054] After the imposition process (Figure 2: S250), in S260, the processor 210 executes a process to proceed with the processing (in this case, printing). In this embodiment, the processor 210 displays a message on the display unit 240. Figure 5(G) shows an example of a message. In this embodiment, the processor 210 displays a status screen D10, which includes message MS10, on the display unit 240. Message MS10 prompts the operator to start printing.

[0055] In this embodiment, the processor 210 displays message MS10 on the status screen (for example, status screen D03 (Figure 5(F))) that is displayed when a "Yes" result is obtained in S245. Alternatively, the processor 210 may display message MS10 on the display unit 240 independently of the status screen (for example, the status screen may be cleared and then message MS10 may be displayed).

[0056] In S270 (Figure 2), the operator inputs a print start command to the print control device 490 of the first printing device 400 (Figure 1). The method of inputting the command may be any method. For example, the operator may operate an operation unit (not shown) of the print control device 490. Alternatively, the operator may input a print start command to the print control device 490 via a terminal device (not shown) connected to the local area network NT.

[0057] In response to the instruction to start printing, the print control device 490 starts printing. Specifically, the print control device 490 requests data for the print image IMP (Figure 6) from the data processing device 200. The processor 210 transmits the print image IMP data to the print control device 490. The processor 210 of the data processing device 200 may also voluntarily transmit the print image IMP data to the print control device 490 after generating the print image IMP data. The print control device 490 prints the target images I1, I2, I3, and I3 on ​​the workpieces Wa1, Wa2, Wb1, and Wb2 by controlling the head 412, the moving device 418, and the light source 413 according to the print image IMP data. Upon completion of printing, the print control device 490 moves the platen 416 to an outward position. As shown in Figure 1, this printing process creates workpieces Wa1x, Wa2x, Wb1x, and Wb2x from workpieces Wa1, Wa2, Wb1, and Wb2, each containing the printed target images I1, I2, I3, and I3.

[0058] In S280 (Figure 2), the processor 210 instructs the second printing device 700 to print label sheets for each of the multiple target identifiers (here, identifiers ID1-ID3 (Figure 4)). Figure 7(A) is a diagram showing an example of label sheets. The diagram shows label sheets LS1, LS2, and LS3 corresponding to identifiers ID1, ID2, and ID3. Label sheets LS1, LS2, and LS3 each represent a string representing identifiers ID1, ID2, and ID3, and two-dimensional codes CD1, CD2, and CD3 (for example, QR code®). Two-dimensional codes CD1, CD2, and CD3 may represent various information related to the corresponding identifiers ID1, ID2, and ID3. For example, two-dimensional codes CD1, CD2, and CD3 may represent URLs (Uniform Resource Locators) of a web page describing the next step in printing. The web page may be provided by a server (not shown) connected to the local area network NT.

[0059] The processor 210 generates label image data representing a label for each target identifier and transmits the label image data to the print control device 790 (Figure 1). The two-dimensional code may be an image of a predetermined code. Alternatively, batch information (for example, batch information 233 (Figure 4)) may include image data representing the two-dimensional code of each identifier ID. The processor 210 may then refer to the batch information to obtain the image of the two-dimensional code. The print control device 790 uses the label image data to print the label image onto a roll of paper. In this embodiment, the roll of paper has perforations arranged at equal intervals. The portion of the roll of paper on which the label image is printed is separated from the roll of paper at the perforations and used as a label sheet.

[0060] In S285 (Figure 2), multiple label sheets with multiple target identifiers are placed in different trays. Figures 7(B) to 7(D) show the trays. In this embodiment, the operator places label sheets LS1, LS2, and LS3 into trays TR1, TR2, and TR3, respectively, as shown in Figure 7(B). As will be described later, label sheets LS1, LS2, and LS3 are used to identify the workpieces placed in trays TR1, TR2, and TR3. Alternatively, a robot may place the label sheets into the trays instead of the operator.

[0061] After printing the label sheet (S280), the processor 210 obtains the latest captured image data from the captured image data provided by the camera 500 in S310 (Figure 3). Figures 8(A)-8(G) are diagrams showing examples of images processed by the partial manufacturing process. Figure 8(A) shows an example of a captured image. Captured image IM11 shows the state of the workpieces Wa1x, Wa2x, Wb1x, and Wb2x on the placement area AA before they are picked up.

[0062] In S315 (Figure 3), the processor 210 detects the workpiece from the captured image. Similar to S230 (Figure 2), the processor 210 uses a trained object detection model 232 to detect the workpiece from the captured image. In this embodiment, the object detection model 232 is trained to detect each of the multiple types of workpieces associated with multiple identifier IDs represented by batch information (e.g., batch information 233 (Figure 4)) as a different type of object. For example, the object detection model 232 detects three types of workpieces associated with three identifiers ID1, ID2, and ID3 (Figure 4) as different types of objects. The workpiece associated with the first identifier ID1 is an elliptical WSa workpiece having target image I1. The workpiece associated with the second identifier ID2 is an elliptical WSa workpiece having target image I2. The workpiece associated with the third identifier ID3 is a rectangular WSb workpiece having target image I3.

[0063] When the captured image IM11 in Figure 8(A) is processed, the processor 210 detects the workpieces Wa1x, Wa2x, Wb1x, and Wb2x from the captured image IM11. The object regions OB11, OB12, OB13, and OB14 in the figure show examples of the regions of the detected workpieces. Object regions OB11, OB12, OB13, and OB14 represent the workpieces Wa1x, Wa2x, Wb1x, and Wb2x, respectively.

[0064] In S320 (Figure 3), the processor 210 determines an identifier to be associated with the detected workpiece. In this embodiment, the object detection model 232 detects the object's region and object type. The processor 210 determines the identifier of the workpiece (i.e., the object region) to be an identifier pre-associated with the object type. For example, the type of object region OB14 (Figure 8(A)) is a rectangular WSb workpiece having the target image I3. The processor 210 determines the identifier of object region OB14 to be the third identifier ID3 (Figure 4).

[0065] In S325 (Figure 3), the processor 210 determines the position of each workpiece on the captured image and determines the superposition position near the workpiece's position. In S330, the processor 210 generates a superimposed image by superimposing a string representing an identifier onto the superposition position. Figure 8(B) shows an example of a status screen, which will be described later. Status screen D11 includes the superimposed image IM11o. The superimposed image IM11o is an example of a superimposed image generated using the captured image IM11 (Figure 8(A)) by the processing in S325 and S330. As shown, the string "103" representing the first identifier ID1 is superimposed on the superposition position P1o near the position P1 of the object region OB11. Similarly, the string "104" of the second identifier ID2 is superimposed at superimposed position P2o near position P2 of object region OB12, the string "105" of the third identifier ID3 is superimposed at superimposed position P3o near position P3 of object region OB13, and the string "105" of the third identifier ID3 is superimposed at superimposed position P4o near position P4 of object region OB14. In this embodiment, positions P1, P2, P3, and P4 represent the centroid positions of object regions OB11, OB12, OB13, and OB14, respectively.

[0066] The superposition position may be any of the positions near the workpiece. For example, the superposition position may be determined to a position that satisfies the condition that "the distance between the position of the object region representing the workpiece (e.g., the centroid position) and the superposition position is less than or equal to a predetermined distance threshold, and the superposition position does not overlap with any object region." The processor 210 may randomly select the superposition position from a plurality of candidate positions that satisfy this condition. The distance threshold may be determined experimentally in advance so that a user observing the superimposed image can easily recognize the correspondence between the identifier and the workpiece position. The smaller the distance threshold, the easier it is to distinguish between multiple correspondences between the identifier and the workpiece position.

[0067] In S335 (Figure 3), the processor 210 determines the pickup status for each of the multiple types of workpieces. The processor 210 determines the pickup status for each of the multiple target identifiers (here, identifiers ID1-ID3 (Figure 4)). In this embodiment, the pickup status is determined to be either "not picked up" or "picked up". If one or more workpieces corresponding to identifier IDs are detected, the processor 210 determines the pickup status to be "not picked up". If no workpieces corresponding to identifier IDs are detected, the processor 210 determines the pickup status to be "picked up".

[0068] In S340 (Figure 3), the processor 210 displays the pickup status on the display unit 240. Figure 8(B) shows an example of the displayed screen. The status screen D11 shows the superimposed image IM11o and the status table TSp. The superimposed image IM11o is an image in which the identifier is superimposed on the captured image IM11 acquired in S310. The status table TSp shows the correspondence between the identifier ID and the pickup status STp. The pickup status STp indicates either "OK" or "NG". "OK" indicates "Picked up", and "NG" indicates "Not picked up". In the example in Figure 8(B), the pickup status STp for all identifier IDs is "NG". By observing the status screen D11, the operator can easily recognize the identifiers of the workpieces that remain unpicked.

[0069] In this embodiment, the status table TSp further represents the workpiece shape WS, the number of detected workpieces Nd, the total number Nw, and the target image IMt. By observing the status screen D11, the operator can easily recognize the types of workpieces that have not been picked up.

[0070] In S340 (Figure 3), the processor 210 generates data for the status screen D11 and outputs the generated data to the display unit 240, thereby displaying the status screen D11 on the display unit 240.

[0071] In S345, the processor 210 determines whether or not one or more workpieces are detected from the captured image. In this embodiment, the processor 210 determines that one or more workpieces are detected from the captured image if one or more workpieces of one or more target identifiers (here, one or more identifiers from identifiers ID1-ID3 (Figure 4)) are detected from the captured image. If no workpieces of any of the target identifiers are detected from the captured image, the processor 210 determines that no workpieces are detected from the captured image.

[0072] If it is determined that one or more workpieces are detected (S345: Yes), the processor 210 repeats the process from S310 to S345 until no more workpieces are detected in the captured image. The processor 210 displays the superimposed image and the pickup status STp in near real time. During the repetition of the S310-S345 process, the workpiece pickup (processing S350 and S355 in this embodiment) proceeds.

[0073] In S350, a workpiece is picked up from the placement area AA of the platen 416 (Figure 1). In this embodiment, the operator manually picks up a workpiece (for example, any of workpieces Wa1x, Wa2x, Wb1x, or Wb2x) from the placement area AA. In S355, the picked-up workpiece is placed in a tray associated with the workpiece identifier ID. In this embodiment, the operator manually places the picked-up workpiece into the tray.

[0074] Figure 7(C) shows a single workpiece Wa2x placed in tray TR2. By observing the superimposed image IM11o on the status screen D11 (Figure 8(A)), the worker can easily recognize that the identifier of workpiece Wa2x is "104". The worker places workpiece Wa2x into tray TR2, which contains the label sheet LS2 that represents "104".

[0075] Figure 8(C) shows another example of a captured image. Captured image IM12 shows the state after workpiece Wa2x has been picked up from the workpieces Wa1x, Wa2x, Wb1x, and Wb2x in Figure 8(A). In S315 (Figure 3), the processor 210 detects the remaining workpieces Wa1x, Wb1x, and Wb2x. The processor 210 acquires object regions OB11, OB13, and OB14 corresponding to workpieces Wa1x, Wb1x, and Wb2x.

[0076] Figure 8(D) shows another example of the status screen. Status screen D12 shows an example of the screen displayed in S340 (Figure 3) when the captured image IM12 (Figure 8(C)) is processed. Status screen D12 shows the superimposed image IM12o and the status table TSp. The superimposed image IM12o is an image in which the string of identifiers is superimposed on the captured image IM12. In the example of Figure 8(D), work Wa2x has already been picked up. Therefore, the identifier (104) of work Wa2x is not superimposed on the superimposed image IM12o, and the identifiers of the remaining work Wa1x, Wb1x, and Wb2x are superimposed. The status table TSp shows the detection count Nd and pickup status STp updated from the status table TSp in Figure 8(B). For identifier ID2 of "104", the detection count Nd is set to 0, and the pickup status STp is set to "OK". By observing the status screen D12, the worker can easily recognize that the workpiece associated with the identifier "104" has already been picked up.

[0077] Figure 7(D) shows the state in Figure 7(C) with the workpieces Wa1x, Wb1x, and Wb2x placed in trays TR1 and TR3, respectively. By observing the status screen D12 (Figure 8(D)), the worker can easily recognize that the identifiers for workpieces Wa1x, Wb1x, and Wb2x are "103", "105", and "105", respectively. The worker places workpiece Wa1x into tray TR1, which contains label sheet LS1 labeled "103". The worker also places workpieces Wb1x and Wb2x into tray TR3, which contains label sheet LS3 labeled "105".

[0078] Figure 8(E) shows another example of a captured image. Captured image IM13 shows the state after the workpieces Wa1x, Wb1x, and Wb2x in Figure 8(C) have been picked up. That is, all workpieces have been picked up from the placement area AA. Therefore, in S315 (Figure 3), no workpieces are detected in captured image IM13.

[0079] Figure 8(F) shows another example of the status screen. Status screen D13 shows an example of the screen displayed in S340 (Figure 3) when the captured image IM13 (Figure 8(E)) is processed. Status screen D13 shows the superimposed image IM13o and the status table TSp. The superimposed image IM13o is an image in which the string of the identifier of the detected workpiece is superimposed on the captured image IM13 (since no workpiece is detected from the captured image IM13, the superimposed image IM13o is the same as the captured image IM13). The status table TSp shows the number of detections Nd and the pickup status STp updated from the status table TSp in Figure 8(D). In the example in Figure 8(F), for all identifiers ID, the number of detections Nd is zero and the pickup status STp is "OK". By observing status screen D13, the operator can easily recognize that all workpieces have been picked up.

[0080] If it is determined that no workpiece is detected from the captured image (Figure 3: S345: No), in S360, the processor 210 determines whether all batches have been completed. If there are unprocessed batches remaining (S360: No), the processor 210 proceeds to S210 (Figure 2) and processes new batches.

[0081] If all batches are completed (S360: Yes), in S365, the processor 210 executes the process to proceed to the next step. In this embodiment, the processor 210 displays a message on the display unit 240. Figure 8(G) shows an example of a message. In this embodiment, the processor 210 displays a status screen D14, which includes message MS14, on the display unit 240. Message MS14 prompts the operator to start the next step.

[0082] In this embodiment, the processor 210 displays message MS14 on the status screen (for example, status screen D13 (Figure 8(F))) that is displayed when the determination result of No is obtained in S345 (Figure 3). Alternatively, the processor 210 may display message MS14 on the display unit 240 independently of the status screen (for example, the status screen may be cleared and then message MS14 may be displayed).

[0083] In S370 (Figure 3), the next step after printing is performed. For example, a chain is attached to a workpiece that has a printed image. In this embodiment, the next step is performed by an operator. The content of the next step may vary depending on the type of workpiece (in this case, the identifier ID). For example, the color of the chain may vary depending on the identifier ID. The operator can easily recognize the identifier IDs of each workpiece Wa1x, Wa2x, Wb1x, Wb2x by observing the label sheets LS1, LS2, LS3 in trays TR1, TR2, TR3 (Figure 7(D)). Therefore, the possibility of an error occurring in the next step (for example, attaching the wrong chain) is reduced.

[0084] Furthermore, after S365 (Figure 3), processor 210 terminates the processing shown in Figures 2 and 3 without waiting for the completion of S370.

[0085] As described above, in this embodiment, the processor 210 (Figure 1) executes the following processes according to the program 231. In S310 (Figure 3), the processor 210 acquires an image of the placement area AA for machining multiple workpieces (for example, captured image IM11 (Figure 8(A)). The captured image is an image taken with multiple workpieces (here, workpieces Wa1x, Wa2x, Wb1x, Wb2x) placed in the placement area AA. Here, the printed image differs between workpiece Wa1x and workpiece Wa2x. The shape and the printed image differ between workpiece Wa1x and workpiece Wb1x. The images are different. These pairs of workpieces are examples of pairs of first and second workpieces that differ from each other in one or more of the following items: shape and processing content (in this case, the image to be printed). Thus, the multiple workpieces represented by the captured image include first and second workpieces that differ from each other in one or more of the following items: shape and processing content (in this case, the image to be printed). Such a captured image (e.g., captured image IM11) is an example of a specific image representing placement area AA, captured with multiple workpieces placed in placement area AA.

[0086] In S340 (Figure 3), the processor 210 outputs data for a status screen including an overlaid image to the display unit 240 (for example, status screen D11 including an overlaid image IM11o (Figure 8(B))). The overlaid image represents a workpiece and an identifier placed at an overlaid position near the workpiece's location. For example, the overlaid image IM11o (Figure 8(B)) represents identifier ID1 (103) placed at an overlaid position P1o near the location of workpiece Wa1x (for example, location P1). Such an overlaid image IM11o does not associate "103 (identifier ID1)" with the locations of other workpieces Wa2x, Wb1x, Wb2x (for example, locations P2, P3, P4), but associates "103 (identifier ID1)" with the location of workpiece Wa1x (for example, location P1). In the superimposed image IM11o, the portion WP1 representing workpiece Wa1x and "103 (identifier ID1)" is an example of information that associates the identifier of workpiece Wa1x with the position information of workpiece Wa1x. Note that the position of the workpiece in the superimposed image IM11o indicates the position of the workpiece on the placement area AA. The portion WP1 of the superimposed image IM11o associates the identifier of workpiece Wa1x with the position of workpiece Wa1x on the placement area AA.

[0087] The same applies to the other workpieces. The superimposed image IM11o represents "104 (identifier ID2)" in the vicinity of workpiece Wa2x, "105 (identifier ID3)" in the vicinity of workpiece Wb1x, and "105 (identifier ID3)" in the vicinity of workpiece Wb2x. The superimposed image IM11o associates "104 (identifier ID2)" with the position of workpiece Wa2x (e.g., position P2), and associates "105 (identifier ID3)" with the positions of workpieces Wb1x and Wb2x (e.g., positions P3 and P4). The portion WP2 of the superimposed image IM11o that represents workpiece Wa2x and "104 (identifier ID2)" is an example of information that associates the identifier of workpiece Wa2x with the position information of workpiece Wa2x. Similarly, part WP3, which represents work Wb1x and "105 (identifier ID3)", is an example of information that associates the identifier of work Wb1x with the location information of work Wb1x, and part WP4, which represents work Wb2x and "105 (identifier ID3)", is an example of information that associates the identifier of work Wb2x with the location information of work Wb2x.

[0088] An example of a picker, such as a worker (i.e., a person), refers to the superimposed image IM11o when picking up the processed workpieces Wa1x, Wa2x, Wb1x, and Wb2x on the placement area AA. By referring to the superimposed image IM11o, the worker can easily recognize the position and identifier of each workpiece Wa1x, Wa2x, Wb1x, and Wb2x. Therefore, the possibility of misselecting a workpiece is reduced. Such a superimposed image IM11o is an example of reference information referred to for picking. The process of displaying the superimposed image IM11o on the display unit 240 (Figure 3: S340) is an example of output processing that outputs reference information to the worker. In this embodiment, the processor 210 generates the superimposed image IM11o using the captured image IM11. In this way, the processor 210 uses a specific image (e.g., captured image IM11) to output reference information (e.g., superimposed image IM11o) to the picker.

[0089] Furthermore, in this embodiment, the person picking the processed workpieces Wa1x, Wa2x, Wb1x, and Wb2x (Figure 3: S350) is a human. By outputting reference information (e.g., superimposed image IM11o), the possibility of the picker (i.e., a human) misselecting a workpiece is reduced.

[0090] Furthermore, in this embodiment, the output process for reference information includes the process of displaying the reference information (e.g., superimposed image IM11o) on the display unit 240 (Figure 3: S340). The picker (in this case, a person) can easily refer to the reference information by observing the display unit 240. Therefore, the possibility of incorrect workpiece selection is reduced.

[0091] In this embodiment, the processor 210 performs output processing (Figure 3: S340) after processing multiple workpieces (for example, workpieces Wa1x, Wa2x, Wb1x, Wb2x (Figure 1)) (Figure 2: S270). When such a configuration is adopted, the processor 210 can output reference information that includes information associating workpiece identifiers with their positions. For example, in the superimposed image IM11o of Figure 8(B), portions WP1-WP4 associate the identifiers and positions of the processed workpieces Wa1x, Wa2x, Wb1x, and Wb2x. Such reference information can reduce the possibility of incorrect workpiece selection.

[0092] Furthermore, in this embodiment, after workpiece processing (Figure 2: S270), the processor 210 acquires images from the camera 500 that photographs the placement area AA (Figure 3: S310), detects multiple workpieces from the images (S315), and determines the identifiers and position information of the detected workpieces (S320, S325). Therefore, the processor 210 can output reference information that associates the identifiers and position information of the processed workpieces. Such reference information can reduce the possibility of incorrect selection of processed workpieces. When an operator places workpieces in the placement area AA in S220 (Figure 2), the accuracy of the workpiece positions on the placement area AA may be low. In this case as well, the processor 210 can output reference information that includes appropriate information that associates the identifiers and position information of the processed workpieces. Also, panelization (S250) may be performed by an operator. In this case as well, the processor 210 can output reference information that includes appropriate information that associates the identifiers and position information of the processed workpieces.

[0093] In this embodiment, the processor 210 executes a process PSa which includes repeating steps S310, S315, S335, and S340 in Figure 3. As explained in Figures 8(B), 8(D), and 8(F), this process PSa displays the pickup status STp of each of the multiple processed workpieces on the display unit 240 (S340). As explained in S335, the pickup status STp is selected from a plurality of statuses including not picked up (NG) and picked up (OK). Process PSa is an example of a first display process that displays the pickup status STp.

[0094] Process PSa includes a repetition of S310. Through the repetition of S310, the processor 210 repeatedly acquires captured images from the camera 500 that photographs the placement area AA (Figure 1) (for example, captured images IM11, IM12, IM13 (Figures 8(A), 8(C), 8(E))). In S335, the processor 210 determines the pickup status of each workpiece. The process in S335 includes determining the pickup status STp of workpieces detected from the captured images to not picked up (NG), and determining the pickup status of workpieces not detected from the captured images to picked up (OK).

[0095] With this configuration, the processor 210 can appropriately determine the pickup status of each of the multiple processed workpieces. The processor 210 also displays the pickup status of each workpiece on the display unit 240. The processor 210 can notify the picker (in this case, a person) of the pickup status of each workpiece.

[0096] Furthermore, in this embodiment, the workpiece processing (Figure 2: S270) is image printing. As explained in Figure 4, the processing content (in this case, the image to be printed) differs among multiple workpieces. For example, as shown in Figures 8(A) and 8(B), the image to be printed differs between workpiece Wa1x, which is associated with the first identifier ID1, and workpiece Wa2x, which is associated with the second identifier ID2. That is, the image to be printed differs between the unprocessed workpiece Wa1 and the unprocessed workpiece Wa2. In S220 of Figure 2, multiple unprocessed workpieces (in this case, workpieces Wa1, Wa2, Wb1, Wb2 (Figures 1 and 5(E))) are placed in the placement area AA. After S220, in S250, the image to be printed on each workpiece is determined. For example, as explained in Figure 6, the image to be printed on workpiece Wa1 is determined to be the target image I1 without user specification. The image to be printed on workpiece Wa2 is determined to be a different target image I2 than the target image I1, without user specification. When this configuration is adopted, the processor 210 can output appropriate reference information (e.g., superimposed image IM11o (Figure 8(B))) to the picker. Furthermore, since the correspondence between the workpiece and the target image is determined without user specification, the burden on the operator is reduced. In addition, the actual image on each workpiece can be easily confirmed after printing. As described above, the processor 210 performs output processing (Figure 3: S340) after printing (Figure 2: S270). Therefore, the processor 210 can output appropriate reference information (e.g., superimposed image IM11o) that includes information associating the identifier and position of the processed workpiece.

[0097] Note that the processing in S250 may be performed by another device (for example, the print control device 490 of the first printing device 400) instead of the data processing device 200.

[0098] In this embodiment, the processor 210 instructs the second printing device 700 to print multiple sheets LS1, LS2, and LS3 (Figure 7(A)) representing multiple target identifiers ID1, ID2, and ID3 in S280 (Figure 2). This printing (S280) is performed before the pickup of multiple processed workpieces (Figure 3: S350). As explained in Figure 7(D), the sheets LS1, LS2, and LS3 can be used to distinguish between multiple workpieces (for example, workpiece Wa1x and workpiece Wa2x).

[0099] In this embodiment, the processor 210 executes a process PSb that includes repeating S225, S230, S235, and S240 before processing multiple workpieces (Figure 2: S270). As explained in Figures 5(B), 5(D), and 5(F), this process PSb displays the placement status STa of each of the multiple unprocessed workpieces on the display unit 240 (S240). As explained in S235, the placement status STa is selected from a plurality of statuses, including not placed (NG) and placed (OK). Process PSb is an example of a second display process that displays the placement status STa.

[0100] Process PSb includes a repetition of S225. Through the repetition of S225, the processor 210 repeatedly acquires captured images from the camera 500 that photographs the placement area AA (Figure 1) (for example, captured images IM01, IM02, IM03 (Figures 5(A), 5(C), 5(E))). In S235, the processor 210 determines the placement status of each workpiece. The process in S235 includes determining the placement status STa of workpieces not detected from the captured images to not placed (NG), and determining the placement status STa of workpieces detected from the captured images to placed (OK).

[0101] With this configuration, the processor 210 can appropriately determine the placement status of each of the multiple unprocessed workpieces. The processor 210 also displays the placement status of each workpiece on the display unit 240. The processor 210 can notify the picker (in this case, a person) of the placement status of each workpiece.

[0102] Furthermore, in this embodiment, the processor 210 repeatedly executes S225 before processing multiple workpieces (Figure 2: S270). This causes the processor 210 to repeatedly acquire images from the camera 500 that photographs the placement area AA (Figure 1) (for example, captured images IM01, IM02, IM03 (Figures 5(A), 5(C), 5(E))). If it is determined in S245 that multiple unprocessed workpieces are detected from the captured images (S245: Yes), then in Cc, the processor 210 executes processing to proceed with processing in S260. This reduces the possibility that the processor 210 may erroneously proceed with processing multiple workpieces before they are placed in the placement area AA. Note that in the above case, Cc is an example of a third specific case in which multiple unprocessed workpieces are detected from the captured images and processing to proceed with processing is executed.

[0103] In this embodiment, the processor 210 executes a process PSc (Figure 3) including S315, S320, S325, and S340. In S315, the processor 210 detects each of several workpieces from a specific image. For example, object regions OB11, OB12, OB13, and OB14 representing workpieces Wa1x, Wa2x, Wb1x, and Wb2x are detected from the captured image IM11 (Figure 8(A)) (the captured image IM11 is an example of a specific image). In S320, the processor 210 uses the detection results of each of the several workpieces to determine an identifier associated with the detected workpiece. In S325, the processor 210 uses the detection results of each of the several workpieces to determine positional information associated with the detected workpiece. In this embodiment, the processor 210 determines the positions P1, P2, P3, and P4 of the object regions OB11, OB12, OB13, and OB14. In S340, the processor 210 outputs an overlay image IM11o. The overlay image IM11o is an example of reference information that includes information associating the workpiece identifier with the workpiece location information. Thus, the processor 210 can output reference information that appropriately associates the workpiece identifier with the workpiece location information. Processing PSc is an example of output processing that outputs reference information to the operator.

[0104] The superposition position may be determined based on various positions representing the workpiece. For example, the superposition position may be determined to a position that satisfies the condition that "the shortest distance between a position included in the object region representing the workpiece and the superposition position is less than or equal to a predetermined distance threshold, and the superposition position does not overlap with any object region." In this way, the processor 210 may determine multiple positions included in the object region representing the workpiece as the workpiece position.

[0105] Furthermore, the workpiece shape is not limited to simple shapes such as ellipses, circles, and polygons (rectangles, triangles, hexagons, etc.), but can be of various shapes. For example, if the image to be printed represents a character such as a person or animal, the workpiece may have a shape that is roughly the same as the outline of the character. Also, the material of the workpiece can be of various materials. For example, the material of the workpiece may be various acrylic sheets, such as opaque white acrylic sheets, transparent blue acrylic sheets, transparent pink acrylic sheets, or acrylic sheets that represent a pattern of changing colors like the aurora. Also, the material of the workpiece is not limited to acrylic sheets, but may be various materials such as metal sheets (e.g., aluminum sheets). In any case, it is preferable that the processing in S230 (Figure 2) and the processing in S315 (Figure 3) are configured to detect multiple types of workpieces that differ from each other in one or more items among material, shape, and processing content (in this case, the image to be printed), as different types of workpieces. For example, the object detection model 232 may be trained to detect such multiple types of workpieces as different types of workpieces. This allows the processor 210 to use the work detection results to determine the identifier and location information associated with the detected work.

[0106] B. Second example: B1. System Configuration: Figure 9 is a block diagram representing another embodiment of the manufacturing system. There are two hardware differences between manufacturing system 1000b and manufacturing system 1000 in Figure 1. The first difference is that the second printing device 700 (Figure 1) is omitted. The second difference is that robot 300 (Figure 9) is added. In this embodiment, the placement of workpieces in the placement area AA and the pickup of workpieces from the placement area AA are performed by robot 300. The processing of workpieces using manufacturing system 1000b is the same as the processing of workpieces using manufacturing system 1000 (Figure 1). Program 231b is configured to carry out the partial manufacturing process described later. Program 231c is used in another embodiment described later.

[0107] The robot 300 includes a robot arm 310 and a robot control unit 390 that controls the robot arm 310. The robot arm 310 picks up and places workpieces. The robot arm 310 has an end effector 314 for picking up workpieces. The configuration of the end effector 314 may be any configuration that is capable of picking up workpieces. For example, the end effector 314 may have multiple fingers for gripping the workpiece. Alternatively, the end effector 314 may have a suction part (e.g., a suction cup) for picking up the workpiece. The robot arm 310 further includes one or more joints for moving the end effector 314 and a plurality of arms connected by one or more joints.

[0108] The robot control unit 390 is a computer similar to the data processing unit 200. Although not shown in the diagram, the robot control unit 390 includes a processor, volatile memory, non-volatile memory, and a communication interface. The robot control unit 390 is connected to the local area network NT. The robot control unit 390 controls the robot arm 310 according to instructions received through the local area network NT.

[0109] B2. Partial manufacturing process: Figures 10 and 11 are flowcharts illustrating an example of a partial manufacturing process. Figure 11 shows the process that follows Figure 10. The partial manufacturing process in this embodiment is a modification of the partial manufacturing processes in Figures 2 and 3. In Figures 10 and 11, steps identical to those in Figures 2 and 3 are denoted by the same reference numerals, and their explanations are omitted.

[0110] Hereafter, batch information 233 (Figure 4) is assumed to be the batch information to be processed. S210, S215, S225, S230, and S245 are the same as S210, S215, S225, S230, and S245 in Figure 2, respectively. In this embodiment, S220 in Figure 2 is replaced with S220b in Figure 10. Also, S235 and S240 in Figure 2 are omitted.

[0111] The process in S220b is the same as the process in S220 in Figure 2, except that the robot 300 (Figure 9) moves the workpieces instead of a person. The robot control unit 390 controls the robot arm 310 to place the workpieces Wa1, Wa2, Wb1, and Wb2 in the box BX onto the placement area AA. The workpieces Wa1, Wa2, Wb1, and Wb2 may be placed in predetermined positions within the box BX. The robot control unit 390 may then place the workpieces placed in predetermined positions within the box BX onto predetermined positions on the placement area AA. In S220b, the processor 210 of the data processing device 200 may send a command to the robot control unit 390 to start moving the workpieces. The robot control unit 390 may start moving the workpieces in response to the command.

[0112] If it is determined in S245 that one or more workpieces are not detected (Figure 10: S245: No), the processor 210 repeats the processing of S225, S230, and S245 until all workpieces are detected from the captured image. Similar to the examples in Figures 5(A)-5(F), the workpiece placement (S220b) proceeds during the repetition of the processing of S225, S230, and S245.

[0113] If it is determined that all workpieces have been detected (Figure 10: S245: Yes), the imposition process is executed in S250. In this embodiment, as in the first embodiment, the processor 210 of the data processing device 200 executes the imposition process without using instructions from the operator. For example, the imposition process is performed in the same manner as in the embodiment of Figure 6.

[0114] In S253b (Figure 10), the processor 210 determines the identifier to be associated with the workpiece. In this embodiment, the processor 210 determines the identifier of the object region representing the workpiece according to the result of the panelization process in S250 (Figure 6). The processor 210 refers to the target batch information (here, batch information 233 (Figure 4)) and adopts an identifier ID that is associated with the combination of the workpiece type (here, workpiece shape WS) and the target image assigned to the object region representing the workpiece. For example, identifier ID1 is associated with object region OB01, identifier ID2 is associated with object region OB02, and identifier ID3 is associated with object regions OB03 and OB04.

[0115] In S255b, the processor 210 determines the coordinates associated with the workpiece. These coordinates are used by the robot 300 for workpiece pickup. The coordinates may represent various positions suitable for workpiece pickup. In this embodiment, the processor 210 determines the coordinates of the workpiece (i.e., the object region) in a two-dimensional coordinate system on the captured image that indicates the centroid position of the object region representing the workpiece. Such coordinates indicate the position of the workpiece on the captured image. For example, when the captured image IM03 in Figure 6 is processed, the processor 210 adopts the centroid positions Pw1-Pw4 of the object region OB01-OB04 representing the workpieces Wa1, Wa2, Wb1, Wb2. The coordinates are represented, for example, by the pixel position in a first direction Dx and the pixel position in a second direction Dy in the captured image.

[0116] In S257b, the processor 210 outputs identifier coordinate data DR to the robot 300 (Figure 9), which represents multiple correspondences between identifiers and coordinates. The identifier coordinate data DR represents the correspondence between identifiers and coordinates of multiple object regions detected from the captured image (here, object regions OB01-OB04 (Figure 6)). The robot control unit 390 of the robot 300 stores the received identifier coordinate data DR in a storage device (not shown) of the robot control unit 390 (for example, a non-volatile storage device).

[0117] In S260b, the processor 210 executes the process necessary to carry out the processing (in this case, printing). In this embodiment, the processor 210 outputs a print start instruction to the first printing device 400 (Figure 9).

[0118] In S270, the print control device 490 executes printing in response to the print start instruction. The process in S270 is the same as the process in S270 in Figure 2. For example, as shown in Figure 9, workpieces Wa1x, Wa2x, Wb1x, and Wb2x, each containing the printed target images I1, I2, I3, and I3, are formed from workpieces Wa1, Wa2, Wb1, and Wb2.

[0119] In S305b (Figure 11), the print control device 490 sends a notification of print completion to the data processing device 200. The processor 210 of the data processing device 200 sends a pickup start instruction to the robot 300 in response to the print completion notification.

[0120] After S305b, the processor 210 executes S310, S315, and S345. S310, S315, and S345 are the same as S310, S315, and S345 in Figure 3, respectively. The processor 210 acquires the data of the latest captured image (S310), detects the workpiece from the captured image (S315), and determines whether or not one or more workpieces are detected from the captured image (S345).

[0121] If it is determined that one or more workpieces are detected (S345: Yes), the processor 210 repeats the processes of S310, S315, and S345 until no more workpieces are detected in the captured image. During the repetition of the processes of S310, S315, and S345, the workpiece pickup process (in this embodiment, the processes of S350b and S355b) proceeds.

[0122] In S350b, a workpiece (for example, one of workpieces Wa1x, Wa2x, Wb1x, or Wb2x) is picked up from the placement area AA of the platen 416 (Figure 9). In S355b, the picked-up workpiece is placed in a tray associated with the workpiece identifier ID. In this embodiment, the robot control unit 390 of the robot 300 controls the robot arm 310 to pick up the workpiece on the placement area AA and place the picked-up workpiece in the tray.

[0123] The robot control unit 390 converts the two-dimensional coordinates represented by the identifier coordinate data DR acquired in S257b (Figure 10) into three-dimensional coordinates representing the position of the platen 416 on the placement area AA, according to a predetermined coordinate correspondence. The robot control unit 390 moves the end effector 314 to the position in the three-dimensional coordinates by controlling the robot arm 310 and picks up the workpiece. Note that the two-dimensional coordinates may contain errors. The three-dimensional coordinates calculated from the two-dimensional coordinates may contain errors. The robot arm 310 may include a sensor that detects the reaction force caused by contact between the end effector 314 and other members (e.g., the workpiece). The robot control unit 390 may move the end effector 314 from above to below the position indicated by the three-dimensional coordinates until a reaction force is detected. Then, in response to the detection of the reaction force, the robot control unit 390 may have the end effector 314 pick up the workpiece.

[0124] Furthermore, in this embodiment, the correspondence between identifiers and trays is predetermined. For example, trays TR1, TR2, and TR3 (Figure 9) are associated with identifiers ID1, ID2, and ID3, respectively. The robot control unit 390 controls the robot arm 310 to move the end effector 314 that holds the workpiece to the position of the tray associated with the workpiece identifier and release the workpiece (the workpiece identifier is an identifier associated with coordinates by identifier coordinate data DR). As a result, the workpiece is placed in the tray associated with the workpiece identifier.

[0125] The robot control unit 390 picks up (S350b) and releases (S355b) the workpieces according to the correspondence represented by the identifier coordinate data DR. As a result, as shown in Figure 9, each of the workpieces Wa1x, Wa2x, Wb1x, and Wb2x is placed in its corresponding tray (in this case, one of trays TR1, TR2, or TR3).

[0126] After all workpieces Wa1x, Wa2x, Wb1x, and Wb2x have been picked up, in S345 it is determined that no workpieces are detected from the captured image (S345: No). In S360, the processor 210 determines whether all batches have been completed. If there are unprocessed batches remaining (S360: No), the processor 210 proceeds to S210 (Figure 10) to process new batches.

[0127] If all batches are completed (S360: Yes), in S365b, the processor 210 executes the process to proceed to the next step. If the next step is performed by a robot, the processor 210 may notify the robot of permission to start the next step. If the next step is performed by a human operator, the processor 210 may display a message on the display unit 240. The displayed message may prompt the operator to start the next step, as shown in message MS14 in Figure 8(G).

[0128] In S370b, the next printing step is performed. The content of the next step is the same as that of S370 in Figure 3 (for example, a chain is attached to the workpiece).

[0129] After S365b, processor 210 terminates the processing shown in Figures 10 and 11 without waiting for the completion of S370b.

[0130] As described above, in this embodiment, the processor 210 (Figure 9) executes the following processes according to program 231b. In S225 (Figure 10), the processor 210 acquires an image of the placement area AA for processing multiple workpieces (for example, image IM03 (Figure 5(E), Figure 6)). The image is an image taken with multiple workpieces (here, workpieces Wa1, Wa2, Wb1, Wb2) placed in the placement area AA. Here, the printed image differs between workpiece Wa1 and workpiece Wa2. The shape and the printed image differ between workpiece Wa1 and workpiece Wb1. The shape and the printed image differ between workpiece Wa2 and workpiece Wb1. These pairs of workpieces are examples of pairs of first and second workpieces in which one or more items among the shape and processing content (here, the printed image) differ from each other. Thus, the multiple workpieces represented by the captured image include a first workpiece and a second workpiece, each differing from the others in one or more of the following aspects: shape and processing content (in this case, the image to be printed). Such a captured image (for example, captured image IM03) is an example of a specific image representing the placement area AA, captured with multiple workpieces placed in the placement area AA.

[0131] In S257b (Figure 10), the processor 210 outputs identifier coordinate data DR to the robot 300 (Figure 9). The identifier coordinate data DR represents the correspondence between the positional information of each of the multiple object regions detected from the captured image (for example, object regions OB01-OB04 (Figure 6)) and their identifiers. For example, the identifier coordinate data DR represents the following correspondence: (1) Correspondence between position Pw1 and identifier ID1 of object region OB01 representing work Wa1 (2) Correspondence between the position Pw2 of the object region OB02 representing work Wa2 and the identifier ID2 (3) Correspondence between the position Pw3 of the object region OB03 representing work Wb1 and the identifier ID3 (4) Correspondence between the position Pw4 of the object region OB04 representing work Wb2 and the identifier ID3

[0132] As explained in S350b (Figure 11), the robot 300, which is an example of a picker, refers to the identifier coordinate data DR when picking up the processed workpieces Wa1x, Wa2x, Wb1x, and Wb2x on the placement area AA. By referring to the identifier coordinate data DR, the robot 300 can easily obtain the position and identifier of each workpiece Wa1x, Wa2x, Wb1x, and Wb2x. Therefore, the possibility of misselecting a workpiece is reduced. Such identifier coordinate data DR is an example of reference information referred for picking. In this embodiment, the processor 210 generates the identifier coordinate data DR using the captured image IM03. In this way, the processor 210 uses a specific image (e.g., captured image IM03) to output reference information (e.g., identifier coordinate data DR) to the picker.

[0133] Furthermore, in this embodiment, the picker of the processed workpieces Wa1x, Wa2x, Wb1x, and Wb2x (Figure 11: S350b) is the robot 300. By outputting reference information (specifically, identifier coordinate data DR), the possibility of the picker (i.e., the robot 300) misselecting a workpiece is reduced.

[0134] Furthermore, in this embodiment, as explained in S255b and S257b (Figure 10), the identifier coordinate data DR represents the coordinates indicating the position of the workpiece as workpiece position information. For example, the identifier coordinate data DR represents the following coordinates. (1) Coordinates indicating the position Pw1 of the object region OB01 representing the workpiece Wa1 (2) Coordinates indicating the position Pw2 of the object region OB02 representing the workpiece Wa2 (3) Coordinates indicating the position Pw3 of the object region OB03 representing the workpiece Wb1 (4) Coordinates indicating the position Pw4 of the object region OB04 representing the workpiece Wb2 Since the robot 300 references identifier coordinate data DR representing these coordinates, the possibility of incorrect workpiece selection is reduced.

[0135] Furthermore, in this embodiment, the processor 210 performs output processing (S257b) before processing (Figure 10: S270) multiple workpieces (for example, workpieces Wa1, Wa2, Wb1, Wb2 (Figure 9)). Also, in this embodiment, the position of the workpieces on the placement area AA is the same before and after processing. The robot 300 can pick up the appropriate workpiece by referring to the position of the unprocessed workpieces without referring to the image printed on the workpieces. In this way, when the processor 210 outputs reference information (specifically, identifier coordinate data DR) representing the position of the unprocessed workpieces, the possibility of the robot 300 misselecting a workpiece is reduced.

[0136] In this embodiment, multiple workpieces (for example, workpieces Wa1, Wa2, Wb1, Wb2 (Figure 9)) are subject to multiple processes, including machining (Figure 10: S270) and the next step after machining (Figure 11: S370b). After machining the multiple workpieces (S270), the processor 210 repeats steps S310, S315, and S345 in Figure 11. By repeating S310, the processor 210 repeatedly acquires images from the camera 500 that photographs the placement area AA (for example, captured images IM11, IM12, IM13 (Figure 8(A), Figure 8(C), Figure 8(E))). If it is determined in S345 that no machined workpieces are detected from the captured images (S345: No), and all batches have finished (S360: Yes), then in Ca2, the processor 210 executes a process in S365b to proceed to the next step. This reduces the possibility that the next process may erroneously proceed before the pickup of the processed workpiece is complete. In the above case, Ca2 is an example of a first specific case in which the processed workpiece is not detected from the captured image and processing to proceed to the next process is performed.

[0137] Furthermore, in this embodiment, the workpiece processing (Figure 10: S270) is image printing. As explained in Figure 4, the processing content (in this case, the image to be printed) differs among multiple workpieces. For example, as shown in Figures 8(A) and 8(B), the image to be printed differs between workpiece Wa1x, which is associated with the first identifier ID1, and workpiece Wa2x, which is associated with the second identifier ID2. That is, the image to be printed differs between the unprocessed workpiece Wa1 and the unprocessed workpiece Wa2. In S220b of Figure 10, multiple unprocessed workpieces (in this case, workpieces Wa1, Wa2, Wb1, Wb2 (Figure 9)) are placed on the placement area AA. After S220b, in S250, the image to be printed on each workpiece is determined. For example, as shown in Figure 6, the image to be printed on workpiece Wa1 is determined to be the target image I1 without user specification. The image to be printed on workpiece Wa2 is automatically determined to be a different image I2 than the target image I1, without user specification. When this configuration is adopted, the processor 210 can output appropriate reference information (e.g., identifier coordinate data DR) to the picker. Furthermore, since the correspondence between the workpiece and the target image is determined without user specification, the burden on the operator is reduced.

[0138] Furthermore, in this embodiment, as described in S220b, multiple unprocessed workpieces (for example, workpieces Wa1, Wa2, Wb1, Wb2 (Figure 9)) are placed in the placement area AA by the robot 300. When such a configuration is adopted, the processor 210 can output appropriate reference information (for example, identifier coordinate data DR) to the picker. In addition, the burden on the operator is reduced.

[0139] In this embodiment, the processor 210 repeatedly executes S225 before processing multiple workpieces (Figure 10: S270). This causes the processor 210 to repeatedly acquire images from the camera 500 that photographs the placement area AA (for example, captured images IM01, IM02, IM03 (Figures 5(A), 5(C), 5(E))). If it is determined in S245 that multiple unprocessed workpieces have been detected from the captured images (S245: Yes), then in Cb, the processor 210 executes a process to proceed with processing in S260b. This reduces the possibility that the processor 210 may erroneously proceed with processing multiple workpieces before they are placed in the placement area AA. In the above case, Cb is an example of a second specific case in which multiple workpieces are detected from the captured images and a process to proceed with processing is executed when the workpiece picker is a robot. Furthermore, the process to proceed with processing may be executed regardless of the type of picker. In the above case, Cb is an example of a third specific case in which, regardless of the type of picker, multiple workpieces are detected from the captured image, and processing is performed to proceed with the machining.

[0140] In this embodiment, the processor 210 executes a process PSd (Figure 10) which includes S230, S253b, S255b, and S257b. In S230, the processor 210 detects each of several workpieces from a specific image. For example, object regions OB01, OB02, OB03, and OB04 representing workpieces Wa1, Wa2, Wb1, and Wb2 are detected from the captured image IM03 (Figures 5(E) and 6) (the captured image IM03 is an example of a specific image). In S253b, the processor 210 uses the detection results of each of the several workpieces to determine an identifier associated with the detected workpiece. In S255b, the processor 210 uses the detection results of each of the several workpieces to determine positional information associated with the detected workpiece. In this embodiment, the processor 210 determines the positions Pw1-Pw4 of object regions OB01-OB04. In S257b, the processor 210 outputs identifier coordinate data DR. The identifier coordinate data DR is an example of reference information that includes information linking the workpiece identifier to the workpiece's position information. Based on this, the processor 210 can output reference information that appropriately links the workpiece identifier to the workpiece's position information. The processing PSd is an example of output processing that outputs the reference information to the robot 300.

[0141] C. Third embodiment: Figure 12 is a flowchart illustrating an example of a partial manufacturing process. In this embodiment, the manufacturing system 1000b shown in Figure 9 is used. The process in Figure 12 is executed in place of the process in Figure 10. The process following Figure 12 is the process shown in Figure 11. The processor 210 proceeds with the partial manufacturing processes in Figures 12 and 11 according to program 231c, instead of program 231b.

[0142] The only difference between the process in Figure 10 and the process in Figure 12 is that steps S253b-S257b (Figure 10) are omitted, and instead, steps S275c-S295c (Figure 12) are added. The other parts of the process in Figure 12 are the same as the corresponding parts in Figure 10. In Figure 12, steps that are the same as those in Figure 10 are denoted by the same reference numerals, and their explanations are omitted.

[0143] After imposition (S250), in S260b, the processor 210 performs processing to proceed with the manufacturing (in this case, printing). In S270, in response to the instruction to start printing, the print control device 490 performs printing.

[0144] S275c, S280c, and S285c following S270 are the same as S310, S315, and S320 in Figure 3, respectively. Similar to the examples in Figures 8(A) and 8(B), the processor 210 determines an identifier to be associated with the workpiece detected from the captured image.

[0145] In S290c, the processor 210 determines the coordinates associated with the workpiece. Similar to the processing in S255b (Figure 10), the processor 210 determines the coordinates of the workpiece (i.e., the object region) in two-dimensional coordinates on the captured image that indicate the centroid position of the object region representing the workpiece. For example, when the captured image IM11 in Figures 8(A) and 8(B) is processed, the processor 210 adopts the centroid positions P1-P4 of the object regions OB11-OB14 representing the workpieces Wa1x, Wa2x, Wb1x, and Wb2x.

[0146] In S295c (Figure 12), the processor 210 outputs identifier coordinate data DRc to the robot 300 (Figure 9), which represents multiple correspondences between identifiers and coordinates. The identifier coordinate data DRc represents the correspondence between identifiers and coordinates for each of the multiple object regions detected from the captured image (here, object regions OB11-OB14 (Figure 8(A), Figure 8(B))). The robot control unit 390 of the robot 300 stores the received identifier coordinate data DRc in a storage device (not shown) of the robot control unit 390 (for example, a non-volatile storage device).

[0147] After S295c, the process shown in Figure 11 is executed. In this embodiment, identifier coordinate data DRc is used instead of identifier coordinate data DR. Identifier coordinate data DRc is an example of reference information that is referenced for pickup, similar to identifier coordinate data DR. The partial manufacturing process is completed when the process shown in Figure 11 is finished.

[0148] As described above, in this embodiment, the processor 210 outputs reference information (in this case, identifier coordinate data DRc) to the robot 300 after processing multiple workpieces (Figure 12: S270) (S295c). When such a configuration is adopted, the processor 210 can output reference information that associates the identifier and position of the workpiece. Such reference information can reduce the possibility of incorrect workpiece selection.

[0149] In this embodiment, after workpiece processing (Figure 12: S270), the processor 210 acquires images from the camera 500 that photographs the placement area AA (S275c), detects multiple workpieces from the images (S280c), and determines the identifiers and position information of the detected workpieces (S285c, S290c). Therefore, the processor 210 can output reference information (here, identifier coordinate data DRc) that associates the identifiers and position information of the processed workpieces. Such reference information can reduce the possibility of misselecting processed workpieces. Also, if an operator places the workpieces in the placement area AA instead of the robot 300 in S220b, the accuracy of the workpiece positions in the placement area AA may be low. In this case as well, the processor 210 can output reference information that includes appropriate information that associates the identifiers and position information of the processed workpieces. Furthermore, the panelization (S250) may be performed by an operator. In this case as well, the processor 210 can output reference information that includes appropriate information to associate the identifier and location information of the processed workpiece.

[0150] In this embodiment, the processor 210 executes a process PSe (Figure 12) which includes S280c, S285c, S290c, and S295c. In S280c, the processor 210 detects each of several workpieces from a specific image. For example, from the captured image IM11 (Figure 8(A)), object regions OB11, OB12, OB13, and OB14 representing workpieces Wa1x, Wa2x, Wb1x, and Wb2x are detected (the captured image IM11 is an example of a specific image). In S285c, the processor 210 uses the detection results of each of the several workpieces to determine an identifier associated with the detected workpiece. In S290c, the processor 210 uses the detection results of each of the several workpieces to determine positional information associated with the detected workpiece. In this embodiment, the processor 210 determines the positions P1-P4 of object regions OB11-OB14. In S295c, the processor 210 outputs identifier coordinate data DRc. The identifier coordinate data DRc is an example of reference information that includes information linking the workpiece identifier to the workpiece's position information. Based on this, the processor 210 can output reference information that appropriately links the workpiece identifier to the workpiece's position information. The process PSe is an example of output processing that outputs the reference information to the robot 300.

[0151] The only difference between the partial manufacturing process of this embodiment (Figures 12 and 11) and the partial manufacturing process of the second embodiment (Figures 10 and 12) is that steps S253b-S257b (Figure 10) are omitted, and instead, steps S275c-S295c (Figure 12) are added. Therefore, this embodiment can provide the same various advantages as those provided by the second embodiment.

[0152] C. Fourth Example: Figures 13(A) and 13(B) illustrate another embodiment of the reference information. These figures show status screens D11d and D11e, respectively, which are displayed instead of status screen D11 (Figure 8(B)). The only difference from status screen D11 in Figure 8(B) is that the superimposed image IM11o has been replaced by superimposed images IM11d and IM11e, respectively.

[0153] The superimposed image IM11d in Figure 13(A) represents the identifier-bearing arrows L1-L3. In S330 (Figure 3), the processor 210 superimposes a string representing the identifier at the superimposed position, and further superimposes arrows extending from the identifier to the object region associated with the identifier. If the same identifier is common to multiple object regions, multiple arrows associating one identifier with multiple object regions may be superimposed (for example, the identifier-bearing arrow L3). By observing the superimposed image IM11d, the operator can easily recognize the correspondence between the identifier and the workpiece position.

[0154] The superimposed image IM11e in Figure 13(B) represents the identifier-attached frames FD1-FD3. In S330 (Figure 3), the processor 210 superimposes a string representing the identifier at the superimposed position. Furthermore, the processor 210 superimposes a frame that overlaps with the identifier and encloses the object region associated with the identifier. If the same identifier is common to multiple object regions, a single frame enclosing the multiple object regions may be superimposed (for example, identifier-attached frame FD3). By observing the superimposed image IM11e, the operator can easily recognize the correspondence between the identifier and the workpiece position.

[0155] Thus, the information that associates a work identifier with location information may include various types of information, such as arrows that associate identifiers with work objects, and borders that represent groups of identifiers.

[0156] D. Variations: (1) In S250 (Figures 2, 10, and 12), the processor 210 may arrange multiple target images for multiple workpieces using user specifications. For example, the processor 210 may allow the user to specify the position and orientation of each target image on a captured image representing multiple workpieces (e.g., captured image IM03 (Figure 6)). The operator places target images I1, I2, I3, and I3 on ​​workpieces Wa1, Wa2, Wb1, and Wb2 by operating the operation unit 250. In any case, the processing of S250 may be executed according to a program other than programs 231, 231b, and 231c. Furthermore, the processing of S250 may be executed by a device other than the data processing device 200 (e.g., the print control device 490 of the first printing device 400).

[0157] (2) The object detection model 232 (Figure 2) may be any model capable of detecting a workpiece from a captured image, instead of RTMDet (e.g., YOLO (You only look once), Mask R-CNN). Also, the method for detecting a workpiece from a captured image may be any other method, such as template matching using a template image of the workpiece, instead of using a machine learning model. In any case, workpiece detection (e.g., S230 (Figures 2, 10, 12), S315 (Figures 3, 11), S280c (Figure 12)) may be performed according to a program other than programs 231, 231b, 231c. Also, workpiece detection may be performed by a device other than the data processing device 200 (e.g., the print control device 490 of the first printing device 400).

[0158] (3) The placement status (S235 (Figure 2)) may be selected from a plurality of statuses, including not placed and placed, as well as other statuses. The plurality of statuses may include, for example, "Placement in progress (i.e., workpiece being moved)". The processor 210 may also notify a person (e.g., an operator) of the placement status by various other methods, instead of using the display unit 240. For example, the processor 210 may use an image projector (not shown) to project an image representing the placement status onto the placement area AA.

[0159] (4) The pickup status (S335 (Figure 3)) may be selected from a plurality of statuses, including not picked up and picked up, as well as other statuses. The plurality of statuses may include, for example, "in the process of pickup (i.e., moving the workpiece)". The processor 210 may also notify a person (e.g., an operator) of the pickup status by various other methods, instead of using the display unit 240. For example, the processor 210 may use an image projector (not shown) to project an image representing the pickup status onto the placement area AA.

[0160] (5) The reference information may be various types of information that associate the work identifier with the location information. For example, the identifier may be various strings such as the file name of the target image data or the product name of the work. The reference information may also be various images. When the reference information is an image, the correspondence between the work identifier and the location information may be represented not only by the proximity of the distance on the image between the work's location and the identifier (superimposed image IM11o (Figure 8(B))), identifier-attached arrows L1-L3 (Figure 13(A)), identifier-attached frames FD1-FD3 (Figure 13(B)), but also by various configurations of the image. In addition, the reference information may be various types of information that associate the work identifier with the location information, such as identifier coordinate data DR (S257b (Figure 10)) or identifier coordinate data DRc (S295c (Figure 12)), instead of an image. When a robot (for example, robot 300 (Figure 9)) is used to pick up the workpiece, the position information may represent three-dimensional coordinates representing the position on the placement area AA instead of two-dimensional coordinates representing the position on the captured image. The processor 210 may convert the two-dimensional coordinates on the captured image to three-dimensional coordinates according to a predetermined coordinate correspondence.

[0161] (6) The method for outputting the reference information may be any method that allows the picker (e.g., a worker or robot 300) to recognize the content of the reference information. For example, if the picker is a person (e.g., a worker), the processor 210 may use an image projector (not shown) to project an image representing the reference information onto the placement area AA.

[0162] (7) The output of reference information may be performed at various timings. The processor 210 may output reference information before or after machining the workpiece. In the embodiment shown in Figure 2, the processor 210 may generate and output reference information after panelization (S250) and before machining (S260).

[0163] (8) The processes for carrying out the processing (e.g., printing) (S260 (Figure 2), S260b (Figures 10, 12)) may be various processes that promote the progress of the processing. If the processing is carried out by a person (e.g., an operator), the processor 210 may perform various processes to encourage the person to proceed with the processing (e.g., turning on a lamp, outputting sound). The processor 210 may also store data of a flag that allows processing in a storage device referenced by the processing device (e.g., the first printing device 400). Such a storage device may be the storage device 215 of the data processing device 200 (e.g., the non-volatile storage device 230), or alternatively, it may be the storage device of the processing device.

[0164] In the case where the robot is the workpiece picker, the second specific case in which processing to proceed with machining is executed is not limited to the case where the judgment result of S245 is Yes Cb (Figure 10), but may be any case in which the second specific condition is met. The second specific condition may be a variety of conditions, including the detection of multiple unprocessed workpieces from the captured image. For example, the second specific condition may include the completion of panelization (Figure 10: S250). The second specific condition may also include the completion of the transmission of identifier coordinate data DR to the robot 300 (Figure 10: S257b).

[0165] Furthermore, the processing for advancing the machining may be performed regardless of the type of picker. The third specific case in which the processing for advancing the machining is performed regardless of the type of picker is not limited to cases where the judgment result of S245 is Yes Cb, Cc (Figures 10, 2), but may also be any case in which the third specific condition is met. The third specific condition may be a variety of conditions, including the detection of multiple unprocessed workpieces from the captured image. For example, the third specific condition may include the completion of panelization (Figures 2, 10: S250).

[0166] Furthermore, the processes necessary to carry out the manufacturing process may be omitted.

[0167] (9) The process for proceeding to the next step of processing (e.g., printing) (S365 (Figure 3), S365b (Figure 11)) may be various processes that prompt the next step to proceed. If the next step is to be carried out by a person (e.g., an operator), the processor 210 may perform various processes to prompt the person to proceed to the next step (e.g., turning on a lamp, outputting sound). The processor 210 may also store data of a flag that allows the next step in a storage device referenced by the device that performs the next step (e.g., a robot not shown). Such a storage device may be the storage device 215 of the data processing device 200 (e.g., the non-volatile storage device 230), or alternatively, it may be the storage device of the device that performs the next step.

[0168] In the first specific case in which processing to proceed to the next step is executed, the judgment result of S345 is No and the judgment result of S360 is Yes, not limited to Ca1, Ca2 (Figures 3 and 11), but also when the first specific condition is met. The first specific condition may be a variety of conditions, including not detecting a processed workpiece from the captured image. For example, the judgment result of S360 may be omitted from the first specific condition. That is, the next step may be performed for each batch. Note that processing to proceed to the next step may be omitted.

[0169] (10) When the picking of workpieces from the placement area AA is performed by the robot 300 (Figure 9), the placement of workpieces in the placement area AA may be performed by a person (e.g., an operator). Also, when the picking of workpieces from the placement area AA is performed by a person (e.g., an operator), the placement of workpieces in the placement area AA may be performed by a robot (e.g., robot 300).

[0170] (11) The specific image used for outputting the reference information may be various images representing the placement area (for example, placement area AA (Figures 1 and 9)) taken with multiple unprocessed or processed workpieces placed in the placement area. The timing of acquiring the specific image (for example, acquiring the specific image from a camera that photographs the placement area) may be determined independently of the timing of outputting the reference information. For example, if the reference information is output after the workpieces have been processed, the specific image may be acquired before the workpieces have been processed. If the specific image is acquired before the workpieces have been processed, it is preferable that the position of the workpieces on the placement area AA does not change between before and after processing.

[0171] (12) A sheet representing an identifier (for example, label sheets LS1-LS3 (Figure 7(A)) may represent various images including the identifier. For example, the sheet may represent a description of the workpiece. In any case, it is preferable that the identifier is represented by human-readable characters. However, the identifier may be represented by a code image such as a one-dimensional code or a two-dimensional code.

[0172] If the next step after workpiece processing is performed by a person (e.g., an operator), it is preferable that the workpiece picked up from the placement area and the sheet representing the workpiece identifier are placed in close proximity (e.g., in the same tray). For example, in the embodiment shown in Figures 9-12, if the next step after workpiece processing (S370b (Figure 11)) is performed by a person, it is preferable that the processor 210 causes the second printing device 700 to print sheets LS1-LS3, similar to the embodiment shown in Figures 1 and 7(A). Then, as shown in Figure 7(D), it is preferable that sheets LS1-LS3 are placed in trays TR1-TR3.

[0173] If the next step after workpiece processing is performed by a machine (e.g., a robot), the sheet representing the identifier may be omitted. For example, in the embodiment shown in Figures 1-3, if the next step after workpiece processing (S370 (Figure 3)) is performed by a machine, the printing and placement of sheets LS1-LS3 may be omitted. However, if the workpieces are picked up from the placement area AA by a person, the sheet representing the identifier may be used to distinguish between the multiple picked-up workpieces.

[0174] (13) Multiple workpieces to be processed may include multiple types of workpieces that differ from each other in one or more of the following aspects: material, shape, and processing content. Various material properties, such as color and substance (acrylic, aluminum, paper, cloth, wood, rubber, etc.), may differ among the multiple types of workpieces. Processing of workpieces is not limited to printing images, but may include various other processing methods (e.g., laser engraving, drilling, etc.). The configuration of the apparatus for processing workpieces may be various configurations suitable for processing. Various processing details, such as the image to be printed, the pattern formed by laser engraving, and the size of the holes to be drilled, may differ among the multiple types of workpieces. In any case, the placement area may be various areas in which the workpieces are placed for processing. For example, workpieces may be supported by jigs for processing. In this case, the placement area may include the area containing the jigs.

[0175] (14) The configuration of the manufacturing system used for partial manufacturing is not limited to the configurations of manufacturing systems 1000 and 1000b in Figures 1 and 9, but may be various configurations. For example, the configuration of the robot 300 may be various configurations suitable for workpiece pickup and placement of the picked-up workpieces. The print control device 490 (Figure 1) may be omitted, and the data processing device 200 may control the print execution unit 410. The print control device 790 may be omitted, and the data processing device 200 may control the print execution unit 710. The robot control unit 390 (Figure 9) may be omitted, and the data processing device 200 may control the robot arm 310.

[0176] (15) The data processing device 200 (Figures 1 and 9) may be a different type of device from a personal computer (e.g., a digital camera, a smartphone). The data processing device may also be part of a printing device or part of a digital camera. Alternatively, multiple devices (e.g., computers) that can communicate with each other via a network may each share a portion of the data processing function performed by the data processing device, and together they may provide the data processing function (a system comprising these devices corresponds to the data processing device).

[0177] In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, the processing by the object detection model 232 may be performed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC) instead of a program module.

[0178] Furthermore, if some or all of the functions of this disclosure are implemented by a computer program, that program may be provided in the form of a computer-readable recording medium (e.g., a non-temporary recording medium). The program may be used while stored on the same or a different recording medium (computer-readable recording medium) as it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as hard disk drives.

[0179] The above embodiments and modifications can be combined as appropriate. Furthermore, the above embodiments and modifications are provided to facilitate understanding of this disclosure and do not limit the present invention. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are included. [Explanation of Symbols]

[0180] 200...Data processing device, 210...Processor, 215...Storage device, 220...Volatile storage device, 230...Non-volatile storage device, 231,231b,231c...Program, 232...Object detection model, 240...Display unit, 250...Operation unit, 270...Communication interface, 300...Robot, 310...Robot arm, 314...End effector, 390...Robot control unit, 400...First printing device, 410...Print execution unit, 412...Head, 413...Light source, 416...Platen, 418...Moving device, 490...Print control device, AA...Placement area, 500...Camera, 700...Second printing device, 710...Print execution unit, 790...Print control device, 1000,1000b...Manufacturing system, NT...Local area network, 233...Batch information, 250r...Result image, DR,DRc...Identifier coordinate data, Dx...First direction, D y...Second direction, L1-L3...Arrow with identifier, FD1-FD3...Frame with identifier, ID, ID1-ID3...Identifier, IM01, IM02, IM03, IM11, IM12, IM13...Captured image, IM01o, IM02o, IM03o, IM11o, IM12o, IM13o, IM11d, IM11e,...Superimposed image, IMP...Image for printing, LS, LS1-LS3...Label sheet, MS10, MS14...Mesh Sage, OB01-OB04, OB11-OB14, ... object area, P1-P4, Pw1-Pw4 ... center of gravity position, P1o, P2o, P3o, P4o ... superposition position, STa ... placement status, STp ... pickup status, TSa, TSP ... status table, TR1-TR3 ... tray, Wa1, Wa2, Wb1, Wb2 ... unprocessed workpiece, Wa1x, Wa2x, Wb1x, Wb2x ... processed workpiece

Claims

1. It is a program, A function to acquire a specific image which is an image of a placement area for processing multiple workpieces, wherein the specific image is an image taken with multiple unprocessed or processed workpieces placed in the placement area, and the multiple workpieces include a first workpiece and a second workpiece which differ from each other in one or more items among material, shape, and processing content, and the function is described above. A function that performs output processing to output reference information used to pick up multiple processed workpieces to a picker using the specified image, wherein the reference information includes first information which is information relating a first identifier and first position information of the first workpiece, and second information which is information relating a second identifier and second position information of the second workpiece, A program that enables a computer to realize this.

2. The program according to claim 1, The aforementioned picker is a person. program.

3. The program according to claim 2, The output process includes the process of displaying the reference information on a display device. program.

4. A program according to claim 2 or 3, The function that performs the output processing performs the output processing after the machining of the plurality of workpieces. program.

5. The program according to claim 4, further, A first display process that displays the pickup status of each of the aforementioned processed workpieces on a display device, wherein the pickup status is selected from a plurality of statuses including not picked up and picked up, and a computer is provided with a function to perform the first display process. The first display process is, The process involves repeatedly acquiring images from a camera that photographs the aforementioned arrangement area, A process to determine the pickup status of the workpiece detected from the aforementioned captured image to be "not picked up", A process to determine the pickup status of workpieces not detected in the aforementioned captured image as "picked up", A program that includes this.

6. The program according to claim 4, The aforementioned process is printing. The processing details mentioned above refer to the image to be printed. The printed images differ between the first workpiece and the second workpiece. After multiple unprocessed workpieces are placed in the placement area, the image to be printed on the first unprocessed workpiece is determined as the first image without user specification, and the image to be printed on the second unprocessed workpiece is determined as a second image different from the first image without user specification. program.

7. A program according to claim 1 or 2, further, A program that causes a computer to perform the function of causing a printing device to print a sheet representing the first identifier and a sheet representing the second identifier before the pickup of the multiple processed workpieces.

8. The program according to claim 1, The aforementioned picker is a robot. program.

9. L according to claim 8, The first position information represents coordinates indicating the position of the first workpiece, The second position information represents coordinates indicating the position of the second workpiece. program.

10. A program according to claim 8 or 9, The function that performs the output processing performs the output processing before the machining of the plurality of workpieces. program.

11. A program according to claim 8 or 9, The aforementioned workpieces are the subject of a plurality of processes, including the machining and the subsequent process following the machining. The aforementioned program, further, A function to repeatedly acquire images from a camera that photographs the arrangement area after the processing of the aforementioned multiple workpieces, In the first specific case where no processed workpiece is detected from the captured image, a function is provided to execute a process to proceed to the next step. A program that enables a computer to realize something.

12. A program according to claim 8 or 9, The aforementioned process is printing. The processing details mentioned above refer to the image to be printed. The printed images differ between the first workpiece and the second workpiece. After multiple unprocessed workpieces are placed in the placement area, the image to be printed on the first unprocessed workpiece is determined as the first image without user specification, and the image to be printed on the second unprocessed workpiece is determined as a second image different from the first image without user specification. program.

13. A program according to claim 8 or 9, Multiple unprocessed workpieces are placed in the aforementioned placement area by a robot. program.

14. The program according to claim 13, further, A function to repeatedly acquire images from a camera that photographs the arrangement area before the processing of the aforementioned multiple workpieces, A function to execute a process for proceeding with the processing in a second specific case in which multiple unprocessed workpieces are detected from the captured image, A program that enables a computer to realize something.

15. A program according to claim 8 or 9, The function that performs the output processing performs the output processing after the machining of the plurality of workpieces. program.

16. A program according to claim 1 or 2, further, A second display process, which displays the placement status of each of the multiple workpieces on a display device before the processing of the multiple workpieces, wherein the placement status is selected from a plurality of statuses including not placed and placed, and a computer is provided with a function to perform the second display process. The second display process is as follows: The process involves repeatedly acquiring images from a camera that photographs the aforementioned arrangement area, A process to determine the placement status of workpieces not detected in the captured image as "not placed", A process to determine the placement status of the workpiece detected from the captured image as "placed," A program that includes this.

17. A program according to claim 1 or 2, further, A function to repeatedly acquire images from a camera that photographs the arrangement area before the processing of the aforementioned multiple workpieces, A function to execute a process for proceeding with the processing in a third specific case in which multiple unprocessed workpieces are detected from the captured image, A program that enables a computer to realize something.

18. A program according to claim 1 or 2, The output processing described above is: A process for detecting each of the multiple workpieces from the specified image, A process to determine an identifier associated with a detected work using the detection results of each of the aforementioned multiple workpieces, A process to determine position information associated with the detected workpiece using the detection results of each of the aforementioned multiple workpieces, A program that includes this.

19. A data processing device, An acquisition unit that acquires a specific image which is an image of a placement area for processing multiple workpieces, wherein the specific image is an image taken with multiple unprocessed or processed workpieces placed in the placement area, and the multiple workpieces include a first workpiece and a second workpiece, each having one or more different items among material, shape, and processing content, and the acquisition unit An output unit that performs output processing to output reference information used to pick up multiple processed workpieces to a picker using the specified image, wherein the reference information includes first information which is information relating a first identifier and first position information of the first workpiece, and second information which is information relating a second identifier and second position information of the second workpiece, A data processing device equipped with the following features.

Images