Projection control device, projection control method, and program

Abstract

It displays where on the circuit board the components that the worker has freely selected should be mounted. [Solution] The projection control device 100 includes a control unit 110 that identifies the type of circuit board and the components specified by the user, obtains the mounting position of the components on the circuit board from the storage unit 120, and projects a guide image indicating the mounting position onto the projection unit 150.

Description

Technical Field

[0001] The present invention relates to a projection control device, a projection control method, and a program.

Background Art

[0002] As a technology for automating work in factories, FA (Factory Automation) technology has been spreading. However, it is difficult to completely automate all work, and many parts of the work are still often done manually. Also, the development of technologies for improving the efficiency and reducing errors in manual work is progressing. For example, Patent Document 1 discloses an assembly support device that can support the assembly work of products even when the position of an article is not specified.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The assembly support device disclosed in Patent Document 1 reads procedure data stored in a storage unit, which sets the location where parts required for assembly exist, for each assembly procedure of a product, and irradiates light at the location where the part exists, so that an operator can save the time for searching for the part and can also prevent a selection error of the part. However, in such a conventional technology, since the parts are attached in order according to the procedure data, there is a problem that an operator cannot freely select the part to be attached.

[0005] In view of the above circumstances, the present invention has been made, and an object thereof is to provide a projection control device, a projection control method, and a program that can display where an operator can attach a freely selected part to a substrate. [Means for solving the problem]

[0006] To achieve the above objective, one embodiment of the projection control device according to the present invention includes a control unit that identifies the type of substrate and the type of component specified by the user, obtains the mounting position of the component on the substrate from a storage unit, and projects a guide image indicating the mounting position onto a projection unit. [Effects of the Invention]

[0007] According to the present invention, it is possible to display where on the circuit board a component freely selected by the worker should be attached. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the functional configuration of a projection control device according to an embodiment. [Figure 2] This figure shows an example of the installation of the projection control device according to the embodiment. [Figure 3] This is an example of a flowchart for projection control processing according to the embodiment. [Figure 4] This figure shows an example of a projected message prompting the start of work. [Figure 5] This figure shows an example of a substrate information database according to an embodiment. [Figure 6] This figure shows an example where a message prompting the scanning of a part barcode is projected. [Figure 7] This figure shows an example of a parts information database according to the embodiment. [Figure 8] This figure shows an example where a message prompting the installation of a component is projected. [Figure 9] This figure shows an example where a message indicating that the parts have been installed is projected. [Figure 10] This figure shows an example of a projected message encouraging the installation of multiple parts, which can be installed in any order. [Figure 11]This figure shows an example where a message is projected indicating that the installation of one of several parts, whose installation order can be freely chosen, has been completed. [Figure 12] This figure shows an example where a message is projected prompting the installation of multiple parts in a specified order. [Figure 13] This figure shows an example where a message is projected indicating that the first of several parts, whose installation order is specified, has been successfully installed. [Modes for carrying out the invention]

[0009] The projection control device and the like according to the embodiment will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals. The projection control device 100 according to this embodiment is a device for reducing the burden on the user (worker) during assembly work by mapping and projecting guide images indicating the positions for attaching parts onto a circuit board.

[0010] The projection control device 100, as shown in Figure 1, comprises a control unit 110, a storage unit 120, an operation input unit 130, a detection unit 140, and a projection unit 150. The control unit 110 is composed of a processor, such as a CPU (Central Processing Unit). The control unit 110 executes projection control processing and other processes described later by executing a program stored in the storage unit 120. The storage unit 120 stores programs executed by the control unit 110 and necessary data. The storage unit 120 may include, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), flash memory, etc. The operation input unit 130 is equipped with a user interface such as a keyboard, mouse, and push-button switches, and accepts operation input from the user.

[0011] The detection unit 140 includes a 3D (Dimension) camera that can simultaneously acquire a two-dimensional image and depth direction information, and a barcode reader. The 3D camera detects the three-dimensional shape of the substrate and components, and the barcode reader reads the ID (Identification), which is unique information indicating the types of the substrate and components. The projection unit 150 is a projector (projection device), which includes a light source, a projection device, a lens unit, etc., and projects the given video based on the control from the control unit 110.

[0012] The functional configuration of the projection control device 100 has been described above. The projection control device 100 is configured, for example, as shown in FIG. 2, in a form where a 3D camera 141, a barcode reader 142, and a projector 151 are connected to a PC (Personal Computer) 111, and is installed on a workbench 300 for performing the operation of attaching components 201 to a substrate 200. In the example shown in FIG. 2, two component trays 301 storing a large number of components 201 and one substrate 200 are placed on the workbench 300. Actually, however, a large number of substrates 200 and component trays 301 are stored in a location (such as a storage shelf) different from the workbench 300. The user (operator) brings the substrate 200 for performing the current operation and the component tray 301 storing the components 201 to be attached to the substrate 200, and performs the operation on the workbench 300.

[0013] Also, as shown in FIG. 2, the 3D camera 141 of the detection unit 140 and the projector 151 of the projection unit 150 are installed to face the same direction. The projection control device 100 projects a video (guidance video) indicating the component attachment position at the position where the component 201 of the substrate 200 is to be attached. In order to perform such processing, it is necessary to establish a correspondence between the component attachment position (coordinates) in the image captured by the 3D camera 141 of the detection unit 140 and the position (coordinates) in the video projected by the projector 151 of the projection unit 150. Therefore, after being installed on the workbench 300, the projection control device 100 performs calibration processing to establish a correspondence between the coordinates of the projector 151 of the projection unit 150 (projector coordinates) and the coordinates of the 3D camera 141 of the detection unit 140 (camera coordinates).

[0014] Although it is arbitrary how the calibration process is performed, for example, the control unit 110 projects a checkerboard pattern check image onto the work machine 300 with the projector 151, and captures the projected check image with the 3D camera 141 to obtain a captured image. Then, the control unit 110 obtains the correspondence between the coordinates of each intersection point in the captured image (camera coordinates) and the coordinates of each intersection point in the original check image (projector coordinates). By using this correspondence, the control unit 110 can know what values should be set as the projector coordinates for projection onto a certain area (the area represented by camera coordinates) on the work machine 300.

[0015] Next, the projection control process will be described with reference to FIG. 3. This process is started when the user (operator) starts the operation of attaching the component 201 to the substrate 200. For example, when the operator places the substrate 200 on the work machine 300 and instructs the projection control device 100 to execute the projection control process, the execution of the projection control process is started.

[0016] When the projection control process is started, the control unit 110 performs a work start process (step S101). Specifically, the control unit 110 performs various initial settings (which may include, for example, the calibration process described above), and projects a message prompting the user 400 to start work, such as "Start work" onto the work machine 300 as shown in FIG. 4. Also, in step S101, the control unit 110 may obtain information about the work to be performed from the operation input unit 130, or obtain the user 400's unique information (operator ID) with a barcode reader 142 or the like. By obtaining the operator ID, the control unit 110 can store the work log data in the subsequent attachment work in association with the operator ID in the storage unit 120.

[0017] Next, the control unit 110 acquires the unique information (board ID) of the board 200 using a barcode reader 142 or the like (it may also be read by a 3D camera 141, not just the barcode reader 142) (step S102). Here, it is assumed that a barcode indicating the board ID is attached to the board 200, but the control unit 110 may acquire the unique information of the board 200 by having the user 400 input information such as the model number or serial number of the board 200 from the operation input unit 130. The storage unit 120 has in advance stored the 3D data of each board 200 and the mounting position data of each component 201 (component position data) linked to the board ID as a board information database 121 as shown in Figure 5, and the control unit 110 acquires the 3D data of the board 200 and the mounting position of each component 201 corresponding to the board ID from the board information database 121 based on the board ID acquired in step S102. Although not shown in the diagram, the board information database 121 also stores mounting order information for each component 201 (information indicating whether the mounting order is specified (determined) or whether it can be mounted in any order, and if the mounting order is specified, information indicating that order), linked to the board ID.

[0018] Then, the control unit 110 obtains the unique information (part ID) of the part 201 that the user 400 is trying to attach to the circuit board 200 (step S103). For example, as shown in Figure 6, the control unit 110 projects a message onto the workbench 300 prompting the user 400 to scan the part barcode, such as "Please scan the part barcode," and obtains the part ID of the part 201 specified (selected) by the user using the barcode reader 142. The part barcode is affixed to the part tray 301 in which the part 201 is stored, and the user scans the part barcode affixed to the part tray 301 using the barcode reader 142 or the like. However, the part barcode may also be affixed to the part 201, in which case the user scans the part barcode affixed to the part 201. The memory unit 120 has pre-stored the names and 3D data of each component 201 linked to the component ID as a component information database 122 as shown in Figure 7. The control unit 110 retrieves the type (name (switch, capacitor, resistor, coil, etc.), rated value (capacitance value, resistance value, inductance value, etc.), size, etc.) and 3D data of the component 201 corresponding to the component ID from the component information database 122 based on the component ID acquired in step S103. If the component 201 specified (selected) by the user in step S103 is a component that cannot be attached to the circuit board 200 at this time, the control unit 110 may project a message onto the workbench 300 prompting the user 400 to select a different component, such as "This component cannot be attached at this time. Please select the correct component," and return to step S103.

[0019] Next, the control unit 110 uses the 3D camera 141 to photograph the circuit board 200 placed on the workbench 300 (step S104). Here, the control unit 110 compares the 3D data of the circuit board 200 acquired in step S102 with the photographic data including 3D information acquired in step S104 to determine the position and orientation of the circuit board 200 on the workbench 300. Then, based on the component position data acquired in step S102, the control unit 110 projects a guide image onto the circuit board 200 at the position where the component 201 will be attached (step S105). In this step, the control unit 110 projects a marker 152 indicating the position where the component 201 will be attached onto the circuit board 200 as a guide image, along with a message prompting the attachment of the component 201, such as "Attach the switch (0 / 1 Complete)," on the workbench 300. The messages displayed in this step consist of a message instructing the user on what to do (a "work message"), such as "Install the switch," and a message indicating the progress ("(0 / 1 completed)"), such as "(0 / 1 completed)," indicating that 0 out of a total of 1 part have been installed (a "progress message"). In this example, the progress message shows the progress as a fraction of "number of completed parts / total number that need to be installed" for part 201, but the way progress is shown is not limited to this example.

[0020] This guide video may, for example, indicate the location of parts using shapes (rectangles, circles, etc.) of a predetermined color (red, green, etc.), or the shapes may be made to flash to highlight the location of the parts. Alternatively, the location of the parts may be highlighted by projecting shapes that change in size (for example, repeatedly narrowing the size of the shapes towards the center of the part mounting area) or by projecting shapes that change in brightness (for example, repeatedly making them gradually darker and then gradually brighter). Furthermore, in this projection, the control unit 110 calculates coordinates while constantly tracking the position and orientation of the substrate acquired in step S104 or step S106 (described later), and then maps and projects the marker 152 to the component mounting position. Therefore, even if the user 400 moves the substrate 200 by hand, the control unit 110 projects a guide image, which is an image that maps the marker 152 to the mounting position of the component 201, following the movement of the substrate 200. Furthermore, if the control unit 110 has not installed the part 201 at the installation location even after a predetermined amount of time has elapsed since the start of projecting the guide image (i.e., the determination in step S107 described later is not Yes), it may project a time-elapsed image to indicate that time has passed (for example, an image displaying the message "It is taking time" or an image showing the marker 152 flashing rapidly). By projecting the time-elapsed image, the user 400 can become aware that it is taking time to install the part 201, thereby reducing delays in the work.

[0021] The control unit 110 then takes another photograph of the circuit board 200, which is placed on the workbench 300, with the 3D camera 141 (step S106). Here, the control unit 110 compares the 3D data of the circuit board 200 acquired in step S102 and the 3D data of the component 201 acquired in step S103 with the photographic data including the 3D information acquired in step S106, thereby acquiring information on the position and orientation of the circuit board 200 and the position and orientation of the component 201. The control unit 110 then checks the position and orientation of the component 201 and determines whether the component 201 is correctly mounted at its mounting position on the circuit board 200 (step S107). This determination method is arbitrary, but for example, in step S106, the control unit 110 captures a predetermined number of frames (e.g., 10 frames) of video data with the 3D camera 141 and determines that the component 201 is "mounted" if it remains stationary at its mounting position on the circuit board 200 during that time. Alternatively, in step S106, the control unit 110 may capture still image data instead of video data with the 3D camera 141 and repeat the loop from steps S105 to S107 several times. If it can determine that the component 201 is stationary at its mounting position on the circuit board 200 for a predetermined time (e.g., 1 second), it may determine that the component is "mounted". Here, a relatively simple method using image processing was introduced for determining whether or not component 201 is installed. However, more complex determination methods (such as determination using machine learning, determination by comparing 3D data in the board information database 121 and component information database 122 with 3D data detected by the detection unit 140, etc.) may also be used.

[0022] If component 201 is not mounted in the mounting position for component 201 on the circuit board 200 (step S107; No), return to step S105. If component 201 is mounted to its mounting position on the circuit board 200 (step S107; Yes), the control unit 110 terminates the projection of the guide image (step S108) and projects the progress status (step S109). For example, as shown in Figure 9, the control unit 110 projects a message indicating that the component has been mounted, such as "Switch mounting complete" (mounting completion message), along with a message indicating the progress, such as "(1 / 1 complete)" (one of the one components has been mounted). In step S108, instead of simply terminating the projection of the guide image, a predetermined marker indicating that the mounting work of the component has been completed (for example, a marker hatched with diagonal lines on component 201 on the circuit board 200 as shown in Figure 9) may be mapped and projected onto component 201 on the circuit board 200 as a completion display image. By projecting the completion display image, the user 400 can confirm at a glance that the component 201 has been mounted and proceed to the next task with confidence.

[0023] Then, the control unit 110 determines whether or not all components 201 to be attached to the substrate 200 have been attached (step S110). This determination method is arbitrary, but for example, the control unit 110 compares the component position data in the substrate information database 121 with the current image data of the substrate 200 and components 201 taken in step S106 to determine whether or not all components 201 to be attached to the substrate 200 have been attached. If all components have not yet been attached (step S110; No), the process returns to step S103. If all components have been attached (step S110; Yes), the control unit 110 projects a mounting completion message, such as "All components have been attached" (step S111), and terminates the projection control process.

[0024] By performing the above projection control process, the projection control device 100 displays where on the circuit board 200 the component 201 that the user 400 has freely selected should be mounted, so that the user 400 can easily proceed with the task of mounting the component 201 on the circuit board 200.

[0025] In the above explanation, we described the case where one component 201 with component ID PT01 is attached to board 200 with board ID BD01 (only one component 201 (switch) needed to be attached to board 200), so in step S105, the progress message "(0 / 1 Complete)" was projected along with the task content message "Please attach the switch". However, depending on the component 201, it may be necessary to attach multiple identical components 201. By referring to the board information database 121 shown in Figure 5, the control unit 110 can determine, for example, that three components 201 with component ID PT03 need to be attached to board 200 with board ID BD01. Therefore, in this case, the progress message projected in step S105 will be "(0 / 3 Complete)". Also, by referring to the component information database 122 shown in Figure 7, the control unit 110 can determine that the name of component 201 with component ID PT03 is a resistor. Therefore, in this case, the task message projected in step S105 will be "Install the resistor".

[0026] Furthermore, although not shown in Figure 5, the board information database 121 also stores mounting order information as described above, and the control unit 110 can refer to the mounting order information to determine, for example, whether the mounting order of component 201, whose component ID is PT03, is flexible or whether it must be mounted in a specified order. If the mounting order of the three components 201 is flexible, the control unit 110 projects a guide image displaying markers 152 at all (three locations in this example) mounting positions for the components 201 on the board 200, as shown in Figure 10. This allows the user 400 to understand that they only need to mount the component 201 at one of these three marker positions 152. Once the user 400 has mounted one component 201, the control unit 110 stops projecting the marker 152 at the mounted position, as shown in Figure 11, and the user 400 understands that they only need to mount the component 201 at one of the remaining two marker positions 152. Then, once all three components 201 have been installed, the control unit 110 projects a message indicating that the component 201 has been installed, which reads "Resistor installation complete (3 / 3 complete)".

[0027] On the other hand, if the mounting order of component 201 is not flexible and it must be mounted in a specified order, the control unit 110 projects a guide video that displays markers 152 one by one in the specified order. For example, let's consider the case where three component 201 with component ID PT03 need to be mounted in order from the top (the side furthest from the user 400 in the example shown in Figure 12). In this example as well, component 201 (the component with component ID PT03) is a resistor, and three of them need to be mounted, so in step S105 the work content message becomes "Please mount the resistors" and the progress message becomes "(0 / 3 Complete)". Therefore, as shown in Figure 12, the control unit 110 projects a guide video that displays marker 152 at the mounting position of the first component 201 to be mounted on the circuit board 200, along with the message "Please mount the resistors (0 / 3 Complete)". This allows the user 400 to understand that they just need to mount the component 201 at the position of this marker 152. Then, when user 400 installs component 201, control unit 110 stops projecting marker 152 at the position where component 201 was installed, as shown in Figure 13, and projects marker 152 at the next position where component 201 will be installed. This lets user 400 know that they should then install component 201 at the position of marker 152. When all three components 201 have been installed, control unit 110 projects a message indicating that the installation of component 201 has been completed: "Resistor installation complete (3 / 3 complete)".

[0028] Thus, when it is necessary to install multiple identical parts 201, if the installation order is specified (determined), a guide video is projected showing the markers 152 in that order, allowing the user to install the parts 201 without confusion about the installation location. Also, if the installation order is flexible, a guide video is projected showing the markers 152 at all installation locations for the parts 201, allowing the user to install the parts 201 in any order from among the multiple displayed markers 152, without any restrictions on the installation order.

[0029] Furthermore, in the projection control process described above (Figure 3), the control unit 110 proceeded with the process assuming that the component 201 was correctly mounted on the substrate 200 if the determination in step S107 was Yes. However, it may also determine whether the component 201 is mounted in the wrong position or in the wrong orientation, and if the mounting position or orientation is incorrect, project a warning message such as "The mounting position is incorrect" or "The mounting orientation is incorrect" and return to step S105. In addition, if the mounted component 201 is not the assumed component 201 (the component indicated by the component ID obtained in step S103), it may project a warning message such as "The wrong component has been mounted" and return to step S105. Furthermore, if an installation error occurs as described above, the control unit 110 may store information related to such installation errors of part 201 (error information) in the storage unit 120 as work log data. Also, if the worker ID was obtained in step S101, the control unit 110 may store the error information in the work log data linked to the worker ID.

[0030] The control unit 110 then searches the work log data for error information related to installation errors of the currently installed part 201, and in step S105, depending on the error information found, it may project a warning video that includes a message to alert the user 400, such as "N errors have occurred in the past" or "Please pay attention to the orientation." When projecting such a warning video, the projection state of the marker 152 may also be changed (e.g., blinking, enlargement / reduction, color change, etc.) to further alert the user 400. Furthermore, if error information is recorded in the work log data linked to the worker ID, the control unit 110 may search for error information linked to the worker ID of the currently working user 400 when searching for error information, thereby enabling the acquisition of more accurate error information. Furthermore, information on common errors, such as those that are known to occur particularly frequently, may be stored not only as error information in the work log data, but also as general error information in the parts information database 122. In step S105, the control unit 110 may retrieve the general error information for the part 201 from the parts information database 122 and project it. In this way, the projection control device 100 can project warning images and the like based on error information, thereby performing projections appropriate to the difficulty level of the task. When warning images and the like are projected, the user 400 can recognize that the installation of the part 201 is difficult and prone to errors, and can proceed with the work carefully.

[0031] Furthermore, in the above-described embodiment, the control unit 110 always acquired unique information of the substrate 200 in step S102. However, if there is no need to use the unique information of the substrate 200, such as when there is only one type of substrate 200, the control unit does not need to acquire the unique information of the substrate 200. In other words, in this case, step S102 of the projection control process can be skipped, and subsequent processing can proceed while ignoring the unique information of the substrate 200. Furthermore, while the above-described embodiment assumed a case where one user selects components one by one and attaches them to the circuit board, it is also possible to assume cases where one user selects multiple components simultaneously using both hands, or where multiple components are selected simultaneously by multiple users working at the same time. In this case, the control unit 110 may change the color of the guide video that shows the mounting position of each of the multiple components selected by the user (for example, projecting a red frame on the mounting position of the component selected by user 1, and a blue frame on the mounting position of the component selected by user 2), or change the shape, frame type, line thickness, and appearance (dotted line, dashed line, blinking, etc.) of the guide video so that it is possible to distinguish which mounting position each component should be attached to.

[0032] Furthermore, the projection control device 100 does not necessarily have to be a standalone device. It can also be implemented as a projection control system composed of multiple devices, such as a PC or other computer (corresponding to the control unit 110, memory unit 120, and operation input unit 130), a 3D camera (corresponding to the detection unit 140), and a projector (corresponding to the projection unit 150). In addition, if the 3D camera or projector is equipped with a control unit 110, a PC is not required. For example, the control unit 110 of the projector may perform the projection control processing described above.

[0033] Furthermore, in the above embodiment, it was described that the program for projection control processing and other processes executed by the control unit 110 is pre-stored in the storage unit 120. However, a computer capable of executing the above-mentioned processes may be configured by distributing the program on a non-temporary computer-readable recording medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), DVD (Digital Versatile Disc), MO (Magneto-Optical disc), memory card, or USB memory, and then loading and installing the program into the computer.

[0034] Furthermore, the program can be superimposed on a carrier wave and applied via a communication medium such as the Internet. For example, the program could be posted and distributed on a bulletin board system (BBS) on a communication network. This program could then be launched and executed under the control of the operating system (OS), just like any other application program, to perform the aforementioned processes.

[0035] Furthermore, the control unit 110 may consist of any single processor, such as a single processor, multi-processor, or multi-core processor, or it may be configured by combining any of these processors with processing circuits such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array).

[0036] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these specific embodiments, and the present invention includes the invention described in the claims and its equivalents. [Explanation of Symbols]

[0037] 100...Projection control device, 110...Control unit, 120...Storage unit, 130...Operation input unit, 140...Detection unit, 150...Projection unit

Claims

1. Identify the type of circuit board and user-specified components, The mounting position of the component on the substrate is obtained from the storage unit, A guide image showing the aforementioned mounting position is projected onto the projection unit. Equipped with a control unit, Projection control device.

2. The control unit, Determine whether the component is attached to the aforementioned mounting position. If the part is not installed at the aforementioned installation location, a guide image indicating the installation location is projected onto the projection unit. The projection control device according to claim 1.

3. The control unit, If there are multiple mounting positions, the mounting order of the component is obtained from the storage unit. If the acquired mounting order is arbitrary, the guide video showing all of the multiple mounting positions is projected onto the projection unit. If the acquired installation order is specified, the guide images showing the installation positions are projected onto the projection unit one by one in the specified order. The projection control device according to claim 1.

4. The control unit, Confirm the position and orientation of the aforementioned parts, If the component is correctly installed in the aforementioned mounting position, a completion display image indicating that the installation work is complete will be projected onto the projection unit. The projection control device according to claim 1.

5. The control unit, Error information related to the installation error of the aforementioned parts is retrieved from the storage unit. In accordance with the error information retrieved, a warning video to alert the worker is projected onto the projection unit. The projection control device according to claim 1.

6. The control unit, Obtain the worker's unique information, The error information by the worker, represented by the acquired unique information, is retrieved from the storage unit. In accordance with the error information retrieved, a warning video to alert the worker is projected onto the projection unit. The projection control device according to claim 5.

7. The control unit, If the component specified by the user is a component that cannot be mounted on the circuit board, To project that information onto the projection unit. The projection control device according to claim 1.

8. The control unit, Identify the type of circuit board and user-specified components, The mounting position of the component on the substrate is obtained from the storage unit, A guide image showing the aforementioned mounting position is projected onto the projection unit. Projection control method.

9. In the control unit, Identify the type of circuit board and user-specified components, The mounting position of the component on the substrate is obtained from the storage unit, A guide image showing the aforementioned mounting position is projected onto the projection unit. A program that executes a process.

Images