Component mounting system, image processing apparatus, image processing method, computer program product, and image processing system

Abstract

A component mounting system includes a plurality of production devices provided in a component mounting line that mounts components on a substrate, a photographing unit provided in the component mounting line and that photographs the plurality of production devices and one or more workers provided in the component mounting line, and an image processing unit that performs image processing on images photographed by the photographing unit during a predetermined production period in which products are produced using the component mounting line. The image processing unit visualizes and displays the working status of at least one of the plurality of production devices at each of a plurality of times or each of a plurality of periods set in advance by at least one of the one or more workers during the production period.

Description

Technical Field

[0001] This specification relates to component mounting lines for mounting components onto a substrate. More specifically, it relates to techniques for visualizing the work status of operators working on component mounting lines. Background Technology

[0002] In a component mounting line where components are mounted onto a substrate, multiple production equipment (such as printers, component mounting machines, reflow ovens, and substrate inspection machines) are installed. The substrate is sequentially transported to these multiple production equipment, where it undergoes predetermined processing to mount components. Each production equipment requires preparatory work (such as supplying raw materials (components, etc.)) to mount components onto the substrate. Therefore, operators are assigned to the component mounting line, performing necessary tasks based on the operating conditions of the production equipment. To improve the production efficiency of the component mounting line, the operators' work conditions must be appropriate relative to the operating conditions of the production equipment. Therefore, techniques for evaluating the work conditions of operators have been developed (e.g., Japanese Patent Application Publication No. 2019-191748). Summary of the Invention

[0003] The problem that the invention aims to solve

[0004] In the aforementioned disclosed technology, images are taken of the production equipment being analyzed and the workers operating on that equipment. Furthermore, by analyzing the captured images, the operating status of the production equipment and the position and orientation of the workers are obtained to determine whether the workers' operation on the production equipment is appropriate. While the aforementioned disclosed technology can evaluate the appropriateness of the workers' operation on the production equipment being analyzed, multiple production machines are typically installed on a component assembly line, and workers operate on these multiple machines. Therefore, even if the operation on one production machine is appropriate, the operation on other production machines may not be appropriate. The aforementioned disclosed technology cannot evaluate the appropriateness of the workers' operation relative to the multiple production machines installed on a component assembly line.

[0005] This specification discloses a technique for evaluating the suitability of an operator's work performance relative to multiple production devices installed on a component assembly line.

[0006] Methods for solving problems

[0007] The component mounting system disclosed in this specification includes: multiple production devices provided on a component mounting line for mounting components onto a substrate; an imaging unit provided on the component mounting line for capturing images of the multiple production devices and one or more operators on the component mounting line; and a display unit that, by image processing the images captured by the imaging unit during a predetermined production period of producing products using the component mounting line, visualizes and displays the operating status of at least one of the multiple production devices for at least one of the one or more operators at each of a plurality of pre-set times or periods during the production period.

[0008] In the aforementioned component assembly system, multiple production devices and one or more operators on the component assembly line are photographed. The photographed images are then processed to visualize the operators' work status. By photographing multiple production devices, it is possible to evaluate whether the operators' work is appropriate relative to the operation of these devices. Attached Figure Description

[0009] Figure 1 This is a diagram showing the overall structure of the component mounting system.

[0010] Figure 2 This is a flowchart illustrating the processes performed in an image processing apparatus.

[0011] Figure 3 It is shown Figure 2 A flowchart of the image processing performed in S20.

[0012] Figure 4 It is a diagram used to illustrate the steps for calculating the distance between the operator and the production equipment.

[0013] Figure 5 This is an example of a timeline diagram that visualizes the work status of operators.

[0014] Figure 6 This is another example of a timeline diagram that visualizes the work status of operators. Detailed Implementation

[0015] In the technology disclosed in this specification, the component mounting system may further include an operation status acquisition unit, which acquires the operation status of at least one of a plurality of production equipment at various times or during various periods. The display unit may also correlate and display the acquired operation status of the production equipment with the work status. With this structure, the operation status of the production equipment and the work status of the operator are correlated and displayed, thus making it easy to evaluate whether the operator's work status is appropriate.

[0016] In the technology disclosed in this specification, the operating status acquisition unit can acquire the operating status through image processing. For example, a production equipment may have signal lights that notify operators of specific tasks. In this case, when photographing the production equipment, the signal lights are photographed along with the production equipment, thereby acquiring the operating status of the production equipment.

[0017] In the technology disclosed in this specification, at least one of a plurality of production devices may also have a status output unit that outputs its own status. The operating status acquisition unit may also acquire the operating status based on the status output from the status output unit. With this structure, the operating status of the production device can be accurately acquired based on the information output from the production device.

[0018] In the technology disclosed in this specification, the component mounting system may further include a judgment unit that determines whether there is any delay in the operator's work based on the changes in the operating status of the production equipment and the changes in the operator's work status. With this structure, the judgment unit determines whether there is any delay in the operator's work, thus making it easier to evaluate the operator's work status.

[0019] In the technology disclosed in this specification, the component mounting system may further include a first designation device for selecting a production equipment from multiple production equipment as the object of analysis. The operation status acquisition unit can also acquire the operation status of the production equipment designated by the first designation device. With this configuration, it is possible to evaluate the working status of the operator relative to the designated production equipment.

[0020] In the technology disclosed in this specification, the component mounting system may further include a second designation device for selecting an operator from one or more operators as the object of analysis. The display unit may also visualize and display the work status of the operator designated by the second designation device. With this configuration, the work status of the designated operator can be evaluated.

[0021] In the image processing disclosed in this specification, when multiple production devices are captured near a worker at any time or during any period of time, the distance between the worker and each of the multiple production devices is calculated, and the worker is determined to be working on the production device with the smallest calculated distance. Based on this structure, even if multiple production devices appear in the captured image, the production device on which the worker is working can be identified.

[0022] In the image processing disclosed in this specification, it is possible to abandon the determination that a worker is working on the production equipment at the closest distance when the determination is made, if the time spent working on the production equipment is less than a predetermined time, and if the time spent working on the production equipment is more than the predetermined time, the determination that a worker is working on the production equipment is maintained. Based on this structure, even if a worker is captured simply passing in front of the production equipment, it is possible to avoid determining that the worker is working on the production equipment.

[0023] In the image processing disclosed in this specification, it is possible to determine the content of the work performed by the operator on a specific production equipment based on the operating status of the specific production equipment when it is determined that the operator is performing work on that equipment. According to this structure, since the operator's work content is determined considering the operating status of the production equipment, the operator's work status can be accurately evaluated.

[0024] (Example) Hereinafter, the component mounting system 10 of Example 1 will be described with reference to the accompanying drawings. Figure 1 As shown, the component mounting system 10 includes multiple production devices 12-28 installed on the component mounting line, a management device 30 for managing the multiple production devices 12-28, a camera (34a, 34b) installed on the component mounting line, and an image processing device 36 for processing images captured by the camera (34a, 34b).

[0025] Production equipment 12-28 constitutes a component mounting line for mounting components onto substrates. The component mounting line mounts components onto the substrates that are fed in, manufacturing substrates with mounted components. Hereinafter, the substrate after component mounting is sometimes referred to as a circuit board, and the substrate before and during component mounting is simply referred to as a substrate. In this embodiment, the component mounting line manufactures multiple types of circuit boards. Different types of circuit boards have different sizes, different solder patterns printed on them, and different types of components mounted on them. Therefore, when changing the type of circuit board produced in the component mounting line, changeover adjustments (preparation work) are required in each of production equipment 12-28. Furthermore, when starting circuit board production using the component mounting line, substrates, solder, components, etc., used in the production of the circuit boards need to be replenished. For this purpose, multiple operators 50a-50c (an example of "operators") are provided on the component mounting line. Figure 1 The installation line shown is equipped with three workers, 50a~50c.

[0026] The component mounting line includes a substrate loader 12, a printer 14, a printed circuit board (SPI) inspection machine 16, a first component mounter 18, a second component mounter 20, a first substrate appearance inspection machine (AOI) 22, a reflow oven 24, a second substrate appearance inspection machine (AOI) 26, and a substrate unloader 28. These production devices 12-28 can use known machinery used in known component mounting lines, and will therefore be described simply. The substrate loader 12 feeds substrates into the component mounting line. The substrate loader 12 holds multiple substrates and moves them one by one to the printer 14. The printer 14 prints solder patterns onto the substrates moved in from the substrate loader 12. The substrates with solder patterns are moved from the printer 14 to the printed circuit board (SPI) inspection machine 16. The SPI inspection machine 16 checks whether the solder patterns printed on the substrates are normal. If an abnormality occurs in the printed solder pattern (e.g., a printing defect caused by mask blockage), the substrate is discarded. On the other hand, if the printed solder pattern is normal, the substrate is moved from the printing inspection machine 16 to the first component mounting machine 18. The first component mounting machine 18 mounts a predetermined number of components onto the substrate moved from the printing inspection machine 16. Specifically, the first component mounting machine 18 mounts multiple component feeders in a detachable manner and mounts the components supplied from these component feeders onto the substrate. The substrate with components mounted by the first component mounting machine 18 is moved to the second component mounting machine 20. The second component mounting machine 20, like the first component mounting machine 18, mounts the predetermined number of components onto the substrate moved from the first component mounting machine 18. The substrate with components mounted is moved to the first substrate appearance inspection machine 22. The first substrate appearance inspection machine 22 checks whether the components are properly mounted on the substrate. If the components are not properly mounted on the substrate (e.g., the components are mounted in different places), the substrate is discarded. On the other hand, if the components are properly mounted on the substrate, the substrate is moved from the first substrate appearance inspection machine 22 to the reflow oven 24. The reflow oven 24 heats the incoming substrate to melt the solder, soldering components onto the substrate. The substrate removed from the reflow oven 24 is then fed into the second substrate inspection machine 26. The second substrate inspection machine 26 checks whether the substrate is soldered in the correct position. If components are not soldered in the correct position on the substrate (e.g., if components shifted due to some reason during heating in the reflow oven 24), the substrate is discarded. On the other hand, if components are soldered in the correct position on the substrate, the substrate is removed from the second substrate inspection machine 22 to the substrate unloading machine 28. The substrate unloading machine 28 removes the substrate (i.e., the circuit board with components mounted) transported from the second substrate inspection machine 22 from the component mounting line.

[0027] It should be noted that each production device 12-28 is equipped with communication circuits 12a-28a (an example of a "status output unit"). Communication circuits 12a-28a are connected to the management device 30 in a communicative manner. Each communication circuit 12a-28a outputs status information indicating the status of the production device 12-28 equipped with that communication circuit to the management device 30. For example, the substrate loader 12 outputs the number of substrates it has collected to the management device 30. Thus, the management device 30 can determine whether substrates need to be replenished to the substrate loader 12. Furthermore, for example, the first component mounting machine 18 and the second component mounting machine 20 output the number of components used according to each type of component to the management device 30. Thus, the management device 30 can determine whether components need to be replenished to the first component mounting machine 18 and the second component mounting machine 20.

[0028] Furthermore, the component feeders installed in the first component mounting machine 18 and the second component mounting machine 20 are transported by the feeder rack 32. That is, before use, the component feeders are loaded onto the feeder rack 32 in a storage location (not shown) and transported to the first component mounting machine 18 and / or the second component mounting machine 20 via the feeder rack. After use, the component feeders in the first component mounting machine 18 and / or the second component mounting machine 20 are loaded into the feeder rack 32. The used component feeders loaded onto the feeder rack 32 are then transported to a storage location (not shown) for preparation for future use.

[0029] The management device 30 is a computer equipped with a CPU and memory. The management device 30 controls the production of the circuit board by controlling the operation of the production equipment 12-28. For example, the management device 30 sends a solder printing pattern to the printing inspection machine 16, which inspects the board based on the received printing pattern. Furthermore, for example, the management device 30 sends an installation procedure (installation work) to the first component mounting machine 18 and the second component mounting machine 20, specifying the type, sequence, and location of the components to be installed. The first component mounting machine 18 and the second component mounting machine 20 install components onto the board based on the received installation procedure.

[0030] Furthermore, the management device 30 determines the operating status of each production device based on the status information output from each production device 12-28. For example, when the first component mounting machine 18 and the second component mounting machine 20 need to replace their component feeders, they output this intention information to the management device 30. Based on the information output from the first component mounting machine 18 and the second component mounting machine 20, the management device 30 can determine that the first component mounting machine 18 and the second component mounting machine 20 need to replace their component feeders.

[0031] The imaging devices (34a, 34b (an example of an "imaging unit")) are installed in a factory where a component mounting line is provided. The imaging devices (34a, 34b) are capable of photographing multiple production machines 12-28 installed on the component mounting line, multiple workers 50a-50c working on the component mounting line, and the feeder frame 32 used on the component mounting line. Specifically, the imaging devices (34a, 34b) include a first camera 34a installed on the upstream side of the component mounting line (the side of the substrate loader 12) and a second camera 34b installed on the downstream side of the component mounting line (the side of the substrate unloader 28). The first camera 34a is configured such that its optical axis is tilted relative to the component mounting line (the direction in which the production machines 12-28 are located), enabling it to photograph the entire production machines 12-28 from the upstream side of the component mounting line. The second camera 34b is configured such that its optical axis is tilted relative to the component mounting line, enabling it to photograph the entire production machines 12-28 from the downstream side of the component mounting line. By capturing images of the component mounting line from different directions using the first camera 34a and the second camera 34b, the positions (x, y) of the workers 50a~50c can be determined, as described later. The first camera 34a and the second camera 34b are connected to the image processing device 36. The animation data captured by the first camera 34a and the second camera 34b is input to the image processing device 36.

[0032] The image processing device 36 is a computer equipped with a CPU and memory. By executing programs (including image processing programs) stored in its memory, the image processing device 36 functions as a "display unit" and a "operational status acquisition unit." The image processing device 36 is connected to the first camera 34a, the second camera 34b, and the management device 30. The image processing device 36 receives inputs of animations captured by the first camera 34a and the second camera 34b, and receives inputs of the operational status of production equipment 12-28 from the management device 30. Based on the input animations and the input operational status of production equipment 12-28, the image processing device 36 visualizes the operational status of operators 50a-50c. The processing performed in the image processing device 36 will be described in detail later.

[0033] It should be noted that the image processing apparatus 36 is also connected to a display device 38, an input device 40, and an image registration device 42. The display device 38 displays various information output from the image processing apparatus 36 (e.g., a timeline visualizing the operating status of production equipment 12-28 and the work status of operators 50a-50c). The display device 38 can use a known display. The input device 40 can specify (input) the conditions for image processing performed by the image processing apparatus 36, and the production equipment or operators as the objects of image processing. That is, the input device 40 is an example of a "first specifying device" and an example of a "second specifying device." The information input from the input device 40 is stored in the memory of the image processing apparatus 36. The input device 40 can use a known pointing device such as a keyboard or mouse. The image registration device 42 inputs (registers) images (reference images) of the production equipment 12-28 and operators 50a-50c as the objects of identification during the image processing performed by the image processing apparatus 36. The reference images input from the image registration device 42 are stored in the memory of the image processing apparatus 36. The image registration device 42 can use digital cameras, smartphones with digital cameras, tablet PCs with digital cameras, scanners, etc. The image registration device 42 is an example of an "image input unit".

[0034] Next, the processing performed by the image processing device 36 when visualizing the work status of workers 50a to 50c based on the animation captured by the imaging devices (34a, 34b) will be explained. Figure 2 As shown, the image processing device 36 first registers (inputs) reference images of the production equipment 12-28 and the feeder frame 32, which are the objects of image processing (S10). In this embodiment, reference images of all the production equipment 12-28 constituting the component mounting line and the reference image of the feeder frame 32 used in the component mounting line are input. These reference images are input using the image registration device 42.

[0035] Next, the image processing device 36 registers (inputs) reference images of the workers 50a-50c, which are the objects of image processing (S12). In this embodiment, reference images of all workers 50a-50c equipped on the component mounting line are input. These reference images are input using the image registration device 42, just as in S10.

[0036] Next, the image processing unit 36 ​​acquires the animation data captured by the imaging devices (34a, 34b) (S14). Specifically, the imaging devices (34a, 34b) capture images of the production equipment 12-28, the feeder rack 32, and the operators 50a-50c during a predetermined production period for manufacturing circuit boards using the component mounting line. The animation data captured by the imaging devices (34a, 34b) is stored in the memory of the imaging devices (34a, 34b). In S14, the animation data stored in the memory of the imaging devices (34a, 34b) is input to the image processing unit 36. It should be noted that the period for capturing images of the component mounting line using the imaging devices (34a, 34b) can be arbitrarily set by the user. For example, the working hours of the factory with the component mounting line (e.g., 9:00 AM to 5:00 PM) can be set as the shooting period. If the factory with the component mounting line operates on a 24-hour system, the shooting period can be set as a day starting from the time the operators change shifts.

[0037] Next, the image processing device 36 acquires the operating status of production equipment 12-28 during the period when the animation was captured by the shooting devices (34a, 34b) (S16). That is, production equipment 12-28 outputs status information indicating its own status to management device 30 via communication circuits 12a-28a. Therefore, management device 30 obtains the operating information of production equipment 12-28 (e.g., normal operation, operation stopped (including error codes), maintenance operation, etc.) based on the status information output from production equipment 12-28, and stores the operating information in its memory. In S16, the operating information of production equipment 12-28 stored in the memory of management device 30 is input to image processing device 36.

[0038] Next, the image processing device 36 performs image processing on the animation data input in S14 to create a timeline (S18) that visualizes the work status of workers 50a to 50c. Regarding the image processing in S18, see [link to relevant documentation]. Figure 3 Let me explain in detail.

[0039] like Figure 3As shown, firstly, the image processing device 36 selects an image data (an animation data) as the object of image processing (analysis object) from the animation data input in S14 (S22). That is, the shooting devices (34a, 34b) shoot the animation data at a predetermined time interval (predetermined frame rate). Therefore, the animation data input in S14 consists of multiple image data captured at predetermined time intervals. In S22, one image data from the multiple image data contained in the animation data is selected as the object of image processing. For example, all the image data contained in the shot animation data can also be selected for image processing. In this case, all the image data is processed, so the working status of the workers 50a to 50c can be visualized with high precision. On the other hand, image processing can also be performed only on the image data selected from the image data contained in the shot animation data at a predetermined time interval (i.e., a time interval longer than the shooting time interval of the shooting devices (34a, 34b)). In this case, the number of image data to be processed is reduced, so the time required for image processing can be shortened.

[0040] Next, the image processing device 36 automatically identifies whether the production equipment 12-18, etc., which are the objects of identification, appear in the image data selected in S22 (S24). In this embodiment, all of the production equipment 12-28 constituting the component assembly line, the feeder frame 32 used in the component assembly line, and all the workers 50a-50c assigned to the component assembly line are the objects of identification. The first camera 34a and the second camera 34b of the imaging device (34a, 34b) are configured to capture all of the production equipment 12-28. Therefore, all of the production equipment 12-28 appear in the animation data captured by the first camera 34a and the second camera 34b. On the other hand, the feeder frame 32 is sometimes located outside the shooting range of the imaging device (34a, 34b), and the workers 50a-50c are sometimes located outside the shooting range of the imaging device (34a, 34b). Therefore, in S24, when production equipment 12-28 appearing in the image data selected in S22, and feeder frames 32 and operators 50a-50c are within the shooting range of the shooting device (34a, 34b), these feeder frames 32 and operators 50a-50c will be identified. It should be noted that the method for automatically identifying whether the identified object appears in the image data can use a known automatic identification program.

[0041] Next, the image processing device 36 determines whether any of the operators 50a to 50c has been identified in S24 (S26). If the image processing device 36 has not identified any of the operators 50a to 50c (i.e., there are no operators 50a to 50c on the component mounting line), S26 is negative, and the process proceeds to S36.

[0042] If the image processing device 36 identifies any one of the workers 50a to 50c (i.e., if any worker 50a to 50c is present on the component mounting line (S26 is true)), it proceeds to S28 and selects one of the identified workers. For example, if three workers 50a to 50c are identified, one worker is selected from among them as the object of analysis.

[0043] Next, the image processing device 36 calculates the distance between the selected worker and each production device 12-28 in S28 (S30). (Refer to...) Figure 4 This explains the steps for calculating the distance between workers and production equipment. Figure 4 The diagram shows operator 50a and production equipment 22 and 24. (For example...) Figure 4As shown, the position (x1, y1) of the first camera 34a and the position (x2, y2) of the second camera 34b are known. The optical axis of the first camera 34a is fixed, and the optical axis of the second camera 34b is also fixed. Therefore, the direction (angle θ1) of the worker 50a relative to the first camera 34a can be calculated based on the position of the worker 50a in the image captured by the first camera 34a. Similarly, the direction (angle θ2) of the worker 50a relative to the second camera 34b can be calculated based on the position of the worker 50a in the image captured by the second camera 34b. Knowing the positions (x1, y1) of the first camera 34a and (x2, y2) of the second camera 34b, and calculating the aforementioned angles θ1 and θ2, the position (x3, y3) of the worker 50a can be calculated. Here, the positions (x4, y4) of the center of the front of production equipment 22 and the positions (x5, y5) of the center of the front of production equipment 24 are also known, so the distance l1 between operator 50a and production equipment 22 and the distance l2 between operator 50a and production equipment 24 can be calculated. In S30, the distances between the operators selected in S28 and each production equipment 12-28 are calculated according to the above steps. As can be clearly seen from the above explanation, the distances calculated in S30 refer to the distances when viewed from above from production equipment 12-28 and operators 50a-50c. It should be noted that when the feeder frame 32 is identified in S24, the distance between the operator and the feeder frame 32 is also calculated. The position of the feeder frame 32 can be calculated in the same way as the operator, so the distance between the feeder frame 32 and the operator can also be calculated.

[0044] In S30, the image processing device 36 establishes a correspondence between the operator selected in S28 and the nearest production equipment (which may be the feeder frame 32, depending on the situation) (S32). That is, since the distance between the operator and each production equipment 12-28 is calculated in S30, the production equipment with the smallest calculated distance is matched with the operator selected in S28. For example, Figure 4 The situation shown establishes a correspondence between the operator 50a and the production equipment (first substrate appearance inspection machine) 22.

[0045] Next, the image processing device 36 determines whether the processing steps S28 to S32 (S34) have been performed on all the workers identified in S24. If the processing has not been performed on all the workers identified in S24 (no in S34), it returns to S28 and performs the processing starting from S28. Thus, the distance between all the workers identified in S24 and the production equipment 12 to 28 is calculated, and each worker is associated with the production equipment with the shortest distance.

[0046] If all the workers identified in S24 have been processed (yes in S34), proceed to S36. In S36, the image processing device 36 determines whether all image data that is the object of image processing in the animation data input in S14 has been processed. If not all image data has been processed (no in S36), the image processing device 36 returns to S22 and repeats the processing that started from S22. Thus, all image data that is the object of image processing in the animation data input in S14 has been processed.

[0047] After processing all the image data (as in S36), the image processing device 36 selects one operator as the object of analysis (S38). In this embodiment, three operators 50a to 50c, each assigned to the component mounting line, become the objects of analysis. Therefore, in S38, one operator (e.g., operator 50a) is selected from the three operators 50a to 50c.

[0048] Next, the image processing device 36 determines whether the worker selected in S38 has continuously corresponded with the same production equipment 12-28, 32 for a predetermined period of time or more during the period when the animation was captured by the shooting devices (34a, 34b) (S40). That is, through the processing of S22-S36 described above, all image data of the animation data input in S14 that are the objects of analysis are processed, and at the moment when each image data is captured, the workers 50a-50c are associated with any of the production equipment 12-28, 32. (However, if the worker does not appear in the image data (i.e., if the worker is not identified), the worker is not associated with any of the production equipment 12-28, 32 at that moment.) When a worker is associated with a production equipment, it can be determined that the worker is near the production equipment and is performing some work on the production equipment. However, sometimes, although the worker is near a certain production equipment at a certain moment, he / she only passes in front of the production equipment and does not perform any work on the production equipment. Therefore, in this embodiment, if an operator continuously interacts with the same production equipment for a predetermined time (e.g., 20 seconds) or more, it is determined that the operator is performing work on the production equipment. Therefore, in S40, it is determined whether there is a period during which an operator continuously interacts with the same production equipment 12-28, 32 for a predetermined time or more.

[0049] It should be noted that the determination in S40 can be based on the time interval of the image data selected in S22. For example, if the time interval of the image data selected in S22 is 5 seconds and the predetermined time used in the determination in S40 is 20 seconds, when the operator establishes a correspondence with the same production equipment for 5 consecutive image data points, it can be determined that the operator has continuously corresponded with the same production equipment for a period of more than the predetermined time. It should also be noted that, from the perspective of reducing the processing load of image processing, the time interval of the image data selected in S22 can also be consistent with the predetermined time used in the determination in S40. In this case, when the operator establishes a correspondence with the same production equipment for 2 consecutive image data points, it can be determined that the operator has continuously corresponded with the same production equipment for a period of more than the predetermined time.

[0050] If there is no period in which an operator has continuously worked with the same production equipment for a predetermined time or longer (no in S40), it is considered that the operator has not performed any work on any of the production equipment 12-28, 32, and the result of the operator-production equipment pairing is reset (S44). On the other hand, if there is a period in which an operator has continuously worked with the same production equipment for a predetermined time or longer (yes in S40), it is considered that the operator was working on the production equipment during that period, and the result of the operator-production equipment pairing is maintained (S42). It should be noted that for periods other than the period in which an operator has continuously worked with the same production equipment for a predetermined time or longer, it is considered that the operator has not performed any work on any of the production equipment 12-28, 32, and the result of the operator-production equipment pairing is reset.

[0051] Next, the image processing device 36 determines whether processes S40 to S44 (S46) have been performed on all the workers being analyzed. If no processes have been performed on all the workers being analyzed (no in S46), the process returns to S38 and executes the process starting from S38. Thus, processes S40 to S44 are performed on all the workers being analyzed to determine which production equipment each worker worked on.

[0052] After S46 ends, return to Figure 2In step S20, the image processing device 36 displays a timeline on the display device 38 corresponding to the workers 50a-50c, which are the subjects of analysis, and each production device 12-28, 32. That is, the display device 38 displays a timeline visualizing the work status of which production device the workers 50a-50c are working on at each moment during the period when animation was captured by the imaging devices (34a, 34b) (the production period). It should be noted that the image processing device 36 obtains the operating status of each production device 12-28 in step S16, so the operating status of each production device 12-28 can also be displayed in the timeline displayed in step S20. By displaying the operating status of each production device 12-28 together, the work performed by the workers 50a-50c on each production device 12-28 can be estimated. For example, if the first component installation machine 18 issues a warning that the component is exhausted when the operator is working on it, it can be inferred that the operator is replenishing the component.

[0053] Here, refer to Figure 5 , 6 Here is an example of the time map displayed on the display device 38 in S20.

[0054] Figure 5 This is a time-sharing diagram showing the operational status of workers 50a-50c on production equipment 12-28 and 32 from 9:50 to 11:05. It should be noted that the operational status of worker 50a is displayed as "Worker 1" in the diagram, the operational status of worker 50b is displayed as "Worker 2", and the operational status of worker 50c is displayed as "Worker 3".

[0055] exist Figure 5 In this embodiment, the work status of operators 50a to 50c is displayed every minute, showing which production equipment they worked on. For example, the 9:50 column shows which production equipment an operator worked on during the period from 9:50:00 to 9:51:00. In this embodiment, if an operator establishes a connection with production equipment 12 to 28 and 32 for more than 20 consecutive seconds, it is determined that the operator is working on that production equipment. Therefore, it is possible for multiple production equipment to be worked on within one minute. For example, operator 50b (operator 2) worked on both the first component mounting machine 18 and the second component mounting machine 20 at 10:08.

[0056] Furthermore, in Figure 5The operating status of production equipment 12-28 is also shown in the diagram. Specifically, at 9:51, the printing press 14 outputs a changeover adjustment operation guide. Similarly, at 9:52, the printing inspection machine (SPI) 16 outputs a changeover adjustment operation guide (alarm); at 9:53, the first component mounting machine 18 and the second component mounting machine 20 output a changeover adjustment operation guide (alarm); at 9:54, the first substrate appearance inspection machine (AOI) 22 outputs a changeover adjustment operation guide (alarm); at 9:55, the reflow oven 24 and the second substrate appearance inspection machine (AOI) 26 output a changeover adjustment operation guide (alarm); and at 9:56, the substrate unloading machine 28 outputs a changeover adjustment operation guide (alarm). Therefore, Figure 5 The timeline serves as a guide for outputting changeover adjustment operations to production equipment 12-28, and according to this guide, operators 50a-50c perform changeover adjustment operations on each production equipment 12-28.

[0057] Depend on Figure 5 The timeline shown indicates the need for specific countermeasures to improve the working conditions of workers in periods 50a-50c. For example, ... Figure 5 As shown in A, operator 50a (operator 1) needs a long time to adjust the printing press 14 during the changeover process. Therefore, it can be seen that in order to shorten the changeover time of operator 50a on the printing press 14, countermeasures such as preparing solder in advance and preparing replacement masks in advance are needed.

[0058] And, as Figure 5 As shown in B, in the first component mounting machine 18 and the second component mounting machine 20, a long time is required from the start of the output changeover adjustment operation until the operator 50b (operator 2) actually begins the operation. Therefore, it can be seen that in order to shorten the time until the operator 50b starts the operation, countermeasures such as preparing the feeder frame 32 in advance are needed.

[0059] And, as Figure 5 As shown in Figure C, worker 50b (worker 2) has a high workload during this period, working on multiple production machines 18-28 and 32. On the other hand, worker 50c (worker 3) does not work during this period. Therefore, it can be seen that in order to reduce the workload of worker 50b, other workers 50c need to share the workload.

[0060] And, as Figure 5As shown in diagrams D and E, from 10:44 to 10:54, operators 50a (operator 1) and 50b (operator 2) jointly operate the first component mounting machine 18 and the second component mounting machine 20. Therefore, operator 50a's operation on the printing press 14 is postponed until 10:57. Thus, it can be seen that by changing the operation to operators 50b and 50c (operator 3) jointly operating the first component mounting machine 18 and the second component mounting machine 20, operator 50a's operation on the printing press 14 can begin quickly.

[0061] Figure 6 This is a timeline showing the operational status of workers 50a-50c on production equipment 12-28 and 32 from 13:15 to 13:54. Figure 6 The display also shows the operating status of production equipment 12-28 (specifically, based on displays from signal towers and monitors). Figure 6 What is clear is that in Figure 6 The timeline shows the operational status of workers 50a to 50c, which are generated from the production equipment and displayed on signal towers or monitors.

[0062] according to Figure 6 The timeline shown can also be used to derive countermeasures for improving the working conditions of workers in periods 50a-50c. For example, such as... Figure 6 As shown in Figure F, at time 13:33, both printing press 14 and the first component mounting machine 18 generate alarms simultaneously, thus demonstrating a delay in the work of operator 50a (operator 1) opposite to the first component mounting machine 18. Therefore, it can be seen that when alarms occur simultaneously in multiple production devices, multiple operators are required to handle the situation. In the example above, it would be sufficient to change the operation of the first component mounting machine 18 to operators 50b and 50c (operators 2 and 3).

[0063] And, as Figure 6 As shown in G, approximately two minutes of operation time elapses from the alarm generated by the Printing Inspection Machine (SPI) 16 until operator 50c (operator 3) completes the task. Therefore, the possibility that operator 50c may be unfamiliar with handling the Printing Inspection Machine 16 can be considered. Consequently, as a countermeasure, measures such as standardizing (manualizing) the operation of the Printing Inspection Machine 16 or providing training to operator 50c could be considered.

[0064] As explained above, in the component mounting system 10 of this embodiment, multiple production devices 12-28, 32 and multiple operators 50a-50c equipped on the component mounting line are photographed. The working status of operators 50a-50c is visualized by performing image processing on the photographed animation. Therefore, it is possible to evaluate whether the work of operators 50a-50c is appropriate relative to the operation of the multiple production devices 12-28, 32 provided in the component mounting system. Furthermore, if the work of operators 50a-50c is inappropriate, the cause can be easily deduced, and as a result, improvement measures can be discovered.

[0065] The above provides a detailed description of specific examples of the technology disclosed in this specification. However, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes technical solutions resulting from various modifications and alterations to the specific examples illustrated above.

[0066] For example, in the above embodiment, each production device 12-28 outputs status information indicating its own status, and the management device 30 obtains the operating status of each production device 12-28 from the output status information and inputs the obtained operating status to the image processing device 36. However, the technology disclosed in this specification is not limited to this method. For example, the image processing device 36 can also obtain the operating status of each production device by performing image processing on animation data captured by the imaging device. That is, the operating status of each production device can also be obtained by performing image processing on the animation data and following the guidance of the signal tower or monitor installed in each production device. In this case, it is not necessary to obtain the operating status from the management device 30, thus simplifying the system results.

[0067] Furthermore, in the above embodiments, the manufacturer producing circuit boards using component mounting lines and the person using the image processing device 36 to visualize the workers' work status are the same person; however, the technology disclosed in this specification is not limited to this method. For example, the manufacturer producing circuit boards using component mounting lines and the person using the image processing device 36 to visualize the workers' work status can be different people. In this case, the manufacturer producing circuit boards using component mounting lines may, for example, only register reference images of the production equipment or workers as the objects of image processing and input animation data obtained by photographing the component mounting lines. The registration of reference images and the input of animation can be performed by the manufacturer from their own PC or tablet PC. On the other hand, the person using the image processing device 36 to visualize the workers' work status analyzes the registered reference images and the input animation data, and outputs a timeline that visualizes the workers' work status to the manufacturer. In this way, the user of the image processing device 36 can remotely support the manufacturer producing circuit boards using component mounting lines.

[0068] Furthermore, in the above embodiments, only whether operators 50a-50c performed work on production equipment 12-28, 32 was determined; however, the technology disclosed in this specification is not limited to this method. For example, the image processing apparatus may also be configured to estimate (determine) both the operating status of the production equipment operated by the operator and the work content. By estimating the work content of the operator even through the image processing apparatus, the user can easily evaluate the operator's work status. Moreover, the image processing apparatus may also be configured to determine whether there is any delay in the operator's work based on the changes in the operating status of the production equipment and the changes in the operator's work status. With this structure, it is easy to grasp the delay in the operator's work, which can improve the convenience for manufacturers producing circuit boards using component mounting lines.

[0069] Furthermore, in the above embodiments, operators 50a to 50c are always associated with the production equipment at the nearest distance; however, the technology disclosed in this specification is not limited to this method. For example, if the distance between the operator and the production equipment at the nearest distance is longer than a predetermined distance, it is possible not to associate the operator with that production equipment. That is, the operator is not associated with any production equipment. This is because if the distance from the operator to the production equipment exceeds a predetermined distance, it is difficult to consider that the operator should perform operations on that production equipment.

[0070] Furthermore, in the above embodiments, all production equipment and all personnel working on the component assembly line are considered as the objects of analysis; however, the technology disclosed in this specification is not limited to this approach. For example, the working conditions of specific personnel working on specific production equipment may also be considered as the objects of analysis.

[0071] Furthermore, in the above embodiment, the positions of workers 50a-50c are calculated by taking pictures of them with two cameras 34a and 34b. However, the technology disclosed in this specification is not limited to this method. For example, a single camera and a distance sensor (laser sensor) that measures the distance to the workers can be used, and the positions of the workers can be calculated using the camera images and the distance measured by the distance sensor.

[0072] It should be noted that in the above embodiments, images captured by a camera were used to calculate the location of the workers. However, the location of the workers can also be obtained by having them carry GPS sensors. In this case, it is not necessarily necessary to photograph the production equipment installed on the component assembly line; the workers' working status can be inferred solely from their location.

[0073] Furthermore, the technical elements described in this specification or drawings, individually or in various combinations, contribute to the technical utility and are not limited to the combinations recited in the claims at the time of application. Moreover, the technologies illustrated in this specification or drawings simultaneously achieve multiple objectives, and achieving one of these objectives is itself technically practical.

Claims

1. A component mounting system, comprising: Multiple production equipment are installed on a component mounting line that mounts components onto a substrate; A camera unit, located on the component assembly line, is used to photograph the plurality of production devices and one or more workers operating on the component assembly line; and The display unit processes images captured by the imaging unit during a predetermined production period using the component assembly line to visualize and display the operational status of at least one of the multiple production devices for at least one of the one or more operators during various predetermined times or periods within the production period. The component mounting system further includes an operation status acquisition unit, which acquires the operation status of at least one of the plurality of production devices at each of the plurality of times or during each of the plurality of periods. The display unit corresponds and displays the obtained operating status of the production equipment with the work status. The component installation system also includes a judgment unit, which determines whether there is any delay in the operator's work based on the changes in the operating status of the production equipment and the working status of the operator. When at least two of the multiple production devices generate alarms simultaneously, the judgment unit determines that the operator's work is delayed.

2. The component mounting system according to claim 1, wherein, The operation status acquisition unit acquires the operation status through the image processing.

3. The component mounting system according to claim 1, wherein, At least one of the plurality of production devices has a status output unit that outputs its own status. The operation status acquisition unit acquires the operation status based on the status output from the status output unit.

4. The component mounting system according to any one of claims 1 to 3, wherein, The component mounting system also includes a first designation device for selecting the production equipment from the plurality of production equipment as the object of analysis. The operation status acquisition unit acquires the operation status of the production equipment designated by the first designation device.

5. The component mounting system according to any one of claims 1 to 3, wherein, The component mounting system also includes a second designation device for selecting the operator from among the one or more operators as the object of analysis. The display unit visualizes and displays the work status of the operator designated by the second designation device.

6. The component mounting system according to any one of claims 1 to 3, wherein, In the image processing, At any one of the plurality of moments or during any one of the plurality of periods, when multiple production devices are photographed near the operator, the distance between the operator and each of the plurality of production devices is calculated. It is determined that the operator is working on the production equipment with the smallest distance among the calculated distances.

7. The component mounting system according to claim 6, wherein, In the image processing, if it is determined that the operator is working on the production equipment with the smallest distance, If the time spent operating the production equipment is less than the predetermined time, the determination that the operator was operating the production equipment is invalidated. If the time spent working on the production equipment exceeds the predetermined time, the determination that the operator is working on the production equipment is maintained.

8. The component mounting system according to any one of claims 1 to 3, wherein, In the image processing, when it is determined that the operator is performing work on a specific production equipment, the content of the work performed by the operator on the specific production equipment is determined based on the operating status of the specific production equipment.

9. An image processing apparatus comprising: The image input unit inputs images captured during a predetermined production period of a component mounting line that mounts components onto a substrate, observing multiple production devices installed on the component mounting line and one or more operators working on the component mounting line; and The image processing unit processes the image input from the image input unit to visualize the operational status of at least one of the multiple production devices for at least one of the one or more operators during the production period, at each of a plurality of pre-set times or periods. The image processing unit acquires the operating status of at least one of the multiple production devices at each of the multiple times or during each of the multiple periods, and matches the acquired operating status of the production device with the work status and displays it. The image processing unit determines whether there is any delay in the operator's work based on the changes in the operating status of the production equipment and the working status of the operator. When at least two of the multiple production devices generate alarms simultaneously, the image processing unit determines that the operator's work is delayed.

10. An image processing method, comprising: The image capture step involves taking pictures of multiple production devices set up on the component mounting line and one or more operators on the component mounting line during a predetermined production period when products are manufactured using a component mounting line that mounts components onto a substrate. as well as The image processing step involves processing the images captured in the image acquisition step to visualize the operational status of at least one of the multiple production devices for at least one of the one or more operators during the production period, at various times or periods pre-set by the operator. In the image processing step, the operating status of at least one of the multiple production devices is obtained at each of the multiple time points or during each of the multiple periods, and the obtained operating status of the production device is correlated with the work status and displayed. In the image processing step, the operator's work is judged to be delayed based on the progress of the operation status of the production equipment and the progress of the operator's work status. If at least two of the multiple production equipment generate alarms at the same time, it is judged that the operator's work is delayed.

11. A computer program product storing a program for causing a computer to execute: The image capture step involves taking pictures of multiple production devices installed on the component mounting line and one or more operators working on the component mounting line during a predetermined production period using a component mounting line that mounts components onto a substrate; and The image processing step involves processing the images captured in the image acquisition step to visualize the operational status of at least one of the multiple production devices for at least one of the one or more operators during the production period, at various times or periods pre-set by the operator. In the image processing step, the operating status of at least one of the multiple production devices is obtained at each of the multiple time points or during each of the multiple periods, and the obtained operating status of the production device is correlated with the work status and displayed. In the image processing step, the operator's work is judged to be delayed based on the progress of the operation status of the production equipment and the progress of the operator's work status. If at least two of the multiple production equipment generate alarms at the same time, it is judged that the operator's work is delayed.

12. An image processing system, comprising: The input device inputs images captured during a predetermined production period of a component mounting line that mounts components onto a substrate, observing multiple production devices installed on the component mounting line and one or more operators on the component mounting line; and An image processing device, communicatively connected to the input device, performs image processing on images input from the input device to visualize the operational status of at least one of the multiple production devices for at least one of the one or more operators during the production period, at various times or periods pre-set within the production period. The image processing device acquires the operating status of at least one of the multiple production devices at each of the multiple moments or during each of the multiple periods, and displays the acquired operating status of the production device in relation to the work status. The image processing device determines whether there is any delay in the operator's work based on the changes in the operating status of the production equipment and the working status of the operator. When at least two of the multiple production devices generate alarms simultaneously, the image processing device determines that the operator's work is delayed.

Images