Detection method, detection processing device, and program

Abstract

The present invention provides a detection method, detection processing device, and program that can more appropriately identify foundation members without being affected by various conditions. [Solution] (1) Obtain multiple reference images of base members with different component mounting states. (2) For each of the multiple reference images, set multiple reference feature ranges that each include multiple feature points on the base member. (3) Detect feature ranges similar to each of the multiple reference feature ranges from the captured image of the base member to be detected. For each of the multiple reference feature ranges, associate the position of the feature point in the reference image selected according to the degree of similarity between the reference feature range and the feature range in the multiple reference images with the position of the feature point in the captured image of the feature range. (4) Detect the position or orientation of the base member in the captured image based on the positional relationship of the feature points associated with each of the multiple reference feature ranges.

Description

Technical Field

[0006] ,

[0001] This invention relates to a detection method, a detection processing device, and a program.

Background Art

[0002] There are many products obtained by mounting various components on a base member such as a substrate. In order to assist in mounting the components on such a base member or to perform it mechanically as much as possible, it is necessary to accurately grasp the orientation of the base member and the mounting positions on the base member. Patent Document 1 discloses a technique for detecting the position and orientation of a printed circuit board and projecting an image onto the mounting position of the mounted component according to the detection result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] According to the present invention, it is possible to more appropriately identify the foundation members without being affected by various conditions. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows the configuration of the implementation support system of this embodiment. [Figure 2] This is a block diagram showing the functional configuration of the control device. [Figure 3] This diagram schematically shows an example of an image of a substrate that is the target of position and orientation detection. [Figure 4] This figure shows an example of a reference image in this embodiment. [Figure 5] This flowchart shows the control procedure for implementation support control processing. [Figure 6] This diagram illustrates another example related to setting a reference image. [Figure 7] This is a diagram illustrating the generation of a reference image. [Figure 8] This flowchart shows another example of implementation support control processing. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. As shown in the configuration diagram of Figure 1, the implementation support system 100 includes a control device 10, a shooting device 20, and a projection device 30.

[0010] The imaging device 20 is a device for imaging a substrate S, which is the base material of an object, placed at a predetermined position such as a workbench T. The imaging device 20 has a spatial resolution that allows for the identification of the mounting positions of mounted components on the substrate S with the required accuracy. The imaging device 20 may, for example, capture full-color images in visible light.

[0011] The projection device 30 emits light based on image data and projects an image onto the substrate S. The projection device 30 may also be capable of projecting a full-color image using RGB light emitted by an LD (Laser Diode) or an LED (Light Emitting Diode).

[0012] The imaging device 20 and projection device 30 are capable of imaging and projecting from above the substrate S. However, imaging and projection may also be possible from an oblique direction to prevent workers performing mounting work on the substrate S from getting in between the device and the substrate S. Furthermore, the imaging range of the imaging device 20 and the projection range of the projection device 30 are predetermined and fixed. The correspondence between the imaging range and the projection range is predetermined and stored in the control device 10.

[0013] The control device 10 analyzes the image captured by the imaging device 20 of the substrate S during the mounting process. The control device 10, which is a detection processing device in this embodiment, identifies the position and orientation of the substrate S based on the captured image. The control device 10 may also identify the position of the next component to be mounted on the identified substrate S and output control information to the projection device 30 for outputting an image to that position.

[0014] As shown in the block diagram of FIG. 2, the control device 10 includes a CPU 11 (Central Processing Unit), a RAM 12 (Random Access Memory), a storage unit 13, a communication unit 14, and the like. The control device 10 may be an electronic computer such as an ordinary PC (Personal Computer).

[0015] The CPU 11 is a processor that performs arithmetic processing and overall controls the entire operation of the control device 10. The CPU 11 is included in the computer of the present embodiment. The CPU 1 eleven may be a general-purpose processor, or may include a dedicated processor suitable for the processing of the present embodiment. The processor may be single, or a plurality of processors may operate in parallel or independently according to their uses.

[0016] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data. The RAM 12 may be a DRAM, or may be another memory capable of operating equivalently to an SRAM or a volatile memory.

[0017] The storage unit 13 is a non-volatile memory that stores a program 131, setting data, and the like. The non-volatile memory may be, for example, a flash memory. Alternatively, an HDD (Hard Disk Drive) may also be included in the non-volatile memory of the present embodiment. The program 131 may include a recognition program for a captured image related to mounting support of components on the substrate S, a mounting support program for specifying the mounting position of the next component according to the position and orientation of the substrate S obtained by the recognition program, and the like. The setting data includes feature point data 132, reference image data 133, reference relative position information 134, and the like.

[0018] The feature point data 132 is information on a characteristic range to be detected from a captured image. A plurality of characteristic ranges are set, and each may be shown as a pixel range in a first reference image I1 described later. The pixel range is a range in which a structure or pattern can be specified with a certain degree of accuracy at the time of image detection, and may be determined according to the individual sizes of the mounted components, etc. The pixel ranges may have different sizes according to the individual characteristic structures. Representative feature points may be determined for each pixel range. The feature points may be the geometric centroid positions of each pixel range, or may be determined according to the structure related to the characteristic range.

[0019] The reference image data 133 includes an image serving as a reference for detection. The reference images include two images, a first reference image I1 and a second reference image I2. These reference images will be described later.

[0020] The reference relative position information 134 stores each element of a transformation matrix indicating the amount of misalignment between the first reference image I1 and the second reference image I2, etc. Further, the reference relative position information 134 includes each element of a transformation matrix indicating the correspondence between each pixel position on the first reference image I1 and the pixel position on the projection device 30 that irradiates light, etc.

[0021] The communication unit 14 controls communication between the imaging device 20 and the projection device 30. The communication unit 14 may communicate with the imaging device 20 and the projection device 30 via a network such as a LAN (Local Area Network). Alternatively, the communication unit 14 may control direct communication using USB (Universal Serial Bus), etc. Further, the communication unit 14 may be able to control wireless communication using a wireless LAN, etc.

[0022] The control device 10 may also have a display unit, an operation reception unit, etc. The display unit may have, for example, a liquid crystal screen or an organic EL (Electro-Luminescent) screen as a digital display screen. The operation reception unit may have a keyboard, a pointing device, for example, a mouse or a touch panel, and various operation switches, etc.

[0023] Next, the detection operation of the substrate S in the mounting support system 100 will be described. This detection operation includes the detection method of this embodiment. In the mounting support system 100, the imaging device 20 photographs the substrate S on the workbench T, and the control device 10 acquires this image. The substrate S corresponds to the base member of this embodiment. The control device 10 recognizes the substrate S in the image and identifies the position and orientation of each of the many substrates S. The substrate S may be a normal rigid substrate, or any other type of substrate as long as it does not undergo significant deformation during component mounting. The components to be mounted may be, for example, electronic components connected to wiring on the substrate. The substrate S may experience some displacement in position and orientation when it is first placed on the workbench T before component mounting, and each time a component is mounted. The control device 10 identifies the position and orientation of each substrate S, including this displacement, from the image captured by the imaging device 20. In addition to the position and orientation within the imaging plane, the tilt in the direction perpendicular to the imaging plane may also be considered. In this case, since the image includes a portion of the substrate S that is not in contact with the workbench T, the imaging device 20 stereographs the substrate S from two or more different positions in order to determine its three-dimensional position. Alternatively, even if the tilt of the substrate S is not taken into consideration, its three-dimensional position may be determined by stereo imaging.

[0024] As shown in Figure 3, the mounting order of components mounted on a substrate S, which is the target of position and orientation detection, can be predetermined according to constraints in the manufacturing process. For example, as shown in Figure 3(a), component P1 may be mounted first, followed by component P2 as shown in Figure 3(b).

[0025] To determine the position of the substrate S in the captured image, characteristic structures on the substrate S are used. The position and orientation of the substrate S in the captured image are determined from the positions of multiple feature points, each representing a different characteristic structure. Furthermore, the spatial position and orientation of the substrate S are detected by comparing the positions of the identified feature points with the positions of multiple feature points in the reference image. The features can be predetermined based on the reference image, which is a captured image of the substrate S. The range related to the determined features is stored in the feature point data 132 of the storage unit 13 as a reference feature range that includes the feature points of the features. Features may include, for example, the peripheral shape of the substrate S, markings and wiring on the substrate S, etc. In addition, features may include the shape, color and positional relationship of the mounted components, etc. The features used in the captured image are limited to those on the substrate S, and features such as the workbench T that are reflected around the substrate S are not considered in principle. The area outside the substrate S in the first reference image I1 and the second reference image I2 may be masked with a single color such as white or black so that even if the feature range includes the area outside the substrate S, that area does not affect the matching during matching. The reference feature range may be defined in particular for each mounting range of the mounted components.

[0026] However, as components are sequentially mounted on the circuit board S, the captured image of the circuit board S also changes sequentially. Therefore, characteristic structures in the captured image of the circuit board S often appear and disappear depending on the mounting status. As a result, simply calculating the degree of agreement while shifting the relative position between the reference image and the captured image, as in the conventional method, and identifying the most matching relative positional relationship does not necessarily yield accurate results.

[0027] As shown in Figure 4, in this embodiment, a first reference image I1 and a second reference image I2 are prepared as multiple reference images, for example, two reference images, taken of substrates S with different component mounting states. These reference images can be captured and acquired in advance by the camera 20 before starting the mounting support process. The acquired multiple reference images are stored as reference image data 133. This reference image data 133 can be continuously used for multiple identical mounting operations. If the mounting operation is changed to a different substrate S or different mounted components, new first reference image I1 and second reference image I2 are prepared. The first reference image I1 shown in Figure 4(a) may be an image of the substrate S with no components mounted. The second reference image I2 shown in Figure 4(b) may be an image of the substrate S with all components mounted. Ultimately, in addition to the above-mentioned components P1 and P2, component P3 may also be mounted on the substrate S.

[0028] The captured images I3, which are acquired sequentially during implementation, show that the presence or absence of each component is related to the presence or absence of the characteristic structure described above. That is, within a certain range of the captured image I3, there is a portion that roughly matches either the first reference image I1 or the second reference image I2, depending on whether or not a component is implemented. When a reference feature range containing a characteristic structure is defined in a portion of the multiple reference images, namely the first reference image I1 and the second reference image I2, as described above, a reference feature range that does not have a characteristic structure is also set for the reference images other than the portion mentioned above, corresponding to the reference feature range in that portion of the reference image. The matching of the image content with the respective reference feature range is detected near the ranges corresponding to the multiple reference feature ranges in the captured image. A range of the same size as the reference feature range is set as a feature range for the captured image, and the reference image with the greatest degree of image similarity between this feature range and the reference feature range is selected. As described above, it is assumed that the degree of matching with one of the reference feature ranges will increase depending on whether or not a component is implemented. The difference between the reference point related to the reference image with a higher degree of similarity and the reference point related to the feature range of the captured image provides data for detecting the difference between the reference image and the captured image, i.e., the position and orientation of the substrate S in the captured image.

[0029] The similarity and degree of matching referred to here can be quantitatively evaluated numerically. For example, similarity can be measured using the difference in brightness values ​​of each pixel or the square of the difference, as well as detecting the edges of each image and using or combining the distance to the nearest edge position. In addition, various other indicators that represent the similarity of images can be used.

[0030] At this time, there may be some discrepancy between the position and orientation of the substrate S in the first reference image I1 and the position and orientation of the substrate S in the second reference image I2. Therefore, the relative positional relationship, represented by the amount of discrepancy between the coordinates of the substrate S in the first reference image I1 and the coordinates of the substrate S in the second reference image I2, is predetermined and stored in the storage unit 13 as reference relative position information 134. Using this transformation matrix, a correction process can be performed to unify to one of the reference images. Since the mounting state of both images is known, a transformation matrix representing the discrepancy may be defined using only common feature points that are independent of the mounting state. Furthermore, if it is difficult to obtain an accurate transformation matrix using only such feature points, the amount of discrepancy between the first reference image I1 and the second reference image I2 may be determined by the operator by superimposing the captured images on a display screen and visually inspecting them.

[0031] For example, consider a case where a feature range in captured image I3 better matches the reference feature range of the second reference image I2 than the reference feature range of the first reference image I1. By transforming the coordinates of the reference feature range in the second reference image I2 to the coordinates of the first reference image I1, the coordinates of the feature points in captured image I3 can be associated with the coordinates on the first reference image I1. If the content of the feature range in captured image I3 better matches the feature range of the first reference image I1 than the feature range of the second reference image I2, the coordinates of the feature points in the first reference image I1 are associated with the coordinates of the feature points in captured image I3. In this way, a correspondence between the coordinates of the feature points in the captured image and the coordinates relative to the first reference image I1 is obtained. Based on this correspondence, for example, the amount of translation and rotation of the substrate S can be determined by a transformation matrix from the first reference image I1 to captured image I3 that minimizes the error. The reference may be the second reference image I2 instead of the first reference image I1. A simple least squares method may be used to determine such a transformation matrix. Alternatively, Random Sample Consensus (RANSAC) or similar methods may be used.

[0032] Along with the coordinates of the substrate S in the captured image I3, the position of the next component to be mounted on the substrate S is identified. The mounting position of this component is predetermined as the coordinates of the first reference image I1. The predetermined mounting order and mounting position information may be associated with the first reference image I1 of the reference image data 133. Furthermore, the correspondence of coordinates on the projection device 30 when light is projected by the projection device 30 to each coordinate of the first reference image I1 is stored in the reference relative position information 134.

[0033] If the mounting order of multiple components on the substrate S is predetermined, it is easy to determine how many of the multiple components have already been mounted and the mounting position of the next component to be mounted. The number of components already mounted may be determined, for example, simply by the number of times the captured image I3 has been taken. Alternatively, the number of mounted components may be determined according to the number or area of ​​feature parts that have a high degree of agreement with the second reference image I2.

[0034] The control device 10 converts the mounting position of the identified next mounted component into coordinates related to the projection position by the light emitted from the projection device 30, and generates a control signal to display a specific image at those coordinates. When this control signal is transmitted to the projection device 30, the projection device 30 projects a specific image onto the mounting position of the next mounted component on the substrate S according to the control signal. The specific image only needs to allow the operator to easily recognize the next mounting position, and may also identify the type of component to be mounted at that position. As the specific image, for example, a marker indicating the next mounting position, such as a cross, arrow, or circle, may be projected. In addition, the specific image may include numbers or symbols that identify the component.

[0035] As shown in the flowchart of Figure 5, in the implementation support control process, the CPU 11 reads reference image data 133, which includes feature point data 132, a first reference image I1, and a second reference image I2, from the storage unit 13 (S1; acquisition means). The CPU 11 sends a control signal to the imaging device 20 to photograph the substrate S and acquires the captured image I3 relating to the substrate S from the imaging device 20 (S2).

[0036] The CPU 11 divides the first reference image I1 into feature ranges based on the feature point data 132 and identifies the range in the captured image I3 that best matches (S3). The CPU 11 divides the second reference image I2 into feature ranges based on the feature point data 132 and the reference relative position information 134 read out as needed. The CPU 11 identifies the range in the captured image I3 that best matches each of the feature ranges of the second reference image I2 (S4). Steps S3 and S4 correspond to the setting means in the program of this embodiment.

[0037] The CPU 11 compares the matching range in the first reference image I1 with the matching range in the second reference image I2 for each feature point and adopts the one with the higher degree of matching (S5). If the matching range overlaps with a range that has already been adopted, the other may be automatically adopted, or the feature range may be deemed undeterminable. On the other hand, if there is a slight overlap with adjacent feature ranges due to rotational shifts or other reasons, the range may be adopted as is. If the matching range in the second reference image I2 is adopted, the CPU 11 converts the adopted matching range to the coordinates of this feature range in the first reference image I1 (S6).

[0038] The CPU 11 determines a representative point for each of the adopted feature ranges. The CPU 11 identifies and temporarily stores the corresponding ranges in the captured image I3 between the adopted feature ranges and the feature ranges obtained from the first reference image I1 (S7; correspondence means).

[0039] The CPU 11 calculates a transformation matrix between the coordinates of the feature points in the first reference image I1 and the coordinates of the feature points in the captured image I3 so as to minimize the difference between the feature positions in the first reference image I1 and the captured image I3 obtained in step S6 (S8; detection means). The CPU 11 then identifies the position of the next component to be mounted on the captured image I3 and determines the relative positional shift and the amount of the shift between the first reference image I1 and the captured image I3.

[0040] The CPU 11 identifies the mounting position of the next component based on pre-held component mounting order information and, if necessary, the captured image I3 (S9). The CPU 11 converts the mounting position on the captured image I3 into coordinates for the projection position by the projection device 30 (S10). The CPU 11 causes the projection device 30 to project the content of a predetermined guide display onto the next mounting position on the substrate S (S11). This projection output may continue for a predetermined time. Alternatively, the CPU 11 may acquire captured images from the camera 20 at appropriate intervals and stop the projection after determining that the operator has correctly mounted or started mounting the component to the mounting position.

[0041] The CPU 11 determines whether the mounting of all components has been completed (S12). If it is determined that the mounting of components has not been completed (S12;N), the CPU 11 returns to process S2. If it is determined that the mounting of all components has been completed (S12;Y), the CPU 11 terminates the mounting support control process.

[0042] On a circuit board S, the components are not necessarily mounted in independent positions. Components may overlap, or, depending on the direction of the photograph, one component may be obscured by another. In such cases, a situation may arise where the resulting image does not match either the image without any components or the image with all components mounted.

[0043] In one embodiment, three or more captured images, including states that do not appear in the two captured images, may be used as reference images. As shown in Figure 6(a), when all mounted components are installed, components P21 and P31 overlap component P11, and part of other component P11 is hidden behind component P31. In such a case, for example, as shown in Figure 6(b), by adding a reference image I4 taken when only components P1 and P11 are installed, a range of features that match the captured image with greater accuracy can be obtained.

[0044] Furthermore, in cases where systematically adjacent components tend to overlap in the captured image, or where shadows overlap due to lighting direction even if they do not actually overlap, a reference image may be included that shows at least one of the components in the set closest to each component in terms of mounting position, and that at least one of the components in that set has not yet been mounted. If the components are arranged in a regular pattern, the reference image may include captured images in which adjacent components are not mounted consecutively. That is, in this reference image, one of the adjacent components is mounted and the other is not, or one is not mounted and the other is mounted.

[0045] In one embodiment, it may also be possible to pre-determine whether a component is mounted or not for each feature range corresponding to a component. As shown in Figure 7(a), an image of the feature range R3 where no component is mounted may be obtained from the first reference image I1, and an image of the feature range where a component is mounted may be obtained from the second reference image I2, and these images may be combined to generate a single reference image I12. Here, with component P3 not mounted in feature range R3, an image of the range other than component P3 is obtained from the second reference image I2. The obtained image is converted to the coordinates of the first reference image I1 and incorporated into the corresponding portion of the first reference image I1. This reference image may be compared with the captured image I3, which is the target for detecting the substrate S, to determine the amount of positional shift of the captured image I3. Note that if the second reference image I2 has a relative positional shift with respect to the first reference image I1, the obtained region will not coincide with the corresponding feature range in the first reference image I1. In this case, the remaining shifted region may remain as the first reference image I1. Alternatively, as shown in Figure 7(b), the remaining shifted area may be masked in the resulting reference image I12 as an area that does not correspond to either the feature range in the original first reference image I1 or the feature range in the added second reference image I2.

[0046] Once a single reference image I12 is obtained in this way, the amount of misalignment with the highest degree of agreement can be identified by comparing the entire reference image I12 with the entire captured image I3. In other words, alignment is not limited to the comparison of feature points. In this case, for example, the difference in brightness values ​​or the difference in the square of brightness values ​​between corresponding pixels may be calculated and added together while sequentially changing the rotation angle and the amount of translation to obtain a total difference value. The amount of misalignment in the captured image I3 may be identified by the rotation angle and translation amount that result in the smallest total difference value or the average difference value obtained by dividing the total difference value by the number of pixels to which the total difference values ​​are added.

[0047] In the flowchart of this embodiment shown in Figure 8, steps S21 to S25 are executed instead of steps S3 to S8 in the implementation support control process shown in Figure 5. Other steps are the same in both flowcharts, and the same reference numerals are used for identical steps, and detailed explanations are omitted.

[0048] After step S2, the CPU 11 identifies the mounted components (S21). The CPU 11 obtains the image portion of the feature range corresponding to the mounted components from the second reference image I2 (S22). The CPU 11 converts the obtained image portion of the second reference image I2 into coordinates of the first reference image I1 (S23). The CPU 11 inserts the image portion of the second reference image I2, whose coordinates have been converted, into the first reference image I1 to generate a reference image (S24).

[0049] The CPU 11 compares and matches the acquired captured image I3 with the reference image while moving it relative to it, and identifies the amount of shift that minimizes the error between the two images (S25). Then, the CPU 11 proceeds to process S9.

[0050] Furthermore, depending on the substrate S, similar to the detection of positional misalignment between the first reference image I1 and the second reference image I2, a sufficient number of feature points may be obtained when comparing the captured image I3 with the reference images, regardless of whether components are mounted or not. In this case, regardless of the above, it is sufficient to simply compare and match the captured image I3 with a reference image in any mounting state using feature ranges that do not change depending on whether components are mounted or not, and to determine the amount of misalignment in the captured image I3.

[0051] Alternatively, not limited to the feature points mentioned above, the area in the reference image where the component is mounted may be uniformly masked, and the masked reference image and the captured image I3 may be compared and matched. In this case, when calculating the error between the reference image and the captured image I3, the combined portion of the masked area and the captured image I3 is excluded. The masked reference image may be generated manually in advance by an operator or the like.

[0052] As described above, the detection method of this embodiment includes the following steps: (1) Acquire multiple reference images of substrates S with different component mounting states. (2) Set multiple reference feature ranges for each of the multiple reference images, each containing multiple feature points on the substrate S. (3) Detect feature ranges similar to each of the multiple reference feature ranges from the captured image of the substrate S to be detected, and for each of the multiple reference feature ranges, associate the position of the feature point in the selected reference image with the position of the feature point in the feature range in the captured image, according to the degree of similarity with the feature range of the reference feature range in the multiple reference images. (4) Detect the position and orientation of the substrate S in the captured image based on the positional relationship of the feature points associated with each of the multiple reference feature ranges. By detecting the substrate S in this way, even if the captured image gradually changes due to component mounting, the decrease in detection accuracy of the substrate S can be reduced regardless of the situation throughout the mounting work. By specifying the position of the substrate S in this way, this detection method makes it possible to more appropriately specify the position of the substrate S, and consequently the mounting position of components on the substrate S, without being affected by conditions.

[0053] Furthermore, in this detection method, the relative positional relationship of multiple reference images may be determined based on the positional relationship of multiple feature points in the multiple reference images. The positional relationship of the associated feature points may be corrected according to the reference image with the greatest degree of similarity and the relative positional relationship. When multiple reference images are set as described above, relative positional shifts may occur between the multiple reference images. By determining this positional shift in advance, it becomes possible to accurately correct the coordinate shift between images and use multiple reference images to determine the position of the substrate S in the captured image.

[0054] Furthermore, the correction may include rotational movement. By considering not only the translational movement but also the rotational movement of the substrate S, it becomes possible to more practically and accurately determine the position of the substrate S.

[0055] Furthermore, the different mounting states may include a state in which no components are mounted on the substrate S and a state in which all components are mounted on the substrate S. By using the first and last images taken during the mounting process as reference images, as long as the mounted components are simply arranged in a planar manner, appropriate image portions can be extracted from the reference image and compared and reflected, regardless of whether all components have been mounted or not. Therefore, the decrease in detection accuracy of the substrate S during the mounting process can be reduced.

[0056] Furthermore, there may be three or more reference images. In particular, if overlaps or shadows of multiple components to be implemented may occur in the captured image, intermediate reference images that do not yet show such overlaps may be used in conjunction. This allows for the acquisition of reference images that more accurately match each part of the captured image.

[0057] Furthermore, different mounting states may include states in which at least one of the components and the component closest to each of those components is not mounted. By determining the third reference image while avoiding states in which all the nearest components are mounted, the effects of overlap between components and between components and their shadows are more effectively reduced, as described above, and the position of the substrate S can be obtained with greater accuracy.

[0058] Furthermore, the reference feature range may be defined for each mounting range of the components to be mounted. The reference image selected according to the degree of similarity with the feature range may be a reference image in which the presence or absence of components in the captured image matches. By defining a feature range for each mounting range of components in this way, the decrease in matching accuracy during the mounting process due to combinations of the presence or absence of components in multiple components is reduced. On the other hand, the number of feature ranges corresponding to the mounted components prevents the feature range area from being increased unnecessarily. Therefore, this detection method allows for more efficient acquisition of information such as the mounting position of the substrate S and components.

[0059] Furthermore, this detection method may include a step of acquiring information related to the mounting order of multiple components. It may also output information related to the mounting position of the component to be mounted after the component already mounted on the substrate S in the captured image. Based on the captured image during mounting and the information on the mounting order of the components, the next item to be mounted and its mounting position can be easily obtained. By outputting this information to the necessary locations, this detection method can reduce the workload of workers and reduce errors in the process order, thereby supporting the completion of the component mounting process more efficiently.

[0060] Furthermore, in this detection method, control information that indicates the mounting position of the next component to be mounted using the light emitted from the projection device 30 may be output to the projection device 30. By outputting a control signal that directly indicates the next mounting position using the projection device 30 in this way, the burden on the worker in confirming the work content is reduced. In addition, the possibility of work errors can also be reduced.

[0061] Furthermore, the control device 10, which is a detection processing device in this embodiment, includes a CPU 11. The CPU 11 acquires a plurality of reference images taken of substrates S with different component mounting states. For each of the plurality of reference images, the CPU 11 sets a plurality of reference feature ranges, each containing a plurality of feature points on the substrate S. The CPU 11 detects feature ranges from the captured image of the substrate S that are the target of detection, which are similar to the plurality of reference feature ranges. For each of the plurality of reference feature ranges, the CPU 11 associates the position of the feature point in the reference image selected according to the degree of similarity to the feature range in the plurality of reference images with the position of the feature point in the feature range in the captured image. Based on the positional relationship of the feature points associated with each of the plurality of reference feature ranges, the CPU 11 detects the position and orientation of the substrate S in the captured image. With this control device 10, by using multiple reference images in combination, the correspondence of feature points can be identified more accurately for each part of the substrate S according to the mounting status. Based on the positional relationship of the identified feature points, the control device 10 can accurately obtain the position of the substrate S.

[0062] Furthermore, by installing the program 131 of this embodiment on a computer, the processing related to the detection method described above can be easily executed in software. Therefore, with this program 131, the position of the substrate S, and consequently each mounting position on the substrate S, can be identified more appropriately, regardless of the conditions.

[0063] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are possible. For example, although the above description assumes that reference images I1 and I2 are always combined, the processing is not limited to this. Immediately after the start of mounting or just before the end of mounting, the image will match with either reference image I1 or reference image I2 with nearly sufficient accuracy even without combining them. Therefore, the position of the substrate S may be determined by combining reference images I1 and I2 only during the middle of the mounting process.

[0064] Furthermore, although the above assumes that both the position and orientation of the substrate S are specified, this is not the only limitation. In situations where rotational displacement cannot occur, for example, when a guide member that defines the orientation of the substrate S is located on the workbench T, rotational displacement may not be considered, and the orientation of the substrate S may not need to be determined.

[0065] Furthermore, although the above explanation assumes a relative positional misalignment of the substrate S between two reference images, this is not limited to this. When capturing the reference images, precise alignment can be performed, or after capturing the reference images, these positions can be precisely aligned and cropped. This eliminates the need to correct for positional misalignment between multiple reference images.

[0066] Furthermore, the reference image does not need to include the state in which no components are mounted or the state in which all components are mounted. This means that reference images of situations where some components are not mounted or where some components are mounted will not be included, but this is acceptable as long as it does not cause problems with the detection accuracy of the substrate S. Alternatively, a reference feature range for those some components may not even be set in the first place. In other words, some of the mounted components may not be included in the reference feature range.

[0067] Furthermore, the number of reference images may be more than three. However, since the effort required to identify the substrate S increases with the number of reference images, the number of reference images may be determined appropriately depending on the number and complexity of mounted components and the performance of the control device 10.

[0068] Furthermore, the feature range does not have to be defined for each component being mounted. The feature range may be set for only a portion of the component. Also, depending on the situation, there may be parts of the circuit board that are not included in the feature range. In addition, if overlap occurs between components as described above, and there are situations where there is overlap and situations where there is no overlap during mounting, the feature range that includes multiple overlapping components may be set. In this case, reference images for both the overlapping and non-overlapping situations may be prepared.

[0069] Furthermore, although the above describes an embodiment in which the projection device 30 emits light to indicate the next mounting position, it is not limited to this. The light may be emitted by a laser pointer or the like. Alternatively, the mounting position does not have to be indicated by the projection device 30 or the like. The next mounting position may simply be indicated by voice, or the system may not indicate the mounting position at all. In addition, the projection device 30 may be used not to indicate the mounting position in advance, but to point out errors during or after mounting.

[0070] Furthermore, while the above uses a substrate S as an example of a base material to describe the situation in which electronic components are mounted on the substrate S, the invention is not limited to this. The base material may be, for example, a support member or housing that supports other articles. Accordingly, the components that are attached may be metal fittings, fasteners, connectors, decorative items, buttons, switches and knobs, and other functional components. For example, the contents of this disclosure may be applied to a situation in which many buttons and switches are attached to the housing of various adjustment devices.

[0071] Furthermore, while the above description has used a storage unit 13 consisting of non-volatile memory such as an HDD or flash memory as an example of a computer-readable medium for storing the program 131 related to the control of substrate detection and mounting support of the present invention, the invention is not limited to these. Other computer-readable media can include other non-volatile memories such as MRAM, or portable recording media such as CD-ROMs and DVD discs. In addition, a carrier wave can also be used as a medium for providing the program data according to the present invention via a communication line.

[0072] Furthermore, the specific configurations, processing operations, and procedures shown in the above embodiments can be modified as appropriate without departing from the spirit of the present invention. The scope of the present invention includes the scope of the invention described in the claims and its equivalents. [Explanation of Symbols]

[0073] I1 First reference image, I2 Second reference image, I12, I4 Reference image, I3 Captured image, P1, P11, P2, P21, P3, P31 Components, S Circuit board

Claims

1. By acquiring multiple reference images of base materials with different component mounting configurations, For each of the aforementioned multiple reference images, a plurality of reference feature ranges are set, each including a plurality of feature points on the base member. From the captured image of the base member that is the target of detection, feature ranges similar to each of the multiple reference feature ranges are detected, and for each of the multiple reference feature ranges, the position of the feature point in the reference image selected according to the degree of similarity between the reference feature range and the feature range in the multiple reference images is associated with the position of the feature point in the feature range in the captured image. Based on the positional relationship of the feature points associated with each of the aforementioned plurality of reference feature ranges, the position or orientation of the base member in the captured image is detected. Detection method.

2. Based on the positional relationship of the multiple feature points in the multiple reference images, the relative positional relationship of the multiple reference images is identified. The positional relationship of the associated feature points is corrected according to the selected reference image and the relative positional relationship. The detection method according to claim 1.

3. The detection method according to claim 2, wherein the correction includes rotational movement.

4. The detection method according to claim 1, wherein the aforementioned different mounting states include a state in which none of the components are mounted and a state in which all of the components are mounted.

5. The detection method according to claim 4, wherein the plurality of reference images are three or more.

6. The aforementioned different mounting states include a state in which at least one of the multiple components and the component that is mounted closest to each of those multiple components is not mounted. The detection method according to claim 5.

7. The aforementioned reference feature range is determined for each mounting range of the component to be mounted, The reference image selected according to the degree of similarity with the feature range is the reference image in which the presence or absence of the component in the captured image matches. The detection method according to claim 1.

8. Information relating to the mounting order of multiple aforementioned components is obtained, The system outputs information relating to the mounting position of the component to be mounted next to the component already mounted on the base member in the captured image. The detection method according to claim 1.

9. The detection method according to claim 8, wherein control information is output to the projection device to indicate the mounting position of the next component to be mounted by the light emitted from the projection device.

10. By acquiring multiple reference images of base materials with different component mounting configurations, For each of the aforementioned multiple reference images, a plurality of reference feature ranges are set, each including a plurality of feature points on the base member. From the captured image of the base member that is the target of detection, feature ranges similar to each of the multiple reference feature ranges are detected, and for each of the multiple reference feature ranges, the position of the feature point in the reference image selected according to the degree of similarity between the reference feature range and the feature range in the multiple reference images is associated with the position of the feature point in the feature range in the captured image. Based on the positional relationship of the feature points associated with each of the aforementioned plurality of reference feature ranges, the position or orientation of the base member in the captured image is detected. A detection and processing device equipped with a control unit.

11. Computers, Acquisition means for acquiring multiple reference images of base members with different component mounting states, Setting means for setting a plurality of reference feature ranges for each of the plurality of reference images, each including a plurality of feature points on the base member, A matching means that detects feature ranges similar to each of the multiple reference feature ranges from the captured image of the base member that is the target of detection, and associates, for each of the multiple reference feature ranges, the position of the feature point in the reference image selected according to the degree of similarity between the reference feature range and the feature range in the multiple reference images, with the position of the feature point in the feature range in the captured image. A detection means for detecting the position or orientation of the base member in the captured image based on the positional relationship of the feature points associated with each of the aforementioned plurality of reference feature ranges, A program that makes it function as such.

Images