Determination method of projected position, and determination device of projected position

Abstract

To detect a positional deviation of a position image projected on a working object.SOLUTION: A reference point image associated with a position image for a working object 6 is projected from a projection device 12 in association with physical markers Mk0 to Mk4 for defining the position of the working object 6. At least one photographed image is acquired by photographing the physical markers Mk0 to Mk4 and the reference point image with a photographing device 14. Whether a coordinate indicated by the physical markers Mk0 to Mk4 and a reference point image coordinate satisfy a previously determined relation is determined by a computer 10. When the relation is not satisfied, the computer 10 detects that the position of the position image is deviated from the working object 6.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a method and an apparatus for determining a projection position. [Background technology]

[0002] For example, there is known a technology for projecting an image onto a work object on which a worker is working, such as on a production line, and more specifically, a technology for supporting the worker's work by projecting a position image onto the work object that indicates the work position and work procedure for marking, drilling, etc. (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-000726 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above technology has a problem in that if the position image projected onto the work object is misaligned for some reason, work will be performed on the work object at an incorrect position thereafter. [Means for solving the problem]

[0005] In order to solve the above problem, a method for determining a projection position according to one embodiment of the present disclosure includes projecting a reference point image associated with a position image for a work object on which work is to be performed from a projection device, in correspondence with a reference point possessed by a base for defining the position of the work object; acquiring at least one captured image by photographing the reference point and the reference point image with a photographing device; determining by a computer whether the position of the reference point and the position of the reference point image match based on the at least one captured image; and detecting by the computer that the position of the position image has shifted relative to the work object if the position of the reference point and the position of the reference point image do not match.

[0006] A method for determining a projection position according to another aspect of the present disclosure includes: reproducing, in a virtual space of a computer, an arrangement of a work object on which work is to be performed in real space; a base having a reference point in the real space and for determining the position of the work object; and a projection device that projects a position image for the work object in the real space onto the work object; projecting the position image projected from the projection device onto the work object in the virtual space from the projection device onto the work object in the real space; arranging a reference point image associated with the position image in the virtual space at the position of the reference point; and projecting the reference point image associated with the position image in the real space. The method includes projecting a point image from the projection device so as to correspond to the reference point; acquiring at least one captured image by capturing images of the reference point and the reference point image in the real space with an imaging device; determining by the computer a first positional relationship between the position of the reference point and the position of the reference point image based on the at least one captured image; determining by the computer whether the first positional relationship matches a second positional relationship between the position of the reference point and the position of the reference point image in the virtual space; and detecting by the computer that a deviation of the positional image has occurred if the first positional relationship and the second positional relationship do not match.

[0007] A projection position determination device according to an aspect of the present disclosure is a projection position determination device realized by a computer that reproduces in a virtual space an arrangement of a work object on which work is to be performed in a real space, a base having a reference point in the real space and for determining the position of the work object, and a projection device that projects a position image for the work object in the real space onto the work object, wherein the computer projects the position image that is projected from the projection device onto the work object in the virtual space from the projection device onto the work object in the real space, and places a reference point image associated with the position image in the virtual space at the position of the reference point. projecting a reference point image associated with the position image in the real space from the projection device so as to correspond to the reference point; acquiring at least one captured image by capturing an image of the reference point and the reference point image in the real space with an imaging device; determining a first positional relationship between the position of the reference point and the position of the reference point image based on the at least one captured image; determining whether or not the first positional relationship matches a second positional relationship between the position of the reference point and the position of the reference point image in the virtual space; and detecting that a deviation of the position image has occurred if the first positional relationship and the second positional relationship do not match. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an overview of a system to which a projection position determination method according to an embodiment is applied; [Figure 2] FIG. 10 is a diagram showing an example in which a position image is projected onto a work target. [Figure 3] FIG. 1 is a diagram illustrating a system configuration. [Figure 4] FIG. 1 is a functional block diagram constructed by a computer control device in the system. [Figure 5] FIG. 10 is a diagram illustrating an example of a physical marker. [Figure 6] FIG. 10 is an explanatory diagram of coordinates etc. obtained by a physical marker. [Figure 7]10A and 10B are explanatory diagrams of reference point images projected onto a physical marker, etc. FIG. [Figure 8] 10 is a flowchart showing the operation of projecting a position image and a reference point image. [Figure 9] 10 is a flowchart showing a projection position determination operation. [Figure 10] FIG. 10 is an explanatory diagram showing generation of a position image. [Figure 11] FIG. 10 is a diagram showing an example of an analysis range set for a physical marker. [Figure 12] FIG. 10 is a diagram illustrating an example in which it is determined that there is no positional deviation. [Figure 13] 10A and 10B are diagrams illustrating examples of cases where it is determined that there is a positional deviation. [Figure 14] FIG. 10 is a diagram showing a first modified example. [Figure 15] FIG. 10 is a diagram showing a second modified example. [Figure 16] FIG. 10 is a diagram showing a fourth type deformation. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a method for determining a projection position according to an embodiment will be described with reference to the drawings. Note that in each drawing, the dimensions and scale of each part are appropriately different from those of the actual parts. Furthermore, since the embodiments described below are preferred specific examples, various technically preferable limitations are applied, but the scope of the present disclosure is not limited to these embodiments unless otherwise specified in the following description to the effect that the present disclosure is limited.

[0010] 1 is a diagram showing an overview of a system 1 including a computer 10 that realizes a projection position determination method according to an embodiment. The system 1 includes the computer 10, a projection device 12, and an image capture device 14. The system 1 projects a position image from the projection device 12 onto a work object, and detects whether the position image has been projected in a misaligned state relative to the work object.

[0011] The computer 10 provides image data representing the position image to the projection device 12 and, in cooperation with software, determines the projection position of the position image projected by the projection device 12 . The projection device 12 is a projector that is fixed to a predetermined position (described later) on the ceiling of a workroom, which is the real space, or on a tripod or the like installed in the workroom, and projects a position image based on image data supplied from the computer 10 onto the work object. In other words, the arrangement of the projection device 12 (the installation position and projection direction of the projection device 12) has a predetermined relationship with the work object.

[0012] The image capturing device 14 is a digital still camera that is installed on the ceiling of the work room or in the work room, captures an image projected onto a work object, and supplies the computer 10 with image data indicating the captured image. It should be noted that while the projection device 12 needs to be placed in a specified relationship to the work object, the image capture device 14 can be installed anywhere as long as it can capture the projected positional image.

[0013] FIG. 2 is a diagram for explaining the installation state of the work target 6 in the system 1, a position image projected onto the work target 6, and the like. The work object 6 is placed in a positioned state on a fixed work table 5. In detail, a positioning jig 3 is placed on the work table 5, and the work object 6 is positioned by the positioning jig 3. The positioning jig 3 is composed of, for example, rectangular parallelepipeds 3a, 3b, and 3c that are at right angles to each other, and the position of an example long work object 6 is determined by a worker 7 pressing the work object 6 against two sides of the rectangular parallelepipeds 3a and 3b.

[0014] In the positioning jig 3, a physical marker Mk0 is provided near the intersection 3Rp of the rectangular parallelepipeds 3a and 3b where the work object 6 is positioned. In addition, physical markers Mk1 to Mk4 are provided in a row at intervals on the rectangular parallelepiped 3c. The physical markers Mk0 to Mk4 are a kind of mark used to determine whether or not the position image is projected at the correct position on the work object 6. In this embodiment, ArUco markers, which are simple two-dimensional barcodes, are used as the physical markers Mk0 to Mk4. Each of the physical markers Mk0 to Mk4 is assigned an identifier for uniquely identifying itself.

[0015] The projection device 12 alternately projects the position image and the superimposed image onto the work object 6 . The position image is an image that shows, for example, a black background, marking lines on the work object 6, processing points, work content, work procedures, etc. Fig. 2 shows an example in which line diagrams 122a and 122b indicating processing points on the work object 6 are projected as the position image. The superimposed image is an image in which projected bright points for determining the positions of the physical markers Mk0 to Mk4, that is, reference point images, are superimposed on the position image, and will be described in detail later. The image capturing device 14 captures the alternately displayed position images and superimposed images at, for example, 30 FPS (frames per second).

[0016] 3 is a block diagram showing the electrical configuration of the system 1. In the system 1, a computer 10 includes various devices, specifically, a control device 100, a storage device 112, an input IF (interface) 116, and an output IF 118. These devices are electrically connected to each other via, for example, a bus Bs.

[0017] The control device 100 is configured with one or more processing circuits such as a CPU (Central Processing Unit), and comprehensively controls each element of the computer 10. Note that the control device 100 may be configured with circuits such as a DSP (Digital Signal Processor) and an ASIC (Application Specific Integrated Circuit) in addition to the CPU.

[0018] The storage device 112 is a single or multiple memories configured with known storage media such as magnetic storage media or semiconductor storage media, and stores programs for position determination executed by the control device 100, various data used by the control device 100, position image data, superimposed image data, etc. The storage device 112 may also be configured with a combination of multiple types of storage media. Alternatively, a portable storage medium that is detachable from the computer 10, or an external storage medium with which the computer 10 can communicate via a network, such as online storage, may be used as the storage device 112. The input IF 116 inputs imaging data from the imaging device. The output IF 118 supplies the projection device 12 with data on the position image and the superimposed image.

[0019] 4 is a block diagram showing the functional configuration of the control device 100. As shown in the figure, the control device 100 functions as a plurality of elements (a processing control unit 102, an image acquisition unit 104, an image output unit 106, and a determination unit 108) by executing a program stored in the storage device 112, more specifically, a program for determining the projection position of a position image.

[0020] The process control unit 102 controls the elements of each unit. The image acquisition unit 104 acquires photographic data taken by the photographing device 14 input via the input IF. The image output unit 106 alternately outputs the position image data and the superimposed image data read out from the storage device 112 by the processing control unit 102 to the projection device 12 via the output IF 118. The determination unit 108 determines whether or not there is a positional deviation based on the photographic data acquired by the image acquisition unit 104.

[0021] Next, the physical markers Mk0 to Mk4 and the reference point images projected toward the physical markers Mk0 to Mk4 will be described using the physical marker Mk0 as an example.

[0022] Figure 5 is a diagram showing an example of a physical marker Mk0, Figure 6 is an explanatory diagram of the coordinates to be identified using the physical marker Mk0, and Figure 7 is a diagram showing a reference point image projected toward the physical marker Mk0.

[0023] The positions of the physical markers Mk0 to Mk4 can be determined with a certain degree of accuracy using the coordinate information associated with the identifiers of the physical markers Mk0 to Mk4. However, in this embodiment, the following method is adopted to determine the positions of the physical markers Mk0 to Mk4 with higher accuracy.

[0024] First, in a state where no reference point image is superimposed, the coordinates indicated by the physical marker Mk0 are obtained from the photographing data captured by the photographing device 14. In detail, as shown in Fig. 6, the coordinates of the four corners indicated by the arrows are identified in the physical marker Mk0 from the photographing data of the position image where no reference point image is superimposed. Then, the diagonal center A0 of the identified coordinates of the four corners is obtained as the coordinates indicated by the physical marker Mk0.

[0025] Second, the projection device 12 projects reference point images toward the four corners of the physical marker Mk0, while the center of gravity of the reference point images photographed by the photographing device 14 is determined. FIG. 7 shows a state in which the reference point image projected from the projection device 12 is accumulated over multiple frames. When the reference point image is accumulated over multiple frames, the reference point image has a certain degree of spatial spread due to shaking caused by micro-vibrations, etc. Therefore, the barycentric coordinates of the spread reference point image are obtained. The barycentric coordinates of the reference point image are the black circle indicated by the arrow in FIG. 7. Then, the diagonal center B0 of the obtained barycentric coordinates is obtained as the reference point image coordinates. In addition, in FIG. 7, the background is gray to show the extent of the reference point image.

[0026] Note that the projection device 12 is not necessarily positioned directly in front of the physical marker Mk0, but is usually positioned at an angle. Furthermore, the reference point image projected by the projection device 12 is an image obtained by photographing (viewing) point images of the four corners of a physical marker placed in virtual space, as will be described later. Therefore, since the reference point image projected in real space is often a distorted rectangle, the reference point image photographed by the photographing device 14 is also often a distorted rectangle. However, for ease of explanation, the reference point image in FIG. 7 is shown as a square.

[0027] Although the physical marker Mk0 has been described here, the coordinates indicated by the other physical markers Mk1 to Mk4 and the coordinates of the reference point images projected toward the physical markers Mk0 to Mk4 can also be calculated in a similar manner. In this embodiment, four reference point images are projected onto one physical marker, but this is not particularly limited. For example, an O-shaped reference point image connecting the four corners of one physical marker may be projected onto one physical marker. For example, only one reference point image may be projected onto only one corner of one physical marker.

[0028] Next, the operation of the system 1 will be described.

[0029] Fig. 8 is a flowchart showing the operation of projecting a position image executed by the computer 10, and Fig. 9 is a flowchart showing the operation of determining the position of the position image executed by the computer 10. Note that steps S11 to S13 in Fig. 8 are operations for generating a position image and a superimposed image before projecting the position image etc.

[0030] First, as a preliminary operation for generating a position image, the process control unit 102 executes three-dimensional rendering of the work object 6 in the virtual space in accordance with an instruction from the operator of the computer 10 (step S11). Specifically, as shown in Fig. 10 , the process control unit 102 places a 3D model of the work object 6, generated based on three-dimensional CAD information or the like, in the virtual space 200. The process control unit 102 also aligns and places a position image to be projected onto the work object 6, specifically a position image indicating a marking line on the work object 6, a processing point, work content, work procedure, etc., with the 3D model placed in the virtual space. In other words, a state in which the position image is virtually projected onto the 3D model is created in the computer 10. The process control unit 102 places a physical marker Mk on the work object 6 in the virtual space 200 in alignment with the physical marker in real space, and further places reference point images at each of the four corners of the physical marker Mk in the virtual space 200.

[0031] Next, as shown in FIG. 10, the process control unit 102 places the projection device 12 in the virtual space 200 in accordance with the projection device 12 placed in the real space (step S12). The processing control unit 102 obtains images showing what a position image in the virtual space 200 and a superimposed image in which a reference point image is superimposed on the position image would look like when viewed from a projection device 12 placed in the virtual space 200 (step S13). That is, when the position of the projection device 12 in the virtual space 200 is set as the camera position, the process control unit 102 obtains the position image and the superimposed image projected onto the image coordinate system 250 of the camera. Note that FIG. 10 is merely a diagram for explaining the generation of the position image and the superimposed image, and does not correspond to FIGS. 1 and 2 which show the arrangement of the work object 6 and the projection device in the real space.

[0032] Here, the determined position image and reference point image are images viewed from the projection device 12 when the projection device 12 and work object 6 are positioned in the same way as in real space, even though they are in virtual space. Therefore, the position image and reference point image viewed from the position of the projection device 12 in virtual space 200 become the position image and reference point image to be projected onto the work object 6 positioned in the same way in real space. The processing control unit 102 stores the position image data and the superimposed image data obtained in the virtual space in the storage device 112.

[0033] Next, the process control unit 102 reads out the position image data stored in the storage device 112 and transfers it to the image output unit 106, which then supplies it to the projection device 12 in real space (step S14). As a result, the projection device 12 in real space projects the position image onto the work object 6. For example, one second after the start of step S14, the processing control unit 102 reads out the data of the superimposed image stored in the storage device 112 and transfers it to the image output unit 106, which then supplies it to the projection device 12 in real space (step S15). As a result, the projection device 12 in real space projects the reference point image onto the work object 6, superimposing the reference point image on the position image.

[0034] The process control unit 102 returns the process procedure to step S14, for example, one second after the start of step S15. As a result, steps S14 and S15 are executed alternately, so that the position image and the superimposed image are displayed alternately every second. Since the superimposed image is an image in which the reference point image is superimposed on the position image, the position image is constantly projected, while the reference point image is repeatedly turned on for one second and turned off for one second. In other words, the reference point image is projected as if it were flashing every two seconds.

[0035] Independently from the operation of projecting the reference point image on and blinking it onto such a position image, the operation of determining the next position image is executed.

[0036] First, the process control unit 102 accumulates the photographic data acquired via the image acquisition unit 104 for, for example, 2 seconds, which is the blinking period, and calculates the average luminance L1 of the accumulated images (step S21). As described above, the position image and the superimposed image are projected alternately every second, while the image capturing device 14 captures images at 30 FPS, so that images accumulated over two seconds will include 30 frames of the position image and 30 frames of the superimposed image. The difference between a positional image and a superimposed image is that the positional image does not include a reference point image, while the superimposed image does. Therefore, when comparing a positional image in one frame with a superimposed image of the same frame, the average brightness of the superimposed image is higher than that of the positional image by the amount of the reference point image, which is a bright point. Therefore, when a frame of image is acquired, if the average brightness of the captured image is less than the average brightness L1 of a stored image containing the same number of frames of positional images and superimposed images, the acquired captured image can be determined to be a positional image. On the other hand, when a frame of image is acquired, if the average brightness of the captured image is higher than the average brightness L1 of the stored image, the acquired captured image can be determined to be a superimposed image with a reference point image superimposed.

[0037] Using this, first, the process control unit 102 acquires one frame of captured image via the image acquisition unit 104 (step S22). Next, the process control unit 102 calculates the average brightness L2 of the acquired one frame of captured image (step S23). Then, the process control unit 102 determines whether the average brightness L2 calculated in step S23 is less than the average brightness L1 of the stored image calculated in step S21 (step S24).

[0038] If the average luminance L2 is less than the average luminance L1 (if the determination result in step S24 is "Yes"), the photographed image of one frame acquired in step S22 is a positional image without a reference point image. Therefore, the process control unit 102 calculates the coordinates of each of the four corners of the physical marker from the position image (step S25), and further calculates the coordinates of the diagonal center defined by the coordinates of the four corners as the coordinates of the physical marker (step S26). Note that the process control unit 102 performs the calculation operation for each of the physical markers Mk0 to Mk4.

[0039] Next, the process control unit 102 sets a range centered on the coordinates of the four corners of the physical marker calculated in step S26 as an analysis range of the reference point image (step S27). Specifically, in the case of the physical marker Mk0, regions R1 to R4 centered on the coordinates of the four corners are set as the analysis range of the reference point image, as shown in Fig. 11. Although the explanation has been given for the physical marker Mk0 here, analysis ranges are set similarly for the four corners of the other physical markers Mk1 to Mk4. If the analysis of the reference point image, which will be described next, is performed in all areas of the shooting range of the shooting device 14, the load on the calculation processing will be heavy, so the calculation load is reduced by narrowing the analysis range of the reference point image in advance.

[0040] In the example of steps S25 to S27, position images for one frame are processed, but position images for a plurality of frames, for example, 20 frames, may be stored and processed.

[0041] If the average brightness L2 is higher than the average brightness L1 (if the judgment result in step S24 is "No"), the captured image of one frame acquired in step S22 is a superimposed image in which a reference point image of a bright point is superimposed on a position image. Immediately after the determination result in step S24 switches from "Yes" to "No," the superimposed image continues for 30 frames. Therefore, the process control unit 102 accumulates, for example, 20 frames of reference point images included in the set analysis range from the superimposed images, calculates the coordinates of the centers of gravity at the four corners (step S29), and further calculates the diagonal center defined by the coordinates of the four corners as the reference point image coordinates (step S30). The process control unit 102 executes the operation of calculating the reference point image coordinates for each of the reference point images projected corresponding to the physical markers Mk0 to Mk4.

[0042] Next, the process control unit 102 determines whether or not the distance between the coordinates of the physical marker calculated in step S26 and the reference point image coordinates calculated in step S30 is less than a threshold value (step S31). Here, it is determined whether the distances between the coordinates of not just one physical marker but all physical markers Mk0 to Mk4 and the reference point image coordinates obtained from the reference point image projected corresponding to the physical markers Mk0 to Mk4 are all less than a threshold value. Note that the distance being less than the threshold is an example of "the position of the reference point and the position of the reference point image agreeing."

[0043] If the distance is less than the threshold (if the determination result in step S31 is "Yes"), this means that the position image is being projected onto the work object 6 without any positional deviation or with a positional deviation that does not affect the work, and so there is no need to issue a warning. Therefore, the process control unit 102 returns the process procedure to step S22 to repeat the same operation in preparation for the occurrence of positional deviation.

[0044] Note that Figure 12 shows an example in which the distance between the diagonal center A0, which is the central coordinate of the physical marker Mk0, and the diagonal center B0, which is the reference point image coordinate obtained in the reference point image corresponding to the physical marker Mk0, is less than the threshold value.

[0045] When the processing procedure returns to step S22, the processing up to step S31 is repeated, but this position image determination operation ends after the processing of step S32 described below is executed, or when the operator of computer 10 performs an operation to stop it.

[0046] The "positional deviation" referred to here does not only occur when the projection device 12 is not positioned in a predetermined manner relative to the work object 6 in real space. For example, it occurs when the position of the projection device 12 relative to the work object 6 in virtual space and the position of the projection device 12 relative to the work object 6 in real space do not satisfy a predetermined relationship. In other words, it occurs when the first positional relationship between a physical marker in real space and a reference point image positioned corresponding to the physical marker, and the second positional relationship between a physical marker in virtual space and a reference point image positioned corresponding to the physical marker in virtual space do not satisfy a predetermined relationship, i.e., they do not match, or are not similar enough to be considered to match.

[0047] On the other hand, if the distance between the center coordinates of the physical marker and the reference point image coordinates is equal to or greater than the threshold value (if the judgment result of step S31 is "No"), the processing control unit 102 detects that a positional shift has occurred and notifies the operator 7 of the occurrence of the positional shift (step S32). FIG. 13 shows an example in which the distance between the diagonal center A0, which is the central coordinate of the physical marker Mk0, and the diagonal center B0, which is the reference point image coordinate corresponding to the physical marker Mk0, is equal to or greater than the threshold value. As an example of notifying the occurrence of a positional deviation, it is considered preferable to have the projection device 12 project an image that is different from the display mode of the positional image and the superimposed image, such as projecting the positional image in red or a pre-created warning image, since this does not require any additional configuration to the system 1. The examples of the notification are not limited to these, and any notification may be used as long as the operator 7 is able to recognize the occurrence of a positional deviation.

[0048] Upon receiving notification that a positional deviation has occurred, the worker 7 can perform calibration to eliminate the positional deviation of the projection device 12 relative to the work object 6. Specifically, the worker 7 can re-arrange the position of the projection device 12 relative to the work object 6 in real space so that it matches the position of the projection device 12 relative to the work object 6 in virtual space, or conversely, can re-generate the position image in a state where the position of the projection device 12 relative to the work object 6 in virtual space matches the position of the projection device 12 relative to the work object 6 in real space. The operation of determining the positional deviation ends after the notification in step S32 has been executed for a certain period of time, for example, 10 seconds, or when the operator 7 operates the computer 10 to indicate that he or she has acknowledged the notification.

[0049] According to this embodiment, if the position image is not projected at a position relative to the work object 6 as specified, an alert to that effect is issued, allowing the work to be stopped thereafter, thereby preventing a series of defective products due to work errors. Furthermore, in this embodiment, there is a restriction that the placement of the projection device 12 must have a predetermined relationship with the work object 6. However, the placement of the image capture device 14, i.e., the installation position and image capture direction of the image capture device 14, is not subject to the same restrictions as the projection device 12, as long as it can capture the reference point images projected in correspondence with the physical markers Mk0 to Mk4. In extreme cases, the placement of the image capture device 14 may be changed from its initial placement when determining misalignment. Therefore, according to this embodiment, not only can the degree of freedom in the placement of the image capture device 14 be increased, but the robustness of the misalignment determination can also be improved.

[0050] In the embodiment, it is possible that a worker 7 may appear in the captured image, but the worker 7 appearing in the captured image should be included in both the position image and the superimposed image, so the impact of the worker 7 appearing in the image can be ignored.

[0051] In the embodiment, the positional deviation of the projection position is determined by alternately projecting the position image and the superimposed image. As described above, the coordinates of the physical markers Mk0 to Mk4 can be obtained by reading out coordinate information previously associated with the identifiers of the physical markers Mk0 to Mk4. Therefore, the positional deviation of the projection position can be determined by comparing the reference point image coordinates obtained in the superimposed image with the read coordinates of the physical markers Mk0 to Mk4. In other words, the positional deviation of the projection position can be determined using only one frame of an image obtained by capturing a superimposed image accompanied by a reference point image.

[0052] The projection position determination method according to the embodiment can be understood as a projection position determination device realized by the computer 10. In addition, the following modifications are possible in the embodiment.

[0053] 14 is a diagram showing an overview of the first modified example. In the embodiment, the computer 10 was configured to determine the projection position of the position image, but in recent years, the performance of control devices in cameras, which are the image capturing devices 14, has improved significantly. For this reason, the first modified example is configured so that the image capturing device 14 determines the positional deviation.

[0054] In detail, the image capturing device 14 includes a control device 140 , a storage device 142 , an image sensor 144 and an output IF 146 . The image sensor 144 is a semiconductor element that converts an image captured through a lens (not shown) into an electrical signal. Similar to the control device 100 in the computer 10, the control device 140 is configured with a processing control unit, an image acquisition unit, and a determination unit (not shown), and similar to the computer 10, determines the positional deviation of the position image relative to the work object 6.

[0055] In the first modified example, the calculation process for determining the positional deviation is transferred to the imaging device 14, and therefore the transfer of the imaging data from the imaging device 14 to the computer 10 is omitted. Therefore, according to the first modified example, not only can the load on the computer 10 be reduced, but also the effect of delays associated with the transfer of the imaging data can be suppressed.

[0056] 15 is a diagram showing an overview of the second modified example. In the embodiment, the computer 10 installed near the projection device 12 is configured to determine the projection position of the position image. In the second modified example, the computer 10 functions simply as a relay device, and the projection position of the position image is determined by a computer 20 located in a remote location that is connected to the computer 10 via a network N. In short, the computer 10 is remotely operated by the computer 20.

[0057] In the second modified example, not only the worker 7 but also the operator of the computer 20 at a remote location can know the positional deviation of the position image relative to the work object 6. The operator at a remote location cannot reposition the projection device 12 relative to the work object 6. However, the operator at a remote location can regenerate the position image so that the position of the projection device 12 relative to the work object 6 in real space matches the position of the projection device 12 relative to the work object 6 in virtual space. Therefore, according to the second modified example, either the worker 7 or the operator of the computer 20 can perform calibration to eliminate the positional deviation of the position image relative to the work object 6.

[0058] In the embodiment, the position image and the superimposed image are alternately projected to constantly determine the positional deviation, but it is not necessary to constantly determine the positional deviation. Therefore, a third modified example will be described, in which the positional deviation is determined only at specific timings.

[0059] The specific timing in the third modified example can be the following: Specifically, the specific timing can be the timing when the position image showing the processing position or work procedure is switched, a relatively long time interval such as a periodic timing of every 5 minutes, the timing when the worker 7 performs a specific operation on the computer 10, or the timing when vibrations above a threshold are detected as in the fourth modified example described next.

[0060] If the positional deviation is constantly determined as in the embodiment, the reference point image will be turned on and off repeatedly at two-second intervals, which may be visually bothersome to the worker 7 and may lead to a decrease in work efficiency. In contrast, according to the third modification, it is possible to reduce the visual annoyance to the worker 7. Furthermore, in the third modification, when it is determined at a certain timing that there is a positional deviation, the work defect is limited to the period from the previous timing at which it was determined that there is no positional deviation until the current timing. This makes it easier to take measures against the work defect.

[0061] Even if no misalignment occurs in the initial state, a situation in which it is determined that a misalignment has occurred later may be when vibrations occur in the workroom, causing the placement of the projection device 12 to change relative to the work object 6. Therefore, in the fourth modified example shown in Fig. 16, the output of a sensor 16 that detects vibrations in the workbench 5 is supplied to the computer 10, and when the computer 10 detects that vibrations equal to or greater than a threshold have occurred, it executes the processing of steps S21 to S31 described above and determines the misalignment of the position image. Note that the sensor 16 may be installed in the positioning jig 3. According to the fourth variant, a positional deviation is detected only when vibrations exceeding a threshold value occur, so similar to the third variant, it is possible to reduce the visual annoyance to the worker 7 and also to make it easier to take measures against work defects.

[0062] In the embodiments, the positioning jig 3 or the work table 5 is an example of a “base,” and the physical markers Mk0 to Mk4 are an example of a “reference point.” Furthermore, although five physical markers Mk0 to Mk4 are used in the embodiments, only one physical marker or six or more physical markers may be used. An image captured of a position image, i.e., an image captured including the position image and physical markers Mk0 to Mk4, is an example of a "first captured image," and an image captured of a superimposed image in which a reference point image is superimposed on a position image, i.e., an image captured including the position image, physical markers Mk0 to Mk4, and reference point image, is an example of a "second captured image."

[0063] A summary of this disclosure is provided below.

[0064] The method for determining a projection position according to Supplementary Note 1 includes projecting a reference point image associated with a position image for a work object on which work is to be performed from a projection device, in correspondence with a reference point on a base for defining the position of the work object; acquiring at least one captured image by photographing the reference point and the reference point image with an imaging device; determining by a computer whether the position of the reference point and the position of the reference point image match based on the at least one captured image; and detecting by the computer that the position of the position image has shifted relative to the work object if the position of the reference point and the position of the reference point image do not match. According to the projection position determination method of Supplementary Note 1, if the projection position of the position image projected onto the work object is shifted, it is possible to prevent subsequent production of defective products.

[0065] A projection position determination method according to Supplementary Note 2, which is another aspect of the present disclosure, includes: reproducing, in a virtual space of a computer, an arrangement of a work object on which work is to be performed in real space; a base having a reference point in the real space and for determining the position of the work object; and a projection device that projects a position image of the work object in the real space onto the work object; projecting the position image projected from the projection device onto the work object in the virtual space from the projection device onto the work object in the real space; arranging a reference point image associated with the position image in the virtual space at the position of the reference point; The method includes projecting the associated reference point image from the projection device so that it corresponds to the reference point; acquiring at least one captured image by photographing the reference point and the reference point image in the real space with a photographing device; determining by the computer a first positional relationship between the position of the reference point and the position of the reference point image based on the at least one captured image; determining by the computer whether or not the first positional relationship matches a second positional relationship between the position of the reference point and the position of the reference point image in the virtual space; and detecting by the computer that a deviation of the positional image has occurred if the first positional relationship and the second positional relationship do not match. According to the projection position determination method of Supplementary Note 2, when the projection position of the position image projected onto the work object is shifted, it is possible to prevent subsequent production of defective products. Also, it is possible for the worker (or the computer operator) to know the discrepancy between the first positional relationship between the reference point and the reference point image in the real space and the second positional relationship between the reference point and the reference point image in the virtual space.

[0066] The method for determining the projection position according to Supplementary Note 3 is the same as that according to Supplementary Note 1 or 2, wherein acquiring at least one captured image includes acquiring a first captured image by capturing an image of the reference point with a camera, and acquiring a second captured image by capturing an image of the reference point and the reference point image with the camera, and determining by the computer whether or not there is a match is based on the results of analyzing a data set including the first captured image and the second captured image.

[0067] The projection position determination method according to Supplementary Note 4 includes, in Supplementary Note 1 or 2, changing the display mode of the position image after the computer detects that a deviation of the position image relative to the work object has occurred.

[0068] The method for determining the projection position according to Supplementary Note 5 includes, in Supplementary Note 1 or 2, determining by the computer whether the base has moved based on a signal from a sensor that supports the work object and detects at least one of the movement of the base or the movement of the projection device, and acquiring the at least one captured image when the computer determines that the base has moved.

[0069] The projection position determination device according to Supplementary Note 6 is a projection position determination device realized by a computer that reproduces in a virtual space an arrangement of a work object on which work is performed in a real space, a base having a reference point in the real space and for determining the position of the work object, and a projection device that projects a position image for the work object in the real space onto the work object, wherein the computer projects the position image projected from the projection device onto the work object in the virtual space from the projection device onto the work object in the real space, and places a reference point image associated with the position image in the virtual space at the position of the reference point. projecting a reference point image associated with the position image in the real space from the projection device so as to correspond to the reference point; acquiring at least one captured image by capturing images of the reference point and the reference point image in the real space with an imaging device; determining a first positional relationship between the position of the reference point and the position of the reference point image based on the at least one captured image; determining whether or not the first positional relationship matches a second positional relationship between the position of the reference point and the position of the reference point image in the virtual space; and detecting that a deviation of the position image has occurred if the first positional relationship and the second positional relationship do not match. According to the projection position determination device of Supplementary Note 6, when the projection position of the position image projected onto the work object is shifted, it is possible to prevent subsequent production of defective products. Also, it is possible for the worker (or the computer operator) to know the discrepancy between the first positional relationship between the reference point and the reference point image in the real space and the second positional relationship between the reference point and the reference point image in the virtual space. [Explanation of symbols]

[0070] 1...judgment system, 6...work object, 10, 20...computer, 12...projection device, 14...imaging device, 100...control device

Claims

1. projecting, from a projection device, a reference point image associated with a position image for a work object on which work is to be performed, in correspondence with a reference point of a base for defining the position of the work object; acquiring at least one photographed image by photographing the reference point and the reference point image with an imaging device; determining by a computer whether or not the position of the reference point and the position of the reference point image coincide with each other based on the at least one captured image; If the position of the reference point does not match the position of the reference point image, the computer detects that the position of the position image has shifted relative to the work object; A method for determining a projection position, comprising:

2. Reproducing in a virtual space of a computer an arrangement of a work object to be worked on in a real space, a base having a reference point in the real space and for defining the position of the work object, and a projection device that projects a position image of the work object onto the work object in the real space; projecting the position image projected from the projection device onto the work object in the virtual space from the projection device onto the work object in the real space; placing a reference point image associated with the position image in the virtual space at the position of the reference point; projecting a reference point image associated with the position image in the real space from the projection device so as to correspond to the reference point; acquiring at least one captured image by capturing the reference point and the reference point image in the real space with an imaging device; Identifying, by the computer, a first positional relationship between a position of the reference point and a position of the reference point image based on the at least one captured image; determining, by a computer, whether or not the first positional relationship coincides with a second positional relationship between the position of the reference point and the position of the reference point image in the virtual space; detecting by the computer that a deviation of the position image has occurred if the first positional relationship and the second positional relationship do not match; A method for determining a projection position, comprising:

3. The acquiring of the at least one captured image includes: capturing an image of the reference point with a camera to obtain a first captured image; capturing an image of the reference point and the reference point image with the camera to obtain a second captured image; Including, Determining by the computer whether or not there is a match based on a result of analyzing a data set including the first captured image and the second captured image. The projection position determination method according to claim 1 or 2.

4. After the computer detects that a deviation of the position image relative to the work object has occurred, changing a display mode of the position image; The projection position determination method according to claim 1 or 2.

5. the base supports the workpiece; determining by the computer whether the substrate has moved based on signals from a sensor detecting at least one of the substrate movement and the projection device movement; acquiring the at least one captured image when the computer determines that the substrate has moved; The method for determining a projection position according to claim 1 or 2, comprising:

6. A projection position determination device implemented by a computer that reproduces in a virtual space an arrangement of a work object to be worked on in a real space, a base having a reference point in the real space and for determining the position of the work object, and a projection device that projects a position image of the work object onto the work object in the real space, The computer projecting the position image projected from the projection device onto the work object in the virtual space from the projection device onto the work object in the real space; placing a reference point image associated with the position image in the virtual space at the position of the reference point; projecting a reference point image associated with the position image in the real space from the projection device so as to correspond to the reference point; acquiring at least one captured image by capturing the reference point and the reference point image in the real space with an imaging device; Identifying a first positional relationship between a position of the reference point and a position of the reference point image based on the at least one captured image; determining whether or not the first positional relationship coincides with a second positional relationship between the position of the reference point and the position of the reference point image in the virtual space; detecting that a deviation of the position image has occurred if the first positional relationship and the second positional relationship do not match; A projection position determination device including:

Images