Photo guide device
The photography guide device addresses alignment challenges by calculating and displaying a tilt index based on a plane projection transformation matrix, enhancing camera orientation adjustments for precise image matching.
Patent Information
- Application Number
- JP2023542138
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-08-19
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-08-19
AI Technical Summary
Existing technologies face challenges in aligning camera orientations during registration and verification due to projective distortion, particularly with complex shapes like trapezoids and rhombuses, making it difficult to adjust camera positions accurately for individual identification and fixed-point observation.
A photography guide device and method that calculates a plane projection transformation matrix, generates a tilt index based on this matrix, and displays it superimposed on an image to assist users in aligning camera orientations, using visual indicators to guide adjustments.
Enables easy determination of camera orientation adjustments based on projective distortion, improving matching performance by visually guiding users to align camera directions accurately.
Smart Images

Figure 0007586332000001 
Figure 0007586332000002 
Figure 0007586332000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a photography guide device, a photography guide method, and a recording medium. [Background technology]
[0002] Attempts have been made to utilize individual differences in images obtained by photographing an object with a camera for individual authentication and matching (see, for example, Patent Document 1).
[0003] When using the individual differences in images obtained by photographing an object with a camera in this way for individual authentication, if the camera orientation when photographing the object to be registered differs from the camera orientation when photographing the object to be matched, the authentication performance will deteriorate due to the effects of projection distortion (also called trapezoidal distortion). Therefore, it is important to match the camera orientation during registration and matching.
[0004] Patent Documents describing techniques related to the present invention include Patent Document 2 and Patent Document 3.
[0005] In the technology described in Patent Document 2 (hereinafter referred to as the first related technology), when the matching area of the object to be matched is photographed by a camera during matching, a registration image or its contour line (hereinafter referred to simply as the registration image) captured in advance of the matching area by a camera set in a specified orientation and position is superimposed on a through image (also called a preview image or live view image).
[0006] The technology described in Patent Document 3 (hereinafter, referred to as the second related technology) is a technology that supports fixed-point observation in which inspection points of structures such as bridges are photographed by a camera at the same position and orientation every time. In the second related technology, a planar projective transformation matrix (also referred to as a homography matrix) is estimated for a candidate image of a structure photographed by a camera, and a registered image of the same structure photographed in the past by a camera set at a predetermined orientation and position. Then, in the second related technology, a composite image obtained by combining an image obtained by transforming the registered image based on the estimated planar projective transformation matrix with the candidate image is displayed on a display device. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] Patent No. 6217886 [Patent Document 2] Patent No. 6206645 [Patent Document 3] WO / 2020 / 145004 Summary of the Invention [Problem to be solved by the invention]
[0008] According to the first related technology, a user can adjust the camera orientation so that the matching area of the through image and the registered image match, thereby adjusting the camera orientation between registration and matching. However, a user cannot easily determine in what direction the camera should be adjusted so that the matching area of the through image and the registered image match, just by looking at the matching area of the through image and the registered image. Of course, if the original shape of the matching area is known, it is theoretically possible to estimate the adjustment direction of the camera orientation from the direction of the projective distortion of the shape of the matching area of the through image and the direction of the projective distortion of the shape of the registered image superimposed on the through image. However, there are cases where the original shape of the matching area is unknown. Even if it is known, there are shapes such as trapezoids, diamonds, and stars, rather than simple shapes such as rectangles, for which it is difficult to estimate the adjustment direction of the camera orientation from the direction of projective distortion. For such reasons, in the first related technology, it is not easy to adjust the camera orientation at matching so that it matches the orientation at registration.
[0009] On the other hand, consider applying the second related technology in the field of fixed-point observation to photography in individual authentication. In that case, the planar projective transformation matrix of an image of a matching region of an object to be matched taken by a camera at the time of matching and a registered image of a matching region taken in advance by a camera set in a predetermined direction and position is estimated, and a composite image obtained by combining an image obtained by converting the registered image based on the planar projective transformation matrix and an image obtained by photographing the matching region of the object to be matched is displayed on a display device. With such a configuration, a user determines the direction in which to adjust the orientation of the camera based on the shape of the image obtained by converting the registered image based on the planar projective transformation matrix and the original shape of the matching region. However, as described above, there are cases in which the original shape of the matching region is unknown. Even if it is known, there are shapes such as trapezoids, diamonds, and stars, rather than simple shapes such as rectangles, for which it is difficult to estimate the adjustment direction of the camera orientation from the direction of the projective distortion. For such reasons, when the second related technology is applied to photography in individual authentication, it is not easy to adjust the orientation of the camera at the time of matching so that it matches the orientation at the time of registration.
[0010] The present invention provides a photography guide device that solves the above-mentioned problems. [Means for solving the problem]
[0011] A photography guide device according to one aspect of the present invention includes: A calculation means for calculating a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photographing device and a second image obtained by photographing the plane with a second photographing device; an acquisition means for acquiring a second figure obtained by transforming a first figure, which is virtually drawn in a predetermined orientation and a predetermined shape on the first image, based on the planar projective transformation matrix; a generating means for generating an inclination indicator representing an inclination of an imaging direction of the second imaging device with respect to the first imaging device based on the second graphic, and generating a display item including the inclination indicator; a display control means for displaying the display item superimposed on the second image; The device is configured to include:
[0012] A photography guide method according to another aspect of the present invention includes: Calculating a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photographing device and a second image obtained by photographing the plane with a second photographing device; acquiring a second figure obtained by transforming a first figure, which is virtually drawn in a predetermined orientation and a predetermined shape on the first image, based on the planar projective transformation matrix; generating an inclination indicator representing an inclination of an imaging direction of the second imaging device relative to the first imaging device based on the second graphic, and generating a display item including the inclination indicator; displaying the display item superimposed on the second image; It is structured as follows.
[0013] A computer-readable recording medium according to another aspect of the present invention includes: On the computer, A process of calculating a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photographing device and a second image obtained by photographing the plane with a second photographing device; A process of acquiring a second figure obtained by transforming a first figure, which is virtually drawn in a predetermined orientation and a predetermined shape on the first image, based on the planar projective transformation matrix; generating an inclination indicator representing an inclination of a shooting direction of the second photographing device relative to the first photographing device based on the second figure, and generating a display item including the inclination indicator; a process of displaying the display item superimposed on the second image; The recording medium is configured to record a program for causing the recording medium to perform the above steps. Effect of the Invention
[0014] By having the above-described configuration, the present invention enables a user to easily determine, from the direction of projection distortion of a figure of a specified shape, to what direction the camera should be adjusted so that the camera orientation during matching can be matched to the orientation during registration. [Brief description of the drawings]
[0015] [Figure 1] 1 is a block diagram of an individual identification system according to a first embodiment of the present invention. [Diagram 2] 1 is a schematic diagram showing an example of a relative positional relationship between a camera and a product when photographing the product in a first process processing device of a production line that constitutes an individual identification system according to a first embodiment of the present invention. FIG. [Diagram 3] 1 is a diagram showing an example of a registered image captured in a first process processing device of a production line constituting an individual identification system according to a first embodiment of the present invention. FIG. [Figure 4] 2 is a schematic diagram showing an example of a relative positional relationship between a camera and a product when photographing the product in a second process processing device of a production line that constitutes the individual identification system according to the first embodiment of the present invention. FIG. [Diagram 5] 4 is a diagram showing an example of a collation image captured in a second process processing device of a production line constituting the individual identification system according to the first embodiment of the present invention. FIG. [Figure 6] 10 is a diagram showing another example of a collation image captured in a second process processing device of a production line constituting an individual identification system according to the first embodiment of the present invention. FIG. [Figure 7] 1 is a block diagram of an individual identification device according to a first embodiment of the present invention. [Figure 8] 3 is a diagram showing an example of a guide image presented to a user by the individual identification device according to the first embodiment of the present invention. FIG. [Figure 9] 2 is a diagram showing an example of the configuration of a product DB in the individual identification device according to the first embodiment of the present invention; FIG. [Figure 10] 5 is a flowchart showing an example of an operation of a photography guide section in the individual identification device according to the first embodiment of the present invention. [Figure 11] 13 is a flowchart showing an example of detailed processing of step S5. [Figure 12] FIG. 13 is a diagram showing an example of a reference image in which a reference figure is drawn. [Figure 13]13 is a flowchart showing an example of a process for calculating a fourth index. [Figure 14] 1A and 1B are diagrams showing examples of a reference figure, a figure obtained by projective transformation of the reference figure, a figure in which the projective distortion of the reference figure is exaggerated, and a figure obtained by rotating the reference figure. [Figure 15] 13 is a flowchart showing an example of a process for coloring each side of a graphic according to a projection distortion. [Figure 16] 5 is a flowchart showing an example of the operation of a registration unit of the individual identification device according to the first embodiment of the present invention. [Figure 17] 5 is a flowchart showing an example of the operation of a matching unit of the individual identification device according to the first embodiment of the present invention. [Figure 18] 5 is a flowchart showing an example of processing performed by a matching unit of the individual identification device according to the first embodiment of the present invention to match a matching image with a registered image. [Figure 19] FIG. 11 is a block diagram of a photography guide device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Next, an embodiment of the present invention will be described in detail with reference to the drawings. [First embodiment] 1 is a block diagram of an individual identification system 10 to which a photography guide device according to a first embodiment of the present invention is applied. The individual identification system 10 is composed of an individual identification device 100 and a production line 200.
[0017] The production line 200 is a line for producing industrial products. The industrial products to be produced are not particularly limited. For example, the industrial products may be electrical products such as printed wiring boards, metal products such as screws and washers, food products such as canned beer, and medicines such as drugs. The production line 200 is configured to include at least two process processing devices, namely, a first process processing device 201 and a second process processing device 202, and conveyors 203 to 205 such as conveyors. For example, in the case of an SMT (Surface Mount Technology) line for printed wiring boards, the first process processing device 201 is, for example, a solder printing machine that applies solder onto a board, and the second process processing device 202 is a component mounting machine that mounts multiple electronic components at predetermined positions on the board after solder printing. Alternatively, the first process processing device 201 may be a component mounting machine, and the second process processing device 202 may be a reflow machine that flows the board through a thermostatic chamber to melt and solidify the solder in order to fix the mounted components to the board. The conveyors 203 to 205 are means for conveying products one by one from the upstream side to the downstream side, in the order of the first process processing device 201 and the second process processing device 202.
[0018] The production line 200 has at least two phases, an adjustment phase and an operation phase. The adjustment phase is performed prior to the operation phase, and processing such as adjustment of the camera position and attitude described below is performed. In the operation phase, processing is performed to produce a large amount of products in an assembly line manner. In this operation phase, a plurality of products that have passed through the first process processing device 201 are then carried into the second process processing device 202. However, a buffer unit that temporarily accumulates a plurality of products in random order may be provided between the first process processing device 201 and the second process processing device 202. Therefore, the plurality of products that have passed through the first process processing device 201 do not necessarily pass through the second process processing device 202 while maintaining the passing order. Therefore, individual identification is performed to identify which product that has passed through the first process processing device 201 is the same as the product carried into the second process processing device 202.
[0019] The individual identification device 100 is an information processing device that manages individual products produced on a production line 200 for manufacturing process management, quality control, shipping management, sales management, and the like. The individual identification device 100 is connected to the first process processing device 201 and the second process processing device 202 so that they can communicate with each other by wire or wirelessly. The first process processing device 201 and the second process processing device 202 are provided with cameras 211 and 212 for photographing products 241 and 242 carried into the first process processing device 201 and the second process processing device 202 for individual identification, camera position and attitude adjusters 221 and 222 for adjusting the positions and attitudes of the cameras 211 and 212, and illuminators 231 and 232 for irradiating the products with illumination for photographing. The cameras 211 and 212 may be, for example, visible light and color cameras or black and white cameras equipped with a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor having a pixel capacity of about several million pixels. The camera position / attitude adjusters 221 and 222 may be, for example, a pan-tilt head, a six-axis robot arm, etc. The illuminators 231 and 232 may be, for example, a ring light, a dome light, etc.
[0020] FIG. 2 is a schematic diagram showing an example of the relative positional relationship between the camera 211 and the product 241 when photographing the product in the first process processing device 201. The camera 211 is placed at a position where the top surface of the product 241 can be photographed from above. The position of the camera 211 can be adjusted by the camera position / attitude adjuster 221 in the left / right direction of the paper, the front / back direction of the paper, and the up / down direction of the paper. In addition, the attitude of the camera 211 can be adjusted by the camera position / attitude adjuster 221 in the left / right direction (pan direction), the up / down direction (tilt direction), and the camera rotation direction (roll direction). When the product 241 is carried into the first process processing device 201 by the conveyor 203, the conveyance is temporarily stopped at the point where the leading end of the product 241 reaches a predetermined position, and the product 241 is brought to a stationary state. The user adjusts the position / attitude of the camera 211 by the camera position / attitude adjuster 221 so that the random pattern formed on the top surface of the stationary product 241 can be photographed as a light / dark pattern. At this time, tools such as a spirit level or a sensor may be used as necessary to determine the optimal position and orientation of the camera 211. Then, when the user completes the adjustment using the camera position and orientation adjuster 221, the camera 211 is fixed in that position and orientation. The camera 211 acquires a registration image, which is an image obtained by capturing the collation area of the product 241 in a stationary state with the position and orientation fixed.
[0021] 3 shows an example of a registered image obtained by photographing a matching region of a product 241 by a camera 211. In this example, the product 241 is a metallic washer having a ring shape in a plan view. The matching region is the entire upper surface of the product 241. The upper surface of the product 241 forms approximately the same plane. In general, the surfaces of multiple products 241 manufactured through the same manufacturing process have fine patterns unique to each product 241, as well as fine patterns and non-fine patterns common to the multiple products 241.
[0022] On the other hand, FIG. 4 is a schematic diagram showing an example of the relative positional relationship between the camera 212 and the product 242 when photographing the product in the second process processing device 202. The camera 212 is placed at a position where the top surface of the product 242 can be photographed from above. The position of the camera 212 can be adjusted by the camera position / attitude adjuster 222 in the left / right direction of the paper, the front / back direction of the paper, and the up / down direction of the paper. In addition, the attitude of the camera 212 can be adjusted by the camera position / attitude adjuster 222 in the left / right direction (pan direction), the up / down direction (tilt direction), and the camera rotation direction (roll direction). When the product 242 is carried into the second process processing device 202 by the conveyor 204, the conveyance is temporarily stopped and the product 242 is brought to a stationary state when the leading end of the product 242 reaches a predetermined position. The camera 212 photographs the top surface of the stationary product 242 from above. At this time, the user initially sets the position and orientation of the camera 212 using the camera position and orientation adjuster 222 so that the position and orientation of the camera 212 relative to the product 242 is the same as the position and orientation of the camera 211 relative to the product 241 .
[0023] However, due to various factors such as manual camera position setting errors and individual differences in the camera position and attitude adjusters, the initially set position and attitude of the camera 212 with respect to the product 242 in the second process processing device 202 may differ from the position and attitude of the camera 211 with respect to the product 241 in the first process processing device 201. For example, even if the same amount of movement as that given to the camera position and attitude adjuster 221 is given to the camera position and attitude adjuster 222 in order to set the position of the camera 212 to the same position as the camera 211, the camera 212 may not actually be set to the same position due to individual differences in the camera position and attitude adjusters 221 and 222. Also, even if the same amount of rotation as that given to the camera position and attitude adjuster 221 is given to the camera position and attitude adjuster 222 in order to set the attitude of the camera 212 to the same attitude as the camera 211, the camera 212 may not actually be set to the same amount of rotation due to individual differences in the camera position and attitude adjusters 221 and 222.
[0024] If the position and orientation of the camera 212 differs from that of the camera 211, the entire upper surface of the product 241 may fit within the image of the camera 211, whereas the entire upper surface of the product 242 may not fit within the image of the camera 212, with some parts cut off, as shown in FIG. 5, for example. If the position and orientation of the camera 212 differs from that of the camera 211, whereas the image of the upper surface of the product 241 may not have projection distortion in the image of the upper surface of the product 242 in the camera 212, as shown in FIG. 6, for example. A positional deviation in which the entire matching area does not fit within the image as shown in FIG. 5 can be visually recognized by the position on the product image, and therefore the user can deal with it relatively easily. However, differences in camera orientation as shown in FIG. 6 are difficult to visually recognize when the difference in orientation is 10 degrees or less. Also, even a slight projection distortion that causes an image difference of a few pixels can cause deterioration in matching performance. Therefore, it is important to support the user so that the position and orientation of the camera 212 can be adjusted so that there is no difference in camera orientation that causes projection distortion. Therefore, the individual identification device 100 is configured to present to the user information indicating the difference between the position and orientation of the camera 212 and the position and orientation of the camera 211. The individual identification device 100 will be described in detail below.
[0025] Fig. 7 is a block diagram of the individual identification device 100. Referring to Fig. 7, the individual identification device 100 is communicably connected to a first process processing device 201 and a second process processing device 202. The individual identification device 100 is also configured to include a communication I / F (interface) unit 110, an operation input unit 120, a screen display unit 130, a storage unit 140, and an arithmetic processing unit 150.
[0026] The communication I / F unit 110 is composed of a data communication circuit, and is configured to perform data communication with an external device wirelessly or by wire. The operation input unit 120 is composed of devices such as a keyboard and a mouse, and is configured to detect an operator's operation and output it to the arithmetic processing unit 150. The screen display unit 130 is composed of devices such as an LCD (Liquid Crystal Display), and is configured to display various information on the screen in response to instructions from the arithmetic processing unit 150.
[0027] The storage unit 140 is composed of a storage device such as a hard disk or a memory, and is configured to store processing information and a program 141 required for various processes in the arithmetic processing unit 150. The program 141 is a program that is read into the arithmetic processing unit 150 and executed to realize various processing units, and is read in advance from an external device or recording medium (not shown) via a data input / output function such as the communication I / F unit 110 and stored in the storage unit 140. Main processing information stored in the storage unit 140 includes a reference image 142, a guide image 143, and a product DB (database) 144.
[0028] Reference image 142 is an image of a collation area of a product for position / attitude adjustment carried into first process processing device 201 (hereinafter referred to as reference product) captured by camera 211 whose position / attitude has been adjusted. The reference product is one of a plurality of products produced in production line 200. The reference product may be subjected to predetermined processes in first process processing device 201 and second process processing device 202, like other products produced in production line 200, or these processes may be omitted. For example, when the reference product is a metal washer having a ring shape in a plan view, an example of reference image 142 is the same as the example of the registered image shown in FIG. 3.
[0029] The guide image 143 is an image in which a display item indicating the degree of deviation of the position and orientation of the camera 212 relative to the camera 211 is superimposed on an image captured by the camera 212 of the comparison area of the reference product brought into the second process processing device 202.
[0030] 8 shows an example of the guide image 143. In this example, the guide image 143 has display items including a first index 1431, a second index 1432, a third index 1433, and a fourth index 1434 superimposed on an image captured by the camera 212. The first index 1431 is also called a non-tilted index. The second index 1432 is also called an arrow index. The third index 1433 is also called a text index. The fourth index 1434 is also called a tilted index.
[0031] The first index 1431 is made up of a square figure, and the position and orientation of the square represent the positional deviation and rotational deviation of the camera 212 with respect to the camera 211. For example, if the camera 212 has no rotational deviation, the square is drawn in an upright state. On the other hand, if the camera 212 has a rotational deviation, the square is drawn in a state rotated from the upright state according to the amount of rotational deviation. The second index 1432 is made up of one arrow, and the length and orientation of the arrow represent the positional deviation and rotational deviation of the camera 212 with respect to the camera 211. For example, if the camera 212 has no positional deviation, the length of the arrow is zero, and the arrow is not actually drawn. On the other hand, if the camera 212 has a positional deviation, an arrow with a length according to the amount of positional deviation is drawn. At this time, the tip of the arrow is drawn at the same location as the center of the square that constitutes the first index 1431. Therefore, if the camera 212 has a positional deviation, the square that constitutes the first index 1431 is drawn shifted by the length and orientation of the arrow. The third index 1433 consists of three lines of text, the first and second lines representing the amount of positional deviation in the X-axis and Y-axis directions of camera 212 relative to camera 211, and the third line representing the rotational deviation angle of camera 212 relative to camera 211. In this way, the first index 1431, the second index 1432, and the third index 1433 all represent the positional deviation and rotational deviation of camera 212 relative to camera 211.
[0032] The fourth index 1434 is made up of a quadrangle shape, and represents the inclination of the shooting direction of the camera 212 with respect to the camera 211. When there is no deviation in the inclination of the shooting direction of the camera 212 with respect to the camera 211, the quadrangle constituting the fourth index 1434 is drawn as a square with each side colored in a first color (e.g., green). On the other hand, when there is a deviation in the inclination of the shooting direction of the camera 212 with respect to the camera 211, the quadrangle is drawn as a trapezoid having a projection distortion oriented according to the deviation direction of the inclination. Furthermore, of the four sides of the trapezoid, the side having the same length as the opposing side is colored in a first color (e.g., green), the side having a shorter length than the opposing side is colored in a second color (e.g., red), and the side having a longer length than the opposing side is colored in a third color (e.g., blue).
[0033] However, the indices constituting the display items of the guide image 143 are not limited to the above. For example, the display items may include only the fourth index 1434. Alternatively, the display items may include the fourth index 1434 and any one or two of the first to third indexes. Alternatively, the display items may include other indices different from the first to fourth indices. As an example of the other indices, an index constituted by an arrow indicating the direction of the perspective distortion of the fourth index 1434 is considered.
[0034] The product DB 144 is a database that stores information such as images of individual products mass-produced on the production line 200. FIG. 9 shows an example of the configuration of the product DB 144. The product DB 144 in this example is composed of multiple entries, and each entry is composed of the following items: product ID, manufacturing information and registered image of the first process, and manufacturing information and match image of the second process. In the item of product ID, an ID such as a serial number assigned to the product is set. In the items of manufacturing information and registered image of the first process, manufacturing information consisting of the work history of the first process for the product, the type of parts used, the lot number, etc., and a registered image which is an image of the product photographed by the camera 211 are set. In the items of manufacturing information and match image of the second process, manufacturing information consisting of the work history of the second process for the product, the type of parts used, the lot number, etc., and a match image which is an image of the product photographed by the camera 212 are set.
[0035] The arithmetic processing unit 150 has a processor such as an MPU and its peripheral circuits, and is configured to read and execute the program 141 from the storage unit 140 to realize various processing units by making the above hardware and the program 141 cooperate with each other. The main processing units realized by the arithmetic processing unit 150 are an imaging guide unit 151, a registration unit 152, and a matching unit 153.
[0036] The photography guide unit 151 is configured to support the user in adjusting the positions and attitudes of the cameras 211 and 212 in the adjustment phase of the production line 200. For example, the photography guide unit 151 is configured to present information indicating the difference between the position and attitude of the camera 212 and the position and attitude of the camera 211 to the user through the screen display unit 130.
[0037] The registration unit 152 is configured to acquire images of products carried into the first process processing device 201 during the operation phase of the production line 200, and to store the registered images and the like in the product DB 144.
[0038] The matching unit 153 is configured to acquire an image of the product brought into the second process processing device 202 during the operation phase of the production line 200, and perform individual identification to identify which product that passed through the first process processing device 201 the product brought into the second process processing device 202 is, by matching it with registered images of products that passed through the first process processing device 201 recorded in the product DB 144.
[0039] Next, a description will be given of the operation of the individual identification device 100. First, a description will be given of the operation of the individual identification device 100 in the adjustment phase.
[0040] FIG. 10 is a flowchart showing an example of a process executed by the photographing guide unit 151 in the adjustment phase. Referring to FIG. 10, first, the photographing guide unit 151 adjusts the position and attitude of the camera 211 by the camera position and attitude adjuster 221 according to an instruction input by the user through the operation input unit 120 (step S1). Since the optimal collation area of a product differs depending on the type of product, the user performs the adjustment by inputting the position and attitude information of the camera 211 according to the type of product to be manufactured through the operation input unit 120. Here, according to the position and attitude information input by the user, the camera 211 is adjusted and fixed in a position and attitude such that the entire upper surface of the product 241 fits within the image from almost directly above the product 241, as described with reference to FIG. 3.
[0041] When the adjustment of the position and attitude of the camera 211 is completed, the user places the reference product on the conveyor 203 and conveys it toward the first process processing device 201. When the reference product is conveyed into the first process processing device 201, it is detected by a sensor (not shown) installed in the first process processing device 201, and when the reference product reaches a predetermined position, the conveyance is temporarily stopped and the reference product is made stationary. In addition, the first process processing device 201 notifies the individual identification device 100 by communication that the product has been conveyed into the first process processing device 201. When the photographing guide unit 151 detects the conveyance of the reference product by this communication, it acquires an image of the reference product by the camera 211 as a reference image 142 and stores it in the storage unit 140 (step S2). At this time, the photographing guide unit 151 may display the reference image 142 on the screen display unit 130 so that the user can confirm it.
[0042] In this manner, by the operation of the photographing guide unit 151, the position and posture of the camera 211 of the first process processing device 201 of the production line 200 is adjusted, and a reference image 142, which is an image of the reference product photographed by the camera 211 in the adjusted position and posture, is stored in the memory unit 140.
[0043] Next, the photography guide unit 151 adjusts the position and attitude of the camera 212 by the camera position and attitude adjuster 222 according to instructions input by the user through the operation input unit 120 (step S3). When adjusting the position and attitude of the camera 212 for the first time, the user may, for example, input from the operation input unit 120 position and attitude information that will result in the same position and attitude as the adjusted position and attitude of the camera 211, and perform the adjustment.
[0044] After adjusting the position and attitude of the camera 212, the user removes the reference product carried into the first process processing device 201 from the first process processing device 201, places it on the conveyor 204, and conveys it toward the second process processing device 202. When the reference product is carried into the second process processing device 202 by the conveyor 204, it is detected by a sensor (not shown) installed in the second process processing device 202, and when the reference product reaches a predetermined position, the conveyance is temporarily stopped and the reference product is made stationary. In addition, the second process processing device 202 notifies the individual identification device 100 by communication that the reference product has been carried into the second process processing device 202. When the photographing guide unit 151 detects the carrying in of this reference product, it photographs an image of the reference product by the camera 212 and acquires the photographed image (step S4).
[0045] Next, the photographing guide unit 151 generates a guide image 143 as illustrated in FIG. 8 from the reference image 142 acquired in step S2 and the photographed image (hereinafter, referred to as photographed image SG) acquired in the immediately preceding step S4, and displays it on the screen display unit 130 (step S5). The user visually checks the displayed guide image 143 and checks whether the position and orientation of the camera 212 are not shifted relative to the camera 211. If the user does not need to readjust the position and orientation of the camera 212, he or she inputs an instruction indicating OK from the operation input unit 120, for example, and otherwise inputs an instruction indicating NG. When the instruction of OK is input, the photographing guide unit 151 ends the process shown in FIG. 10. On the other hand, when the instruction of NG is input, the photographing guide unit 151 returns to the process of step S3 and repeats the same process as the above-mentioned process again.
[0046] Next, a detailed description will be given of step S5 for generating the guide image 143. Fig. 11 is a flowchart showing an example of detailed processing of step S5.
[0047] First, the photographing guide unit 151 calculates a planar projection transformation matrix Mh of the reference image 142 acquired in step S2 and the photographed image SG acquired in the immediately preceding step S4 (steps S11 to S12). Specifically, first, the photographing guide unit 151 calculates a rigid transformation matrix Mr of the reference image 142 and the photographed image SG (step S11). The reference image 142 and the photographed image SG are images obtained by photographing the same reference product. On the surface of the reference product, there is a fine pattern specific to the individual product, and there are also fine patterns and non-fine patterns common to a plurality of products resulting from the product mold. Therefore, the reference image 142 and the photographed image SG have the same light and dark patterns due to the same fine patterns and non-fine patterns. The photographing guide unit 151 calculates the rigid transformation matrix Mr of the reference image 142 and the photographed image SG based on such the same light and dark patterns. For example, the photographing guide unit 151 calculates the rigid transformation matrix Mr by using a phase-only correlation method. Specifically, the photographing guide unit 151 first performs FFT transformation on each of the reference image 142 and the photographed image SG, and then performs logarithmic polar coordinate transformation on each of them. Next, the photographing guide unit 151 calculates the rotation angle and the amount of enlargement / reduction between the reference image 142 and the photographed image SG from the frequency spectrum images of the reference image 142 and the photographed image SG after the logarithmic polar coordinate transformation, using phase-only correlation. Next, the photographing guide unit 151 corrects the rotational deviation and the enlargement / reduction deviation between the reference image 142 and the photographed image SG using the calculated rotation angle and the amount of enlargement / reduction, and calculates the positional deviation amount (parallel movement amount) between the reference image 142 and the photographed image SG from the frequency spectrum images of the reference image 142 and the photographed image SG after the correction, using phase-only correlation. The rigid transformation matrix Mr is uniquely determined from the calculated positional deviation amount, rotation angle, and enlargement / reduction amount.
[0048] Next, the photography guide unit 151 obtains an image (hereinafter, referred to as a photographed image SG') obtained by performing rigid transformation on the photographed image SG using the calculated rigid transformation matrix Mr (step S12). The photographed image SG' becomes an image that almost overlaps with the reference image 142. Next, the photography guide unit 151 calculates a planar projection transformation matrix Mh' between the two images from the reference image 142 and the photographed image SG' (step S13). Specifically, first, the photography guide unit 151 extracts a plurality of feature points from each of the reference image 142 and the photographed image SG' using any method such as SIFT or Random. Next, the photography guide unit 151 associates a plurality of feature points between the reference image 142 and the photographed image SG' using the Lucas-Kanade method. Next, the photography guide unit 151 calculates a planar projection transformation matrix Mh' between the reference image 142 and the photographed image SG' using the coordinates of the associated feature points.
[0049] The projective transformation matrix Mh of the reference image 142 and the captured image SG is given by the product of the rigid transformation matrix Mr and the projective transformation matrix Mh'. The projective transformation matrix Mh can be decomposed into a similarity transformation matrix, an affine transformation matrix, and a projective transformation matrix. Therefore, the translation, rotation, and scaling transformations using the rigid transformation matrix Mr are the same as the translation, rotation, and scaling transformations using the similarity transformation matrix and the affine transformation matrix obtained by decomposing the projective transformation matrix Mh. Also, the transformation using the projective transformation matrix Mh' is the same as the transformation using the projective transformation matrix obtained by decomposing the projective transformation matrix Mh.
[0050] In the above, the planar projective transformation matrix Mh of the reference image 142 and the photographed image SG is divided and calculated into a rigid transformation matrix Mr of the reference image 142 and the photographed image SG, and a planar projective transformation matrix Mh' of the reference image 142 and the photographed image SG' transformed by the rigid transformation matrix Mr. However, the method of calculating the planar projective transformation matrix Mh is not limited to the above. For example, in an environment in which it is guaranteed that the rotational deviation, positional deviation, and enlargement / reduction deviation between the reference image 142 and the photographed image SG are small, the planar projective transformation matrix Mh may be calculated directly from the reference image 142 and the photographed image SG. That is, the photographing guide unit 151 first extracts a plurality of feature points from each of the reference image 142 and the photographed image SG using any method such as SIFT or Random. Next, the photographing guide unit 151 associates a plurality of feature points between the reference image 142 and the photographed image SG using the Lucas-Kanade method. Next, the photography guide unit 151 calculates a planar projective transformation matrix Mh between the reference image 142 and the photographed image SG using the coordinates of the associated feature points.
[0051] Next, the photographing guide unit 151 virtually draws a reference figure 1421 of a predetermined shape and orientation on the plane of the reference product in the reference image 142 (step S14). FIG. 12 shows an example of the reference image 142 on which the reference figure 1421 is drawn. In this example, the reference figure 1421 is a square. In this example, the reference figure 1421 is drawn in an upright orientation so that the center coincides with the center of the reference product. The upright orientation is an orientation in which each side of the reference figure 1421 is parallel to the horizontal direction (X axis) or vertical direction (Y axis) of the reference image 142. The position of each vertex of the reference figure 1421 is specified by the XY coordinate values of the reference image 142. However, the reference figure 1421 is not limited to a square, and may be a figure having another shape such as a rectangle. In addition, the orientation in which the reference figure 1421 is established is not limited to the above.
[0052] Next, the photography guide unit 151 calculates a first index 1431 consisting of a square that represents a positional deviation and a rotational deviation depending on a position and an orientation by applying rigid transformation by the rigid transformation matrix Mr calculated in step S11 to the reference figure 1421 (step S15). The size of the square that constitutes the first index 1431 is determined by the size of the reference figure 1421 and the scaling component of the rigid transformation matrix Mr. Note that, in order to visualize the amount of deviation due to the scaling of the camera 212, a square without any amount of deviation due to the scaling may be created as one index and included in the display items.
[0053] Next, the photography guide unit 151 calculates a second index 1432 representing the positional deviation and rotational deviation by the length and direction of an arrow from the positional deviation amount and rotational angle that are components of the rigid transformation matrix Mr calculated in step S11 (step S16).
[0054] Next, the photography guide unit 151 calculates a third index 1433 representing the positional deviation and rotational deviation by text from the positional deviation amount and rotational angle that are components of the rigid transformation matrix Mr calculated in step S11 (step S17).
[0055] Next, the photographing guide unit 151 calculates the fourth index 1434 (step S18). Details of this step S18 will be described later. Next, the photographing guide unit 151 acquires the guide image 143 obtained by superimposing the display items including the first to fourth indexes on the photographed image SG (step S19). Next, the photographing guide unit 151 displays the guide image 143 on the screen display unit 130 (step S20).
[0056] Fig. 13 is a flow chart showing an example of a process for calculating the fourth index 1434. Referring to Fig. 13, the photography guide unit 151 first calculates a figure obtained by projectively transforming the reference figure 1421 using the planar projection transformation matrix Mh' calculated in step S13 (step S31). Fig. 14 shows an example of the reference figure 1421 and a figure 1422 obtained by projectively transforming the reference figure 1421. The reference figure 1421 has vertices a 1 ,a 2 ,a3 ,a 4 The figure 1422 after projective transformation is a square with vertex b 1 ,b 2 ,b 3 ,b 4 It is a trapezoid with
[0057] Next, the photography guide unit 151 calculates a figure in which the perspective distortion of the figure 1422 after the projective transformation calculated in step S31 is exaggerated (step S32). The figure 1423 shown in Fig. 14 is an example of a figure in which the perspective distortion of the figure 1422 is exaggerated. The photography guide unit 151 calculates a figure in which the perspective distortion is exaggerated, for example, by using the following method.
[0058] First, the photographing guide portion 151 is located at the vertex b of the figure 1422. n For each vertex a of the reference figure 1421, n The difference between the X coordinates (b n Xa n X) and Y coordinate value difference (b n Ya n Y) is calculated. Next, the photography guide unit 151 multiplies the difference by e, where e is a preset exaggeration coefficient. Next, the photography guide unit 151 calculates the difference multiplied by e at the corresponding vertex a of the reference figure 1421. n Adding this to the coordinates of the exaggerated perspective distortion figure 1423, vertex c 1 ,c 2 ,c 3 ,c 4 Calculate the coordinates of vertex c n The X and Y coordinate values are given by the following equations: c n X=(b n Xa n X)×e+a n X (1) c n Y=(b n Ya n Y)×e+a n Y (2)
[0059] 13 again, the photography guide unit 151 calculates a figure obtained by applying the transformation based on the rigid transformation matrix Mr calculated in step S11 to the figure 1423 in which the perspective distortion calculated in step S32 has been exaggerated (step S33). The figure 1424 shown in Fig. 14 is an example of a figure obtained by applying the above transformation to the figure 1423. Note that the photography guide unit 151 may calculate the figure 1424 obtained by applying only the rotational component of the rigid transformation matrix Mr to the figure 1423 in step S33.
[0060] Next, the photography guide unit 151 calculates a figure obtained by applying coloring to each side of the figure 1424 calculated in step S33 according to the projection distortion, as a fourth index 1434 (step S34).
[0061] 15 is a flowchart showing an example of a process for coloring each side of a figure according to the projection distortion. Referring to FIG. 15, the photographing guide unit 151 first calculates the length of each side h of a figure 1422 obtained by projectively transforming a reference figure 1421 using the following equation: 1 ,h 2 ,h 3 ,h 4 The length of is calculated (step S41). h n =SQRT((b n Xb n+1 X) 2 +(b n Yb n+1 Y) 2 ) ···(3)
[0062] Next, the photographing guide unit 151 calculates each side h 1 ,h 2 ,h 3 ,h 4 The difference in length between the opposite sides is multiplied by a predetermined exaggeration coefficient f, and the gamma-corrected value g is obtained. n is calculated (step S42). g n =((h n -h n+2 )×f / 255) 1 / gamma ×255 (4)
[0063] Next, the photographing guide unit 151 calculates the hue: g in the HLS color space for the sides of the figure 1424 that correspond one-to-one to the sides of the figure 1422. n The sides are colored with a color corresponding to +75, brightness: 128, and saturation: 255 (step S43). Here, the range of hue, brightness, and saturation is 0 to 255. A color with a hue of 75 is green. When the hue is smaller than 75, it becomes red, and when it is larger, it becomes blue. From the above formula, the side that is the same length as the opposite side is colored green, the side that is shorter than the opposite side is colored red, and the side that is longer than the opposite side is colored blue.
[0064] The above is the operation of the individual identification device 100 in the adjustment phase.
[0065] Next, a description will be given of the operation of the operation phase of the individual identification device 100. Fig. 16 is a flowchart showing an example of the operation of the registration unit 152 in the operation phase of the individual identification device 100. Fig. 17 is a flowchart showing an example of the operation of the matching unit 153 in the operation phase of the individual identification device 100.
[0066] First, the operation of the registration unit 152 in the operation phase of the individual identification device 100 will be described with reference to FIG. 16. In the operation phase, when a product is carried into the first process processing device 201, it is detected by a sensor (not shown) installed in the first process processing device 201, and when the product reaches a predetermined position, the transport is temporarily stopped and the product is made stationary. In addition, the first process processing device 201 notifies the individual identification device 100 by communication that the product has been carried into the first process processing device 201. When the registration unit 152 detects the carrying-in of the product by this communication (YES in step S51), it acquires an image of the product as a registered image (step S52). Next, the registration unit 152 stores the registered image and the manufacturing information of the product acquired from the first process processing device 201 in the product DB 144 of the storage unit 140 (step S53). The above operation is repeated every time a product is carried into the first process processing device 201.
[0067] Next, the operation of the matching unit 153 in the operation phase will be described. When the product is carried into the second process processing device 202 by the conveyor 204, it is detected by a sensor (not shown) installed in the second process processing device 202, and when the product reaches a predetermined position, the conveyance is temporarily stopped and the product is made stationary. In addition, the second process processing device 202 notifies the individual identification device 100 by communication that the product has been carried into the second process processing device 202. When the matching unit 153 detects the carrying in of the product (step S61), it acquires an image of the product as a matching image (step S52). Next, the matching unit 153 matches the above matching image with one or more registered images stored in the product DB 144, and calculates a matching score (step S63). Here, the matching unit 153 uses registered images registered in the product DB 144 that have not yet been successfully matched with the matching image of the second process processing device, as one or more registered images to be matched with the above matching image. For example, when the product DB 144 is in the state shown in FIG. 9 at the time of matching the above match images, the registered image G101 has been successfully matched with the match image G201, and so is matched in order with the registered images G102 to G108 which have not been successfully matched.
[0068] When there is a registered image whose matching score with the match image is greater than the threshold (YES in step S64), the matching unit 153 determines that the product related to the match image is the same as the product related to the registered image whose matching score is greater than the threshold (step S65). In this case, the matching unit 153 stores the match image and the manufacturing information acquired from the second process processing device 202 in the product DB 144 in association with the matched registered image (step S66). Then, the process returns to step S61 and the same process as described above is repeated.
[0069] On the other hand, if there is no registered image in the product DB 144 whose matching score with the matching image is greater than the threshold value (NO in step S64), the matching unit 153 performs error processing (step S67), and then returns to step S61 to repeat the same processing as described above.
[0070] Next, step S63 of the collation unit 153 will be described in detail.
[0071] There are various possible specific methods for the matching unit 153 to match two images, the match image and the registered image. For example, the matching unit 153 may compare the two images themselves, or may compare feature amounts obtained by performing some kind of transformation on the two images. For example, the matching unit 153 may compare frequency spectrum images obtained by performing frequency transformation such as Fourier transform on each of the two images. Alternatively, the matching unit 153 may first perform frequency transformation such as Fourier transform on each of the two images to convert them into frequency spectrum images, and then compare polar coordinate images (Fourier-Mellin feature images) obtained by performing polar coordinate transformation or log-polar coordinate transformation on the frequency spectrum images. Alternatively, the matching unit 153 may first perform frequency transformation such as Fourier transform on each of the two images to convert them into frequency spectrum images, and then perform polar coordinate transformation or log-polar coordinate transformation on the frequency spectrum images to convert them into Fourier-Mellin features, and further compare phase images obtained by performing frequency transformation such as Fourier transform on the Fourier-Mellin features. An example of a matching method performed by matching unit 153 will be described below.
[0072] FIG. 18 is a flowchart showing an example of a process in which the matching unit 153 matches the match image with the registered image. Referring to FIG. 18, the matching unit 153 first performs a discrete Fourier transform on the match image and the registered image to obtain a frequency spectrum image of the match image and a frequency spectrum image of the registered image (step S71). When there are a plurality of registered images, the matching unit 153 obtains a frequency spectrum image from each of the registered images. Next, the matching unit 153 calculates a normalized cross power spectrum between the frequency spectrum image of the match image and the frequency spectrum image of the registered image for each registered image (step S72). Next, the matching unit 153 performs an inverse Fourier transform on the normalized cross power spectrum for each registered image to calculate a correlation coefficient map (step S73). Next, the matching unit 153 calculates a matching score indicating the similarity between the match image and the registered image from the correlation coefficient map for each registered image (step S74). Next, the matching unit 153 matches the match image with the registered image based on the matching score calculated for each registered image (step S75). For example, if the best score among the multiple matching scores calculated for each registered image is equal to or greater than a predetermined threshold, matching unit 153 determines that the match image matches (is identical to) the registered image with the best matching score. On the other hand, if the best matching score is less than the threshold, matching unit 153 determines that the match image does not match (is not identical to) any registered image.
[0073] In this manner, in the operation phase, under the cameras 211, 212 whose positions and attitudes have been adjusted in the adjustment phase, a registered image of the product brought into the first process processing device 201 and a matching image of the product brought into the second process processing device 202 are acquired and matched, and individual products are identified.
[0074] As described above, according to this embodiment, the user can easily determine, from the direction of the projection distortion of the figure constituting the fourth index 1434 displayed on the guide image 143, in what direction the camera 212 should be adjusted to match the direction of the registration camera 211.
[0075] In this embodiment, since the projection distortion of the figure constituting the fourth index 1434 is exaggerated, the user can easily check the direction of the projection distortion of the figure with the naked eye even when the posture difference is 10 degrees or less, at which the projection distortion is difficult to check with the naked eye. As a result, the adjustment direction of the orientation of the camera 212 can be easily recognized with the naked eye.
[0076] In addition, in this embodiment, since each side of the figure constituting the fourth indicator 1434 is colored according to the direction of the perspective distortion, the user can visually and easily confirm the direction of the perspective distortion of the figure. As a result, the user can visually and easily recognize how to adjust the direction of the camera 212.
[0077] Even if the misalignment between the images is exaggerated to improve the visibility of the difference between the two images, simply exaggerating the entire misalignment will result in exaggerating not only the perspective distortion but also the rotational distortion. However, if the rotational distortion is exaggerated, it becomes difficult to understand the direction in which the camera should be corrected. Therefore, in this embodiment, the perspective distortion of the figure constituting the fourth index 1434 is exaggerated, but the rotational distortion of the figure is not exaggerated.
[0078] Furthermore, according to this embodiment, the user can recognize the positional deviation and rotational deviation of the camera 212 relative to the camera 211 based on the positions and orientations of the squares constituting the first index 1431 displayed on the guide image 143.
[0079] Furthermore, according to this embodiment, the user can recognize the positional deviation and rotational deviation of the camera 212 relative to the camera 211 from the length and direction of the arrow constituting the second indicator 1432 displayed on the guide image 143.
[0080] Furthermore, according to this embodiment, the user can recognize the amount of positional deviation of camera 212 relative to camera 211 in the X-axis and Y-axis directions, and the rotation angle of camera 212 relative to camera 211, by the text constituting the third indicator 1433 displayed on guide image 143.
[0081] [Second embodiment] Next, a photography guide device according to a second embodiment of the present invention will be described with reference to Fig. 19. Fig. 19 is a block diagram of the photography guide device in this embodiment.
[0082] With reference to FIG. 19, the imaging guide device 300 according to this embodiment includes a calculation unit 301 , an acquisition unit 302 , a generation unit 303 , and a display control unit 304 .
[0083] The calculation means 301 is configured to calculate a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photographing device and a second image obtained by photographing the same plane with a second photographing device.
[0084] The acquiring means 302 is configured to acquire a second figure obtained by transforming a first figure, which is virtually drawn in a predetermined orientation and a predetermined shape in the first image, based on the planar projective transformation matrix.
[0085] The generating means 303 is configured to generate an inclination indicator representing an inclination of the shooting direction of the second photographing device relative to the first photographing device based on the second figure, and to generate a display item including the inclination indicator.
[0086] The display control means 304 is configured to display the display items in a superimposed manner on the second image.
[0087] The photography guide device 300 configured in this manner operates as follows. That is, first, the calculation means 301 calculates a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photography device and a second image obtained by photographing the plane with a second photography device. Next, the acquisition means 302 acquires a second figure obtained by transforming a first figure virtually drawn in a predetermined direction and shape in the first image based on the planar projection transformation matrix. Next, the generation means 303 generates an inclination indicator representing the inclination of the photography direction of the second photography device relative to the first photography device based on the second figure, and generates a display item including the inclination indicator. Next, the display control means 304 displays the display item by superimposing it on the second image.
[0088] With the photography guide device 300 configured and operating as described above, the user can easily determine, from the direction of the projection distortion of the figure that constitutes the inclined indicator, in what direction the second photography device should be adjusted to match the orientation of the first photography device.
[0089] Although the present invention has been described with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.
[0090] For example, in the above first embodiment, the present invention is applied to a photography guide device for supporting an operation of aligning the position and orientation of camera 212 set on a production line with the position and orientation of camera 211. However, the present invention may also be applied to a photography guide device for camera position and orientation when a user compares an image of a product photographed by a camera with an image of the product photographed in advance and registered to perform individual identification and authenticity determination. [Industrial Applicability]
[0091] INDUSTRIAL APPLICABILITY The present invention can be used in the field of identifying individuals, for example, in the field of identifying individual products flowing on a production line.
[0092] A part or all of the above-described embodiments may be described as, but is not limited to, the following supplementary notes. [Appendix 1] A calculation means for calculating a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photographing device and a second image obtained by photographing the plane with a second photographing device; an acquisition means for acquiring a second figure obtained by transforming a first figure, which is virtually drawn in a predetermined orientation and a predetermined shape on the first image, based on the planar projective transformation matrix; a generating means for generating an inclination indicator representing an inclination of an imaging direction of the second imaging device with respect to the first imaging device based on the second graphic, and generating a display item including the inclination indicator; a display control means for displaying the display item superimposed on the second image; A photography guide device comprising: [Appendix 2] the calculation means calculates a rigid transformation matrix of the first image and the second image, and calculates a planar projection transformation matrix of an image obtained by transforming the first image and the second image based on the rigid transformation matrix; 2. A photography guide device as described in appendix 1. [Appendix 3] the generating means generates a figure obtained by exaggerating the projection distortion of the second figure as the inclination indicator; 3. A photography guide device according to claim 1 or 2. [Appendix 4] the first shape is a square; the generating means compares the length of each of the four sides of the second figure with the length of the opposing side, colors the sides having the same length as the opposing sides in a first color, colors the sides having a shorter length than the opposing sides in a second color, and colors the sides having a longer length than the opposing sides in a third color, thereby generating a figure as the inclination indicator; A photography guide device according to any one of appendix 1 to 3. [Appendix 5] the acquiring means further acquires a third figure obtained by transforming the first figure based on the rigid transformation matrix; the generating means generates the display item further including the third graphic as a non-tilted indicator. 3. A photography guide device as described in appendix 2. [Appendix 6] the acquiring means further acquires rotation and positional deviation amounts of the third graphic with respect to the first graphic, the generating means generates the display item further including an arrow indicator representing the amount of deviation of the rotation and position by a direction and a length of an arrow. 6. A photography guide device as described in appendix 5. [Appendix 7] said generating means generating said display item further including a textual indicator representing said rotational and positional offsets in text. 7. A photography guide device as described in appendix 6. [Appendix 8] Calculating a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photographing device and a second image obtained by photographing the plane with a second photographing device; acquiring a second figure obtained by transforming a first figure, which is virtually drawn in a predetermined direction and a predetermined shape on the first image, based on the planar projective transformation matrix; generating an inclination indicator representing an inclination of an imaging direction of the second imaging device relative to the first imaging device based on the second graphic, and generating a display item including the inclination indicator; displaying the display item superimposed on the second image; Photography guide method. [Appendix 9] On the computer, A process of calculating a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photographing device and a second image obtained by photographing the plane with a second photographing device; A process of acquiring a second figure obtained by transforming a first figure, which is virtually drawn in a predetermined orientation and a predetermined shape on the first image, based on the planar projective transformation matrix; generating an inclination indicator representing an inclination of a shooting direction of the second photographing device relative to the first photographing device based on the second figure, and generating a display item including the inclination indicator; a process of displaying the display item superimposed on the second image; A computer-readable recording medium having a program recorded thereon for causing a computer to carry out the above. [Explanation of symbols]
[0093] 10 Individual Identification System 100 Individual Identification Device 110 Communication I / F section 120 Operation input section 130 Screen display section 140 Storage section 141 Programs 142 Reference Image 143 Guide Image 144 Product DB 150 Processing section 151 Photography Guide Section 152 Registration Department 153 Matching Unit 201 First process treatment device 202 Second process treatment equipment 203~205 Conveyor 211, 212 Camera 221, 222 Camera position and attitude adjuster 231, 232 Illuminators 241, 242 products
Claims
1. a calculation means for calculating a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photographing device and a second image obtained by photographing the plane with a second photographing device; an acquisition means for acquiring a second figure obtained by transforming a first figure, which is virtually drawn in a predetermined orientation and a predetermined shape on the first image, based on the planar projective transformation matrix; a generation means for generating an inclination indicator representing an inclination of a shooting direction of the second photographing device with respect to the first photographing device based on the second graphic, and generating a display item including the inclination indicator; a display control means for displaying the display item superimposed on the second image; A photography guide device comprising:
2. the calculation means calculates a rigid transformation matrix of the first image and the second image, and calculates a planar projection transformation matrix of an image obtained by transforming the first image and the second image based on the rigid transformation matrix; The photography guide device according to claim 1.
3. the generating means generates, as the inclination indicator, a figure obtained by exaggerating a projection distortion of the second figure; 3. The photography guide device according to claim 1 or 2.
4. the first shape is a square; the generating means compares the length of each of the four sides of the second figure with the length of an opposing side, colors the sides having the same length as the opposing sides in a first color, colors the sides having a shorter length than the opposing sides in a second color, and colors the sides having a longer length than the opposing sides in a third color, thereby generating a figure as the inclination indicator; 4. The photographing guide device according to claim 1.
5. the acquiring means further acquires a third figure obtained by transforming the first figure based on the rigid transformation matrix; the generating means generates the display item further including the third graphic as a non-tilted indicator. The photographing guide device according to claim 2.
6. the acquiring means further acquires rotation and positional deviation amounts of the third graphic with respect to the first graphic, the generating means generates the display item further including an arrow indicator representing the amount of deviation of the rotation and position by a direction and a length of an arrow.
6. The photography guide device according to claim 5.
7. said generating means generating said display item further including a textual indicator representing said rotational and positional offsets in text.
7. The photography guide device according to claim 6.
8. Calculating a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photographing device and a second image obtained by photographing the plane with a second photographing device; acquiring a second figure obtained by transforming a first figure, which is virtually drawn in a predetermined orientation and a predetermined shape on the first image, based on the planar projective transformation matrix; generating an inclination indicator representing an inclination of an imaging direction of the second imaging device relative to the first imaging device based on the second figure, and generating a display item including the inclination indicator; displaying the display item superimposed on the second image; Photography guide method.
9. On the computer, A process of calculating a planar projection transformation matrix of a first image obtained by photographing a plane of an object with a first photographing device and a second image obtained by photographing the plane with a second photographing device; A process of acquiring a second figure obtained by transforming a first figure, which is virtually drawn in a predetermined orientation and a predetermined shape on the first image, based on the planar projective transformation matrix; generating an inclination indicator representing an inclination of an imaging direction of the second imaging device relative to the first imaging device based on the second figure, and generating a display item including the inclination indicator; a process of displaying the display item by superimposing it on the second image; A program to carry out the above.
Citation Information
Patent Citations
Rice ball and production thereof
JP1987006645A
Control of coupon issuing apparatus
JP1987017886A
Image processing device and image processing program
JP2012193980A
Guide method, information processing apparatus, and guide program
JP2015082830A
Photography guide device
WO2020145004A1
