MOBILE BODY POSITION DETERMINATION APPARATUS, MOBILE BODY POSITION DETERMINATION METHOD, AND PROGRAM

The mobile body position determination device enhances the flexibility and precision of calculating position and orientation by using multiple light-emitting devices with known coordinates and adjusting vectors for height differences and image distortions.

JP7582825B2Active Publication Date: 2024-11-13CASIO COMPUTER CO LTD

Patent Information

Application Number
JP2020157122
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-09-18
Publication Date
2024-11-13
Estimated Expiration
2040-09-18

AI Technical Summary

Technical Problem

Existing technologies restrict the degree of freedom in setting markers and calculating the position and orientation of moving objects, limiting their flexibility and accuracy.

Method used

A mobile body position determination device that uses an imaging device to select and image multiple light-emitting devices with known three-dimensional coordinates, calculates three-dimensional direction vectors, adjusts two-dimensional vectors based on height differences, and processes these to determine the position and orientation of the moving object.

Benefits of technology

Improves the degree of freedom and accuracy in calculating the position and orientation of moving objects by utilizing multiple light-emitting devices with known coordinates and adjusting vectors to account for height differences and image distortions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582825000001
    Figure 0007582825000001
  • Figure 0007582825000002
    Figure 0007582825000002
  • Figure 0007582825000003
    Figure 0007582825000003
Patent Text Reader

Abstract

To improve flexibility in the calculation of a position and an azimuth.SOLUTION: A server 200 obtains an image obtained by imaging of a marker 102a and a marker 102b with a camera 101a. Then, the server 200 corrects the distortion of the obtained image, thereby correcting positions of the marker 102a and the marker 102b. Furthermore, the server 200 converts image positions of the marker 102a and the marker 102b in the images after correction into three-dimensional positions of a camera 101a (forklift 100) by using a camera internal parameter and the like.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a mobile object position determining device, a mobile object position determining method, and a program. [Background technology]

[0002] Conventionally, there has been technology in which an imaging device attached to a moving object captures an image of a marker, which is a light-emitting device in a space, and calculates the position and orientation of the moving object in three-dimensional space based on the image obtained by imaging (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-157051 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned technology has a restriction that the markers must be placed on the same plane, and there is a problem in that the degree of freedom in placing the markers and further in calculating their positions and orientations is low.

[0005] The present invention has been made in consideration of these problems, and has an object to improve the degree of freedom in calculating position and orientation. [Means for solving the problem]

[0006] In order to achieve the above object, a mobile object position determining device according to the present invention comprises: Among a plurality of light-emitting devices, including two light-emitting devices installed at arbitrary positions having different heights in a three-dimensional space, and whose coordinate positions in the three-dimensional space are known, based on an image including at least two light-emitting devices captured by an imaging device attached to a moving body, coordinate positions of the at least two light-emitting devices in the image are acquired; obtaining a plurality of three-dimensional direction vectors from the imaging device to coordinate positions in an image of the at least two light-emitting devices; The acquired three-dimensional direction vectors; The imaging device. The at least two light emitting devices Each and Based on the height difference in the three-dimensional space, a plurality of two-dimensional vectors different in dimension from the height difference in the three-dimensional space are obtained; a positional relationship between the imaging device and each of the at least two light emitting devices in a two-dimensional space and each of the plurality of two-dimensional vectors corresponding to the positional relationship are substantially consistent with each other, At least one of the plurality of two-dimensional vectors At least one of the length and direction of a two-dimensional vector Adjust the At least one of the length and direction Adjusted The plurality of determining at least one of a coordinate position of the moving object and a direction in which the moving object is facing in the three-dimensional space based on the two-dimensional vector and the attitude data of the imaging device; Execute the process. Effect of the Invention

[0007] According to the present invention, it is possible to improve the degree of freedom in calculating the position and orientation. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a visible light communication system according to an embodiment of the present invention. [Diagram 2] FIG. 2 illustrates an example of a server configuration. [Diagram 3] FIG. 1 is a diagram showing an example of an arrangement of cameras and markers in space. [Figure 4] FIG. 11 is a diagram showing an example of a marker three-dimensional position table. [Diagram 5] FIG. 4 is a diagram illustrating an example of camera internal parameters. [Figure 6] FIG. 1A is a diagram showing an example of an image before distortion correction, and FIG. 1B is a diagram showing an example of an image after distortion correction. [Figure 7] 11A and 11B are diagrams illustrating an example of image positions and three-dimensional positions of markers. [Figure 8] FIG. 2 is a diagram illustrating an example of an XY plane vector. [Figure 9]FIG. 13A is a diagram showing an example of an XY plane vector before adjustment, and FIGS. 13B and 13C are diagrams showing an example of an XY plane vector after adjustment. [Figure 10] FIG. 13 is a diagram showing an example of a camera position candidate line. [Figure 11] 13 is a diagram showing an example of acquiring an intersection between a camera position candidate line and an extension vector; FIG. [Figure 12] 11 is a first flowchart showing an example of an operation of a server. [Figure 13] 11 is a second flowchart showing an example of an operation of the server. [Figure 14] 13 is a third flowchart showing an example of an operation of the server. [Figure 15] FIG. 13 is a diagram showing an example of an XY plane vector when there are two cameras. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a visible light communication system serving as a mobile object position determining device according to an embodiment of the present invention will be described with reference to the drawings.

[0010] FIG. 1 is a diagram showing the configuration of a visible light communication system. As shown in FIG. 1, the visible light communication system 1 includes a forklift 100 as a moving object moving in a space 500, a camera 101a attached to the forklift 100, markers 102a, 102b, and 102c (hereinafter, when the markers 102a, 102b, and 102c are not limited, they are appropriately referred to as "markers 102") as light sources attached to the ceiling, side, and the like of the space 500, and a server 200 corresponding to a moving object position determination device. The marker 102 includes an LED (Light Emitting Diode) not shown. The space 500 can be defined by three-dimensional parameters consisting of an X-axis coordinate (X-coordinate), a Y-axis coordinate (Y-coordinate), and a Z-axis coordinate (Z-coordinate), which makes it possible to identify the position of an object existing in the space 500. Here, the X-axis and the Y-axis are both horizontal and perpendicular to each other, and the Z-axis is vertical and perpendicular to the X-axis and the Y-axis.

[0011] In this embodiment, a camera 101a attached to a forklift 100 captures an image of a marker 102. The three-dimensional position of the marker 102 in a space 500 is known, and the marker 102 performs color modulation with various pieces of information to be transmitted, and transmits the information by changing the emitted light color in a time series. The modulation method used is a method in which the information to be transmitted is converted into bit data, and this data is assigned to the order of the three primary colors of light (R (red), G (green), B (blue)) with a predetermined regularity.

[0012] Meanwhile, the server 200 demodulates the change in the emitted color in the light image obtained by chronologically continuous imaging by the camera 101a to acquire information emitted by the marker 102, and calculates the three-dimensional position of the camera 101a in the space 500 as the three-dimensional position of the forklift 100.

[0013] Fig. 2 is a diagram showing an example of the configuration of the server 200. As shown in Fig. 2, the server 200 includes a control unit 202, a wireless communication unit 203, an image processing unit 204, a memory 205, an operation unit 206, a display unit 207, and a communication unit 208.

[0014] 3 is a diagram showing an example of the arrangement of the camera 101a and the marker 102 in the space 500. In this embodiment, the server 200 calculates the unknown three-dimensional position of the camera 101a based on the known three-dimensional position of the marker 102 in the space 500.

[0015] Camera 101a captures (receives) an optical image incident through a lens (not shown), and converts image signals within the imaging angle of view into digital data to generate an image. Camera 101a also captures and generates images continuously in time, and transmits the continuous images to server 200 via wireless communication. Wireless communication unit 203 in server 200 receives the images and outputs them to image processing unit 204. The position of each pixel or image area in the image is defined on a two-dimensional coordinate plane consisting of X and Y coordinates.

[0016] 2 for further explanation. The image processing unit 204 outputs the image (digital data) output from the wireless communication unit 203 to the control unit 202 as is, and also adjusts the image quality and image size of the image and outputs it to the control unit 202 so that the image is displayed on the display unit 207. The image processing unit 204 also has a function of encoding the image using a compression encoding method such as JPEG (Joint Photographic Experts Group) or MPEG (Moving Photographic Experts Group), and converting the image into a still image file or a moving image file.

[0017] The control unit 202 is configured by a CPU (Central Processing Unit) which is a processor. The control unit 202 controls various functions of the server 200 by executing software processing in accordance with a program (for example, a program for realizing the operation of the server 200 shown in Figs. 12, 13, and 14 described later) stored in the memory 205.

[0018] The memory 205 is, for example, a random access memory (RAM) or a read only memory (ROM). The memory 205 stores various information (programs, etc.) used for control in the server 200.

[0019] Furthermore, the memory 205 stores a marker three-dimensional position table shown in Fig. 4 and camera internal parameters shown in Fig. 5. The marker three-dimensional position table shown in Fig. 4 associates IDs (Identifications) that can uniquely identify the markers 102a, 102b, and 102c with the three-dimensional positions (X coordinates, Y coordinates, and Z coordinates) of the markers 102a, 102b, and 102c in the space 500.

[0020] In addition, the camera internal parameters shown in FIG. 5 are composed of image distortion correction data used to correct image distortion, the height (Z coordinate) of the camera 101a, a camera orientation matrix Rz(θ) which is a 3×3 matrix indicating rotation of the camera 101a about the Z axis as the rotation axis, which is attitude data indicating the direction in which the camera 101a is facing, and a camera installation matrix Rc which is a 3×3 matrix indicating rotation of the camera 101a about the X axis and about the Y axis as the rotation axis, which is attitude data indicating the direction in which the camera 101a is facing.

[0021] Returning to FIG. 2 for further explanation, the operation unit 206 is an interface configured with a numeric keypad, function keys, etc., and used for inputting user operation contents. The display unit 207 is configured with, for example, an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an EL (Electroluminescence) display, etc. The display unit 207 displays an image according to an image signal output from the control unit 202. The communication unit 208 is, for example, a LAN (Local Area Network) card. The communication unit 208 communicates with an external communication device.

[0022] The control unit 202 includes the functions of an image acquisition unit 230, a marker identification unit 232, an image position correction unit 234, and a three-dimensional position acquisition unit 236.

[0023] The image acquisition unit 230 acquires an image from the image processing unit 204. The marker identification unit 232 detects the position of the marker 102 in the image (image position: X coordinate, Y coordinate in a two-dimensional plane corresponding to the image). Here, the markers 102a, 102b, and 102c in the space 500 emit light that changes cyclically in a three-color pattern of R (red), G (green), and B (blue), modulated with an ID that can uniquely identify the markers.

[0024] The marker identification unit 232 detects a cyclical three-color pattern of light included in each image captured by the camera 101a. Furthermore, the marker identification unit 232 attempts to detect an ID corresponding to this three-color light emission pattern and further demodulate it into an ID. When an ID is detected, the marker identification unit 232 acquires the image position of the marker 102 corresponding to that ID in the image.

[0025] Furthermore, the marker identification unit 232 refers to the marker three-dimensional position table stored in the memory 205 to acquire the three-dimensional position in the space 500 of the marker 102 corresponding to the acquired ID.

[0026] The image position correction unit 234 corrects the image position of the marker 102. Specifically, as shown in FIG. 6(a), a distortion occurs in the normal image 601a, and the distortion increases as the distance from the center of the image increases. Therefore, the image position correction unit 234 corrects the distortion using image distortion correction data of the camera internal parameters in FIG. 5. The image distortion correction data is data in which the amount of correction increases as the distance from the center of the image increases. This results in an image 601b in which the distortion is corrected as shown in FIG. 6(b), and the positions (image positions) of the markers 102a, 102b, and 102c in the image 601b are corrected.

[0027] Furthermore, the image position correction unit 234 sets the markers 102 present at the four corners of the image after distortion correction so as not to be used when acquiring the three-dimensional position of the camera 101a. For example, as shown in Fig. 6(b), triangular invalid areas 603a, 603b, 603c, and 603d are set at the four corners of the image 601b. Since the marker 102c exists within the invalid area 603a, the marker 102c is not used when acquiring the three-dimensional position of the camera 101a, and only the markers 102a and 102b are used when acquiring the three-dimensional position of the camera 101a.

[0028] The three-dimensional position acquisition unit 236 acquires the three-dimensional position of the camera 101a in the space 500. Specifically, the three-dimensional position acquisition unit 236 calculates the three-dimensional position C=(cx, cy, cz) of the camera 101a in the space 500 by a known method using the image position 610a of the marker 102a and the image position of the marker 102b in the image area 700 shown in Fig. 7, the three-dimensional positions of the marker 102a and the marker 102b obtained by referring to the marker three-dimensional table, and the camera orientation matrix Rz(θ) and the camera installation matrix Rc, which are the posture data stored in the memory 205.

[0029] Specifically, in a camera-based relative system, which is a coordinate system in three-dimensional space centered on the camera 101a, the three-dimensional position acquisition unit 236 calculates a three-dimensional direction vector (vx1, vy1, vz1) pointing from the camera 101a to the marker 102a based on the three-dimensional position of the marker 102a and the camera internal parameters and orientation data, and calculates a three-dimensional direction vector (vx2, vy2, vz2) pointing from the camera 101b to the marker 102 based on the image position and orientation data of the marker 102b.

[0030] Next, in order to fit the three-dimensional direction vector (vx1, vy1, vz1) pointing from the camera 101a to the marker 102a to the positional relationship between the camera 101a and the marker 102a in three-dimensional space, the three-dimensional position acquisition unit 236 calculates an XY plane vector V1 by projecting the three-dimensional direction vector (vx1, vy1, vz1) onto the XY plane using a known method based on the elevation difference between the marker 102a and the camera 101a and the camera installation matrix Rc1. Similarly, in order to fit the three-dimensional direction vector (vx2, vy2, vz2) from the camera 101a to the marker 102b to the positional relationship between the camera 101a and the marker 102b in the three-dimensional space, the three-dimensional position acquisition unit 236 calculates an XY plane vector V2 by projecting the three-dimensional direction vector (vx2, vy2, vz2) onto the XY plane based on the height difference between the marker 102b and the camera 101a and the camera installation matrix Rc2. Fig. 8 is a diagram showing an example of the XY plane vectors V1 and V2.

[0031] Furthermore, the three-dimensional position acquisition unit 236 adjusts the XY plane vector V1 and the XY plane vector V2. This adjustment is for reducing an error in the calculated three-dimensional position C=(cx, cy, cz) of the camera 101a in the space 500 due to an error in the rotation angle θ around the Z axis indicated by the camera orientation matrix Rz(θ) or the like.

[0032] For example, as shown in FIG. 9(a), if the positional relationship between the camera 101a and the markers 102a and 102b on the XY plane does not match the XY plane vector V1 and the XY plane vector V2, the three-dimensional position acquisition unit 236 adjusts the XY plane vector V1 and the XY plane vector V2.

[0033] For example, the three-dimensional position acquisition unit 236 attempts to match the positional relationship between the camera 101a and the markers 102a and 102b with the XY plane vector V1 and the XY plane vector V2 by adjusting the rotation angle θ around the Z axis indicated by the camera orientation matrix Rz(θ).

[0034] Furthermore, in Figure 9(a), since the XY plane vector V1 is longer than the XY plane vector V2, the three-dimensional position acquisition unit 236 determines that the XY plane vector V1 has a larger error, and as shown in Figure 9(b), matches the positional relationship between the camera 101a and marker 102b with the XY plane vector V2 without adjusting the length and direction of the XY plane vector V2, while adjusting the length and direction of the XY plane vector V1 to match the positional relationship between the camera 101a and marker 102b with the XY plane vector V2.

[0035] In addition, as shown in Figure 9 (c), the three-dimensional position acquisition unit 236 adjusts the length and direction of both the XY plane vectors V1 and V2 by matching the ratio of the adjustment amount of the length and direction of the XY plane vector V1 to the ratio of the lengths of the XY plane vectors V1 and V2, thereby matching the positional relationship between the camera 101a and the marker 102b with the XY plane vector V2.

[0036] The three-dimensional position acquisition unit 236 uses the adjusted XY plane vectors V1, V2 and camera orientation matrix Rz(θ) to calculate the three-dimensional direction vector (vx1, vy1, vz1) and further the three-dimensional position C=(cx, cy, cz) of the camera 101a in space 500, and calculates the orientation of the camera 101a based on the camera orientation matrix Rz(θ) and the camera installation matrix Rc.

[0037] Furthermore, the three-dimensional position acquisition unit 236 selects the best three-dimensional position C=(cx,cy,cz) from among the three-dimensional positions C=(cx,cy,cz) of the camera 101a stored in the memory 205, and sets it as the final three-dimensional position of the camera 101a (the coordinate position of the moving body). The best three-dimensional position is, for example, a position among the three-dimensional positions C of the camera 101a where the corresponding reprojection error is the smallest, or a position calculated by a weighted average according to the inverse of the reprojection error. The three-dimensional position acquisition unit 236 stores the obtained three-dimensional position and orientation of the camera 101a in association with the ID of the marker 102 included in the image in the memory 205.

[0038] In addition, if two or more markers 102 are initially captured by the camera 101a, but the camera 101a subsequently moves so that only one marker 102 is captured, the final three-dimensional position and orientation of the camera 101a are calculated based on the image position of that one marker 102.

[0039] Specifically, the three-dimensional position acquisition unit 236 obtains a three-dimensional direction vector from the camera 101a to the marker 102 in a camera reference relative system, which is a coordinate system in a three-dimensional space centered on the camera 101a. The three-dimensional direction vector is obtained based on the image position and posture data of the marker 102. As shown in Fig. 10, when the three-dimensional direction vector V100 is obtained, the position (three-dimensional position) of the camera 101a in the three-dimensional space becomes any position on a camera position candidate line 710, which is the outer edge of a circle centered on a position C10 directly below the marker 102 and on the same plane (XY plane) as the camera 101a.

[0040] Furthermore, the three-dimensional position acquisition unit 236 calculates the inclination of the three-dimensional direction vector with respect to the XY plane, and calculates the horizontal distance (distance on the XY plane) between the camera 101a and the marker 102 based on the inclination and the elevation difference between the camera 101a and the marker 102.

[0041] Next, based on the history of the three-dimensional position and orientation of the camera 101a corresponding to the ID of the marker 102 stored in the memory 205, the three-dimensional position acquisition unit 236 obtains a movement vector V110 having the three-dimensional position (previous position) P1 of the camera 101a obtained by the measurement before last as the start point and the three-dimensional position (previous position) of the camera 101a obtained by the previous measurement as the end point, as shown in Fig. 11, and obtains an extension vector V120 by extending the movement vector V110. Furthermore, the three-dimensional position acquisition unit 236 determines an intersection C20 of the extension vector V120 and the camera position candidate line 710 as the current three-dimensional position C20 of the camera 101a.

[0042] Next, the three-dimensional position acquisition unit 236 obtains the change in the orientation of the camera 101a from the lateral movement (movement on the XY plane) of the marker 102 in the image, and determines the current orientation of the camera 101a by adding the amount of change in the orientation to the orientation in the previous measurement. Furthermore, the three-dimensional position acquisition unit 236 stores the obtained three-dimensional position and orientation of the camera 101a in memory 205 in association with the ID of the marker 102 included in the image.

[0043] Next, the operation of the server 200 will be described with reference to a flowchart. Figures 12, 13 and 14 are flowcharts showing an example of the operation of the server 200.

[0044] The image acquisition unit 230 in the control unit 202 acquires an image captured by the camera 101a from the image processing unit 204 (step S101). Next, the marker identification unit 232 analyzes the image to attempt to acquire the ID of the marker 102, and determines whether the ID has been acquired (step S102). If the ID has not been acquired (step S102; NO), the series of operations ends.

[0045] On the other hand, if an ID has been acquired (step S102; YES), the marker identification unit 232 refers to the marker three-dimensional position table and acquires the three-dimensional position in the space 500 of the marker 102 corresponding to the acquired ID (step S103).

[0046] The image position correction unit 234 corrects the image distortion based on the image distortion correction data of the camera internal parameters (step S104). Furthermore, the image position correction unit 234 sets the markers 102 present in the invalid area 603a, etc., of the image after the distortion correction not to be used when acquiring the three-dimensional position of the camera 101a (step S105).

[0047] Next, the three-dimensional position acquisition unit 236 judges whether or not two or more markers 102 are present in the image (step S106). If two or more markers 102 are present in the image (step S106; YES), the process proceeds to the operation shown in Fig. 13, and the three-dimensional position acquisition unit 236 calculates a three-dimensional direction vector from the camera 101a to the marker 102 in the camera reference relative system based on the three-dimensional position of the marker 102a, the camera internal parameters, and the attitude data (step S201).

[0048] Next, the three-dimensional position acquisition unit 236 calculates an XY plane vector by projecting the three-dimensional direction vector onto the XY plane, based on the height difference between the marker 102 and the camera 101a and the camera installation matrix Rc (step S202).

[0049] Next, the three-dimensional position acquisition unit 236 selects two XY plane vectors having two markers 102 as end points from among the two or more markers 102 (step S203).

[0050] Next, the three-dimensional position acquisition unit 236 adjusts the two XY plane vectors (step S204). Next, the three-dimensional position acquisition unit 236 calculates the three-dimensional position of the camera 101a in the space 500 using the adjusted XY plane vector and the camera orientation matrix Rz(θ), and calculates the orientation of the camera 101a based on the camera orientation matrix Rz(θ) and the camera installation matrix Rc, and stores it in the memory 205 together with the ID of the marker 102 in the image (step S205).

[0051] Furthermore, the three-dimensional position acquisition unit 236 judges whether the processing of steps S203 to S205 has been completed for all combinations of two XY plane vectors (step S206). If not completed (step S206; NO), the operations from step S203 onwards are repeated. On the other hand, if completed (step S206; YES), the three-dimensional position acquisition unit 236 selects the best three-dimensional position and orientation of the camera 101a (step S207).

[0052] On the other hand, in step S106 of Fig. 11, if there are two or more markers 102 in the image (step S106; NO), that is, if there is only one marker 102 in the image, the process proceeds to the operation shown in Fig. 14, and the three-dimensional position acquisition unit 236 determines whether or not the ID of that one marker 102 has already been acquired when the three-dimensional position and orientation of the camera 101a were previously acquired (step S301). If the ID has not been acquired (step S301; NO), the series of operations ends.

[0053] On the other hand, if the ID has been acquired (step S301; YES), the three-dimensional position acquisition unit 236 obtains a three-dimensional direction vector from the camera 101a to the marker 102 in a camera-based relative system, which is a coordinate system of a three-dimensional space centered on the camera 101a. Furthermore, the three-dimensional position acquisition unit 236 obtains the inclination of the three-dimensional direction vector with respect to the XY plane, and obtains the horizontal distance (distance on the XY plane) between the camera 101a and the marker 102 based on the inclination and the elevation difference between the camera 101a and the marker 102 (step S302).

[0054] Next, based on the history of the three-dimensional position and orientation of camera 101a corresponding to the ID of marker 102 stored in memory 205, the three-dimensional position acquisition unit 236 calculates a movement vector from the three-dimensional position of camera 101a obtained by the measurement before last (the position before last) to the three-dimensional position of camera 101a obtained by the previous measurement (the previous position), and calculates an extension vector by extending that movement vector (step S303).

[0055] Furthermore, the three-dimensional position acquisition unit 236 determines the intersection point between the extension vector and the camera position candidate line as the current three-dimensional position of the camera 101a (step S304).

[0056] Next, the three-dimensional position acquisition unit 236 obtains the change in the orientation of the camera 101a from the lateral movement of the marker 102 in the image, and adds the amount of change in the orientation to the orientation in the previous measurement to obtain the current orientation of the camera 101a (step S305). Furthermore, the three-dimensional position acquisition unit 236 associates the obtained three-dimensional position and orientation of the camera 101a with the ID of the marker 102 included in the image and stores them in the memory 205 (step S306).

[0057] Thus, in this embodiment, when the server 200 acquires an image obtained by capturing an image of the marker 102 by the camera 101a, the server 200 corrects the position of the marker 102 by correcting distortion of the image, and converts the image position of the marker 102 in the corrected image into the three-dimensional position of the camera 101a (forklift 100). In this embodiment, there are no restrictions on the installation position of the marker 102, and the degree of freedom in calculating the three-dimensional position and orientation of the forklift 100 can be improved.

[0058] In this embodiment, in consideration of the fact that image distortion increases with distance from the center of the image, the server 200 performs image correction using image distortion correction data in which the amount of correction increases with distance from the center of the image, and does not use the markers 102 present in the invalid areas 603a at the four corners of the image when acquiring the three-dimensional position of the camera 101a. This improves the accuracy of calculation of the three-dimensional position and orientation of the camera 101a.

[0059] Furthermore, in this embodiment, the server 200 adjusts the XY plane vector, and if an error occurs in the three-dimensional position of the camera 101a due to an error in the rotation angle θ around the Z axis indicated by the camera orientation matrix Rz(θ), the error can be reduced.

[0060] In addition, in this embodiment, the best one is selected or acquired based on the three-dimensional position and orientation of the camera 101a acquired for each combination of two markers 102, thereby improving the calculation accuracy of the three-dimensional position and orientation of the camera 101a.

[0061] Furthermore, in this embodiment, even if an image contains only one marker 102, it is possible to calculate the current three-dimensional position and orientation of the camera 101a if the three-dimensional position and orientation of the camera 101a have been calculated in the past using the three-dimensional position of the marker 102. This improves the versatility of the calculation of the three-dimensional position and orientation of the camera 101a.

[0062] Furthermore, in this embodiment, since each marker 102 has an independent ID and the number of IDs in a set can be extremely large, it is possible to arrange markers 102 having many different IDs over a very wide range. Therefore, within the space 500, the absolute distance to each marker 102 can be found immediately upon receiving the IDs of two markers 102, regardless of the previous state.

[0063] The present invention is not limited to the above-described embodiment and drawings, and appropriate modifications can be made to the above-described embodiment and drawings.

[0064] In the above-described embodiment, only the camera 101a is used, but multiple cameras may be used. For example, as shown in Fig. 12, when a camera 101b is provided in addition to the camera 101a, the camera 101a captures an image of the marker 102a, the camera 101b captures an image of the marker 102b, and an XY plane vector V3 from the camera 101a to the camera 101b on the XY plane is defined, whereby the three-dimensional position of the forklift 100 can be calculated with high accuracy.

[0065] In the above embodiment, the marker 102 transmits the ID by changing the emission color in a time series, but the marker 102 may transmit the three-dimensional position of the marker 102. In this case, the server 200 can detect the three-dimensional position of the marker 102 corresponding to the emission pattern. Therefore, the marker three-dimensional position table shown in FIG. 4 is not necessary.

[0066] Furthermore, the marker 102 is not limited to an LED. For example, a light source may be configured as part of an LCD, PDP, EL display, or the like that configures the display device.

[0067] Furthermore, the server 200 may be any device that can acquire images from the cameras 101a and 101b.

[0068] In addition, in the above embodiment, the program to be executed may be stored and distributed on a computer-readable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read-Only Memory), a DVD (Digital Versatile Disc), or an MO (Magneto-Optical Disc), and a system that executes the above-mentioned processing may be configured by installing the program.

[0069] Also, the program may be stored in a disk device or the like of a predetermined server on a network such as the Internet, and may be downloaded, for example, by being superimposed on a carrier wave.

[0070] In addition, when the above-mentioned functions are shared and realized by the OS (Operating System) or by the OS working together with an application, etc., only the parts other than the OS may be stored on a medium and distributed, or may be downloaded, etc.

[0071] Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the specific embodiment, and the present invention includes the inventions described in the claims and their equivalents. The inventions described in the original claims of this application are described below.

[0072] (Appendix 1) Among a plurality of light-emitting devices, including two light-emitting devices installed at arbitrary positions having different heights in a three-dimensional space, and whose coordinate positions in the three-dimensional space are known, based on an image including at least two light-emitting devices captured by an imaging device attached to a moving body, coordinate positions of the at least two light-emitting devices in the image are acquired; determining at least one of a coordinate position of the moving object in the three-dimensional space and a direction in which the moving object is facing, based on the coordinate position in the three-dimensional space and the coordinate position in the image; A mobile positioning device that executes the process.

[0073] (Appendix 2) identifying the at least two light-emitting devices that emit light in different light-emitting patterns based on a plurality of images including the image and captured at different timings; 2. A mobile positioning device according to claim 1 for performing the process.

[0074] (Appendix 3) Acquire a plurality of three-dimensional direction vectors having a focal point of the imaging device as a starting point and a coordinate position in the image of the at least two light-emitting devices as an ending point, and attitude data indicating a direction in which the imaging device is facing; acquiring distances in the three-dimensional space from the imaging device to the at least two light-emitting devices and an optical axis direction in which the imaging device faces, based on the acquired three-dimensional direction vectors and a height difference in the three-dimensional space of the at least two light-emitting devices; 3. A mobile positioning device according to claim 1 or 2, for executing a process.

[0075] (Appendix 4) adjusting the longer of the obtained distances; 4. A mobile positioning device according to claim 3, adapted to perform processing.

[0076] (Appendix 5) adjusting each of the distances according to a ratio of an amount of deviation between the plurality of three-dimensional direction vectors and a coordinate position in the three-dimensional space calculated based on each of the acquired distances; 4. A mobile positioning device according to claim 3, adapted to perform processing.

[0077] (Appendix 6) and controlling the image capturing apparatus so as not to use coordinate positions of light emitting devices present at an outer edge of the image in order to determine the coordinate position of the moving object. 6. A mobile object positioning device according to any one of claims 1 to 5, which executes a process.

[0078] (Appendix 7) determining a coordinate position of the moving body based on a calculated movement vector of the first light-emitting device moving body when an image acquired after acquiring a first image showing an image including the first light-emitting device and other light-emitting devices is a second image including only the first light-emitting device and not any other light-emitting devices; 7. A mobile object positioning device according to any one of claims 1 to 6, which executes a process.

[0079] (Appendix 8) determining a coordinate position of the moving object according to images acquired from the plurality of imaging devices attached to the moving object; 8. A mobile position determination device according to any one of claims 1 to 7, which executes a process.

[0080] (Appendix 9) A method for causing a computer to execute the processing of the mobile positioning device according to any one of appendixes 1 to 8.

[0081] (Appendix 10) A program for causing a computer to execute the processing of the mobile object position determining device according to any one of appendixes 1 to 8. [Explanation of symbols]

[0082] 1...visible light communication system, 100...forklift, 101a, 101b...camera, 102, 102a, 102b, 102c...marker, 200...server, 202...control unit, 203...wireless communication unit, 204...image processing unit, 205...memory, 206...operation unit, 207...display unit, 208...communication unit, 230...image acquisition unit, 232...marker identification unit, 234...image position correction unit, 236...three-dimensional position acquisition unit, 50 0...space, 601a, 601b...image, 603a, 603b, 603c, 603d...invalid area, 610a, 610b...image position of marker, 700...image area, 710...candidate line for camera position, C10...center, C20...3D position of camera, P1...position before last, P2...previous position, V1, V2, V3...XY plane vector, V100...3D direction vector, V110...movement vector, V120...extension vector

Claims

1. Among a plurality of light-emitting devices including two light-emitting devices installed at arbitrary positions having different heights in a three-dimensional space and whose coordinate positions in the three-dimensional space are known, based on an image including at least two light-emitting devices captured by an imaging device attached to a moving body, coordinate positions of the at least two light-emitting devices in the image are acquired; obtaining a plurality of three-dimensional direction vectors from the imaging device to coordinate positions in an image of the at least two light-emitting devices; acquiring a plurality of two-dimensional vectors different in dimension from the height difference in the three-dimensional space based on the acquired plurality of three-dimensional direction vectors and a height difference in the three-dimensional space between the imaging device and each of the at least two light-emitting devices; adjusting at least one of a length and a direction of at least one of the plurality of two-dimensional vectors so that a positional relationship in a two-dimensional space between the imaging device and each of the at least two light-emitting devices substantially coincides with each of the plurality of two-dimensional vectors corresponding to the positional relationship, and determining at least one of a coordinate position of the moving body and a direction in which the moving body is facing in the three-dimensional space based on the plurality of two-dimensional vectors whose length and / or direction have been adjusted and attitude data of the imaging device; A mobile positioning device that executes the process.

2. identifying the at least two light-emitting devices that emit light in different light-emitting patterns based on a plurality of images including the image and captured at different timings; 2. The mobile positioning device of claim 1, further comprising:

3. acquiring the plurality of three-dimensional direction vectors having a focal point of the imaging device as a starting point and a coordinate position in an image of the at least two light-emitting devices as an ending point, and the attitude data indicating a direction in which the imaging device is facing; acquiring distances in the three-dimensional space from the imaging device to the at least two light-emitting devices and an optical axis direction in which the imaging device is facing, based on the acquired three-dimensional direction vectors and a height difference in the three-dimensional space of the at least two light-emitting devices; 3. A mobile position determination device according to claim 1, which executes a process.

4. adjusting the longer of the obtained distances; 4. The mobile position determining device of claim 3, further comprising:

5. and controlling the image capturing apparatus so as not to use coordinate positions of light emitting devices present at an outer edge of the image in order to determine the coordinate position of the moving object.

5. A mobile position determining apparatus according to claim 1, which performs processing.

6. determining a coordinate position of the moving body based on a calculated movement vector of the moving body when an image acquired after acquiring a first image showing an image including the first light-emitting device and other light-emitting devices is a second image including only the first light-emitting device and not any other light-emitting device; 6. A mobile position determining apparatus according to claim 1, which performs processing.

7. determining a coordinate position of the moving object according to images acquired from the plurality of imaging devices attached to the moving object; 7. A mobile position determining device according to claim 1, which performs processing.

8. A method for causing a computer to execute the process of a mobile positioning device according to any one of claims 1 to 7.

9. A program for causing a computer to execute the processing of the mobile object position determining device according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Self position-locating system of movable body

    JP2000337887A

  • Own position identifying method, and device therefor

    JP2005315746A

  • Device and method for estimation of position of moving body and generation of map, and computer-readable recording medium storing computer program controlling the device

    JP2006209770A

  • Optical self-position detection apparatus and method

    JP2011134058A

  • Position detection device

    JP2014157051A

Cited By

  • Mobile object positioning device, mobile object positioning method and program

    JP2023126497A