Position positioning device for mobile object, position positioning method for mobile object

Abstract

The present invention provides a device for positioning a mobile object and a method for positioning a mobile object. A camera (201a-201d) whose imaging direction changes from a first direction to a second direction obtains a camera image including at least one marker (300a-300e) arranged in a space (500) and at least one mobile object (100a-100c) moving in the space (500), detects the change of the imaging direction from the first direction to the second direction based on the camera image and initial data, and obtains the position data of at least one mobile object (100a-100c) based on the camera image without returning the imaging direction of the camera (201a-201d) in the real world from the second direction to the first direction.

Description

Technical Field

[0001] The present invention relates to a position positioning device for a mobile object and a position positioning method for a mobile object. Background Art

[0002] In the past, measurement / position detection technology using visible light was generally calibrated for each camera setting and measured in a relatively short period of time. Therefore, when the camera is used statically, the camera setting conditions will deviate from the initial value due to external factors such as camera cleaning, earthquakes, and time factors such as loose screws, resulting in a decrease in position detection accuracy. In the current situation, recalibration is required to restore the position detection accuracy.

[0003] Japanese Patent Gazette No. 2013-219628 discloses a camera system that stores a reference image of a detection area captured when the camera is not in use and position information of a reference point set in the reference image, periodically detects a deviation of the reference point between the camera's captured image and the reference image, and drives the pan / tilt head based on the detected deviation.

[0004] However, if the gimbal is driven or recalibrated, the system will need to be stopped for a long time, which will affect on-site operations. Summary of the invention

[0005] The present invention can continue to perform positioning well even when the direction of the camera is changed from the initially set state due to various factors.

[0006] The first embodiment of the present invention involves a position positioning device for a mobile object, which comprises: a communication device; and at least one processor, wherein the at least one processor performs the following processing: obtaining a camera image including at least one marker arranged in a space and at least one mobile object moving in the space from a camera whose camera direction changes from a first direction to a second direction via the communication device, detecting the change of the camera direction from the first direction to the second direction based on the camera image and initial data, and obtaining the position data of the at least one mobile object based on the camera image without returning the camera direction of the camera in the real world from the second direction to the first direction.

[0007] In addition, the position positioning device of a mobile body involved in the second embodiment of the present invention comprises: a communication device; and at least one processor, and the at least one processor performs the following processing: obtaining a camera image obtained by photographing multiple markers arranged in a space from a camera fixed to the side of the mobile body and whose camera direction changes from the first direction to the second direction via the communication device, detecting the change of the camera direction from the first direction to the second direction based on the camera image and initial data, and obtaining the position data of the mobile body based on the camera image obtained from the camera without returning the camera direction of the camera in the real world from the second direction to the first direction.

[0008] In the first embodiment, a sign is provided on the moving body side, but a camera is not provided on the moving body side. On the other hand, in the second embodiment, a camera is provided on the moving body side, but a sign is not provided on the moving body side. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a diagram showing an example of a visible light communication system involved in the first embodiment of the present invention.

[0010] Figure 2 This is a diagram showing an example of a server according to the first embodiment of the present invention.

[0011] Figure 3 This is a flowchart showing an example of the imaging direction change detection / correction process according to the first embodiment of the present invention.

[0012] Figure 4 This is a flowchart showing an example of reliability determination processing of a server according to the second embodiment of the present invention.

[0013] Figure 5 This is a diagram showing an example of a visible light communication system involved in the third embodiment of the present invention.

[0014] Figure 6 This is a flowchart showing an example of imaging direction change detection / correction processing according to the third embodiment of the present invention.

[0015] Figure 7 This is a flowchart showing an example of imaging direction change detection / correction processing according to the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0016] Hereinafter, a visible light communication system will be described with reference to the accompanying drawings as a position positioning system involved in an embodiment of the present invention.

[0017] (First embodiment)

[0018] Figure 1 is a diagram showing the structure of the visible light communication system 1. Figure 1 As shown, the visible light communication system 1 includes mobile bodies 100a, 100b, 100c (hereinafter referred to as "mobile body 100" when not limiting each mobile body 100a, 100b, 100c separately) arranged in a space 500 and a server 200 corresponding to the position positioning device.

[0019] The moving body 100a is equipped with a light source 102a, the moving body 100b is equipped with a light source 102b, and the moving body 100c is equipped with a light source 102c (hereinafter, when the light sources 102a, 102b, and 102c are not limited separately, they are appropriately referred to as "light sources 102"). The server 200 is equipped with cameras 201a, 201b, 201c, and 201d corresponding to the camera device (hereinafter, when the cameras 201a, 201b, 201c, and 201d are not limited separately, they are appropriately referred to as "cameras 201"). The camera 201 is fixed in the space 500 by means of a setting device such as a pan / tilt head. In addition, fixed points whose positions do not change, namely markers 300a, 300b, 300c, 300d, and 300e (hereinafter, when the markers 300a, 300b, 300c, 300d, and 300e are not limited to each other, they are appropriately referred to as "markers 300"). The marker 300 and the light source 102 include an LED (Light Emitting Diode) not shown. The light source 102 corresponds to the position positioning object.

[0020] In this embodiment, the light source 102 installed in the mobile body 100 transmits information by emitting light corresponding to various information of the transmission object such as the state of the mobile body 100. On the other hand, the server 200 demodulates the change of the luminous color of the light in the image obtained by the continuous imaging of the camera 201 in time series to obtain the information emitted by the light source 102.

[0021] In this embodiment, the positions and shooting directions of the cameras 201a to 201d are not clear at first. Therefore, before the server 200 obtains the state of the moving body 100, etc., the camera parameters of each camera are obtained, and calibration is performed to obtain the correspondence between the virtual space recognized by the cameras 201a to 201d and the space of the real world. In this way, the camera image obtained by the camera 201 is corrected so that the distortion is reduced. The camera parameters are internal parameters and external parameters of each camera. The internal parameters are matrices that transform camera coordinates into image coordinates, that is, parameters inherent to the camera, including focal length and the optical center position of the lens, and will not change by calibration. The external parameters are matrices that transform world coordinates into camera coordinates, and include information on the position / direction of the camera 201 in the space 500. The server 200 calculates the camera parameters based on the positions (two-dimensional coordinate information) of the images of the markers 300a, 300b, 300c, 300d, and 300e in the images obtained by the cameras 201a to 201d. The calculated camera parameters are stored in the server 200 as initial data.

[0022] Figure 2 2 is a diagram showing an example of the structure of the server 200. Figure 2 As shown, the server 200 includes a control unit 202, an image input unit 204, a memory 205, an operation unit 206, a display unit 207, and a communication unit 208. In addition, cameras 201a to 201d are installed via wiring in the server 200. The server 200 and the cameras 201a to 201d may be connected wirelessly.

[0023] The camera 201a includes a lens 203a, the camera 201b includes a lens 203b, the camera 201c includes a lens 203c, and the camera 201d includes a lens 203d (hereinafter, when the lenses 203a, 203b, 203c, and 203d are not limited, they are appropriately referred to as "lenses 203" respectively). The lens 203 is composed of a zoom lens or the like. The lens 203 moves by the zoom control operation from the operation unit 206 and the focus control of the control unit 202. The camera 201 controls the camera angle of view and the optical image captured by the camera 201 by the movement of the lens 203.

[0024] Cameras 201a to 201d are composed of a plurality of light-receiving elements regularly arranged in two dimensions on a light-receiving surface. The light-receiving elements are, for example, imaging devices such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor). Cameras 201a to 201d capture (receive light) the optical image received through lens 203 within a given range of the camera angle of view based on a control signal from control unit 202, convert the image signal within the camera angle of view into digital data, and generate frames. In addition, cameras 201a to 201d continuously perform imaging and frame generation in time, and output continuous frames to image input unit 204 in server 200.

[0025] The frame (digital data) output from the camera 201 is input to the image input unit 204 based on a control signal from the control unit 202 .

[0026] The control unit 202 is composed of, for example, a CPU (Central Processing Unit). The control unit 202 executes the program stored in the memory 205 (for example, for implementing the above-mentioned Figure 3 The program for controlling the actions of the server 200 shown in the figure executes software processing to control various functions of the server 200.

[0027] The memory 205 is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The memory 205 stores various information (programs, etc.) used for control, etc. in the server 200. In addition, the memory 205 stores camera images captured during calibration of the initial setting of the cameras 201a to 201d and camera parameters calculated as initial data.

[0028] The operation unit 206 is composed of a keypad, function keys, etc., and is an interface for inputting the user's operation content. The display unit 207 is composed of, for example, an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an EL (Electro Luminescence) display, etc. The display unit 207 displays an image in accordance with the image signal output from the control unit 202. The communication unit 208 is, for example, a LAN (Local Area Network) card. The communication unit 208 communicates with an external communication device based on the control of the communication control unit 240.

[0029] The control unit 202 includes an image acquisition unit 231 , an imaging direction detection unit 232 , a correction determination unit 234 , a correction value calculation unit 236 , a position data acquisition unit 238 , and a communication control unit 240 .

[0030] The image acquisition unit 231 performs peripheral bleaching correction and distortion correction, and adjusts the image quality and image size for the frames (digital data) outputted from the camera 201 and inputted to the image input unit 204, so as to display them as live view images on the display unit 207. In addition, when a control signal based on a recording instruction operation from the operation unit 206 is inputted, the image acquisition unit 231 has a function of encoding and filing an optical image within the camera angle of view of the camera 201 or within the display range of the display unit 207 at the time of the recording instruction, for example, in a compression encoding format such as JPEG (Joint Photographic Experts Group).

[0031] After obtaining the camera parameters for each combination (camera pair) of two cameras 201 , the position data acquisition unit 238 detects the cyclic three-color pattern light included in each image obtained from the image acquisition unit 231 by the imaging of the cameras 201 a to 201 d .

[0032] The light sources 102a, 102b, and 102c respectively mounted on the moving bodies 100a, 100b, and 100c emit light modulated to an ID (Identification) that can be identified and uniquely identified, and that changes cyclically in an arbitrary pattern of the three colors of R (red), G (green), and B (blue). This is an embodiment, and as long as the moving body can be uniquely identified, it is not necessary to change the light emission color in an arbitrary pattern of the three colors of R (red), G (green), and B (blue).

[0033] The position data acquisition unit 238 detects the three-color light pattern and attempts to demodulate the ID. In one embodiment, when the same ID can be detected from both images captured by the two cameras 201 included in the camera pair, it is considered that the position data acquisition unit 238 can detect the light source 102 corresponding to the ID. In addition, even if the image is captured by one camera 201, as long as any one of the fixed points, that is, the markers, is captured in the captured image, the position of the moving object having the light source included in the captured image can be calculated.

[0034] Next, the position data acquisition unit 238 acquires, for each camera pair, the position (Xga2, Yga2) of the image of the light source 102 in the imaging plane of one of the two cameras 201 included in the camera pair, and the position (Xgb2, Ygb2) of the image of the light source 102 in the imaging plane of the other camera 201. Furthermore, the position data acquisition unit 238 calculates the installation position (Xk2, Yk2, Zk2) of the light source 102 in the space 500 in world coordinates using a combination of the positions (Xga2, Yga2) and (Xgb2, Ygb2) of both images and the camera parameters.

[0035] The camera direction detection unit 232 detects changes in the camera directions of the cameras 201a to 201d periodically or according to the user's instructions after the cameras 201a to 201d are installed. By detecting changes in the camera directions of the cameras 201a to 201d, the camera direction detection unit 232 can detect deviations from the initial camera installation conditions due to external factors or time elapsed factors after the camera installation.

[0036] The imaging direction detection unit 232 detects the position (two-dimensional coordinate information) of the image of the marker 300 in each image obtained by the imaging of the cameras 201a to 201d. Here, the installation positions (three-dimensional coordinate information of the world coordinate system) of the markers 300a, 300b, 300c, 300d, and 300e in the space 500 are known. In addition, the markers 300a, 300b, 300c, 300d, and 300e emit light modulated to a three-color pattern of R (red), G (green), and B (blue) that changes in a circular manner so that their IDs can be identified and uniquely determined.

[0037] The imaging direction detection unit 232 sets a combination (camera pair) of two cameras 201 for the cameras 201a to 201d. There are six patterns (six types) of combinations (camera pairs) of any two cameras 201 from the four cameras 201.

[0038] The camera direction detection unit 232 detects the three-color pattern light contained in each image obtained by the camera 201a to 201d. Furthermore, the camera direction detection unit 232 attempts to demodulate the ID corresponding to the three-color light pattern. In the memory 205, the installation positions of the markers 300a, 300b, 300c, 300d, and 300e are associated with the IDs and stored as initial data (including fixed points, i.e., world coordinate data of each marker).

[0039] The imaging direction detection unit 232 attempts to detect light modulated corresponding to the ID, i.e., a modulated light area (a pixel area having a high brightness value greater than a preset value and composed of a specific size / shape) from the image obtained from the image acquisition unit 231 through the imaging of each camera 201a to 201d. Then, if the light modulated corresponding to the ID can be detected, it is considered that the mark 300 corresponding to the ID can be detected. The imaging direction detection unit 232 obtains the installation position of the mark 300 corresponding to the ID.

[0040] The camera direction detection unit 232 compares the images captured by cameras 201a~201d with the initial data, i.e., the camera images initially set by cameras 201a~201d, and detects changes in the camera directions of cameras 201a~201d based on the image position or setting position of the mark 300 contained in the camera images.

[0041] The correction determination unit 234 determines whether to correct the change in the camera direction of the cameras 201a to 201d based on the change in the camera direction detected by the camera direction detection unit 232. Here, the correction refers to a process of calculating a correction value for correcting the information on the position / direction of the camera 201 in the space 500, i.e., an external parameter, and performing correction so as to eliminate the change in the camera direction.

[0042] The correction judgment unit 234 judges whether to perform correction based on two viewpoints: (1) whether correction is necessary and (2) whether correction can be performed. The correction judgment unit 234 judges whether correction is necessary based on whether the change (deviation) of the image position or installation position of the mark 300 contained in the camera image from the image position or installation position of the mark 300 contained in the camera image when the camera was initially set reaches a set threshold. The threshold is pre-stored in the memory 205 for each camera, and the correction judgment unit 234 reads the threshold from the memory 205 to perform judgment processing. In addition, the threshold includes a threshold for judging whether correction is necessary and a threshold for judging whether correction can be performed.

[0043] The correction value calculation unit 236 calculates the correction value for the case where the correction judgment unit 234 determines that the correction is possible and necessary. As the correction value, the correction value parameter of the external parameter, which is the information of the position / direction of the camera 201, is calculated. Specifically, the correction value is calculated based on the inverse matrix of the variation parameter of the external parameter. This is described below.

[0044] Generally, the relationship between the world coordinates [Wp] of a three-dimensional space and the image coordinates [u, v] of a two-dimensional image is represented as follows using a perspective projection matrix, etc., and by using intrinsic parameters [A] and extrinsic parameters [E] of a camera.

[0045] (Equation 1)

[0046] [u, v] = [A] [E] [Wp]…(1)

[0047] Next, after the camera is set up, due to external factors / time factors, the camera's shooting direction, which is a setting condition of the camera, changes, so that the image coordinates of the two-dimensional image change from [u, v] to [u', v']. In this case, the relationship between the world coordinates [Wp] of the three-dimensional space and the image coordinates [u', v'] of the two-dimensional image is represented by the following formula.

[0048] (Equation 2)

[0049] [u', v'] = [A] [E] [E'] [Wp]...(2)

[0050] Here, [E'] is a parameter indicating the amount of change in the external parameter [E] caused by the change in the camera's shooting direction. In addition, the camera's internal parameter [A] is a parameter inherent to the camera and does not change even when the camera's installation conditions change. In equation (2), the factor that causes the image coordinates to change from [u, v] to [u', v'] is due to the parameter [E'] indicating the amount of change in the external parameter [E]. Therefore, the correction value [E'] is calculated by obtaining the inverse matrix of the parameter [E'] indicating the amount of change. -1 , and the coordinates are corrected by multiplying the correction value.

[0051] To calculate the correction value [E'] by finding the inverse matrix -1 , it is necessary for cameras to shoot common signs, or for one camera to shoot more than two signs.

[0052] In cases other than the above, that is, when one camera is photographing one mark, the correction value [E'] cannot be calculated by finding the inverse matrix. -1 In this case, the correction value calculation unit 236 selects the correction value [E'] by brute forcing the combination of rotation angles around the coordinate axes of the camera coordinate system. -1 That is, parameters. Here, if the pan / tilt platform that fixes the camera is a rotating platform that can adjust pan and pitch among the three axes of the camera's left-right movement, that is, pan, the camera's up-and-down movement, that is, tilt, and the rotation around the camera's optical axis, that is, roll, the number of combinations to be enumerated can be reduced by setting the combination of only the pan and pitch rotation angles. Furthermore, the number of combinations to be enumerated can be reduced according to the movable range and movable unit of the pan / tilt adjustment, that is, the adjustment resolution.

[0053] The correction value calculation unit 236 calculates the inverse matrix or obtains the correction value [E'] by performing an enumeration test on the combination of rotation angles. -1 Stored in memory 205.

[0054] Next, the operation of the server 200 will be described with reference to the flowchart. Figure 3 2 is a flowchart showing an example of the operation of the imaging direction change detection / correction process of the server 200. This process may be always executed while the position of the moving object 100 in the space 500 is being located, may be executed periodically, or may be executed in response to a user's instruction.

[0055] The cameras 201a to 201d perform video recording in the space 500. The video recording direction detection unit 232 in the control unit 202 compares the video recording images of the cameras 201a to 201d acquired by the image acquisition unit 231 with the video recording images of the cameras 201a to 201d initially set as the initial data read from the memory 205. Specifically, the image positions or setting positions of the marks 300 of the common ID contained in the video recording images are compared. Based on the comparison, the video recording directions of the cameras 201a to 201d newly acquired by the image acquisition unit 231 are detected to change from the video recording directions of the cameras 201a to 201d initially set by the cameras, i.e., the first direction to the second direction (step S101).

[0056] The so-called first direction defined in this specification refers to the imaging direction of the camera fixed by the fixing unit when the system is first introduced or when the camera calibration is performed during system operation.

[0057] The camera 201 is initially fixed in the first direction by the fixing unit for fixing the camera 201. However, due to external factors such as earthquakes and time factors such as loosening of the fixing part including the screws, the direction in which the camera 201 is facing may deviate from the first direction to a direction other than the first direction (second direction). In this embodiment, even if the fixing unit physically deviates the direction in which the camera is fixed, there is no need to physically change (return) the deviation.

[0058] Next, the correction determination unit 234 in the control unit 202 determines whether correction of the external parameters is necessary based on the change in the imaging direction detected by the imaging direction detection unit 232 (step S102 ).

[0059] The correction judgment unit 234 compares the change (deviation) of the image position or setting position of the mark 300 contained in the camera image relative to the image position or setting position of the mark 300 initially set by the camera 201 with the threshold value. When the amount of change from the initially set camera direction of the camera 201, i.e., the first direction, to the current camera direction, i.e., the second direction, is greater than the threshold value, it is determined that the correction of the external parameters is necessary (step S102 "Yes"), and then it is determined whether the correction of the external parameters can be performed (step S103). The amount of change from the first direction to the second direction is determined based on the amount of movement of the image position or setting position of the mark 300. In addition, when the camera direction of the camera 201 deviates greatly and the mark 300 contained in the initially set camera image is not included in the current camera image and there is no common mark 300, the camera direction detection unit 232 cannot detect the amount of change. Therefore, the correction judgment unit 234 determines that correction is necessary even when the amount of change cannot be detected.

[0060] When the change from the initially set shooting direction of the camera 201, i.e., the first direction, to the current shooting direction, i.e., the second direction, is small or nonexistent and thus smaller than the threshold, the correction judgment unit 234 judges that no correction of the external parameters is required (step S102 "No") and ends the processing.

[0061] Next, in step S103, the correction judgment unit 234 judges whether the external parameters can be corrected based on the camera image of the current camera 201. Specifically, it is judged whether the image position of the mark 300 contained in the camera image deviates significantly from the center of the image and is located in the corner of the image, or whether it is out of view. In the case where the position of the mark 300 deviates significantly from the center of the image and cannot be adjusted by the correction of the external parameters, the correction judgment unit 234 judges that the correction cannot be performed (step S103 "No"), and displays an error on the display unit 207 to warn the user (step S104). In addition, the warning is not limited to a visual display, but can also be a sound-based warning. After warning the user, the control unit 202 ends the processing. The user accepts the warning and adjusts the setting position of the camera 201.

[0062] When the position of the marker 300 is greatly deviated from the center of the image to such an extent that it cannot be adjusted by correction of the external parameters, the correction determination unit 234 determines that correction is possible (Yes in step S103 ), and proceeds to correction value calculation processing.

[0063] Before calculating the correction value, the correction value calculation unit 236 of the control unit 202 determines whether there is a camera pair that is photographing the common mark 300, or whether a single camera 201 is photographing more than two marks 300 (step S105). When there is a camera pair that is photographing the common mark 300, or when one camera 201 is photographing more than two marks 300 (step S105 "Yes"), the correction value [E'] is calculated by obtaining an inverse matrix based on the camera parameters (internal parameters and external parameters), the known world coordinates (true value) of the mark 300 stored in the memory 205, and the world coordinates (after deviation) of the mark 300 calculated based on the current camera image. -1 (Step S106).

[0064] When there is no camera pair that is photographing the common mark 300 and the camera 201 is photographing only one mark 300 (step S105: No), the correction value calculation unit 236 enumerates and attempts combinations of rotation angles about pan and tilt, assuming that there is no deviation in the rolling direction, among the rotation angles about the coordinate axes of the camera coordinate system, and selects the correction value [E']. -1 In addition, in the case of a pan / tilt that can also be adjusted in the rolling direction, it is also possible to enumerate and try combinations that can also change the rolling direction to select the correction value [E']. -1 .

[0065] If the correction value [E'] is obtained in step S106 or step S107 -1 , the position data acquisition unit 238 performs correction of the camera parameters and acquires the position data of the moving object 100 based on the corrected camera parameters (step S108). Correction value [E'] -1 It is the inverse matrix of the parameter [E'] corresponding to the amount of change of the external parameter [E] caused by the change of the imaging direction of the camera 201. Therefore, the coordinates are corrected by multiplying the parameter [E'] corresponding to the amount of change by the correction value. The position data acquisition unit 238 calculates the installation position of the world coordinates of the light source 102 in the space 500 using the position of the image of the light source 102 of the moving body 100 in the imaging plane of the camera 201 and the camera parameters corrected by multiplying by the correction value. Thus, even if the imaging direction changes from the first direction initially set by the camera 201 to the second direction, the imaging direction of the camera 201 in the real world does not need to be returned from the second direction to the first direction, and the position data of the moving body 100 is acquired based on the captured image. In this way, even if the direction facing the camera 201 changes from the initially set state due to various factors, calibration is not required, and the positioning accuracy can be restored without stopping the operating system.

[0066] (Second embodiment)

[0067] Next, other embodiments are described. In this embodiment, the visible light communication system 1 and Figure 1 Likewise, the server 200 and Figure 2 Similarly, in the present embodiment, the installation positions of the marker 300 and the light source 102 are calculated for each camera pair, and the reliability of the calculated installation positions is determined.

[0068] Figure 4 This is a flowchart showing an example of the operation of the reliability determination process of the server 200 according to another embodiment. Figure 4 The actions shown are performed for each symbol 300 .

[0069] For each camera pair, if the two cameras 201 included in the camera pair take a picture of the same sign 300, the picture image is obtained via the image input unit 204, and the camera direction detection unit 232 of the control unit 202 attempts to identify the sign 300 by obtaining the ID (step S201).

[0070] Next, the imaging direction detection unit 232 selects a camera pair that can image the sign 300 and obtain the ID in step S301 (step S202 ).

[0071] Next, the camera direction detection unit 232 calculates the installation position of the sign 300 based on the images captured by the two cameras 201 included in the camera pair for each camera pair selected in step S202 (step S203). Specifically, the camera direction detection unit 232 obtains the position of the image of the sign 300 in the image captured by the camera 201 of one of the camera pairs and the position of the image of the sign 300 in the image captured by the camera 201 of the other camera pair. Furthermore, the camera direction detection unit 232 calculates the installation position of the sign 300 using a combination of the two positions obtained and the camera parameters corresponding to the camera pair.

[0072] Next, the camera direction detection unit 232 obtains the installation reliability of the mark 300 whose installation position is calculated in step S203 (step S204). For example, with camera 201a as the center, the installation position of the mark 300c calculated by the camera pair of camera 201a and camera 201b is set to (Xb, Yb, Zb), the installation position of the mark 300c calculated by the camera pair of camera 201a and camera 201c is set to (Xc, Yc, Zc), and the installation position of the mark 300c calculated by the camera pair of camera 201a and camera 201d is set to (Xd, Yd, Zd). Here, it is assumed that any camera pair can shoot the mark 300c when the camera is initially set. Since any camera pair calculates the installation position of the mark 300c, the calculated value should be the same. The installation position of the mark 300c is known and stored in the memory 205 in advance. The camera direction detection unit 232 reads the setting position of the mark 300c from the memory 205, and obtains the difference between it and the setting positions (Xb, Yb, Zb), (Xc, Yc, Zc), and (Xd, Yd, Zd) of the mark 300c calculated by each camera pair.

[0073] The camera direction detection unit 232 determines whether the camera direction detection unit 232 needs to be adjusted based on whether the difference between the camera pairs is 0 or within the error range. Figure 3 If there is a camera pair whose difference obtained for each camera pair exceeds the error range (step S205 "Yes"), the camera direction detection unit 232 is set to need to perform camera direction change detection / correction processing, and transfers to Figure 3 For example, in the example described above, when the difference in the setting position (Xd, Yd, Zd) of the mark 300c calculated by the camera pair 201a and the camera 201d exceeds the error range, and the difference in the setting position of the mark 300c calculated by the other camera pairs is within the error range, it is determined that the setting reliability of the camera 201d is low, and the process is transferred to Figure 3 Change detection / correction processing of camera direction.

[0074] When the difference obtained for each camera pair is 0 or within the error range (No in step S205), it is determined that there is no problem with the installation reliability of the cameras 201a to 201d, and the processing ends.

[0075] In the above example, the difference between the installation position of the mark 300 calculated by each camera pair and the known installation position of the mark 300 read from the memory 205 is obtained, but the present invention is not limited to this. The installation positions of the mark 300 calculated by each camera pair may be compared, and the camera pair that calculates the installation position different from the installation position of the mark 300 calculated by the other plurality of camera pairs may be judged to have low installation reliability. For example, when the installation position (Xb, Yb, Zb) of the mark 300c calculated by the camera pair of camera 201a and camera 201b is the same as the installation position (Xc, Yc, Zc) of the mark 300c calculated by the camera pair of camera 201a and camera 201c, and the installation position (Xd, Yd, Zd) of the mark 300c calculated by the camera pair of camera 201a and camera 201d is different from the installation positions calculated by the other camera pairs by exceeding the error range, it is judged that the installation reliability of camera 201d is low.

[0076] In addition, the above reliability determination process can also be replaced by Figure 3 In this case, in step S103, if the difference from the known installation position of the mark 300 or the difference from the installation position of the mark 300 calculated based on multiple camera pairs is so large that it cannot be dealt with by the correction of external parameters, it is determined that the correction cannot be performed and a warning of the error is issued.

[0077] (Third embodiment)

[0078] In the first and second embodiments described above, the camera 201 is fixedly installed in the space 500 (hereinafter referred to as a fixed camera type). In contrast, in the third embodiment, a case where the camera 201 is fixedly installed on the moving body 100 and the camera 201 moves with the movement of the moving body 100 (hereinafter referred to as a moving camera type) is described.

[0079] Figure 5 1 is a diagram showing the structure of the visible light communication system 1 according to the present embodiment. Here, a forklift is taken as an example as the mobile body 100. Figure 5 As shown in FIG. 1 , the visible light communication system 1 includes a moving body 100 moving in a space 500, a camera 201 installed on the moving body 100, markers 300a, 300b, 300c, 300d as light sources installed on the ceiling, side, etc. of the space 500, and a server 200. The server 200 and the camera 201 are connected by wireless, and the server 200 is set outside the space 500. Of course, the server 200 can also be set in the space 500. In addition, for Figure 1The same structure is marked with the same number and the description is omitted. In addition, the structure of the server 200 is the same as that of the camera 201a~201d except that the server 200 and the camera 201a~201d are wirelessly connected. Figure 2 The structure shown is the same, so the description is omitted.

[0080] The camera 201 moves along with the movement of the mobile body 100, and photographs a plurality of marks 300 to measure the position of the camera, that is, the position of the mobile body 100. In the mobile camera type, the camera position can be measured by bringing two or more marks into the camera's field of view. Compared with the fixed camera type, since the number of observation points is reduced, the accuracy requirements for the camera parameters are increased. Since the camera 201 is installed on the mobile body 100, the setting conditions of the camera 201 may deviate from the initial value due to vibrations during the travel of the mobile body 100, and the frequency of the problem of reduced position detection accuracy is higher than that of the fixed camera type. Therefore, in the mobile camera type, it is important to correct the deviation of the setting conditions of the camera 201 on a daily basis. For example, when the mobile body 100 is a forklift, the position where the forklift is to be parked is often determined based on operational considerations at the end of the operation. In this case, the deviation of the direction in which the camera is facing can be corrected when returning to the parking position (starting position) of the forklift determined in operation. The position of the mark that can be visually recognized from the start position should always be the same. Therefore, when the moving body 100 returns to the start position, the change detection / correction process of the imaging direction of the camera 201 is executed.

[0081] Figure 6 2 is a flowchart showing an example of the operation of the imaging direction change detection / correction process of the server 200. This process is automatically executed when the moving object 100 returns to the home position, but may also be executed by a user's instruction.

[0082] When the moving body 100 returns to the starting position, the camera 201 starts to shoot the space 500 from the starting position. The shooting direction detection unit 232 in the control unit 202 compares the shooting image of the camera 201 obtained by the image acquisition unit 231 with the shooting image of the camera 201 at the starting position where the camera was initially set, which is read from the memory 205 as initial data. The shooting image shot at the starting position during the calibration of the initial setting of the camera 201 and the calculated camera parameters are pre-stored in the memory 205 as initial data. The image position or setting position of the mark 300 of the common ID contained in the shooting image is compared, and the shooting direction of the camera 201 newly obtained by the image acquisition unit 231 is detected to change from the shooting direction of the camera 201 initially set, that is, the first direction, to the current shooting direction, that is, the second direction (step S301).

[0083] Next, the correction determination unit 234 in the control unit 202 determines whether correction of the external parameters is necessary based on the change in the imaging direction detected by the imaging direction detection unit 232 (step S302 ).

[0084] When the detected change amount from the first direction to the second direction is greater than the threshold, the correction judgment unit 234 judges that the external parameter needs to be corrected (step S302 "Yes"), and then judges whether the external parameter can be corrected (step S303). When the detected change amount from the first direction to the second direction is small or does not change, so that the change amount is smaller than the threshold, the correction judgment unit 234 judges that the external parameter does not need to be corrected (step S302 "No"), and the processing ends. The judgment on whether correction is needed is the same as Figure 3 The same is true for step S102.

[0085] Next, in step S303, the correction determination unit 234 determines whether the external parameters can be corrected based on the current camera image of the camera 201. Figure 3 The same is true for step S103. When the position of the mark 300 deviates greatly from the center of the image and cannot be adjusted by correction of external parameters, the correction judgment unit 234 judges that correction cannot be performed (step S303 "No"), and displays an error on the display unit 207 to warn the user (step S304). After warning the user, the control unit 202 ends the processing. When the position of the mark 300 deviates greatly from the center of the image to the extent that it cannot be adjusted by correction of external parameters, the correction judgment unit 234 judges that correction is possible (step S303 "Yes") and moves to the correction value calculation process.

[0086] The correction value calculation unit 236 of the control unit 202 calculates the correction value [E'] by obtaining an inverse matrix based on the camera parameters (internal parameters and external parameters), the known world coordinates (true values) of the markers 300 stored in the memory 205 for two or more markers 300 in the camera image, and the world coordinates (after deviation) of the marker 300 calculated based on the current camera image. -1 (Step S305).

[0087] If the correction value [E'] is obtained in step S305 -1 , the position data acquisition unit 238 performs correction of the camera parameters, and based on the corrected camera parameters, performs self-positioning of the position of the camera 201, i.e., the position of the moving object 100, and acquires the position data (step S306). Correction value [E'] -1It is the inverse matrix of the parameter [E'] corresponding to the amount of change of the external parameter [E] caused by the change of the imaging direction of the camera 201. Therefore, by multiplying the parameter [E'] corresponding to the amount of change by the correction value, the coordinates are corrected.

[0088] If the position coordinates of the corrected moving body 100 are obtained, they are compared with the position coordinates of the starting position to determine whether the obtained position coordinates are consistent with the position of the starting position, that is, to determine whether the obtained position coordinates are correct (step S307). The position coordinates of the starting position are pre-stored in the memory 205. The control unit 202 reads the position coordinates of the starting position from the memory 205 to perform the above determination. If the obtained position coordinates are correct (step S307 "Yes"), the processing is terminated. If the obtained position coordinates are incorrect (step S307 "No"), the process returns to step S301 and the processing after step S301 is repeated.

[0089] According to the third embodiment, when the camera 201 is fixed to the mobile body 100, even if the shooting direction changes from the first direction in which the camera 201 is initially set to the second direction, the shooting direction of the camera 201 in the real world does not need to be returned from the second direction to the first direction. In this state, the position data of the mobile body 100 is obtained based on the camera image. In this way, even if the direction to which the camera 201 is directed changes from the initially set state due to various factors, calibration is not required, and the position positioning accuracy can be restored without stopping the system in operation. In addition, since the correction process is performed on the starting position that must be returned at the end of the operation, the correction process is performed on a daily basis. Therefore, in an environment where the posture of the camera 201 is easily changed due to the camera 201 being set on the mobile body 100, the position positioning accuracy can be maintained.

[0090] (Fourth embodiment)

[0091] In the third embodiment, the correction value [E'] for correcting all the rotation angles of the three axes of pan, pitch, and roll is obtained based on the captured image. -1 In contrast, in the fourth embodiment, an inertial sensor is provided in the camera 201, and a portion of the pan, pitch, and roll is corrected based on the sensor value, and then a correction value [E'] is obtained from the captured image. -1 From this, the correction value [E'] is obtained -1 The calculation process becomes simple.

[0092] As inertial sensors, an acceleration sensor and a gyro sensor are provided in the camera 201. The acceleration sensor and the gyro sensor measure the moving state of the camera 201. In addition, the inertial sensor may be any one of the acceleration sensor and the gyro sensor, as long as it can detect the posture of the camera 201. In addition, the inertial sensor may be provided in a setting device of the camera 201 that moves together with the camera 201. The acceleration sensor is a three-axis acceleration sensor, and measures the change in the movement speed of the camera 201 during movement by detecting the acceleration in the three-axis directions orthogonal to each other. The gyro sensor is a three-axis angular velocity sensor, and measures the change in the movement direction of the camera 201 during movement by detecting the angular velocity of the three axes rotating around each axis of the specified acceleration in the acceleration sensor.

[0093] Figure 7 2 is a flowchart showing an example of the operation of the imaging direction change detection and correction process of the server 200 in this embodiment. Similar to the third embodiment, this process is executed when the moving body 100 returns to the home position.

[0094] When the moving object 100 returns to the starting position, the amount of change is detected based on the sensor values ​​of the acceleration sensor and the gyro sensor provided in the camera 201 (step S401).

[0095] In addition, the camera 201 captures images in the space 500 from the starting position. The camera direction detection unit 232 in the control unit 202 compares the camera image of the camera 201 acquired by the image acquisition unit 231 with the camera image of the camera 201 at the starting position where the camera was initially set, which is read from the memory 205 as initial data. Based on the comparison, the camera direction of the camera 201 newly acquired by the image acquisition unit 231 is detected to be changed from the first direction where the camera 201 was initially set, that is, to the second direction (step S402).

[0096] Next, the correction judgment unit 234 in the control unit 202 judges whether it is necessary to correct the external parameters based on the change in the camera direction detected by the camera direction detection unit 232 (step S403). When the amount of change from the camera direction originally set by the camera 201, that is, the first direction, to the second direction is greater than the threshold, it is judged that the external parameters need to be corrected (step S403 "Yes"), and then it is judged whether the external parameters can be corrected (step S404). When the amount of change from the camera direction originally set by the camera 201, that is, the first direction, to the second direction is small or does not change, so that the amount of change is smaller than the threshold, the correction judgment unit 234 judges that the external parameters do not need to be corrected (step S403 "No"), and the processing ends.

[0097] Next, in step S404, the correction judgment unit 234 judges whether the external parameters can be corrected based on the current camera image of the camera 201. In the case where the position of the mark 300 deviates greatly from the center of the image and cannot be adjusted by the correction of the external parameters, the correction judgment unit 234 judges that the correction cannot be performed (step S404 "No"), and displays an error on the display unit 207 to warn the user (step S405). The control unit 202 ends the processing. In the case where the position of the mark 300 deviates greatly from the center of the image to such an extent that it cannot be adjusted by the correction of the external parameters, the correction judgment unit 234 judges that the correction cannot be performed (step S404 "Yes"), and transfers to the pitch and roll correction processing based on the sensor values ​​of the acceleration sensor and the gyro sensor (step S406).

[0098] In step S406, the pitch and roll are corrected based on the sensor values, and the pan is not corrected. Regarding the pan, since the correction process based on the sensor value becomes complicated, the pan is corrected by the correction value calculation process based on the camera image that is performed next. In step S406, the 3-axis output of the acceleration sensor and the 3-axis output of the gyro sensor are input to the Kalman filter and the low-pass filter to calculate the 3-axis data of the acceleration relative to the ground and the 3-axis data of the angular velocity, so as to estimate the direction of gravity. In addition, the estimation of the direction of gravity can also be performed by using an axis estimation method other than the Kalman filter and the low-pass filter. If the direction of gravity is estimated, the posture is corrected in the direction of gravity estimated by the data of the acceleration sensor and the gyro sensor. Pitch and roll are corrected in this way.

[0099] Next, the correction value calculation unit 236 of the control unit 202 calculates the correction value [E'] by obtaining an inverse matrix based on the camera parameters (internal parameters and external parameters), the known world coordinates (true values) of the markers 300 stored in the memory 205 for two or more markers 300 in the camera image, and the world coordinates (after deviation) of the marker 300 calculated based on the current camera image. -1 (Step S407) Since the pitch and roll have been corrected in step S406, the correction value [E'] to be obtained is reduced. -1 of computational processing burden.

[0100] If the correction value [E'] is obtained in step S407 -1 The position data acquisition unit 238 performs correction of the camera parameters, obtains the position of the camera 201 based on the corrected camera parameters, that is, performs self-positioning of the position of the moving body 100, and obtains the position data (step S408).

[0101] If the position coordinates of the moving body 100 are obtained, they are compared with the position coordinates of the starting position read from the memory 205 to determine whether the obtained position coordinates are consistent with the position of the starting position, that is, to determine whether the obtained position coordinates are correct (step S409). If the obtained position coordinates are correct (step S409 "Yes"), the process ends. If the obtained position coordinates are incorrect (step S409 "No"), the process returns to step S401 and the process after step S401 is repeated.

[0102] According to the fourth embodiment, in addition to the effects of the third embodiment, the sensor value correction by the inertial sensor is used as the correction value for calculating the correction value [E']. -1 The pitch and roll angles of the parameters are reduced and used to calculate the correction value [E'] -1 The processing becomes simple. In addition, in the correction of the sensor value based on the inertial sensor, by using the correction value [E'] -1 The calculation process for correcting the panning rotation angle, which is complicated, can be simplified by the calculation process for correcting the panning rotation angle based on the sensor value of the inertial sensor.

[0103] In the second, third, and fourth embodiments described above, the parking accuracy of the moving object 100 at the starting position is not considered. In contrast, video content analysis may be used to ease the parking accuracy of the moving object 100. For example, by setting a chessboard-like pattern at the starting position, the parking posture of the moving object 100 is derived to correct the deviation of the left-right rotation caused by the deviation of the parking direction of the moving object 100.

[0104] In the above-mentioned embodiment, the CPU functions as the control unit 202 by executing the program stored in the RAM and ROM. However, instead of the CPU executing the program stored in the RAM and ROM, dedicated hardware such as ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and various control circuits may be provided to function as the control unit 202. In this case, a part may be implemented by the dedicated hardware, and another part may be implemented by software or firmware.

[0105] Although the embodiment of the present invention has been described, the scope of the present invention is not limited to the above-described embodiment, and includes the scope of the invention described in the claims and the scope equivalent thereto.

Claims

1. A positioning device for a mobile object, characterized in that: have communications device; and At least 1 processor, The at least one processor performs the following processing: A camera whose shooting direction changes from a first direction to a second direction acquires a shooting image including at least one marker arranged in a space and at least one moving object moving in the space via the communication device, detecting a change in the imaging direction from the first direction to the second direction based on the imaging image and the initial data, When a change in the camera direction toward the second direction is detected, information related to a change in the position information of the at least one marker is used to determine whether the camera direction can be corrected. When it is determined that the camera direction can be corrected, the position data of the at least one moving object is obtained using the information related to the change in the position information of the at least one marker without returning the camera direction of the camera in the real world from the second direction to the first direction.

2. The device for positioning a mobile object according to claim 1, wherein: The initial data includes the world coordinate data of the configuration mark, When it is determined that the camera direction can be corrected, the camera whose camera direction has changed among the multiple cameras is determined based on the first information including multiple images obtained by each camera photographing the mark and the position of the mark in the space, the camera direction of the determined camera is corrected based on the first information, and the position data of the at least one moving object is obtained using the corrected camera direction.

3. The position positioning device for a mobile object according to claim 1 or 2, characterized in that: The imaging direction of the camera changes from the first direction to the second direction due to an external factor such as vibration or a time lapse factor.

4. The position positioning device for a mobile object according to claim 1 or 2, characterized in that: The at least one processor performs the following processing: When detecting the change in the second direction, a correction value for correcting the change in the imaging direction of the camera is calculated based on the image coordinate data of the marker included in the captured image and the world coordinate data of the marker.

5. The device for positioning a mobile object according to claim 4, characterized in that: The at least one processor performs the following processing: The position data is acquired based on the correction value.

6. A device for positioning a mobile object, characterized in that: have: communications device; and At least 1 processor, The at least one processor performs the following processing: A camera fixed to the moving body and changing its imaging direction from a first direction to a second direction acquires, via the communication device, an image of a plurality of markers arranged in the space. detecting a change in the imaging direction from the first direction to the second direction based on the imaging image and the initial data, When a change of the camera direction toward the second direction is detected, information related to the change of the position information of the plurality of markers is used to determine whether the camera direction can be corrected. When it is determined that the camera direction can be corrected, the position data of the moving object is acquired using the information related to the change in the position information of the plurality of markers without returning the camera direction of the camera in the real world from the second direction to the first direction.

7. The device for positioning a mobile object according to claim 6, characterized in that: The initial data includes the world coordinate data of the configuration mark, When it is determined that the camera direction can be corrected, the camera whose camera direction has changed among the multiple cameras is determined based on the first information including multiple images obtained by each camera photographing the mark and the position of the mark in the space, the camera direction of the determined camera is corrected based on the first information, and the position data of the moving object is obtained using the corrected camera direction.

8. The position positioning device for a mobile object according to claim 6 or 7, characterized in that: The imaging direction of the camera changes from the first direction to the second direction due to an external factor such as vibration or a time lapse factor.

9. The device for positioning a mobile object according to claim 6 or 7, characterized in that: The at least one processor performs the following processing: When detecting the change in the second direction, a correction value for correcting the change in the imaging direction of the camera is calculated based on the image coordinate data of the marker included in the captured image and the world coordinate data of the marker.

10. The device for positioning a mobile object according to claim 9, characterized in that: The at least one processor performs the following processing: The position data is acquired based on the correction value.

11. The position positioning device for a mobile object according to claim 6 or 7, characterized in that: The captured image is captured when the moving object is located at a start position.

12. The device for positioning a mobile object according to claim 6 or 7, characterized in that: The mobile body includes an inertial sensor for detecting the posture of the camera. The posture of the camera except at least the rolling is corrected based on the sensor value of the inertial sensor.

13. A method for positioning a mobile object, characterized in that: At least one processor of the position positioning device performs the following processing: A camera whose imaging direction changes from a first direction to a second direction acquires, via a communication device, an image including at least one marker arranged in a space and at least one moving object moving in the space, detecting a change in the imaging direction from the first direction to the second direction based on the imaging image and the initial data, When a change in the camera direction toward the second direction is detected, information related to a change in the position information of the at least one marker is used to determine whether the camera direction can be corrected. When it is determined that the camera direction can be corrected, the position data of the at least one moving object is obtained using the information related to the change in the position information of the at least one marker without returning the camera direction of the camera in the real world from the second direction to the first direction.

14. The method for locating a position of a moving object according to claim 13, characterized in that: The initial data includes the world coordinate data of the configuration mark, When it is determined that the camera direction can be corrected, the camera whose camera direction has changed among the multiple cameras is determined based on the first information including multiple images obtained by each camera photographing the mark and the position of the mark in the space, the camera direction of the determined camera is corrected based on the first information, and the position data of the at least one moving object is obtained using the corrected camera direction.

15. The method for positioning a mobile object according to claim 13 or 14, characterized in that: The imaging direction of the camera changes from the first direction to the second direction due to an external factor such as vibration or a time lapse factor.

16. The method for locating a position of a mobile object according to claim 13 or 14, characterized in that: The at least one processor performs the following processing: When detecting the change in the second direction, a correction value for correcting the change in the imaging direction of the camera is calculated based on the image coordinate data of the marker included in the captured image and the world coordinate data of the marker.

17. The method for locating a position of a moving object according to claim 16, characterized in that: The at least one processor performs the following processing: The position data is acquired based on the correction value.

18. A method for positioning a mobile object, characterized in that: At least one processor of the position positioning device performs the following processing: A camera fixed to the moving body and changing its imaging direction from a first direction to a second direction acquires, via a communication device, an image obtained by imaging a plurality of markers arranged in a space. detecting a change in the imaging direction from the first direction to the second direction based on the imaging image and the initial data, When a change of the camera direction toward the second direction is detected, information related to the change of the position information of the plurality of markers is used to determine whether the camera direction can be corrected. When it is determined that the camera direction can be corrected, the position data of the moving object is acquired using the information related to the change in the position information of the plurality of markers without returning the camera direction of the camera in the real world from the second direction to the first direction.

19. The method for locating a position of a moving object according to claim 18, characterized in that: The initial data includes the world coordinate data of the configuration mark, When it is determined that the camera direction can be corrected, the camera whose camera direction has changed among the multiple cameras is determined based on the first information including multiple images obtained by each camera photographing the mark and the position of the mark in the space, the camera direction of the determined camera is corrected based on the first information, and the position data of the moving object is obtained using the corrected camera direction.

20. The method for locating a position of a mobile object according to claim 18 or 19, characterized in that: The imaging direction of the camera changes from the first direction to the second direction due to an external factor such as vibration or a time lapse factor.

21. The method for locating a position of a mobile object according to claim 18 or 19, characterized in that: The at least one processor performs the following processing: When detecting the change in the second direction, a correction value for correcting the change in the imaging direction of the camera is calculated based on the image coordinate data of the marker included in the captured image and the world coordinate data of the marker.

22. The method for locating a position of a moving object according to claim 21, characterized in that: The at least one processor performs the following processing: The position data is acquired based on the correction value.

23. The method for positioning a mobile object according to claim 18 or 19, characterized in that: The captured image is captured when the moving object is located at a start position.

24. The method for locating a position of a mobile object according to claim 18 or 19, characterized in that: The mobile body includes an inertial sensor for detecting the posture of the camera. The posture of the camera except at least the rolling is corrected based on the sensor value of the inertial sensor.

Images