Information processing device, program, and positioning method

Abstract

To provide an information processor, a program, and a method for positioning that can keep the number of markers small and can derive an accurate positioning result.SOLUTION: The information processor has a processor which acquires an image marker position showing the position of a marker 5 in an image coordinate system and an image auxiliary marker position showing the position of an auxiliary marker 6 by image processing from a taken image 7 including the marker 5 and the auxiliary marker 6 provided in a space imaged by an imaging unit 2 provided on a side of a mobile body 3, acquires a one-dimensional world imaging position showing the three-dimensional position of the imaging unit 2 in a world coordinate system on the basis of the image marker position and a world known marker position showing a known three-dimensional position of the marker in the world coordinate system, relates the image auxiliary marker position and the world known auxiliary marker position to each other on the basis of the one-dimensional world imaging position and the world known auxiliary marker position showing a known three-dimensional position of the auxiliary marker 6 in the world coordinate system, and determines the position of the imaging unit 2 on the basis of the auxiliary marker position and the world known auxiliary marker position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, a program, and a positioning method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a technique for estimating the position of a moving object or the like by using a captured image.

[0003] Patent Document 1 relates to a method for identifying a self-location of a moving body that travels autonomously in an environment where multiple markers are installed indoors and above. In the method for identifying a self-location of a moving body in Patent Document 1, the positions of multiple markers installed on a ceiling or the like are preregistered. To identify the self-location, marker candidate points are extracted from a ceiling image captured from the moving body, and two-dimensional candidate point coordinates are calculated. In addition, multiple arbitrary virtual points are set within the moving body travel area, and the two-dimensional coordinates of the markers on the image obtained when the moving body is present at the virtual points are derived from the three-dimensional registered positions, and the candidate point coordinates and the two-dimensional coordinates are compared. Then, the virtual point of the moving body that corresponds to the two-dimensional coordinates that are most similar to the candidate point coordinates is estimated as the self-location. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-315746 A Summary of the Invention [Problem to be solved by the invention]

[0005] The more markers there are, the higher the accuracy of position measurement. For example, in the technology described in Patent Document 1, the number of markers installed in the movement range is directly related to the position accuracy. However, if you try to realize positioning at multiple locations, it becomes necessary to install many markers in places where movement is expected.

[0006] An object of the present invention is to provide an information processing device, a program, and a positioning method that reduce the number of markers and derive highly accurate positioning results. [Means for solving the problem]

[0007] In order to achieve the above object, an information processing device according to one aspect of the present invention is provided in a space captured by an imaging unit provided in a moving object. It is a light-emitting body that transmits optical communication data by changing the light-emitting mode. First marker and The light-emitting body does not transmit the optical communication data. The imaging device is characterized in that it comprises a processing unit that acquires, by image processing, a first image marker position indicating the position of the first marker in an image coordinate system and a second image marker position indicating the position of the second marker from an captured image including the second marker, acquires a world imaging position indicating the three-dimensional position of the imaging unit in the world coordinate system based on the first image marker position and a first world known marker position indicating the known three-dimensional position of the first marker in the world coordinate system, matches the second image marker position with the second world known marker position based on the world imaging position and a second world known marker position indicating the known three-dimensional position of the second marker in the world coordinate system, and determines the position of the imaging unit based on the second image marker position and the second world known marker position. Effect of the Invention

[0008] According to the present invention, it is possible to provide an information processing device, a program, and a positioning method that reduce the number of markers and derive a positioning result with high position accuracy. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic diagram showing an overall configuration of a positioning system including an information processing device according to a first embodiment of the present invention. [Diagram 2] 3 is a schematic diagram of a captured image of a ceiling including markers and auxiliary markers acquired by the information processing device according to the first embodiment. FIG. [Diagram 3] 1 is a block diagram showing the hardware configurations of an information processing device and an imaging unit according to a first embodiment. [Figure 4] 1 is a functional block diagram showing a functional configuration of an information processing device according to a first embodiment. [Diagram 5] 4 is a schematic diagram showing an example of the positions of a marker and an auxiliary marker according to the first embodiment. FIG. [Figure 6] 1 is a diagram showing, in a table format, an example of the positions of a marker and an auxiliary marker according to the first embodiment. FIG. [Figure 7] 4 is a flowchart illustrating the flow of a positioning process executed by the information processing device according to the first embodiment. [Figure 8] 1A and 1B are schematic diagrams illustrating a process of extracting a high luminance area from a captured image. [Figure 9] 11 is a schematic diagram illustrating a process of identifying an auxiliary marker to be used for positioning from an extracted region. FIG. [Figure 10] 10A and 10B are schematic diagrams illustrating a process of identifying a marker from a captured image. [Figure 11] FIG. 4 is a diagram showing a list of image marker positions, image auxiliary marker positions, and world known marker positions according to the first embodiment. [Figure 12] FIG. 11 is a diagram showing a table of the relationship between world known auxiliary marker positions and calculated auxiliary marker positions according to the first embodiment. [Figure 13] 6 is a diagram showing a relationship between a calculated assistant marker position and an image assistant marker position on an image coordinate system according to the first embodiment. FIG. [Figure 14] 13 is a diagram showing an example of a table associating an image assistant marker position with a world known assistant marker position based on an image assistant marker position and a calculated assistant marker position according to the first embodiment. FIG. [Figure 15] 13 is a schematic diagram showing an example in which the auxiliary marker according to the second embodiment is a rectangle. FIG. [Figure 16] 13 is a schematic diagram showing an example of assigning auxiliary marker positions to vertices of a polygonal auxiliary marker according to the second embodiment. FIG. [Figure 17] 13 is a flowchart illustrating a process for acquiring an image assistant marker position when the assistant marker according to the second embodiment is a polygon. [Figure 18] 13A is a schematic diagram showing an example of a case in which the size of an auxiliary marker according to the second embodiment is smaller than a predetermined size, and FIG. 13B is a schematic diagram showing an example of a case in which the boundary portion of the auxiliary marker is unclear. [Figure 19] FIG. 11 is a diagram showing an example of a table of world known auxiliary marker positions when the auxiliary marker according to the second embodiment is a rectangle. [Figure 20] 11 is a conceptual diagram showing an example in which an auxiliary marker in a captured image is identified in correspondence with a world-known auxiliary marker position, and the identified world-known auxiliary marker position is used as a marker. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] <Positioning System 1> First, an overview of a positioning system 1 will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing the overall configuration of a positioning system 1 including an information processing device 10 according to a first embodiment of the present invention. Fig. 2 is a schematic diagram of a captured image 7 of a ceiling 4 including a marker 5 and an auxiliary marker 6 acquired by the information processing device 10 according to the first embodiment.

[0012] As shown in FIG. 1, the positioning system 1 includes an imaging unit 2 mounted on the side of a moving object 3 rather than on the side of a building 9 including the ceiling, a marker 5 installed inside the building 9 through which the moving object 3 moves, and an information processing device 10 that locates the position of the moving object 3 based on an image 7 captured by the imaging unit 2.

[0013] The imaging unit 2 acquires captured images 7 while mounted on the moving body 3. The imaging unit 2 of this embodiment continuously captures images of the ceiling 4 above at a predetermined frame rate, and acquires multiple captured images 7 of the ceiling 4 that are continuous in time.

[0014] The imaging unit 2 is also connected to a communication device 28 for communicating with the information processing device 10. The captured image 7 acquired by the imaging unit 2 or various information based on the captured image 7 is transmitted to the information processing device 10 via the communication device 28. In this example, the moving object 3 on which the imaging unit 2 is mounted is a work vehicle such as a forklift.

[0015] The marker 5 is an object including a device or a sign that can obtain three-dimensional position information in the world coordinate system 70 where the marker 5 is located from the database 19 by either transmitting data from the marker 5 itself or by having the imaging unit 2 image the marker 5 and processing the image.

[0016] The marker 5 of this embodiment is a light-emitting device capable of controlling the light emission mode, such as the color and timing of visible light. The marker 5 optically communicates identification information, for example, by changing color or blinking according to a predetermined pattern. The light emitted by the marker 5 may be near-infrared light instead of visible light. In other words, it may be any light waveband that can be captured by a camera.

[0017] In this embodiment, a plurality of markers 5 are installed to identify the position and orientation of the imaging unit 2. In the example shown in Fig. 1, the markers 5 are installed at two locations on the ceiling 4. Note that the markers 5 do not necessarily need to be installed on the ceiling 4, and may be installed inside the building 9 in which the mobile object 3 that can be imaged by the imaging unit 2 moves.

[0018] The auxiliary marker 6 does not transmit data itself. Furthermore, when the imaging unit 2 captures an image of the auxiliary marker 6, the three-dimensional position information in the world coordinate system 70 in which the auxiliary marker 6 is set cannot be acquired from the database 19 by image processing alone. The auxiliary marker 6 is, for example, a light source or lighting installed on the ceiling 4. In this specification, the auxiliary marker 6 means an area that can be identified from the captured image 7 by image processing. For example, luminance or chromaticity can be used to identify the area.

[0019] A window 8 is installed in the building 9, and when the window 8 is captured in the captured image 7, the luminance condition may be met depending on the time of day, but image processing is performed to control not to process the window 8 area as the auxiliary marker 6. Note that the auxiliary marker 6 in this embodiment is not limited to a self-luminous object. A white reflective object that reflects light, or an area painted green on a brown plane, etc. may be identified as the auxiliary marker 6 from the captured image 7 by image processing.

[0020] The information processing device 10 locates the position of the imaging unit 2 based on the known positions of the marker 5 and the auxiliary marker 6, and the image marker position 51 of the marker 5 and the image auxiliary marker position 61 of the auxiliary marker 6 included in the captured image 7. The known positions of the marker 5 and the auxiliary marker 6 may be specified in advance from design drawings, for example, or may be specified by another positioning device. In this specification, the position of the marker 5 or the auxiliary marker 6 is expressed in a three-dimensional world coordinate system 70.

[0021] The information processing device 10 of this embodiment is connected to a communication device 18 for communicating with the imaging unit 2. The information processing device 10 acquires the captured image 7 from the imaging unit 2 by establishing communication between the communication device 18 on the information processing device 10 side and the communication device 28 on the imaging unit 2 side.

[0022] <Hardware configuration> FIG. 3 is a block diagram showing the hardware configuration of the information processing device 10 and the imaging unit 2 according to the first embodiment.

[0023] In the example of FIG. 3, the information processing device 10 includes a processor 11, a read only memory (ROM) 12, a random access memory (RAM) 13, an input / output unit 14, a communication unit 15, and a storage unit 16.

[0024] The processor 11 performs various calculations and processes. The processor 11 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA). Alternatively, the processor 11 is a combination of a plurality of these. The processor 11 may also be a combination of these with a hardware accelerator or the like.

[0025] The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus. The processor 11 executes various processes according to a program recorded in the ROM 12 or a program loaded into the RAM 13. A part or the whole of the program may be incorporated in the circuitry of the processor 11.

[0026] The input / output unit 14 is composed of a keyboard, various buttons, a microphone, etc., and inputs various information in response to user's instruction operations. The input / output unit 14 is also composed of a display, a speaker, etc., and outputs images and sounds. An information terminal with a touch panel is included. The communication unit 15 is a network interface for communicating with the imaging unit 2 etc. via the communication device 28. The storage unit 16 is an area for storing various information such as the captured image 7 and known information.

[0027] Next, a description will be given of an example of a hardware configuration of the imaging unit 2. The imaging unit 2 includes an optical lens unit 21 and an image sensor 22.

[0028] The optical lens unit 21 is composed of lenses that collect light in order to capture an image of a subject, such as a focus lens or a zoom lens. The focus lens is a lens that forms an image of a subject on the light receiving surface of the image sensor 22. The zoom lens is a lens that can freely change the focal length within a certain range. The optical lens unit 21 is also provided with peripheral circuits that adjust setting parameters such as focus, exposure, and white balance as necessary.

[0029] The image sensor 22 is composed of a photoelectric conversion element, an AFE (Analog Front End), and the like. The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element. An object image is incident on the photoelectric conversion element from the optical lens unit 21. The photoelectric conversion element photoelectrically converts (captures) the object image, accumulates the image signal for a certain period of time, and sequentially supplies the accumulated image signal as an analog signal to the AFE. The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on this analog image signal. A digital signal is generated by this various signal processing, and the captured image 7 is output as an output signal of the imaging unit 2.

[0030] <Functional configuration of information processing device> Fig. 4 is a functional block diagram showing the functional configuration of the information processing device 10 according to the first embodiment. Fig. 4 shows the functional configuration as described above, and also shows the relationship of the processing flow.

[0031] The information processing device 10 performs various controls of the information processing device 10 using a processing unit 100 that is realized by the processor 11 executing arithmetic processing based on a predetermined program.

[0032] The processing unit 100 of this embodiment has, by function, an image processing unit 101, a marker identification unit 102, a known information acquisition unit 103, a primary positioning unit 104, a calculation unit 105, an auxiliary marker identification unit 106, a final positioning unit 107, and an output unit 108.

[0033] The image processing unit 101 acquires the captured image 7 and performs pre-processing such as distortion processing. The marker identification unit 102 executes processing to identify the marker 5 from the captured image 7, thereby acquiring the image marker position 51.

[0034] The known information acquisition unit 103 executes a process of acquiring pre-set known information such as the imaging unit internal parameters 25, the imaging unit installation parameters 26, the world known marker positions 52, and the world known auxiliary marker positions 62 from the database 19. The database 19 may be constructed in the storage unit 16 of the information processing device 10, or may be constructed in an external server of the information processing device 10.

[0035] The primary positioning unit 104 executes a process of primarily locating the position of the imaging unit 2 based on the image marker positions 51 acquired from the captured image 7 and the world known marker positions 52 acquired from the database 19, and acquiring the primary world imaging position 23. This primary world imaging position 23 includes the position of the imaging unit 2 and the direction in which the imaging unit 2 is facing.

[0036] The calculation unit 105 executes a projection calculation process that calculates, by projecting, from the world known auxiliary marker position 62, a position on the captured image 7 obtained when the imaging unit 2 captures an image. The auxiliary marker identification unit 106 executes a process of associating and identifying the image auxiliary marker position 61 of the auxiliary marker 6 acquired from the captured image 7 by image processing with the world known auxiliary marker position 62. In this embodiment, this association process is executed, and the position of the moving object 3 is calculated as a PnP problem including the image auxiliary marker position 61 and the world known auxiliary marker position 62 in addition to the image marker position 51 and the world known marker position 52. As a result, even if only a small number of markers 5, for example two, are installed, the position of the moving object 3 can be calculated with high accuracy by utilizing the auxiliary markers 6 such as existing lighting.

[0037] The confirmed positioning unit 107 executes a process of acquiring a confirmed world imaging unit position 24, which is the final position of the imaging unit 2, based on the primary position of the imaging unit 2, the world known marker position 52, the world known auxiliary marker position 62, etc. This confirmed world imaging unit position 24 includes the position of the imaging unit 2 and the direction in which the imaging unit 2 is facing. The output unit 108 executes a process of outputting the positioning result.

[0038] In this example, the information processing device 10 is located at a location away from the imaging unit 2, but is not limited to this configuration. For example, the information processing device 10 may be configured to be mounted on the moving object 3 or the imaging unit 2.

[0039] <Known information> Next, examples of known information registered in advance in the database 19 will be described with reference to FIGS.

[0040] Fig. 5 is a schematic diagram showing the planar installation of the marker 5 and auxiliary marker 6 according to the first embodiment. Fig. 5 shows the positional relationship of the marker 5 and auxiliary marker 6 installed on the ceiling 4 in a plan view. In this example, the two markers 5 are given identification numbers 5a and 5b and are each treated as individual markers. In this embodiment, the 16 auxiliary markers 6 are similarly given identification numbers 6a to 6p and are each treated as individual markers.

[0041] FIG. 6 is a diagram showing an example of positions of the marker 5 and the auxiliary marker 6 in the world coordinate system 70 according to the first embodiment in a table format. As described above, the positions of the marker 5 and the auxiliary marker 6 are specified in advance by design drawings, positioning, or the like. FIG. 6 shows world coordinates as three-dimensional positions for the identification numbers 5a and 5b of the marker 5 and the identification numbers 6a to 6p of the auxiliary marker 6. Here, for convenience of explanation, one of the corners of the ceiling 4 is set as the origin, the x-axis and the y-axis are set in a planar view, the vertical direction is set as the z-axis, and the floor surface is set as the origin of the Z-axis. With the three-dimensional positions of the marker 5 and the auxiliary marker 6 registered in advance in the database 19 as known information, a process of specifying the position of the imaging unit 2 is executed based on the captured image 7 acquired by capturing an image of the ceiling 4 by the imaging unit 2.

[0042] <Positioning processing> Next, a description will be given of positioning processing using the captured image 7. Fig. 7 is a flowchart illustrating the flow of positioning processing executed by the information processing device 10 according to the first embodiment.

[0043] When the positioning process starts, the processing unit 100 acquires the captured image 7 captured by the imaging unit 2 and performs distortion correction processing (step S101). The distortion correction processing corrects, for example, barrel aberration of an image captured by a wide-angle lens.

[0044] After the process of step S101, the processing unit 100 performs a process of specifying the position of the auxiliary marker 6 (step S102).

[0045] An example of the process of identifying the auxiliary marker position will be described with reference to Figs. 8 and 9. Fig. 8 is a schematic diagram for explaining the process of extracting a high-luminance area from a captured image 7. As shown in Fig. 8, in order to identify the auxiliary marker 6 from the captured image 7, the image processing unit 101 of the processing unit 100 performs preprocessing by binarizing the captured image 7 using a predetermined threshold value, making only the high-luminance area white and the rest black. As a result, a high-luminance area that is in a saturated area is extracted. However, in this state, the position of the window 8 may also be extracted as a high-luminance area depending on the time of day when there is sunlight, such as during the daytime.

[0046] FIG. 9 is a schematic diagram for explaining a process of identifying the auxiliary marker 6 to be used for positioning from the region extracted in FIG. 8. As shown in FIG. 9, the image processing unit 101 executes a filtering process based on the actual shape and area of ​​the extracted region in order to eliminate the region that does not contribute to positioning due to the window 8, and identifies only those that are likely to be the marker 5 and the auxiliary marker 6. Then, the region center position for each of the multiple auxiliary markers 6 is obtained by image processing, and the region center position is stored as the image auxiliary marker position 61. Then, the auxiliary marker identification unit 106 creates a list of the image auxiliary marker positions 61 of the auxiliary marker 6. Here, the center position is the center of gravity of the region recognized as the auxiliary marker 6 in the world coordinate system 70, the average of each of the x, y, and z coordinates, the intermediate value between the maximum coordinate and the minimum coordinate, etc. are adopted. It is not limited to these, and the center position represents the position of the auxiliary marker 6 as one position.

[0047] The image assistant marker position 61 on the captured image 7 is specified by setting two-dimensional coordinates on the captured image 7. In this specification, the two-dimensional coordinate system on the captured image 7 is referred to as an image coordinate system 71.

[0048] Fig. 11 shows an example of a list of image auxiliary marker positions 61. A primary identification number such as L1 or L2 is assigned to the auxiliary marker 6 given the image auxiliary marker position 61. At this stage, the positions of the multiple auxiliary markers 6 are merely identified from the captured image 7, and it is not possible to determine whether the multiple auxiliary markers 6 correspond to any of the auxiliary markers 6a to 6p shown in Fig. 6, and they cannot be uniquely identified. For this reason, as shown in Fig. 11, at this point, the auxiliary marker 6 is not linked to the world known auxiliary marker position 62.

[0049] Next, the marker identification unit 102 performs a marker identification process (step S103). Fig. 10 is a schematic diagram illustrating a process for identifying the marker 5 from the captured image 7. In the marker identification process, the marker identification unit 102 identifies the position of the marker 5 from the captured image 7 by image processing.

[0050] 11 shows an example of a list of image marker positions 51 and world known marker positions 52. A primary identification number such as M1 or M2 is assigned to a marker 5 given an image marker position 51. Since the markers 5 are identified, for example, the image marker positions 51 of M1 and M2 are linked to the markers 5a and 5b shown in FIG.

[0051] An example of the marker identification process will now be described. The marker identification unit 102 acquires a light emission pattern based on the successive captured images 7, and identifies the position of the marker 5 in the captured images 7 by comparing the light emission pattern with a preset light emission pattern.

[0052] Next, the primary positioning unit 104 performs a positioning possibility determination process (step S104). As the positioning possibility determination process, for example, the primary positioning unit 104 determines whether two or more markers 5 are present in the captured image 7 and whether positioning is possible.

[0053] If positioning is possible using the marker 5 (step S104: Yes), the primary positioning unit 104 performs a primary positioning process (step S105).

[0054] An example of the primary positioning process will be described below. As described above, since the markers 5a and 5b can be individually identified from the captured image 7, the primary positioning unit 104 acquires the world known marker positions 52 of the markers 5a and 5b based on the correspondence relationship in FIG.

[0055] As the primary positioning process, for example, the primary positioning unit 104 performs primary positioning of the imaging unit 2 through PnP positioning process based on the image marker position 51 and the world known marker position 52.

[0056] As is conventionally known, when the positions of a certain point group and a position in the captured image 7 are given, the position and attitude, etc. of the imaging unit 2 can be obtained by PnP positioning processing. When positioning is performed based on six points, the position and imaging direction (in-plane orientation, elevation angle) of the imaging unit 2 can be obtained. Even when the number of points is less than six, for example when the imaging unit 2 is located on a plane parallel to the ceiling 4 on which the marker 5 and auxiliary marker 6 are provided, the position and elevation angle of the imaging unit 2 can be obtained by positioning using only two points, although not with high accuracy.

[0057] Since the moving object 3 moves on the floor surface parallel to the ceiling 4, the primary positioning unit 104 determines the primary world imaging position 23, which is the primary position and elevation angle of the imaging unit 2, through the P2P method. The primary positioning unit 104 locates the primary world imaging position 23, which is the primary position of the imaging unit 2, through the P2P method using the image marker positions 51 represented by identification numbers M1 and M2 and the world known marker positions 52 represented by identification numbers 5a and 5b shown in Fig. 11.

[0058] If positioning is not possible using the marker 5 (step S104: No), the primary positioning unit 104 performs a position estimation process (step S106). As the position estimation process, for example, the primary positioning unit 104 estimates the position of the imaging unit 2 based on the previously confirmed world imaging unit position 24 and the movement vector history, and sets the result as a primary positioning result.

[0059] Next, the calculation unit 105 performs an auxiliary marker recognition process (step S107). As the auxiliary marker recognition process, for example, the calculation unit 105 determines whether the auxiliary marker 6 is present in the captured image .

[0060] If the auxiliary marker 6 is not present in the captured image 7 (step S107: No), the primary positioning unit 104 performs a primary positioning result application process (step S111). In the primary positioning result application process, for example, the primary positioning unit 104 applies the primary positioning result as a confirmed positioning result.

[0061] If the auxiliary marker 6 is present in the captured image 7 (step S107: Yes), the calculation unit 105 performs a calculated auxiliary marker position calculation process (step S108). As the calculated auxiliary marker position calculation process, for example, when the world known auxiliary marker position 62 is imaged from the primary world imaging position 23 obtained based on the primary positioning result, the calculation unit 105 calculates a position of the world known auxiliary marker position 62 imaged in the captured image 7, in other words, a calculated auxiliary marker position 63 indicating a projection position.

[0062] This calculation is a known process that reversely calculates how a specified arbitrary three-dimensional point is drawn on an image from the position and orientation of the imaging unit 2. The calculation itself is a simple matrix calculation. Therefore, the projection is in an infinite field of view, and the back side of the world imaging unit position is also projected onto a plane. If an attempt is made to execute the PnP positioning process as is, the calculation will not be valid or a large error will occur. Therefore, the calculation is performed after selecting only the auxiliary marker 6 at the world known auxiliary marker position 62 that is approximately within the imaging angle of view when imaged from the primary world imaging position 23 obtained by the primary positioning.

[0063] 12 shows a world known auxiliary marker position 62, whose position is known, and a calculated auxiliary marker position 63, which is its drawing position on the captured image 7. The identification number of the calculated auxiliary marker position 63 calculated for the identification number 6a of the world known auxiliary marker position 62 is written as 6a-C.

[0064] The final positioning unit 107 performs a linking process (step S109). As the linking process, for example, the final positioning unit 107 associates the image auxiliary marker position 61 with the calculated auxiliary marker position 63 based on their positional relationship on the same captured image, and links the auxiliary marker 6 to the world known auxiliary marker position 62 from which the calculated auxiliary marker position 63 is calculated.

[0065] Fig. 13 shows the relationship between the calculated auxiliary marker position 63 and the image auxiliary marker position 61. As shown in Fig. 13a, the calculated auxiliary marker position 63 on the captured image 7 of the world known auxiliary marker position 62 including the world known marker position 52 is plotted in the image coordinate system 71. On the other hand, as shown in Fig. 13b, the image auxiliary marker position 61 of the auxiliary marker 6 including the image marker position 51 is plotted in the image coordinate system 71. Fig. 13c is an enlarged view of the upper right of the image coordinate system 71 when these are superimposed. The calculated auxiliary marker position 63 is similar to the image auxiliary marker position 61, but there is a slight deviation due to the positioning error. Points that are the nearest or surrounding points and have a positional relationship that does not destroy the mutual positional relationship are linked to each other. 13C, the calculated assistant marker position 63 of the identification number 6p-C and the image assistant marker position 61 of the identification number L13 are plotted nearby, and the calculated assistant marker position 63 of the identification number 6o-C and the image assistant marker position 61 of the identification number L14 are plotted nearby. These are treated as corresponding coordinate points and are linked or associated with each other.

[0066] 14 shows a list of calculated auxiliary marker positions 63 and image auxiliary marker positions 61 linked in a similar manner by the confirmed positioning unit 107. The world known auxiliary marker position 62 that was the basis for calculating the calculated auxiliary marker position 63 is also displayed. The image auxiliary marker position 61 is linked to the world known auxiliary marker position 62. At this point, the auxiliary marker 6 that has been imaged by the imaging unit 2 but has not been identified is identified as an auxiliary marker 6 whose position is known. This identified auxiliary marker 6 is similar to the marker 5 because it has been identified and its position is known.

[0067] The primary positioning unit 104 determines the primary world imaging position 23 of the imaging unit 2 based on the two markers 5. In contrast, by linking the auxiliary markers 6 to the world known auxiliary marker positions 62 as shown in Fig. 14, the number of markers 5 increases from two to 18, including the 16 identified auxiliary markers 6 from 6a to 6p.

[0068] Furthermore, the final positioning unit 107 performs a PnP positioning process (step S110). In the PnP positioning process, for example, the final positioning unit 107 performs the PnP positioning process based on the world known marker position 52, the world known auxiliary marker position 62, the image marker position 51, and the image auxiliary marker position 61.

[0069] What was a P2P positioning calculation in the primary positioning can be made into a P18P problem in which positioning is performed from the imaging results of 18 3D known points, resulting in high accuracy.

[0070] Then, the output unit 108 outputs the result of the PnP positioning process as a confirmed positioning result.

[0071] The processing unit 100 performs a positioning end determination process (step S112).

[0072] As the positioning end determination process, if the positioning is to be ended (step S112: Yes), the processing unit 100 ends the positioning process. If the positioning is not to be ended (step S112: No), the processing unit 100 returns the positioning process to A in the flowchart.

[0073] In the first embodiment, P2P positioning processing is adopted for primary positioning, and two markers 5 are the minimum number for primary positioning. The minimum number of markers 5 may be six, i.e., P6P. This is because P6P makes it possible to derive the position and imaging direction of the imaging unit 2. Furthermore, this embodiment is not limited to the above two points or six points, and is effective for all PnP positioning processing.

[0074] <Effects of the embodiment> In the first embodiment, the marker 5 is identified by the imaging unit 2 as a method of primary positioning, so there is no need to input initial states such as the position and imaging direction of the imaging unit 2 to the information processing device 10, and the imaging unit 2 can be positioned definitively at any point where the marker 5 can be imaged. The identification function of the marker 5 is capable of receiving even an image that is close to a very small dot on the image. Therefore, the positioning system 1 can be established by adding a minimum number of markers 5 necessary to practically establish primary positioning. As a result, the positioning system 1 can be constructed at low cost and in a short period of time.

[0075] The information processing device 10 includes a processing unit 100 that acquires, by image processing, an image marker position 51 indicating the position of the marker 5 in an image coordinate system 71 and an image auxiliary marker position 61 indicating the position of the auxiliary marker 6 from an captured image 7 including a marker 5 and an auxiliary marker 6 provided in a space captured by an imaging unit 2 provided on the side of a moving object 3, acquires a primary world imaging position 23 indicating the three-dimensional position of the imaging unit 2 in the world coordinate system 70 based on the image marker position 51 and a world known marker position 52 indicating the known three-dimensional position of the marker 5 in the world coordinate system 70, associates the image auxiliary marker position 61 with the world known auxiliary marker position 62 based on the primary world imaging position 23 and the world known auxiliary marker position 62 indicating the known three-dimensional position of the auxiliary marker 6 in the world coordinate system 70, and determines the position of the imaging unit 2 based on the image auxiliary marker position 61 and the world known auxiliary marker position 62.

[0076] An information processing device 10 is provided that reduces the number of markers 5 and derives a positioning result with high position accuracy.

[0077] There is no difficulty in pursuing high accuracy due to the use of radio waves. In addition, there is no problem with the difficulty in guaranteeing definite positioning, as occurs with Visual-SLAM and LiDAR-SLAM, and there is no problem of being trapped in the loop. A practical positioning system can be constructed.

[0078] A processing unit 100 included in the information processing device 10 identifies, as a marker 5, a distinguishable light source of visible light that is distinguished based on the light emission mode in a captured image 7, and acquires an image marker position 51.

[0079] Each marker 5 can be reliably identified from the individual light emission patterns of the imaged markers 5.

[0080] The processing unit 100 provided in the information processing device 10 identifies two or more markers 5 from the captured image 7, obtains multiple image marker positions 51, and obtains a primary world imaging position 23 based on the multiple image marker positions 51 and world known marker positions 52 corresponding to each of the markers 5.

[0081] Even a small number of markers 5 is sufficient to acquire the primary world imaging position 23. Since a small number of markers 5 is sufficient, costs, start-up man-hours, and the like can be kept low.

[0082] The processing unit 100 provided in the information processing device 10 identifies the imaging position and imaging direction of the captured image 7 based on the primary world imaging position 23, calculates the calculated auxiliary marker position 63 by performing a projection calculation process when the world known auxiliary marker position 62 is projected onto the captured image 7, and corresponds the image auxiliary marker position 61 to the world known auxiliary marker position 62 by corresponding the image auxiliary marker position 61 to the calculated auxiliary marker position 63.

[0083] The position of the auxiliary marker 6 that is the basis of the image auxiliary marker position 61 is identified as a world known auxiliary marker position 62. The auxiliary marker 6 can be used in the PnP positioning process in the same way as the marker 5.

[0084] The processing unit 100 provided in the information processing device 10 performs a matching process by matching the world known auxiliary marker position 62, which is the calculation source of the calculated auxiliary marker position 63 located at the image auxiliary marker position 61 or around the image auxiliary marker position 61, to the image auxiliary marker position 61 as a result of the projection calculation process.

[0085] Correspondence can be reliably performed by a simple method of acquiring the image assistant marker positions 61 located around the image coordinate system 71 .

[0086] The program causes the computer to execute a marker 5 and auxiliary marker position acquisition function of acquiring, by image processing, an image marker position 51 indicating the position of the marker 5 in an image coordinate system 71 and an image auxiliary marker position 61 indicating the position of the auxiliary marker 6 from an captured image 7 including a marker 5 and an auxiliary marker 6 provided in a space captured by an imaging unit 2 provided on the side of the moving object 3; a primary positioning function of acquiring a primary world imaging position 23 indicating the three-dimensional position of the imaging unit 2 in the world coordinate system 70 based on the image marker position 51 and a world known marker position 52 indicating the known three-dimensional position of the marker 5 in the world coordinate system 70; and a determination positioning function of matching the image auxiliary marker position 61 with the world known auxiliary marker position 62 based on the primary world imaging position 23 and the world known auxiliary marker position 62 indicating the known three-dimensional position of the auxiliary marker 6 in the world coordinate system 70, and determining the position of the imaging unit 2 based on the image auxiliary marker position 61 and the world known auxiliary marker position 62.

[0087] A positioning program is provided that reduces the number of markers 5 and derives positioning results with high position accuracy.

[0088] The position measurement method executed by the computer includes a marker 5 and auxiliary marker position acquisition step of acquiring, by image processing, an image marker position 51 indicating the position of the marker 5 in an image coordinate system 71 and an image auxiliary marker position 61 indicating the position of the auxiliary marker 6 from an captured image 7 including a marker 5 and an auxiliary marker 6 provided in a space captured by an imaging unit 2 provided on the side of the moving object 3; a primary positioning step of acquiring a primary world imaged position 23 indicating the three-dimensional position of the imaging unit 2 in the world coordinate system 70 based on the image marker position 51 and a world known marker position 52 indicating the known three-dimensional position of the marker 5 in the world coordinate system 70; and a determining positioning step of associating the image auxiliary marker position 61 with the world known auxiliary marker position 62 based on the primary world imaged position 23 and the world known auxiliary marker position 62 indicating the known three-dimensional position of the auxiliary marker 6 in the world coordinate system 70, and determining the position of the imaging unit 2 based on the image auxiliary marker position 61 and the world known auxiliary marker position 62.

[0089] A positioning method is provided that reduces the number of markers 5 and derives positioning results with high position accuracy.

[0090] Next, a description will be given of an embodiment different from the first embodiment. In the following description, the same reference numerals will be used to designate configurations common to or similar to those described in the first embodiment, and the description thereof may be omitted.

[0091] Second embodiment In the first embodiment, an example has been described in which the position of the circular auxiliary marker 6 is set based on the center of the area. In the second embodiment described below, the method of identifying the position of the auxiliary marker 6 is different.

[0092] FIG. 15 is a schematic diagram showing an example in which the auxiliary marker 6 according to the second embodiment is a rectangle. Identification numbers 100 to 130 are assigned to each auxiliary marker 6. In the figure, arrows are used to indicate the identification numbers. The same applies below. FIG. 16 is a schematic diagram showing an example in which auxiliary marker positions are assigned to vertices of a quadrangular auxiliary marker 6 according to the second embodiment. As shown in FIG. 15, a rectangular light in a plan view is also used as the auxiliary marker 6. In the second embodiment, as shown in FIG. 16(a), a process of adding vertex part codes 111 to 114 to the world known auxiliary marker position 62 is executed. In FIG. 16(a), arrows are used for the codes. The same applies below. As a result, in the case of a quadrangular auxiliary marker 6, a total of five world known auxiliary marker positions 62, including the center position and the four corners of the auxiliary marker 6, can be obtained as three-dimensional positions in the world coordinate system 70. In the following description, the positions of the vertices are referred to as world known outer shape positions 65. Moreover, the position of the auxiliary marker 6 in the image coordinate system 71 on the captured image 7 is referred to as an image outer shape position 64 .

[0093] A world known auxiliary marker position 62 can also be assigned to the vertices of an auxiliary marker 6 that is not rectangular but has a polygonal shape such as a trapezoid as shown in FIG. 16(b). As shown in FIG. 16(b), for example, a world known auxiliary marker position 62 is assigned to the center of the auxiliary marker 6, and a world known outer shape position 65 is assigned to the positions of the vertices, and each is given an identification number of 200 to 204. In the example of FIG. 16, both are a form of a quadrangle, but the shape is not limited to this. For example, even in the case of a triangle or a shape with pentagons or more, each vertex can be added as a world known outer shape position 65, and the image outer shape positions 64 of the vertices can be obtained, and secondary positioning can be performed.

[0094] <Extraction process of known auxiliary marker positions> 17 is a flowchart for explaining the flow of processing when the auxiliary marker 6 according to the second embodiment is a polygon. By the processing executed by the processing unit 100, a world known auxiliary marker position 62 and a world known outer shape position 65 are extracted.

[0095] When the creation of the table of image auxiliary marker positions 61 starts, first, the image processing unit 101 performs auxiliary marker recognition processing (step S201). In the auxiliary marker recognition processing, for example, the image processing unit 101 extracts an area where the brightness is saturated in the captured image 7, and assigns a primary identification number to the extracted area.

[0096] Furthermore, the image processing unit 101 performs an auxiliary marker selection process (step S202). As the auxiliary marker selection process, for example, the image processing unit 101 selects the obtained regions based on their shapes. For example, the image processing unit 101 extracts ellipses and rectangles, and excludes those that overlap the edges of the image.

[0097] Next, the image processing unit 101 performs table creation processing (step S203). In the table creation processing, for example, the image processing unit 101 registers the extracted candidate regions in a table.

[0098] The auxiliary marker identifying unit 106 performs a selection process (step S204). In the extraction process, for example, the auxiliary marker identifying unit 106 selects one of the regions.

[0099] The auxiliary marker identification unit 106 performs a size determination process. As the size determination process, for example, the auxiliary marker identification unit 106 determines whether the size of the auxiliary marker 6 in the captured image 7 is equal to or larger than a predetermined size. The predetermined size is, for example, a size of 7×7 pixels in the captured image 7.

[0100] The size determination process will be described with reference to Fig. 18. In cases where the image is far away and small, using the points on the image at the four corners may result in errors. Therefore, in the size determination process, to avoid a determination that is unlikely to result in errors, it is determined whether the area that is a candidate for the auxiliary marker 6 is a predetermined size.

[0101] If the extracted region is equal to or larger than the specified size (step S205: Yes), the auxiliary marker identifying unit 106 performs vertex registration processing (step S206). As the vertex registration processing, for example, the auxiliary marker identifying unit 106 registers four vertices of the quadrangular auxiliary marker 6 as the image outer shape position 64. As shown in FIG. 18(a), when there is a candidate for obtaining the image auxiliary marker position 61 as one of the auxiliary markers 6, only a partial region 612 located at the center of the region 611 may be used. Also, as shown in FIG. 18(b), although it appears to be a polygon, the boundary may be blurred and the vertices may not be clear. In such a case, the partial region 612 located at the center of the region 611 may be used.

[0102] Next, the auxiliary marker identifying unit 106 performs a center registration process (step S207). In the center registration process, for example, the auxiliary marker identifying unit 106 registers the center of the extracted area as the image auxiliary marker position 61 regardless of the size of the extracted area.

[0103] The processing unit 100 performs an end determination process (step S208).

[0104] If the process does not end (step S208: No), the processing unit 100 returns the positioning process to step S204. If the process ends (step S208: Yes), the processing unit 100 ends the creation of the table of image assistant marker positions 61.

[0105] 19 shows an example in which a world known outer shape position 65 is identified and registered as a world known auxiliary marker position 62 at the position of a vertex of such a polygonal auxiliary marker 6. A tens digit or higher is used for the center position, and a single digit is used for the world known outer shape position 65, which is the vertex.

[0106] In this embodiment, the auxiliary marker 6 is a light source or a light source installed in the environment, and the auxiliary marker 6 can be used as a complement when the number of markers 5 is small. When indoor lighting is normally captured as the auxiliary marker 6, the brightness is captured as a saturated area, so that the image signal is very stable as the first auxiliary marker area candidate is extracted by simply binarizing it at a saturation value. In addition, lighting is generally installed in a place with good visibility in the environment, and its shape is often a simple shape such as a circle or a square, which is suitable as the auxiliary marker 6 in this embodiment. The auxiliary marker 6 is not limited to lighting, and other methods may be used as long as it is easy to detect stably and simply in the usage scene. For example, a high-saturation object with good visibility in a low-saturation environment can be binarized at a specific high saturation threshold, so it can be considered as another method.

[0107] As described above, in the second embodiment, an auxiliary marker 6 having a polygonal shape is used, and the world known outer shape positions 65 are increased as one form of the world known auxiliary marker positions 62. It becomes possible to provide a large number of world known auxiliary marker positions 62 with one auxiliary marker or illumination.

[0108] The processing unit 100 included in the information processing device 10 acquires the image auxiliary marker position 61 of the auxiliary marker 6 based on the positions of the vertices of the polygonal shape of the auxiliary marker 6 in the captured image 7 .

[0109] The number of world known auxiliary marker positions 62 can be increased without increasing the number of auxiliary markers 6, thereby improving the accuracy of the PnP positioning process.

[0110] When the size of the auxiliary marker 6 in the captured image 7 is smaller than a predetermined size, the processing unit 100 included in the information processing device 10 acquires the center position of the auxiliary marker 6, and acquires the image auxiliary marker position 61 based on the center position.

[0111] The world known auxiliary marker positions 62 used in the PnP positioning process are optimized, resulting in higher positioning accuracy.

[0112] (Modification of marker) In the above-mentioned positioning process, the marker 5 having an identification function is used as one of the methods for primary positioning. Other methods may be used for primary positioning. Examples are listed below.

[0113] This is not possible in the initial state, but if PnP positioning processing has been performed by the positioning confirmation unit 107 up until just before, the current position can be estimated from the past position and the last vector using the difference, and this can be used as the primary world imaging position 23.

[0114] The two-dimensional code is captured by the imaging unit 2, and the position of the imaging unit 2 can be primarily set based on the code and the imaging angle of the imaging unit 2. When determining position by recognizing a two-dimensional code or the like, it is preferable to display the two-dimensional code in a relatively large area on the imaging screen.

[0115] For example, an infrared sensor is prepared as the specific point, and the position of the imaging unit 2 can be primarily set based on the detection of the passage of the specific point. When the imaging unit 2 is installed on the moving body 3, if it is guaranteed that the starting point of the moving body 3 is a specific point, the position of the imaging unit 2 can be primarily set based on the position of the specific point. Information on this specific point may be stored in the database 19 as the imaging unit installation parameters 26 shown in FIG. 1. For example, if the starting point is a charging station, the position of the imaging unit 2 can be primarily set if the identification number of the charging station is known. In addition, the elevation angle, focal length, presence or absence of an autofocus function, etc. of the imaging unit 2 may be stored in the database 19 as the imaging unit internal parameters 25 shown in FIG. 1. In addition, a combination of these various methods may be used. However, a method using a marker 5 having an identification function, particularly a method using a marker 5 identified by a color change signal of a light-emitting body, is suitable for performing primary positioning over a wide area with relatively high accuracy.

[0116] As explained above, the marker 5 is not limited to the marker 5 of the above embodiment that performs optical communication by changing the light emission pattern, and other types of markers 5 can also be used, as shown in the modified example.

[0117] Next, an example of processing after the world known auxiliary marker position 62 is recognized on the captured image 7 will be described.

[0118] 20 shows a more specific positioning process using a ceiling 4 on which a marker 5 and an auxiliary marker 6 are installed. The recognized auxiliary marker 6 is identified in the same way as the marker 5. In the following description, the identified world known auxiliary marker position 62 will be used by the primary positioning unit 104 for primary positioning in the same way as the marker 5.

[0119] For ease of explanation, it is assumed in Fig. 20 that the markers 5 and auxiliary markers 6 are installed in an orderly fashion, and the imaging unit 2 is installed perpendicular to these. In the figure, the rectangular area indicates the imaging range of the captured image 7. Fig. 20a shows the initial state. The image of the marker 5 is captured, and the image marker position 51 is recognized. Then, the image auxiliary marker position 61 is obtained. The marker 5 is colored black in Fig. 20a to indicate that it has been identified.

[0120] As in the first embodiment, the primary world imaging position 23 of the imaging unit 2 is calculated based on the image marker position 51 and the world known marker position 52. Then, a calculated auxiliary marker position 63 is calculated when an auxiliary marker 6 located at a world known auxiliary marker position 62 is imaged from the primary world imaging position 23. That is, a projection calculation process is performed. Then, for example, by associating nearby points, the auxiliary marker 6 that produces the image auxiliary marker position 61 is linked to the world known auxiliary marker position 62 that is the calculation source. As a result, the auxiliary marker 6 in the captured image 7 is recognized as an identified auxiliary marker 6 whose position is known. Here, the auxiliary marker 6 becomes an identified index similar to the marker 5. To show this situation, the auxiliary marker 6 that was represented by a white circle in FIG. 19a is painted black in FIG. 20b.

[0121] From this, when the moving object 3 moves continuously, the captured image 7 captured by the imaging unit 2 changes as shown in FIG. 20c. Here, the image auxiliary marker position 61 in the captured image 7 is linked to the world known auxiliary marker position 62 by image processing. In FIG. 20C, the image marker position 51 and the image auxiliary marker position 61 of the marker 5 and the auxiliary marker 6 are maintained by blackening, which means identification. The four blackened points in the captured image 7 in FIG. 20c are identified, so they can be used for the primary positioning of the imaging unit 2, similar to the image marker position 51 in FIG. 20a. Then, through comparison with the calculated auxiliary marker position 63, the image auxiliary marker position 61 in the captured image 7 in FIG. 20C is newly linked to the world known auxiliary marker position 62, and becomes an identified index similar to the marker 5. To show this situation, the auxiliary marker 6 shown as a white circle in FIG. 20C is blackened in FIG. 20d.

[0122] This process is repeated. Fig. 20e shows the captured image 7 when the moving object 3 has moved further following Fig. 20c. At this point, the image marker position 51 caused by the marker 5 is not present in the captured image 7. The primary positioning unit 104 performs primary positioning of the position of the imaging unit 2 based on the identified auxiliary marker 6, and further, the final positioning unit 107 performs final positioning of the position of the imaging unit 2.

[0123] As a result, the markers 5 are placed only near the initial operation position, and thereafter, the auxiliary markers 6 are continuously located and identified, making it possible to place the markers 5 only in the initial positions even in a wide area. That is, the process transition using the markers 5 makes it possible to derive the confirmed world imaging unit position 24 of the imaging unit 2 even if the marker 5 is not included in the captured image 7, as long as the auxiliary marker 6 is included.

[0124] The above describes the positioning process using the marker 5. Note that the present invention is not limited to the above-described embodiment and modified examples, and the present invention includes modifications, improvements, etc. within the scope of achieving the object of the present invention.

[0125] In the above embodiment, the information processing device 10 to which the present invention is applied has been described using a forklift as an example of the moving object 3, but is not particularly limited thereto. For example, the present invention can be applied to electronic devices in general that have an image processing function. Specifically, the present invention can be applied to, for example, notebook personal computers, portable navigation devices, mobile phones, smartphones, portable game consoles, and the like.

[0126] The above-mentioned series of processes can be executed by hardware or software. In other words, the functional configuration of Fig. 4 is merely an example and is not particularly limited. That is, it is sufficient that the information processing device 10 has a function capable of executing the above-mentioned series of processes as a whole, and the type of functional block used to realize this function is not particularly limited to the example of Fig. 4.

[0127] Moreover, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination of both. The functional configuration in this embodiment is realized by a processor that executes arithmetic processing, and processors that can be used in this embodiment include those configured as single pieces of various processing devices such as single processors, multiprocessors, and multicore processors, as well as those in which these various processing devices are combined with processing circuits such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field-Programmable Gate Arrays).

[0128] When a series of processes is executed by software, the programs constituting the software are installed into a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a general-purpose personal computer.

[0129] A recording medium including such a program is not only distributed separately from the device main body in order to provide the program to the user, but is also configured as a recording medium provided to the user in a state in which it is pre-installed in the device main body. The removable medium is, for example, a magnetic disk (including a floppy disk), an optical disk, or a magneto-optical disk. The optical disk is, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), a Blu-ray (registered trademark) Disc, etc. The magneto-optical disk is, for example, an MD (Mini-Disk), etc. Also, the recording medium provided to the user in a state in which it is pre-installed in the device main body is, for example, a ROM 12 in which the program is recorded, a hard disk (not shown), etc.

[0130] In this specification, the steps describing the program recorded on the recording medium include not only processes performed in chronological order, but also processes that are not necessarily performed in chronological order but are performed in parallel or individually. In addition, in this specification, the term "system" refers to an overall device composed of multiple devices, multiple means, etc.

[0131] Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and further, various modifications such as omissions and substitutions can be made without departing from the gist of the present invention. These embodiments and their modifications are included in the scope and gist of the invention described in this specification, etc., and are included in the scope of the invention described in the claims and their equivalents.

[0132] The invention as described in the claims of the present application as originally filed is set forth below.

[0133] [Appendix 1] acquiring an image marker position indicating a position of the marker and an image auxiliary marker position indicating a position of the auxiliary marker in an image coordinate system from a captured image including the marker and the auxiliary marker that are captured by an imaging unit provided on the moving body side, by image processing; obtaining a primary world imaging position indicating a three-dimensional position of the imaging unit in the world coordinate system based on the image marker position and a world known marker position indicating a known three-dimensional position of the marker in the world coordinate system; Correlating the image assistant marker position with the world known assistant marker position based on the primary world imaging position and a world known assistant marker position indicating a known three-dimensional position of the assistant marker in a world coordinate system; determining a position of the imaging unit based on the image assistant marker position and the world known assistant marker position; An information processing device comprising a processing unit.

[0134] [Appendix 2] The processing unit includes: A light source of visible light that is identified based on a light emission pattern is identified as the marker from the captured image, and the position of the image marker is acquired. 2. An information processing device according to claim 1.

[0135] [Appendix 3] The processing unit includes: Identifying two or more of the markers from the captured image and obtaining a plurality of image marker positions; Obtaining the primary world imaging position based on a plurality of the image marker positions and the world known marker positions corresponding to each of the markers. 3. An information processing device according to claim 1 or 2.

[0136] [Appendix 4] The processing unit includes: The imaging position and imaging direction of the captured image are identified based on the primary world imaging position, a calculated auxiliary marker position is calculated by performing a projection calculation process in the case where the world known auxiliary marker position is projected onto the captured image, and the image assistant marker position is associated with the calculated assistant marker position, thereby associating the image assistant marker position with the world known assistant marker position. 4. An information processing device according to any one of claims 1 to 3.

[0137] [Appendix 5] The processing unit includes: As a result of the projection calculation process, the world known auxiliary marker position from which the calculated auxiliary marker position located at or around the image auxiliary marker position is calculated is associated with the image auxiliary marker position, thereby executing a matching process. 5. The information processing device according to claim 4.

[0138] [Appendix 6] The processing unit includes: The image auxiliary marker position of the auxiliary marker is acquired based on the positions of vertices of the polygonal shape of the auxiliary marker in the captured image. 6. An information processing device according to any one of appendix 1 to 5.

[0139] [Appendix 7] The processing unit includes: When the size of the auxiliary marker in the captured image is smaller than a predetermined size, the center position of the auxiliary marker is acquired, and the image auxiliary marker position is acquired based on the center position. 7. The information processing device according to claim 6.

[0140] [Appendix 8] To your computer a marker and auxiliary marker position acquisition function for acquiring an image marker position indicating a position of the marker and an image auxiliary marker position indicating a position of the auxiliary marker in an image coordinate system from an image including the marker and the auxiliary marker captured by an imaging unit provided on the moving body side; a primary positioning function that acquires a primary world imaging position indicating a three-dimensional position of the imaging unit in a world coordinate system based on the image marker position and a world known marker position indicating a known three-dimensional position of the marker in a world coordinate system; Correlating the image assistant marker position with the world known assistant marker position based on the primary world imaging position and a world known assistant marker position indicating a known three-dimensional position of the assistant marker in a world coordinate system; and a positioning function for determining the position of the imaging unit based on the image assistant marker position and the world known assistant marker position.

[0141] [Appendix 9] 1. A computer implemented method for position determination, comprising: a marker and auxiliary marker position acquisition step of acquiring an image marker position indicating a position of the marker and an image auxiliary marker position indicating a position of the auxiliary marker in an image coordinate system from an image including the marker and the auxiliary marker captured by an imaging unit provided on the moving body side; a primary positioning step of acquiring a primary world imaging position indicating a three-dimensional position of the imaging unit in a world coordinate system based on the image marker position and a world known marker position indicating a known three-dimensional position of the marker in a world coordinate system; Correlating the image assistant marker position with the world known assistant marker position based on the primary world imaging position and a world known assistant marker position indicating a known three-dimensional position of the assistant marker in a world coordinate system; and determining a position of the imaging unit based on the image assistant marker position and the world known assistant marker position. [Explanation of symbols]

[0142] 1. Enthronement System 2. Imaging unit 3. Mobile 4. Ceiling 5 Markers 6 Auxiliary Markers 8. Windows 9. Building 10. Information processing device 18 Communication Equipment 28 Imaging unit communication device

Claims

1. from a captured image including a first marker, which is an illuminant that transmits optical communication data by changing an emission mode provided in a space captured by an imaging unit, and a second marker, which is an illuminant that does not transmit the optical communication data, a first image marker position indicating a position of the first marker and a second image marker position indicating a position of the second marker in an image coordinate system are acquired by image processing; acquiring a world imaging position indicating a three-dimensional position of the imaging unit in the world coordinate system based on the first image marker position and a first world known marker position indicating a known three-dimensional position of the first marker in a world coordinate system; Correlating the second image marker position with the second world known marker position based on the world imaging position and a second world known marker position indicating a known three-dimensional position of the second marker in a world coordinate system; An information processing device comprising: a processing unit that determines a position of the imaging unit based on the associated second image marker position and the second world known marker position.

2. The imaging unit is provided on an object present in the space. The information processing device according to claim 1 .

3. The processing unit includes: A light source identified based on a light emission mode is identified as the first marker from the captured image, and the position of the first image marker is acquired. The information processing device according to claim 2 .

4. The processing unit includes: Identifying two or more of the first markers from the captured image and obtaining a plurality of the first image marker positions; Obtaining the world imaging position based on the first image marker positions and the first world known marker positions corresponding to each of the first markers.

4. The information processing device according to claim 1.

5. The processing unit includes: The imaging position and imaging direction of the captured image are identified based on the world imaging position, a projection calculation process is performed when the second world known marker position is projected onto the captured image to calculate a second calculated marker position, and the second image marker position is associated with the second calculated marker position, thereby associating the second image marker position with the second world known marker position.

5. An information processing device according to claim 1.

6. The processing unit includes: A second image marker position of the second marker is acquired based on the positions of vertices of the polygonal shape of the second marker in the captured image.

6. An information processing device according to claim 1.

7. The processing unit includes: When the size of the second marker in the captured image is smaller than a predetermined size, the center position of the second marker is acquired, and the second image marker position is acquired based on the center position. The information processing device according to claim 6.

8. To your computer a first marker and second marker position acquisition function for acquiring, from an image captured by an imaging unit provided in a moving object, a first marker, which is an illuminant that transmits optical communication data by changing an emission mode, and a second marker, which is an illuminant that does not transmit the optical communication data, by image processing to acquire a first image marker position indicating the position of the first marker in an image coordinate system and a second image marker position indicating the position of the second marker from an image captured by the imaging unit provided in the moving object; a primary positioning function that acquires a world imaging position indicating a three-dimensional position of the imaging unit in a world coordinate system based on the first image marker position and a first world known marker position indicating a known three-dimensional position of the first marker in a world coordinate system; Correlating the second image marker position with the second world known marker position based on the world imaging position and a second world known marker position indicating a known three-dimensional position of the second marker in a world coordinate system; and a positioning function for determining the position of the imaging unit based on the second image marker position and the second world known marker position.

9. 1. A computer implemented positioning method, comprising: a first marker and second marker position acquisition step of acquiring, from an image captured by an imaging unit provided in a moving object, a first marker, which is an illuminant that transmits optical communication data by changing a light emission mode, and a second marker, which is an illuminant that does not transmit the optical communication data, a first image marker position indicating the position of the first marker in an image coordinate system and a second image marker position indicating the position of the second marker by image processing from an image captured by the imaging unit provided in the moving object; a primary positioning step of acquiring a world imaging position indicating a three-dimensional position of the imaging unit in a world coordinate system based on the first image marker position and a first world known marker position indicating a known three-dimensional position of the first marker in a world coordinate system; Correlating the second image marker position with the second world known marker position based on the world imaging position and a second world known marker position indicating a known three-dimensional position of the second marker in a world coordinate system; A positioning method including a determining positioning step of determining a position of the imaging unit based on the second image marker position and the second world known marker position.

Images