Information Processing Apparatus, Recording Medium, and Positioning Method

Abstract

The present invention provides an information processing apparatus, a recording medium, and a positioning method. The information processing apparatus includes a processing unit that performs the following processing: obtaining the positions of a marker (5) and an auxiliary marker (6) in an image coordinate system from a captured image (7) captured by a camera unit (2) provided on one side of a moving body (3), the captured image including the marker (5) and the auxiliary marker (6) provided in a space; obtaining temporary positioning information related to the camera unit (2) in a world coordinate system based on the position of the marker (5) in the image coordinate system and the position of the marker in the world coordinate system; and determining the position of the camera unit (2) based on the temporary positioning information and the position of the auxiliary marker (6) in the world coordinate system.

Description

[0001] Cross - reference to related applications

[0002] This application claims the priority and benefits of Japanese Patent Application No. 2021 - 151352, filed on September 16, 2021. The entire specification, claims, and drawings of Japanese Patent Application No. 2021 - 151352 are incorporated herein by reference. Technical Field

[0003] The present invention relates to an information processing apparatus, a recording medium, and a positioning method. Background Art

[0004] In the past, techniques for estimating the position of a moving body or the like using captured images have been known.

[0005] In Japanese Patent Application Laid - Open No. 2005 - 315746, a technique related to a method for identifying the own position of a moving body that autonomously travels in an environment where a plurality of markers are provided above the interior is disclosed. In the own - position identification method of Japanese Patent Application Laid - Open No. 2005 - 315746, the positions of a plurality of markers provided on the ceiling or the like are registered in advance. In the identification of the own position, marker candidate points are extracted from a ceiling image captured by the moving body, and two - dimensional candidate point coordinates are calculated. Also, a plurality of arbitrary imaginary points are set within the moving region of the moving body, and the two - dimensional coordinates of the markers on the image obtained when the moving body is present at the imaginary points are respectively derived from the three - dimensional registered positions, and the candidate point coordinates and the two - dimensional coordinates are compared respectively. Then, the imaginary point of the moving body corresponding to the two - dimensional coordinates that are most similar to the candidate point coordinates is determined as the own position.

[0006] If the number of markers is large, the measurement accuracy of the position is improved. For example, in the technique described in Japanese Patent Application Laid - Open No. 2005 - 315746, the number of markers set within the moving range is directly related to the position accuracy. However, if positioning at multiple locations is to be achieved, many markers need to be set in the assumed moving places. Summary of the Invention

[0007] An object of the present invention is to provide an information processing apparatus, a recording medium, and a positioning method that suppress the number of markers and derive a positioning result with high accuracy.

[0008] To achieve the above object, an information processing apparatus according to one aspect of the present invention includes one or more processing units, and the one or more processing units perform the following processing: obtaining the position of a marker provided in a space from an image captured by a imaging unit, obtaining information at the time of positioning related to the imaging unit before determination in the world coordinate system based on the position of the marker in the image coordinate system and the position of the marker in the world coordinate system, and determining information at the time of positioning related to the imaging unit based on the positioning information of the imaging unit before determination in the world coordinate system and the position of the auxiliary marker in the world coordinate system.

[0009] Effects of the Invention

[0010] According to the present invention, an information processing apparatus, a recording medium, and a positioning method that suppress the number of markers and derive a positioning result with high position accuracy can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a schematic diagram showing the overall structure of a positioning system including an information processing apparatus according to the first embodiment of the present invention.

[0012] Figure 2 It is a schematic diagram of a captured image of a ceiling including a marker and an auxiliary marker obtained by the information processing apparatus according to the first embodiment.

[0013] Figure 3 It is a block diagram showing the hardware structure of the information processing apparatus and the imaging unit according to the first embodiment.

[0014] Figure 4 It is a functional block diagram showing the functional structure of the information processing apparatus according to the first embodiment.

[0015] Figure 5 It is a schematic diagram showing an example of the positions of the marker and the auxiliary marker according to the first embodiment.

[0016] Figure 6 It is a diagram showing an example of the positions of the marker and the auxiliary marker according to the first embodiment in tabular form.

[0017] Figure 7 It is a flowchart for explaining the process flow of the positioning process executed by the information processing apparatus according to the first embodiment.

[0018] Figure 8 It is a schematic diagram for explaining the process of extracting a region with high brightness from a captured image.

[0019] Figure 9 It is a schematic diagram for explaining the process of determining an auxiliary marker used for positioning from the extracted region.

[0020] Figure 10 It is a schematic diagram illustrating the process of determining a marker from a captured image.

[0021] Figure 11 It is a diagram showing a list of the image marker position, the image auxiliary marker position, and the world-known marker position related to the first embodiment.

[0022] Figure 12 It is a diagram showing a table of the relationship between the world-known auxiliary marker position and the calculated auxiliary marker position related to the first embodiment.

[0023] Figure 13A It is a diagram showing the relationship on the image coordinate system between the calculated auxiliary marker position and the image auxiliary marker position related to the first embodiment.

[0024] Figure 13B It is a diagram showing the relationship on the image coordinate system between the calculated auxiliary marker position and the image auxiliary marker position related to the first embodiment.

[0025] Figure 13C It is a diagram showing the relationship on the image coordinate system between the calculated auxiliary marker position and the image auxiliary marker position related to the first embodiment.

[0026] Figure 14 It is a diagram showing an example of a table that correlates the world-known auxiliary marker position with the image auxiliary marker position based on the image auxiliary marker position and the calculated auxiliary marker position related to the first embodiment.

[0027] Figure 15 It is a schematic diagram showing an example of the case where the auxiliary marker related to the second embodiment is a quadrilateral.

[0028] Figure 16A It is a schematic diagram showing an example of assigning auxiliary marker positions to vertices for a polygonal auxiliary marker related to the second embodiment.

[0029] Figure 16B It is a schematic diagram showing an example of assigning auxiliary marker positions to vertices for a polygonal auxiliary marker related to the second embodiment.

[0030] Figure 17 It is a flowchart explaining the process of obtaining the image auxiliary marker position in the case where the auxiliary marker related to the second embodiment is a polygon.

[0031] Figure 18A It is a schematic diagram showing an example of the case where the size of the auxiliary marker related to the second embodiment is smaller than a given value.

[0032] Figure 18BIt is a schematic diagram showing an example where the boundary portion of the auxiliary mark related to the present embodiment is unclear.

[0033] Figure 19 It is a diagram showing an example of a table of known world positions of auxiliary marks where the auxiliary mark related to the second embodiment is quadrilateral.

[0034] Figure 20A It is a conceptual diagram showing an example of identifying the auxiliary mark in the captured image corresponding to the known world position of the auxiliary mark and using the identified known world position of the auxiliary mark as a mark.

[0035] Figure 20B It is a conceptual diagram showing an example of identifying the auxiliary mark in the captured image related to the present embodiment corresponding to the known world position of the auxiliary mark and using the identified known world position of the auxiliary mark as a mark.

[0036] Figure 20C It is a conceptual diagram showing an example of identifying the auxiliary mark in the captured image related to the present embodiment corresponding to the known world position of the auxiliary mark and using the identified known world position of the auxiliary mark as a mark.

[0037] Figure 20D It is a conceptual diagram showing an example of identifying the auxiliary mark in the captured image related to the present embodiment corresponding to the known world position of the auxiliary mark and using the identified known world position of the auxiliary mark as a mark.

[0038] Figure 20E It is a conceptual diagram showing an example of identifying the auxiliary mark in the captured image related to the present embodiment corresponding to the known world position of the auxiliary mark and using the identified known world position of the auxiliary mark as a mark. Detailed Embodiment

[0039] The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0040] <Positioning System 1>

[0041] First, with reference to Figure 1 the outline of the positioning system 1 will be described. Figure 1 It is a schematic diagram showing the overall structure of the positioning system 1 including the information processing device 10 related to the first embodiment of the present invention. Figure 2 It is a schematic diagram of a captured image 7 of the ceiling 4 including the mark 5 and the auxiliary mark 6 obtained by the information processing device 10 related to the first embodiment.

[0042] As Figure 1As shown in the figure, the positioning system 1 includes: a camera unit 2 mounted on the side of the moving body 3 rather than on the side of the building 9 including the ceiling; a marker 5 provided in the building 9 where the moving body 3 moves; and an information processing device 10 that locates the position of the moving body 3 based on the captured image 7 captured by the camera unit 2.

[0043] The camera unit 2 obtains the captured image 7 in the state of being mounted on the moving body 3. The camera unit 2 of the present embodiment continuously captures the upper ceiling 4 at a given frame rate, and obtains a plurality of temporally consecutive captured images 7 of the ceiling 4.

[0044] In addition, the camera unit 2 is connected to a communication device 28 for communicating with the information processing device 10. The captured image 7 captured by the camera unit 2 or various information based on the captured image 7 is transmitted to the information processing device 10 via the communication device 28. In addition, in this example, the moving body 3 equipped with the camera unit 2 is a work vehicle such as a forklift.

[0045] The marker 5 is an object including a device or identification as follows: by the marker 5 itself sending data, or the camera unit 2 capturing the marker 5 for image processing, three-dimensional position information in the world coordinate system 70 where the marker 5 is set can be obtained from the database 19.

[0046] The marker 5 of the present embodiment is, for example, a light-emitting device that can control the light-emitting form such as the color and timing of visible light. The marker 5 changes the color, blinks, etc. according to a given pattern, for example, to perform optical communication of identification information. In addition, the light emitted by the marker 5 may not be visible light, but near-infrared light. That is, as long as it is a light waveband that can be captured by a camera.

[0047] In the present embodiment, a plurality of markers 5 are provided to determine the position and orientation of the camera unit 2. In Figure 1 the example shown, two markers 5 are provided on the ceiling 4. In addition, the marker 5 does not necessarily have to be provided on the ceiling 4, as long as it is provided in the building 9 where the moving body 3 equipped with the camera unit 2 can move and be captured.

[0048] In the auxiliary marker 6, the auxiliary marker 6 itself does not send data. In addition, when the camera unit 2 captures the auxiliary marker 6, three-dimensional position information in the world coordinate system 70 where the auxiliary marker 6 is set cannot be obtained from the database 19 only by image processing. The auxiliary marker 6 is, for example, a light source or lighting provided in multiple places on the ceiling 4. In this specification, the auxiliary marker 6 refers to an area that can be determined from the captured image 7 through image processing. The determination of the area can utilize, for example, brightness or chromaticity.

[0049] A window 8 is provided in a building 9. When the window 8 is captured in a captured image 7, control is executed according to a time period. Although there are cases where the condition of brightness is satisfied, the area of the window 8 is not processed as an auxiliary marker 6 through image processing. In addition, the auxiliary marker 6 of the present embodiment is not limited to a self-luminous body. It is also possible to determine, through image processing from the captured image 7, a white reflecting object that reflects light, a green-painted area existing in a brown plane, etc. as the auxiliary marker 6.

[0050] The information processing device 10 locates the position of the imaging unit 2 based on the known positions of the marker 5 and the auxiliary marker 6, and the position 51 of the marker 5 (hereinafter referred to as the in-image marker position 51) and the position 61 of the auxiliary marker 6 (hereinafter referred to as the in-image auxiliary marker position 61) included in the captured image 7. The known positions of the marker 5 and the auxiliary marker 6 can be determined in advance from a design drawing, for example, or can be determined by other positioning devices.

[0051] In this specification, the positions of the marker 5 or the auxiliary marker 6 are marked using a three-dimensional world coordinate system 70.

[0052] The information processing device 10 of the present embodiment is connected to a communication device 18 for communicating with the imaging unit 2. Communication is established between the communication device 18 on the information processing device 10 side and the communication device 28 on the imaging unit 2 side, and the information processing device 10 acquires the captured image 7 from the imaging unit 2.

[0053] <Hardware Structure>

[0054] Figure 3 It is a block diagram showing the hardware structure of the information processing device 10 and the imaging unit 2 according to the first embodiment.

[0055] In Figure 3 this example, the information processing device 10 includes a processor 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output unit 14, a communication unit 15, and a storage unit 16.

[0056] The processor 11 performs various operations and processes. The processor 11 is, for example, a CPU (central processing unit), an MPU (micro processing unit), an SoC (system on a chip), a DSP (digital signal processor), a GPU (graphics processing unit), an ASIC (application specific integrated circuit), a PLD (programmable logic device), or an FPGA (field-programmable gate array), etc. Alternatively, the processor 11 is a processor obtained by combining multiple ones of these. In addition, the processor 11 may also be a processor obtained by combining a hardware accelerator or the like among these.

[0057] The processor 11, the ROM 12, and the RAM 13 are interconnected via a bus. The processor 11 executes various processes according to the program recorded in the ROM 12 or the program loaded into the RAM 13. A part or all of the program can be loaded into the circuit of the processor 11.

[0058] The input / output unit 14 is composed of a keyboard, various buttons, a microphone, etc., and inputs various information corresponding to the user's instruction operations. In addition, the input / output unit 14 is composed of a display, a speaker, etc., and outputs images and sounds. It includes an information terminal with a touch panel. The communication unit 15 is a network interface for communicating with the imaging unit 2, etc. via the communication device 28. The storage unit 16 is an area for storing various information such as the captured image 7 and known information.

[0059] Next, an example of the hardware structure of the imaging unit 2 will be described. The imaging unit 2 includes an optical lens unit 21 and an image sensor 22.

[0060] The optical lens unit 21 is composed of lenses that converge light, such as a focusing lens, a zoom lens, etc., for imaging a subject. The focusing lens is a lens that forms an image of the subject on the light-receiving surface of the image sensor 22. The zoom lens is a lens that allows the focal length to freely vary within a certain range. In addition, the optical lens unit 21 is also provided with peripheral circuits for adjusting setting parameters such as focus, exposure, and white balance as needed.

[0061] The image sensor 22 is composed of a photoelectric conversion element, an AFE (Analog Front End), etc. The photoelectric conversion element is composed of a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element, etc. The object image is incident on the photoelectric conversion element from the optical lens unit 21. Therefore, the photoelectric conversion element performs photoelectric conversion (imaging) on the object image, accumulates the image signal for a certain period of time, and sequentially supplies the accumulated image signal to the AFE as an analog signal. The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. Through this various signal processing, a digital signal is generated, and the captured image 7 is output as the output signal of the imaging unit 2.

[0062] <Functional Structure of Information Processing Device>

[0063] Figure 4 It is a functional block diagram showing the functional structure of the information processing device 10 according to the first embodiment. Figure 4 As described above, the functional structure is shown, and the relationship of the processing flow is shown.

[0064] The information processing device 10 performs various controls of the information processing device 10 by a processing unit 100 implemented by the processor 11 executing arithmetic processing based on a given program.

[0065] The processing unit 100 of the present embodiment has, by functional distinction, an image processing unit 101, a marker recognition unit 102, a known information acquisition unit 103, a primary positioning unit 104, a calculation unit 105, an auxiliary marker recognition unit 106, a determination positioning unit 107, and an output unit 108.

[0066] The image processing unit 101 acquires the captured image 7 and performs preprocessing such as distortion processing. The marker recognition unit 102 can acquire the marker position 51 in the image by performing the process of recognizing the marker 5 from the captured image 7.

[0067] The known information acquisition unit 103 performs a process of acquiring preset known information such as the internal parameters 25 of the imaging unit, the imaging unit setting parameters 26, the known marker position 52 in the world coordinate system 70 (hereinafter referred to as the world known marker position 52), and the known auxiliary marker position 62 in the world coordinate system 70 (hereinafter referred to as the world known auxiliary marker position 62) from the database 19. In addition, the database 19 can be constructed in the storage unit 16 of the information processing device 10 or in a server external to the information processing device 10.

[0068] The primary positioning unit 104 performs the following process: Based on the in-image marker position 51 obtained from the captured image 7 and the world-known marker position 52 obtained from the database 19, the position of the imaging unit 2 is positioned once, and the position 23 of the imaging unit 2 before determination in the world coordinate system 70 (hereinafter referred to as the primary world imaging position 23) is obtained. The primary world imaging position 23 includes the position of the imaging unit 2 and the direction in which the imaging unit 2 is oriented.

[0069] The calculation unit 105 performs a projection calculation process, and performs a projection process on the position on the captured image 7 obtained when the imaging unit 2 captures an image according to the world-known auxiliary marker position 62, thereby calculating.

[0070] The auxiliary marker recognition unit 106 performs the following process: The position 61 of the auxiliary marker 6 obtained from the captured image 7 through image processing (hereinafter referred to as the in-image auxiliary marker position 61) is associated with the world-known auxiliary marker position 62 for recognition. In the present embodiment, this association process is performed, and a PnP problem including the in-image marker position 51, the world-known marker position 52, the in-image auxiliary marker position 61, and the world-known auxiliary marker position 62 is set to calculate the position of the moving body 3. Thus, even with a small number of markers 5 such as 2, the position of the moving body 3 can be calculated with high accuracy by utilizing the existing auxiliary marker 6 such as illumination.

[0071] The determination positioning unit 107 performs the following process: Based on the primary position of the imaging unit 2, the world-known marker position 52, the world-known auxiliary marker position 62, etc., the final position of the imaging unit 2 in the world coordinate system 70 (hereinafter referred to as the determined world imaging unit position 24) is determined and obtained. The determined world imaging unit position 24 includes the position of the imaging unit 2 and the direction in which the imaging unit 2 is oriented. The output unit 108 performs a process of outputting the positioning result.

[0072] In addition, in this example, the information processing device 10 is located at a place far from the imaging unit 2, but is not limited to this structure. For example, the information processing device 10 may also be configured to be installed in the moving body 3 or the imaging unit 2.

[0073] <Known information>

[0074] Next, with reference to Figure 5 and Figure 6 an example of the known information pre-registered in the database 19 will be described.

[0075] Figure 5 is a schematic diagram showing the planar arrangement of the marker 5 and the auxiliary marker 6 according to the first embodiment. In Figure 5In [the figure], the positional relationship between the sign 5 and the auxiliary sign 6 provided on the ceiling 4 is shown in a top-down view. In this example, two signs 5 are given 5a and 5b as identification numbers and are individually handled. In the present embodiment, the same applies to the auxiliary sign 6 with 16 lights, which is given 6a to 6p as identification numbers and is individually handled.

[0076] Figure 6 is a diagram showing an example of the positions of the sign 5 and the auxiliary sign 6 in the world coordinate system 70 according to the first embodiment. As described above, the positions of the sign 5 and the auxiliary sign 6 are determined in advance by design drawings, positioning, etc. In Figure 6 it shows the world coordinates as three-dimensional positions for the identification numbers 5a and 5b of the sign 5 and the identification numbers 6a to 6p of the auxiliary sign 6 respectively. Here, for the convenience of explanation, one of the corners of the ceiling 4 is set as the origin, the x-axis and y-axis are set in a top-down view, the vertical direction is set as the z-axis, and the ground is set as the origin of the Z-axis. The following processing is performed: In a state where the three-dimensional positions of the sign 5 and the auxiliary sign 6 are pre-registered in the database 19 as known information, the position of the imaging unit 2 is determined based on the captured image 7 obtained by the imaging unit 2 capturing the ceiling 4.

[0077] <Positioning process>

[0078] Next, the positioning process using the captured image 7 will be described. Figure 7 is a flowchart showing the flow of the positioning process executed by the information processing device 10 according to the first embodiment.

[0079] When the positioning process starts, the processing unit 100 acquires the captured image 7 captured by the imaging unit 2 and performs distortion correction processing (step S101). The distortion correction processing corrects, for example, the barrel distortion of the image captured by the wide-angle lens.

[0080] After the processing of step S101, the processing unit 100 performs the determination process of the position of the auxiliary sign 6 (step S102).

[0081] Refer to Figure 8 and Figure 9 to describe an example of the determination process of the auxiliary sign position. Figure 8 is a schematic diagram showing the process of extracting a high-brightness area from the captured image 7. As Figure 8As shown, in order to determine the auxiliary marker 6 from the captured image 7, the image processing unit 101 of the processing unit 100 performs binarization on the captured image 7 using a given threshold as preprocessing, setting only the regions with high brightness to white and the rest to black. As a result, regions with high brightness such as those in the saturation region are extracted. However, in this state, depending on the time period with sunlight such as during the day, the position of the window 8 is also extracted as a region with high brightness.

[0082] Figure 9 is a schematic diagram illustrating the process of determining the auxiliary marker 6 used in positioning from the Figure 8 extracted region. As Figure 9 shown, in order to exclude the regions that contribute to misalignment due to the window 8, the image processing unit 101 performs filtering processing based on the actual shape, area, etc. of the extracted region, and determines only the regions where the marker 5 and the auxiliary marker 6 are highly likely. Then, for each of the multiple auxiliary markers 6, the region center position is obtained through image processing and stored as the in-image auxiliary marker position 61. Then, the auxiliary marker recognition unit 106 creates a list of the in-image auxiliary marker positions 61 of the auxiliary marker 6. Here, the center position in the world coordinate system 70 is the centroid of the region recognized as the auxiliary marker 6, the average of the xyz coordinates, the intermediate value between the maximum coordinate and the minimum coordinate, etc. Not limited to these, the center position can represent the position of the auxiliary marker 6 as one position.

[0083] Two-dimensional coordinates are set on the captured image 7 to determine the in-image auxiliary marker position 61 on the captured image 7. In this specification, the two-dimensional coordinate system on the captured image 7 is called the image coordinate system 71.

[0084] Figure 11 shows an example of the list of the in-image auxiliary marker positions 61. Recognition numbers are assigned once to the auxiliary markers 6 that give the in-image auxiliary marker positions 61, such as L1 and L2. At this stage, only the positions of the multiple auxiliary markers 6 are determined from the captured image 7, and it is not possible to determine which of the auxiliary markers 6a to 6p shown in Figure 6 corresponds, and unique identification is not possible. Therefore, as Figure 11 shown, at this time point, the association between the auxiliary marker 6 and the world-known auxiliary marker position 62 does not hold.

[0085] Next, the marker recognition unit 102 performs marker recognition processing (step S103). Figure 10 is a schematic diagram illustrating the process of determining the marker 5 from the captured image 7. In the marker recognition processing, the marker recognition unit 102 determines the position of the marker 5 from the captured image 7 through image processing.

[0086] Figure 11 An example of a list showing the in-image marker positions 51 and the world-known marker positions 52. Identification numbers are assigned once to the markers 5 such as M1 and M2 at the in-image marker positions 51 given in the image. Since the markers 5 are recognized, for example, the in-image marker positions 51 of M1 and M2 are associated with Figure 5 the markers 5a and 5b shown.

[0087] An example of marker recognition processing is described. The marker recognition unit 102 obtains a light emission pattern based on consecutive captured images 7 and compares it with a preset light emission pattern to determine the position of the marker 5 in the captured image 7.

[0088] Next, the primary positioning unit 104 performs a positioning possibility determination process (step S104). As the positioning possibility determination process, for example, the primary positioning unit 104 determines whether there are two or more markers 5 in the captured image 7 and whether positioning can be performed.

[0089] When positioning can be performed using the marker 5 (step S104 "Yes"), the primary positioning unit 104 performs a primary positioning process (step S105).

[0090] An example of the primary positioning process is described. As described above, since the markers 5a and 5b can be individually recognized from the captured image 7, the primary positioning unit 104 obtains the world-known marker positions 52 of the markers 5a and 5b based on Figure 11 the corresponding relationship.

[0091] As the primary positioning process, for example, the primary positioning unit 104 performs primary positioning on the imaging unit 2 through PnP positioning processing based on the in-image marker position 51 and the world-known marker position 5.

[0092] As has been known in the past, when the positions of a certain point group and the positions in the captured image 7 are given, the position and attitude of the imaging unit 2 are obtained through PnP positioning processing. When positioning is performed based on six points, the position and imaging direction (in-plane orientation, elevation angle) of the imaging unit 2 are obtained. Even when the number of points is less than six, for example, when the imaging unit 2 is located on a plane parallel to the ceiling 4 provided with the marker 5 and the auxiliary marker 6, the position and elevation angle of the imaging unit 2 can be obtained with low accuracy by positioning based on only two points.

[0093] Since the moving body 3 moves on the ground parallel to the ceiling 4, the primary world imaging position 23, which is the primary position and elevation angle of the imaging unit 2, is obtained by the primary positioning unit 104 through the P2P method. Using Figure 11The image marker positions 51 represented by the identification numbers M1 and M2 and the world-known marker positions 52 represented by the identification numbers 5a and 5b are located by the P2P method, and the primary positioning unit 104 locates the primary world imaging position 23, which is the position of the imaging unit 2 as the primary.

[0094] When positioning cannot be performed using the marker 5 (step S104 "No"), the primary positioning unit 104 performs a position estimation process (step S106). As the position estimation process, for example, the primary positioning unit 104 estimates the position of the imaging unit 2 based on the previous determined world imaging unit position 24 and the movement vector history, and sets it as the primary positioning result.

[0095] Next, the calculation unit 105 performs an auxiliary marker recognition process (step S107). As the auxiliary marker recognition process, for example, the calculation unit 105 determines whether an auxiliary marker 6 exists in the captured image 7.

[0096] When the auxiliary marker 6 does not exist in the captured image 7 (step S107 "No"), the primary positioning unit 104 performs a primary positioning result application process (step S111). In the primary positioning result application process, for example, the primary positioning unit 104 applies the primary positioning result as the determined positioning result.

[0097] When the auxiliary marker 6 exists in the captured image 7 (step S107 "Yes"), the calculation unit 105 performs an auxiliary marker 6 position calculation process (hereinafter referred to as the auxiliary marker position calculation process) (step S108). As the auxiliary marker position calculation process, when imaging the world-known auxiliary marker position 62 from the primary world imaging position 23 obtained based on the primary positioning result, the calculation unit 105 calculates the position where the world-known auxiliary marker position 62 is imaged in the captured image 7. In other words, it calculates the position 63 of the auxiliary marker 6 representing the projection position (hereinafter referred to as the calculated auxiliary marker position 63).

[0098] This calculation is a well-known process of calculating how an arbitrary three-dimensional point specified by the position / attitude of the imaging unit 2 is depicted on the image. In addition, the calculation itself is a simple matrix calculation. For this reason, it becomes a projection in an infinite field of view, and the back side of the world imaging unit position will also be projected onto a plane. And if the PnP positioning process is executed as it is, the calculation will not hold or a large error will occur. For this reason, when imaging is performed from the primary world imaging position 23 obtained by primary positioning, after only selecting the auxiliary marker 6 located at the world-known auxiliary marker position 62 that roughly enters the imaging angle of view, the calculation is performed.

[0099] Figure 12The known world auxiliary marker position 62 and its drawing position on the captured image 7 represent the calculated auxiliary marker position 63. The identification number of the calculated auxiliary marker position 63 calculated for the identification number 6a of the known world auxiliary marker position 62 is expressed as 6a-C.

[0100] The determination and positioning unit 107 performs an association process (step S109). As an association process, the determination and positioning unit 107 establishes a correspondence between the auxiliary marker position 61 in the image and the calculated auxiliary marker position 63 based on the positional relationship on the same camera image, and associates the auxiliary marker 6 with the world known auxiliary marker position 62 which is the calculation source of the calculated auxiliary marker position 63.

[0101] Figure 13A , Figure 13B , Figure 13C = represents the calculated relationship between the auxiliary mark position 63 and the auxiliary mark position 61 in the image. Figure 13A As shown in FIG. 5 , the calculated auxiliary marker positions 63 of the world known auxiliary marker positions 62 on the captured image 7 including the world known marker positions 52 are plotted on the image coordinate system 71. Figure 13B As shown, the in-image auxiliary marker position 61 of the auxiliary marker 6 including the in-image marker position 51 is drawn on the image coordinate system 71 . Figure 13C This is an enlarged view of the upper right of the image coordinate system 71 when these are overlapped. The calculated auxiliary mark position 63 is similar to the auxiliary mark position 61 in the image, but there is a slight deviation in the amount of positioning error. The points that are closest to or around the points and are in a positional relationship that does not destroy the mutual positional relationship are associated with each other. Figure 13C In the figure, the calculated auxiliary mark position 63 of the identification number 6p-C and the in-image auxiliary mark position 61 of the identification number L13 are drawn adjacent to each other, and the calculated auxiliary mark position 63 of the identification number 6o-C and the in-image auxiliary mark position 61 of the identification number L14 are drawn adjacent to each other. These are respectively regarded as corresponding coordinate points, and are associated or associated.

[0102] Figure 14 The calculated auxiliary marker position 63 and the in-image auxiliary marker position 61 are shown in a similar manner. The world-known auxiliary marker position 62, which is the basis for calculating the calculated auxiliary marker position 63, is also shown. The in-image auxiliary marker position 61 is associated with the world-known auxiliary marker position 62. The auxiliary marker 6, which was not recognized at this point in time although it was captured by the imaging unit 2, is recognized as the auxiliary marker 6 with a known position. The recognized auxiliary marker 6 is the same as the marker 5 because it is recognized and has a known position.

[0103] The primary positioning unit 104 obtains the primary world imaging position 23 of the imaging unit 2 based on two markers 5. In contrast, by Figure 14 associating the auxiliary marker 6 with the known world auxiliary marker position 62 as shown, the number of markers equivalent to the two markers 5 becomes 18 recognized auxiliary markers 6 including 6a to 6p.

[0104] Furthermore, the determination positioning unit 107 performs PnP positioning processing (step S110). Regarding the PnP positioning processing, the determination positioning unit 107 performs PnP positioning processing based on the known world marker position 52, the known world auxiliary marker position 62, the in-image marker position 51, and the in-image auxiliary marker position 61.

[0105] Although it is a P2P positioning calculation in the primary positioning, since a P18P problem of positioning based on the imaging result of 18 three-dimensional known points can be created, it becomes highly accurate.

[0106] Then, the output unit 108 outputs the result of the PnP positioning processing as the determination positioning result.

[0107] The processing unit 100 performs positioning end determination processing (step S112).

[0108] As the positioning end determination processing, when the positioning ends (step S112 "Yes"), the processing unit 100 ends the positioning processing. When the positioning has not ended (step S112 "No"), the processing unit 100 returns the positioning processing to A in the flowchart.

[0109] In the first embodiment, P2P positioning processing is adopted in the primary positioning, and two markers 5 are used as the minimum number for primary positioning. The minimum number of markers 5 can be in the form of P6P with six markers as the minimum. This is because, through P6P, the position and imaging direction of the imaging unit 2 can be derived. In addition, this embodiment is not limited to the above two points or six points and is effective for all PnP positioning processing.

[0110] <Effect of the Embodiment>

[0111] In the first embodiment, as a primary positioning method, the imaging unit 2 identifies and determines the marker 5. Therefore, it is not necessary to input the initial state such as the position and imaging direction of the imaging unit 2 to the information processing device 10, and the determination positioning of the imaging unit 2 can be performed at any point where the marker 5 can be imaged. The recognition function of the marker 5 can receive even an imaging of a point close to a very small point on the image. Thus, by adding the minimum number of markers 5 required for the primary positioning to be practical, the positioning system 1 is established. As a result, the positioning system 1 can be constructed at low cost and in a short period.

[0112] The information processing device 10 includes a processing unit 100 that performs the following processing: obtaining, through image processing, an in-image marker position 51 indicating the position of a marker 5 in an image coordinate system 71 and an in-image auxiliary marker position 61 indicating the position of an auxiliary marker 6 from a captured image 7 that includes the marker 5 and the auxiliary marker 6 provided in a space and captured by a imaging unit 2 provided on the side of a moving body 3; obtaining a primary world imaging position 23 indicating the three-dimensional position of the imaging unit 2 in a world coordinate system 70 based on the in-image marker position 51 and a world-known marker position 52 indicating the known three-dimensional position of the marker 5 in the world coordinate system 70; establishing a correspondence between the in-image auxiliary marker position 61 and the world-known auxiliary marker position 62 based on the primary world imaging position 23 and a world-known auxiliary marker position 62 indicating the known three-dimensional position of the auxiliary marker 6 in the world coordinate system 70; and determining the position of the imaging unit 2 based on the in-image auxiliary marker position 61 and the world-known auxiliary marker position 62.

[0113] Provided is an information processing device 10 that suppresses the number of markers 5 and derives a positioning result with high position accuracy.

[0114] There will be no difficulty in high-precision tracing caused by radio wave utilization. In addition, unlike Visual-SLAM and LiDAR-SLAM, there will be no problem of difficult to ensure definite positioning, and no closed-loop problem will occur. A practical positioning system can be constructed.

[0115] The processing unit 100 included in the information processing device 10 sets a visible light recognition light source based on light emission form as the marker 5, performs recognition from the captured image 7, and obtains the in-image marker position 51.

[0116] Each marker 5 can be reliably recognized according to the individual light emission form of the captured marker 5.

[0117] The processing unit 100 included in the information processing device 10 recognizes two or more markers 5 from the captured image 7, obtains a plurality of in-image marker positions 51, and obtains the primary world imaging position 23 based on the plurality of in-image marker positions 51 and the world-known marker positions 52 corresponding to the markers 5 respectively.

[0118] Even with a small number of markers 5, it is sufficient to obtain the primary world imaging position 23. It is sufficient to prepare a small number of markers 5, and the cost, startup man-hours, etc. can be suppressed to a low level.

[0119] The processing unit 100 included in the information processing device 10 determines the imaging position and imaging direction of the captured image 7 based on the first-world imaging position 23, calculates the calculated auxiliary marker position 63 by performing projection calculation processing for projecting the world-known auxiliary marker position 62 onto the captured image 7, and establishes a correspondence between the in-image auxiliary marker position 61 and the world-known auxiliary marker position 62 by establishing a correspondence between the in-image auxiliary marker position 61 and the calculated auxiliary marker position 63.

[0120] The position of the auxiliary marker 6 that is the source of the in-image auxiliary marker position 61 is recognized as the world-known auxiliary marker position 62. The auxiliary marker 6 can be used in the PnP positioning process in the same way as the marker 5.

[0121] The processing unit 100 included in the information processing device 10 performs the correspondence establishment process by establishing a correspondence between the result of the projection calculation processing, the world-known auxiliary marker position 62 that is the calculation source of the calculated auxiliary marker position 63 located at or around the in-image auxiliary marker position 61, and the in-image auxiliary marker position 61.

[0122] The correspondence establishment is surely performed only by a simple method of obtaining the in-image auxiliary marker positions 61 located around in the image coordinate system 71.

[0123] The program causes a computer to execute: a marker 5 and auxiliary marker position acquisition function that acquires, through image processing, the in-image marker position 51 indicating the position of the marker 5 in the image coordinate system 71 and the in-image auxiliary marker position 61 indicating the position of the auxiliary marker 6 from the captured image 7 captured by the imaging unit 2 provided on the side of the moving body 3 and including the marker 5 and the auxiliary marker 6 provided in the space; a primary positioning function that acquires the first-world imaging position 23 indicating the three-dimensional position of the imaging unit 2 in the world coordinate system 70 based on the in-image marker position 51 and the world-known marker position 52 indicating the known three-dimensional position of the marker 5 in the world coordinate system 70; and a determination positioning function that establishes a correspondence between the in-image auxiliary marker position 61 and the world-known auxiliary marker position 62 based on the first-world imaging position 23 and the world-known auxiliary marker position 62 indicating the known three-dimensional position of the auxiliary marker 6 in the world coordinate system 70, and determines the position of the imaging unit 2 based on the in-image auxiliary marker position 61 and the world-known auxiliary marker position 62.

[0124] A positioning program is provided that suppresses the number of markers 5 and derives a positioning result with high position accuracy.

[0125] The position positioning method executed by a computer causes the computer to execute: a marker 5 and an auxiliary marker position acquisition step, obtaining an in-image marker position 51 indicating the position of the marker 5 in the image coordinate system 71 and an in-image auxiliary marker position 61 indicating the position of the auxiliary marker 6 from a captured image 7 including the marker 5 and the auxiliary marker 6 provided in the space captured by the imaging unit 2 provided on one side of the moving body 3 through image processing; a primary positioning step, obtaining a primary world imaging position 23 indicating the three-dimensional position of the imaging unit 2 in the world coordinate system 70 based on the in-image marker position 51 and a world-known marker position 52 indicating the known three-dimensional position of the marker 5 in the world coordinate system 70; and a determination positioning step, establishing a correspondence between the in-image auxiliary marker position 61 and the world-known auxiliary marker position 62 based on the primary world imaging position 23 and a world-known auxiliary marker position 62 indicating the known three-dimensional position of the auxiliary marker 6 in the world coordinate system 70, and determining the position of the imaging unit 2 based on the in-image auxiliary marker position 61 and the world-known auxiliary marker position 62.

[0126] A positioning method is provided that suppresses the number of markers 5 and derives a positioning result with high position accuracy.

[0127] Next, a form different from the first embodiment will be described. In the following description, for structures common or the same as those described in the first embodiment, the same reference numerals are given, and the description may be omitted sometimes.

[0128] (Second Embodiment)

[0129] In the first embodiment, an example of setting the position of the circular auxiliary marker 6 based on the center of the region has been described. In the second embodiment described below, the method of determining the position of the auxiliary marker 6 is different.

[0130] Figure 15 It is a schematic diagram showing an example in which the auxiliary marker 6 according to the second embodiment is quadrilateral. Identification numbers 100 to 130 are assigned to each auxiliary marker 6. In the figure, arrows are used to indicate the identification numbers. The same applies hereinafter. Figure 16A , Figure 16B It is a schematic diagram showing an example of assigning auxiliary marker positions to the vertices of the quadrilateral auxiliary marker 6 according to the second embodiment. As Figure 15 shown, when viewed from above, a rectangular illumination is also used as the auxiliary marker 6. In the second embodiment, as Figure 16A shown, a process of adding the reference numerals 111 to 114 of the vertex portions to the world-known auxiliary marker position 62 is executed. In Figure 16AIn this case, for the reference signs, an arrow is used. The same applies hereinafter. Thus, in the case of the auxiliary sign 6 in the shape of a quadrilateral, the center position of the auxiliary sign 6 and the world-known auxiliary sign positions 62 at the four corners, a total of five positions, can be obtained in the three-dimensional position in the world coordinate system 70. In the following description, the position of the vertex portion is referred to as the world-known outer shape position 65. In addition, the position in the image coordinate system 71 on the captured image 7 of the auxiliary sign 6 is referred to as the image outer shape position 64.

[0131] For Figure 16B a polygon-shaped auxiliary sign 6 such as a trapezoid that is not rectangular as shown, world-known auxiliary sign positions 62 can also be assigned to its vertex portions. As Figure 16B shown, for example, a known sign position 62 in the world coordinate system 70 (hereinafter referred to as the world-known auxiliary sign position 62) is assigned to the center of the auxiliary sign 6, and world-known outer shape positions 65 are assigned to the positions of the vertex portions, and identification numbers from 200 to 204 are respectively given. In Figure 16A , Figure 16B the examples, they are all one form of a quadrilateral, but are not limited to this shape. For example, in a triangle, a pentagon or more, each vertex can also be added as the world-known outer shape position 65, the image outer shape positions 64 of the vertices can be respectively obtained, and secondary positioning can be performed.

[0132] <World-known auxiliary sign position extraction process>

[0133] Figure 17 is a flowchart showing the process flow of the case where the auxiliary sign 6 according to the second embodiment is a polygon. Through the process executed by the processing unit 100, the world-known auxiliary sign position 62 and the world-known outer shape position 65 are extracted.

[0134] If the creation of the table of the auxiliary sign positions 61 in the image starts, first, the image processing unit 101 performs an auxiliary sign recognition process (step S201). In the auxiliary sign recognition process, for example, the image processing unit 101 extracts the regions with saturated brightness in the captured image 7 and assigns a primary identification number to the extracted regions.

[0135] Furthermore, the image processing unit 101 performs an auxiliary sign selection process (step S202). As the auxiliary sign selection process, for example, the image processing unit 101 performs shape-based picking on the obtained regions. The image processing unit 101 extracts, for example, ellipses and quadrilaterals and excludes the regions overlapping the image edge.

[0136] Next, the image processing unit 101 performs a table creation process (step S203). The table creation process is, for example, a process of registering the candidate regions extracted by the image processing unit 101 in a primary table.

[0137] The auxiliary mark recognition unit 106 performs a selection process (step S204). The extraction process is, for example, one of the areas selected by the auxiliary mark recognition unit 106.

[0138] The auxiliary mark recognition unit 106 performs a size determination process. As the size determination process, for example, the auxiliary mark recognition unit 106 determines whether the size in the captured image 7 of the auxiliary mark 6 is equal to or greater than a given size. The so-called given size is, for example, a size of 7×7 pixels in the captured image 7.

[0139] Reference Figure 18A 、 Figure 18B To illustrate the size determination process. In a case where imaging is taken from a distance and appears small, if the points on the image at the four corners are used, errors may sometimes occur. Therefore, in the size determination process, in order to avoid determination with a high possibility of generating errors, it is determined whether the area that is a candidate for the auxiliary mark 6 is a given size.

[0140] When the extracted area is equal to or greater than the specified size (step S205 “Yes”), the auxiliary mark recognition unit 106 performs a vertex registration process (step S206). As the vertex registration process, for example, the auxiliary mark recognition unit 106 registers the 4 vertices of the auxiliary mark 6 having a quadrilateral shape as the image outer shape position 64. As Figure 18A shown, in a case where there is a candidate for obtaining the in-image auxiliary mark position 61 among the auxiliary marks 6, only the local area 612 located at the center of the area 611 may be used. In addition, there is also Figure 18B shown a case where it can be seen as a polygon but the boundary is blurred and the vertices are not distinct. In such a case, the local area 612 located at the center of the area 611 may also be used.

[0141] Next, the auxiliary mark recognition unit 106 performs a center registration process (step S207). As the center registration process, the auxiliary mark recognition unit 10 registers the center of the extracted area as the in-image auxiliary mark position 61 regardless of the size of the extracted area.

[0142] The processing unit 100 performs an end determination process (step S208).

[0143] In the case of not ending (step S208 “No”), the processing unit 100 returns the positioning process to step S204. In the case of ending (step S208 “Yes”), the processing unit 100 ends the creation of the table of the in-image auxiliary mark position 61.

[0144] Figure 19The position of the vertex of the auxiliary marker 6 in such a polygon is identified and registered, and the known external shape position 65 in the world is represented as an example of the known auxiliary marker position 62 in the world. More than 10 bits are used for the center position, and 1 bit is used for the known external shape position 65 in the world as the vertex.

[0145] In the present embodiment, the lighting or light source provided in the environment is set as the auxiliary marker 6, and this auxiliary marker 6 can be used as a supplement in the case of few markers 5. If the indoor lighting is generally imaged as the auxiliary marker 6, the brightness is reflected as a saturated area, and when taking out the initial auxiliary marker area candidate, it is only necessary to perform binarization according to the saturation value, and the image signal is very stable. In addition, lighting is generally installed in an open area of the environment, and the shape is mostly a simple shape such as a circle or a quadrilateral, which is suitable as the auxiliary marker 6 in the present embodiment. The auxiliary marker 6 is not limited to lighting, and as long as it is a marker that can be easily detected stably in a simple manner in the scene, it can also be other methods. For example, a marker that can be seen at a glance because it is a high-chroma object in a low-chroma environment, etc., can be considered as other methods because it can be binarized according to a specific high chroma threshold.

[0146] In this way, in the second embodiment, an auxiliary marker 6 having a polygonal shape is used as a form of the known auxiliary marker position 62 in the world, and the known external shape position 65 in the world is increased. A large number of known auxiliary marker positions 62 in the world can be provided by using one auxiliary marker or lighting.

[0147] The processing unit 100 included in the information processing apparatus 10 obtains the in-image auxiliary marker position 61 of the auxiliary marker 6 based on the position of the vertex portion of the polygonal shape of the auxiliary marker 6 in the captured image 7.

[0148] The known auxiliary marker position 62 in the world can be increased without increasing the number of auxiliary markers 6, and the accuracy of the PnP positioning process can be improved.

[0149] When the size of the auxiliary marker 6 in the captured image 7 is smaller than a given size, the processing unit 100 included in the information processing apparatus 10 obtains the center position of the auxiliary marker 6, and obtains the in-image auxiliary marker position 61 based on this center position.

[0150] Optimizing the known auxiliary marker position 62 in the world used in the PnP positioning process can obtain higher positioning accuracy.

[0151] (Modification example of the marker)

[0152] In the above positioning process, as one of the methods for primary positioning, a marker 5 having an identification function is used. Other methods can also be used for primary positioning. Examples are listed below.

[0153] In the case where the PnP positioning process for the positioning unit 107 could not be performed in the initial state but has just been completed, since the current position can be estimated by difference based on the past position and the last vector, it can be set as the first world camera position 23.

[0154] The imaging unit 2 can image the QR code, and the position of the imaging unit 2 can be initially set based on the code and the imaging angle of the imaging unit 2. In addition, in the case of recognizing the positioning of a QR code or the like, it is preferable to display the QR code over a relatively large area on the imaging screen.

[0155] As a specific point, for example, an infrared sensor is prepared, and the position of the imaging unit 2 can be initially set based on the detection of the passage of the specific point. When the imaging unit 2 is provided on the moving body 3, if it is ensured that the starting point of the moving body 3 is the specific point, the position of the imaging unit 2 can be initially set based on the position of the specific point. The information of this specific point can be stored in the database 19 as the Figure 1 shown imaging unit setting parameter 26. For example, if starting from a charging station, as long as the identification number of the charging station is known, the position of the imaging unit 2 can be initially set. In addition, the elevation angle, focal length, presence or absence of the autofocus function, etc. of the imaging unit 2 can be stored in the database 19 as the Figure 1 shown imaging unit internal parameter 25. In addition, these various methods can also be combined. However, the method of using the marker 5 with an identification function, especially the marker 5 that is identified by the color change signal of the light emitter, is suitable for performing the first positioning of a relatively large area with relatively high accuracy.

[0156] As described above, it is not limited to the marker 5 in the above-described embodiment that performs optical communication by changing the light emission form, and as shown in the modification example, other types of markers 5 can also be used.

[0157] Next, an example of the processing after recognizing the position 62 of the world-known auxiliary marker on the captured image 7 will be described.

[0158] Figure 20A 、 Figure 20B 、 Figure 20C 、 Figure 20D 、 Figure 20E shows a more specific positioning process using the ceiling 4 provided with the marker 5 and the auxiliary marker 6. The recognized auxiliary marker 6 is identified in the same way as the marker 5. In the following description, the first positioning unit 104 uses the recognized position 62 of the world-known auxiliary marker in the first positioning in the same way as the marker 5.

[0159] In addition, in Figure 20A 、 Figure 20B 、 Figure 20C 、 Figure 20D 、 Figure 20EIn this case, the marker 5 and the auxiliary marker 6 are neatly arranged for the convenience of explanation, and it is assumed that the imaging unit 2 is provided perpendicular to these arrangements. In the figure, the regions at the four corners represent the imaging range of the captured image 7. Figure 20A Indicates the initial state. The marker 5 is imaged to recognize the position 51 of the marker in the image. Then, the position 61 of the auxiliary marker in the image is obtained. In the sense of being recognized, the marker 5 is Figure 20A blackened in

[0160] Similar to that described in the first embodiment, the first world imaging position 23 of the imaging unit 2 is calculated based on the position 51 of the marker in the image and the known world position 52 of the marker. Then, the calculated auxiliary marker position 63 in the case of imaging the auxiliary marker 6 located at the known world auxiliary marker position 62 from the first world imaging position 23 is calculated. That is, the projection calculation process is performed. Then, by establishing correspondence between adjacent points, the auxiliary marker 6 in the position 61 of the auxiliary marker in the image is associated with the known world auxiliary marker position 62 of the calculation source. Thus, the auxiliary marker 6 captured in the captured image 7 is recognized as the recognized auxiliary marker 6 with a known position. Here, the auxiliary marker 6 becomes the same recognized index as the marker 5. As a schematic showing this situation, Figure 20A the auxiliary marker 6 marked with a white circle in Figure 20B is blackened in

[0161] In the case of continuous movement from the moving body 3, the captured image 7 captured by the imaging unit 2 changes as Figure 20C shown. Here, through image processing, the association between the position 61 of the auxiliary marker in the image in the captured image 7 and the known world auxiliary marker position 62 is maintained. In Figure 20C the position 51 of the marker in the image and the position 61 of the auxiliary marker in the image of the blackened marker 5 and auxiliary marker 6 indicating recognition are maintained. The Figure 20C four blackened points in the captured image 7 of this Figure 20A can be used in the primary positioning of the imaging unit 2 in the same way as the position 51 of the marker in the image in Figure 20C Then, through comparison with the calculated auxiliary marker position 63, the position 61 of the auxiliary marker in the image in the captured image 7 of Figure 20C is newly associated with the known world auxiliary marker position 62 and becomes the same recognized index as the marker 5. As a schematic showing this situation, the auxiliary marker 6 marked with a white circle in Figure 20D is blackened in

[0162] This process is repeated. Figure 20E Immediately following Figure 20CFurther shown is a captured image 7 when the moving body 3 moves. At this point in time, the in-image marker position 51 due to the marker 5 is not in the captured image 7. The primary positioning unit 104 performs primary positioning of the position of the imaging unit 2 based on the recognized auxiliary marker 6, and further, the determination positioning unit 107 determines the position of the imaging unit 2.

[0163] Thereby, the marker 5 is provided only near the action initial position. Thereafter, by continuously positioning and recognizing the auxiliary marker 6, it is possible to achieve the setting of the marker 5 such that only the initial position is provided with the marker 5 while covering a large area. That is, regarding the transfer of the process using the marker 5, even if the marker 5 is not included in the captured image 7, as long as the auxiliary marker 6 is included, the determined world imaging unit position 24 of the imaging unit 2 can be derived.

[0164] The above has described the positioning process using the marker 5. In addition, the present invention is not limited to the above-described embodiments and modified examples, and variations, improvements, etc. within the scope that can achieve the object of the present invention are included in the present invention.

[0165] In addition, in the above-described embodiment, the information processing device 10 applying the present invention has been described by taking a forklift as an example of the moving body 3, but it is not particularly limited thereto. For example, the present invention can generally be applied to electronic devices having an image processing function. Specifically, for example, it can be applied to a notebook personal computer, a portable navigation device, a portable telephone, a smartphone, a handheld game console, etc.

[0166] The above series of processes can be executed by hardware or by software. In other words, Figure 4 the functional structure is only an example and is not particularly limited. That is, it is sufficient that the information processing device 10 has a function capable of executing the above series of processes as a whole, and it is not particularly limited to Figure 4 the example of.

[0167] In addition, one functional block can be constituted by hardware alone, by software alone, or by a combination thereof. The functional structure in the present embodiment is realized by a processor that executes arithmetic processing. Processors that can be used in the present embodiment include, in addition to processors constituted by various processing devices alone such as a single processor, a multi-processor, and a multi-core processor, a solution obtained by combining these various processing devices and processing circuits such as an ASIC (application specific integrated circuit) and an FPGA (Field-Programmable Gate Array).

[0168] In the case where a series of processes are executed by software, the program constituting the software is installed from a network or a recording medium onto a computer or the like. The computer may be a computer incorporated into dedicated hardware. In addition, the computer may be a computer that can execute various functions by installing various programs, such as a general-purpose personal computer.

[0169] The recording medium containing such a program is distributed separately from the device main body not only to provide the program to the user, but also consists of a recording medium or the like provided to the user in a state of being pre-installed in the device main body. The removable medium is constituted by, for example, a magnetic disk (including a floppy disk), an optical disk, or an optical magnetic disk. The optical disk is constituted by, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), a Blu-ray (registered trademark) Disc, etc. The optical magnetic disk is constituted by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being pre-installed in the device main body is constituted by, for example, a ROM 12 recording the program, a hard disk (not shown), etc.

[0170] In addition, in this specification, the steps of the program recorded on the recording medium naturally include the processes performed in sequence in time series, but the processes are not necessarily performed in time series, and also include the processes performed in parallel or individually. In addition, in this specification, the term "system" means the entire device constituted by a plurality of devices, a plurality of means, etc.

[0171] The above has described several embodiments of the present invention, but these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and furthermore, various omissions, substitutions, etc. can be made without departing from the gist of the present invention. These embodiments and their modifications are included in the scope and gist of the invention described in this specification, etc., and are included in the scope of the invention described in the claims and its equivalents.

Claims

1. An information processing device, characterized in that, it is provided with one or more processing units, and the one or more processing units perform the following processing: Obtain the position of the marker in the image coordinate system from an image captured by an imaging unit and including the marker provided in a space, Based on the position of the marker in the image coordinate system and the position of the marker in the world coordinate system, obtain information at the time of positioning related to the imaging unit before determination in the world coordinate system, Based on the positioning information of the imaging unit before determination in the world coordinate system and the position of the auxiliary marker in the world coordinate system, determine the positioning information related to the imaging unit, The auxiliary marker is an auxiliary marker different from the marker and existing in the space.

2. The information processing device according to claim 1, characterized in that, The marker is a light-emitting body that performs optical communication by changing its light-emitting form and transmits its own data, The auxiliary marker is a light-emitting body different from the marker and does not transmit its own data.

3. The information processing device according to claim 2, characterized in that, The image includes the auxiliary marker in addition to the marker, The one or more processing units perform the following processing: Obtain the position of the marker in the image coordinate system and the position of the auxiliary marker in the image coordinate system, Based on the positioning information related to the imaging unit before determination and the position of the auxiliary marker in the world coordinate system, establish a correspondence between the position of the auxiliary marker in the image coordinate system and the position of the auxiliary marker in the world coordinate system, Based on the established corresponding position of the auxiliary marker in the image coordinate system and the position of the auxiliary marker in the world coordinate system, determine the positioning information related to the imaging unit.

4. The information processing device according to claim 3, characterized in that, The one or more processing units perform the following processing: Establish a correspondence between the position of the auxiliary marker in the image coordinate system and the position of the auxiliary marker in the world coordinate system without coordinate transformation, Based on the established corresponding position of the auxiliary marker in the image coordinate system and the position of the auxiliary marker in the world coordinate system, determine the positioning information related to the imaging unit.

5. The information processing device according to claim 4, characterized in that, The one or more processing units perform the following processing: Based on the positioning information before determination, determine the imaging position and imaging direction of the image, Based on the image auxiliary marker position representing the position of the auxiliary marker in the image coordinate system and the known position of the auxiliary marker in the world coordinate system, calculate a calculated auxiliary marker representing the position of the auxiliary marker in the image coordinate system calculated by projecting the position of the auxiliary marker in the world coordinate system onto the image coordinate system, Based on the comparison between the image auxiliary marker position and the position of the calculated auxiliary marker, establish a correspondence between the position of the auxiliary marker in the image coordinate system and the position of the auxiliary marker in the world coordinate system.

6. The information processing device according to claim 1, characterized in that, The above-mentioned one or more processing units perform the following processing: Identify the light source recognized based on the light emission form as the said marker from the said image, and obtain the marker position in the said image coordinate system.

7. The information processing apparatus according to claim 1, wherein, The above-mentioned one or more processing units perform the following processing: Identify two or more of the said markers from the said image, obtain a plurality of marker positions in the said image coordinate system, Based on the plurality of marker positions in the said image coordinate system and the marker positions in the said image coordinate system respectively corresponding to the two or more markers identified, obtain the positioning information related to the imaging unit before determination.

8. The information processing apparatus according to claim 1, wherein, The above-mentioned one or more processing units perform the following processing: When the size of the auxiliary marker in the said image is smaller than a given size, obtain the center position of the auxiliary marker, and based on this center position, obtain the image auxiliary marker position indicating the position of the auxiliary marker in the said image coordinate system.

9. A storage medium, which is a non-temporary computer-readable recording medium for recording a program, wherein, The said program causes a computer to perform the following processing: Obtain the position of the marker in the image coordinate system from a captured image captured by an imaging unit and including a marker provided in a space, Based on the marker position in the said image coordinate system and the marker position in the world coordinate system, obtain the temporary positioning information related to the imaging unit in the said world coordinate system, Based on the positioning information of the imaging unit before determination in the said world coordinate system and the auxiliary marker position in the said world coordinate system, determine the positioning information related to the imaging unit, The said auxiliary marker is an auxiliary marker different from the said marker and existing in the said space.

10. A positioning method, which is a positioning method executed by a computer, wherein, includes the following steps: Obtain the position of the marker in the image coordinate system from an image captured by an imaging unit and including a marker provided in a space; Based on the position of the marker in the said image coordinate system and the position of the marker in the world coordinate system, obtain the temporary positioning information of the imaging unit in the world coordinate system; and Based on the positioning information of the imaging unit before determination in the said world coordinate system and the position of the auxiliary marker in the world coordinate system, determine the position of the imaging unit, The said auxiliary marker is an auxiliary marker different from the said marker and existing in the said space.

Images