Estimation device, imaging device, moving body, estimation system, estimation method, and program

Abstract

This estimation device comprises a processor. The processor: acquires position information of a plurality of reference stations that respectively have light emitting bodies that emit light at different wavelength bands from each other; acquires image data obtained as a result of an image capture device mounted on a moving body capturing an image capture scene including the light emitting bodies; and estimates at least one of the position of the moving body and the orientation of the moving body on the basis of an in-image position that is the position of the light emitting body in the image represented by the image data and the position information of the plurality of reference stations.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an estimation device, an imaging device, a moving body, an estimation system, an estimation method, and a program. [Background technology]

[0002] JP 2019-095963 A discloses a position control system for a mobile body, which includes a mobile body that can be remotely or automatically controlled and is equipped with a plurality of light-emitting elements that emit light of different wavelengths, and a ground body installed on the ground, wherein the ground body includes a light-receiving unit that images the plurality of light-emitting elements, a processing unit that determines the two-dimensional coordinates of the plurality of light-emitting elements on a two-dimensional image captured by the light-receiving unit and determines the three-dimensional coordinates of the mobile body based on the determined two-dimensional coordinates of the plurality of light-emitting elements, and a control unit that controls the movement of the mobile body so that the three-dimensional coordinates determined by the processing unit match the three-dimensional coordinates of a predetermined target.

[0003] Japanese Patent Application Laid-Open Publication No. 2000-055657 discloses a positioning device that identifies multiple moving objects moving on a floor in a room and determines the two-dimensional position of each moving object, the positioning device comprising a base station and a transmitting means, the base station transmitting an identification signal corresponding to a specific moving object to all moving objects via the transmitting means, each moving object having a receiving means and a light source, the receiving means receiving the identification signal and determining whether the identification signal corresponds to its own identification signal or not, and emitting the light source if it corresponds to its own identification signal, and an imaging camera that captures an image of the light source emitted by the moving object and transmits the image to the base station, the base station correlating the position of the image with the two-dimensional position on the floor to determine the position of the moving object.

[0004] JP 2019-144183 A discloses a three-dimensional coordinate measuring device comprising: six or more reflective markers that are arranged on the surface of an object placed in a natural light environment and reflect one monochromatic light contained in the natural light; a light source that irradiates the object with the monochromatic light; a filter that has a transmittance for the wavelength of the monochromatic light lower than the transmittance for other monochromatic lights contained in the natural light; a camera that takes three or more images of the object through the filter from different positions; and a three-dimensional coordinate extraction unit that performs photogrammetry at the positions of the reflective markers in the three or more images, performs SfM using the results of the photogrammetry and feature points in the images excluding the reflective markers, and extracts the three-dimensional coordinates of the object. Summary of the Invention

[0005] One embodiment of the technology disclosed herein provides an estimation device, an imaging device, a moving body, an estimation system, an estimation method, and a program that can estimate at least one of the position and attitude of a moving body even in an environment where it is difficult to use GNSS. [Means for solving the problem]

[0006] A first aspect of the technology of the present disclosure is an estimation device that includes a processor, which acquires position information of a plurality of reference stations each having an illuminant that emits light in a different wavelength band from the others, acquires image data obtained by capturing an image scene including the illuminant by an imaging device mounted on a moving body, and estimates at least one of the position of the moving body and the attitude of the moving body based on the in-image position, which is the position of the illuminant within the image shown by the image data, and the position information of the plurality of reference stations.

[0007] A second aspect of the technology disclosed herein is an estimation device according to the first aspect, in which the image data includes a plurality of groups of image data obtained by capturing an image scene in a plurality of wavelength bands using an imaging device, and the processor identifies the position within the image of the light-emitting element possessed by one of the plurality of reference stations based on the result of subtraction processing between first image data obtained by capturing an image in an light-emitting element wavelength band, which is a wavelength band in which a light-emitting element possessed by one of the plurality of reference stations emits light, and second image data obtained by capturing an image in a first adjacent wavelength band having a central wavelength that is a specified wavelength away from the central wavelength of the light-emitting element wavelength band on either the longer or shorter wavelength side.

[0008] A third aspect of the technology of the present disclosure is an estimation device according to the second aspect, in which a processor identifies the position within an image of an illuminant possessed by a reference station based on the result of subtraction processing between first image data and third image data obtained by imaging in a second adjacent wavelength band having a central wavelength on the other side of the first adjacent wavelength band by a specified wavelength from the central wavelength of the illuminant wavelength band.

[0009] A fourth aspect of the technology of the present disclosure is an estimation device according to the first aspect, in which the image data includes a plurality of groups of image data obtained by capturing an image scene in a plurality of wavelength bands using an imaging device, and the processor identifies the position within the image of the light-emitting element possessed by one of the plurality of reference stations based on the result of subtraction processing between first image data obtained by capturing an image in an light-emitting element wavelength band, which is a wavelength band in which a light-emitting element possessed by one of the plurality of reference stations emits light, and fourth image data obtained by capturing an image in a third adjacent wavelength band having the most intense wavelength on either the longer or shorter wavelength side by a specified wavelength from the most intense wavelength in the light-emitting element wavelength band.

[0010] A fifth aspect of the technology of the present disclosure is an estimation device according to the fourth aspect, in which a processor identifies the position in an image of an illuminant possessed by a reference station based on the result of subtraction processing between the first image data and fifth image data obtained by imaging in a fourth adjacent wavelength band having the most intense wavelength on the other side of the third adjacent wavelength band by a specified wavelength from the most intense wavelength in the illuminant wavelength band.

[0011] A sixth aspect of the technology of the present disclosure is an estimation device according to any one of the first to fifth aspects, wherein the image data includes a plurality of groups of image data obtained by capturing an image scene in a plurality of wavelength bands using an imaging device, and the processor identifies the position within the image of an illuminant possessed by one of the plurality of reference stations based on the sixth image data obtained by capturing an image in a wavelength band different from the illuminant wavelength band emitted by the illuminant possessed by one of the plurality of reference stations.

[0012] A seventh aspect of the technology of the present disclosure is an estimation device according to any one of the first to sixth aspects, wherein the location information is obtained by converting the latitude and longitude of each of a plurality of reference stations into coordinates in a two-dimensional Cartesian coordinate system.

[0013] An eighth aspect of the technology of the present disclosure is an estimation device according to any one of the first to sixth aspects, wherein the location information is obtained by converting the latitude, longitude, and altitude of each of a plurality of reference stations into coordinates in a three-dimensional Cartesian coordinate system.

[0014] A ninth aspect according to the technique of the present disclosure is the estimation device according to any one of the first to eighth aspects, in which the plurality of reference stations is three or more.

[0015] A tenth aspect of the technology of the present disclosure is an imaging device comprising an image sensor and an estimation device relating to any one of the first to ninth aspects, wherein the estimation device estimates at least one of the position of a moving body and the attitude of the moving body using an image obtained by capturing an imaging scene with the image sensor.

[0016] An eleventh aspect of the technique of the present disclosure is the imaging device according to the tenth aspect, in which the image sensor is an image sensor capable of capturing images using a multispectral method.

[0017] A twelfth aspect of the technology of the present disclosure is a moving body equipped with an imaging device and an estimation device, wherein the estimation device acquires position information of a plurality of reference stations each having an illuminant that emits light in a different wavelength band from each other, acquires image data obtained by capturing an image scene including the illuminant by the imaging device, and estimates at least one of the position of the moving body and the attitude of the moving body based on the in-image position, which is the position of the illuminant within the image shown by the image data, and the position information of the plurality of reference stations.

[0018] A thirteenth aspect of the technology of the present disclosure is an estimation system comprising a moving body equipped with an imaging device, an estimation device, and a plurality of reference stations each having an illuminant that emits light in a different wavelength band from each other, wherein the estimation device acquires position information of the plurality of reference stations, acquires image data obtained by capturing an image scene including the illuminant by the imaging device, and estimates at least one of the position of the moving body and the attitude of the moving body based on the in-image position, which is the position of the illuminant within the image shown by the image data, and the position information of the plurality of reference stations.

[0019] A fourteenth aspect of the technology of the present disclosure is an estimation method that includes acquiring position information of a plurality of reference stations, each having an illuminant that emits light in a different wavelength band from the other, acquiring image data obtained by capturing an image scene including the illuminant by an imaging device mounted on a moving body, and estimating at least one of the position of the moving body and the attitude of the moving body based on the in-image position, which is the position of the illuminant within the image shown by the image data, and the position information of the plurality of reference stations.

[0020] A fifteenth aspect of the technology of the present disclosure is a program that causes a computer to execute processing including acquiring position information of multiple reference stations, each having an illuminant that emits light in a different wavelength band from the other, acquiring image data obtained by capturing an image scene including the illuminant by an imaging device mounted on a moving body, and estimating at least one of the position of the moving body and the attitude of the moving body based on the in-image position, which is the position of the illuminant within the image shown by the image data, and the position information of the multiple reference stations. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a conceptual diagram showing an example of a usage state of a moving object. [Figure 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a moving body. [Figure 3] FIG. 1 is a schematic diagram illustrating an example of the configuration of an optical system of an imaging device. [Figure 4] FIG. 1 is a schematic diagram illustrating an example of the configuration of an image sensor. [Figure 5] FIG. 2 is a block diagram showing an example of main functions of a processor. [Figure 6] FIG. 3 is a conceptual diagram illustrating an example of processing content of an image processing unit. [Figure 7] FIG. 3 is a conceptual diagram illustrating an example of processing content of an image processing unit. [Figure 8] FIG. 10 is a conceptual diagram illustrating an example of processing content of an estimation unit. [Figure 9] FIG. 1 is a conceptual diagram illustrating an example of estimating the position and orientation of a moving body. [Figure 10] 10 is a flowchart illustrating an example of the flow of an estimation process. [Figure 11] 10 is a flowchart showing an example of the flow of a light-emitting body identification process. [Figure 12] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the estimation system. [Figure 13] FIG. 10 is a conceptual diagram showing an example of how an estimation processing program is installed in a computer from a storage medium. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, exemplary embodiments of an estimation device, an imaging device, a moving body, an estimation system, an estimation method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". RAM is an abbreviation for "Random Access Memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". GUI is an abbreviation for "Graphical User Interface." CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor." CCD is an abbreviation for "Charge Coupled Device." GNSS is an abbreviation for "Global Navigation Satellite System." LED is an abbreviation for "Light Emitting Diode."

[0025] [First embodiment] The mobile object 10 is used for surveying and / or inspecting land and / or infrastructure, etc. Examples of infrastructure include road facilities (e.g., bridges, road surfaces, tunnels, guardrails, traffic lights, and / or windbreak fences), waterway facilities, airport facilities, port facilities, water storage facilities, gas facilities, power supply facilities, medical facilities, and / or firefighting facilities, etc. As an example, as shown in FIG. 1 , the mobile object 10 is an unmanned aerial vehicle (e.g., a drone) flying inside a tunnel 1.

[0026] As an example, as shown in FIG. 1, a moving body 10 includes a main body 32 and a plurality of propellers 34 (four propellers in the example shown in FIG. 1). The moving body 10 flies or hovers in three-dimensional space by controlling the rotation of each of the plurality of propellers 34. The moving body 10 is also equipped with an imaging device 30. The moving body 10 is an example of a "moving body" according to the technology of the present disclosure, and the imaging device 30 is an example of an "imaging device" according to the technology of the present disclosure.

[0027] Here, an unmanned aerial vehicle is cited as an example of the moving body 10, but the technology of the present disclosure is not limited thereto. For example, the moving body 10 may be a vehicle. Examples of vehicles include a vehicle with a gondola, a vehicle for high-altitude work, and a bridge inspection vehicle. The moving body 10 may also be a slider or a dolly on which the imaging device 30 can be mounted. The moving body 10 may also be a person. Here, a person refers to, for example, a worker who performs surveying and / or inspection of land and / or infrastructure. Note that when the moving body 10 is a person, the imaging device 30 being mounted on the moving body 10 includes a case where the imaging device 30 is held by the person and / or the imaging device 30 is attached to equipment worn by the person (e.g., a helmet, work clothes, etc.). The moving body 10 may also be a ship or an underwater drone.

[0028] Incidentally, when an inspection or the like of infrastructure or the like is performed using a mobile object 10, it is necessary to know the position and attitude of the mobile object 10 in order to control the movement of the mobile object 10. It is known that GNSS is used to know the position and attitude of the mobile object 10. However, as shown in FIG. 1 as an example, when the mobile object 10 is placed in an obstructed environment such as inside a tunnel 1, communication between the satellites used in the GNSS and the mobile object 10 is interrupted, making it difficult to estimate the position and attitude of the mobile object 10.

[0029] Therefore, in this embodiment, in order to realize estimation of the position and attitude of the moving body 10 even in an environment where communication with the outside (e.g., a satellite) is difficult, as an example, as shown in Figure 1, an imaging device 30 captures an image of an illuminant 37 provided at a reference station 36, thereby estimating the position and attitude of the moving body 10.

[0030] The reference station 36 is used to estimate the position and attitude of the moving body 10. The reference station 36 is provided at a position where the imaging device 30 of the moving body 10 can capture an image of the light-emitting body 37. In the example shown in FIG. 1, the reference station 36 is provided near the entrance and exit of the tunnel 1. Three or more reference stations 36 are provided. In the example shown in FIG. 1, three reference stations 36 are provided. In the example shown in FIG. 1, the three reference stations 36 are a first reference station 36A, a second reference station 36B, and a third reference station 36C.

[0031] Although three reference stations 36 are shown here as an example, this is merely an example, and four reference stations 36 may be installed around the mobile body 10, and the technology of the present disclosure is valid as long as three or more reference stations 36 are installed around the mobile body 10.

[0032] The reference station 36 is provided with a light emitter 37. The light emitter 37 emits light in a predetermined wavelength band. The light emitter 37 is, for example, an LED light source. The light emitters 37 provided in each of the three reference stations 36 emit light in wavelength bands that differ from one another among the reference stations 36. For example, each of the three reference stations 36 has a blue LED light source, a yellow LED light source, and a red LED light source as the light emitter 37.

[0033] Although an example has been described in which an LED light source is used as the light emitter 37, this is merely one example. The light emitter 37 may be any light source that can emit light in a predetermined wavelength band, and for example, an electroluminescence light source or a fluorescent lamp may be used as the light emitter 37.

[0034] 1, a first light emitter 37A is provided in the first reference station 36A. A second light emitter 37B is provided in the second reference station 36B. A third light emitter 37C is provided in the third reference station 36C. The first light emitter 37A is a blue LED light source, the second light emitter 37B is a yellow LED light source, and the third light emitter 37C is a red LED light source.

[0035] As an example, as shown in FIG. 2, the moving body 10 includes an imaging device 30, a controller 17, and an estimation device 38. The controller 17 is realized by, for example, an IC chip. A main body 32 is provided with a plurality of motors 17A. The plurality of motors 17A are connected to a plurality of propellers 34. The controller 17 controls the plurality of motors 17A to control the flight of the moving body 10.

[0036] The estimation device 38 includes a computer 39 and an external I / F 46. The computer 39 includes a processor 40, a storage 42, and a RAM 44. The processor 40, the storage 42, the RAM 44, and the external I / F 46 are connected to a bus 48. The estimation device 38 is an example of an "estimation device" according to the technology of the present disclosure. The computer 39 is an example of a "computer" according to the technology of the present disclosure. The processor 40 is an example of a "processor" according to the technology of the present disclosure.

[0037] The processor 40 has, for example, a CPU and a GPU, and controls the entire estimation device 38. The GPU operates under the control of the CPU and is responsible for screen display and / or image processing, etc. The processor 40 may be one or more CPUs that have integrated GPU functionality, or one or more CPUs that do not have integrated GPU functionality.

[0038] The storage 42 is a non-volatile storage device that stores various programs, various parameters, etc. Examples of the storage 42 include a flash memory (for example, an EEPROM and / or an SSD) and / or an HDD. Note that the flash memory and / or the HDD are merely examples, and non-volatile storage devices such as a magnetoresistive memory and / or a ferroelectric memory may be used instead of or in addition to the HDD and / or the SSD. The RAM 44 is a memory that temporarily stores information and is used as a work memory by the processor 40.

[0039] The imaging device 30 captures an image of an image scene around the moving object 10. The image scene includes light emitting elements 37A, 37B, and 37C (see FIG. 1). The imaging device 30 is a so-called multispectral camera, and includes an imaging optical system 11 and an image sensor 20.

[0040] The imaging optical system 11 includes a fisheye lens 11A as an objective lens. By using the fisheye lens 11A, it becomes possible to capture images of a wide range around the moving object 10. The image sensor 20 is an image sensor capable of capturing images using a multispectral method.

[0041] The imaging optical system 11 also includes a lens 12 and a pupil division filter 14. The lens 12 forms an optical image of a subject, including a target object, on a light receiving surface 20A of the image sensor 20. The pupil division filter 14 is provided at or near the pupil position of the imaging optical system 11 and divides the pupil portion of the imaging optical system 11 into nine optical regions. As shown in FIG. 3 as an example, the pupil division filter 14 is a filter formed by overlapping a narrow-band filter 16 and a polarizing filter 18. The pupil division filter 14 has nine optical regions Sj (j = a natural number from 1 to 9) that are equally divided into nine in the circumferential direction. Light of different wavelength bands is transmitted through each optical region Sj. Of the nine optical regions Sj, an optical region group including optical regions S1 to S3, an optical region group including optical regions S4 to S6, and an optical region group including optical regions S7 to S9 transmit light of different polarization directions (i.e., transmission polarization orientations). The nine optical regions Sj are realized by a combination of narrow-band filters 16 and polarizing filters 18, for example.

[0042] The narrow-band filter 16 has nine narrow-band filter sections F1 to F9 that are equally divided into nine sections in the circumferential direction. Each narrow-band filter section F1 to F9 corresponds to a corresponding optical region S1 to S9 of the pupil division filter 14. Each narrow-band filter section F1 to F9 includes a band-pass filter that transmits light of a different narrow band.

[0043] That is, the first narrowband filter section F1 transmits light in the first wavelength band Δf1. The second narrowband filter section F2 transmits light in the second wavelength band Δf2. The third narrowband filter section F3 transmits light in the third wavelength band Δf3. The fourth narrowband filter section F4 transmits light in the fourth wavelength band Δf4. The fifth narrowband filter section F5 transmits light in the fifth wavelength band Δf5. The sixth narrowband filter section F6 transmits light in the sixth wavelength band Δf6. The seventh narrowband filter section F7 transmits light in the seventh wavelength band Δf7. The eighth narrowband filter section F8 transmits light in the eighth wavelength band Δf8. The ninth narrowband filter section F9 transmits light in the ninth wavelength band Δf9.

[0044] The second wavelength band Δf2 is a wavelength band corresponding to the wavelength band of the blue LED light source. The center wavelength of the second wavelength band Δf2 is identical to the center wavelength of the blue LED light source (i.e., 503 nm, hereinafter also referred to simply as the "blue center wavelength"). Furthermore, it is preferable that the bandwidth of the second wavelength band Δf2 is within the bandwidth of the wavelength band of the blue LED light source. In this embodiment, "match" refers not only to a perfect match, but also to a match that includes an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure.

[0045] That is, the central wavelength of the second wavelength band Δf2 does not need to exactly match the blue central wavelength, but only needs to be within a range from the blue central wavelength to less than the half-width of the emission spectrum of the blue LED light source. The second wavelength band Δf2 is an example of the "light emitter wavelength band" according to the technology of the present disclosure.

[0046] The first wavelength band Δf1 has a center wavelength that is shorter than the center wavelength of the second wavelength band Δf2 (i.e., the blue center wavelength) by a specified wavelength. For example, the center wavelength of the first wavelength band Δf1 is 503 nm-λ1, which is shorter than the center wavelength of the second wavelength band Δf2 by λ1 (see FIG. 6). Furthermore, the third wavelength band Δf3 has a center wavelength that is longer than the center wavelength of the second wavelength band Δf2 (i.e., the blue center wavelength) by a specified wavelength. The center wavelength of the third wavelength band Δf3 is 503 nm+λ3, which is longer than the center wavelength of the second wavelength band Δf2 by λ3 (see FIG. 6). In other words, the third wavelength band Δf3 differs from the first wavelength band Δf1 and the second wavelength band Δf2, and is a wavelength band on the opposite side of the second wavelength band Δf2 from the first wavelength band Δf1 in the emission spectrum. The first wavelength band Δf1 is an example of a "first adjacent wavelength band" according to the technique of the present disclosure, and the third wavelength band Δf3 is an example of a "second adjacent wavelength band" according to the technique of the present disclosure.

[0047] Furthermore, it is preferable that λ1 and λ3 each have a value that is at least the half-width of the emission spectrum of the blue LED light source away from the center wavelength of the second wavelength band Δf2.

[0048] The fifth wavelength band Δf5 is a wavelength band corresponding to the wavelength band of the yellow LED light source. The center wavelength of the fifth wavelength band Δf5 coincides with the center wavelength of the yellow LED light source (i.e., 592 nm, hereinafter also referred to simply as the "yellow center wavelength"). The bandwidth of the fifth wavelength band Δf5 is preferably within the bandwidth of the wavelength band of the yellow LED light source.

[0049] The fourth wavelength band Δf4 and the sixth wavelength band Δf6 are wavelength bands on the shorter and longer wavelength sides, respectively, of the fifth wavelength band Δf5. The center wavelength of the fourth wavelength band Δf4 is 592 nm - λ4, and the sixth wavelength band Δf6 is 592 nm + λ6 (see FIG. 6). It is preferable that λ4 and λ6 are each values ​​that are away from the center wavelength of the fifth wavelength band Δf5 by at least the half-width of the emission spectrum of the yellow LED light source.

[0050] The eighth wavelength band Δf8 is a wavelength band corresponding to the wavelength band of the red LED light source. The center wavelength of the eighth wavelength band Δf8 coincides with the center wavelength of the red LED light source (i.e., 630 nm, hereinafter also referred to simply as the "red center wavelength"). The bandwidth of the eighth wavelength band Δf8 is preferably within the bandwidth of the wavelength band of the red LED light source.

[0051] The seventh wavelength band Δf7 and the ninth wavelength band Δf9 are wavelength bands on the shorter and longer wavelength sides, respectively, of the eighth wavelength band Δf8. The center wavelength of the seventh wavelength band Δf7 is 630 nm-λ7, and the ninth wavelength band Δf9 is 630 nm+λ9 (see FIG. 6). It is preferable that λ7 and λ9 are each values ​​that are away from the center wavelength of the eighth wavelength band Δf8 by at least the half-width of the emission spectrum of the red LED light source.

[0052] Polarizing filter 18 has three polarizing filter sections G1 to G3 that are divided into three equal parts in the circumferential direction. The first polarizing filter section G1 corresponds to the first optical region S1 to the third optical region S3 of pupil division filter 14. The second polarizing filter section G2 corresponds to the fourth optical region S4 to the sixth optical region S6 of pupil division filter 14. The third polarizing filter section G3 corresponds to the seventh optical region S7 to the ninth optical region S9 of pupil division filter 14.

[0053] Each of the polarizing filter sections G1 to G3 transmits light of a different polarization direction (i.e., transmission polarization orientation). Here, the polarization direction of light transmitted by the first polarizing filter section G1 is denoted as α1, the polarization direction of light transmitted by the second polarizing filter section G2 is denoted as α2, and the polarization direction of light transmitted by the third polarizing filter section G3 is denoted as α3. In the imaging device 30, the first polarizing filter section G1 transmits light with an azimuth angle of 0° (i.e., α1=0°). The second polarizing filter section G2 transmits light with an azimuth angle of 60° (i.e., α2=60°). The third polarizing filter section G3 transmits light with an azimuth angle of 120° (i.e., α3=120°).

[0054] The pupil division filter 14 obtained by overlapping the narrow-band filter 16 and polarizing filter 18 described above on the same axis operates as follows.

[0055] Light passing through the first optical region S1 of the pupil division filter 14 passes through the first narrow-band filter portion F1 of the narrow-band filter 16 and the first polarizing filter portion G1 of the polarizing filter 18. Therefore, light in the first wavelength band Δf1 is linearly polarized in the polarization direction α1 and emitted from the first optical region S1. Light passing through the second optical region S2 of the pupil division filter 14 passes through the second narrow-band filter portion F2 of the narrow-band filter 16 and the first polarizing filter portion G1 of the polarizing filter 18. Therefore, light in the second wavelength band Δf2 is linearly polarized in the polarization direction α1 and emitted from the second optical region S2. Light passing through the third optical region S3 of the pupil division filter 14 passes through the third narrow-band filter portion F3 of the narrow-band filter 16 and the first polarizing filter portion G1 of the polarizing filter 18. Therefore, light in the third wavelength band Δf3 is linearly polarized in the polarization direction α1 and emitted from the third optical region S3.

[0056] Furthermore, light passing through the fourth optical region S4 of the pupil division filter 14 passes through the fourth narrowband filter portion F4 of the narrowband filter 16 and the second polarizing filter portion G2 of the polarizing filter 18. Therefore, light in the fourth wavelength band Δf4 is linearly polarized in the polarization direction α2 and emitted from the fourth optical region S4. Light passing through the fifth optical region S5 of the pupil division filter 14 passes through the fifth narrowband filter portion F5 of the narrowband filter 16 and the second polarizing filter portion G2 of the polarizing filter 18. Therefore, light in the fifth wavelength band Δf5 is linearly polarized in the polarization direction α2 and emitted from the fifth optical region S5. Light passing through the sixth optical region S6 of the pupil division filter 14 passes through the sixth narrowband filter portion F6 of the narrowband filter 16 and the second polarizing filter portion G2 of the polarizing filter 18. Therefore, light in the sixth wavelength band Δf6 is linearly polarized in the polarization direction α2 and emitted from the sixth optical region S6.

[0057] Furthermore, light passing through the seventh optical region S7 of the pupil division filter 14 passes through the seventh narrow-band filter portion F7 of the narrow-band filter 16 and the third polarizing filter portion G3 of the polarizing filter 18. Therefore, light in the seventh wavelength band Δf7 is linearly polarized in the polarization direction α3 and emitted from the seventh optical region S7. Light passing through the eighth optical region S8 of the pupil division filter 14 passes through the eighth narrow-band filter portion F8 of the narrow-band filter 16 and the third polarizing filter portion G3 of the polarizing filter 18. Therefore, light in the eighth wavelength band Δf8 is linearly polarized in the polarization direction α3 and emitted from the eighth optical region S8. Light passing through the ninth optical region S9 of the pupil division filter 14 passes through the ninth narrow-band filter portion F9 of the narrow-band filter 16 and the third polarizing filter portion G3 of the polarizing filter 18. Therefore, light in the ninth wavelength band Δf9 is linearly polarized in the polarization direction α3 and emitted from the ninth optical region S9.

[0058] The entire imaging optical system 11 is provided so as to be movable back and forth along the optical axis L. This allows focus adjustment.

[0059] 4, the image sensor 20 has a plurality of pixels Pi (i=a natural number from 1 to 9) on the light receiving surface 20A. The pixels Pi are regularly arranged at a constant pitch along the horizontal direction (i.e., the x direction shown in FIG. 4) and the vertical direction (i.e., the y direction shown in FIG. 4).

[0060] The image sensor 20 has a pixel block PB including nine adjacent pixels Pi (i.e., 3 x 3). For ease of explanation, any pixel block PB will be referred to as PB(x, y) below. The notation (x, y) indicates that a pixel block PB is arranged at the xth position in the horizontal direction (i.e., the x direction shown in FIG. 4) and at the yth position in the vertical direction (i.e., the y direction shown in FIG. 4).

[0061] The pixel blocks PB(x, y) are regularly arranged along the horizontal direction (i.e., the x direction shown in FIG. 4) and the vertical direction (i.e., the y direction shown in FIG. 4). Each pixel Pi receives light with different characteristics.

[0062] The image sensor 20 has a pixel array layer 21, a polarization filter element array layer 23, a spectral filter element array layer 25, and a microlens array layer 27. The layers are arranged in the following order from the image plane side to the object side: pixel array layer 21, polarization filter element array layer 23, spectral filter element array layer 25, and microlens array layer 27.

[0063] The pixel array layer 21 has a large number of photodiodes 22 arranged two-dimensionally. One photodiode 22 corresponds to one pixel. The photodiodes 22 are regularly arranged in the horizontal direction (i.e., the x direction shown in FIG. 4) and the vertical direction (i.e., the y direction shown in FIG. 4).

[0064] The polarization filter element array layer 23 has three types of polarization filter elements 24A, 24B, and 24C that differ from one another in polarization direction (i.e., transmission polarization orientation). The polarization filter elements 24A, 24B, and 24C are arranged two-dimensionally. The polarization filter elements 24A, 24B, and 24C are arranged at the same intervals as those in the pixel array layer 21. Each polarization filter element 24A, 24B, and 24C is provided for each pixel. The polarization direction of light transmitted by the first polarization filter element 24A is denoted as β1, the polarization direction of light transmitted by the second polarization filter element 24B is denoted as β2, and the polarization direction of light transmitted by the third polarization filter element 24C is denoted as β3.

[0065] The first polarizing filter element 24A transmits light with an azimuth angle of 0° (i.e., β=0°). The second polarizing filter element 24B transmits light with an azimuth angle of 60° (i.e., β2=60°). The third polarizing filter element 24C transmits light with an azimuth angle of 120° (i.e., β3=120°).

[0066] In each pixel block PB(x, y), the polarization filter elements 24A, 24B, and 24C are arranged regularly. The first polarization filter element 24A is arranged in the first pixel P1, fourth pixel P4, and seventh pixel P7, which are pixels in the first column within the pixel block PB(x, y). The second polarization filter element 24B is arranged in the second pixel P2, fifth pixel P5, and eighth pixel P8, which are pixels in the second column within the pixel block PB(x, y). The third polarization filter element 24C is arranged in the third pixel P3, sixth pixel P6, and ninth pixel P9, which are pixels in the third column within the pixel block PB(x, y).

[0067] The spectral filter element array layer 25 has three types of spectral filter elements 26A, 26B, and 26C, each with a different spectral transmittance. The spectral filter elements 26A, 26B, and 26C are arranged two-dimensionally. The spectral filter elements 26A, 26B, and 26C are arranged at the same intervals as the photodiodes 22, and are arranged for each pixel.

[0068] The spectral filter elements 26A, 26B, and 26C transmit light passing through each of the narrowband filter portions F1 to F9 of the narrowband filter 16 at different transmittances. The first spectral filter element 26A has a characteristic of transmitting more light in a short wavelength band within the wavelength band of visible light. The second spectral filter element 26B has a characteristic of transmitting more light in an intermediate wavelength band. The third spectral filter element 26C has a characteristic of transmitting more light in a long wavelength band. For example, the spectral filter elements 26A, 26B, and 26C may be B, G, and R color filters that are provided in a general color image sensor.

[0069] In each pixel block PB(x, y), the spectral filter elements 26A, 26B, and 26C are arranged regularly. The first spectral filter element 26A is arranged in the first pixel P1, second pixel P2, and third pixel P3, which are pixels in the first row within the pixel block PB(x, y). The second spectral filter element 26B is arranged in the fourth pixel P4, fifth pixel P5, and sixth pixel P6, which are pixels in the second row within the pixel block PB(x, y). The third spectral filter element 26C is arranged in the seventh pixel P7, eighth pixel P8, and ninth pixel P9, which are pixels in the third row within the pixel block PB(x, y).

[0070] The microlens array layer 27 has a large number of microlenses 28 arranged two-dimensionally. The microlenses 28 are arranged at the same intervals as the photodiodes 22, and are provided for each pixel. The microlenses 28 are arranged for the purpose of efficiently collecting light from the imaging optical system 11 onto the photodiodes 22.

[0071] In the image sensor 20 described above, in each pixel block PB(x, y), each pixel Pi has a different combination of spectral filter elements 26A, 26B, and 26C and polarization filter elements 24A, 24B, and 24C, which causes the pixels Pi in the pixel block PB(x, y) to receive light with different characteristics.

[0072] Incidentally, when the image capture device 30 captures an image of the light-emitting body 37 in order to estimate the position and orientation of the moving body 10, it is necessary to identify an image 50 showing the light-emitting body 37 (hereinafter also referred to as "light-emitting body image 50") in the image obtained by capturing an image of an image scene including the light-emitting body 37. However, depending on the image capture conditions (for example, the distance from the moving body 10 to the light-emitting body 37, or the ambient brightness at the time of capturing the image), it may be difficult to distinguish the light-emitting body image 50 from images of subjects other than the light-emitting body 37 in the image obtained by capturing an image of the image scene.

[0073] In view of these circumstances, in this embodiment, as an example shown in FIG. 5, the estimation process is performed by a processor 40 of an estimation device 38. An estimation process program 42A is stored in a storage 42. The processor 40 reads the estimation process program 42A from the storage 42 and executes the read estimation process program 42A on a RAM 44 to perform the estimation process. The estimation process is realized by the processor 40 operating as an acquisition unit 40A, an image processing unit 40B, and an estimation unit 40C. The estimation process program 42A is an example of a "program" according to the technology of the present disclosure.

[0074] The acquisition unit 40A acquires the position information of the reference station 36 from the storage 42. The storage 42 stores the position information acquired via the external I / F 46 in advance. The position information of the reference station 36 is obtained, for example, by converting the GNSS coordinates of the reference station 36, which are longitude, latitude, and altitude, into coordinates in a three-dimensional Cartesian coordinate system. However, using the coordinates converted into the three-dimensional Cartesian coordinate system as the position information of the reference station 36 is merely one example. For example, the GNSS coordinates of the reference station 36 may be used as the position information as is.

[0075] The acquisition unit 40A acquires image data from the imaging device 30. The image data is data representing an image that includes an image of an imaging scene. The image data is obtained by imaging the imaging scene using the imaging device 30. The image data includes a plurality of image data groups representing images obtained by imaging the imaging scene around the moving object 10 in a plurality of wavelength bands. That is, the image data includes a plurality of image data groups obtained by imaging the imaging scene in each of the first wavelength band Δf1, the second wavelength band Δf2, the third wavelength band Δf3, the fourth wavelength band Δf4, the fifth wavelength band Δf5, the sixth wavelength band Δf6, the seventh wavelength band Δf7, the eighth wavelength band Δf8, and the ninth wavelength band Δf9.

[0076] The image processing unit 40B performs image processing on the image data acquired by the acquisition unit 40A to identify the light-emitting body 37 included as an image in the image represented by the image data. First, as an example, a case will be described in which a light-emitting body image 50A representing the first light-emitting body 37A, which is a blue LED light source, is identified in the image.

[0077] 6, the image processing unit 40B first extracts a determination region including the light emitter image 50 based on six pieces of image data in the fourth wavelength band Δf4 to the ninth wavelength band Δf9. The six pieces of image data in the fourth wavelength band Δf4 to the ninth wavelength band Δf9 are image data that do not include the blue emission spectrum emitted from the first light emitter 37A, among the nine pieces of image data in the first wavelength band Δf1 to the ninth wavelength band Δf9. In this embodiment, the concept of "not including" not only includes the concept of "not including" completely, but also the concept of "not including" in the sense of including an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure.

[0078] The image processing unit 40B performs a non-determination area determination process based on six pieces of image data of the fourth wavelength band Δf4 to the ninth wavelength band Δf9, determining areas where the light emitter image 50A does not exist as non-determination areas. The fourth wavelength band Δf4 to the ninth wavelength band Δf9 are an example of a "wavelength band different from the light emitter wavelength band" according to the technology of the present disclosure. The six pieces of image data of the fourth wavelength band Δf4 to the ninth wavelength band Δf9 are an example of "sixth image data" according to the technology of the present disclosure.

[0079] For example, for a certain pixel Pi, six pieces of pixel data (i.e., pixel values) at a position corresponding to the pixel Pi are obtained from six pieces of image data in the fourth wavelength band Δf4 to the ninth wavelength band Δf9, which do not include the emission spectrum of the first light emitter 37A, out of the nine pieces of image data. If any one of the obtained six pieces of pixel data exceeds a threshold, the position corresponding to the pixel Pi is determined to be an area where the light emitter image 50A does not exist (hereinafter also referred to as a "non-determination area"). The image processing unit 40B determines whether or not all pixel data of the image data showing the captured scene is a non-determination area, thereby realizing the determination of a non-determination area where the light emitter image 50A does not exist.

[0080] Based on the results of determining the non-judgment area where the light-emitting body image 50A does not exist, the image processing unit 40B extracts the area excluding the non-judgment area from the captured scene as a judgment area where the light-emitting body image 50A may exist.

[0081] The image processing unit 40B determines whether or not the object shown in the extracted determination area is the light-emitting body image 50A. As an example, as shown in Fig. 7, the image processing unit 40B determines whether or not the object in the determination area is the light-emitting body image 50A based on three pieces of image data of the first wavelength band Δf1 to the third wavelength band Δf3.

[0082] Incidentally, it is conceivable that images of other light sources or reflectors that emit blue light may exist within the determination area, in addition to the light-emitting body image 50A. On the other hand, even if the other light sources or reflectors that emit blue light are blue, they do not emit light with a central wavelength of 503 nm and a narrow wavelength width of about 30 to 50 nm, like a blue LED light source.

[0083] Therefore, the image processing unit 40B subtracts image data of a second wavelength band Δf2, whose center wavelength coincides with the blue center wavelength, from image data of a first wavelength band Δf1, whose center wavelength is a specified wavelength shorter than the center wavelength of the second wavelength band Δf2. For example, the image processing unit 40B subtracts image data of the first wavelength band Δf1 from image data of the second wavelength band Δf2. The image processing unit 40B determines whether the object in the determination area is the light-emitting body image 50A based on the result of the subtraction process. The image data of the second wavelength band Δf2 is an example of "first image data" according to the technology of the present disclosure, and the image data of the first wavelength band Δf1 is an example of "second image data" according to the technology of the present disclosure.

[0084] Similarly, the image processing unit 40B subtracts image data of a second wavelength band Δf2, whose center wavelength coincides with the blue center wavelength, from image data of a third wavelength band Δf3, whose center wavelength is longer than the center wavelength of the second wavelength band Δf2 by a specified wavelength. For example, the image processing unit 40B subtracts image data of the third wavelength band Δf3 from the image data of the second wavelength band Δf2. Based on the result of the subtraction process, the image processing unit 40B determines whether the object shown in the determination area is the light-emitting body image 50A. The image data of the third wavelength band Δf3 is an example of "third image data" according to the technology of the present disclosure.

[0085] If the object in the determination area is the light-emitting body image 50A, the emission intensity in the subtraction result will not be significantly reduced. On the other hand, if the object is a light source or a reflector that does not have an emission spectrum, such as a blue LED light source, the emission intensity in the subtraction result will be significantly reduced or will be a negative value. Therefore, the image processing unit 40B can determine whether the object shown in the determination area is the light-emitting body image 50A from the subtraction result.

[0086] Furthermore, when identifying the second light emitter 37B, which is a yellow LED light source, the image processing unit 40B performs image processing similar to that performed when identifying the first light emitter 37A. That is, first, the image processing unit 40B extracts a determination region based on six pieces of image data: the first wavelength band Δf1 to the third wavelength band Δf3 and the seventh wavelength band Δf7 to the ninth wavelength band Δf9. Furthermore, in the extracted determination region, the image processing unit 40B subtracts image data of the fifth wavelength band Δf5, whose center wavelength coincides with the yellow center wavelength, from image data of the fourth wavelength band Δf4, whose center wavelength is a specified wavelength shorter than the center wavelength of the fifth wavelength band Δf5. The image processing unit 40B also subtracts image data of the sixth wavelength band Δf6, whose center wavelength is a specified wavelength longer than the center wavelength of the fifth wavelength band Δf5, from image data of the fifth wavelength band Δf5, whose center wavelength coincides with the yellow center wavelength. Based on the result of the subtraction process, the image processing unit 40B determines whether or not the object shown in the determination area is an image 50B showing the second light emitter 37B.

[0087] Furthermore, when identifying the third light emitter 37C, which is a red LED light source, the image processing unit 40B performs image processing similar to that performed when identifying the first light emitter 37A and the second light emitter 37B. Specifically, first, the image processing unit 40B extracts a determination region based on six pieces of image data for the first wavelength band Δf1 to the sixth wavelength band Δf6. Furthermore, in the extracted determination region, the image processing unit 40B subtracts image data for an eighth wavelength band Δf8, whose center wavelength coincides with the red wavelength, from image data for a seventh wavelength band Δf7, whose center wavelength is a specified wavelength shorter than the center wavelength of the eighth wavelength band Δf8. The image processing unit 40B also subtracts image data for an eighth wavelength band Δf8, whose center wavelength coincides with the red center wavelength, from image data for a ninth wavelength band Δf9, whose center wavelength is a specified wavelength longer than the center wavelength of the eighth wavelength band Δf8. Based on the result of the subtraction process, the image processing unit 40B determines whether or not the object shown in the determination area is the image 50C showing the third light emitter 37C.

[0088] Based on the result of identifying the light-emitting body image 50, the image processing unit 40B identifies an in-image position GP (hereinafter also simply referred to as "in-image position GP"), which is the position of the light-emitting body image 50 within the image. Identifying the in-image position GP refers to calculating the two-dimensional coordinates of the pixels that form the light-emitting body image 50 within the image represented by the image data. The example shown in Figure 7 shows an example in which the in-image position GP of the light-emitting body image 50A is identified.

[0089] 8, the estimation unit 40C acquires the intra-image position GP calculated by the image processing unit 40B. The estimation unit 40C also acquires the position information of the reference station 36 from the acquisition unit 40A.

[0090] Furthermore, the estimation unit 40C estimates the position and attitude of the moving body 10 based on the in-image position GP and the position information of the reference station 36. Estimating the position and attitude of the moving body 10 means calculating the position coordinates of the moving body 10 in a three-dimensional Cartesian coordinate system in three-dimensional space, as well as the roll angle, pitch angle, and yaw angle of the moving body 10.

[0091] When estimating the position and orientation of an object based on its position in three-dimensional space and its position in an image coordinate system, the position and orientation of the object can generally be estimated by solving the PnP (Perspective n-Point) problem. For example, the position and orientation of the moving object 10 can be estimated using the solution described in "Nakano et al., 'A Unified Solution Using Groebner Basis for the PnP Problem of a General Camera Model,' Image Recognition and Understanding Symposium (MIRU2011), pp. 845-851, 2011." The estimation unit 40C estimates the position and orientation of the moving object 10 using an arithmetic expression with the image position GP and position information (e.g., three-dimensional Cartesian coordinates) of the reference station 36 corresponding to the light-emitting object 37 as independent variables, and the position coordinates and Euler angles of the moving object 10 in the three-dimensional Cartesian coordinate system as dependent variables. The method for estimating the position and attitude of the moving body 10 described here is merely an example, and various methods for solving PnP problems can be used as appropriate to estimate the position and attitude of the moving body 10. Furthermore, when estimating the attitude of the moving body 10, the inertial coordinate system may be transformed into a body coordinate system using a direction cosine matrix, so that the attitude of the moving body 10 may be estimated as the attitude in the body coordinate system. FIG. 9 shows an example of the body coordinate system, in which the center of gravity G of the moving body 10 is the origin and the axis X L , axis Y L , and axis Z L A coordinate system consisting of

[0092] Once the position and attitude of the moving body 10 are estimated, as shown in FIG. 9 as an example, the estimation device 38 outputs the results of estimating the position and attitude of the moving body 10 to the controller 17. The controller 17 controls the position and attitude of the moving body 10 based on the results of estimating the position and attitude of the moving body 10. For example, when the moving body 10 is in a position deviating from a predetermined path, the controller 17 controls the operation of the motor 17A so that the moving body 10 returns to the predetermined path.

[0093] Next, the operation of the estimation device 38 will be described with reference to FIGS.

[0094] 10 and 11 show an example of the flow of the estimation process performed by the processor 40. The flow of the estimation process shown in Fig. 10 and 11 is an example of an "estimation method" according to the technology of the present disclosure.

[0095] As an example, in the estimation process shown in FIG. 10, first, in step ST10, the acquisition unit 40A determines whether or not the estimation timing for estimating the position and attitude of the moving object 10 has arrived. One example of the estimation timing is whether a pre-specified time has elapsed since the time when the previous estimation was performed. If the estimation timing has not arrived in step ST10, the determination is negative, and the estimation process proceeds to step ST10. If the estimation timing has arrived in step ST10, the determination is positive, and the estimation process proceeds to step ST12.

[0096] In step ST12, the acquisition unit 40A acquires the position information of the reference station 36 from the storage 42. After the processing of step ST12 is executed, the estimation processing proceeds to step ST14.

[0097] In step ST14, the acquisition unit 40A causes the imaging device 30 to capture an image. After the processing of step ST14 is executed, the estimation processing proceeds to step ST16.

[0098] In step ST16, the acquisition unit 40A acquires image data obtained by causing the imaging device 30 to capture an image in step ST14. After the process of step ST16 is executed, the estimation process proceeds to step ST18.

[0099] In step ST18, the image processing unit 40B performs the light-emitting body identification process shown in Fig. 11 as an example. In the light-emitting body identification process shown in Fig. 11, first, in step ST20, the image processing unit 40B extracts a determination region based on image data obtained by capturing an image in a wavelength band different from the wavelength band emitted by the light-emitting body 37. After the process of step ST20 is executed, the light-emitting body identification process proceeds to step ST22.

[0100] In step ST22, the image processing unit 40B performs subtraction processing between image data obtained by capturing an image in the wavelength band emitted by the light emitter 37 in the determination area extracted in step ST20 and image data obtained by capturing an image in a wavelength band having a central wavelength on the shorter or longer wavelength side of the central wavelength of the wavelength band emitted by the light emitter 37. After the processing of step ST22 is executed, the light emitter identification processing proceeds to step ST24.

[0101] In step ST24, the image processing unit 40B identifies the light-emitting body image 50 in the image based on the subtraction result calculated in step ST22. After the processing of step ST24 is executed, the light-emitting body identification processing proceeds to step ST26.

[0102] In step ST26, the image processing unit 40B calculates the intra-image position GP based on the result of identifying the light-emitting body image 50 in step ST24. After the processing of step ST26 is executed, the light-emitting body identification processing proceeds to step ST28.

[0103] In step ST28, the image processing unit 40B determines whether all in-image positions GP have been identified. If in step ST28, all in-image positions GP have not been identified, the determination is negative, and the light-emitting object identification process proceeds to step ST20. If in step ST28, all in-image positions GP have been identified, the determination is positive, and the light-emitting object identification process proceeds to step ST30 of the estimation process shown in FIG. 10, as an example.

[0104] In step ST30, the estimation unit 40C estimates the position and attitude of the moving object 10 based on the in-image position GP calculated by the image processing unit 40B and the position information of the reference station 36 acquired by the acquisition unit 40A. After the processing of step ST30 is executed, the estimation processing proceeds to step ST32.

[0105] In step ST32, the estimation unit 40C determines whether or not a condition for terminating the estimation process (hereinafter referred to as the "termination condition") has been satisfied. One example of the termination condition is that the moving object 10 has terminated movement along a pre-specified route. If the termination condition has not been satisfied in step ST32, the determination is negative, and the estimation process proceeds to step ST10. If the termination condition has been satisfied in step ST32, the determination is positive, and the estimation process ends.

[0106] As described above, the moving body 10 according to this embodiment can estimate at least one of the position and attitude of the moving body 10 even in an environment where it is difficult to use GNSS.

[0107] Furthermore, in the moving object 10, for example, when the light emitter 37 is a blue LED light source, the intra-image position GP is identified based on the result of subtraction processing of image data obtained by imaging in the second wavelength band Δf2 and image data obtained by imaging in the first wavelength band Δf1. Therefore, according to this configuration, it is possible to identify the intra-image position GP more accurately and quickly than when the intra-image position GP is identified using only the image data obtained by imaging in the second wavelength band Δf2.

[0108] Furthermore, in the moving object 10, for example, when the light emitter 37 is a blue LED light source, the intra-image position GP is identified based on the result of subtraction processing of image data obtained by imaging in the second wavelength band Δf2 and image data obtained by imaging in the third wavelength band Δf3. Therefore, according to this configuration, it is possible to identify the intra-image position GP more accurately and quickly than when the intra-image position GP is identified using only the image data obtained by imaging in the second wavelength band Δf2.

[0109] Furthermore, in the moving object 10, for example, when the light emitter 37 is a blue LED light source, the intra-image position GP is identified based on six pieces of image data in the fourth wavelength band Δf4 to the ninth wavelength band Δf9, which are wavelength bands different from the second wavelength band Δf2. Therefore, with this configuration, it is possible to identify the intra-image position GP more accurately and quickly than when the intra-image position GP is identified using only image data obtained by capturing an image in the second wavelength band Δf2.

[0110] Furthermore, the position information of the moving body 10 is obtained by converting the latitude, longitude, and altitude of each of the multiple reference stations 36 into coordinates in a three-dimensional Cartesian coordinate system. Therefore, according to this configuration, the calculation required to estimate the position and attitude of the moving body 10 is easier than when the latitude, longitude, and altitude are directly used as the position information of the reference station 36, and at least one of the position and attitude of the moving body 10 can be estimated more quickly.

[0111] Furthermore, since the position and attitude of the moving body 10 are estimated using three or more reference stations 36, at least one of the position and attitude of the moving body 10 can be estimated more accurately than when the number of reference stations 36 is less than three.

[0112] Furthermore, in the moving body 10, the imaging device 30 is equipped with an image sensor 20 capable of capturing images using a multispectral method, so the position GP within the image can be identified more accurately and quickly compared to when the image sensor 20 is not a multispectral method.

[0113] [Second embodiment] In the first embodiment, an example is described in which the estimation device 38 is provided in the moving body 10 and the position and attitude estimation process is performed in the moving body 10, but the technology of the present disclosure is not limited to this. In the second embodiment, the estimation process is performed in an information processing device 71 provided outside the moving body 10A.

[0114] As an example, as shown in FIG. 12 , an estimation system 70 includes a mobile object 10A, a reference station 36, and an information processing device 71. The estimation system 70 is an example of an “estimation system” according to the technology of the present disclosure. The information processing device 71 is an example of an “estimation device” according to the technology of the present disclosure. An example of the information processing device 71 is a laptop personal computer, but this is merely an example and may be a tablet terminal, a smartphone, or a desktop personal computer. The information processing device 71 is not limited to a personal computer and may also be a server. The server may be a mainframe used on-premises together with the mobile object 10A, or an external server realized by cloud computing. The server may also be an external server realized by network computing such as fog computing, edge computing, or grid computing. The information processing device 71 may also be a terminal for operating the mobile object 10A.

[0115] The information processing device 71 includes a reception device 76 and a display 75. The reception device 76 has a keyboard, a mouse, a touch panel, etc., and receives instructions from a user. The display 75 displays various information (e.g., images, text, etc.). The display 75 is, for example, an EL display (e.g., an organic EL display or an inorganic EL display). Note that the display is not limited to an EL display, and may be another type of display such as a liquid crystal display.

[0116] The mobile object 10A is connected to the information processing device 71 so as to be able to communicate wirelessly, and various information is exchanged wirelessly between the mobile object 10A and the information processing device 71. For example, when the mobile object 10A is inspecting the inside of a tunnel, the information processing device 71 is installed within the tunnel that the mobile object 10A is inspecting, within a distance where wireless communication is possible with the mobile object 10A. The communication I / F 77 receives image data from the mobile object 10A by wirelessly communicating with the communication I / F 19 of the mobile object 10A. The image data received by the communication I / F 77 is acquired and processed by the processor 40.

[0117] The information processing device 71 acquires the position information of the reference station 36. The method of acquiring the position information of the reference station 36 is not particularly limited, but for example, the position information may be acquired by exchanging information via wired or wireless communication between the information processing device 71 and the reference station 36, or may be acquired via a portable storage medium on which the position information of the reference station 36 is stored. The position information of the reference station 36 acquired by the information processing device 71 is stored in the storage 42.

[0118] The processor 40 estimates the position and attitude of the moving body 10A based on the image data acquired via the communication I / F 77 and the position information of the reference station 36 acquired from the storage 42. The processor 40 performs GUI control to cause the display 75 to display the estimated results of the position and attitude of the moving body 10A. The processor 40 also causes the display 75 to output the estimated results of the position and attitude of the moving body 10A to the moving body 10A via the communication I / F 77.

[0119] The communication I / F 19 of the moving body 10A receives the estimation results of the position and attitude of the moving body 10A by wirelessly communicating with the communication I / F 77 of the information processing device 71. The controller 17 controls the flight of the moving body 10A based on the estimation results of the position and attitude of the moving body 10A acquired via the communication I / F 19.

[0120] As described above, the estimation system 70 according to the second embodiment can estimate at least one of the position and attitude of the moving object 10A even in an environment where it is difficult to use GNSS.

[0121] In the second embodiment, the estimation processing program 42A is stored in the storage 42 of the information processing device 71. However, the technology of the present disclosure is not limited to this. For example, the estimation processing program 42A is stored in a storage device of another computer or server device connected to the information processing device 71 via a network. The estimation processing program 42A may be downloaded in response to a request from the information processing device 71 and installed on the computer 39.

[0122] Furthermore, it is not necessary to store the entire estimation processing program 42A in a storage device of another computer or server device connected to the information processing device 71, or in the storage 42; only a part of the estimation processing program 42A may be stored therein.

[0123] Furthermore, as an example, the information processing device 71 shown in FIG. 12 has a built-in computer 39, but the technology of the present disclosure is not limited to this, and for example, the computer 39 may be provided outside the information processing device 71.

[0124] [First Modification] In the first and second embodiments, multiple wavelength bands are distinguished by their center wavelengths. However, the technology of the present disclosure is not limited to this. In the first modification, multiple wavelength bands are distinguished by using the wavelength with the greatest intensity in a wavelength band (hereinafter also referred to as the peak wavelength). Specifically, the first wavelength band Δf1 is a wavelength band having a peak wavelength that is a specified wavelength shorter than the peak wavelength of the second wavelength band Δf2, whose peak wavelength coincides with the peak wavelength of the blue LED light source. Similarly, the third wavelength band Δf3 is a wavelength band having a peak wavelength that is a specified wavelength longer than the peak wavelength of the second wavelength band Δf2. The first wavelength band Δf1 is an example of a "third adjacent wavelength band" according to the technology of the present disclosure, and the third wavelength band Δf3 is an example of a "fourth adjacent wavelength band" according to the technology of the present disclosure.

[0125] Below, we will explain the subtraction process when distinguishing between multiple wavelength bands using peak wavelengths. First, image processing unit 40B subtracts image data of the second wavelength band Δf2 from image data of the first wavelength band Δf1, which has a peak wavelength that is a specified wavelength shorter than the peak wavelength of the second wavelength band Δf2. Image processing unit 40B identifies the intra-image position GP based on the result of the subtraction process. The image data of the first wavelength band Δf1, which has a peak wavelength that is a specified wavelength shorter than the peak wavelength of the second wavelength band Δf2, is an example of the "fourth image data" according to the technology of the present disclosure.

[0126] Furthermore, image processing unit 40B subtracts image data of the second wavelength band Δf2 from image data of a third wavelength band Δf3, which has a peak wavelength that is longer than the peak wavelength of the second wavelength band Δf2 by the specified wavelength. Image processing unit 40B identifies an intra-image position GP based on the subtraction result. Image data of the third wavelength band Δf3, which has a peak wavelength that is longer than the peak wavelength of the second wavelength band Δf2 by the specified wavelength, is an example of "fifth image data" according to the technology of the present disclosure.

[0127] The fourth wavelength band Δf4 to the sixth wavelength band Δf6 and the seventh wavelength band Δf7 to the ninth wavelength band Δf9 can also be distinguished by using peak wavelengths in the same way as the first wavelength band Δf1 to the third wavelength band Δf3. Similarly, the wavelength bands distinguished by peak wavelengths can be used in the process of extracting the determination region by the image processing unit 40B.

[0128] As described above, in the first modified example, when the light emitter 37 is a blue LED light source, for example, the intra-image position GP is identified based on the result of subtracting image data obtained by imaging in the second wavelength band Δf2 from image data obtained by imaging in the first wavelength band Δf1. Therefore, with this configuration, it is possible to identify the intra-image position GP more accurately and quickly than when the intra-image position GP is identified using only image data obtained by imaging in the second wavelength band Δf2.

[0129] Furthermore, in this first modified example, when the light emitter 37 is a blue LED light source, the intra-image position GP is identified based on the result of subtracting image data obtained by imaging in the second wavelength band Δf2 from image data obtained by imaging in the third wavelength band Δf3. Therefore, with this configuration, it is possible to identify the intra-image position GP more accurately and quickly than when the intra-image position GP is identified using only image data obtained by imaging in the second wavelength band Δf2.

[0130] [Second Modification] In the above-described first and second embodiments, an example was described in which the results of converting the latitude, longitude, and altitude of the GNSS coordinates into coordinates in a three-dimensional Cartesian coordinate system were used as the position information of the reference station 36. However, the technology of the present disclosure is not limited to this. In the present second modification, the position information of the reference station 36 is obtained by converting the latitude and longitude of the GNSS coordinates into coordinates in a two-dimensional Cartesian coordinate system. As an example, when the mobile object 10 moves at a constant height, the latitude and longitude of the GNSS coordinates can be converted into coordinates in a two-dimensional Cartesian coordinate system and used as the position information. Examples of cases in which the mobile object 10 moves at a constant height include when the mobile object 10 runs on flat ground or when the mobile object 10 flies at a constant height.

[0131] According to this second variant, the calculations required to estimate the position and attitude of the moving body 10 are easier than when latitude and longitude are directly used as the position information of the reference station 36, and at least one of the position and attitude of the moving body 10 can be estimated more quickly.

[0132] In the above embodiment, an example has been described in which the position and attitude of the mobile body 10 are estimated with the reference station 36 installed, but the technology of the present disclosure is not limited to this. For example, the reference station 36 may be movable. As an example, the reference station 36 may include a traveling mechanism, and the mobile body 10 may perform the position and attitude estimation process after the reference station 36 has moved to a pre-designated position. Alternatively, the reference station 36 may be moved by an operator and installed at a pre-designated position, after which the position and attitude of the mobile body 10 may perform the position and attitude estimation. Furthermore, the reference station 36 does not have to be newly prepared for estimating the position and attitude of the mobile body 10, but may be an existing facility (e.g., a lamp, a traffic light, etc.) equipped with a light source corresponding to the light-emitting body 37.

[0133] In the above embodiment, the image processing unit 40B identifies the intra-image position GP based on the result of subtracting image data in the first wavelength band Δf1 or the third wavelength band Δf3 from image data in the second wavelength band Δf2 when the light emitter 37 is a blue LED light source, for example. However, the technology of the present disclosure is not limited to this. For example, the image processing unit 40B may identify the intra-image position GP based on the result of subtracting image data in the second wavelength band Δf2 from image data in the first wavelength band Δf1 or the third wavelength band Δf3. Furthermore, the image processing unit 40B may perform a similar subtraction process when identifying the intra-image position GP when the light emitter 37 is a yellow LED light source or a red LED light source, for example.

[0134] In addition, in the above embodiment, an example in which the position and attitude of the moving body 10 are estimated has been described, but the technology of the present disclosure is not limited to this. For example, only the position of the moving body 10 may be estimated, or only the attitude of the moving body 10 may be estimated.

[0135] Furthermore, in the above embodiment, an example was described in which the image capture device 30 includes the pupil division filter 14, and each pixel of the image sensor 20 has a different combination of spectral filter elements 26A, 26B, and 26C and polarization filter elements 24A, 24B, and 24C, thereby receiving light with different characteristics, but the technology of the present disclosure is not limited to this. It is sufficient for the image capture device 30 to be able to capture images using a multispectral method, and the image capture device may also be an image capture device that is capable of capturing images using a multispectral method using a filter wheel.

[0136] In the above embodiment, the imaging device 30 is described as having a fisheye lens 11A as an objective lens, but the technology of the present disclosure is not limited to this. For example, a plurality of imaging devices 30 may be provided, and each imaging device 30 may capture an image of the periphery of the moving object 10 in synchronization with the other imaging devices 30.

[0137] In the above embodiment, the light emitter 37 provided in the reference station 36 emits light in the visible light band, but the technology of the present disclosure is not limited to this. The light emitter 37 may emit light in a wavelength band that can be imaged by the imaging device 30, for example, in the infrared wavelength band.

[0138] Furthermore, in the above embodiment, an example in which the estimation processing program 42A is stored in the storage 42 has been described, but the technology of the present disclosure is not limited to this. As an example, as shown in FIG. 13 , the estimation processing program 42A may be stored in a portable storage medium 100 such as an SSD or a USB memory. The storage medium 100 is a non-transitory computer-readable storage medium. The estimation processing program 42A stored in the storage medium 100 is installed in the computer 39 of the estimation device 38. The processor 40 executes the estimation processing in accordance with the estimation processing program 42A.

[0139] Furthermore, in the above embodiment, a computer 39 is exemplified, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 39. Furthermore, instead of the computer 39, a combination of a hardware configuration and a software configuration may be used.

[0140] Furthermore, the hardware resources for executing the estimation process described in the above embodiments may be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing the estimation process by executing software, i.e., a program. Examples of processors include dedicated electronic circuits, such as FPGAs, PLDs, or ASICs, which are processors with a circuit configuration specifically designed to execute specific processes. Each processor has a built-in or connected memory, and each processor uses the memory to execute the estimation process.

[0141] The hardware resource that executes the estimation process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the estimation process may be a single processor.

[0142] As an example of configuring a system using a single processor, there is a first form in which one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes the estimation process. A second form is a form in which a processor is used that realizes the functions of the entire system, including multiple hardware resources that execute the estimation process, on a single IC chip, as typified by SoCs. In this way, the estimation process is realized using one or more of the various processors described above as hardware resources.

[0143] Furthermore, the hardware structure of these various processors can be, more specifically, an electronic circuit that combines circuit elements such as semiconductor devices. The above estimation process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0144] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0145] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0146] The disclosure of Japanese Patent Application No. 2021-161787, filed on September 30, 2021, is incorporated herein by reference in its entirety.

[0147] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. a processor; The processor: acquiring position information of a plurality of reference stations each having a light emitter that emits light in a different wavelength band; acquiring image data obtained by capturing an image of an image scene including the light-emitting body by an imaging device mounted on a moving body, the image data including a plurality of image data groups obtained by capturing an image of the image scene by the imaging device in a plurality of wavelength bands; and based on the result of subtracting first image data obtained by imaging in an illuminant wavelength band, which is a wavelength band in which an illuminant possessed by one of the plurality of reference stations emits light, from second image data obtained by imaging in a first adjacent wavelength band having a central wavelength that is located on either the longer wavelength or shorter wavelength side by a specified wavelength from the central wavelength of the illuminant wavelength band, a position within the image that is the position of the illuminant possessed by the one reference station is identified, At least one of the position of the moving object and the attitude of the moving object is estimated based on the position in the image and the position information of the plurality of reference stations. Estimation device.

2. The processor: The position in the image of the light-emitting body of the one reference station is identified based on a result of subtraction processing between the first image data and third image data obtained by imaging in a second adjacent wavelength band having a center wavelength on the other side of the first adjacent wavelength band by a wavelength specified from the center wavelength of the light-emitting body wavelength band. The estimation device according to claim 1 .

3. the image data includes a plurality of image data groups obtained by capturing images of the scene by the imaging device in a plurality of wavelength bands, The processor: A light emitter having a wavelength band emitted by a light emitter of one of the plurality of reference stations The position of the light-emitting body of the one reference station in the image is identified based on a result of subtraction processing between first image data obtained by imaging in a wavelength band and fourth image data obtained by imaging in a third adjacent wavelength band having a wavelength with the strongest intensity on either the longer wavelength side or the shorter wavelength side by a specified wavelength from the wavelength with the strongest intensity in the light-emitting body wavelength band. The estimation device according to claim 1 .

4. The processor: The position in the image of the light-emitting device of the one reference station is identified based on a result of subtraction processing between the first image data and fifth image data obtained by imaging in a fourth adjacent wavelength band having a wavelength with the strongest intensity on the other side of the third adjacent wavelength band by a wavelength specified from the wavelength with the strongest intensity in the light-emitting device wavelength band. The estimation device according to claim 3 .

5. the image data includes a plurality of image data groups obtained by capturing images of the scene by the imaging device in a plurality of wavelength bands, The processor: determining a non-determined region, which is a region in the image represented by the image data where no light-emitting body image representing the light-emitting body exists, based on sixth image data obtained by capturing an image of the light-emitting body of one of the plurality of reference stations in a wavelength band different from the light-emitting body wavelength band in which the light-emitting body has; extracting a determination region from the image, the determination region being a region excluding the non-determination region within the image; Within the determination area, a position within the image of the light-emitting body of the one reference station is identified based on a result of the subtraction process. The estimation device according to claim 1 .

6. The position information is obtained by converting the latitude and longitude of each of the plurality of reference stations into coordinates in a two-dimensional orthogonal coordinate system. The estimation device according to claim 1 .

7. The position information is obtained by converting the latitude, longitude, and altitude of each of the plurality of reference stations into coordinates in a three-dimensional orthogonal coordinate system. The estimation device according to claim 1 .

8. The plurality of reference stations is three or more. The estimation device according to claim 1 .

9. An image sensor; The estimation device according to claim 1, The estimation device estimates at least one of the position of the moving body and the attitude of the moving body using an image obtained by capturing an image of the image capture scene by the image sensor. Imaging device.

10. The image sensor is an image sensor capable of capturing images in a multispectral manner. The imaging device according to claim 9 .

11. A moving object including an imaging device and an estimation device, The estimation device includes: acquiring position information of a plurality of reference stations each having a light emitter that emits light in a different wavelength band; acquiring image data obtained by capturing an image of an image scene including the light-emitting body by the imaging device, the image data including a plurality of image data groups obtained by capturing an image of the image scene by the imaging device in a plurality of wavelength bands; and based on the result of subtracting first image data obtained by imaging in an illuminant wavelength band, which is a wavelength band in which an illuminant possessed by one of the plurality of reference stations emits light, from second image data obtained by imaging in a first adjacent wavelength band having a central wavelength that is located on either the longer wavelength or shorter wavelength side by a specified wavelength from the central wavelength of the illuminant wavelength band, a position within the image that is the position of the illuminant possessed by the one reference station is identified, At least one of the position of the moving object and the attitude of the moving object is estimated based on the position in the image and the position information of the plurality of reference stations. Mobile object.

12. An estimation system including a moving object equipped with an imaging device, an estimation device, and a plurality of reference stations each having a light emitter that emits light in a different wavelength band, The estimation device includes: acquiring position information of the plurality of reference stations; acquiring image data obtained by capturing an image of an image scene including the light-emitting body by the imaging device, the image data including a plurality of image data groups obtained by capturing an image of the image scene by the imaging device in a plurality of wavelength bands; and based on the result of subtracting first image data obtained by imaging in an illuminant wavelength band, which is a wavelength band in which an illuminant possessed by one of the plurality of reference stations emits light, from second image data obtained by imaging in a first adjacent wavelength band having a central wavelength that is located on either the longer wavelength or shorter wavelength side by a specified wavelength from the central wavelength of the illuminant wavelength band, a position within the image that is the position of the illuminant possessed by the one reference station is identified, At least one of the position of the moving object and the attitude of the moving object is estimated based on the position in the image and the position information of the plurality of reference stations. Estimation system.

13. acquiring position information of a plurality of reference stations each having a light emitter that emits light in a different wavelength band; acquiring image data obtained by capturing an image of an image scene including the light-emitting body by an image capturing device mounted on a moving body, the image data including a plurality of image data groups obtained by capturing an image of the image scene by the image capturing device in a plurality of wavelength bands; Identifying an in-image position, which is the position of the light-emitting body in an image shown by the image data of the light-emitting body possessed by one of the plurality of reference stations, based on a result of subtraction processing between first image data obtained by imaging in a light-emitting body wavelength band, which is a wavelength band in which a light-emitting body possessed by one of the plurality of reference stations emits light, and second image data obtained by imaging in a first adjacent wavelength band having a central wavelength that is on either the longer wavelength side or the shorter wavelength side by a specified wavelength from the central wavelength of the light-emitting body wavelength band; and Estimating at least one of the position of the moving object and the attitude of the moving object based on the position in the image and the position information of the plurality of reference stations. Estimation methods including:

14. On the computer, acquiring position information of a plurality of reference stations each having a light emitter that emits light in a different wavelength band; acquiring image data obtained by capturing an image of an image scene including the light-emitting body by an image capturing device mounted on a moving body, the image data including a plurality of image data groups obtained by capturing an image of the image scene by the image capturing device in a plurality of wavelength bands; First image data obtained by capturing an image in a light-emitting element wavelength band, which is a wavelength band emitted by a light-emitting element of one of the plurality of reference stations; and a center wavelength of the light-emitting element wavelength band. and determining an in-image position, which is a position of the light-emitting body in an image shown by the image data of the light-emitting body possessed by the one reference station, based on a result of subtraction processing between the first image data and second image data obtained by capturing an image in a first adjacent wavelength band having a center wavelength on either the longer wavelength side or the shorter wavelength side by a wavelength specified by the first adjacent wavelength band; and Estimating at least one of the position of the moving object and the attitude of the moving object based on the position in the image and the position information of the plurality of reference stations. A program that executes processing including

Images