Mobile object position estimation system and mobile object position estimation method

Abstract

To provide a mobile object position estimating method capable of improving the accuracy of estimating an actual position of the mobile object.SOLUTION: A mobile object position estimation system 1 is provided with a computer 7 that stores three-dimensional coordinates 41 of a specific object 32 in an actual physical space 40. The computer 7 extracts a plurality of characteristic points of an object captured in a video image captured by a camera 4, generates an environment map 50 in which three-dimensional coordinates of the respective characteristic points are recorded and which includes information on the environment surrounding a mobile object 2, estimates three-dimensional coordinates 53 indicating the position of the mobile object 2 in the environment map 50 based on the characteristic points recorded in the environment map 50, and corrects the coordinate system and scale of the environment map 50 to match the coordinate system and size of the physical space 40 based on the coordinates 41 in the physical space 40 and the position in the video of the specific object 32 captured in the video image captured by the camera 4.SELECTED DRAWING: Figure 3

Description

Technical Field

[0004] , , , ,

[0005] , , ,

[0003] , , ,

[0001] Embodiments of the present invention relate to mobile object position estimation technology.

Background Art

[0002] Conventionally, in power plants or factories, in order to operate safely, inspectors regularly conduct patrol inspections and perform maintenance inspections on a vast number of devices. However, in the case of power plants located in remote areas or during emergencies such as disasters, unmanned inspections using robots or drones are desired. However, indoors, since satellite positioning systems cannot be used, position estimation by other means is required.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional maintenance inspections, there are many manual operations such as handwritten recording of the results confirmed by inspectors on-site and creating inspection records in the office, and efforts have been made to streamline the work. Here, there is a service that uses a mobile terminal to assist with manual work. However, it is difficult for inspectors to go to the site remotely or during emergencies. Therefore, consideration is being given to automatic inspections using robots or drones. To perform this automatic inspection, it is necessary to set the travel route of the robot or drone and estimate the self-position of the robot or drone.

[0005] As a method for estimating the self-position using inexpensive sensors, there is a method for estimating the self-position based on an image captured by a camera. However, the estimation of the self-position based on an image may result in an indefinite scale. Also, in order to collate inspection points or perform superimposition with drawings, it is necessary to improve the accuracy of the self-position estimation.

[0006] Embodiments of the present invention have been made in consideration of these circumstances, and aim to provide a mobile object position estimation technique that can improve the accuracy of estimating the actual position of a mobile object. [Means for solving the problem]

[0007] An embodiment of the present invention provides a mobile object position estimation system comprising: one or more cameras mounted on a mobile object and capturing images of the area surrounding the mobile object; and one or more computers that store the three-dimensional coordinates in actual physical space of at least one specific subject that is fixed in advance at the location where the mobile object is moving. The aforementioned specific subject includes a marker on which a readable code is printed, The computer extracts multiple feature points of at least one object captured in the image taken by the camera, records the three-dimensional coordinates of each feature point, generates an environment map containing information about the surrounding environment of the moving object, and estimates the three-dimensional coordinates indicating the position of the moving object in the environment map based on the feature points recorded in the environment map. The coordinates of the markers are stored, with each marker associated with an individually identifiable identification piece. , The number of markers detected when estimating the coordinates of the moving object is recorded, and the coordinates of the moving object estimated when a second number of markers is detected (which is greater than the first number) are used as a priority in processing to estimate the position of the moving object, rather than the coordinates of the moving object estimated when a first number of markers is detected. The system is configured to correct the coordinate system and scale of the environment map to match the coordinate system and size of the physical space, based on the coordinates in the physical space and the position in the image of the specific subject captured by the camera. [Effects of the Invention]

[0008] Embodiments of the present invention provide a mobile object position estimation technique that can improve the accuracy of estimating the actual position of a mobile object. [Brief explanation of the drawing]

[0009] [Figure 1] A side view showing the moving object position estimation system of the first embodiment. [Figure 2] A block diagram showing a mobile object position estimation system. [Figure 3] An explanatory diagram showing the processing flow of a mobile object position estimation system. [Figure 4] Plan view showing the inspection location. [Figure 5] Explanatory drawing showing a mode of correcting an environmental map to fit the physical space. [Figure 6] Front view showing a marker. [Figure 7] Explanatory drawing showing a marker management table. [Figure 8] Explanatory drawing showing a position recording table. [Figure 9] Explanatory drawing showing the processing flow of the mobile body position estimation system of the second embodiment. [Figure 10] Explanatory drawing showing a site marker management table. [Figure 11] Image diagram showing an image of the inspection location photographed by a camera. [Figure 12] Explanatory drawing showing the processing flow of the mobile body position estimation system of the third embodiment. [Figure 13] Explanatory drawing showing an object management table.

Mode for Carrying Out the Invention

[0010] (First Embodiment) Hereinafter, embodiments of a mobile body position estimation system and a mobile body position estimation method will be described in detail with reference to the drawings. First, the first embodiment will be described using FIGS. 1 to 8.

[0011] Reference numeral 1 in FIG. 1 is a mobile body position estimation system of the first embodiment. This mobile body position estimation system 1 estimates the position of a mobile body 2 which is an inspection robot traveling on the floor surface.

[0012] The mobile body 2 includes a plurality of wheels 3 provided at the lower part of its housing. The mobile body 2 travels by rotationally driving the wheels 3. Further, a camera 4 as imaging equipment is mounted on the upper part of the housing of the mobile body 2. This camera 4 photographs an image of the periphery of the mobile body 2. For example, the camera 4 photographs a detailed image of a subject such as a device or a structure to be inspected.

[0013] The mobile body position estimation system 1 estimates the position of the mobile body 2 based on the video captured by the camera 4. Since the camera 4 is provided at the upper center of the mobile body 2, the description will be made on the assumption that the shooting position of the camera 4 is the same as the position of the mobile body 2.

[0014] As shown in FIG. 4, the mobile body 2 travels indoors in a predetermined building 30 as an inspection location. A number of devices or structures to be inspection objects 31 are provided indoors, and the videos of these are captured by the camera 4. Then, an inspector located at a remote location checks the inspection object 31 by viewing the video. In addition, a plurality of markers 32 as specific subjects are attached to the indoor wall surface. The mobile body position estimation system 1 refers to the positions of these markers 32 and corrects the position of the mobile body 2.

[0015] As shown in FIG. 1, the camera 4 exemplifies an omnidirectional camera (360° camera) capable of simultaneously shooting in all directions. This camera 4 guides the surrounding scenery to one image sensor using, for example, a convex mirror and shoots an omnidirectional video. With this camera 4, it is possible to simultaneously shoot an all-sky image showing the omnidirectional top, bottom, left, and right around the mobile body 2. Note that the captured video does not have to be an all-sky image, and may also be a panoramic image showing a 360-degree range in the horizontal direction (omnidirectional left and right).

[0016] Here, a form in which one camera 4 shoots an omnidirectional video is exemplified, but other forms may also be used. For example, an omnidirectional video may be generated by synthesizing the videos captured by a plurality of cameras 4. Also, videos around the mobile body 2 may be simultaneously captured by a plurality of image sensors with fisheye lenses.

[0017] In addition, the camera 4 may be, for example, a pan-tilt-zoom camera. A pan-tilt-zoom camera has a pan function capable of panning horizontally, a tilt function capable of tilting vertically, and a zoom function capable of zooming in (telephoto) and zooming out (wide angle).

[0018] The example shown is a video captured at an arbitrary frame rate. Alternatively, the video captured by camera 4 could be a still image taken at regular intervals.

[0019] Next, the system configuration of the mobile object position estimation system 1 will be explained with reference to the block diagram shown in Figure 2.

[0020] The mobile body position estimation system 1 comprises a camera 4, a driving motor 5, a communication unit 6, and a control computer 7. The control computer 7 comprises a processing circuit 8 and a memory unit 9. These are mounted on the mobile body 2.

[0021] In this embodiment, for the sake of understanding, we illustrate a configuration in which all devices constituting the mobile body position estimation system 1 are mounted on the mobile body 2, but other configurations are also possible. For example, at least some of the functions of the control computer 7 may be provided on another computer located at a remote location other than the mobile body 2.

[0022] The driving motor 5 is mounted inside the housing of the mobile body 2 and rotates the wheels 3 (Figure 1). By controlling the rotation of each wheel 3 with the driving motor 5, the mobile body 2 can move forward, backward, and turn.

[0023] The communication unit 6 is a wireless communication device that communicates wirelessly. This communication unit 6 allows the control computer 7 to communicate with other computers. For example, the control computer 7 and other computers are connected to each other via a predetermined communication line such as the Internet, LAN (Local Area Network), WAN (Wide Area Network), or mobile communication network.

[0024] The control computer 7 controls the movement of the mobile body 2. In this embodiment, the control computer 7 automatically controls the movement of the mobile body 2, but other configurations are also possible. For example, the control computer 7 may receive input operations from a user in a remote location and control the movement of the mobile body 2. In other words, the control computer 7 may also be a remote control unit for controlling the mobile body 2 through manual operation by a user.

[0025] The control computer 7 has hardware resources such as a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and SSD (Solid State Drive). The CPU executes various programs, thereby enabling software-based information processing using these hardware resources. Furthermore, the moving object position estimation method of this embodiment is implemented by having the control computer 7 execute various programs.

[0026] Furthermore, each component of the mobile object position estimation system 1 does not necessarily have to be installed on a single computer. For example, one mobile object position estimation system 1 may be implemented using multiple computers connected to each other via a network. Alternatively, each mobile object position estimation system 1 may be installed on a separate computer.

[0027] The processing circuit 8 is, for example, a circuit equipped with a CPU, a GPU (Graphics Processing Unit), or a dedicated or general-purpose processor. This processor realizes various functions by executing various programs stored in the memory unit 9. The processing circuit 8 may also be composed of hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Various functions can also be realized by this hardware. Furthermore, the processing circuit 8 can realize various functions by combining software processing by the processor and programs with hardware processing.

[0028] The memory unit 9 stores various information necessary for estimating the position of the moving object 2 based on information stored in a predetermined database. For example, the memory unit 9 stores a marker management table (Figure 7) and a position record table (Figure 8). The database is a collection of information that is stored in memory, an HDD, an SSD, or the cloud and organized so that it can be searched or stored.

[0029] Although the example shows the memory unit 9 mounted on the mobile body 2, other configurations are also possible. For example, at least some of the functions of the memory unit 9 may be provided on another computer connected to the control computer 7 via a network. Alternatively, at least some of the functions of the memory unit 9 may be provided on the cloud.

[0030] As shown in Figure 3, the processing circuit 8 includes a feature point extraction unit 10, a feature point comparison unit 11, a shooting position estimation unit 12, a distance measurement unit 13, a shooting position correction unit 14, a marker detection unit 15, a marker tracking unit 16, and a marker shooting position estimation unit 17. The storage unit 9 includes a feature point recording unit 20, a shooting position recording unit 21, a corrected shooting position recording unit 22, a marker detection result recording unit 23, a marker shooting position recording unit 24, and a marker position recording unit 25. These functions of the processing circuit 8 and the storage unit 9 are realized by the CPU executing programs stored in memory, HDD, or SSD.

[0031] As shown in Figure 4, the inspection location is, for example, the interior of a building 30 located in a designated plant such as a power plant, chemical plant, or factory. The plant contains numerous objects 31 to be inspected. Alternatively, the inspection location may be a designated commercial or public facility containing a large building 30.

[0032] Multiple markers 32, which serve as specific objects, are provided in predetermined positions and fixed in place at the inspection site where the mobile object 2 is moving. These markers 32 are, for example, attached to several locations on the walls of an indoor building and are used to correct the position of the mobile object 2 when estimating its own position.

[0033] The control computer 7 stores the three-dimensional coordinates of these markers 32 in the actual physical space 40 (Figure 5). The control computer 7 also generates an environment map 50 (Figure 5) containing information about the surrounding environment in response to the movement of the mobile object 2. For example, the control computer 7 performs self-position estimation of the mobile object 2 and records the position and trajectory of the mobile object 2 in the environment map 50.

[0034] For self-localization, known techniques such as SLAM (Simultaneous Localization and Mapping) are used. This technique allows for the determination of the self-localization and movement amount of the moving object 2. In particular, VSLAM (Visual Simultaneous Localization and Mapping) is used. By using VSLAM, it becomes possible to record both video and position using only camera 4.

[0035] The control computer 7 calculates the changes in the position and orientation of the mobile object 2, which is the position where camera 4 is capturing images, based on the images captured by camera 4. Therefore, self-position estimation is possible even in places such as indoors where satellite positioning systems cannot be used. The orientation of the mobile object 2 includes information about its attitude, such as its tilt.

[0036] VSLAM is a technique that uses information acquired by camera 4 to extract feature points of surrounding objects. For example, it analyzes the video captured by camera 4 and extracts feature points that stand out in contrast to the surroundings, such as corners of objects, and generates point cloud data using multiple feature points. When camera 4 moves, the video changes with its movement, and the feature points in the video also move. By tracking the movement of these feature points in three dimensions, it is possible to simultaneously determine the 3D point cloud data of the inspection area and the position of camera 4. Furthermore, the orientation of camera 4 can also be determined from the positional relationship between the video from camera 4 and each feature point.

[0037] In this embodiment, feature points of objects captured in the omnidirectional video are extracted. For example, by calculating the trajectory of the camera 4 (mobile body 2) from a predetermined location whose position is known, the current position and orientation are calculated. This VSLAM is a technology that can estimate its own position even if 3D point cloud data of the inspection site has not been acquired in advance. Furthermore, at the inspection site, by calculating the feature points of each object captured in multiple frames that make up the omnidirectional video, the self-position can be estimated from the amount of movement of those feature points.

[0038] The 3D point cloud data acquired by VSLAM is recorded in the environment map 50 (Figure 5). In other words, the 3D coordinates of multiple feature points of the object are recorded in the environment map 50. The control computer 7 estimates the 3D coordinates indicating the position of the moving object 2 in the environment map 50 based on the feature points recorded in the environment map 50.

[0039] The position and orientation obtained by VSLAM represent the accumulation of relative position and orientation changes from the start of VSLAM processing. Therefore, if there is an error (positional misalignment) between the environment map 50 and the physical space 40 (Figure 5) at the start of VSLAM processing, it becomes impossible to estimate the accurate position of the mobile body 2. By correcting the misalignment of the environment map 50 in relation to the physical space 40, the position and orientation of the mobile body 2 corresponding to the inspection location can be estimated.

[0040] In the first embodiment, a marker 32 is used to align the environmental map 50 with the physical space 40. As shown in Figure 6, the marker 32 is a graphic that can be recognized by the control computer 7, and a readable code 33 is printed on a predetermined base sheet 34. For example, a matrix-type two-dimensional code, a so-called QR code (registered trademark), is printed on the base sheet 34. The marker 32 may also be a known AR marker. This marker 32 is attached to the wall surface of the inspection location.

[0041] Furthermore, the backing sheets 34 of each marker 32 are different colors. For example, the backing sheets 34 are colored red, blue, yellow, and green. It is preferable to arrange the backing sheets 34 of at least multiple markers 32 installed in the same room so that they are all different colors. It is also acceptable to have multiple markers 32 of the same color in the same room, but even in that case, it is preferable to arrange the colors of the multiple markers 32 installed on the same wall so that they are all different.

[0042] As shown in Figure 7, the marker management table associates each marker 32 with a marker ID, which is an identification piece of information that allows for individual identification. The table also registers the color of the marker 32, the location ID indicating the building 30 where the marker 32 is located, and the coordinates of the marker 32.

[0043] The location ID is identification information that can individually identify multiple buildings 30 or multiple rooms. In this embodiment, a coordinate system of the physical space 40 is set for each location ID.

[0044] Furthermore, the coordinates of the marker 32 registered in the marker management table are the coordinates of the pre-defined physical space 40 (Figure 5). In this way, the coordinates in the physical space 40 can be identified from the marker 32 that appears in the video captured by the camera 4, and the position of the moving object 2 can be estimated from that marker 32.

[0045] Next, the processing flow of the mobile object position estimation system 1 will be explained using Figure 3. Refer to the previously mentioned diagrams as appropriate.

[0046] Note that the arrows in Figure 3 are just one example of a processing flow, and there may be other processing flows besides those indicated by the arrows. Also, the order of each process is not necessarily fixed, and the order of some processes may be reversed. Furthermore, some processes may be executed in parallel with other processes. In addition, the mobile object position estimation system 1 may include components other than those shown in Figure 3, and some of the components shown in Figure 3 may be omitted.

[0047] First, camera 4 captures images of the area around the moving object 2, and these images are sent to processing circuit 8 (Figure 2). The feature point extraction unit 10 in processing circuit 8 extracts multiple feature points of objects in the images from the images obtained from camera 4, based on the brightness gradient and similarity of the images.

[0048] The feature point recording unit 20 records the three-dimensional coordinates of the extracted feature points in the environment map 50 (Figure 5). The feature point comparison unit 11 compares the feature points already recorded in the feature point recording unit 20 with the feature points newly extracted by the feature point extraction unit 10 based on the vectors of each feature point. In other words, the feature point comparison unit 11 compares whether the feature points depict the same object. For example, if the feature points extracted from the first frame of the video are already recorded in the environment map 50, the feature point comparison unit 11 compares whether the newly acquired feature points extracted from the second frame are related to the feature points from the first frame that have already been recorded.

[0049] In other words, the feature point comparison unit 11 compares the feature point to be newly recorded in the environment map 50 with known feature points. If the feature point comparison unit 11 determines that the new feature point is related to the known feature point, the feature point recording unit 20 associates the new feature point with an object related to the known feature point and records it in the environment map 50.

[0050] The shooting position estimation unit 12 estimates the location where the video was captured from the three-dimensional coordinates of each feature point of the object recorded in the environment map 50. In other words, the shooting position estimation unit 12 estimates the coordinates of the moving object 2 at the time the video from which the newly extracted feature points were captured, based on the coordinates of the newly extracted feature points. The shooting position recording unit 21 records the coordinates of the moving object 2 estimated by the shooting position estimation unit 12 in the environment map 50.

[0051] The distance measurement unit 13 measures the distance from the moving object 2 to a new feature point based on the coordinates of the moving object 2 and the position of the new feature point in the video. The position in the video refers to the two-dimensional coordinates, where the vertical and horizontal dimensions of a single video frame are the dimensions. For example, if there is an unrecorded feature point in the environment map 50, the distance measurement unit 13 calculates the distance from the feature point to the camera 4 using the principle of triangulation, based on the position of the feature point in the video. The feature point recording unit 20 records the distance measured by the distance measurement unit 13 along with the coordinates in the environment map 50. In this way, the precise position of objects can be recorded in the environment map 50.

[0052] The marker detection unit 15 detects markers 32 from the video footage of camera 4, from which the relative coordinates with respect to camera 4 can be determined. The markers 32 to be detected are registered in advance in the marker management table (Figure 7).

[0053] The marker tracking unit 16 tracks the marker 32 that appears in each frame that makes up the video. If the marker 32 cannot be detected from the video, the marker tracking unit 16 estimates the coordinates of the marker 32 from the amount of movement of the moving object 2. Then, based on the estimated coordinates of the marker 32 and the environment map 50, the marker tracking unit 16 identifies the area in the video where the marker 32 may be visible, and detects the marker 32 from the identified area. In this way, the accuracy of detecting the marker 32 from the video can be improved.

[0054] For example, as the distance from camera 4 to marker 32 increases, the detection accuracy of marker 32 decreases. Therefore, the marker tracking unit 16 reads the shooting position, which is the position of the moving object 2, from the environment map 50, extracts the range (region) of the image where marker 32 is estimated to be present, and performs detection processing of marker 32 using parameters that match that range. In this way, the detection accuracy of marker 32 can be improved by adjusting the parameters for image processing.

[0055] The marker detection result recording unit 23 records the coordinates of the marker 32 and the shooting position, which are the detection results of the marker 32, on the environment map 50.

[0056] The marker position recording unit 25 pre-records the coordinates of the fixed locations of each marker 32 in the coordinate system to be managed. This coordinate system to be managed is, for example, the coordinate system of physical space 40.

[0057] The marker shooting position estimation unit 17 associates the relative coordinates of the shooting position obtained from the marker 32 with the coordinates of the location where the marker 32 is fixed. The marker shooting position recording unit 24 records the shooting position obtained from the marker 32 by the marker shooting position estimation unit 17 in the environment map 50.

[0058] The shooting position correction unit 14 calculates an arbitrary transformation matrix so that the shooting positions obtained from the object's feature points and the shooting positions obtained from the markers 32 match within the same video frame. Then, the shooting position correction unit 14 applies the shooting position obtained from the markers 32 to the shooting position obtained from the feature points and corrects it.

[0059] Here, the shooting position correction unit 14 corrects the coordinate system and scale of the environment map 50 (Figure 5) to match the coordinate system and size of the physical space 40, based on the coordinates of the marker 32 in the physical space 40 (Figure 5) and its position in the image captured by the camera 4. This correction includes aligning the origin and orientation of the coordinate axes of the environment map 50 with the origin and orientation of the coordinate axes of the physical space 40.

[0060] As shown in Figure 5, the physical space 40 is, for example, the indoor space of building 30. A predetermined location in this physical space 40 is set as the origin of the coordinate system, and the coordinates 41 of each marker 32 in the physical space 40 are stored in the control computer 7. The coordinates 41 of each marker 32 are acquired in advance by the user at the inspection site and input into the control computer 7.

[0061] Furthermore, the environment map 50 is a virtual space that is generated and updated by the control computer 7 as the mobile object 2 moves. A predetermined position in this environment map 50 is used as the origin of the coordinate system, and the coordinates 51 of the marker 32, which are estimated in advance based on the coordinates 41 of the marker 32 in the physical space 40, are also recorded in the environment map 50.

[0062] Here, when the moving object 2 moves through the actual physical space 40, an error occurs between the actual movement path 42 and the movement path 52 recorded in the environment map 50. Therefore, the coordinate system and scale of the environment map 50 are corrected so that the coordinates 51 of the marker 32 recorded in the environment map 50 match the coordinates 41 of the marker 32 in the actual physical space 40.

[0063] The shooting position correction unit 14 (Figure 3) first determines the coordinates 43 of the camera 4's shooting position in physical space 40 using a projection matrix, based on the coordinates 41 of the marker 32 in physical space 40 and the position of the marker 32 in the video. Then, the shooting position correction unit 14 corrects the coordinate system and scale of the environment map 50 so that the coordinates 53 of the moving object 2 in the environment map 50 match the coordinates 43 of the actual shooting position. In this way, the coordinate system and scale of the environment map 50 can be corrected from the coordinates 43 of the camera 4's shooting position.

[0064] The scale of the environment map 50 refers to the magnification or reduction ratio of the environment map 50. This scale also depends on the scale at the start of VSLAM processing. Therefore, by correcting the scale, it is possible to estimate the precise shooting position of camera 4, that is, the precise position of the moving object 2 in physical space 40.

[0065] In this way, during inspection by the mobile unit 2, its own position (shooting position) is measured from the image of camera 4, and at the same time, the coordinate system is corrected and the scale of the coordinate system is corrected.

[0066] Conventional feature point-based self-localization (VSLAM) is possible even without markers 32, but the coordinate system and scale of the environment map 50 become undefined. On the other hand, self-localization using markers 32 can determine the self-localization using the coordinate system and scale of the environment map 50 only within the range where markers 32 are visible. In this embodiment, the accuracy of self-localization can be improved by estimating the self-localization using VSLAM and correcting the self-localization using markers 32.

[0067] As shown in Figure 3, the corrected shooting position recording unit 22 records the shooting position, which is the coordinates of the corrected environment map 50 and the moving object 2. Here, the corrected shooting position recording unit 22 registers the coordinates of the moving object 2 in the position recording table (Figure 8).

[0068] As shown in Figure 8, the position recording table registers the coordinates of the moving object 2 at the time each frame was acquired, and the marker ID of the marker 32 that appears in each frame, associated with the frame number of the video. The number of marker IDs associated with one frame number is equivalent to the number of markers 32 detected when estimating the coordinates of the moving object 2.

[0069] In this way, the corrected shooting position recording unit 22 (Figure 3) stores the estimated coordinates of the moving object 2 in association with the frame at the time the image of the object was captured. This makes it easier for the marker tracking unit 16 to find the coordinates of feature points that overlap each other in multiple frames, and also makes it easier to estimate the coordinates of the moving object 2.

[0070] Furthermore, the corrected shooting position recording unit 22 (Figure 3) stores the estimated coordinates of the moving object 2 in association with the frame at the time the video showing the marker 32 was captured. This makes it easier for the marker tracking unit 16 (Figure 3) to track the marker 32 as it appears in the video of each frame.

[0071] As shown in Figure 3, the feature point comparison unit 11 uses, for example, the frame number of the video in the position recording table as a key to determine transformed coordinates such that each feature point overlaps. In this way, the shooting position estimation unit 12 can estimate the shooting position based on each feature point. Based on the result, the shooting position correction unit 14 can correct the coordinate system and scale of the environment map 50. Therefore, inspectors can easily compare and confirm the inspection target object 31 in the video of the inspection location drawing.

[0072] Furthermore, the control computer 7 prioritizes using the coordinates of the moving object 2 estimated when a second number of markers 32 are detected (which is more than the first number) over the coordinates of the moving object 2 estimated when a first number of markers 32 are detected, in order to estimate the position of the moving object 2. In other words, the control computer 7 prioritizes using the coordinates of the moving object 2 estimated when there are many markers 32 in a single frame, in order to estimate the position of the moving object 2. The more markers 32 detected, the more accurately the position (shooting position) of the moving object 2 can be estimated.

[0073] For example, the feature points recorded by the feature point recording unit 20 are recorded in correspondence with the number of markers 32 that were visible in the video when the distance measurement unit 13 measured the distance. The shooting position estimation unit 12 then prioritizes processing the markers 32 that were visible in the video with the highest number, thereby deriving an estimation result that is more reliable in terms of positional relationships.

[0074] The shooting position correction unit 14 corrects the coordinate system and scale of the environment map 50 to match the coordinate system and size of the physical space 40, based on the coordinates of at least three markers 32 and their positions in the image. In this way, the correction of the coordinate system and scale of the three-dimensional environment map 50 can be performed more accurately.

[0075] Furthermore, conventionally, when the distance from camera 4 to marker 32 increases, the size of marker 32 that appears in the image becomes smaller, and it may become undetectable. In this case, the accuracy of estimating the shooting position using marker 32 decreases. While it is possible to increase the range in which marker 32 can be detected by attaching a large number of markers 32 to the wall surface, the process of attaching the markers 32 is time-consuming.

[0076] In this embodiment, the backing paper 34 of each marker 32 is a different color. Therefore, even if a marker 32 is far from the camera 4, its position can be extracted based on the color of the marker 32. For example, as shown in the marker management table (Figure 7), the control computer 7 stores the color information of the marker 32 in association with the identification information of the marker 32. The control computer 7 then detects the position of the marker 32 in the video based on these colors.

[0077] The marker detection unit 15 or marker tracking unit 16 extracts a specified color range (multiple pixels) from the area (region) in the video where the marker 32 may exist, and identifies the position of the marker 32 in the video by assuming that the marker 32 is located at the centroid of that range. The coordinates of this marker 32 are recorded in the environment map 50. In this way, even if the marker 32 is far away, it can be identified by its color.

[0078] Furthermore, if the marker 32 is recorded not as the marker 32 itself, but as a colored point, the marker position estimation unit 17 may process the positions of multiple points as a PnP problem. For example, the relationship between the coordinates of the marker 32 in physical space 40 and the position of the marker 32 in the video (two-dimensional coordinates in the video) may be analyzed as a PnP problem, and the shooting position may be estimated using this marker 32.

[0079] In the first embodiment, the user attaches a small number of markers 32 to the wall surface of the building 30. By performing this task, the user can correct the scale and coordinate system of the environmental map 50. This reduces the amount of manual correction work that was previously required, and simplifies the operation of the mobile unit 2 that performs automated inspections.

[0080] (Second Embodiment) Next, the mobile object position estimation system 1A of the second embodiment will be described with reference to Figures 9 to 11. Note that components identical to those shown in the previously described embodiments are denoted by the same reference numerals, and redundant explanations are omitted.

[0081] As shown in Figure 9, the processing circuit 8 (Figure 2) of the second embodiment includes an object detection unit 60, an object tracking unit 61, and an object position estimation unit 62, in addition to the configuration of the first embodiment. The storage unit 9 (Figure 2) of the second embodiment also includes a field marker recording unit 63, in addition to the configuration of the first embodiment. These functions of the processing circuit 8 and storage unit 9 are realized by the execution of programs stored in memory, HDD, or SSD by the CPU.

[0082] In the second embodiment, a predetermined object located at the inspection site is used instead of the marker 32 (Figure 4). For example, any type of object is provided in a fixed position in advance at the location where the mobile body 2 (Figure 4) is moving. When the coordinates of this arbitrary type of object are newly recorded, that object becomes a site marker as a specific subject.

[0083] As shown in Figure 11, the types of objects that can be used as field markers include, for example, equipment such as motors or pumps 70, instruments 71 or valves 72 installed on the equipment, and nameplates 73 attached to the equipment that display the equipment's brand name. In other words, anything that does not move at the inspection site becomes a field marker. These objects that appear in the video of the inspection site captured by camera 4 are processed by image recognition technology, and the position of the object in the video is identified. The types of objects that will be used as field markers are set in advance by the user.

[0084] The field marker recording unit 63 (Figure 9) stores a field marker management table. As shown in Figure 10, the field marker management table associates each field marker object with a marker ID, which is identification information that allows for individual identification. The table registers the type of object, a location ID indicating the building 30 where the object is located, and the coordinates of the object. This information is registered each time a predetermined object is detected from the camera 4's video feed. In other words, the field marker recording unit 63 stores the location and type of any object.

[0085] Furthermore, the coordinates of objects registered in the field marker management table are the coordinates of a pre-defined physical space 40 (Figure 5). In other words, the field marker recording unit 63 (Figure 9) stores the type and coordinates of objects, associating them with identification information that allows for individual identification of each object. In this way, the coordinates of an object in the physical space 40 can be identified from the image captured by the camera 4, and the position of the moving object 2 can be estimated from that object.

[0086] For example, a site marker registered when the mobile unit 2 first moves through an inspection area can be used to estimate the mobile unit 2's own position when it next moves through the inspection area.

[0087] Next, the processing flow of the mobile object position estimation system 1A will be explained using Figure 9. Refer to the previously mentioned diagrams as appropriate.

[0088] In the second embodiment, in addition to the processing of the first embodiment, a process is performed to detect objects of a type that will serve as a field marker. Here, the object detection unit 60 detects a rectangular region of the image (Figure 11) surrounding an object in the image captured by the camera 4. Furthermore, the object tracking unit 61 tracks objects that appear in each frame that makes up the image.

[0089] The object position estimation unit 62 estimates the position of the object. The field marker recording unit 63 first records the position of the object estimated by the object position estimation unit 62 in the environment map 50. This environment map 50 is corrected to match the physical space 40, so that the coordinates of the corrected object match the physical space 40.

[0090] In other words, when the object position estimation unit 62 detects any type of object from the video captured by the camera 4, it calculates three-dimensional coordinates indicating the object's position in physical space 40 based on the coordinates of a known marker 32 or a known field marker. The field marker recording unit 63 registers these corrected object coordinates as a new field marker (specific subject) in the field marker management table. In this way, newly detected objects can be used as field markers.

[0091] Subsequently, the coordinates of the field marker (object) are used to estimate the self-position of the mobile body 2, similar to the coordinates of marker 32. In this way, even if marker 32 cannot be detected, the object can be used in place of marker 32 to estimate the self-position of the mobile body 2.

[0092] In the second embodiment, the user attaches a small number of markers 32 to the walls of the building 30, or acquires the types of objects fixed at the inspection location and inputs them into the control computer 7. By simply performing these tasks, the user can correct the scale and coordinate system of the environmental map 50. This reduces the amount of manual correction work that was previously required, and simplifies the operation of the mobile unit 2 that performs automated inspections.

[0093] (Third embodiment) Next, the mobile object position estimation system 1B of the third embodiment will be described with reference to Figures 12 to 13. Note that components identical to those shown in the previously described embodiments are denoted by the same reference numerals, and redundant explanations are omitted.

[0094] As shown in Figure 12, the processing circuit 8 (Figure 2) of the third embodiment includes a feature point extraction unit 10, a feature point comparison unit 11, a shooting position estimation unit 12, a distance measurement unit 13, a shooting position correction unit 14, an object detection unit 60, an object tracking unit 61, and an object shooting position estimation unit 64. The storage unit 9 (Figure 2) of the third embodiment includes a feature point recording unit 20, a shooting position recording unit 21, a corrected shooting position recording unit 22, an object detection result recording unit 65, an object shooting position recording unit 66, and a drawing information recording unit 67. These functions of the processing circuit 8 and storage unit 9 are realized by the execution of a program stored in memory, HDD, or SSD by the CPU.

[0095] In the third embodiment, the marker 32 (Figure 4) of the first embodiment is not used, and an object located at the inspection site (site marker: specific subject) is used instead. The drawing information recording unit 67 stores an object management table. The information registered in this object management table is information about any type of object that is fixed in place beforehand.

[0096] As shown in Figure 13, the object management table associates each object (site marker) with an object ID, which is an identification piece of information that allows for individual identification. The table registers the type of object, the location ID indicating the building 30 where the object is located, and the coordinates of the object.

[0097] Furthermore, the coordinates of objects registered in the object management table are the coordinates of a pre-defined physical space 40 (Figure 5). In this way, the coordinates of an object in the physical space 40 can be identified from the image captured by camera 4, and the position of the moving object 2 can be estimated from that object.

[0098] Next, the processing flow of the mobile object position estimation system 1B will be explained using Figure 12. Refer to the previously mentioned diagrams as appropriate.

[0099] In the third embodiment, in addition to the process of extracting feature points as in the first embodiment, a process is performed to detect pre-set objects that serve as field markers.

[0100] The object detection unit 60 detects objects (field markers) from the video footage of camera 4, from which the relative coordinates with respect to camera 4 can be determined. The objects to be detected are pre-registered in the object management table (Figure 13). Here, the object detection unit 60 detects a rectangular region of the video footage (Figure 11) that surrounds the object.

[0101] The object tracking unit 61 tracks objects that appear in each frame that makes up the video. If an object cannot be detected from the video, the object tracking unit 61 estimates the object's coordinates from the amount of movement of the moving object 2. Then, based on the estimated object coordinates and the environment map 50, the object tracking unit 61 identifies the area in the video where an object may be visible and detects the object from that area. In this way, the accuracy of detecting objects from the video can be improved.

[0102] The object detection result recording unit 65 records the coordinates of the object and the shooting location, which are the object detection results, in the environment map 50.

[0103] The drawing information recording unit 67 pre-records the coordinates of the fixed locations of each object in the coordinate system to be managed. This coordinate system to be managed is, for example, the coordinate system of physical space 40.

[0104] The object shooting position recording unit 66 associates the relative coordinates of the shooting position obtained from the object with the coordinates of the location where the object is fixed. The object shooting position recording unit 66 records the shooting position obtained from the object by the object shooting position estimation unit 64 on the environment map 50.

[0105] The drawing information recording unit 67 records the type and coordinates of objects on drawings, 3D point cloud data, or 3D CAD. The object detection unit 60 calculates the position (2D coordinates) and type of the object in the video. These recorded and calculated results may be compared and processed as a PnP problem. For example, the relationship between the 3D coordinates and the position in the video of objects of the same type may be analyzed to estimate the camera 4's shooting position when the object was captured.

[0106] Furthermore, if an object not described in the drawing is detected, and the same object is detected in multiple frames, the three-dimensional coordinates of the object may be estimated from the shooting position of camera 4, and the estimated result may be recorded in the drawing information recording unit 67.

[0107] In the third embodiment, the user acquires the object type and coordinates and inputs them into the control computer 7. By performing this task alone, the user can correct the scale and coordinate system of the environment map 50. This reduces the amount of manual correction work that was previously required, and simplifies the operation of the mobile unit 2 that performs automated inspections.

[0108] Although the mobile object position estimation systems 1, 1A, and 1B and the mobile object position estimation method have been described above based on the first to third embodiments, a configuration applied in any one embodiment may be applied to another embodiment, or the configurations applied in each embodiment may be combined.

[0109] The system of the aforementioned embodiment comprises a control device with highly integrated processors such as FPGAs, GPUs, CPUs, and dedicated chips; storage devices such as ROMs and RAMs; external storage devices such as HDDs and SSDs; a display device such as a display; input devices such as a mouse and keyboard; and a communication interface. This system can be realized with a hardware configuration using a standard computer.

[0110] The program to be executed in the system of the above-described embodiment is provided pre-installed in ROM or the like. Additionally or alternatively, this program may be provided as an installable or executable file stored on a computer-readable non-temporary storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD).

[0111] Furthermore, programs executed by this system may be stored on computers connected to a network such as the Internet and provided for download via the network. Alternatively, this system can be configured by combining separate modules, each independently performing its respective function, and interconnected via a network or dedicated line.

[0112] The above-described embodiment exemplifies an inspection robot that travels on the floor as the mobile body 2, but other embodiments are also possible. For example, the mobile body 2 may be an inspection drone that flies in the air. Alternatively, the mobile body 2 may be a portable terminal that can be carried by an inspector. The inspector may carry the camera 4 (mobile body 2) and estimate the shooting position at a later date from the image from the camera 4. The mobile body 2 may also be a helmet worn by the inspector, and the camera 4 may be attached to this helmet. Alternatively, the camera 4 may be attached somewhere on the inspector's body. In other words, the inspector themselves may be configured as the mobile body 2.

[0113] According to at least one embodiment described above, the accuracy of estimating the actual position of the moving object 2 can be improved by correcting the coordinate system and scale of the environment map 50 to match the coordinate system and size of the physical space 40, based on the coordinates of a specific subject in the physical space 40 and its position in the image captured by the camera 4.

[0114] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, modifications, and combinations are possible without departing from the spirit of the invention. These embodiments or their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0115] 1, 1A, 1B... Mobile object position estimation system, 2... Mobile object, 3... Wheels, 4... Camera, 5... Driving motor, 6... Communication unit, 7... Control computer, 8... Processing circuit, 9... Memory unit, 10... Feature point extraction unit, 11... Feature point comparison unit, 12... Shooting position estimation unit, 13... Distance measurement unit, 14... Shooting position correction unit, 15... Marker detection unit, 16... Marker tracking unit, 17... Marker shooting position estimation unit, 20... Feature point recording unit, 21... Shooting position recording unit, 22... Corrected shooting position recording unit, 23... Marker detection result recording unit, 24... Marker shooting position recording 25...Marker position recording unit, 30...Building, 31...Inspection target, 32...Marker, 33...Code, 34...Backing sheet, 40...Physical space, 41...Coordinates, 42...Movement path, 43...Coordinates, 50...Environment map, 51...Coordinates, 52...Movement path, 53...Coordinates, 60...Object detection unit, 61...Object tracking unit, 62...Object position estimation unit, 63...Site marker recording unit, 64...Object shooting position estimation unit, 65...Object detection result recording unit, 66...Object shooting position recording unit, 67...Drawing information recording unit, 70...Pump, 71...Instrument, 72...Valve, 73...Nameplate.

Claims

1. One or more cameras mounted on a mobile device to capture images of the area surrounding the mobile device, One or more computers that store the three-dimensional coordinates in actual physical space of at least one specific object that is fixed in place in the location where the moving object is traveling, Equipped with, The aforementioned specific subject includes a marker on which a readable code is printed, The aforementioned computer, Multiple feature points of at least one object captured in the image taken by the camera are extracted. The three-dimensional coordinates of each of the aforementioned feature points are recorded, and an environment map including information about the surrounding environment of the moving object is generated. Based on the feature points recorded in the environmental map, the three-dimensional coordinates indicating the position of the moving object in the environmental map are estimated. The coordinates of the markers are stored by associating each marker with identification information that allows for individual identification of the markers. The number of markers detected when estimating the coordinates of the moving object is recorded, and the coordinates of the moving object estimated when a second number of markers is detected (which is greater than the first number) are used as a priority in processing to estimate the position of the moving object, rather than the coordinates of the moving object estimated when a first number of markers is detected. Based on the coordinates in physical space and the position in the image of the specific subject captured by the camera, the coordinate system and scale of the environment map are corrected to match the coordinate system and size of the physical space. It is structured in such a way. A system for estimating the position of a moving object.

2. One or more cameras mounted on a mobile body and capturing images of the area around the mobile body, One or more computers that store the three-dimensional coordinates in actual physical space of at least one specific object that is fixed in place in the location where the moving object is traveling, Equipped with, An object of any type is provided in a fixed state in the location where the moving body is moving, and the specific subject includes the object of a predetermined type as a site marker. The aforementioned computer, Multiple feature points of at least one object captured in the image taken by the camera are extracted. The three-dimensional coordinates of each of the aforementioned feature points are recorded, and an environment map including information about the surrounding environment of the moving object is generated. Based on the feature points recorded in the environmental map, the three-dimensional coordinates indicating the position of the moving object in the environmental map are estimated. The type and coordinates of the aforementioned field markers are stored, associated with identification information that allows for individual identification of each field marker. The number of on-site markers detected when estimating the coordinates of the moving object is recorded, and the coordinates of the moving object estimated when a second number of on-site markers is detected (which is greater than the first number) are used as the priority for processing to estimate the position of the moving object, compared to the coordinates of the moving object estimated when a first number of on-site markers is detected. Based on the coordinates in physical space and the position in the image of the specific subject captured by the camera, the coordinate system and scale of the environment map are corrected to match the coordinate system and size of the physical space. It is structured in such a way. A system for estimating the position of a moving object.

3. The aforementioned computer, The feature point newly recorded in the environment map is compared with the known feature point, and if it is determined that the new feature point is related to the known feature point, the new feature point is recorded in association with the object related to the known feature point. From the coordinates of the new feature points, estimate the coordinates of the moving object at the time the video in which the new feature points were extracted was filmed. Based on the coordinates of the moving body and the position of the new feature point in the video, the distance from the moving body to the new feature point is measured. The measured distance is recorded on the aforementioned environmental map. It is structured in such a way. A moving object position estimation system according to claim 1 or 2.

4. The aforementioned video is a video shot at an arbitrary frame rate, The aforementioned computer, The estimated coordinates of the moving object are stored in association with the frame at the time the video of the object was captured. It is structured in such a way. A moving object position estimation system according to claim 1 or claim 2.

5. The aforementioned video is a video shot at an arbitrary frame rate, The aforementioned computer, The estimated coordinates of the moving object are stored in association with the frame at the time the video of the specific subject was taken. It is structured in such a way. A moving object position estimation system according to claim 1 or claim 2.

6. The aforementioned computer, If the specific subject cannot be detected from the video, the coordinates of the specific subject are estimated from the amount of movement of the moving object. Based on the estimated coordinates of the specific subject and the environment map, the area in the video in which the specific subject may be visible is identified. The specific subject is detected from the specified range. It is structured in such a way. A moving object position estimation system according to claim 1 or claim 2.

7. The aforementioned computer, Based on the coordinates of the specific subject and its position in the video, the coordinates of the camera's shooting position in the physical space are determined. The coordinate system and scale of the environment map are corrected so that the coordinates of the moving object in the environment map match the coordinates of the shooting position. It is structured in such a way. A moving object position estimation system according to claim 1 or claim 2.

8. The aforementioned computer, Based on the coordinates of at least three of the specified subjects and their positions in the video, the coordinate system and scale of the environment map are corrected to match the coordinate system and size of the physical space. It is structured in such a way. A moving object position estimation system according to claim 1 or claim 2.

9. The aforementioned computer, The system stores the color information of the marker in association with the identification information of the marker. Based on the aforementioned color, the position of the marker in the video is detected. It is structured in such a way. The mobile object position estimation system according to claim 1.

10. The aforementioned computer, When an object of any type is detected from the image captured by the camera, three-dimensional coordinates indicating the position of the object in physical space are calculated based on the coordinates of a known specific subject. The object whose coordinates have been calculated is stored as a new site marker. It is structured in such a way. The mobile object position estimation system according to claim 2.

11. One or more cameras mounted on a mobile device to capture images of the area surrounding the mobile device, One or more computers that store the three-dimensional coordinates in actual physical space of at least one specific object that is fixed in place in the location where the moving object is traveling, This method uses The aforementioned specific subject includes a marker on which a readable code is printed, The aforementioned computer, Multiple feature points of at least one object captured in the image taken by the camera are extracted. The three-dimensional coordinates of each of the aforementioned feature points are recorded, and an environment map including information about the surrounding environment of the moving object is generated. Based on the feature points recorded in the environmental map, the three-dimensional coordinates indicating the position of the moving object in the environmental map are estimated. The coordinates of the markers are stored by associating each marker with identification information that allows for individual identification of the markers. The number of markers detected when estimating the coordinates of the moving object is recorded, and the coordinates of the moving object estimated when a second number of markers is detected (which is greater than the first number) are used as a priority in processing to estimate the position of the moving object, rather than the coordinates of the moving object estimated when a first number of markers is detected. Based on the coordinates in physical space and the position in the image of the specific subject captured by the camera, the coordinate system and scale of the environment map are corrected to match the coordinate system and size of the physical space. Mobile object position estimation method.

12. One or more cameras mounted on a mobile body and capturing images of the area around the mobile body, One or more computers that store the three-dimensional coordinates in actual physical space of at least one specific object that is fixed in place in the location where the moving object is traveling, This method uses An object of any type is provided in a fixed state in the location where the moving body is moving, and the specific subject includes the object of a predetermined type as a site marker. The aforementioned computer, Multiple feature points of at least one object captured in the image taken by the camera are extracted. The three-dimensional coordinates of each of the aforementioned feature points are recorded, and an environment map including information about the surrounding environment of the moving object is generated. Based on the feature points recorded in the environmental map, the three-dimensional coordinates indicating the position of the moving object in the environmental map are estimated. The type and coordinates of the aforementioned field markers are stored, associated with identification information that allows for individual identification of each field marker. The number of on-site markers detected when estimating the coordinates of the moving object is recorded, and the coordinates of the moving object estimated when a second number of on-site markers is detected (which is greater than the first number) are used as the priority for processing to estimate the position of the moving object, compared to the coordinates of the moving object estimated when a first number of on-site markers is detected. Based on the coordinates in physical space and the position in the image of the specific subject captured by the camera, the coordinate system and scale of the environment map are corrected to match the coordinate system and size of the physical space. Mobile object position estimation method.

Images