Storage device and storage method

Abstract

To appropriately estimate a self position.SOLUTION: A storage device comprises: a sensor unit which captures a peripheral image of a predetermined position and measures a distance from the predetermined position to a peripheral object; a key point extraction unit which extracts key points from a visible region and an invisible region in the image captured by the sensor unit; a self-position estimation unit which estimates a position based on the key points in the visible region and the key points in the invisible region; and a storage unit which stores information regarding a position estimation result of the self-position estimation unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a storage device and a storage method. [Background technology]

[0002] There are known techniques for inspecting the presence or absence of cracks in concrete walls during construction at construction sites such as tunnels. For example, Patent Document 1 discloses a technique for inspecting defects on the surface of road structures using a vehicle. For example, Patent Document 2 discloses a technique for associating the state of a deformed part of a structure with its coordinates. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6068099 [Patent Document 2] Patent No. 6829846 Summary of the Invention [Problem to be solved by the invention]

[0004] In an environment such as a construction site where various objects are placed in various locations and there are many workers and construction vehicles moving around, it can be difficult to detect cracks in a structure and properly associate the cracks with location information.

[0005] An object of the present disclosure is to provide a storage device and a storage method capable of appropriately performing self-location estimation. [Means for solving the problem]

[0006] The storage device according to the present disclosure includes a sensor unit that captures an image of the surroundings of a predetermined position and measures the distance from the predetermined position to surrounding objects, a key point extraction unit that extracts key points from visible and invisible areas in the image captured by the sensor unit, a self-position estimation unit that performs position estimation based on the key points in the visible area and the key points in the invisible area, and a memory unit that stores information regarding the position estimation result of the self-position estimation unit.

[0007] The storage method of the present disclosure includes the steps of capturing an image of the surroundings of a predetermined position and measuring the distance from the predetermined position to surrounding objects, extracting key points from visible and invisible areas in the captured image, performing position estimation based on the key points in the visible area and the key points in the invisible area, and storing information regarding the position estimation result in a memory unit. Effect of the Invention

[0008] According to the present disclosure, self-location estimation can be performed appropriately. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an overview of the storage device according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the storage device according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of the storage device according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing an overview of the self-location estimation process according to the first embodiment. [Diagram 5] FIG. 5 is a diagram for explaining an example of the self-position estimation process according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of imaging information stored in the captured image storage unit according to the first embodiment. [Figure 7]FIG. 7 is a flowchart showing the flow of the self-position estimation process according to the second embodiment. [Figure 8A] FIG. 8A is a diagram showing an example of RGB-D data according to the second embodiment. [Figure 8B] FIG. 8B is a diagram showing an example of RGB-D data according to the second embodiment. [Figure 9] FIG. 9 is a diagram for explaining an example of a method for setting the radar angle of view. [Figure 10] FIG. 10 is a diagram for explaining a method for generating three-dimensional radar information according to the second embodiment. [Figure 11] FIG. 11 is a flowchart showing the contents of a process for setting the luminance of an object region according to the second embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of three-dimensional radar information according to the second embodiment. [Figure 13] FIG. 13 is a diagram showing an example of three-dimensional radar information projected on a virtual screen according to the second embodiment. [Figure 14A] FIG. 14A is a diagram showing an example of transparency information according to the second embodiment. [Figure 14B] FIG. 14B is a diagram showing an example of transparency information according to the second embodiment. [Figure 15] FIG. 15 is a flowchart showing the flow of the self-position estimation process according to the third embodiment. [Figure 16] FIG. 16 is a diagram showing depth data of transmission information according to the third embodiment. [Figure 17] FIG. 17 is a diagram for explaining a method for generating three-dimensional radar information according to the third embodiment. [Figure 18A] FIG. 18A is a diagram showing an example of transparency information according to the third embodiment. [Figure 18B] FIG. 18B is a diagram showing an example of transparency information according to the third embodiment. [Figure 19] FIG. 19 is a flowchart showing the flow of the crack information output process according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the present disclosure is not limited to this embodiment, and in the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] [Comparative Example] Before describing the embodiments of the present disclosure, a technique according to a comparative example will be described.

[0012] In a construction site such as a tunnel, when a crack occurs in the concrete of a wall during construction, the crack needs to be repaired during the construction period. Therefore, workers inspect the cracks, and when they find a crack, they record the condition and location of the crack. The workers record the condition of the crack, for example, by sketching and taking a photo. Usually, imaging devices such as cameras and smartphones include a function of adding location information of the shooting location to the shooting data. The camera acquires location information by receiving satellite signals such as the Global Navigation Stellite System (GNSS) with a receiving device, and associates the location information with the shooting data. However, in a construction site where the sky is blocked, such as a tunnel, the camera cannot receive satellite signals and cannot acquire location information. In such a case, the worker manually identifies the location of the crack using a tape measure.

[0013] A worker uses, for example, a vehicle to inspect concrete cracks in a tunnel (see, for example, Patent Document 1). For example, a lighting means and a photographing means are installed on the loading platform of the vehicle. While the vehicle is traveling, the lighting means illuminates the wall surface and the photographing means photographs the condition of the wall surface. In this case, based on the image data photographed by the photographing means, it is automatically detected whether or not there is deterioration such as a crack on the wall surface. The photographing means photographs the wall surface of one tunnel as one image, making it easy to identify the position of the crack or the like. For example, a line sensor is used as the photographing means. For this reason, in order to uniformly measure the wall surface of the tunnel, the vehicle is required to keep the distance between the photographing means and the wall surface constant and to travel at a constant speed along the wall surface. For this reason, the inspection of cracks using a vehicle is not suitable for an environment where various components are placed in various places, scaffolding is set up, and many workers are working.

[0014] In recent years, with the development of HMDs (Head Mounted Displays), a maintenance management system for structures such as tunnels that utilizes MR (Mixed Reality) technology has been proposed (see, for example, Patent Document 2). Such a maintenance management system uses a camera and a depth sensor to sense the surrounding conditions. Based on the sensing results, the maintenance management system generates an image showing information corresponding to the surrounding conditions from the drawing information and maintenance management information of the structure. The maintenance management system displays and outputs the generated image so as to be superimposed on the actual surrounding scenery. This allows a worker to observe the structure itself while superimposing information on the piping arranged behind it and the soil obtained in a pre-construction investigation without destroying the structure. Furthermore, by superimposing and displaying information on cracks and the like previously investigated, a worker can observe the progress of deterioration of the cracks.

[0015] In a maintenance management system, for example, Visual-SLAM (Simultaneous Localization and Mapping) is used to estimate position. In Visual-SLAM, characteristic images (keypoints) are extracted from surrounding images captured by a camera. Of the extracted keypoints, Visual-SLAM stores those that are deemed optimal for use in self-location estimation calculations as map points. Visual-SLAM estimates the position of the HMD from changes in the appearance of multiple map points between frames. As a result, it is possible to estimate position by capturing surrounding images with the HMD camera, identify the position on the structure's blueprint, and read line-of-sight information from blueprint information and maintenance management information, and visualize it.

[0016] In the maintenance management system, workers can wear an HMD on their head to check for cracks, etc. Therefore, there is no need to secure a driving space for vehicles to run, so workers can walk to the area to be inspected.

[0017] In the Visual-SLAM of the HMD of the maintenance management system, self-location estimation is performed assuming that map points are stationary objects. Therefore, when Visual-SLAM recognizes a moving object as a map point, it may estimate an incorrect self-location. Also, when the number of detectable map points becomes extremely small, it may become difficult to estimate the self-location.

[0018] During inspection work for maintaining and managing a structure after construction, vehicle restrictions and other measures make it possible to perform the inspection work in an environment with good visibility and few moving objects. However, during the construction of a structure, map points in the distance may not be detected due to the presence of many workers and construction vehicles, or map points for workers and construction vehicles may be assigned. Therefore, during the construction of a structure, self-location estimation may be performed incorrectly or may not be possible.

[0019] When self-location estimation using Visual-SLAM is not possible, a method using an inertial measurement unit (IMU) can be considered. The IMU obtains the angular velocity and acceleration of the measurement target. In this way, the IMU measures the motion state and attitude of the measurement target. However, the measurement value of the IMU contains internal errors. Therefore, when the self-location estimation process is repeated using the measurement value of the IMU, errors may accumulate. As a result, the method using the inertial measurement unit will obtain an incorrect self-location.

[0020] Another type of SLAM is Lidar-SLAM, which uses a laser sensor (distance sensor) called Lidar (Light Detection and Ranging). Lidar-SLAM measures the distance to surrounding objects using Lidar, and obtains a point cloud based on the measured direction and distance to the object. When moving, Lidar-SLAM estimates the current position relative to the past position by comparing the past point cloud with the current point cloud. Infrared sensors, ultrasonic sensors, and radar sensors using radio waves may be used instead of Lidar. Lidar-SLAM requires that the object measured by Lidar is stationary and has a distinctive shape. For this reason, Lidar-SLAM may fail to process self-location estimation if the measurement direction is relatively close and the object is surrounded by moving objects.

[0021] By combining Visual-SLAM and Lidar-SLAM, even if it is difficult to estimate one's position, the other can be used to compensate. For example, in the case of Visual-SLAM, the image becomes dark in a dark place, making it difficult to estimate the position. On the other hand, Lidar-SLAM measures distances with a laser sensor, so it is possible to estimate the position even in a dark place. However, in an environment such as a construction site, the surroundings are surrounded by many workers. Therefore, in an environment such as a construction site, it is difficult to estimate the position even with Visual-SLAM and Lidar-SLAM.

[0022] In recent years, research has been conducted into technology that can see through invisible areas behind walls, for the purpose of saving lives at disaster sites. Conventional radar detects distant objects by emitting radio waves toward the object and receiving the direct reflected waves from the object. In contrast, technology that can see through invisible areas forms multipaths between the object and the object by emitting multiple radio waves toward the object. Technology that can see through invisible areas then uses the multipaths to estimate the position of objects that are hidden behind obstructions and cannot be directly recognized.

[0023] Instead of Lidar-SLAM, it is possible to estimate the self-location using a method that uses multipath sensing to sense parts hidden by obstructions, etc. In this case, the position information of objects hidden by obstructions, etc. is expressed as a radar image of the object's reflection characteristics. Therefore, it is possible to estimate the self-location based on the radar image in the same way as Lidar-SLAM. When calculating reflectors in all directions like Lidar-SLAM, the amount of calculations becomes larger than that of radar using direct reflected waves, depending on the number of multipaths to be considered and the diffusion state at the time of reflection on the object surface. Also, like Lidar-SLAM, it becomes difficult to estimate the self-location when there are many moving objects in the vicinity.

[0024] [First embodiment] A first embodiment of the present disclosure will be described. FIG. 1 is a diagram for explaining an overview of a storage device according to the first embodiment. FIG. 1 shows a schematic diagram of a construction site such as a tunnel where a worker U1 and a work vehicle 1 such as a shovel are working. The storage device 10 is worn on the head of a recorder U2 at a construction site, for example. The recorder U2 wearing the storage device 10 checks the condition of a concrete wall 2 that has been constructed. When the recorder U2 finds an abnormality such as a crack 3 in the concrete wall 2, he operates the storage device 10 to record an image of the found crack 3. The storage device 10 performs a self-location estimation process at the position where the crack 3 is imaged.

[0025] (storage device) Fig. 2 is a block diagram showing a configuration example of the storage device according to the first embodiment. As shown in Fig. 2, the storage device 10 includes a sensor unit 12, a radar unit 14, a plurality of antenna units 16, an operation unit 18, a display unit 20, a storage unit 22, and a control unit 24.

[0026] The sensor unit 12 captures RGB image data of the surroundings of the storage device 10. The area included in the RGB image data captured by the sensor unit 12 may also be called a visible area. The sensor unit 12 acquires depth data in a direction corresponding to the RGB image data. The sensor unit 12 operates under the control of the control unit 24. The sensor unit 12 may include, for example, an RGB-D sensor. The RGB-D sensor is also called an RGB-D camera.

[0027] The radar unit 14 is connected to a plurality of antenna units 16. The radar unit 14 sends out a transmission signal via the antenna unit 16. The radar unit 14 receives, via the antenna unit 16, a reflected signal that is the transmission signal sent out to the outside and reflected by a surrounding object. The radar unit 14 operates under the control of the control unit 24.

[0028] The operation unit 18 accepts various input operations for the storage device 10. The operation unit 18 outputs an operation signal corresponding to the accepted input operation to the control unit 24. The operation unit 18 may include, for example, a touch panel, a button, a switch, a dial, etc. An operator or the like can input a desired operation to the operation unit 18.

[0029] The display unit 20 displays various types of video including characters and images. The display unit 20 displays, for example, RGB image data captured by the sensor unit 12. The display unit 20 displays, for example, key points and map points, which will be described later. The user can visibly display the RGB image data, key points, map points, and the like on the display unit 20 by, for example, operating the operation unit 18. The display unit 20 can be, for example, a display including a liquid crystal display (LCD), an organic electro-luminescence (EL), or the like.

[0030] The storage unit 22 stores information such as the contents of calculations performed by the control unit 24 and programs. The storage unit 22 can be configured with a main storage device such as a random access memory (RAM) and a read only memory (ROM), a solid state drive (SSD), a hard disk drive (HDD), or the like.

[0031] The storage unit 22 may include a map information storage unit 22a and a captured image storage unit 22b.

[0032] The map information storage unit 22a can store key frames and map points, which will be described later. The captured image storage unit 22b can store image data of the surroundings of the storage device 10 captured by the sensor unit 12.

[0033] The control unit 24 controls each unit of the storage device 10. The control unit 24 has, for example, an information processing device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a storage device such as a RAM or a ROM. The control unit 24 executes a program that controls the operation of the storage device 10 according to the present disclosure. The control unit 24 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 24 may be realized by a combination of hardware and software.

[0034] The control unit 24 includes a sensor control unit 30, a radar control unit 32, a key point extraction unit 34, a self-position estimation unit 36, a three-dimensional radar information generation unit 38, an obstruction determination unit 40, a transmission information generation unit 42, and an output control unit 44.

[0035] The sensor control unit 30 controls the sensor unit 12. The sensor control unit 30 controls the sensor unit 12 to capture RGB image data around the storage device 10. The sensor control unit 30 controls the sensor unit 12 to detect the depth in a direction corresponding to the RGB image data. The sensor control unit 30 acquires the RGB image data captured by the sensor unit 12. The sensor control unit 30 acquires depth data relating to the depth of the RGB image data detected by the sensor unit 12.

[0036] The radar control unit 32 controls the radar unit 14. The radar control unit 32 controls the radar unit 14 to send a transmission signal via the antenna unit 16. The radar control unit 32 controls the radar unit 14 to receive, via the antenna unit 16, a reflected signal that is a reflection of the transmission signal from an object around the storage device 10. The radar control unit 32 acquires the reflected signal received by the radar unit 14 as a received signal.

[0037] The key point extraction unit 34 performs a process of extracting key points of the imaging target included in the RGB image data acquired by the sensor control unit 30. A key point is a point that can be a feature point of the imaging target. The key point extraction unit 34 performs the process of extracting key points and extracts key points of the imaging target included in the RGB image data. Details of the key point extraction unit 34 will be described later.

[0038] The self-location estimation unit 36 ​​performs a self-location estimation process of the storage device 10 based on the key points extracted by the key point extraction unit 34. The self-location estimation unit 36 ​​performs the self-location estimation process using, for example, a method based on Visual-SLAM. The self-location estimation unit 36 ​​performs the self-location estimation process and generates a key frame and a map point. The key frame refers to a frame that can be a key in estimating the self-location of the storage device 10. For example, the key frame can be a frame when the field of view changes significantly. The map point refers to a point among the key points that serves as a landmark when performing the self-location estimation process. The self-location estimation unit 36 ​​stores the generated key frame and map point in the map information storage unit 22a of the storage unit 22.

[0039] The three-dimensional radar information generating unit 38 estimates the positions of reflecting objects located around the storage device 10 based on the received signal acquired by the radar control unit 32 from the radar unit 14. The three-dimensional radar information generating unit 38 generates three-dimensional radar information based on the estimation results of the positions of reflecting objects located around the storage device 10.

[0040] The obstructing object determination unit 40 determines whether an object located around the storage device 10 is an obstructing object. An obstructing object is an object that is not a recording target of the storage device 10. The obstructing object determination unit 40 determines an obstructing object based on, for example, distance information related to the distance to the object measured by the sensor unit 12. The obstructing object determination unit 40 identifies an obstructing object based on the RGB image data and depth data acquired by the sensor control unit 30 and the three-dimensional radar information generated by the three-dimensional radar information generation unit 38.

[0041] The transmission information generating unit 42 generates transmission information in which the RGB image data is removed from the obstruction, based on the 3D radar information generated by the 3D radar information generating unit 38 and the obstruction identified by the obstruction determining unit 40. In this embodiment, the area in the RGB image data that is obstructed by the obstruction may be called an invisible area.

[0042] The key point extraction unit 34 performs a process of extracting key points from the transparency information generated by the transparency information generation unit 114. In other words, the key point extraction unit 34 performs a process of extracting key points of an object located in an invisible area.

[0043] The self-position estimation unit 36 ​​performs the self-position estimation process based on the key points extracted from the RGB image data by the key point extraction unit 34 and the key points extracted from the transparent information generated by the transparent information generation unit 114. In other words, the self-position estimation unit 36 ​​performs the self-position estimation process based on the key points in the visible area and the key points in the invisible area.

[0044] The output control unit 44 outputs various types of information. The output control unit 44 outputs various types of information stored in the storage unit 22.

[0045] 3 is a diagram showing an example of the configuration of the storage device according to the first embodiment. As shown in FIG. 3, the storage device 10 may include a helmet 50 and a work vest 60.

[0046] The helmet 50 is worn on the head of the worker. The helmet 50 is fixed to the head of the worker by a chin strap 52 and a headband 54. The helmet 50 has a sensor unit 12, an antenna unit 16, and a display unit 20 on the front. The antenna unit 16 may be configured by arranging a plurality of patch antennas along the left side to the right side of the helmet 50. The antenna unit 16 may be configured by changing the direction of the plurality of patch antennas at a certain angle so as to draw an up-down arc. As a result, the antenna unit 16 can receive a plurality of reflected signals reflected in different directions in the horizontal direction at different patch antennas, and can also receive reflected signals reflected at different angles in the vertical direction. In this embodiment, by arranging the sensor unit 12 and the antenna unit 16 on the helmet 50, it is possible to suppress blocking by the worker's own hand, etc. The display unit 20 displays an image captured by the sensor unit 12. The worker wearing the helmet 50 on his head visually recognizes the image displayed on the display unit 20. The helmet 50 and the work vest 60 are connected by a wire 56. The helmet 50 and the work vest 60 may be connected wirelessly.

[0047] The work vest 60 has, at its lower part, a main body 62 of the storage device 10 including the storage unit 22 and the control unit 24. The operation unit 18 can be disposed on the main body 62. The worker checks the concrete wall via the display unit 20. When the worker finds a crack in the concrete wall, the worker instructs via the operation unit 18 to save an image of the crack and its position. The work vest 60 is configured to be capable of storing a battery and electronic circuits necessary for the storage device 10. By storing the battery and the like in the work vest 60, the weight of the helmet 50 can be reduced, thereby reducing the strain on the worker's neck when wearing the helmet 50.

[0048] (Self-location estimation processing content) An overview of the processing contents of self-location estimation according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an overview of the processing contents of self-location estimation according to the first embodiment.

[0049] The process shown in FIG. 4 may be executed when a user mounts the storage device 10 and starts inspecting cracks in a construction site such as a tunnel.

[0050] The sensor control unit 30 acquires sensing data from the sensor unit 12 (step S10). Specifically, the sensor control unit 30 acquires RGB image data and depth data of the surroundings of the storage device 10 from the sensor unit 12. Then, the process proceeds to step S12.

[0051] The radar control unit 32 acquires sensing data from the radar unit 14 (step S12). Specifically, the radar control unit 32 acquires reflected signals from objects around the storage device 10 from the radar unit 14. Then, the process proceeds to step S14.

[0052] The three-dimensional radar information generating unit 38 generates three-dimensional radar information (step S14). Specifically, the three-dimensional radar information generating unit 38 generates three-dimensional radar information of the surroundings of the storage device 10 based on the reflected signal acquired by the radar unit 14. Then, the process proceeds to step S16.

[0053] The shielding object determination unit 40 identifies a shielding object (step S16). Specifically, the shielding object determination unit 40 identifies a shielding object based on the RGB image data and depth data acquired by the sensor control unit 30 and the three-dimensional radar information generated by the three-dimensional radar information generation unit 38. The shielding object determination unit 40 determines that an object located at a point within a predetermined distance is a shielding object based on distance information relating to the distance to the object measured by the sensor unit 12, for example. The predetermined distance may be set arbitrarily. Then, the process proceeds to step S18.

[0054] The transmission information generating unit 42 generates transmission information (step S18). Specifically, the transmission information generating unit 42 generates transmission information by removing the obstruction from the RGB image data, based on the three-dimensional radar information generated by the three-dimensional radar information generating unit 38 and the obstruction identified by the obstruction determining unit 40. Then, the process proceeds to step S20.

[0055] The key point extraction unit 34 extracts key points (step S20). Specifically, the key point extraction unit 34 extracts key points from the RGB image data and the transparency information. Then, the process proceeds to step S22.

[0056] The sensor control unit 30 determines whether or not an instruction to photograph a crack has been given (step S22). Specifically, the sensor control unit 30 determines that an instruction to photograph a crack has been given when an operation to photograph a crack has been input to the operation unit 18. If it is determined that an instruction to photograph a crack has been given (step S22; Yes), the process proceeds to step S24. If it is not determined that an instruction to photograph a crack has been given (step S22; No), the process proceeds to step S42.

[0057] If the result of the determination in step S22 is Yes, the self-position estimation unit 36 ​​sets the current frame as a key frame (step S24). The frame set as the key frame in step S24 may be called a key frame Fk. Then, the process proceeds to step S26.

[0058] The self-position estimation unit 36 ​​registers the key frame Fk as the shooting position information (step S26), and then proceeds to step S28.

[0059] The key point extraction unit 34 judges whether or not a key point has been extracted within the focus area for photographing the crack (step S28). A key point within the focus area may be called a key point Pf. If it is judged that a key point has been extracted within the focus area (step S28; Yes), the process proceeds to step S30. If it is judged that a key point has not been extracted within the focus area (step S28; No), the process proceeds to step S34.

[0060] If it is determined as Yes in step S28, the self-position estimation unit 36 ​​sets the key point Pf as a map point (step S30). The key point Pf set as the map point in step S30 may be called a map point Mf. Then, the process proceeds to step S32.

[0061] The self-position estimating unit 36 ​​registers the map point Mf as the main value of the crack position information (step S32), and then proceeds to step S44.

[0062] If the result of the determination in step S28 is No, the key point extraction unit 34 detects key points near the focus area (step S34). The key points near the focus area may be called key points Pn. Then, the process proceeds to step S36.

[0063] The self-location estimation unit 36 ​​sets the key point Pn as a map point (step S36). The key point Pn set as a map point in step S36 may be called a map point Mn. Then, the process proceeds to step S38.

[0064] The self-position estimating unit 36 ​​registers the map point Mn as an auxiliary value of the crack position information (step S38), and then proceeds to step S40.

[0065] The self-position estimation unit 36 ​​calculates the difference between the auxiliary value and the median value of the focus area (step S40). The self-position estimation unit 36 ​​stores the difference between the auxiliary value and the crack position information together with the map point Mn as an auxiliary value of the crack position information. Then, the process proceeds to step S44.

[0066] If it is determined that the current frame is a key frame (step S42), the self-position estimation unit 36 ​​determines whether the current frame is a key frame or not (step S42). If it is determined that the current frame is a key frame (step S42; Yes), the process proceeds to step S44. If it is determined that the current frame is not a key frame (step S42; No), the process of FIG. 4 ends.

[0067] After step S32, after step S40, or if it is determined as Yes in step S42, the self-location estimation unit 36 ​​executes a self-location estimation process (step S44). After executing the self-location estimation process, the self-location estimation unit 36 ​​associates various information with each other and stores them as imaging information in the captured image storage unit 22b. The imaging information will be described later.

[0068] Fig. 5 is a diagram for explaining an example of the self-position estimation process according to the first embodiment. Fig. 5 shows image data 70 displayed on the display unit 20. In Fig. 5, the X-axis direction is the horizontal direction, the Y-axis direction is the vertical direction, and the Z-axis direction is the depth direction.

[0069] The image data 70 shows a concrete wall 2. The image data 70 includes a focus area 71 of the sensor unit 12. The image data 70 includes key points 72, 73, 74, 75, 76, 77, and 78 extracted by the key point extraction unit 34. A crack 3 is displayed in the focus area 71. When the user of the storage device 10 instructs the image storage of the crack 3 using the operation unit 18, the key point extraction unit 34 searches for key points in the focus area 71. When the key points 73 and 78 in the focus area 71 are detected, the self-location estimation unit 36 ​​sets the key points 73 and 78 as map points. The self-location estimation unit 36 ​​stores the key points 73 and 78 set as map points in the captured image storage unit 22b as main values ​​of the position information of the crack 3.

[0070] If the key point 73 and the key point 78 do not exist, the key point extraction unit 34 selects the nearby key points 74, 75, and 77 that surround the focus area 71. In this case, the self-position estimation unit 36 ​​sets the key points 74, 75, and 77 as map points. The self-position estimation unit 36 ​​stores the key points 74, 75, and 77 set as map points in the captured image storage unit 22b as auxiliary values ​​for the position information of the crack 3. The self-position estimation unit 36 ​​stores the positional relationships between the key points 74, 75, and 77 and the focus of the subject (crack 3) located at the center of the focus area 71 in the captured image storage unit 22b. The self-position estimation unit 36 ​​acquires, for example, the coordinates of the center point of the focus area 71, the key points 74, 75, and 77 in the X-axis direction, the Y-axis direction, and the Z-axis direction on the image data 70. The coordinates of the center point of focus area 71 are (X0, Y0, Z0). The position coordinates of key point 74 are (X1, Y1, Z1). The position coordinates of key point 75 are (X2, Y2, Z2). The position coordinates of key point 77 are (X3, Y3, Z2). Self-position estimation unit 36 ​​calculates difference values ​​between the coordinates of the center point of focus area 71 and key point 74, key point 75, and key point 77, and stores them in captured image storage unit 22b.

[0071] FIG. 6 is a diagram showing an example of imaging information stored in the captured image storage unit 22b according to the first embodiment.

[0072] As shown in FIG. 6, the imaging information includes items such as "identification information," "date and time information," "person in charge information," "imaging location information," "crack location information," and "image information."

[0073] The "identification information" is information for uniquely identifying an image. The "identification information" may be, for example, an identification number for uniquely identifying an image. In FIG. 6, the "identification information" is conceptually shown as "A1" or the like, but in reality, it indicates specific information.

[0074] "Date and time information" is information indicating the date and time when an image was captured. "Date and time information" may be information including, for example, year, month, day, hour, and minute. In FIG. 6, "date and time information" is conceptually shown as "B1", but in reality, it indicates specific information.

[0075] "Personnel information" is information for uniquely identifying the person who inspected the crack and took the image. "Personnel information" may be, for example, a name or an identification number. In FIG. 6, "Personnel information" is conceptually shown as "C1" or the like, but in reality, it shows specific information.

[0076] The "photographing position information" is information indicating the position of the storage device 10 at the time of photographing. The "photographing position information" may be, for example, a key frame in self-location estimation. In FIG. 6, the "photographing position information" is conceptually shown as "D1", but in reality, it indicates specific information.

[0077] "Crack position information" is information indicating the position of the subject at the time of photographing. "Crack position information" may be, for example, a map point in self-location estimation. In FIG. 6, "crack position information" is conceptually shown as "E1" or the like, but in reality, it indicates specific information.

[0078] "Image information" refers to RGB image data captured by the sensor unit 12. In Fig. 6, "image information" is conceptually shown as "F1" etc., but actually indicates specific information.

[0079] In the first embodiment, a crack in a concrete wall is inspected, and a self-location estimation process is performed to identify the position of the crack based on key points in two areas, a visible area and an invisible area. This makes it possible to improve the accuracy of self-location estimation in the first embodiment.

[0080] [Second embodiment] Next, the self-position estimation process according to the second embodiment will be described.

[0081] FIG. 7 is a flowchart showing the flow of the self-position estimation process according to the second embodiment.

[0082] The sensor control unit 30 acquires the RGB-D data la from the sensor unit 12 (step S50). The sensor control unit 30 acquires lens information including the angle of view and lens characteristics from the sensor unit 12. FIG. 8A and FIG. 8B are diagrams showing an example of RGB-D data according to the second embodiment. As shown in FIG. 8A, the RGB image data shows a state in which a worker 81, a worker 82, and a worker 83 are working with their backs turned in a space surrounded by a concrete wall 84. FIG. 8B shows depth data in FIG. 8A. The brighter the luminance of the depth data, the farther away it is. In FIG. 8B, of the three workers, the worker closest to the storage device 10 is the worker 82. A point 85 on the back of the worker 82 may be a point showing the shortest distance between the storage device 10 and the worker 82.

[0083] The radar control unit 32 controls the radar unit 14 to send a transmission signal from the antenna unit 16 (step S52). The radar control unit 32 sends a transmission signal from the antenna unit 16 so that objects in the visible and invisible regions can be detected. In this case, the radar unit 14 has a field angle set so that objects in the invisible region can be detected.

[0084] Here, we will explain a method for setting the radar angle of view of the radar unit 14. Fig. 9 is a diagram for explaining an example of a method for setting the radar angle of view.

[0085] In this embodiment, the angle of view 104 of the radar unit 14 can be set to an angle that allows sensing of an object in an invisible area that cannot be seen in the RGB image data even when the angle of view 103 of the sensor unit 12 is blocked by a shielding object 120 (e.g., a worker). Specifically, the angle of view 104 can be set to an angle that allows sensing of an invisible area blocked by the shielding object 120 by only the angle obtained by subtracting the angle of view 103 from the angle of view 104.

[0086] The environment shown in FIG. 9 is, for example, a square space with one side being 10 meters long, and there is an obstruction 120 such as a worker in front of the investigator wearing the storage device 10, and the angle of view of the sensor unit 12 is 87°. In this case, it is preferable to set the angle of view 104 of the radar unit 14 so that the entire area beyond the obstruction 120 can be sensed with the minimum number of reflections. In this case, the angle of view 104 of the radar unit 14 can be 112°. This makes it possible to grasp the state of the area beyond the obstruction 120 even when the sensor unit 12 is obstructed by the obstruction 120. In FIG. 9, the space 110 is a space in which there is no reflection of radio waves on walls. The space 111 is a space in which radio waves are reflected once on walls. The space 112 is a space in which radio waves are reflected twice on walls. The space 113 is a space in which radio waves are reflected three times on walls. In this way, by setting the angle of view 104 to 112° or more, it becomes possible to sense the area beyond the obstruction 120 even if the obstruction 120 is present. Note that setting the angle of view 104 to a wide angle increases the required calculation capacity. For this reason, it is preferable to set the angle of view 104 to the minimum necessary, so that the angle can be set to an angle that allows detection of objects in the entire area beyond the obstruction 120.

[0087] In the example shown in FIG. 9, the radar unit 14 of the storage device 10 worn by the user U transmits multiple transmission signals. The transmission signals transmitted by the radar unit 14 of the storage device 10 are reflected by various reflectors including walls at various locations. In this case, the radar unit 14 of the storage device 10 can receive reflected signals reflected by various reflectors and returned from various directions. In other words, the radar unit 14 of the storage device 10 receives multipath reflected signals. In this embodiment, the storage device 10 can measure the position of an object that is located in an invisible area and cannot be measured directly based on the received multiple reflected signals by taking into account the conditions of each reflection of the transmission signal. As a result, in this embodiment, the storage device 10 can extract key points to be used for self-location estimation from an object located in an invisible area.

[0088] Returning to Fig. 7, the three-dimensional radar information generating unit 38 generates three-dimensional radar information based on the reflected signal of the transmission signal reflected outside (step S54). In this embodiment, the three-dimensional radar information indicates the reflection intensity at each voxel in the voxel space. Then, the process proceeds to step S56.

[0089] The three-dimensional radar information generating unit 38 divides the space into a group of regions St using a threshold value T for the reflection intensity value (step S56). The three-dimensional radar information generating unit 38 divides the space into regions according to the reflection intensity. Then, the process proceeds to step S58.

[0090] The obstructing object determination unit 40 calculates the area A of the shortest distance L based on the depth data of the RGB-D data la (step S58). In other words, the obstructing object determination unit 40 calculates the area of ​​the object located at the shortest distance L from the storage device 10. Then, the process proceeds to step S60.

[0091] The obstructing object determination unit 40 calculates the coordinates Pa of the area A in the three-dimensional virtual space based on the sensor information acquired by the sensor control unit 30 from the sensor unit 12 (step S60). Then, the process proceeds to step S62.

[0092] The obstructing object determination unit 40 calculates an area Sa in the vicinity of the coordinate Pa in the area set St (step S62). The obstructing object determination unit 40 determines the area Sa as a transparent object. The obstructing object determination unit 40 determines the area Sa as a transparent object. Then, the process proceeds to step S64.

[0093] The transmission information generating unit 42 executes a process of removing the region Sa from the region set St (step S64). Specifically, the transmission information generating unit 42 determines that the portion located at the shortest distance L is a region where no object exists, and removes the region where the object exists from the voxel space.

[0094] Fig. 10 is a diagram for explaining a method for generating three-dimensional radar information according to the second embodiment. Fig. 10 shows an example of three-dimensional radar information according to the second embodiment. Fig. 10 may be a schematic diagram of a voxel space viewed from above.

[0095] 10, workers 81, 82, 83, 89, and 90 are each represented by an oval. Stones 86 and 87 are represented by a rectangle. Stone 88 is represented by a circle. Angle of view 104 indicates the angle of view at which reflected radio waves can be received. Angle of view 103 indicates the angle of view of sensor unit 12.

[0096] In this embodiment, it is preferable to set the angle of view 104 wider than the angle of view 103. In this embodiment, for example, by setting the angle of view 104 wider, the transmitted radio waves sent from the antenna unit 16 can be received reflected by various reflecting objects in various places. This makes it possible to sense the invisible area blocked by the blocking object even if the sensor unit 12 is blocked by a worker nearby.

[0097] The three-dimensional radar information generating unit 38 calculates the reflection intensity of voxels written in a three-dimensional voxel space based on the received signal. The three-dimensional radar information generating unit 38 sets a threshold value T for the reflection intensity of each voxel. In this embodiment, a voxel having a reflection intensity smaller than the threshold value T is considered to be a space without an object. In this embodiment, a voxel having a reflection intensity equal to or greater than the threshold value T is considered to be a space where an object is present. When a voxel containing an object has an adjacent voxel that contains an object, they are considered to be the same object and form a cluster of objects.

[0098] The three-dimensional radar information generating unit 38 sets a virtual viewpoint 101 and a virtual screen 102 in the voxel space based on the sensor information acquired from the sensor unit 12. The three-dimensional radar information generating unit 38 projects an image seen from the virtual viewpoint 101 through the virtual screen 102 onto the virtual screen 102. The angle of view of the virtual viewpoint 101 may be the same as the angle of view of the sensor unit 12. The transmission information generating unit 42 determines that the worker 82 located near the point 85 is part of the point 85, and deletes the worker 82 from the voxel space. The transmission information generating unit 42 generates transmission information by deleting the worker 82.

[0099] Returning to Fig. 7, the three-dimensional radar information generation unit 38 sets a different color or brightness to each of the elements si of the region group St (step S66). In the example shown in Fig. 10, the three-dimensional radar information generation unit 38 sets a different color or brightness to each object region. The three-dimensional radar information generation unit 38 projects each object region onto the virtual screen 102. By setting a different color or brightness to each object region, the boundaries of each object region become easier to distinguish.

[0100] Here, a process for setting the luminance of an object region according to the second embodiment will be described. Fig. 11 is a flowchart showing the contents of the process for setting the luminance of an object region according to the second embodiment. In the second embodiment, a process for determining the luminance of each object is executed according to the specific overlapping state of adjacent objects.

[0101] The three-dimensional radar information generating unit 38 sets an identification number for each element (step S80). FIG. 12 is a diagram showing an example of three-dimensional radar information according to the second embodiment. As shown in FIG. 12, the three-dimensional radar information includes a background 130, an object 131, an object 132, an object 133, and an object 134 as elements. The three-dimensional radar information generating unit 38 recognizes the background 130, the object 131, the object 132, the object 133, and the object 134 as different object regions into which the background 130, the object 131, the object 132, the object 133, and the object 134 are each divided. For example, the three-dimensional radar information generating unit 38 sets the identification number of the background 130 to b0. For example, the three-dimensional radar information generating unit 38 sets the identification number of the object 131 to b1. For example, the three-dimensional radar information generating unit 38 sets the identification number of the object 132 to b2. For example, the three-dimensional radar information generating unit 38 sets the identification number of the object 133 to b3. The three-dimensional radar information generator 38 sets the identification number of the object 134 to b4, for example. Then, the process proceeds to step S82.

[0102] The three-dimensional radar information generating unit 38 projects the three-dimensional radar information onto the virtual screen (step S82). FIG. 13 is a diagram showing an example of the three-dimensional radar information projected onto the virtual screen according to the second embodiment. FIG. 13 shows a state in which the image seen from the virtual viewpoint 101 in FIG. 12 is projected onto the virtual screen 102. As shown in FIG. 13, a background 130, an object 131, an object 132, and an object 133 are projected onto the virtual screen 102. Usually, on the virtual screen 102, the color of each object may be displayed in a gradation from white to red, etc., depending on the reflection intensity. Therefore, it becomes difficult to distinguish the boundaries between the background 130, the object 131, the object 132, and the object 133. In this case, it becomes difficult for the key point extracting unit 34 to extract key points from the object 131, the object 132, and the object 133. Then, the process proceeds to step S84.

[0103] The three-dimensional radar information generation unit 38 deletes the identification numbers of the elements that are not projected onto the virtual screen (step S84). In the example shown in Fig. 13, the object 134 is not projected onto the virtual screen 102. In this case, the three-dimensional radar information generation unit 38 deletes the identification number b4 of the object 134. Then, the process proceeds to step S86.

[0104] The three-dimensional radar information generation unit 38 registers one group having all the identification numbers as elements in a set G (step S86). One group having all the identification numbers as elements is expressed as {g0}={b0, b1, b2, b3}, for example. In this case, the three-dimensional radar information generation unit 38 registers one group in the set G as G={g0}. Then, the process proceeds to step S88.

[0105] The three-dimensional radar information generation unit 38 sets the brightness levels of the elements (step S88). Specifically, the three-dimensional radar information generation unit 38 sets the brightness levels of the elements corresponding to the identification numbers included in {g0} to their respective reference values. That is, the three-dimensional radar information generation unit 38 sets the background 130, object 131, object 132, and object 133 projected onto the virtual screen 102 to the same brightness level. Then, the process proceeds to step S90.

[0106] The three-dimensional radar information generator 38 determines whether or not a group is registered in the set G (step S90). If it is determined that a group is registered (step S90; Yes), the process proceeds to step S92. If it is determined that a group is not registered (step S90; No), the process proceeds to step S106.

[0107] The three-dimensional radar information generation unit 38 extracts a group from the set G (step S92). Specifically, it extracts {g0} registered in the set G. Then, the process proceeds to step S94.

[0108] The three-dimensional radar information generation unit 38 updates the brightness levels of the elements of the group (step S94). Specifically, the three-dimensional radar information generation unit 38 updates the brightness levels of the elements corresponding to each identification number included in {g0}. That is, the three-dimensional radar information generation unit 38 brightens the brightness levels of the background 130, the object 131, the object 132, and the object 133 projected onto the virtual screen 102 by one step. Then, the process proceeds to step S96.

[0109] The three-dimensional radar information generation unit 38 determines whether the group contains one element (step S96). Specifically, the three-dimensional radar information generation unit 38 determines whether {g0} contains one element. If it is determined that the group contains one element (step S96; Yes), the process proceeds to step S90. If it is not determined that the group contains one element (step S96; No), the process proceeds to step S98. In other words, if the group contains one element, the luminance setting process for that group ends.

[0110] If the result of step S96 is No, the three-dimensional radar information generation unit 38 identifies the identification number of adjacent elements for each element (step S98). The three-dimensional radar information generation unit 38 identifies the identification numbers of elements adjacent to the element with identification number b0 as identification numbers b1, b2, and b3. The three-dimensional radar information generation unit 38 identifies the identification number of the element adjacent to the element with identification number b1 as identification number b0. The three-dimensional radar information generation unit 38 identifies the identification numbers of elements adjacent to the element with identification number b2 as identification numbers b0 and b3. The three-dimensional radar information generation unit 38 identifies the identification numbers of elements adjacent to the element with identification number b3 as identification numbers b0 and b2. Then, the process proceeds to step S100.

[0111] The three-dimensional radar information generation unit 38 determines a common identification number (step S100). The common identification number may be the identification number of the element that is adjacent to the greatest number of other elements among the identification numbers. The three-dimensional radar information generation unit 38 determines the identification number b0 as the common identification number among the identification numbers b0 to b3. Then, the process proceeds to step S102.

[0112] The three-dimensional radar information generating unit 38 generates a group based on the common identification number (step S102). Specifically, the three-dimensional radar information generating unit 38 generates a group including the identification numbers of adjacent elements, excluding the common identification number. In the case of an element with identification number b1, adjacent identification number b0 is excluded, so there is no element adjacent to the element with identification number b1. In this case, the three-dimensional radar information generating unit 38 generates a group with {g1}={b1}. In the case of an element with identification number b2, adjacent identification number 0 is excluded, so the element adjacent to the element with identification number b2 is the element with identification number b3. In this case, the three-dimensional radar information generating unit 38 generates a group with {g2}={b2, b3}. In the case of an element with identification number b3, adjacent identification number 0 is excluded, so the element adjacent to the element with identification number b3 is the element with identification number b2. In this case, if grouped, the result will be the same as identification number b2, so the three-dimensional radar information generating unit 38 does not group. Then, proceed to step S104.

[0113] The three-dimensional radar information generation unit 38 registers the group in the set G (step S104). Specifically, the three-dimensional radar information generation unit 38 registers the group generated in step S102 in the set G. In this case, the three-dimensional radar information generation unit 38 registers G={g1, g2}, {g1}, and {g2} in the set G. Then, the process proceeds to step S90. In this case, the three-dimensional radar information generation unit 38 executes the processes of steps S90 to S104 for {g1} and {g2}. That is, the three-dimensional radar information generation unit 38 determines the color or brightness of each object according to the specific overlapping between the background 130, the object 131, the object 132, and the object 133.

[0114] Return to FIG. 11. The three-dimensional radar information generating unit 38 sets the luminance value of each element (step S106). Specifically, the three-dimensional radar information generating unit 38 sets the luminance values ​​of the background 130, the object 131, the object 132, and the object 133 projected on the virtual screen 102 as shown in FIG. 13. As a result, different luminance values ​​are set for the background 130, the object 131, the object 132, and the object 133. Specifically, the three-dimensional radar information generating unit 38 sets the luminance values ​​of the background 130, the object 131, the object 132, and the object 133 while comparing the luminance of each adjacent object. As a result, the regions of the background 130, the object 131, the object 132, and the object 133 are easily distinguished from each other. Therefore, the key point extracting unit 34 can easily extract key points from the object 131, the object 132, and the object 133. Then, the process of FIG. 11 ends.

[0115] 11 is described as a process for setting the luminance value of each element, but the present disclosure is not limited to this. In the present disclosure, a process for changing the color of each element may be performed instead of the luminance value.

[0116] Return to FIG. 7. The transmission information generating unit 42 generates the transmission information It for the region group St based on the sensor information of the sensor unit 12 (step S68). FIG. 14A and FIG. 14B are diagrams showing an example of the transmission information according to the second embodiment. FIG. 14A shows an example of RGB-D data of the transmission information according to the second embodiment. FIG. 14B shows an example of the depth data of the transmission information according to the second embodiment. In FIG. 14A, a worker 81, a worker 83, a concrete wall 84, a stone 86, and a stone 87 are displayed. In FIG. 14A, the transmission information generating unit 42 has deleted the worker 82. Therefore, it is possible to grasp the stone 86 and the stone 87, which are located in the invisible area in FIG. 14A. FIG. 14B shows the depth data in FIG. 14A. The brightness of the depth data increases as it gets farther away. In the example shown in FIG. 14B, the worker 83, the worker 81, the stone 86, and the stone 87 are farther away in this order. Usually, when a reflective object is projected on the virtual screen 102 based on the reflection intensity in the voxel space, it is expressed by a gradation that changes from white to red according to the reflection intensity. Therefore, since the reflective object is expressed in red on the virtual screen 102, if the reflective objects overlap, it is difficult to distinguish the boundary between the reflective objects. As a result, when extracting key points from the image projected on the virtual screen 102, it is difficult to distinguish the overlapping parts of the worker 81, the worker 83, the concrete wall 84, the stone 86, and the stone 87, and it is difficult to extract the key points. Therefore, in this embodiment, the three-dimensional radar information generating unit 38 sets different colors or brightness for the worker 81, the worker 83, the concrete wall 84, the stone 86, and the stone 87 so that the boundaries of each area are easily distinguished. As a result, in this embodiment, it is easy for the key point extracting unit 34 to extract the key points of the overlapping parts of the stone 86 and the stone 87.

[0117] Returning to Fig. 7, the key point extraction unit 34 extracts key points Ka from the RGB-D data la acquired by the sensor control unit 30 (step S70). That is, the key points Ka may be key points in the visible region. Then, the process proceeds to step S72.

[0118] The key point extraction unit 34 extracts the key point Kt from the transparent information It generated by the transparent information generation unit 42 (step S72). That is, the key point Kt may be a key point in an invisible region. Then, the process proceeds to step S74.

[0119] The self-position estimation unit 36 ​​performs the self-position estimation process based on the key points Ka and Kt (step S74), and then ends the process of FIG.

[0120] In the second embodiment, key points in the visible area are extracted based on the RGB-D data detected by the sensor unit 12. In the second embodiment, an object located within a predetermined distance from the storage device 10 is determined to be an obstruction, and the obstruction is removed to generate an image of the invisible area. In the second embodiment, key points in the invisible area are extracted based on the generated image of the invisible area. Then, in the second embodiment, a self-location estimation process is performed based on the key points in the visible area and the key points in the invisible area. In this way, the second embodiment can improve the accuracy of the self-location estimation process.

[0121] [Third embodiment] FIG. 15 is a flowchart showing the flow of the self-position estimation process according to the third embodiment.

[0122] The processes from step S110 to step S116 are the same as the processes from step S50 to step S66 shown in FIG. 7, respectively, and therefore will not be described.

[0123] The control unit 24 copies the RGB-D data la acquired by the sensor control unit 30 from the sensor unit 12 to generate RGB-D data la2 (step S118). Then, the process proceeds to step S120.

[0124] The three-dimensional radar information generating unit 38 calculates the area A with the shortest distance L based on the depth data of the RGB-D data la2 (step S120). Then, the process proceeds to step S122.

[0125] The processing from step S122 to step S126 is the same as the processing from step S60 to step S64 shown in FIG. 7, and therefore the description thereof will be omitted.

[0126] The control unit 24 determines whether the shortest distance L exceeds the distance Lth, which is a threshold value (step S128). The distance Lth may be determined arbitrarily according to the design. If it is determined that the shortest distance L exceeds the distance Lth (step S128; Yes), the process proceeds to step S130. If it is determined that the shortest distance L does not exceed the distance Lth (step S128; No), the process proceeds to step S140.

[0127] The processing from step S130 to step S138 is the same as the processing from step S66 to step S74 shown in FIG. 7, and therefore a description thereof will be omitted.

[0128] If the result of step S128 is No, the depth of the RGB-D data la2 corresponding to the region Sa is set to the maximum value (step S140). Then, the process proceeds to step S120. In other words, if it is determined that the shortest distance L does not exceed the distance Lth, the processes from step S120 to step S126 are repeated again.

[0129] The RGB-D data according to the third embodiment will be described. The RGB-D data according to the third embodiment is the same as the RGB-D data shown in FIG. 8A, and therefore the description will be omitted. FIG. 16 is a diagram showing depth data of the transmissive information according to the third embodiment. The brighter the luminance of the depth data, the farther the data is. In the example shown in FIG. 16, the workers 82, 83, and 81 are located further back in this order. Of the worker 81's area, the area that is closest to the storage device 10 is point 91. Of the worker 82's area, the area that is closest to the storage device 10 is point 85. Of the worker 83's area, the area that is closest to the storage device 10 is point 92.

[0130] Fig. 17 is a diagram for explaining a method for generating three-dimensional radar information according to the third embodiment. Fig. 17 may be a schematic diagram of a voxel space viewed from above.

[0131] In FIG. 17, a dotted line 93 indicates the position of the threshold distance Lth from the virtual screen 102. The transparency information generating unit 42 sets an object between the dotted line 93 and the virtual screen 102 as a transparency target. As shown in FIG. 16, in the depth data of the RGB-D data, the shortest distance from the storage device 10 is point 85. In this case, the transparency information generating unit 42 determines that the area of ​​the worker 82 located near point 85 is a part of point 85. The transparency information generating unit 42 deletes the area of ​​the worker 82 from the voxel space. Since point 85 is closer to the virtual screen 102 than the dotted line 93, the depth of the area on the depth data of the RGB-D data la2 corresponding to the area of ​​the worker 82 on the voxel space is set to the maximum value.

[0132] Next, with the region of the worker 82 deleted, the shortest distance from the storage device 10 in the depth data of the RGB-D data is point 92. In this case, the transparency information generating unit 42 determines that the region of the worker 83 located near point 92 is part of point 92. The transparency information generating unit 42 deletes the region of the worker 83 from the voxel space. Since point 92 is located closer to the virtual screen 102 than the dotted line 93, the depth of the region in the depth data of the RGB-D data la2 corresponding to the region of the worker 83 in the voxel space is set to the maximum value.

[0133] Next, with the regions of worker 82 and worker 83 deleted, the point 91 is the shortest distance from storage device 10 in the depth data of the RGB-D data. In this case, the transparency information generating unit 42 determines that the area of ​​worker 81 located near point 91 is part of point 91. The transparency information generating unit 42 deletes the area of ​​worker 81 from the voxel space. Since point 91 is located farther from virtual screen 102 than dotted line 93, the object area deletion process is terminated.

[0134] The three-dimensional radar information generating unit 38 generates transmission information in which a different color or brightness is set for each object region remaining in the voxel space. The three-dimensional radar information generating unit 38 projects the generated transmission information onto the virtual screen 102.

[0135] In the third embodiment, the distance Lth set as the threshold value may be set based on, for example, the angle of view of the sensor unit 12, the shoulder width of the worker set as the transparent target, and the occupancy rate of the worker in the image data. For example, when the angle of view of the camera is 87°, the shoulder width of the worker set as the transparent target is 46.5 cm, and the occupancy rate of the worker in the image data is 25%, the distance Lth may be about 1 m.

[0136] 18A and 18B are diagrams illustrating an example of transparency information according to the third embodiment. Fig. 18A illustrates an example of RGB-D data of the transparency information according to the third embodiment. Fig. 18B illustrates an example of depth data of the transparency information according to the third embodiment.

[0137] In FIG. 18A, a concrete wall 84, a stone 86, and a stone 87 are displayed. FIG. 18B shows depth data in FIG. 18A. The brightness of the depth data increases as it gets farther away. In the example shown in FIG. 18B, the stone 86 is farther away, followed by the stone 87. In FIG. 18A, the transmission information generating unit 42 deletes the workers 81, 82, and 83. Therefore, in FIG. 18A, the stone 86 and the stone 87, which were located in the invisible area, can be visually recognized. In the example shown in FIG. 18A, the workers 81, 82, and 83 are deleted, so that the boundaries between the concrete wall 84, the stone 86, and the stone 87 become clearer. Here, the three-dimensional radar information generating unit 38 sets different colors or brightness for the concrete wall 84, the stone 86, and the stone 87 so that the boundaries between the areas are easily distinguished. As a result, in this embodiment, the key point extraction unit 34 can easily extract key points in the areas where the concrete wall 84, the stone material 86, and the stone material 87 overlap each other.

[0138] In the third embodiment, an obstruction can be set based on the distance from the storage device 10. This allows the third embodiment to extract key points in the invisible area based on the transparency information from which a plurality of obstructions have been removed. As a result, the third embodiment can further improve the accuracy of the self-location estimation process.

[0139] [Fourth embodiment] The storage device 10 according to the present disclosure can output the stored crack information relating to the crack in response to an operation by an operator or the like.

[0140] (Output processing) FIG. 19 is a flowchart showing the flow of the crack information output process according to the fourth embodiment.

[0141] The process shown in FIG. 19 is executed when an operation to output crack information is input to the operation unit 18 by an operator or the like.

[0142] The output control unit 44 judges whether or not the captured image storage unit 22b stores image information related to the crack (step S150). If it is judged that the image information is stored (step S150; Yes), the process proceeds to step S152. If it is not judged that the image information is stored (step S150; No), the process proceeds to step S166.

[0143] If the answer to step S150 is Yes, the output control unit 44 acquires spatial coordinates corresponding to the key frame of the shooting position information (step S152), and then proceeds to step S154.

[0144] The output control unit 44 judges whether or not a map point Mf indicating the position of a crack is stored (step S154). If it is judged that the map point Mf is stored (step S154; Yes), the process proceeds to step S156. If it is not judged that the map point Mf is stored (step S154; No), the process proceeds to step S160.

[0145] If the answer to step S154 is Yes, the output control unit 44 acquires the spatial position coordinates Lf corresponding to the map point Mf (step S156). For example, if there are multiple map points Mf, the output control unit 44 acquires a spatial position coordinate group Lf including multiple spatial position coordinates. Then, the process proceeds to step S158.

[0146] The output control unit 44 identifies the spatial position coordinate Lf as the crack position (step S158), and then proceeds to step S164.

[0147] If it is determined as No in step S154, the output control unit 44 acquires the spatial position coordinate Ln corresponding to the map point Mn indicating the auxiliary value of the crack position information (step S160), and then proceeds to step S162.

[0148] The output control unit 44 calculates the crack position based on the spatial position coordinate Ln and the relative position information Dn (step S162). Specifically, the output control unit 44 calculates the coordinates of the center position of the crack according to formula (1). In formula (1), L0 represents the coordinates of the center position of the crack, Ln represents the spatial position coordinate, Dn represents the relative position information, and α represents an arbitrary coefficient. L0 = Ln + αDn (1)

[0149] After step S158 or step S162, the output control unit 44 outputs crack information indicating the position information of the crack (step S164), and then proceeds to step S168.

[0150] If it is determined as No in step S150, the output control unit 44 outputs information indicating that there is no crack (step S166), and then ends the process of FIG.

[0151] After step S164, the output control unit 44 determines whether or not to end the process (step S168). For example, the output control unit 44 determines to end the process when the output process has been performed for all the imaging information stored in the captured image storage unit 22b. If it is determined to end the process (step S168; Yes), the process of FIG. 19 ends. If it is determined not to end the process (step S168; No), the process proceeds to step S152.

[0152] In the fourth embodiment, position information indicating the position of a crack can be easily output. This makes it possible to easily grasp the position of a crack during construction of a tunnel or the like, and therefore allows appropriate repairs or the like to be performed.

[0153] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the contents of these embodiments. The above-mentioned components include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-mentioned components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the above-mentioned embodiments. [Explanation of symbols]

[0154] 10 Storage device 12 Sensor section 14 Radar section 16 Antenna section 18 Control section 20 Display section 22a Map information storage unit 22b Captured image storage unit 24 Control Unit 30 Sensor control unit 32 Radar control unit 34 Keypoint Extraction Unit 36 Self-position estimation part 38 3D radar information generation unit 40 Obstruction determination section 42 Transparent information generation section 44 Output control section

Claims

1. a sensor unit that captures an image of the surroundings of a predetermined position and measures a distance from the predetermined position to a surrounding object; a key point extraction unit that extracts key points from a visible area and an invisible area in the image captured by the sensor unit; a self-location estimation unit that estimates a location based on key points in the visible area and key points in the invisible area; a storage unit that stores information regarding a position estimation result of the self-position estimation unit; a radar unit having an antenna unit capable of transmitting and receiving radio waves; a three-dimensional radar information generating unit that generates three-dimensional radar information of the surroundings based on a signal received by the radar unit via the antenna unit; an obstruction determination unit that determines whether the object is an obstruction based on distance information regarding a distance to the object measured by the sensor unit; a transmission information generating unit that generates transmission information through which the object determined to be the obstruction is transmitted based on the three-dimensional radar information generated by the three-dimensional radar information generating unit and a determination result from the obstruction determining unit; Equipped with The key point extraction unit extracts key points of the invisible region based on the transparency information generated by the transparency information generation unit. storage device.

2. 2. The storage device according to claim 1, The storage device, wherein the obstructing object determination unit determines that the object located at a point within a predetermined distance is the obstructing object based on distance information regarding the distance to the object measured by the sensor unit.

3. 3. The storage device according to claim 1, The storage device, wherein when the three-dimensional radar information includes a plurality of the objects, the three-dimensional radar information generating unit sets the plurality of the objects to different colors or brightnesses.

4. 4. The storage device according to claim 3, The storage device, wherein the three-dimensional radar information generating unit determines a color or a brightness of each of the plurality of objects in the three-dimensional radar information according to an overlapping state between the objects.

5. capturing an image of a surrounding area of ​​a predetermined position and measuring a distance from the predetermined position to a surrounding object; Extracting key points from visible and invisible areas in the captured image; performing a position estimation based on the key points of the visible area and the key points of the invisible area; storing information relating to a result of the position estimation in a storage unit; generating surrounding three-dimensional radar information based on the received signal; determining whether the object is an obstruction based on distance information relating to the measured distance to the object; generating transmission information of the object determined to be an obstruction based on the generated three-dimensional radar information and a determination result of whether the object is an obstruction; Including, In the step of extracting key points, key points in the invisible region are extracted based on the generated transparency information. Memorization method.

Images