Location determination device, location determination method, and location determination system
By predicting and selectively processing frames with dynamic content, the method addresses the challenge of misidentification in SLAM, reducing resource usage and enhancing map precision for dynamic objects, ensuring accurate position specification.
Patent Information
- Application Number
- JP2021094448
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-06-04
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-06-04
AI Technical Summary
Existing SLAM methods struggle with accurately identifying and excluding dynamic objects from environmental maps, leading to reduced map precision due to misidentification of moving objects as static, and require significant computing resources, especially when computing power is limited.
A method that predicts frames likely to contain dynamic content and performs dynamic object segmentation only on those frames, using selective segmentation and classification to generate high-precision environmental maps while minimizing resource usage.
This approach reduces computing resource requirements and enhances the accuracy of environmental maps by accurately distinguishing between static and dynamic objects, thereby improving the precision of position specification for moving bodies.
Smart Images

Figure 0007577608000001 
Figure 0007577608000002 
Figure 0007577608000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a location determination device, a location determination method, and a location determination system. [Background technology]
[0002] In recent years, with the advancement of IT, many sensors have been installed in society and an extremely large amount of data has been accumulated. In this context, various methods for utilizing accumulated video data are being considered. In particular, as the amount of video content, such as photos, videos, and images, increases, there is a demand for a means to generate an environmental map from this video content and accurately identify the position of a moving object moving within the environment.
[0003] So-called SLAM (Simultaneous Localization and Mapping) has been known as a method for simultaneously estimating the self-position and creating an environmental map. In SLAM, a mobile object equipped with a sensor such as Lidar senses the surrounding environment while moving and creates a two-dimensional or three-dimensional environmental map. By utilizing SLAM, a mobile object can create an environmental map in an unknown environment, and can use the map information to avoid obstacles and perform specific tasks.
[0004] However, in SLAM, only static objects in the video (e.g., non-moving objects such as buildings, furniture, and walls) are used to create the environmental map. When dynamic objects (e.g., moving objects such as humans, animals, and cars) are present in the video, the static objects and background necessary for map creation are partially obscured, reducing the accuracy of the map. Therefore, in order to generate a highly accurate environmental map using SLAM, it is important to identify dynamic objects in the video and exclude them from the SLAM process.
[0005] Several methods have been proposed to identify dynamic objects in video and exclude them from SLAM processing. For example, Chinese Patent Publication No. 111950561 (Patent Document 1) describes a technique that "The present invention discloses a method for removing semantic SLAM dynamic points based on semantic segmentation, including the following steps: 1) perform semantic segmentation on image frames using a PSPNet semantic segmentation network, and perform classification according to pixel segmentation results. Extract ORB features of the new frame I, calculate descriptors, and then match the new frame I with F. 3) Set a threshold M. 4) Classify feature points according to the motion e of dynamic points and static points. 5) Associate semantic labels with the classified static point set to build a local semantic map. 6) Repeat steps 1) to 5) to update the local semantic map until all local maps are created and a global map is obtained. In this method, dynamic feature points in the environment that have a significant impact on the accuracy of the system can be eliminated, and static feature points can be used to build an octomap that can be interpreted with semantic information with high accuracy." [Prior art documents] [Patent documents]
[0006] [Patent Document 1] China Patent Application Publication No. 111950561 Summary of the Invention [Problem to be solved by the invention]
[0007] The method described in Patent Document 1 can identify and eliminate dynamic objects by performing semantic segmentation on all frames that make up the target video. In this way, after eliminating the dynamic objects, a SLAM process can be performed using static objects to generate an environmental map.
[0008] However, the means described in Patent Document 1 requires a huge amount of computing resources because semantic segmentation is performed on all frames of the input video. For this reason, it is difficult to introduce the means described in Patent Document 1 when available computing resources are limited, such as in a smart device. In addition, in many cases, dynamic objects are present only in some frames of the input video, so performing semantic segmentation on frames in which no dynamic objects exist is wasteful. Furthermore, in the means described in Patent Document 1, the determination of whether an object is dynamic or static is made based only on the amount of movement of the object. For this reason, when a movable object such as a human, animal, or automobile (hereinafter, a "dynamic class object") temporarily stops moving, the object may be mistaken for a static object. If an environmental map is subsequently generated in SLAM processing using a dynamic class object that has been mistaken for a static object, the accuracy of the environmental map is significantly impaired, resulting in a decrease in precision.
[0009] Therefore, the present disclosure aims to provide a position identification means capable of generating a highly accurate environmental map and identifying the position of a moving object while reducing the amount of required computing resources by predicting frames (hereinafter referred to as "dynamic frames") that are likely to contain dynamic content in an input video and performing dynamic object segmentation processing only on these dynamic frames. [Means for solving the problem]
[0010] In order to solve the above problems, one representative positioning device disclosed herein is a positioning device that locates a moving object moving in an arbitrary environment, and includes: an image feature extraction unit that analyzes an input video showing the environment and extracts features for each of a plurality of frames included in the input video; a selective division determination unit that predicts a dynamic frame including dynamic content from the input video based on any one of an amount of movement between the plurality of frames included in the input video, a moving speed of a shooting unit that acquired the input video, and a ratio of pixels of an object corresponding to a dynamic class in the frame; a dynamic object division unit that performs a dynamic object division process on the dynamic frame and generates image semantic information indicating whether each pixel in the dynamic frame belongs to an object corresponding to a dynamic class or an object corresponding to a static class; a feature classification unit that classifies each feature as a dynamic feature or a static feature based on the semantic information of the image and the features; and a SLAM unit that generates a 3D map that stereoscopically shows the environment and first moving object position information indicating the position and orientation of the moving object based on the static features and the semantic information of the image. Effect of the Invention
[0011] According to the present disclosure, by predicting dynamic frames in an input video that are likely to contain dynamic content and performing dynamic object segmentation processing only on these dynamic frames, it is possible to provide a position identification means that can generate a highly accurate environmental map and identify the position of a moving object while reducing the amount of computing resources required. Other objects, configurations and effects will become apparent from the following description of the preferred embodiment of the invention. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 illustrates a computer system for implementing an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a position identifying device according to the first embodiment of the present disclosure. [Diagram 3] FIG. 3 is a block diagram illustrating a flow of a map generation process in the position identifying device according to the first embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating a flow of a position identifying process in the position identifying device according to the first embodiment of the present disclosure. [Diagram 5] FIG. 5 is a block diagram illustrating a flow of a dynamic content determination process in the position identifying device according to the first embodiment of the present disclosure. [Figure 6] FIG. 6 is a block diagram illustrating a flow of the feature classification process in the position identifying device according to the first embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating an example of a hardware configuration of a position identifying device according to the second embodiment of the present disclosure. [Figure 8] FIG. 8 is a block diagram illustrating a flow of a moving object position information correction process in a position identifying device according to the second embodiment of the present disclosure. [Figure 9] FIG. 9 is a block diagram illustrating a flow of the GPS association process and the map reading process in the position identifying device according to the second embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating an example of a hardware configuration of a position identifying device according to a third embodiment of the present disclosure. [Figure 11] FIG. 11 is a block diagram illustrating a flow of a risk management process in a position identifying device according to a third embodiment of the present disclosure. [Figure 12] FIG. 12 is a block diagram illustrating the functions of an interface unit according to an embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram illustrating an example of the use of skip mask parameters according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals.
[0014] First, referring to Fig. 1, a computer system 100 for implementing the embodiments of the present disclosure will be described. The mechanisms and devices of the various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 100 include one or more processors 102, memory 104, terminal interface 112, storage interface 113, I / O (input / output) device interface 114, and network interface 115. These components may be interconnected via a memory bus 106, an I / O bus 108, a bus interface unit 109, and an I / O bus interface unit 110.
[0015] Computer system 100 may include one or more general purpose programmable central processing units (CPUs) 102A and 102B, collectively referred to as processors 102. In some embodiments, computer system 100 may include multiple processors, and in other embodiments, computer system 100 may be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include an on-board cache.
[0016] In one embodiment, memory 104 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. Memory 104 may store all or a portion of the programs, modules, and data structures that implement the functions described herein. For example, memory 104 may store location application 150. In one embodiment, location application 150 may include instructions or descriptions that execute on processor 102 the functions described below.
[0017] In some embodiments, location application 150 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, location application 150 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with bus interface unit 109, processor 102, or other hardware of computer system 100.
[0018] Computer system 100 may include a bus interface unit 109 that provides communication between processor 102, memory 104, display system 124, and I / O bus interface unit 110. I / O bus interface unit 110 may couple to an I / O bus 108 for transferring data to and from various I / O units. I / O bus interface unit 110 may communicate via I / O bus 108 with multiple I / O interface units 112, 113, 114, and 115, also known as I / O processors (IOPs) or I / O adapters (IOAs).
[0019] Display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to display device 126. Computer system 100 may also include one or more sensors or other devices configured to collect data and provide the data to processor 102.
[0020] For example, computer system 100 may include biometric sensors to collect heart rate data, stress level data, etc., environmental sensors to collect humidity data, temperature data, pressure data, etc., and motion sensors to collect acceleration data, movement data, etc. Other types of sensors may also be used. Display system 124 may be connected to a display device 126, such as a standalone display screen, a television, a tablet, or a handheld device.
[0021] The I / O interface unit provides the ability to communicate with various storage or I / O devices. For example, the terminal interface unit 112 may be attached to user I / O devices 116, such as user output devices, such as a video display, a television with speakers, and user input devices, such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing device. A user may use a user interface to enter input data or instructions to the user I / O devices 116 and the computer system 100, and receive output data from the computer system 100, by manipulating the user input devices. The user interface may be displayed on a display, played through speakers, or printed via a printer, for example, via the user I / O devices 116.
[0022] Storage interface 113 may be attached to one or more disk drives or direct access storage device 117 (usually a magnetic disk drive storage device, but may also be an array of disk drives or other storage devices configured to appear as a single disk drive). In one embodiment, storage device 117 may be implemented as any secondary storage device. Contents of memory 104 may be stored in storage device 117 and retrieved from storage device 117 as needed. I / O device interface 114 may provide an interface to other I / O devices such as printers, fax machines, etc. Network interface 115 may provide a communications path to allow computer system 100 and other devices to communicate with each other. This communications path may be, for example, network 130.
[0023] In some embodiments, computer system 100 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 100 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device. EXAMPLES
[0024] Next, a hardware configuration of the position identifying device according to the first embodiment of the present disclosure will be described with reference to FIG.
[0025] FIG. 2 is a diagram illustrating an example of a hardware configuration of the position identifying device 200 according to the first embodiment of the present disclosure. The location determination device 200 shown in FIG. 2 is a device configured to execute the location determination means according to an embodiment of the present disclosure, and may be realized as, for example, a smartphone, a tablet, a laptop, a dedicated circuit, etc., but is not limited thereto. Moreover, the position identification device 200 according to the embodiment of the present disclosure is provided in a moving object 275 and is assumed to move in an environment together with the moving object 275. The moving object 275 is not particularly limited as long as it is capable of autonomous movement, such as a human being, an animal, or a robot with an automatic driving function. Furthermore, the form of the position identification device 200 may be determined according to the moving object 275. For example, when the moving object 275 is a human being, the position identification device 200 may be realized as a personal terminal such as a smartphone, a tablet, or a notebook computer, and when the moving object 275 is a robot, the position identification device 200 may be a dedicated circuit included in the robot.
[0026] Additionally, in this disclosure, "environment" refers to the environment for which a map is generated and the location of a moving object 275 is identified, and may include any environment, such as a city, a park, a factory, an office building, a station, an airport, etc. In addition, in this disclosure, an "environmental map" refers to a map that describes information about the surrounding environment, such as buildings, structures, fixed objects, elevation differences, and the presence or absence of obstacles that affect movement.
[0027] As shown in FIG. 2 , the localization device 200 includes an imaging unit 205, a processor 210, a memory 215, an image feature extraction unit 220, a selective segmentation decision unit 225, a dynamic object segmentation unit 230, a feature classification unit 235, a SLAM unit 240, a localization unit 245, and an interface unit 250.
[0028] The photographing unit 205 is a functional unit for acquiring an image showing the environment (hereinafter referred to as an "input image"). For example, the photographing unit 205 may be a laser range scanner (range sensor, Lidar), a camera (RGB camera, infrared camera, 360-degree camera), an encoder, or the like. As an example, when the position identification device 200 is realized using a smartphone, the photographing unit 205 may be a camera built into the smartphone. In addition, it is desirable that the photographing unit 205 is disposed so that the position identification device 200 can photograph an image of the environment while moving. The input video may be data including a number of frames, such as a moving image showing an environment, where a frame is each still image that constitutes the input video. 2 shows an example of a configuration in which the image capturing unit 205 is mounted on the position identifying device 200, but the present disclosure is not limited to this. For example, the position identifying device 200 itself may be installed as a remote server that is geographically separated from the environment, and the image capturing unit 205 may be provided on a moving object 275 that moves within the environment, forming a system configuration.
[0029] 2 shows an example of the configuration of the position identification device 200 including a single image capture unit 205, but the present disclosure is not limited thereto. For example, the position identification device 200 may include a plurality of image capture units 205 arranged to capture images at different angles, or a plurality of different types of image capture units 205 (such as an RGB camera and Lidar). As an example, the position identification device 200 may be configured to include a first image capture unit that captures the front of the moving body 275 and a second image capture unit that captures the rear, or a 360-degree camera that can capture 360 degrees around the moving body 275 and Lidar. In this way, by using a camera that captures not only the front of the moving body 275 but also the surroundings, it is possible to generate a 3D map with higher accuracy.
[0030] The processor 210 is an arithmetic device that executes instructions for implementing the functions of the functional units of the position identification device 200, and is substantially similar to the processor 102 shown in FIG. 1, for example, so a description thereof will be omitted here.
[0031] Memory 215 is a random access semiconductor memory, storage device, or storage medium (either volatile or non-volatile) that stores data and program instructions used by each functional unit of location identification device 200, and is substantially similar to memory 104 shown in FIG. 1, for example, so a description thereof will be omitted here.
[0032] The image feature extraction unit 220 is a functional unit for analyzing the input video acquired by the shooting unit 205 and extracting features for each of a plurality of frames included in the input video. The features here are information (vectors, etc.) that represent the features of an object in a frame, and may include, for example, color, size, shape, etc. The image feature extraction unit 220 here may use any existing means, such as SIFT (Scale Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF).
[0033] The selective division determination unit 225 is a functional unit that predicts frames that are likely to contain dynamic content as dynamic frames to be subjected to dynamic object division processing (so-called semantic segmentation) from among multiple frames included in the input video acquired by the shooting unit 205. More specifically, the selective division determination unit 225 predicts dynamic frames that contain dynamic content from the input video based on any one of the amount of movement between multiple frames included in the input video, the movement speed of the shooting unit 205 that acquired the input video, and the ratio of pixels of objects corresponding to dynamic classes in the frames. In this case, dynamic content refers to image information corresponding to dynamic objects that can move (e.g., humans, animals, automobiles, etc.), whereas static content refers to image information corresponding to static objects that cannot move (e.g., buildings, furniture, walls, etc.). The details of the processing by the selective division determination unit 225 will be described later, and therefore will not be described here.
[0034] The dynamic object splitter 230 is a functional unit that performs dynamic object splitting processing on the dynamic frame predicted by the selective split determination unit 225, and generates image semantic information for each pixel in the dynamic frame indicating whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class. The dynamic object segmentation unit 230 here may use any existing means, such as Mask R-CNN (Mask Region-Based Convolutional Neural Network), PSPNet (Pyramid Scene Parsing Network), and BiSeNet (Bilateral Segmentation Network).
[0035] The feature classification unit 235 is a functional unit that classifies each feature of a frame of an input video as a dynamic feature (i.e., a feature belonging to a dynamic object) or a static feature (i.e., a feature belonging to a static object) based on the semantic information of the image generated by the dynamic object division unit 230 and the features extracted by the image feature extraction unit 220. In one embodiment, the feature classifier 235 may classify a feature as a static feature or a dynamic feature based on the amount of movement of the feature between frames, where the expression "amount of movement between frames" refers to the difference between multiple frames (e.g., difference in pixel information) that occurs due to the movement of the moving object 275.
[0036] The SLAM unit 240 is a functional unit that generates a three-dimensional map that stereoscopically shows the environment based on the static features classified by the feature classification unit 235 and the semantic information of the images generated by the dynamic object division unit 230. The SLAM unit 240 here may use existing Simultaneous Localization and Mapping means, and is not particularly limited in this disclosure.
[0037] The position identification unit 245 is a functional unit for identifying the position of the moving body 275 in the environment based on the 3D map of the environment generated by the SLAM unit 240, the static features classified by the feature classification unit 235, and the semantic information of the image generated by the dynamic object division unit 230.
[0038] The interface unit 250 is a functional unit for providing various information to the user of the position identification device 200 and accepting input from the user. As an example, the interface unit 250 may be a touch screen. The interface unit 250 may display a three-dimensional map generated by the SLAM unit 240, display the position of the mobile object 275 on the three-dimensional map, accept input of a destination of the mobile object, and display a recommended route to the destination.
[0039] According to the position identification device 200 configured as described above, by predicting dynamic frames in an input video that are likely to contain dynamic content and performing dynamic object segmentation processing only on these dynamic frames, it is possible to provide a position identification means that can generate a highly accurate environmental map and identify the position of a moving object while reducing the amount of required computing resources.
[0040] Next, a flow of a map generation process in the position identifying device according to the first embodiment of the present disclosure will be described with reference to FIG.
[0041] 3 is a block diagram showing a flow of a map generation process 300 in the position identification device 200 according to the first embodiment of the present disclosure. The map generation process 300 shown in FIG. 3 is a process for generating a 3D map of an environment in which a moving body (e.g., the moving body 275 shown in FIG. 2) equipped with the position identification device 200 moves, and is executed by each functional unit of the position identification device 200.
[0042] First, an image capturing unit (for example, image capturing unit 205 in position identification device 200 shown in FIG. 2, not shown in FIG. 3) acquires input video 305. As described above, this input video 305 is an image showing the environment in which the mobile object moves, and is composed of a plurality of frames. As an example, the image capturing unit may be disposed on the mobile object in a state in which it can capture the environment, and the input video 305 may be acquired by capturing images of the environment while the mobile object moves. An input video 305 acquired by an imaging unit is input to an image feature extraction unit 220 and a selective division determination unit 225 .
[0043] The image feature extraction unit 220 receives the input video 305 captured by the image capture unit, analyzes the input video 305, and extracts features of each frame. As described above, the features here are information (e.g., vectors) that represent the features of an object in a frame, and may include, for example, color, size, shape, and the like. The features extracted by the image feature extraction unit 220 are input to the selective division decision unit 225 .
[0044] The selective division determination unit 225 predicts, from among a plurality of frames included in the input video 305 acquired by the imaging unit, a frame that is likely to include dynamic content as a dynamic frame to be subjected to dynamic object division processing. More specifically, the selective division determination unit 225 inputs the input video 305, the features of each frame extracted by the image feature extraction unit 220, and moving object position history information 310 (described later) (note that, when the map generation processing 300 is performed for the first time, the moving object position history information 310 does not exist), and predicts a dynamic frame including dynamic content from the input video 305 based on any one of the movement amount between a plurality of frames included in the input video 305, the movement speed of the imaging unit that acquired the input video 305, and the ratio of pixels of an object corresponding to a dynamic class in the frame. Dynamic frames, which are predicted to contain dynamic content, are input to a dynamic object segmenter 230 , and non-dynamic frames, which are predicted to not contain dynamic content, are input to a feature classifier 235 . The details of the processing by the selective division determination unit 225 will be described later, and therefore will not be described here.
[0045] The dynamic object division unit 230 performs dynamic object division processing on the dynamic frame predicted by the selective division determination unit 225, and generates, for each pixel in the dynamic frame, image semantic information 320 indicating whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class. Here, the dynamic object division unit 230 may determine whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class based on dynamic class information 315 indicating information of an object considered to be a dynamic class. The image semantic information 320 generated by the dynamic object segmentation unit 230 is input to the SLAM unit 240 and the feature classification unit 235.
[0046] The feature classifier 235 classifies each feature of a frame of the input video 305 as a dynamic feature (i.e., a feature belonging to a dynamic object) or a static feature (i.e., a feature belonging to a static object) based on the image semantic information 320 generated by the dynamic object segmenter 230 and the features extracted by the image feature extractor 220. In one embodiment, the feature classifier 235 may classify a feature as a static feature or a dynamic feature based on the amount of movement of the feature between frames. Features classified as static features are input to the SLAM unit 240, while features classified as dynamic features are treated as outliers and excluded from SLAM unit 240 processing.
[0047] The SLAM unit 240 generates a three-dimensional map 325 and first moving object position information 330 based on the static features classified by the feature classification unit 235 and the semantic information 320 of the image generated by the dynamic object division unit 230. The 3D map 325 here is a 3D map corresponding to the environment shown in the input video 305. By performing the map generation process 300 shown in Fig. 3 based on input video showing different parts of the same environment and combining each of the generated 3D maps, a 3D map showing the entire environment can be generated.
[0048] The first moving object position information 330 here is information indicating the position (e.g., geographic coordinates, etc.) and orientation (north, east, west, south, etc.) of the moving object. In reality, this first moving object position information 330 is information indicating the position and orientation of the imaging unit derived from the input video acquired by the imaging unit, but since the imaging unit is provided in the moving object, it is also information indicating the position and orientation of the moving object. Moreover, it is possible to generate moving object position history information 310 that indicates the movement of the moving object / image capture unit in a time series from the change in the position of the moving object / image capture unit in a plurality of different frames, which is indicated in the first moving object position information 330. As will be described later, this moving object position history information 310 may be used when predicting dynamic frames. The 3D map 325 generated by the SLAM unit 240 and the first moving object position information 330 are input to the interface unit 250.
[0049] The interface unit 250 outputs the 3D map 325 and the first moving object position information 330 generated by the SLAM unit 240. As an example, the interface unit 250 may display the 3D map 325 and the position of the moving object on the 3D map 325. In addition, in one embodiment, the interface unit 250 may receive an input of a destination of the moving object and then display a recommended route to the destination.
[0050] According to the map generation process 300 described above, dynamic frames that are likely to contain dynamic content are predicted, and dynamic object segmentation processing is performed only on these dynamic frames, thereby making it possible to construct a highly accurate environmental map while reducing the amount of computing resources required.
[0051] Next, a flow of a position identifying process in the position identifying device according to the first embodiment of the present disclosure will be described with reference to FIG.
[0052] Fig. 4 is a diagram showing a flow of a position identification process 400 in the position identification device 200 according to the first embodiment of the present disclosure. The position identification process 400 shown in Fig. 4 is a process for identifying a position of a moving body (human, robot, etc.) equipped with the position identification device 200 in an arbitrary environment by using a 3D map 325 generated in advance by the map generation process 300 described with reference to Fig. 3, and is executed by each functional unit of the position identification device 200. The position identification process 400 shown in FIG. 4 has the same basic processing flow as the map generation process 300 described with reference to FIG. 3, except that a position identification unit 245 that identifies the position of a moving object based on a pre-created 3D map 325 is used instead of the SLAM unit that generates the 3D map 325. Therefore, repeated explanations will be omitted.
[0053] The position identification unit 245 inputs and analyzes the static features classified by the feature classification unit 235, the semantic information 320 of the image generated by the dynamic object division unit 230, and the three-dimensional map 325 generated by the map generation process 300 described with reference to Figure 3 and stored in the memory 215, to generate first moving object position information 330 indicating the position (e.g., geographical coordinates, etc.) and direction (north, east, west, south, etc.) of the moving object on the three-dimensional map 325. This first mobile object location information 330 is transferred to the interface unit 250 and output by the interface unit 250 .
[0054] According to the position determination process 400 described above, by predicting dynamic frames that are likely to contain dynamic content and performing dynamic object segmentation processing only on these dynamic frames, it is possible to accurately determine the position of a moving object in an environment while reducing the amount of computing resources required.
[0055] Next, a flow of a dynamic content determination process in the position identifying device according to the first embodiment of the present disclosure will be described with reference to FIG.
[0056] As described above, in the present disclosure, from among the multiple frames that make up a video frame, dynamic object segmentation processing is performed only on dynamic frames that are predicted to contain dynamic content, and non-dynamic frames that are predicted not to contain dynamic content are excluded from the dynamic object segmentation processing, thereby making it possible to accurately identify the position of a moving object in an environment while reducing the amount of computing resources required. More specifically, the determination of dynamic content according to the embodiment of the present disclosure is based on a skip mask parameter 550. The skip mask parameter 550 is a parameter that specifies the number of frames in the input video to be excluded from the dynamic object segmentation process. The skip mask parameter 550 can be set and updated based on the amount of movement between multiple frames included in the input video, the movement speed of the camera that captured the input video, and the ratio of pixels of objects corresponding to dynamic classes in the frames. By using this skip mask parameter 550, frames that are unlikely to contain dynamic content are skipped (i.e., excluded from the dynamic object segmentation process), and only frames that are likely to contain dynamic content are subjected to the dynamic object segmentation process, thereby significantly saving computing resources compared to performing dynamic object segmentation on all frames of the input video.
[0057] Fig. 5 is a block diagram showing a flow of a dynamic content determination process 500 in the position identification device according to the first embodiment of the present disclosure. The dynamic content determination process 500 shown in Fig. 5 is a process for predicting dynamic frames that are likely to include dynamic content using skip mask parameters 550, and is mainly executed by an image feature extraction unit, a selective division determination unit, and a dynamic object division unit.
[0058] First, the input video 305 is input to the selective division decision unit 225 and the image feature extraction unit 220 .
[0059] As described above, the image feature extraction unit 220 receives the input video 305 captured by the image capture unit, analyzes the input video 305, and extracts features of each frame. These features are used in estimating the amount of movement, which will be described later.
[0060] The selective division determination unit 225 performs a modulo operation for each frame constituting the input video 305 using an image index indicating the order of the frame and the skip mask parameter 550, and if the remainder is "0" (i.e., if the remainder of the division of the image index by the skip mask parameter 550 is 0), it sets the frame as a dynamic frame and inputs it to the dynamic object division unit 230. On the other hand, if the remainder of the division operation is not "0", it performs a movement amount estimation, which will be described later, on the frame. Here, by using this division operation, a frame is input to the dynamic object division unit for each number of frames set in the skip mask parameters, and for the remaining frames, the movement amount estimation described below is performed. It should be noted that for the first frame in the input video, the skip mask parameter 550 has not yet been set, so this first frame is directly input to the dynamic object divider 230 .
[0061] Next, for frames where the remainder of the division operation is "0", the selective division determination unit 225 performs a movement amount estimation based on the features extracted by the image feature extraction unit 220 and information on the previous frame 510 in the input video of the target frame (hereinafter referred to as the "previous frame"). Here, the selective division determination unit 225 estimates the amount of movement e between frames by comparing the features of the target frame with the previous frame 510. As described above, the amount of movement e between frames here means the difference between multiple frames (e.g., the difference in pixel information) that occurs due to the movement of a moving object. If the estimated amount of movement e exceeds a predetermined movement amount standard M, the target frame is considered to be a dynamic frame and is input to the dynamic object division unit 230. On the other hand, if the estimated movement amount e is equal to or less than the predetermined movement amount reference M, the target frame is regarded as a non-dynamic frame and is input to a feature classifier (not shown in FIG. 5). In this way, by estimating the amount of movement for non-dynamic frames, even if dynamic content is present among the frames that were excluded by setting the skip mask parameters, the dynamic content can be detected and the frames containing this dynamic content can be reclassified as dynamic frames. In this way, frames containing dynamic content can be excluded from the SLAM section with high accuracy, making it possible to generate highly accurate 3D maps.
[0062] As described above, the dynamic object splitter 230 uses the dynamic class information 315 to perform dynamic object splitting processing on the dynamic frame predicted by the selective split determination unit 225, and generates image semantic information 320 for each pixel in the dynamic frame indicating whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class.
[0063] Next, the selective split determination unit 225 sets the skip mask parameters 550 based on the ratio of dynamic content in the frame using the semantic information 320 of the image generated by the dynamic object splitter 230. More specifically, the selective split determination unit 225 calculates the ratio of the number of pixels of the object corresponding to the dynamic class to the total number of pixels in the frame, and determines whether the calculated ratio satisfies a predetermined criterion regarding the ratio of dynamic content. If the calculated ratio does not meet the predetermined dynamic content percentage criteria, the skip mask parameter 550 is not set / updated. On the other hand, if the calculated ratio meets the predetermined dynamic content percentage criteria, the skip mask parameter 550 is set to "1".
[0064] Also, the selective division determination unit 225 may set the skip mask parameter 550 using the moving object position history information 310. As described above, this moving object position history information 310 is information indicating the position of the moving object / photographing unit in a plurality of different frames. The selective division determination unit 225 estimates the moving speed of the moving object / photographing unit based on the change in the position of the moving object / photographing unit indicated in this moving object position history information 310. Thereafter, the selective division determination unit 225 may set / update the skip mask parameter 550 based on the estimated moving speed. Here, as a general rule, it is desirable that the selective division determination unit 225 sets the skip mask parameter 550 to a larger value the faster the moving speed of the moving object / photographing unit is, and sets the skip mask parameter 550 to a smaller value the slower the moving speed of the moving object / photographing unit is. For example, in one embodiment, the selective division determination unit 225 may classify the calculated moving speed of the moving object / image capture unit into one of categories such as "Fast", "Medium", "Slow", etc. based on a preset threshold value, and set the skip mask parameter 550 according to the category. As an example, when the selective division determination unit 225 determines the moving speed of the moving object / image capture unit to be "Fast", it may set the skip mask parameter 550 to "2", when the selective division determination unit determines the moving speed of the moving object / image capture unit to be "Medium", it may set the skip mask parameter 550 to "5", and when the selective division determination unit determines the moving speed of the moving object / image capture unit to be "Slow", it may set the skip mask parameter 550 to "10".
[0065] According to the dynamic content determination process 500 described above, the skip mask parameter 550 can be set and updated based on the amount of movement between multiple frames included in the input video, the movement speed of the image capture unit that captured the input video, and the ratio of pixels of objects corresponding to dynamic classes in the frames. Furthermore, by processing the input video using this skip mask parameter 550, it is possible to determine dynamic frames that are likely to contain dynamic content (for example, frames with a lot of movement between frames, frames with a fast movement speed of the moving body / image capture unit, frames with many pixels corresponding to dynamic objects, etc.) while skipping non-dynamic frames that do not contain dynamic content (for example, frames with little movement between frames, frames with a slow movement speed of the moving body / image capture unit, frames with few pixels corresponding to dynamic objects, etc.).
[0066] Next, a flow of a feature classification process in the position identifying device according to the first embodiment of the present disclosure will be described with reference to FIG.
[0067] 6 is a block diagram showing a flow of a feature classification process 600 in the localization device according to the first embodiment of the present disclosure. The feature classification process 600 shown in FIG. 6 is a process for classifying the features of an input video as static features or dynamic features, and is mainly executed by the image feature extraction unit 220, the selective division determination unit 225, and the feature classification unit 235.
[0068] First, the input video 305 is input to the selective division decision unit 225 and the image feature extraction unit 220 .
[0069] As described above, the image feature extraction unit 220 receives the input video 305 captured by the image capture unit, analyzes the input video 305, and extracts features 615 for each frame. These features 615 are used for estimating the amount of movement and for classifying features, which will be described later.
[0070] The selective division determination unit 225 performs a modulo operation for each frame constituting the input video 305 using an image index indicating the order of the frame and the skip mask parameter 550, and if the remainder is "0" (i.e., if the remainder of the division of the image index by the skip mask parameter 550 is 0), it sets the frame as a dynamic frame and inputs it to the dynamic object division unit 230. On the other hand, if the remainder of the division operation is not "0", feature classification is performed on the features of the frame. In addition, in the case of the first frame in the input video, the skip mask parameter 550 has not yet been set, so this first frame is directly input to the dynamic object divider 230 .
[0071] As described above, the dynamic object splitter 230 uses the pre-created dynamic class information 315 to perform dynamic object splitting processing on the dynamic frame predicted by the selective split determination unit 225, and generates image semantic information 320 for each pixel in the dynamic frame indicating whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class. This image semantic information 320 is input to the feature classifier 235 .
[0072] The feature classification unit 235 classifies each feature of the input video frame as a dynamic feature 622 or a candidate static feature 624 based on the image semantic information 320 generated by the dynamic object segmentation unit 230 and the features 615 extracted by the image feature extraction unit 220. Here, features classified as dynamic features 622 are treated as outliers and excluded from the processing of the SLAM unit 240. On the other hand, for features classified as static feature candidates 624, the feature classification unit 235 may perform further feature classification based on the above-mentioned inter-frame movement amount e. More specifically, the feature classification unit 235 compares the movement amount e estimated based on the comparison of features between the target frame and the previous frame 510 with a predetermined movement amount criterion M, and classifies the feature as a dynamic feature 622 if the movement amount e is equal to or less than the predetermined movement amount criterion M, and classifies the feature as a static feature 632 if the movement amount e is equal to or less than the predetermined movement amount criterion M.
[0073] According to the feature classification process 600 described above, the features of the input video can be classified into either static features 631 or dynamic features 622. Furthermore, as described above, by performing feature classification doubly based on the image semantic information 320 and the amount of movement between frames, it is possible to identify dynamic content present in non-dynamic frames skipped by setting the skip mask parameters, and to correctly classify objects that have been misidentified based on the image semantic information 320 (objects that do not move even though they correspond to a dynamic class, and objects that have moved even though they correspond to a static class). EXAMPLES
[0074] Next, a hardware configuration of a position identifying device according to a second embodiment of the present disclosure will be described with reference to FIG.
[0075] FIG. 7 is a diagram illustrating an example of a hardware configuration of a position identifying device 700 according to the second embodiment of the present disclosure. As described above, the SLAM unit 240 and the position identification unit 245 according to the first embodiment of the present disclosure generate first moving object position information indicating the position and orientation of the moving object 275 in a 3D map of an arbitrary environment. The position identification device 700 according to the second embodiment of the present disclosure can generate corrected position information indicating the position and orientation of the moving object 275 more accurately based on the first moving object position information and the second movement position information derived from the movement sensor data related to the movement of the moving object. In addition, the position determining device 700 according to the second embodiment of the present disclosure has substantially the same configuration as the position determining device 200 according to the first embodiment, except that the position determining device 700 includes an inertial measurement unit 705, a GPS unit 710, a filter unit 715, and a position information correction unit 720 in addition to the configuration of the position determining device 200 according to the first embodiment described above. Therefore, repeated explanations will be omitted below.
[0076] The inertial measurement unit 705 is a functional unit for detecting the angles (or angular velocities) and acceleration of three axes governing the motion as moving sensor data. The inertial measurement unit 705 can calculate the three-dimensional angular velocity and acceleration of the moving body 275 using the acquired three-axis gyro and three-direction accelerometer.
[0077] The GPS (Global Positioning System) unit 710 is a functional unit for acquiring geographical position information (coordinates, etc.) of the mobile object 275 based on signals from GPS satellites.
[0078] The filter unit 715 is a functional unit for removing noise from the movement sensor data acquired by the inertial measurement unit 705 by performing a predetermined filtering process on the movement sensor data.
[0079] The position information correction unit 720 is a functional unit for generating corrected moving body position information that more accurately indicates the position and orientation of the moving body 275 based on the first moving body position information generated by the above-mentioned SLAM unit 240 or the position identification unit 245 and the second moving body position information derived from the moving sensor data acquired by the inertial measurement unit 705.
[0080] The position determining device 700 according to the second embodiment of the present disclosure, which is configured as described above, can determine the position of the moving object 275 more accurately than the position determining device 200 according to the first embodiment, and can link the 3D map generated by the SLAM unit 240 to GPS data.
[0081] Next, a flow of a moving object position information correction process in the position identifying device according to the second embodiment of the present disclosure will be described with reference to FIG.
[0082] 8 is a block diagram showing a flow of a moving object position information correction process 800 in a position identification device according to a second embodiment of the present disclosure. The moving object position information correction process 800 shown in FIG. 8 is a process for generating corrected moving object position information that more accurately indicates the position and orientation of the moving object 275 based on the first moving object position information generated by the SLAM unit 240 or the position identification unit 245 and the second moving object position information generated by the inertial measurement unit. The moving object position information correction process 800 is executed by the SLAM unit 240, the position identification unit 245, the inertial measurement unit 705, the filter unit 715, and the position information correction unit 720.
[0083] First, an input image 305 acquired by the imaging unit is input to the SLAM unit 240 or the position identification unit 245 of the position identification device. The SLAM unit 240 or the position identification unit 245 performs a map generation process 300 shown in Fig. 3 or a position identification process 400 shown in Fig. 4 to generate first moving object position information 330 indicating the position (e.g., geographic coordinates, etc.) and direction (north, east, west, south, etc.) of the moving object.
[0084] Furthermore, the inertial measurement unit 705 acquires motion sensor data 805 indicating three-dimensional angular velocity and acceleration of the mobile body equipped with a position specifying device including the inertial measurement unit 705 while the mobile body is moving.
[0085] Next, the filter unit 715 performs a predetermined filtering process on the movement sensor data 805 acquired by the inertial measurement unit 705 to remove noise from the sensor data and generate filtered data 810. Here, any existing noise removal means may be used to remove noise from the mobile sensor data 805, and is not particularly limited in this disclosure.
[0086] Next, the position information correction unit 720 generates second moving object position information 830 indicating the position and orientation of the moving object based on the filtered data 810 generated by the filter unit 715. Note that, in reality, this second moving object position information 830 is information indicating the position and orientation of the inertial measurement unit 705 derived from the mobile sensor data acquired by the inertial measurement unit 705. And, since the inertial measurement unit 705 is provided in the moving object, this second moving object position information 830 is also information indicating the position and orientation of the moving object. Here, to derive the position and orientation of the moving object from the filtered data 810, the position information correction unit 720 may use any existing means, and is not particularly limited in this disclosure.
[0087] Next, the position information correction unit 720 generates corrected moving object position information 850 based on the above-mentioned first moving object position information 330 and second moving object position information 830, which can indicate the position and orientation of the moving object more accurately than the first moving object position information 330 or the second moving object position information 830. The corrected moving object position information 850 generated here may be output, for example, via an interface unit of the position specifying device (for example, the interface unit 250 of the position specifying device 700 shown in FIG. 7).
[0088] As described above, in the position determination device according to the second embodiment of the present disclosure, corrected moving body position information that more accurately indicates the position and orientation of the moving body can be generated based on the first moving body position information 330 generated by the SLAM unit 240 or the position determination unit 245 and the second moving body position information 830 generated by the inertial measurement unit 705.
[0089] Next, a flow of a GPS association process and a map reading process in the position identifying device according to the second embodiment of the present disclosure will be described with reference to FIG.
[0090] As described above, the position identification device 700 according to the second embodiment of the present disclosure includes the GPS unit 710. By using the GPS unit 710 to associate GPS data with the three-dimensional map generated by the SLAM unit 240, it becomes possible to identify not only the position of the moving object in a specific environment but also the global position of the moving object.
[0091] FIG. 9 is a block diagram showing the flow of a GPS association process 900 and a map reading process 950 in a position identifying device according to the second embodiment of the present disclosure.
[0092] First, a description will be given of the GPS association process 900. The GPS association process 900 is a process for storing GPS data in the map database 925 in association with a three-dimensional map generated by the SLAM unit 240.
[0093] First, the SLAM unit 240 generates a 3D map 325 corresponding to the input video 305, for example according to a map generation process 300 shown in FIG.
[0094] After the 3D map 325 is created, the GPS unit 710 acquires GPS data 905 indicating the global position of the moving object. Then, the GPS unit 710 associates the acquired GPS data 905 with the 3D map 325. Here, as a means for associating the GPS data 905 with the 3D map 325, the GPS unit 710 may add a geotag indicating the geographical coordinates (latitude, longitude) of a specific point (such as a park, an office building, or a station) in the 3D map 325. Thereafter, the GPS unit 710 stores the 3D map 325 associated with the GPS data 905 in a map database (hereinafter referred to as "map DB") 925 managed by a server device (such as a cloud server) installed in a geographically distant location via a communication network 915 such as the Internet.
[0095] Next, the map reading process 950 will be described. The map reading process 950 is a process for reading a 3D map associated with GPS data. This map reading process 950 may be automatically performed when a mobile object enters an environment in which a 3D map 325 associated with GPS data 905 has already been created and stored in a map DB 925.
[0096] The GPS unit 710 of the mobile body transmits a map request including geographical information (latitude, longitude, etc.) 955 of the position identification device 700 (i.e., the mobile body) to the map DB 925 via the communication network 915. Note that in this disclosure, a "map request" is information requesting a pre-created environmental map (e.g., a three-dimensional map stored in the map DB 925 by the above-mentioned GPS association process 900) that corresponds to the current geographical information of the mobile body. Then, the map DB 925 searches for a 3D map 325 corresponding to the geographical information 955 included in the map request received from the position identification device 700, and transmits the searched 3D map 325 via the communication network 915. After that, the position identification device 700 that has received the 3D map 325 uses the 3D map 325 to perform the position identification process 400 described with reference to Fig. 4, for example, to identify the position of the mobile object equipped with the position identification device 700 on the entire earth.
[0097] According to the GPS association process 900 and map reading process 950 described above, by storing GPS data in association with a 3D map of the environment generated by the SLAM unit 240, a moving object that later enters the environment can obtain and use the previously generated 3D map from the map DB 925, thereby eliminating the need to generate a 3D map from scratch and enabling the location of the moving object on the entire earth to be identified. EXAMPLES
[0098] Next, a hardware configuration of a position identifying device according to a third embodiment of the present disclosure will be described with reference to FIG.
[0099] FIG. 10 is a diagram illustrating an example of a hardware configuration of a position identifying device 1000 according to a third embodiment of the present disclosure. As described above, the positioning device according to the first and second embodiments of the present disclosure can generate mobile object position information indicating the position and orientation of the mobile object 275 in a three-dimensional map of an arbitrary environment. The positioning device 1000 according to the third embodiment of the present disclosure can notify the mobile object 275 of a risk that may affect the mobile object 275 based on the mobile object position information and risk position information indicating a risk in the environment. In addition, the location identification device 1000 according to the third embodiment of the present disclosure has a substantially similar configuration to the location identification device 700 according to the second embodiment, except that the location identification device 1000 includes a risk management unit 1010 and a notification unit 1020 in addition to the configuration of the location identification device 700 according to the second embodiment described above. Therefore, repeated explanations will be omitted below.
[0100] The risk management unit 1010 is a functional unit for determining risks that may affect the mobile unit 275, based on corrected mobile unit position information indicating the position and orientation of the mobile unit 275 in the environment and risk position information indicating the positions of risk candidates. The risk here means, for example, things that have the danger of preventing the movement of the mobile unit 275 or causing damage or obstruction to the mobile unit 275, and may include, for example, large machinery, harmful substances, areas with poor security, etc.
[0101] The notification unit 1020 is a functional unit for generating and outputting a risk notification to inform the mobile unit 275 of the risk determined by the risk management unit 1010. In one embodiment, the notification may include the location (e.g., geographic coordinates) of the risk determined to have the potential to affect the mobile unit 275, instructions for avoiding the risk, and the like.
[0102] As described above, the positioning device 1000 according to the third embodiment of the present disclosure can determine the risk present in the environment in which the moving object 275 moves, and can notify the moving object 275 of the risk. This can improve the safety of the moving object 275 in the environment.
[0103] Next, a flow of a risk management process in the position identifying device according to the third embodiment of the present disclosure will be described with reference to FIG.
[0104] 11 is a block diagram showing a flow of a risk management process 1100 in a position identification device according to a third embodiment of the present disclosure. The risk management process 1100 is a process for determining a risk that may affect a moving object and outputting a notification of the risk, and is executed by a risk management unit 1010 and a notification unit 1020 in the position identification device 1000 shown in FIG.
[0105] First, the risk management unit 1010 acquires the 3D map 325 generated by the SLAM unit 240, the corrected mobile object position information 850 generated by the position information correction unit 720, and risk position information 1115 indicating the position of a risk in the environment. The risk position information 1115 here may be stored in a server device accessible via a communication network such as the Internet. As an example, when the target environment is a factory, the risk position information 1115 may be stored in a server in the factory. In addition, although a case where the corrected moving object position information 850 is used will be described as an example here, the present disclosure is not limited thereto, and the first moving object position information or the second moving object position information may be used instead of the corrected moving object position information 850. However, since the corrected moving object position information 850 has higher accuracy than the first moving object position information and the second moving object position information, it is desirable to use the corrected moving object position information 850.
[0106] Next, the risk management unit 1010 determines risks that may affect the mobile object based on the acquired 3D map 325, the corrected mobile object position information 850, and the risk position information 1115. Here, the risk management unit 1010 may determine, for example, from among the risk candidates shown in the risk position information 1115, risk candidates whose distance to the current position of the mobile object is less than a predetermined distance standard, as risks that may affect the mobile object.
[0107] Thereafter, the notification unit 1020 generates and outputs a risk notification to inform the mobile unit of the risk determined by the risk management unit 1010. As described above, this notification may include the location (e.g., geographic coordinates) of the risk determined to have the potential to affect the mobile unit, instructions for avoiding the risk, and the like. In one embodiment, the notification unit 1020 may output the created risk notification to an interface unit (for example, the interface unit 250 in the position identification device 1000 shown in FIG. 3) and display it to the user via a touch screen or the like.
[0108] According to the risk management process 1100 described above, it is possible to determine risks present in the environment in which the mobile body moves and to notify the mobile body of the risks, thereby improving the safety of the mobile body in the environment.
[0109] Next, the function of the interface unit according to the embodiment of the present disclosure will be described with reference to FIG.
[0110] As described above, the interface unit according to the embodiment of the present disclosure is a functional unit for providing various information to a user (for example, when the mobile object is a human user having a personal terminal equipped with a position identification device) and accepting input from the user. As an example, the interface unit 250 may be a touch screen. The interface unit 250 may display a three-dimensional map generated by the SLAM unit, display the position of the mobile object on the three-dimensional map (i.e., the user's position), accept input of a destination of the mobile object, and display a recommended route to the destination. 12 is a block diagram showing functions of an interface unit 250 according to an embodiment of the present disclosure. As shown in FIG 12, the interface unit 250 includes a 3D map display function 1205, a mobile object position display function 1210, a destination setting function 1215, a route guidance function 1220, a map search function 1225, a map loading function 1230, and a risk display function 1235.
[0111] The 3D map display function 1205 is a function for displaying a 3D map generated by the map generation process 300 described with reference to Fig. 3 or a 3D map acquired from a map database. Here, the interface unit 250 may display the 3D map to the user via, for example, a touch screen or the like.
[0112] The mobile object position display function 1210 is a function for displaying the position of a mobile object on a three-dimensional map. Here, the mobile object position display function 1210 may display the position of a mobile object on a three-dimensional map according to any one of the above-mentioned first mobile object position information, second mobile object position information, and corrected mobile object position information, but since the corrected mobile object position information has higher accuracy than the first mobile object position information and the second mobile object position information, it is preferable to use the corrected mobile object position information.
[0113] The destination setting function 1215 is a function for setting a destination of a mobile object (i.e., a user) based on an input from a user (e.g., when the mobile object is a human user having a personal terminal equipped with a positioning device). For example, here, the human user may set a destination by selecting a desired destination with a finger or the like on a three-dimensional map displayed on a touch screen or the like by the interface unit 250.
[0114] The route guidance function 1220 is a function for determining and displaying a recommended route for guiding a moving object to a destination set by the destination setting function 1215. Here, the route guidance 1220 may determine a route according to, for example, an existing automatic driving function or the like.
[0115] The map search function 1225 is a function for obtaining a created 3D map corresponding to the geographical information of a moving object obtained by the above-mentioned GPS unit or the like from a map DB (e.g., map DB 925 shown in FIG. 9) stored in the server based on the geographical information of the moving object.
[0116] The map reading function 1230 is a function for reading a created 3D map acquired from the map DB by the map search function 1225 and displaying it via the interface unit.
[0117] The risk display function 1235 is a function for displaying the risk determined and notified by the risk management process 1100 described with reference to Fig. 11. Here, information on the risk that may affect the mobile object may be displayed on a three-dimensional map displayed via the interface unit.
[0118] According to the interface unit described above, a human user who is moving within the environment can easily check various pieces of information output to the position specifying device.
[0119] Next, with reference to FIG. 13, an example of using the skip mask parameters according to an embodiment of the present disclosure will be described.
[0120] As described above, the dynamic content determination process according to the embodiment of the present disclosure (for example, the dynamic content determination process 500 shown in FIG. 5) is performed based on a skip mask parameter. This skip mask parameter is a parameter that specifies the number of frames in the input video to be excluded from the dynamic object segmentation process. By using this skip mask parameter, frames that are unlikely to contain dynamic content are skipped (i.e., excluded from the dynamic object segmentation process), and only frames that are likely to contain dynamic content are subject to the dynamic object segmentation process, which can significantly save computing resources compared to performing the dynamic object segmentation process on all frames of the input video. The skip mask parameters are set and updated based on the amount of movement between multiple frames included in the input video, the movement speed of the image capture unit that captured the input video, and the ratio of pixels of objects corresponding to the dynamic class in the frames. 13 is a diagram showing an example of using skip mask parameters according to an embodiment of the present disclosure. For convenience of explanation, the following example of use describes a case where the skip mask parameters are set based on the ratio of dynamic content of a frame, but as described above, the present disclosure is not limited to this, and the skip mask parameters may be set according to not only the ratio of dynamic content but also the moving speed of the moving object / image capture unit, the movement between frames, etc.
[0121] First, an input video 1300 including at least a plurality of frames such as frame 1301, frame 1307, frame 1308, frame 1309, and frame 1310 is acquired and input to a localization device according to an embodiment of the present disclosure, and is subjected to a dynamic content determination process. As described above, when the dynamic content determination process is performed for the first time, the skip mask parameters have not yet been set, so the selective segmentation determination unit directly inputs frame 1301, which is the first frame, to the dynamic object segmentation unit.
[0122] As a result of analyzing frame 1301 by the dynamic object segmentation unit, it is determined that there is no dynamic object such as a human being, and therefore the dynamic content ratio is low and does not satisfy the dynamic content ratio criterion. Therefore, the skip mask parameter is set to a predetermined value such as "5 frames". Here, the value of the skip mask parameter may be set by, for example, an administrator of the position identification device, or may be determined based on input video used in the past for map generation.
[0123] Next, the selective split decision unit skips five frames (that is, frames 1302 to 1306 not shown in FIG. 13) from frame 1301 in the input video 1300 in accordance with the skip mask parameters, and inputs frame 1307 to the dynamic object splitter.
[0124] Frame 1307 is analyzed by the dynamic object segmentation unit and is determined to have a high dynamic content ratio and meet the dynamic content ratio criterion because a dynamic object (human) is present. Therefore, the skip mask parameter is updated to, for example, "0 frame".
[0125] Next, the selective split decision unit inputs the next frame, frame 1308, from frame 1307 in the input video 1300 to the dynamic object splitter, without skipping any frames, in accordance with the skip mask parameters.
[0126] Frame 1308 is analyzed by the dynamic object segmentation unit and is determined to have a high dynamic content ratio and satisfy the dynamic content ratio criterion because it contains a dynamic object (human) just like frame 1307. Therefore, the skip mask parameters are not changed and remain at "0 frames."
[0127] Frame 1309, like frames 1307 and 1308, is determined to have a high dynamic content ratio and meet the dynamic content ratio criterion due to the presence of a dynamic object (human), so the skip mask parameter is not changed and remains at "0 frames." In this manner, the selective split decision unit inputs each frame to the dynamic object splitter without skipping any frames until the frame no longer meets the dynamic content proportion criterion.
[0128] On the other hand, when frame 1310 is input to the dynamic object segmentation unit and analyzed, it is determined that frame 1310 does not contain a dynamic object such as a human, has a low dynamic content ratio, and does not satisfy the dynamic content ratio criterion. Therefore, the skip mask parameter is set to a predetermined value such as "5 frames", and the selective segmentation determination unit skips 5 frames from frame 1310 in the input video 1300 according to the skip mask parameter, and then inputs the next frame (not shown in FIG. 13) to the dynamic object segmentation unit.
[0129] In this way, the skip mask parameters can be used to determine dynamic frames that are likely to contain dynamic content while skipping non-dynamic frames that do not contain dynamic content, thereby allowing dynamic frames to be excluded from the dynamic object segmentation process while still inputting the non-dynamic frames to the skip unit for map generation, thereby saving significant computing resources compared to performing dynamic object segmentation on all frames of the input video.
[0130] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of the present invention. [Explanation of symbols]
[0131] 200, 700, 1000 locators 205 Filming Department 210 Processor 215 Memory 220 Image feature extraction unit 225 Selective division decision unit 230 Dynamic Object Splitter 235 Feature Classification Department 240 SLAM section 245 Location identification part 250 Interface section 705 Inertial Measurement Unit 710 GPS section 715 Filter section 720 Location information correction section 1010 Risk Management Department 1020 Notification Department
Claims
1. A position determining device for determining the position of a moving object in an arbitrary environment, an image feature extraction unit that analyzes an input video showing the environment and extracts features for each of a plurality of frames included in the input video; a selective division determination unit that predicts a dynamic frame including dynamic content from the input video based on any one of an amount of movement between the plurality of frames included in the input video, a movement speed of a shooting unit that captured the input video, and a ratio of pixels of an object corresponding to a dynamic class in the frame; and a dynamic object segmentation unit that performs a dynamic object segmentation process on the dynamic frame and generates, for each pixel in the dynamic frame, semantic information of the image that indicates whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class; a feature classification unit that classifies each feature as a dynamic feature or a static feature based on semantic information of the image and the features; a SLAM unit that generates a three-dimensional map that stereoscopically illustrates the environment and first moving object position information that indicates a position and an orientation of the moving object based on the static features and semantic information of the image; A location determination device comprising:
2. The selective division determination unit is estimating a moving speed of the imaging unit based on a change in a position of the imaging unit that acquired the input video in the plurality of frames; setting a skip mask parameter that specifies a number of frames to be excluded from the dynamic object segmentation process in the input video based on a moving speed of the image capture unit; processing the input video based on the skip mask parameters to skip non-dynamic frames that do not include dynamic content and to predict the dynamic frames; 2. The position determining device according to claim 1 .
3. The selective division determination unit is determining an amount of motion between the frames by comparing the features extracted from the non-motion frame with features of a frame immediately preceding the non-motion frame in the input video; If the amount of movement between the frames satisfies a predetermined movement amount criterion, the non-dynamic frame is determined to be a dynamic frame.
3. The position determining device according to claim 2 .
4. The feature classification unit is determining the amount of movement of the static features between the frames; inputting, into the SLAM unit, only those static features whose inter-frame movement amount does not satisfy a predetermined movement amount criterion; 4. The position determining device according to claim 3 .
5. The selective division determination unit is According to the semantic information of the image, if the ratio of pixels of the object corresponding to the dynamic class in the target frame included in the input video meets a predetermined dynamic content ratio criterion; setting the skip mask parameter to a lower value; 5. The position determining device according to claim 4.
6. The location identification device is an inertial measurement unit for acquiring movement sensor data relating to the movement of the moving object; a filter unit that performs a filtering process on the movement sensor data to generate filtered movement sensor data; generating second mobile object position information indicative of a position and an orientation of the mobile object based on the filtered motion sensor data; a position information correction unit that generates corrected moving object position information that more accurately indicates a position and an orientation of the moving object based on the first moving object position information and the second moving object position information; The location determination device of claim 1 further comprising:
7. The location identification device is Risk Management Department, A notification unit; Further comprising: The risk management department determining a risk that may affect the moving object based on the corrected moving object position information and risk position information indicating the position of a risk candidate; generating and outputting a risk notification indicating instructions for avoiding said risk; 7. The position determining device according to claim 6,
8. The location identification device is acquiring GPS data indicative of the location of said mobile object on the globe; a GPS unit that stores the GPS data in a map database in association with the three-dimensional map; 8. The position determining device according to claim 7,
9. A method for determining a location of a moving object in an arbitrary environment, comprising: analyzing an input video showing the environment and extracting features for each of a plurality of frames included in the input video; setting a skip mask parameter that specifies the number of frames to be excluded from a dynamic object segmentation process based on any one of an amount of movement between the plurality of frames included in the input video, a moving speed of a shooting unit that captured the input video, and a ratio of pixels of an object corresponding to a dynamic class in the frames; processing the input video based on the skip mask parameters to skip non-dynamic frames that do not contain dynamic content and to predict dynamic frames that contain dynamic content; performing a dynamic object segmentation process on the dynamic frame to generate, for each pixel in the dynamic frame, image semantic information indicating whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class; classifying each feature as a dynamic feature or a static feature based on semantic information of the image and the features; determining the amount of movement of the static features between the frames; generating a three-dimensional map showing the environment in a three-dimensional manner and first moving object position information showing a position and an orientation of the moving object based on the static features, among the static features, a feature where the inter-frame movement amount does not satisfy a predetermined movement amount criterion and on semantic information of the image; A location determination method comprising:
10. A location determination system for determining the location of a moving object in an arbitrary environment, comprising: a position determination device installed on the mobile object and configured to generate a three-dimensional map showing the environment; a map database for storing the three-dimensional map is connected via a communication network; The location identification device is an image feature extraction unit that analyzes an input video showing the environment and extracts features for each of a plurality of frames included in the input video; a selective division determination unit that predicts a dynamic frame including dynamic content from the input video based on any one of an amount of movement between the plurality of frames included in the input video, a movement speed of a shooting unit that captured the input video, and a ratio of pixels of an object corresponding to a dynamic class in the frame; and a dynamic object segmentation unit that performs a dynamic object segmentation process on the dynamic frame and generates, for each pixel in the dynamic frame, semantic information of the image that indicates whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class; a feature classification unit that classifies each feature as a dynamic feature or a static feature based on semantic information of the image and the features; a SLAM unit that generates the three-dimensional map that stereoscopically shows the environment and first mobile object position information that shows a position and an orientation of the mobile object based on the static features and semantic information of the image, and stores the three-dimensional map in the map database via the communication network; A location determination system comprising:
11. The location identification device is acquiring GPS data indicative of the location of said mobile object on the globe; a GPS unit that stores the GPS data in the map database in association with the three-dimensional map; 11. The location system according to claim 10,
12. The GPS unit includes: transmitting a map request to said map database via said communications network, said map request including GPS data indicating a global position of said mobile unit; receiving from the map database a three-dimensional map corresponding to the global location of the moving object; 12. The location system according to claim 11,
Citation Information
Patent Citations
Method for eliminating semantic SLAM dynamic points based on semantic segmentation
CN111950561A
