Synchronous positioning and mapping method for establishing drivable area in map

Abstract

The invention provides a synchronous positioning and mapping method for establishing a drivable area in a map, and the method comprises the steps: obtaining a to-be-processed image sequence, and sequentially detecting the drivable areas of images in the image sequence; after a drivable area is extracted from the current image, determining a corresponding three-dimensional approximate drivable area of the current image in the constructed point cloud map based on the drivable area of the current image; if it is determined that the current image is a key frame based on a preset selection rule, fusing the three-dimensional approximate drivable area corresponding to the current image with a corresponding local fusion result of the previous frame of image in the point cloud map to obtain a corresponding local fusion result of the current image in the point cloud map; and fusing the local fusion result corresponding to the current image in the point cloud map with the global three-dimensional drivable area established in the point cloud map to obtain an updated global three-dimensional drivable area. The method can adapt to various indoor and outdoor scenes and is low in cost.

Description

Technical Field

[0001] The present invention relates to the technical field of synchronous positioning and mapping, and in particular to a synchronous positioning and mapping method for establishing a drivable area in a map. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) technology solves the problem of robots positioning and mapping in unknown environments at the same time, and has been widely used in VR, unmanned driving, robotics and other fields. The maps generated by SLAM are also often used to guide the autonomous movement of intelligent bodies such as robots. Depending on the sensors used, SLAM can be divided into two categories: vision-based and lidar-based methods. Vision-based SLAM methods have become the focus of research due to the maturity of camera technology, low price, long life and rich information collection capabilities. However, the map established by the traditional visual SLAM algorithm is a sparse point cloud map, which is conducive to the establishment of large-scale scene maps and can be used for positioning. However, due to the lack of drivable area information, it is impossible to effectively guide the robot's actions and provide path planning assistance for the robot. In order to effectively guide robots and other intelligent bodies to automatically plan routes, avoid obstacles and finally reach the designated destination, it is of great significance to establish a drivable area in the sparse point cloud map.

[0003] The visual semantic SLAM system uses deep learning neural network images to perform semantic segmentation, maps semantic information to the map, and semantically classifies the point cloud in the map. LiDAR SLAM detects the drivable area through road edge extraction and plane assumption, and divides the point cloud into road surface and non-road surface. Fusion of visual and LiDAR data for drivable area detection improves the accuracy of the detection results. LiDAR-based SLAM systems generally use grid maps to mark drivable areas, divide the map space into grids, and mark the grid occupancy probability to generate drivable and inaccessible areas. Some methods are to perform image segmentation, and then select sparse three-dimensional map points in the ground segmentation in front of the robot based on the assumption that the ground is directly in front of the robot to fit the ground plane to establish a mathematical model of the ground plane, and then use the mathematical model to filter other ground segmentation results, and finally combine the camera pose obtained by SLAM and the ground mathematical model to project the correct ground segmentation image into the SLAM map to achieve dense three-dimensional reconstruction of the ground. However, the SLAM method based on LiDAR is expensive.

[0004] The semantic visual SLAM system currently mainly uses dense point cloud maps, which makes it only suitable for small-scale scenes such as indoors. Raster maps can be converted from point cloud maps, but when sparse point cloud maps are converted to raster maps, due to their sparse characteristics, many unknown areas will be generated, resulting in discrete and discontinuous drivable areas. Although some methods have achieved dense 3D reconstruction of indoor ground, due to the assumption that the ground is flat, this method is only suitable for small-scale scenes such as indoors, and cannot adapt to the undulating ground in large-scale outdoor scenes.

[0005] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a simultaneous positioning and mapping method for establishing a drivable area in a map is provided, aiming to solve the problem in the prior art that when establishing a drivable area in a sparse point cloud map, the cost of the lidar-based SLAM method is high, and the visual semantic SLAM system is not suitable for large-scale outdoor scenes.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] A method for synchronous positioning and mapping of a drivable area in a map, wherein the method comprises:

[0009] Acquire an image sequence to be processed, and detect drivable areas of images in the image sequence in sequence;

[0010] After extracting the drivable area of ​​the current image, determining the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image;

[0011] If the current image is determined to be a key frame based on a preset selection rule, the three-dimensional approximate drivable area corresponding to the current image is fused with the local fusion result corresponding to the previous frame image in the point cloud map to obtain the local fusion result corresponding to the current image in the point cloud map;

[0012] The local fusion result corresponding to the current image in the point cloud map is fused with the global three-dimensional drivable area established in the point cloud map to obtain an updated global three-dimensional drivable area.

[0013] In one embodiment of the present application, after extracting the drivable area for the current image, determining the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image includes:

[0014] After extracting the drivable area of ​​the current image, extracting the boundary of the drivable area of ​​the current image to obtain the boundary of the drivable area of ​​the current image;

[0015] Matching feature points of the drivable area boundary of the current image with the constructed point cloud map to obtain a number of three-dimensional map points in the point cloud map that successfully match the drivable area boundary of the current image;

[0016] A number of three-dimensional map points in the point cloud map that successfully match the boundary of the drivable area of ​​the current image are connected into a closed polygon to form a three-dimensional approximate drivable area corresponding to the current image in the point cloud map.

[0017] In one embodiment of the present application, the drivable area is represented by an image mask; after extracting the drivable area from the current image, the boundary of the drivable area of ​​the current image is extracted to obtain the boundary of the drivable area of ​​the current image, including:

[0018] After extracting the drivable area from the current image, the boundary of the drivable area is extracted from the image mask of the drivable area of ​​the current image to obtain the boundary of the drivable area.

[0019] In one embodiment of the present application, the boundary of the drivable area of ​​the current image is matched with the constructed point cloud map by feature points, and a number of three-dimensional map points in the point cloud map that successfully match the boundary of the drivable area of ​​the current image are obtained, including:

[0020] Feature points within a predetermined radius of the drivable area boundary of the current image are matched with the point cloud map to obtain a number of three-dimensional map points in the point cloud map that successfully match the drivable area boundary of the current image.

[0021] In one embodiment of the present application, the preset selection rule is to select the current image as a key frame every predetermined number of image frames; or, if the number of feature point matching points between the current image and the point cloud map is less than a preset threshold, the current image is selected as the key frame.

[0022] In one embodiment of the present application, after extracting the drivable area for the current image, determining the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image, further comprising:

[0023] If the current image is the next frame image of the key frame, the three-dimensional approximate drivable area corresponding to the current image in the point cloud map is used as the local fusion result corresponding to the current image in the point cloud map.

[0024] In one embodiment of the present application, after extracting the drivable area for the current image, determining the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image, further comprising:

[0025] If the current image is not a key frame, the three-dimensional approximate drivable area corresponding to the current image is fused with the local fusion result corresponding to the previous frame image in the point cloud map to obtain the local fusion result corresponding to the current image in the point cloud map, and continue to extract the drivable area of ​​the next frame image.

[0026] The present application also provides a synchronous positioning and mapping device for establishing a drivable area in a map, wherein the device comprises:

[0027] An area extraction module is used to obtain an image sequence to be processed and sequentially detect drivable areas of images in the image sequence;

[0028] An area determination module is used to determine the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image after extracting the drivable area of ​​the current image;

[0029] A local fusion module is used to fuse the three-dimensional approximate drivable area corresponding to the current image with the local fusion result corresponding to the previous frame image in the point cloud map to obtain the local fusion result corresponding to the current image in the point cloud map;

[0030] A global fusion module is used to fuse the local fusion result corresponding to the current image in the point cloud map with the global three-dimensional drivable area established in the point cloud map if the current image is determined to be a key frame based on a preset selection rule, so as to obtain an updated global three-dimensional drivable area.

[0031] The present application also provides a terminal, which includes: a memory, a processor, and a synchronous positioning and mapping program for establishing a drivable area in a map, which is stored in the memory and can be run on the processor. When the synchronous positioning and mapping program for establishing a drivable area in a map is executed by the processor, the steps of the synchronous positioning and mapping method for establishing a drivable area in a map as described above are implemented.

[0032] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the method for synchronous positioning and mapping of a drivable area in a map as described above.

[0033] The present invention provides a method for synchronous positioning and mapping of a drivable area in a map, and the method for synchronous positioning and mapping of a drivable area in a map includes: obtaining an image sequence to be processed, and sequentially detecting the drivable areas of the images in the image sequence; after extracting the drivable area from the current image, determining the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image; if the current image is determined to be a key frame based on a preset selection rule, the three-dimensional approximate drivable area corresponding to the current image is fused with the local fusion result corresponding to the previous frame image in the point cloud map to obtain the local fusion result corresponding to the current image in the point cloud map; the local fusion result corresponding to the current image in the point cloud map is fused with the global three-dimensional drivable area established in the point cloud map to obtain an updated global three-dimensional drivable area. The present invention extracts the drivable area from the images in the image sequence, determines the three-dimensional approximate drivable area based on the drivable area, and then performs local fusion and global fusion, and finally establishes a continuous drivable area in the point cloud map, which can adapt to a variety of indoor and outdoor scenes and has low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of a preferred embodiment of a method for synchronously positioning and mapping a drivable area in a map according to the present invention;

[0035] Figure 2 is the image mask of the drivable area in the present invention;

[0036] Figure 3 is the drivable area boundary extracted based on the image mask in the present invention;

[0037] Figure 4 It is the local fusion result after the three-dimensional approximate drivable area of ​​the current image and the previous frame image are fused in the present invention;

[0038] Figure 5 It is a schematic diagram of fusing the local fusion result corresponding to the current image in the point cloud map with the global three-dimensional drivable area established in the point cloud map in the present invention;

[0039] Figure 6 It is a flowchart of a method for synchronously positioning and mapping a drivable area in a map according to the present invention;

[0040] Figure 7 It is a functional principle block diagram of a preferred embodiment of a synchronous positioning and mapping device for establishing a drivable area in a map in the present invention;

[0041] Figure 8 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] This application relies solely on visual sensors to achieve the effect of establishing a drivable area in a sparse point cloud map without the need for a priori assumptions, and does not require the addition of additional map points to the map, which is conducive to the storage, fast loading and maintenance of the map. The method of this application can be used not only in small-scale scenes such as indoors, but also in large-scale scenes with undulating outdoor ground. Robots and other intelligent agents can plan paths and avoid obstacles in advance according to the driving area established in the map, laying a good foundation for successfully reaching the destination.

[0044] See also Figure 1 , Figure 1 : is a flow chart of the method for synchronous positioning and mapping of a drivable area in a map in the present invention. Figure 1 As shown, the method for synchronously positioning and mapping a drivable area in a map according to an embodiment of the present invention includes:

[0045] Step S100: Acquire a sequence of images to be processed, and detect drivable areas of images in the sequence of images in sequence.

[0046] Specifically, the traditional visual SLAM system extracts feature points and matches feature points through the camera's input video sequence, and builds and optimizes the point cloud map while optimizing the camera pose. However, the point cloud map does not contain information about the drivable area. Even if it is converted into a raster map, the sparse nature of the point cloud map will produce discrete, discontinuous drivable areas and a large number of unknown areas during the conversion. The embodiment of the present application extracts the drivable area from the images in the image sequence, and then establishes a continuous drivable area in the point cloud map.

[0047] like Figure 1 As shown, the method for synchronous positioning and mapping of a drivable area in a map according to an embodiment of the present invention further includes:

[0048] Step S200: After extracting the drivable area for the current image, determine the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image.

[0049] Specifically, step S200 completes the conversion of the two-dimensional drivable area to the three-dimensional drivable area.

[0050] In the embodiment of the present application, the step S200 specifically includes:

[0051] Step S210: after extracting the drivable area of ​​the current image, extracting the boundary of the drivable area of ​​the current image to obtain the boundary of the drivable area of ​​the current image;

[0052] Step S220, matching feature points of the drivable area boundary of the current image with the constructed point cloud map, and obtaining a number of three-dimensional map points in the point cloud map that successfully match the drivable area boundary of the current image;

[0053] Step S230: Connect several three-dimensional map points in the point cloud map that successfully match the boundary of the drivable area of ​​the current image into a closed polygon to form a three-dimensional approximate drivable area corresponding to the current image in the point cloud map.

[0054] Specifically, the embodiment of the present application detects the drivable area based on texture, road edge, vanishing point and other methods, and can also use deep learning methods to obtain the drivable area and identify the boundary of the drivable area of ​​the current image. The feature points in the current frame that successfully match the three-dimensional map points in the point cloud map are searched along the boundary, and the three-dimensional map points corresponding to the searched feature points of the current frame are connected into closed polygons to form a three-dimensional approximate drivable area.

[0055] That is to say, in the SLAM system, feature points are detected in each frame of the image and matched with the three-dimensional map points in the map. The three-dimensional map points corresponding to the searched feature points of the current frame are connected into closed polygons, which can generate a three-dimensional polygon that is closest to the drivable area, and complete the conversion of the two-dimensional drivable area of ​​the current frame to the three-dimensional drivable area. The three-dimensional approximate drivable area established by this method has no ground plane assumption, can adapt to complex situations such as ground undulations, and does not add any new map points, which is conducive to the storage and fast loading of maps.

[0056] The embodiment of the present application realizes the conversion of a two-dimensional drivable area to a three-dimensional drivable area through the steps of drivable area boundary extraction and feature point matching, thereby providing a basis for establishing a continuous drivable area in a point cloud map.

[0057] In one embodiment of the present application, the drivable area is represented by an image mask; the step S210 is specifically as follows: after extracting the drivable area from the current image, performing boundary extraction on the image mask of the drivable area of ​​the current image to obtain the boundary of the drivable area.

[0058] Specifically, the drivable area obtained in the embodiment of the present application is represented by an image mask, such as Figure 2 As shown. The boundary of the image mask of the drivable area is extracted, as shown Figure 3 As shown, the white part indicates the boundary.

[0059] The drivable area in the embodiment of the present application is represented by an image mask, which can effectively shield the non-target area in the image, thereby accurately identifying the drivable area in the image.

[0060] In one embodiment of the present application, step S220 specifically includes: matching feature points within a predetermined radius of the drivable area boundary of the current image with the point cloud map to obtain a number of three-dimensional map points in the point cloud map that successfully match the drivable area boundary of the current image.

[0061] The embodiment of the present application increases the probability and accuracy of feature point matching by searching for the nearest feature point that successfully matches the three-dimensional map point in the point cloud map along the boundary of the drivable area.

[0062] like Figure 1 As shown, the method for synchronous positioning and mapping of a drivable area in a map according to an embodiment of the present invention further includes:

[0063] Step S300: If the current image is determined to be a key frame based on a preset selection rule, the three-dimensional approximate drivable area corresponding to the current image is fused with the local fusion result corresponding to the previous frame image in the point cloud map to obtain the local fusion result corresponding to the current image in the point cloud map.

[0064] Specifically, the 3D approximate drivable area of ​​the current image is fused with the local fusion result of the previous frame and saved as the local fusion result of the current image, such as Figure 4 shown.

[0065] In an embodiment of the present application, the preset selection rule is to select the current image as a key frame every predetermined number of image frames; or, if the number of feature point matching points between the current image and the point cloud map is less than a preset threshold, the current image is selected as the key frame. In other words, the embodiment of the present application first performs local fusion, and therefore selects a key frame as the end point of a local fusion. When the current frame is selected as a key frame by the SLAM algorithm, the three-dimensional approximate drivable area polygon after the frame is fused is combined with the drivable area polygon in the map to form a global three-dimensional approximate drivable area polygon.

[0066] like Figure 1 As shown, the method for synchronous positioning and mapping of a drivable area in a map according to an embodiment of the present invention further includes:

[0067] Step S400: Fusing the local fusion result corresponding to the current image in the point cloud map with the global three-dimensional drivable area established in the point cloud map to obtain an updated global three-dimensional drivable area.

[0068] Specifically, in order to form a complete drivable area in the map, it is necessary to find the union of the 3D approximate drivable area polygon after the current key frame fusion and the existing 3D drivable area in the point cloud map to achieve global fusion of the drivable area. The process is as follows: Figure 5 As shown, Figure 5 The points in are the point clouds in the point cloud map. The larger area is the existing drivable area, and the smaller area is the local fusion result corresponding to the current key frame.

[0069] The synchronous positioning and mapping method for establishing a drivable area in a map provided in the embodiment of the present application is integrated into the SLAM algorithm framework. The overall flow chart is as follows: Figure 6 As shown. The traditional SLAM algorithm includes three modules: tracking, local optimization and global optimization. The embodiment of the present application integrates the above five steps of establishing a drivable area in a sparse point map into three modules. Among them, the four steps of driving area segmentation, driving area boundary extraction, three-dimensional approximate driving area generation and multi-frame fusion are integrated in the tracking module, and each frame image is processed. When the frame image is selected as a key frame, the accumulated fused three-dimensional approximate drivable area polygon stored in it is combined with the existing drivable area polygons in the point cloud map in the global optimization module to obtain the global drivable area fusion result. That is, the step of global fusion of the driving area is integrated in the global optimization module of the SLAM algorithm, and finally the establishment of the drivable area in the sparse point cloud map is completed.

[0070] The embodiments of the present application realize the establishment of a continuous drivable area in a sparse point cloud map of a visual SLAM system, and simultaneously adapt to a variety of indoor and outdoor scenarios.

[0071] In an embodiment of the present application, step S200 also includes: if the current image is the next frame image of the key frame, the three-dimensional approximate drivable area corresponding to the current image in the point cloud map is used as the local fusion result corresponding to the current image in the point cloud map.

[0072] That is to say, after the calculation of the key frame is completed, the next frame of the image will be used as the first frame of local fusion, and multiple frames will be fused in sequence until the next key frame appears.

[0073] The embodiment of the present application performs local fusion first and saves the local fusion result, which is beneficial to the storage of the drivable area.

[0074] In an embodiment of the present application, step S200 also includes: if the current image is not a key frame, the three-dimensional approximate drivable area corresponding to the current image is fused with the local fusion result corresponding to the previous frame image in the point cloud map to obtain the local fusion result corresponding to the current image in the point cloud map, and continue to extract the drivable area of ​​the next frame image.

[0075] That is to say, if the current image is not a key frame, global fusion is not required, but local fusion is continued.

[0076] This application has achieved the following results:

[0077] First, compared with the dense point cloud generated by semantic visual SLAM, the method of the present application inherits the advantage of sparse point cloud maps that can be used for large-scale mapping while establishing the drivable area.

[0078] Second, compared with the LiDAR SLAM system, the method of the present application relies only on visual sensors, which is low-cost and easy to maintain.

[0079] Third, this application does not have any a priori assumptions and can be applied to a variety of environments, including indoors and outdoors.

[0080] Fourth, the method of the present application does not require adding additional map points to the map, which is conducive to the storage, fast loading and maintenance of the map.

[0081] In one embodiment, if Figure 7 As shown, based on the above-mentioned method for synchronous positioning and mapping of a drivable area in a map, the present invention also provides a synchronous positioning and mapping device for establishing a drivable area in a map, including:

[0082] The region extraction module 100 is used to obtain a sequence of images to be processed and sequentially detect drivable regions of images in the sequence of images;

[0083] The area determination module 200 is used to determine the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image after extracting the drivable area of ​​the current image;

[0084] The local fusion module 300 is used to fuse the three-dimensional approximate drivable area corresponding to the current image with the local fusion result corresponding to the previous frame image in the point cloud map if the current image is determined to be a key frame based on a preset selection rule, so as to obtain the local fusion result corresponding to the current image in the point cloud map;

[0085] The global fusion module 400 is used to fuse the local fusion result corresponding to the current image in the point cloud map with the global three-dimensional drivable area established in the point cloud map to obtain an updated global three-dimensional drivable area.

[0086] Figure 8 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0087] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0088] When the processor 502 executes the program, the synchronous positioning and mapping method for establishing a drivable area in a map provided in the above embodiment is implemented.

[0089] Furthermore, the terminal further includes:

[0090] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0091] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0092] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0093] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0094] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0095] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0096] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for synchronous positioning and mapping of a drivable area in a map as described above is implemented.

[0097] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0098] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0099] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0100] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0101] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0102] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0103] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0104] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0105] In summary, the present invention discloses a method for synchronous positioning and mapping of a drivable area in a map, and the method for synchronous positioning and mapping of a drivable area in a map includes: obtaining an image sequence to be processed, and sequentially detecting the drivable areas of the images in the image sequence; after extracting the drivable area from the current image, determining the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image; if the current image is determined to be a key frame based on a preset selection rule, the three-dimensional approximate drivable area corresponding to the current image is fused with the local fusion result corresponding to the previous frame image in the point cloud map to obtain the local fusion result corresponding to the current image in the point cloud map; the local fusion result corresponding to the current image in the point cloud map is fused with the global three-dimensional drivable area established in the point cloud map to obtain an updated global three-dimensional drivable area. The present invention extracts the drivable area from the images in the image sequence, determines the three-dimensional approximate drivable area based on the drivable area, and then performs local fusion and global fusion, and finally establishes a continuous drivable area in the point cloud map, which can adapt to a variety of indoor and outdoor scenes and has low cost.

[0106] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for synchronous positioning and mapping of a drivable area in a map, characterized in that: The method comprises: Acquire an image sequence to be processed, and detect drivable areas of images in the image sequence in sequence; After extracting the drivable area of ​​the current image, determining the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image; If the current image is determined to be a key frame based on a preset selection rule, the three-dimensional approximate drivable area corresponding to the current image is fused with the local fusion result corresponding to the previous frame image in the point cloud map to obtain the local fusion result corresponding to the current image in the point cloud map; The local fusion result corresponding to the current image in the point cloud map is fused with the global three-dimensional drivable area established in the point cloud map to obtain an updated global three-dimensional drivable area.

2. The method for synchronous positioning and mapping of a drivable area in a map according to claim 1, characterized in that: After extracting the drivable area of ​​the current image, the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map is determined based on the drivable area of ​​the current image, including: After extracting the drivable area of ​​the current image, extracting the boundary of the drivable area of ​​the current image to obtain the boundary of the drivable area of ​​the current image; Matching feature points of the drivable area boundary of the current image with the constructed point cloud map to obtain a number of three-dimensional map points in the point cloud map that successfully match the drivable area boundary of the current image; A number of three-dimensional map points in the point cloud map that successfully match the boundary of the drivable area of ​​the current image are connected into a closed polygon to form a three-dimensional approximate drivable area corresponding to the current image in the point cloud map.

3. The method for synchronous positioning and mapping of a drivable area in a map according to claim 2, characterized in that: The drivable area is represented by an image mask; After extracting the drivable area of ​​the current image, the boundary of the drivable area of ​​the current image is extracted to obtain the boundary of the drivable area of ​​the current image, including: After extracting the drivable area from the current image, the boundary of the drivable area is extracted from the image mask of the drivable area of ​​the current image to obtain the boundary of the drivable area.

4. The method for synchronous positioning and mapping of a drivable area in a map according to claim 2, characterized in that: The boundary of the drivable area of ​​the current image is matched with the constructed point cloud map for feature points, and a number of three-dimensional map points in the point cloud map that successfully match the boundary of the drivable area of ​​the current image are obtained, including: Feature points within a predetermined radius of the drivable area boundary of the current image are matched with the point cloud map to obtain a number of three-dimensional map points in the point cloud map that successfully match the drivable area boundary of the current image.

5. The method for synchronous positioning and mapping of a drivable area in a map according to claim 1, characterized in that: The preset selection rule is to select the current image as a key frame every predetermined number of image frames; or, if the number of matching feature points between the current image and the point cloud map is less than a preset threshold, the current image is selected as the key frame.

6. The method for synchronous positioning and mapping of a drivable area in a map according to claim 1, characterized in that: After extracting the drivable area for the current image, and determining the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image, the method further includes: If the current image is the next frame image of the key frame, the three-dimensional approximate drivable area corresponding to the current image in the point cloud map is used as the local fusion result corresponding to the current image in the point cloud map.

7. The method for synchronous positioning and mapping of a drivable area in a map according to claim 1, characterized in that: After extracting the drivable area for the current image, and determining the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image, the method further includes: If the current image is not a key frame, the three-dimensional approximate drivable area corresponding to the current image is fused with the local fusion result corresponding to the previous frame image in the point cloud map to obtain the local fusion result corresponding to the current image in the point cloud map, and continue to extract the drivable area of ​​the next frame image.

8. A synchronous positioning and mapping device for establishing a drivable area in a map, characterized in that: The device comprises: An area extraction module is used to obtain an image sequence to be processed and sequentially detect drivable areas of images in the image sequence; An area determination module is used to determine the three-dimensional approximate drivable area corresponding to the current image in the constructed point cloud map based on the drivable area of ​​the current image after extracting the drivable area of ​​the current image; A local fusion module is used to fuse the three-dimensional approximate drivable area corresponding to the current image with the local fusion result corresponding to the previous frame image in the point cloud map to obtain the local fusion result corresponding to the current image in the point cloud map; A global fusion module is used to fuse the local fusion result corresponding to the current image in the point cloud map with the global three-dimensional drivable area established in the point cloud map if the current image is determined to be a key frame based on a preset selection rule, so as to obtain an updated global three-dimensional drivable area.

9. A terminal, characterized in that: include: A memory, a processor, and a synchronous positioning and mapping program for establishing a drivable area in a map, which is stored in the memory and can be run on the processor. When the synchronous positioning and mapping program for establishing a drivable area in a map is executed by the processor, the steps of the synchronous positioning and mapping method for establishing a drivable area in a map are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed to implement the steps of the method for simultaneous positioning and mapping of a drivable area in a map as described in any one of claims 1 to 7.

Images