Panoramic image acquisition device, mapping method and mobile robot

Abstract

The present application discloses a panoramic image acquisition device, a mapping method and a mobile robot. The panoramic image acquisition device includes a bracket and a plurality of depth cameras. The bracket is a polygonal bracket or a circular bracket. Each depth camera is respectively arranged on each edge of the polygonal bracket, or evenly arranged on the circular edge of the circular bracket. According to the technical solution provided in the embodiment of the present application, by arranging a panoramic image acquisition device with a depth camera on each edge of the polygonal bracket, the problem of limited field of view of the traditional depth camera system can be solved.

Description

Technical Field

[0001] The present disclosure generally relates to a broad field (eg, the field of computer technology), and more particularly to a panoramic image acquisition device, a mapping method, and a mobile robot. Background Art

[0002] Existing depth camera systems have limited field of view, small feature points within the field of view, and limited applications. Although lidar can provide information with a large field of view, it is expensive, does not provide enough information about the environment, and has very complex positioning operations. In addition, it is difficult to achieve real-time positioning and mapping in complex environments. Summary of the invention

[0003] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a low-cost image acquisition device, a mapping method and a mobile robot that can capture a panoramic view.

[0004] In a first aspect, a panoramic image acquisition device is provided, which includes a bracket and a plurality of depth cameras. The bracket is a polygonal bracket or a circular bracket, and each depth camera is respectively arranged on each edge of the polygonal bracket, or is evenly arranged on the circular edge of the circular bracket.

[0005] In a second aspect, a mapping method based on a mobile robot is provided, wherein the mobile robot includes a panoramic image acquisition device provided in each embodiment of the present application,

[0006] Collect posture information and orientation image information at the same time, and generate panoramic image sequences and raster maps;

[0007] According to the pose information, the panoramic image sequence and the corresponding grid map are fused to obtain a grid map with panoramic visual markings.

[0008] In a third aspect, a mobile robot is provided, which includes the panoramic image acquisition device provided in each embodiment of the present application.

[0009] According to the technical solution provided in the embodiments of the present application, by setting a panoramic image acquisition device of a depth camera on each side of the polygonal support, the problem of limited field of view of the traditional depth camera system can be solved. Furthermore, according to certain embodiments of the present application, by adopting a mapping method that fuses panoramic images with raster images, the problem that the traditional raster map is not very intuitive can also be solved, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0011] Figure 1An exemplary structural diagram of a panoramic image acquisition device according to an embodiment of the present application is shown;

[0012] Figure 2 Shows Figure 1 An exemplary schematic diagram of the field of view angle of a panoramic image acquisition device;

[0013] Figure 3 An exemplary flow chart of a mapping method according to an embodiment of the present application is shown;

[0014] Figure 4 An exemplary structural block diagram of a robot according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0015] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0016] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0017] Please refer to Figure 1 , shows an exemplary structural schematic diagram of a panoramic image acquisition device according to an embodiment of the present application. As shown in the figure, a panoramic image acquisition device includes a bracket 1 and a plurality of depth cameras 2, the bracket 1 is a polygonal bracket or a circular bracket (not marked in the figure), and each depth camera 2 is arranged on each side of the polygonal bracket, or evenly arranged on the circular side of the circular bracket.

[0018] like Figure 1 The panoramic image acquisition device shown includes an octagonal bracket 1 and a depth camera 2 arranged on each side of the octagonal bracket. The depth cameras in various directions are used to collect pictures of corresponding field angles to form a panoramic image. In practical applications, the polygonal bracket can be set to have any number of sides or a circular bracket according to the situation.

[0019] In some embodiments, the depth cameras are arranged on the same horizontal plane. To facilitate calibration, alignment and image stitching, the depth cameras are arranged on the same horizontal plane.

[0020] In some embodiments, the bracket is a regular polygonal bracket.

[0021] Please refer to Figure 2 , showing Figure 1 An exemplary schematic diagram of the field of view of a panoramic image acquisition device. Figure 2As shown, a regular polygon bracket is used, and the field of view of each depth camera in each direction is set to be the same. In the figure, the panoramic acquisition device includes depth cameras 01 to 08, where A, B, C, D, E, F, G, and H are the intersections of the fields of view of the eight depth cameras, and these intersections form an intersection circle 302, and the diameter 303 of the circle is 2R. In a sensor acquisition range greater than 2R, a three-dimensional 360-degree panoramic depth image can be obtained. In the application, a panoramic image can be acquired according to the situation, or an image of a certain or partial orientation can be acquired. It can be seen that in the range that can be acquired, the field of view of each two adjacent depth cameras has an overlapping area. Moreover, the intersection of the field of view of the depth camera is in a circle. It can be seen that in the range that can be acquired, the field of view of each two adjacent depth cameras has an overlapping area. Moreover, the intersection of the field of view of each depth camera is in a circle.

[0022] Please refer to Figure 3 , shows an exemplary flow chart of a mapping method according to an embodiment of the present application. As shown in the figure, the mapping method includes:

[0023] Step S10: simultaneously collecting posture information and orientation image information to generate a panoramic image sequence and a grid map;

[0024] Step S20: According to the position information, the panoramic image sequence and the corresponding grid map are fused to obtain a grid map with panoramic visual markings.

[0025] In step S10, the panoramic image acquisition device acquires image information in all directions, and the posture acquisition device acquires posture information, and generates a panoramic image sequence with posture information and a grid map with posture information. The panoramic image sequence is usually a time series of images acquired at different times.

[0026] In step S20, a grid map with visual identification is established by fusing the three-dimensional panoramic image sequence and the grid map, and accurate positioning can be performed with the assistance of a position acquisition device.

[0027] In some embodiments, before collecting image information, the method includes:

[0028] Step S1: Setting the internal parameters of each depth camera;

[0029] Step S2: Convert the coordinate system of each depth camera into the world coordinate system.

[0030] In step S1, the internal parameters of the depth camera include focal length, distortion parameters, etc. Different depth cameras have different internal parameters, and the internal parameters of the depth camera are set according to the characteristics of the selected depth camera itself and the distribution of the depth camera on the bracket.

[0031] Step S2 is a process of unifying the coordinates of each depth camera. The unified coordinates are helpful for stitching the images acquired by the subsequent depth cameras.

[0032] In some embodiments, step S10 includes:

[0033] Step S11: Associating the position information with the orientation image information;

[0034] Step S12: stitching the images in all directions according to the three-dimensional point cloud information of the direction image information to obtain a panoramic image sequence with positioning information;

[0035] Step S13: Convert the three-dimensional point cloud information of the orientation image into two-dimensional point cloud information to obtain a grid map.

[0036] In step S11, the collected posture information and image information at the same time are associated to generate a panoramic image with posture information.

[0037] In step S12, a method of first matching and then splicing may be adopted.

[0038] Specifically, first determine whether the point cloud matches.

[0039] If they match, determine whether there is an overlapping area:

[0040] If there is an overlapping area, keep one of the overlapping areas. If there is no overlapping area, dock the matching point clouds.

[0041] Otherwise, there is a blind spot, and step S1 is executed again.

[0042] In step S13, the grid map is an information map obtained by grid processing the known environmental information. The environmental information is rectangularized and modeled, and the unit length of the grid is set according to the actual situation of the environment (for example, the grid can be set to 0.1m*0.1m / 0.05m*0.05m or other size values), so as to convert the given environmental information into a rectangular grid map.

[0043] Each grid lattice point is represented by its lattice center node. The entire grid map is mapped to a coordinate system, and each node corresponds to the coordinates on the coordinate system, that is, represented by (x, y), where x represents the column number where the node is located, and y represents the row number where the node is located; and the coordinate unit length of the entire coordinate system is the unit length of the grid. Among them, the nodes containing obstacle information are marked as 1, and the nodes without obstacle information are marked as 0; the grids with obstacles are called 'occupied' grids, and the grids without obstacles are called 'free' grids. Therefore, the entire grid map is composed of 'occupied' grids and 'free' grids. As for how to determine whether a grid contains obstacle information, a method that can be adopted is to set a probability p, project the three-dimensional panoramic point cloud information onto the two-dimensional plane where the robot center is located. If the probability that an obstacle appears in this grid is called p, this probability p is compared with a preset probability threshold p_th. If p >= p_th, this grid is occupied; if p < p_th, this grid is free. The obtained grid map includes pose information.

[0044] In some embodiments, step S20 includes:

[0045] Step S21: Fuse the panoramic image sequence and the grid map with the same position information.

[0046] The grid map represents the information in the environment in the form of grids. Some grids are occupied, and some grids are free. The different states of the grids can represent the state of the environment around the robot, such as where there are obstacles. However, through the grid map, only the distribution of obstacles in the corresponding environment can be understood, and the specific type of obstacle and the specific object cannot be represented by the grid map. Therefore, this application considers making up for it through image information.

[0047] Specifically, first, obtain the grid map from the three-dimensional point cloud information, and save the specific number of grids and grid IDs corresponding to each recorded position; then, record the information of the panoramic pictures corresponding to all positions during the operation of the robot, and all the picture information forms an image sequence; finally, integrate the image sequence and the grid map with the same position information to form a grid map with panoramic visual markers. Obtain a model for three-dimensionally describing the environment. The formed map has a better user experience for users, and the presented environment is more intuitive.

[0048] In some embodiments, the posture information is collected by an IMU sensor. IMU, an inertial measurement unit, is a device that uses an accelerometer and a gyroscope to measure the three-axis attitude angle (or angular rate) and acceleration of an object. An IMU has gyroscopes and accelerometers installed on three orthogonal axes, with a total of 6 degrees of freedom, to measure the angular velocity and acceleration of an object in three-dimensional space, which is called a "6-axis IMU"; IMU can also add a magnetometer on the basis of the accelerometer and gyroscope, which is called a "9-axis IMU".

[0049] Figure 4 An exemplary structural block diagram of a mobile robot according to an embodiment of the present application is shown.

[0050] like Figure 4 As shown, the mobile robot 400 includes one or more central processing units (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the system 400 are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0051] The following components are connected to the I / O interface 405: an acquisition section 406 which may include an image acquisition device, a posture sensing device, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0052] In particular, according to the embodiments of the present disclosure, the above reference Figure 3 The described process may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for executing the mapping method. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 409, and / or installed from a removable medium 411.

[0053] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0054] As another aspect, the present application further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the mapping method described in the present application.

[0055] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

Claims

1. A mapping method based on a mobile robot, characterized in that: The mobile robot includes a panoramic image acquisition device, which includes a bracket and a plurality of depth cameras. The bracket is a polygonal bracket or a circular bracket, and each of the depth cameras is respectively arranged on each edge of the polygonal bracket, or is evenly arranged on the circular edge of the circular bracket; each of the depth cameras is in the same horizontal plane; the polygonal bracket is a regular polygonal bracket; the field of view of each adjacent two depth cameras has an overlapping area; the intersection of the field of view of each depth camera is in a common circle; the method includes: Collecting the position information and the orientation image information at the same time, generating a panoramic image sequence and a grid map having the position information; According to the posture information, a panoramic image sequence having the same position information and the grid map are fused to obtain a grid map having panoramic visual markers; Wherein, before collecting the azimuth image information, the following steps are included: Setting the internal parameters of each depth camera; converting the coordinate system of each depth camera into a world coordinate system; The simultaneous acquisition of position information and orientation image information and generation of a panoramic image sequence and a grid map includes: The posture information is associated with the orientation image information; images in various orientations are spliced ​​according to the three-dimensional point cloud information of the orientation image information to obtain a panoramic image sequence with positioning information; the three-dimensional point cloud information of the orientation image is converted into two-dimensional point cloud information to obtain a grid map.

2. The mobile robot-based mapping method according to claim 1, characterized in that: The position information is collected by an IMU sensor.

3. A mobile robot, characterized in that: The mobile robot includes a panoramic image acquisition device; The panoramic image acquisition device includes a bracket and a plurality of depth cameras, wherein the bracket is a polygonal bracket or a circular bracket, and each of the depth cameras is respectively arranged on each edge of the polygonal bracket, or is evenly arranged on the circular edge of the circular bracket; The depth cameras are on the same horizontal plane; The polygonal bracket is a regular polygonal bracket; The fields of view of each of the two adjacent depth cameras have an overlapping area; The intersection points of the fields of view of the depth cameras are concentric circles; The mobile robot further comprises a posture sensor and one or more processors. The panoramic image acquisition device is used to acquire azimuth images; The posture sensing device is used to collect posture information; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the mobile robot-based mapping method as described in any one of claims 1-2.

4. The mobile robot according to claim 3, characterized in that: The posture sensing device includes an IMU sensor.

Images