Panoramic vision mapping method, device and equipment based on multiple sensors and medium

Abstract

The invention discloses a panoramic vision mapping method and device based on multiple sensors, equipment and a medium, and the method comprises the steps: building a three-dimensional point cloud map of a to-be-mapped space based on three-dimensional point cloud data collected by an AGV, and enabling the three-dimensional point cloud data to be collected through a sensor carried by the AGV; a plurality of RGB images collected by the plurality of camera devices are fused to obtain an RGB panorama of the to-be-mapped space, the camera devices are installed at different positions in the to-be-mapped space, and the RGB images are collected through sensors carried by the camera devices; fusing the three-dimensional point cloud map with the RGB panorama to obtain a fused map; target detection is carried out based on target sensor data collected by the AGV, target information of the to-be-mapped space is obtained, and the target information comprises position information and type information of a target object; and projecting the target information to the fused map to realize the construction of the panoramic map of the to-be-mapped space. In this way, the applicability of the map in a complex scene can be improved, and smooth execution of an AGV task is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of mapping technology, and in particular to a panoramic vision mapping method, device, equipment and medium based on multi-sensor. Background Art

[0002] For scenarios where AGVs (Automated Guided Vehicles) perform operations, it is usually necessary to build a map of the space where AGVs perform operations in advance. The dispatching system issues instructions based on the map. Existing map systems usually only include cargo points and AGV locations. In actual scenarios, the map is changing due to factors such as personnel, obstacles, and cargo point occupancy, which interferes with the execution of AGV tasks and thus triggers traffic control. Therefore, how to improve the applicability of maps in complex scenarios and ensure the smooth execution of AGV tasks is an urgent problem to be solved. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a panoramic vision mapping method, device, equipment and medium based on multi-sensor, which can improve the applicability of maps in complex scenes and ensure the smooth execution of AGV tasks. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a method for panoramic visual mapping based on multiple sensors, including:

[0005] Creating a three-dimensional point cloud map of the space to be mapped based on the three-dimensional point cloud data collected by the AGV, wherein the three-dimensional point cloud data is collected by a sensor carried by the AGV;

[0006] fusing multiple RGB images collected by multiple camera devices to obtain an RGB panoramic image of the space to be mapped, wherein the multiple camera devices are installed at different positions in the space to be mapped, and the RGB images are collected by sensors carried by the camera devices;

[0007] Fusion of the three-dimensional point cloud map with the RGB panoramic image to obtain a fused map;

[0008] Performing target detection based on the target sensor data collected by the AGV to obtain target information of the space to be mapped, wherein the target information includes location information and type information of the target object;

[0009] The target information is projected onto the fused map to realize the construction of a panoramic map of the space to be mapped.

[0010] Optionally, fusing the three-dimensional point cloud map with the RGB panoramic image to obtain a fused map includes:

[0011] Extracting corner points in the RGB panoramic image to obtain first corner points;

[0012] Extracting corner points in the three-dimensional point cloud map to obtain second corner points;

[0013] Matching the first corner point and the second corner point to obtain a matching corner point;

[0014] The three-dimensional point cloud map is fused with the RGB panoramic image based on the matching corner points to obtain a fused map.

[0015] Optionally, extracting a corner point in the RGB panoramic image to obtain a first corner point includes:

[0016] Generating panoramic images of different sizes based on the RGB panoramic image;

[0017] For a panorama of any size, determine the nearest point of a target point in the panorama and when the pixel difference between the nearest point and the target point is greater than a preset threshold, determine the target point as a corner point, and the target point is any point in the panorama;

[0018] Corner points in panoramas of different sizes are fused to obtain corner points in the RGB panorama as first corner points.

[0019] Optionally, extracting a corner point in the three-dimensional point cloud map to obtain a second corner point includes:

[0020] Downsampling the three-dimensional point cloud map to obtain a downsampled point cloud map;

[0021] Constructing a local plane corresponding to any point in the downsampled point cloud map;

[0022] constructing a covariance matrix based on the centroid of the local plane and the points in the local plane;

[0023] Decomposing the covariance matrix into eigenvalues ​​and eigenvectors;

[0024] Determining a normal vector of the arbitrary point from the eigenvector based on the eigenvalue;

[0025] The curvature of the plane is calculated based on the normal vector, and a corner point is determined based on the curvature of the plane to obtain a second corner point.

[0026] Optionally, the fusing the three-dimensional point cloud map with the RGB panoramic image based on the matching corner points to obtain a fused map includes:

[0027] Calculate a translation matrix and a rotation matrix based on the matching corner points;

[0028] The three-dimensional point cloud map is fused with the RGB panoramic image based on the translation matrix and the rotation matrix to obtain a fused map.

[0029] Optionally, fusing the multiple RGB images captured by the multiple camera devices to obtain the RGB panoramic image of the space to be mapped includes:

[0030] Extracting feature points of a plurality of RGB images captured by a plurality of camera devices respectively;

[0031] Match feature points of multiple RGB images to obtain matching feature points;

[0032] Based on the matching feature points, multiple RGB images collected by multiple camera devices are fused to obtain an RGB panoramic image of the space to be mapped.

[0033] Optionally, performing target detection based on target sensor data collected by the AGV to obtain target information of the space to be mapped includes:

[0034] Based on the target sensor data collected by the AGV and using a deep learning target detection model, target information of the space to be mapped is obtained;

[0035] The target objects include obstacles, pedestrians, storage locations, and various objects in goods.

[0036] In a second aspect, the present application discloses a panoramic visual mapping device based on multiple sensors, comprising:

[0037] A point cloud map creation module, used to create a three-dimensional point cloud map of the space to be mapped based on the three-dimensional point cloud data collected by the AGV, wherein the three-dimensional point cloud data is collected by the sensor carried by the AGV;

[0038] A panoramic image generation module, used for fusing multiple RGB images collected by multiple camera devices to obtain an RGB panoramic image of the space to be mapped, wherein the multiple camera devices are installed at different positions in the space to be mapped, and the RGB images are collected by sensors carried by the camera devices;

[0039] An image fusion module, used for fusing the three-dimensional point cloud map with the RGB panoramic image to obtain a fused map;

[0040] A target detection module, used to perform target detection based on the target sensor data collected by the AGV to obtain target information of the space to be mapped, wherein the target information includes location information and type information of the target object;

[0041] The information projection module is used to project the target information onto the fused map to realize the construction of a panoramic map of the space to be mapped.

[0042] In a third aspect, the present application discloses an electronic device, including a memory and a processor, wherein:

[0043] The memory is used to store the computer program;

[0044] The processor is used to execute the computer program to implement the aforementioned multi-sensor based panoramic vision mapping method.

[0045] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned multi-sensor based panoramic visual mapping method.

[0046] It can be seen from the above scheme that the present application provides a panoramic visual mapping method based on multiple sensors, including: creating a three-dimensional point cloud map of the space to be mapped based on the three-dimensional point cloud data collected by the AGV, wherein the three-dimensional point cloud data is collected by the sensor carried by the AGV; fusing multiple RGB images collected by multiple cameras to obtain an RGB panoramic image of the space to be mapped, wherein the multiple cameras are installed at different positions in the space to be mapped, and the sensors carried by the cameras collect RGB images; fusing the three-dimensional point cloud map with the RGB panoramic image to obtain a fused map; performing target detection based on the target sensor data collected by the AGV to obtain target information of the space to be mapped, wherein the target information includes location information and type information of the target object; projecting the target information to the fused map to realize the construction of a panoramic map of the space to be mapped.

[0047] It can be seen that the beneficial effects of the present application are as follows: when building a map, the present application fuses the three-dimensional point cloud map of the space to be mapped, which is created based on the three-dimensional point cloud data collected by the AGV, with the RGB panoramic map of the space to be mapped, which is obtained by fusing multiple RGB images collected by multiple cameras. The sensor data of the local sensors on the vehicle side and the global sensors of the space to be mapped are integrated to improve the map accuracy. Target detection is performed based on the target sensor data collected by the AGV, and the target information is projected onto the fused map to realize the construction of a panoramic map of the space to be mapped, thereby enriching the map information, improving the applicability of the map in complex scenarios, and ensuring the smooth execution of AGV tasks.

[0048] Correspondingly, the present application provides a multi-sensor based panoramic visual mapping device, equipment and readable storage medium, which also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1 A flowchart of a panoramic visual mapping method based on multiple sensors provided in an embodiment of the present application;

[0051] Figure 2 A map construction flow chart provided for an embodiment of the present application;

[0052] Figure 3 An image fusion flow chart provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the structure of a panoramic visual mapping device based on multiple sensors provided in an embodiment of the present application;

[0054] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] For scenarios where AGVs perform operations, it is usually necessary to establish a map of the space corresponding to the AGV's operations in advance, and the scheduling system issues instructions based on the map. Existing map systems usually only contain cargo points and AGV positions, but in actual scenarios, due to factors such as personnel, obstacles, and cargo point occupancy, the map is changing, interfering with the execution of AGV tasks, thereby triggering traffic control. For example, when AGVs are operating in a factory environment, it is usually necessary to establish a map of the factory in advance, and the scheduling system issues instructions based on the map. To this end, this application provides a mapping solution that can improve the applicability of maps in complex scenarios and ensure the smooth execution of AGV tasks.

[0057] See also Figure 1 As shown, the embodiment of the present application discloses a panoramic visual mapping method based on multiple sensors, including:

[0058] Step S11: creating a three-dimensional point cloud map of the space to be mapped based on the three-dimensional point cloud data collected by the AGV, wherein the three-dimensional point cloud data is collected by a sensor carried by the AGV.

[0059] The space to be mapped can be a factory, warehouse, etc. The three-dimensional point cloud data can be obtained through the laser radar of the AGV, or through a visual sensor such as a 3D (i.e. 3 Dimensional) camera or a binocular depth camera. For example, a three-dimensional point cloud map is established through a SLAM (i.e. Simultaneous Localization and Mapping) algorithm. Laser SLAM or visual SLAM methods can be used. The basic process of the SLAM method includes data preprocessing, front-end processing, back-end optimization and map construction. In the data preprocessing stage, odometer calibration, laser radar motion distortion removal and time synchronization between different systems are required. The front-end processing is responsible for realizing the laser odometer, including inter-frame matching and loop detection. The back-end optimization corrects the pose and map through the optimization algorithm to improve the accuracy of the map. Finally, the map construction stage generates a map for navigation. VSLAM (i.e. Visual Simultaneous Localization and Mapping) is a technology that uses cameras for environmental perception and positioning. It estimates the movement of the camera by analyzing the changes in feature points in continuous image frames and constructs a three-dimensional map of the environment. This process includes key steps such as tracking, map building, loop detection, and backend optimization.

[0060] In an optional implementation, it is also possible to combine laser radar and visual sensors such as 3D cameras or binocular depth cameras to obtain three-dimensional point cloud data and create a three-dimensional point cloud map. For example, the embodiment of the present application can fuse the three-dimensional point cloud data collected by the visual sensor and the laser radar, and then establish a three-dimensional point cloud map through the SLAM algorithm. It can reduce the dependence of laser SLAM mapping on environmental features and improve the effect in dynamic environments. It also reduces the light dependence of visual SLAM to avoid the problem of being unable to work in some dark places or areas where visual features are not obvious, and there are cumulative errors, resulting in insufficient accuracy.

[0061] Step S12: Fusing multiple RGB (i.e., Red, Green, Blue) images captured by multiple cameras to obtain an RGB panoramic image of the space to be mapped, wherein the multiple cameras are installed at different positions in the space to be mapped, and the RGB images are captured by sensors carried by the cameras.

[0062] The camera device may be equipped with an RGB sensor to collect RGB images. The multiple cameras are installed at different positions in the space to be mapped, and can cover all work areas as monitoring cameras. For example, the midpoint of the map is used as the world coordinate origin, and all cameras are registered in the current world coordinate system according to the installation position coordinates of the camera devices, and the cameras are uniformly installed in a top-down manner.

[0063] In an optional implementation, feature points of multiple RGB images collected by multiple camera devices can be extracted respectively; the feature points of multiple RGB images are matched to obtain matching feature points; based on the matching feature points, multiple RGB images collected by multiple camera devices are fused to obtain an RGB panoramic image of the space to be mapped. A translation transformation matrix can be calculated based on the matching feature points, and the RGB images can be translated based on the translation transformation matrix, and then spliced ​​to obtain an RGB panoramic image of the space to be mapped.

[0064] For example, the RGB image can be preprocessed first, including image grayscale and image denoising. Converting a color RGB image to a grayscale image can simplify calculations and reduce the amount of data. Removing noise from an image can improve image quality, and denoising methods such as Gaussian filtering and mean filtering can be used. Further, feature extraction and matching are performed: first, feature point detection is performed: feature points such as corners and edges are detected in a grayscale image, and feature point detection algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF) can be used. A feature descriptor is then generated for each feature point, and the feature descriptor contains enough information to distinguish different feature points. Feature descriptors can be used to match feature points of different images, and then the translation transformation matrix between the images is calculated based on the matched feature point pairs. The calculated translation transformation matrix is ​​used to transform one image into the coordinate system of another image, and the transformed images are fused to generate a panorama. The fusion method can be an averaging method, a weighted method, or a more complex image fusion algorithm (such as multi-resolution fusion, Laplace pyramid fusion, etc.). Furthermore, the stitched panorama can be post-processed, such as removing stitching gaps, adjusting brightness and contrast, etc.

[0065] Step S13: Fusing the three-dimensional point cloud map with the RGB panoramic image to obtain a fused map.

[0066] In an optional implementation, step S13 may specifically include the following steps:

[0067] Step 00: extract the corner points in the RGB panoramic image to obtain the first corner points.

[0068] This embodiment can generate panoramas of different sizes based on the RGB panorama; for a panorama of any size, determine the nearest point of a target point in the panorama and when the pixel difference between the nearest point and the target point is greater than a preset threshold, determine the target point as a corner point, and the target point is any point in the panorama; fuse the corner points in panoramas of different sizes to obtain a corner point in the RGB panorama as a first corner point.

[0069] For example, first construct an image pyramid, and pass the RGB panorama through the resize function (i.e., the function used to adjust the size of the image) to obtain the original Figure 1 / 2 size and 1 / 4 size images are used to obtain panoramas of different sizes. The kd (i.e. k-dimensional, in k-dimensional space) tree can be used to search for the nearest neighbor point. If the pixel difference between the nearest neighbor point and the point is greater than the threshold t, the point is considered to be a corner point, and the t value is manually set according to the image pixel value. The corner points identified at different scales can be fused to obtain the final set of corner points, which can be achieved by comparing the similarity of information such as the position, direction and descriptor of the corner points at different scales. In addition, the fused corner points can be subjected to multi-scale consistency verification to ensure that these corner points remain stable at different scales, which can be achieved by calculating the stability of indicators such as the position change and direction change of the corner points at different scales. According to the specific task requirements, a certain number of optimal corner points are selected from the fused set of corner points for subsequent processing. The selection can be based on one or more factors such as the response value, stability, and distribution uniformity of the corner points. The response value of the corner point can be obtained according to the corner point detection algorithm.

[0070] Among them, the kd tree search for the nearest neighbor is a method to efficiently find the data point closest to a given query point in a multidimensional space. The kd tree is a special type of binary tree used to store point sets in k-dimensional space. Each node represents a k-dimensional data point and recursively divides the space into two left and right subspaces. The construction process of the kd tree involves selecting the split axis (split domain) and the split point (node-data) in order to effectively split the data in a multidimensional space.

[0071] Step 01: Extract corner points in the three-dimensional point cloud map to obtain second corner points.

[0072] The embodiment of the present application can downsample the three-dimensional point cloud map to obtain a downsampled point cloud map; construct a local plane corresponding to any point in the downsampled point cloud map; construct a covariance matrix based on the centroid of the local plane and the points in the local plane; decompose the covariance matrix into eigenvalues ​​and eigenvectors; determine the normal vector of the arbitrary point from the eigenvector based on the eigenvalues; calculate the plane curvature based on the normal vector, and determine the corner point based on the plane curvature to obtain a second corner point.

[0073] It should be pointed out that downsampling the point cloud image can highlight the feature points while reducing the amount of calculation. This embodiment can find the normal vector of the downsampled point cloud map. For any point, the nearest preset number of neighboring points can be searched, and the local plane corresponding to the arbitrary point can be constructed based on the normal vector and the nearest preset number of neighboring points. The covariance matrix is ​​constructed based on the centroid of the local plane and the points in the local plane, and the covariance matrix is ​​decomposed into eigenvalues ​​and eigenvectors. The eigenvector corresponding to the minimum eigenvalue is determined as the normal vector of the arbitrary point. The curvature is obtained based on the normal vector of the point, thereby obtaining the corner point. For example, one or more nearest neighboring points of any point can be determined based on the KD algorithm, and it is determined whether the difference between the curvature of any point and the curvature of the nearest point is greater than a preset threshold. If so, the arbitrary point is determined as a corner point.

[0074] Step 02: Match the first corner point and the second corner point to obtain a matching corner point.

[0075] A preset feature point matching algorithm may be used to match the first corner point and the second corner point to obtain a matching corner point, such as a SIFT matching algorithm, a SURF matching algorithm, an ORB matching algorithm, and the like.

[0076] Step 03: Based on the matching corner points, the three-dimensional point cloud map is fused with the RGB panoramic image to obtain a fused map.

[0077] In the embodiment of the present application, a translation matrix and a rotation matrix can be calculated based on the matching corner points; based on the translation matrix and the rotation matrix, the three-dimensional point cloud map can be fused with the RGB panoramic image to obtain a fused map. For example, the three-dimensional point cloud map can be transformed based on the translation matrix and the rotation matrix, and the transformed three-dimensional point cloud map can be fused with the RGB panoramic image.

[0078] Step S14: performing target detection based on the target sensor data collected by the AGV to obtain target information of the space to be mapped, wherein the target information includes location information and type information of the target object.

[0079] This embodiment can perform target detection based on the target sensor data collected by the AGV and use a deep learning target detection model to obtain target information of the space to be mapped; wherein the target objects include obstacles, pedestrians, storage locations, and various objects in goods.

[0080] In an optional implementation, the target sensor data includes lidar data, and the deep learning target detection model may be a model trained using lidar data samples.

[0081] In an optional implementation, the target sensor data includes data collected by a visual sensor, such as data collected by a 3D camera or a binocular depth camera. The deep learning target detection model may be a model trained using data samples collected by a 3D camera or a binocular depth camera.

[0082] In an optional implementation, the target sensor data includes laser radar data and data collected by a 3D camera or a binocular depth camera. The deep learning target detection model can be a model trained using samples consisting of laser radar data and data collected by a 3D camera or a binocular depth camera.

[0083] In an optional embodiment, the target sensor data includes laser radar data and a visual sensor that collects two-dimensional images, such as an RGB sensor, and the deep learning target detection model is a model trained using two-dimensional image samples. The laser radar data can determine the relative position of the target and the AGV.

[0084] Through the AGV's odometer data and the relative position of the target and the AGV, the position of the target in the world coordinate system can be obtained.

[0085] Step S15: Projecting the target information onto the fused map to realize the construction of a panoramic map of the space to be mapped.

[0086] That is, the target information can be added to the fused map, so that real-time global map construction can be achieved to cope with environmental changes in the scene.

[0087] Furthermore, the mapping solution provided by the present application can be executed on an algorithm server, that is, multiple cameras and sensors carried by AGV are responsible for collecting sensor data, and the algorithm server builds the map based on the sensor data. In another optional implementation, the mapping solution provided by the present application can be written and executed by an industrial computer and an algorithm server. The industrial computer can build a three-dimensional point cloud map and perform target detection based on the sensor data collected by the AGV, and the algorithm server builds the map based on the three-dimensional point cloud map, target detection results, and RGB images collected by multiple cameras. The industrial computer can be mounted on an AGV or be an independent device.

[0088] It can be seen that when building a map, the embodiment of the present application fuses the three-dimensional point cloud map of the space to be mapped, which is created based on the three-dimensional point cloud data collected by the AGV, with the RGB panoramic map of the space to be mapped, which is obtained by fusing multiple RGB images collected by multiple cameras. The sensor data of the local sensors on the vehicle side and the global sensors of the space to be mapped are integrated to improve the map accuracy. Target detection is performed based on the target sensor data collected by the AGV, and the target information is projected onto the fused map to realize the construction of a panoramic map of the space to be mapped. This enriches the map information, can improve the applicability of the map in complex scenarios, and ensure the smooth execution of AGV tasks.

[0089] See also Figure 2 As shown, Figure 2 A mapping flow chart provided for the embodiment of the present application. Based on multiple sensors, panoramic visual real-time mapping is performed. The hardware part mainly includes visual sensors such as 3D cameras or binocular depth cameras installed at the fork of the AGV, laser radars and odometers installed on the AGV, monitoring cameras installed at different locations in the factory to cover all operating areas, on-board WIFI (Wireless Fidelity) modules, industrial computers for edge computing tasks such as image processing, and algorithm servers for realizing panoramic visual real-time mapping. The AGV interacts with the algorithm server through the on-board WIFI module. First, a 3D point cloud map of the warehouse is established through the SLAM algorithm. The midpoint of the map is used as the world coordinate origin. According to the installation position coordinates of the surveillance cameras, all cameras are registered in the current world coordinate system. The cameras are uniformly installed in a top-down manner. Registration and fusion are performed according to the translation model to obtain the RGB panoramic image of the warehouse. The corner points in the RGB panoramic image are extracted through the ORB feature point detection algorithm. The corner points of the 3D point cloud map are extracted through the feature point extraction algorithm. The RGB image and the point cloud map are registered and fused according to the corner point information. Based on the warehouse location detection algorithm and the cargo detection algorithm, the warehouse location information and cargo information under the top-down angle can be obtained. According to the AGV coordinates and the calibration information of the AGV and its own sensors, the sensors on the AGV can be registered to the same world coordinates. According to the deep learning target detection model deployed in the AGV, obstacles, pedestrians, storage locations, and cargo on the path can be detected in real time when the AGV is moving. The target type and the relative coordinates of the target and the AGV (i.e., the center point of the AGV) can be obtained by the detection model. The target information is projected onto the map in real time to complete the map information, thereby obtaining a real-time panoramic map of the warehouse.

[0090] For further information, see Figure 3 The image fusion flow chart provided in the embodiment of the present application is shown as follows. Down-sample the point cloud image to reduce the amount of calculation while highlighting the feature points, and calculate the normal vector of the down-sampled point cloud. For each point p in the point cloud, the kd tree is used to search for the nearest K adjacent points, and the local plane P can be expressed as:

[0091]

[0092] in, As the normal vector of the P plane (that is, the normal vector obtained by calculating the normal vector of the downsampled point cloud mentioned above), d represents the distance from P to the origin of the coordinate system. Let the centroid of the P plane be P 0 , construct the covariance matrix:

[0093]

[0094] Among them, M is a 3x3 matrix. The eigenvalue decomposition of the matrix can obtain three eigenvectors and eigenvalues. The eigenvector corresponding to the minimum eigenvalue λ can be considered as the normal vector of point p. By calculating the transformation of the plane normal vector, the curvature of the plane is obtained, and thus the corner points of the 3D plane are obtained.

[0095] At the same time, the ORB algorithm is used to extract feature points from the RGB panoramic bird's-eye view. First, an image pyramid is constructed, and the RGB panoramic image is obtained by the resize function. Figure 1 For images of / 2 size and 1 / 4 size, for each image, select a pixel point pix and use the kd number to search for the nearest neighbor point. If the pixel difference between the neighbor point and the pixel of this point is greater than the threshold t, the point is considered to be a corner point, and the t value is manually set according to the image pixel value.

[0096] With the matching relationship of feature points and the intrinsic parameter matrix of the camera, the PnP (Perspective-n-Point) algorithm can be used to obtain the translation matrix and rotation matrix, thereby realizing image fusion. i =(X i ,Y i ,Z i ), p i (x i ,y i ) can establish the following equation:

[0097] s i ·p i =K·[R|t]·P i ;

[0098] where s i Represents the scale factor, R represents the rotation matrix, and t represents the offset vector, i.e. the translation vector. The expansion yields:

[0099]

[0100] Substituting in the rotation matrix and offset vector, we can obtain the rotation matrix and offset vector. In this way, the model is built by fusing the vehicle-side sensor and the global sensor. The vehicle-side information and the global information are integrated through the target detection algorithm. By fusing the RGB panorama and the 3D point cloud, a high-precision panorama with image information is obtained. The accuracy of the map is improved through multi-sensor fusion. Through the target detection algorithm, the map provides local information of the factory area in real time. It provides decision-making information for the dispatching system to avoid traffic control. Based on real-time mapping of multi-sensor fusion, the factory environment is updated and reported in real time, providing support for AGV route optimization.

[0101] See also Figure 4 As shown, the embodiment of the present application provides a panoramic visual mapping device based on multiple sensors, including:

[0102] A point cloud map creation module 11 is used to create a three-dimensional point cloud map of the space to be mapped based on the three-dimensional point cloud data collected by the AGV, wherein the three-dimensional point cloud data is collected by a sensor carried by the AGV;

[0103] A panoramic image generation module 12 is used to fuse multiple RGB images collected by multiple camera devices to obtain an RGB panoramic image of the space to be mapped, wherein the multiple camera devices are installed at different positions in the space to be mapped, and the RGB images are collected by sensors carried by the camera devices;

[0104] An image fusion module 13, used to fuse the three-dimensional point cloud map with the RGB panoramic image to obtain a fused map;

[0105] A target detection module 14 is used to perform target detection based on the target sensor data collected by the AGV to obtain target information of the space to be mapped, wherein the target information includes location information and type information of the target object;

[0106] The information projection module 15 is used to project the target information onto the fused map to realize the construction of a panoramic map of the space to be mapped.

[0107] The image fusion module 13 may include:

[0108] A first corner point extraction submodule is used to extract corner points in the RGB panoramic image to obtain first corner points;

[0109] A second corner point extraction submodule is used to extract corner points in the three-dimensional point cloud map to obtain second corner points;

[0110] A corner point matching submodule, used for matching the first corner point with the second corner point to obtain a matching corner point;

[0111] The image fusion submodule is used to fuse the three-dimensional point cloud map with the RGB panoramic image based on the matching corner points to obtain a fused map.

[0112] The first corner point extraction submodule can be specifically used to: generate panoramas of different sizes based on the RGB panorama; for a panorama of any size, determine the nearest point of a target point in the panorama and when the pixel difference between the nearest point and the target point is greater than a preset threshold, determine the target point as a corner point, and the target point is any point in the panorama; fuse the corner points in the panoramas of different sizes to obtain the corner point in the RGB panorama as the first corner point.

[0113] The second corner point extraction submodule is specifically used to downsample the three-dimensional point cloud map to obtain a downsampled point cloud map; construct a local plane corresponding to any point in the downsampled point cloud map; construct a covariance matrix based on the centroid of the local plane and the points in the local plane; decompose the covariance matrix into eigenvalues ​​and eigenvectors; determine the normal vector of the arbitrary point from the eigenvector based on the eigenvalues; calculate the plane curvature based on the normal vector, and determine the corner point based on the plane curvature to obtain a second corner point.

[0114] The image fusion submodule is specifically used to: calculate the translation matrix and the rotation matrix based on the matching corner points; and fuse the three-dimensional point cloud map with the RGB panoramic image based on the translation matrix and the rotation matrix to obtain a fused map.

[0115] The panoramic image generation module 12 is specifically used to: respectively extract feature points of multiple RGB images collected by multiple cameras; match the feature points of multiple RGB images to obtain matching feature points; and fuse the multiple RGB images collected by multiple cameras based on the matching feature points to obtain an RGB panoramic image of the space to be constructed.

[0116] The target detection module 14 is specifically used to perform target detection based on the target sensor data collected by the AGV and using a deep learning target detection model to obtain target information of the space to be mapped;

[0117] The target objects include obstacles, pedestrians, storage locations, and various objects in goods.

[0118] It can be seen that when building a map, the embodiment of the present application fuses the three-dimensional point cloud map of the space to be mapped, which is created based on the three-dimensional point cloud data collected by the AGV, with the RGB panoramic map of the space to be mapped, which is obtained by fusing multiple RGB images collected by multiple cameras. The sensor data of the local sensors on the vehicle side and the global sensors of the space to be mapped are integrated to improve the map accuracy. Target detection is performed based on the target sensor data collected by the AGV, and the target information is projected onto the fused map to realize the construction of a panoramic map of the space to be mapped. This enriches the map information, can improve the applicability of the map in complex scenarios, and ensure the smooth execution of AGV tasks.

[0119] See also Figure 5 As shown, an embodiment of the present application discloses an electronic device 20, including a processor 21 and a memory 22; wherein the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program, and the panoramic visual mapping method based on multiple sensors disclosed in the above embodiment.

[0120] For the specific process of the above-mentioned multi-sensor based panoramic visual mapping method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0121] Furthermore, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0122] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26; wherein the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0123] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the multi-sensor based panoramic visual mapping method disclosed in the aforementioned embodiment.

[0124] For the specific process of the above-mentioned multi-sensor based panoramic visual mapping method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0125] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0126] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0127] The above is a detailed introduction to the panoramic visual mapping method, device, equipment and medium based on multi-sensor provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A panoramic visual mapping method based on multiple sensors, characterized in that: include: Creating a three-dimensional point cloud map of the space to be mapped based on the three-dimensional point cloud data collected by the AGV, wherein the three-dimensional point cloud data is collected by a sensor carried by the AGV; fusing multiple RGB images collected by multiple camera devices to obtain an RGB panoramic image of the space to be mapped, wherein the multiple camera devices are installed at different positions in the space to be mapped, and the RGB images are collected by sensors carried by the camera devices; Fusion of the three-dimensional point cloud map with the RGB panoramic image to obtain a fused map; Performing target detection based on the target sensor data collected by the AGV to obtain target information of the space to be mapped, wherein the target information includes location information and type information of the target object; The target information is projected onto the fused map to realize the construction of a panoramic map of the space to be mapped.

2. The method for panoramic visual mapping based on multiple sensors according to claim 1, characterized in that: The step of fusing the three-dimensional point cloud map with the RGB panoramic image to obtain a fused map includes: Extracting corner points in the RGB panoramic image to obtain first corner points; Extracting corner points in the three-dimensional point cloud map to obtain second corner points; Matching the first corner point and the second corner point to obtain a matching corner point; The three-dimensional point cloud map is fused with the RGB panoramic image based on the matching corner points to obtain a fused map.

3. The method for panoramic visual mapping based on multiple sensors according to claim 2, characterized in that: The step of extracting corner points from the RGB panoramic image to obtain a first corner point includes: Generating panoramic images of different sizes based on the RGB panoramic image; For a panorama of any size, determine the nearest point of a target point in the panorama and when the pixel difference between the nearest point and the target point is greater than a preset threshold, determine the target point as a corner point, and the target point is any point in the panorama; Corner points in panoramas of different sizes are fused to obtain corner points in the RGB panorama as first corner points.

4. The method for panoramic visual mapping based on multiple sensors according to claim 2, characterized in that: The step of extracting a corner point in the three-dimensional point cloud map to obtain a second corner point includes: Downsampling the three-dimensional point cloud map to obtain a downsampled point cloud map; Constructing a local plane corresponding to any point in the downsampled point cloud map; constructing a covariance matrix based on the centroid of the local plane and the points in the local plane; Decomposing the covariance matrix into eigenvalues ​​and eigenvectors; Determining a normal vector of the arbitrary point from the eigenvector based on the eigenvalue; The curvature of the plane is calculated based on the normal vector, and a corner point is determined based on the curvature of the plane to obtain a second corner point.

5. The method for panoramic visual mapping based on multiple sensors according to claim 2, characterized in that: The step of fusing the three-dimensional point cloud map with the RGB panoramic image based on the matching corner points to obtain a fused map includes: Calculate a translation matrix and a rotation matrix based on the matching corner points; The three-dimensional point cloud map is fused with the RGB panoramic image based on the translation matrix and the rotation matrix to obtain a fused map.

6. The method for panoramic visual mapping based on multiple sensors according to claim 1, characterized in that: The step of fusing the multiple RGB images collected by the multiple camera devices to obtain the RGB panoramic image of the space to be mapped includes: Extracting feature points of a plurality of RGB images captured by a plurality of camera devices respectively; Match feature points of multiple RGB images to obtain matching feature points; Based on the matching feature points, multiple RGB images collected by multiple camera devices are fused to obtain an RGB panoramic image of the space to be mapped.

7. The method for building panoramic visual maps based on multiple sensors according to any one of claims 1 to 6, characterized in that: The target detection is performed based on the target sensor data collected by the AGV to obtain the target information of the space to be mapped, including: Based on the target sensor data collected by the AGV and using a deep learning target detection model, target information of the space to be mapped is obtained; The target objects include obstacles, pedestrians, storage locations, and various objects in goods.

8. A panoramic visual mapping device based on multiple sensors, characterized in that: include: A point cloud map creation module, used to create a three-dimensional point cloud map of the space to be mapped based on the three-dimensional point cloud data collected by the AGV, wherein the three-dimensional point cloud data is collected by the sensor carried by the AGV; A panoramic image generation module, used for fusing multiple RGB images collected by multiple camera devices to obtain an RGB panoramic image of the space to be mapped, wherein the multiple camera devices are installed at different positions in the space to be mapped, and the RGB images are collected by sensors carried by the camera devices; An image fusion module, used for fusing the three-dimensional point cloud map with the RGB panoramic image to obtain a fused map; A target detection module, used to perform target detection based on the target sensor data collected by the AGV to obtain target information of the space to be mapped, wherein the target information includes location information and type information of the target object; The information projection module is used to project the target information onto the fused map to realize the construction of a panoramic map of the space to be mapped.

9. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store the computer program; The processor is used to execute the computer program to implement the multi-sensor based panoramic vision mapping method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the multi-sensor based panoramic vision mapping method according to any one of claims 1 to 7 is implemented.

Images