Storage location detection method and device, storage medium and electronic device

Abstract

The present invention provides a method and device for detecting storage locations, a storage medium and an electronic device, wherein the method comprises: establishing a three-dimensional voxel probability map based on the relative positions of a plurality of time-of-flight ToF cameras in different orientations and RGB images of the plurality of ToF cameras; determining the semantics of a preset obstacle according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacle; extracting a two-dimensional storage location of a dockable storage location according to a preset storage location model and the determined semantics of the preset obstacle, and determining a three-dimensional storage location corresponding to the two-dimensional storage location in combination with the three-dimensional voxel probability map, thereby solving the problem in the related art that the storage location can only be detected by means of two-dimensional storage location detection.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method and device for detecting a storage location, a storage medium and an electronic device. Background Art

[0002] The automatic parking system is a system that automatically plans a path to park by finding a parking space and controlling the vehicle. This system can greatly improve the automation level of the current assisted parking system, and provide technical guarantees for solving the "last mile" problem of parking and improving the comfort of users' travel. Therefore, it has become a hot research topic in unmanned driving. The system needs to detect the precise location of the parking space and obstacle perception to provide a prerequisite for driving decision-making planning. Among them, in a complex indoor three-dimensional parking environment, due to the complex structures in the vertical area (such as pipes, beams, etc.) that limit the parking suitability of vehicles of different heights and models, accurate three-dimensional parking space detection will greatly improve the safety of the automatic parking system.

[0003] However, due to the selection of sensors or method limitations, the storage location detection in the prior art is often limited to two-dimensional storage location detection, and is easily affected by factors such as sensor distance, angle resolution, and field of view, resulting in a narrow detection range and low detection accuracy.

[0004] There is currently no effective solution to the above-mentioned problems in the related art. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for detecting a storage location, a storage medium and an electronic device, so as to at least solve the problem in the related art that the storage location can only be detected by means of two-dimensional storage location detection.

[0006] According to one embodiment of the present invention, a method for detecting storage locations is provided, comprising: establishing a three-dimensional voxel probability map based on the relative positions of a plurality of time-of-flight ToF cameras in different orientations and RGB images of the plurality of ToF cameras; determining the semantics of a preset obstacle according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacle; extracting a two-dimensional storage location of a dockable storage location according to a preset storage location model and the determined semantics of the preset obstacle, and determining a three-dimensional storage location corresponding to the two-dimensional storage location in combination with the three-dimensional voxel probability map.

[0007] According to one embodiment of the present invention, a detection device for a storage location is provided, comprising: an establishment module for establishing a three-dimensional voxel probability map based on the relative positions of multiple time-of-flight ToF cameras in different orientations and RGB images of the multiple ToF cameras; a determination module for determining the semantics of a preset obstacle based on the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacle; a processing module for extracting a two-dimensional storage location of a dockable storage location based on a preset storage location model and the determined semantics of the preset obstacle, and determining a three-dimensional storage location corresponding to the two-dimensional storage location in combination with the three-dimensional voxel probability map.

[0008] According to yet another embodiment of the present invention, a storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0009] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0010] Through the present invention, a three-dimensional voxel probability map is established based on the relative positions of multiple time-of-flight ToF cameras in different orientations and the RGB images of the multiple ToF cameras, and then the semantics of the preset obstacles are determined according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacles; finally, the two-dimensional storage locations of the dockable storage locations are extracted according to the preset storage location model and the determined semantics of the preset obstacles, and the three-dimensional storage locations corresponding to the two-dimensional storage locations are determined in combination with the three-dimensional voxel probability map, thereby solving the problem in the related art that the storage locations can only be detected by means of two-dimensional storage location detection, and improving the efficiency of automatic parking. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0012] Figure 1 is a flow chart of a method for detecting a storage location according to an embodiment of the present invention;

[0013] Figure 2 is a schematic diagram of the arrangement of multiple ToF devices according to an embodiment of the present invention;

[0014] Figure 3 is a schematic diagram of a single-frame stitching point cloud according to an embodiment of the present invention;

[0015] Figure 4is a schematic diagram of a three-dimensional voxel probability fusion map according to an embodiment of the present invention;

[0016] Figure 5 is a flow chart of storage location detection according to an embodiment of the present invention;

[0017] Figure 6 is a schematic diagram of detection direction according to an embodiment of the present invention;

[0018] Figure 7 is a schematic diagram of a two-dimensional storage location detection result according to an embodiment of the present invention;

[0019] Figure 8 is a schematic diagram of a three-dimensional storage location detection result according to an embodiment of the present invention;

[0020] Fig. 9 4 is a structural block diagram of a storage location detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0023] Example 1

[0024] In this embodiment, a method for detecting a storage location is provided. Figure 1 is a flow chart of a method for detecting a storage location according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0025] Step S102, establishing a three-dimensional voxel probability map based on the relative positions of a plurality of time-of-flight ToF cameras in different directions and the RGB images of the plurality of ToF cameras;

[0026] Step S104, determining the semantics of the preset obstacle according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacle;

[0027] Step S106 , extracting the two-dimensional storage locations of the dockable storage locations according to the preset storage location model and the determined semantics of the preset obstacles, and determining the three-dimensional storage locations corresponding to the two-dimensional storage locations in combination with the three-dimensional voxel probability map.

[0028] Through the above steps S102 to S106, a three-dimensional voxel probability map is established based on the relative positions of multiple time-of-flight ToF cameras in different directions and the RGB images of the multiple ToF cameras; then the semantics of the preset obstacles are determined according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacles; finally, the two-dimensional storage locations of the dockable storage locations are extracted according to the preset storage location model and the determined semantics of the preset obstacles, and the three-dimensional storage locations corresponding to the two-dimensional storage locations are determined in combination with the three-dimensional voxel probability map, thereby solving the problem that the storage locations can only be detected by two-dimensional storage location detection, and improving the efficiency of automatic parking.

[0029] In an optional implementation of this embodiment, the method of establishing a three-dimensional voxel probability map based on the relative positions of multiple time-of-flight ToF cameras in different directions and the RGB images of the multiple ToF cameras involved in step S102 of this embodiment can be implemented in the following manner:

[0030] Step S102-11, obtaining a wide-angle point cloud formed by stitching point clouds within a preset range based on relative position calibration results between multiple time-of-flight ToF cameras in different directions;

[0031] Step S102-12, determining the camera self-motion between adjacent frames in the RGB image based on a feature point method to determine the motion change of the three-dimensional body between previous and next frames;

[0032] Step S102-13: establishing the three-dimensional voxel probability map based on the local point cloud in the wide-angle point cloud and the global position, and updating the voxel probability values ​​in the three-dimensional voxel probability map.

[0033] Furthermore, the method involved in the above step S102-12 of determining the camera self-motion between adjacent frames in the RGB image based on the feature point method to determine the motion change of the three-dimensional body between the previous and next frames can be implemented in the following manner:

[0034] Step S1, acquiring RGB images of adjacent frames through multiple ToF cameras, and extracting feature points in the RGB images of adjacent frames based on the scale-invariant feature transform (SIFT) algorithm;

[0035] Step S2, matching the extracted feature points according to a proximity algorithm;

[0036] Step S3, calculating the change of the positions of adjacent cameras according to the matching results of the feature points to obtain a change matrix, and using the change matrix as the visual odometer.

[0037] In a specific application scenario, steps S1 to S3 can be: for RGB images of adjacent frames, extract the SIFT feature points of the images respectively and use the nearest neighbor algorithm for matching. Since the feature point matching results are the basis for calculating the camera position change between adjacent frames, matches greater than N times the minimum feature distance are deleted in the matching results to ensure the reliability of the feature point matching results. The camera position transformation is expressed using the change matrix T, which consists of the rotation matrix R and the translation vector t. The rotation matrix R is calculated and determined by the rotation angles α, β, and γ of the x, y, and z axes, and the translation vector t is composed of the position difference (Δx, Δy, Δz) of the origins of the two coordinate systems. The camera extrinsic parameter change matrix is ​​calculated by minimizing the reprojection error by using the three-dimensional coordinates and image coordinates of multiple pairs of matching feature points restored using depth information. Since the position transformation is calculated using three-dimensional point projection, the scale of the translation vector can be guaranteed to be the same. Finally, the optimal parameter estimation is obtained. and Form the transformation matrix T as the result of visual odometry.

[0038] In another optional implementation of this embodiment, the method for updating the voxel probability value in the three-dimensional voxel probability map involved in step S102-13 of this embodiment may be: when the voxel has been assigned a value, the voxel probability value is subsequently updated according to the new point cloud result of each frame, wherein the updating method is:

[0039] L(n)=log[p(n) / (1-p(n))]

[0040] L_new(n)=max(min(L_old(n)+L(n),l_max),l_min)

[0041] Where L_new is the updated grid probability value, L_old is the grid probability value before the update, p(n) is the probability value divided by the point cloud of the current frame, and l_min and l_max are probability limiting thresholds.

[0042] In another optional implementation of this embodiment, the method of determining the semantics of the preset obstacle according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacle involved in step S104 of this embodiment may include:

[0043] Step S104-11, setting three height intervals for detection, wherein the three height intervals include at least one of the following: a ground detection interval, a bottom obstacle detection interval, and a high-rise obstacle detection interval;

[0044] Step S104-12, performing through filtering on the three height intervals to obtain a voxel set within a range corresponding to the three height intervals;

[0045] Step S104-13, determining the distribution interval of the preset obstacle and the geometric features of the preset obstacle according to the voxel sets corresponding to the different height intervals, and performing semantic judgment on the obstacle to determine the semantics of the preset obstacle.

[0046] For the above steps S104-11 to S104-13, in a specific application scenario, each voxel records the probability of being an obstacle, so the probability threshold θ is taken obstacle As the basis for obstacle classification, the obstacle perception results in the three-dimensional map are extracted. At the same time, according to the horizontal height h0 of the equipment from the ground, three height intervals (unit: meter) are set:

[0047]

[0048] The three height intervals are filtered through to obtain the voxel set within the corresponding range. The voxel center points of the ground detection interval are extracted to construct a point cloud, and the RANSAC (Random sample consensus) method is used to extract the ground plane parameters and record the ground height value. For the voxels in the high and low-level obstacle detection intervals, they are projected onto a two-dimensional horizontal plane. There are major obstacles in the garage environment, such as pillars, vehicles, and walls, and the distribution intervals and geometric features of obstacles with different semantics are significantly different. For example, walls appear in both high and low layers and have obvious linear features, pillars appear in both high and low layers and are rectangular, and vehicles only appear in the low layer in a rectangular shape. Therefore, the common obstacles in the scene can be quickly semantically judged based on the differences in distribution and geometric features, and the two-dimensional distribution of the corresponding semantic obstacles can be obtained.

[0049] In another optional implementation of this embodiment, the method involved in step S106 of this embodiment of extracting the two-dimensional storage location of the dockable storage location according to the preset storage location model and the semantics of the determined preset obstacles, and determining the three-dimensional storage location corresponding to the two-dimensional storage location in combination with the three-dimensional voxel probability map can be implemented in the following manner:

[0050] Step S106-11, searching for a two-dimensional storage location representing an empty storage location according to the distribution result of the two-dimensional preset obstacles on the horizontal plane in the preset storage location model;

[0051] Step S106-12, based on the plane position of the two-dimensional storage location, grow toward the top surface of the garage with a preset length as a step length to form a three-dimensional bounding box;

[0052] Step S106-13, when the number of voxels of the obstacle contained in the three-dimensional bounding box exceeds a preset threshold, it is determined that the obstacle is touched and recorded as the maximum height value in the vertical direction of the storage location;

[0053] Step S106-14, taking the height value in the ground segmentation result from the maximum height value as the ground height value, and taking the ground height value as the height limit value of the three-dimensional storage location;

[0054] Step S106-15, combining the two-dimensional storage location and the height limit value into a three-dimensional storage location.

[0055] The present application is described below with reference to the optional embodiments of the present application.

[0056] This optional embodiment provides a new solution based on multiple ToF (time of flight) cameras to replace conventional on-board ultrasonic radars and surround-view cameras, which can provide the automatic parking system with real-time three-dimensional environmental perception and storage location detection within a local range, thereby ensuring driving safety during the parking process. In order to break through the limitations of the limited horizontal field of view of the ToF camera and imitate the layout of the on-board surround-view camera, the invention proposes to deploy multiple ToF cameras in different directions to jointly observe the scene, and pre-calibrate the relative positions between the devices to splice the point clouds of multiple ToF cameras into a wide-angle point cloud, and then use the visual odometer to estimate the camera's self-motion, perform real-time three-dimensional voxel probability mapping of the local scene of the indoor parking lot, and fuse multiple frame point clouds through probability updates. Voxels in different height ranges are projected horizontally on the local map. According to the vertical height distribution characteristics of common obstacles in parking lots, semantic results such as walls, columns, and vehicles are assigned to the obstacles. Combining the storage location model hypothesis with the top and ground structures extracted from the three-dimensional map information, a real-time detection method for three-dimensional storage locations is proposed. Automatic parking is assisted by providing three-dimensional environmental perception and storage location detection information within a local range.

[0057] The method steps of the technical solution adopted in this optional embodiment include:

[0058] Step S202, deploying multiple ToF cameras in different directions, and stitching the small-range point cloud into a wide-angle point cloud based on the relative position calibration results between the cameras;

[0059] Step S204, using the RGB image of the ToF camera, using the feature point method to estimate the 6-DOF self-motion of the camera device, and calculating the 3D rigid body motion change between the previous and next frames;

[0060] Step S206, three-dimensional voxel probability mapping is performed based on the local point cloud and the global position, and multi-frame point cloud data is fused by updating the voxel probability value to eliminate dynamic obstacles and reduce the influence of noise points;

[0061] Step S208, by setting different height intervals, obtaining interval through-filter voxels, and quickly distinguishing obstacle semantics and extracting ground structures based on the vertical height distribution characteristics of common obstacles and structures in parking lots;

[0062] Step S210, based on the assumption of the storage location model and the storage location standard parameters, extract the two-dimensional position of the available docking space, and detect the three-dimensional storage location by combining the search limit height in the vertical direction of the corresponding position of the three-dimensional grid map;

[0063] Step S212, selecting a suitable parking location based on vehicle parameters, and obtaining the target parking location and passable range perception results.

[0064] For the above step S202, the ToF camera is limited by the horizontal field of view and it is difficult to perceive the wide-angle range around the vehicle body. By imitating the layout of the on-board surround view camera, multiple ToF cameras with different orientations are deployed to jointly observe the scene. Based on the pre-calibration results of the relative positions between the devices, the point clouds of multiple ToF cameras are spliced ​​into a wide-angle point cloud.

[0065] Therefore, in this optional embodiment, it is preferred to use Kinect-V2 produced by Microsoft as a ToF camera as an experimental instrument. Figure 2 : is a schematic diagram of the arrangement of multiple ToF devices according to an embodiment of the present invention, and the camera is fixed on the platform in the manner as shown in FIG. Figure 2 As shown in the figure, the surround joint observation is realized, and there is a certain common field of view between adjacent cameras to ensure the continuity of stitching. When multiple Kinect-V2 cameras are connected, it is necessary to ensure that the bandwidth of the USB3.0 interface and the control bus is sufficient.

[0066] Among them, Kinect-V2 contains color and infrared cameras, which respectively obtain RGB and depth images (compose RGB-D data). The original depth data contains a lot of noise, and the bilateral filtering method is used to filter out the noise points while maintaining clearer boundary information. Due to the different camera positions, the image is first aligned according to the camera parameters. Then, the point cloud restored by the depth map and camera parameters is converted to the assumed coordinate system according to the pre-calibrated inter-camera variation parameters, and all point clouds are spliced ​​into a wide-angle point cloud. The spliced ​​point cloud is as follows: Figure 3 As shown, Figure 3 4 is a schematic diagram of a single-frame stitching point cloud according to an embodiment of the present invention.

[0067] For step S204, in order to perceive a wider range, combined with the characteristics of ToF camera data, RGB images taken at different times are used to estimate the camera self-motion between adjacent frames using the feature point method to build a visual odometer. In order to adapt to the complex changes in the lighting environment, the more robust SIFT (Scale-Invariant Feature Transform) algorithm is selected to extract feature points on the RGB image and match them.

[0068] First, feature point matching: For RGB images of adjacent frames, SIFT feature points of the images are extracted and matched using the nearest neighbor algorithm. Since the feature point matching results are the basis for calculating the camera position change between adjacent frames, in the matching results, matches greater than N times the minimum feature distance are deleted to ensure the reliability of the feature point matching results.

[0069] Then, the transformation matrix is ​​calculated: the change matrix T is used to express the position transformation of the camera, which is composed of the rotation matrix R and the translation vector t. The rotation matrix R is determined by the rotation angles α, β, γ of the x, y, and z axes, and the translation vector t is composed of the position difference (Δx, Δy, Δz) of the origins of the two coordinate systems. The camera extrinsic parameter change matrix is ​​calculated by minimizing the reprojection error by using the three-dimensional coordinates and image coordinates of multiple pairs of matching feature points restored using depth information. Since the position transformation is calculated using three-dimensional point projection, the scale of the translation vector can be guaranteed to be the same. Finally, the optimal parameter estimation is obtained. and Form the transformation matrix T as the result of visual odometry.

[0070] For the above step S206, the three-dimensional voxel map uses a probability method to describe the possibility of the voxel state being an obstacle point. Within the probability limit, the higher the probability value, the greater the probability that the voxel is an obstacle. By continuously dividing the space into sub-voxels along the three main axes of x, y, and z, a hierarchical data structure is achieved, in which the smallest voxel boundary size represents the resolution of the map, and the voxel indicates whether the area is occupied and records the probability value of the obstacle. Combined with this space division feature, the octree is used to manage the spatial voxel elements, and efficient operations such as query, insertion, deletion, and modification of internal nodes can be implemented.

[0071] First, determine the grid state: according to the spliced ​​wide-angle point cloud and motion estimation of each frame, the inserted point cloud after position transformation is attributed to the nearest intersecting voxel, and a ray is constructed from the current position center and each point cloud. The terminal of the ray corresponds to the state of hitting the obstacle, and the voxel passed by the middle of the ray corresponds to the state of not hitting the obstacle. Based on this, the voxels in the scanning range are divided into two sets: hit and miss. For the initialization of the unobserved voxels, the probability value is assigned as p hit 、p miss .

[0072] Then the probability is updated. The update method is: if the voxel has been assigned a value, the voxel probability value is updated according to the new point cloud result of each frame. The update method is:

[0073]

[0074] L new (n)=max(min(L old(n)+L(n),l max ),l min )

[0075] Where L new is the updated grid probability value, L old is the grid probability value before updating, p(n) is the probability value divided by the point cloud of the current frame, l min With l max The probability threshold is set to ensure the number of voxel state changes. The update strategy ensures that the voxel map can increase the probability value of the target fixed obstacle from the mathematical model, thereby eliminating the ghost of dynamic obstacles and eliminating the influence of spatial noise points on the map.

[0076] It should be noted that in the process of mapping, nodes within a certain distance threshold from the current position are retained. By limiting the spatial range, the number of nodes is reduced to ensure the efficiency of map building and updating. At the same time, areas with excessive error accumulation in the process of mapping are removed to ensure detection accuracy. The three-dimensional voxel probability fusion mapping results are as follows: Figure 4 As shown, Figure 4 is a schematic diagram of a three-dimensional voxel probability fusion map according to an embodiment of the present invention.

[0077] For the above step S208, each voxel records the probability of being an obstacle, so the probability threshold θ obstacle As the basis for obstacle classification, the obstacle perception results in the three-dimensional map are extracted. At the same time, according to the horizontal height h0 of the equipment from the ground, three height intervals (unit: meter) are set:

[0078]

[0079] The three height intervals are filtered through to obtain the voxel set within the corresponding range. The voxel center points of the ground detection interval are extracted to construct a point cloud, and the RANSAC (Random sample consensus) method is used to extract the ground plane parameters and record the ground height value. For the voxels in the high and low-level obstacle detection intervals, they are projected onto a two-dimensional horizontal plane. There are major obstacles in the garage environment, such as pillars, vehicles, and walls, and the distribution intervals and geometric features of obstacles with different semantics are significantly different. For example, walls appear in both high and low layers and have obvious linear features, pillars appear in both high and low layers and are rectangular, and vehicles only appear in the low layer in a rectangular shape. Therefore, the common obstacles in the scene can be quickly semantically judged based on the differences in distribution and geometric features, and the two-dimensional distribution of the corresponding semantic obstacles can be obtained.

[0080] For the above step S210, based on Figure 5 , Figure 5This is a flow chart of the detection of storage locations according to an embodiment of the present invention, and the empty area representing the storage location is found based on the two-dimensional obstacle distribution results on the horizontal plane. In a standard underground garage environment, the parking spaces are often designed to be perpendicular or parallel to the wall. On the other hand, the direction of the parking lot road is usually perpendicular or parallel to the storage location, and the pillars are often distributed along the road direction. Therefore, the detection method of the two-dimensional storage location is based on the following assumptions:

[0081] (1) If there is no wall in the local parking environment, the parking position is detected along the column line, such as Figure 6 The dotted line projection direction in is, where Figure 6 is a schematic diagram of detection direction according to an embodiment of the present invention.

[0082] (2) If there is a wall in the local parking environment, the parking position is detected in the vertical direction of the wall, such as Figure 6 The projection direction of the solid line.

[0083] To facilitate detection, the 2D detection image is rotated with the center of the vehicle as the origin until the image's vertical axis is parallel to the direction line. Then the minimum envelope rectangle of each obstacle is projected to the detection direction, and the overlapping area rectangles are merged. The empty rectangular area where the distance between adjacent obstacle rectangles is greater than the minimum parking space standard, such as 6.0 meters for parallel and 2.4 meters for vertical, is taken as the parking area. The 2D storage location detection effect after rotation is as follows Figure 7 As shown, Figure 7 2. It is a schematic diagram of a two-dimensional storage location detection result according to an embodiment of the present invention.

[0084] Based on the existing two-dimensional storage location, based on the plane position of the two-dimensional storage location, it grows towards the top of the garage with △D as the step length to form a three-dimensional bounding box. If the number of obstacle voxels contained in the three-dimensional bounding box exceeds the set threshold, it is considered to have touched an obstacle, and the maximum height value h above the storage location in the vertical direction is recorded. ceiling ; Take the height value in the ground segmentation result as the ground height h floor , determine the height limit of the three-dimensional storage location. Finally, the horizontal two-dimensional storage location and the vertical limit height are taken to form a three-dimensional bounding box as the three-dimensional storage location that can be parked. The three-dimensional storage location detection result is as follows Figure 8 As shown, Figure 8 3D storage location detection result according to an embodiment of the present invention.

[0085] It can be seen from the above steps S202 to S212 that in this optional embodiment, full use is made of sensor data, and data is fused by establishing a local probability grid map, thereby expanding the perception range and reducing the impact of various types of noise; real-time environmental perception and three-dimensional storage location detection results are achieved, providing a driving decision-making basis for the automatic parking system.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0087] Example 2

[0088] In this embodiment, a detection device for a storage location is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0089] Fig. 9 is a structural block diagram of a storage location detection device according to an embodiment of the present invention, such as Fig. 9 As shown, the device includes: an establishing module 902, which is used to establish a three-dimensional voxel probability map based on the relative positions of multiple time-of-flight ToF cameras in different orientations and the RGB images of the multiple ToF cameras; a determining module 904, which is coupled to the establishing module 902, and is used to determine the semantics of the preset obstacle according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacle; a processing module 906, which is coupled to the determining module 904, and is used to extract the two-dimensional storage location of the dockable storage location according to the preset storage location model and the determined semantics of the preset obstacle, and determine the three-dimensional storage location corresponding to the two-dimensional storage location in combination with the three-dimensional voxel probability map.

[0090] Optionally, the establishment module in this embodiment may further include: an acquisition unit, used to obtain a wide-angle point cloud formed by splicing point clouds within a preset range from the relative position calibration results between multiple time-of-flight ToF cameras in different orientations; a determination unit, used to determine the camera self-motion between adjacent frames in the RGB image based on a feature point method to determine the motion change of the three-dimensional body between previous and next frames; a processing unit, used to establish the three-dimensional voxel probability map based on the local point cloud and the global position in the wide-angle point cloud, and update the voxel probability value in the three-dimensional voxel probability map.

[0091] Optionally, the above-mentioned determination module may further include: a first processing unit, used to acquire RGB images of adjacent frames through multiple ToF cameras, and extract feature points in the RGB images of adjacent frames based on the scale-invariant feature transform SIFT algorithm; a matching unit, used to match the extracted feature points according to the proximity algorithm; a second processing unit, used to calculate the change of the position of adjacent cameras based on the matching results of the feature points to obtain a change matrix, and use the change matrix as a visual odometer.

[0092] Optionally, the processing module in this embodiment is further used to update the voxel probability value according to the new point cloud result of each frame after the voxel has been assigned a value; wherein the updating method is:

[0093]

[0094] L new (n)=max(min(L old (n)+L(n),l max ),l min )

[0095] Where L new is the updated grid probability value, L old is the grid probability value before updating, p(n) is the probability value divided by the point cloud of the current frame, l min With l max Define a threshold for the probability.

[0096] Optionally, the determination module 904 in this embodiment includes: a setting unit, used to set three height intervals for detection, wherein the three height intervals include at least one of the following: a ground detection interval, a bottom obstacle detection interval, and a high-level obstacle detection interval; a third processing unit, used to perform through filtering on the three height intervals to obtain a voxel set within a range corresponding to the three height intervals; and a fourth processing unit, used to determine the distribution interval and geometric features of a preset obstacle based on the voxel set corresponding to different height intervals, and perform semantic judgment on the obstacle to determine the semantics of the preset obstacle.

[0097] Optionally, the processing module 906 in this embodiment includes: a search unit, used to search for a two-dimensional storage location representing an empty storage location based on the distribution results of the two-dimensional preset obstacles in the horizontal plane in the preset storage location model; a fifth processing unit, used to grow toward the top surface of the garage with a preset length as a step length based on the plane position of the two-dimensional storage location to form a three-dimensional bounding box; a sixth processing unit, used to determine that the obstacle is touched when the number of voxels of the obstacle contained in the three-dimensional bounding box exceeds a preset threshold, and record it as the maximum height value in the vertical direction of the storage location; a seventh processing unit, used to take the height value in the ground segmentation result from the maximum height value as the ground height value, and use the ground height value as the height limit value of the three-dimensional storage location; an eighth processing unit, used to combine the two-dimensional storage location and the height limit value to form a three-dimensional storage location.

[0098] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0099] Example 3

[0100] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0101] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0102] S1, establishing a three-dimensional voxel probability map based on the relative positions of a plurality of time-of-flight ToF cameras in different directions and RGB images of the plurality of ToF cameras;

[0103] S2, determining the semantics of the preset obstacle according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacle;

[0104] S3, extracting the two-dimensional storage location of the dockable storage location according to the preset storage location model and the semantics of the determined preset obstacles, and determining the three-dimensional storage location corresponding to the two-dimensional storage location in combination with the three-dimensional voxel probability map.

[0105] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0106] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0107] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0108] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0109] S1, establishing a three-dimensional voxel probability map based on the relative positions of a plurality of time-of-flight ToF cameras in different directions and RGB images of the plurality of ToF cameras;

[0110] S2, determining the semantics of the preset obstacle according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacle;

[0111] S3, extracting the two-dimensional storage location of the dockable storage location according to the preset storage location model and the semantics of the determined preset obstacles, and determining the three-dimensional storage location corresponding to the two-dimensional storage location in combination with the three-dimensional voxel probability map.

[0112] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0113] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting a storage location, characterized in that: include: A three-dimensional voxel probability map is established based on the relative positions of multiple time-of-flight ToF cameras in different orientations and the RGB images of the multiple ToF cameras, including: obtaining a wide-angle point cloud by stitching point clouds within a preset range from relative position calibration results of multiple time-of-flight ToF cameras in different orientations; determining the camera self-motion between adjacent frames in the RGB image based on a feature point method to determine the motion change of the three-dimensional body between the previous and next frames; establishing the three-dimensional voxel probability map based on the local point cloud and the global position in the wide-angle point cloud, and updating the voxel probability value in the three-dimensional voxel probability map; Determining the semantics of the preset obstacle according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacle, including: setting three height intervals for detection, wherein the three height intervals include at least one of the following: a ground detection interval, a bottom obstacle detection interval, and a high-level obstacle detection interval; performing through filtering on the three height intervals to obtain a voxel set within a range corresponding to the three height intervals; determining the distribution interval and geometric characteristics of the preset obstacle according to the voxel set corresponding to the three height intervals, and performing semantic judgment on the obstacle to determine the semantics of the preset obstacle; The two-dimensional storage location of the dockable storage location is extracted according to the preset storage location model and the semantics of the determined preset obstacles, and the three-dimensional storage location corresponding to the two-dimensional storage location is determined in combination with the three-dimensional voxel probability map, including: searching for the two-dimensional storage location representing the empty storage location according to the distribution result of the two-dimensional preset obstacles in the horizontal plane in the preset storage location model; based on the plane position of the two-dimensional storage location, growing toward the top surface of the garage with a preset length as a step length to form a three-dimensional bounding box; when the number of voxels of the obstacle contained in the three-dimensional bounding box exceeds a preset threshold, it is determined that the obstacle is touched and recorded as the maximum height value in the vertical direction of the storage location; the height value in the ground segmentation result is taken from the maximum height value as the ground height value, and the ground height value is used as the height limit value of the three-dimensional storage location; the two-dimensional storage location and the height limit value form the three-dimensional storage location.

2. The method according to claim 1, characterized in that The method of determining the camera self-motion between adjacent frames in the RGB image based on the feature point method to determine the motion change of the three-dimensional body between the previous and next frames includes: Acquire RGB images of adjacent frames through the multiple ToF cameras, and extract feature points in the RGB images of adjacent frames based on a scale-invariant feature transform (SIFT) algorithm; Match the extracted feature points according to the proximity algorithm; The changes in the positions of adjacent cameras are calculated according to the matching results of the feature points to obtain a change matrix, and the change matrix is ​​used as the visual odometer.

3. The method according to claim 1, characterized in that The updating of the voxel probability values ​​in the three-dimensional voxel probability map includes: When the voxel has been assigned a value, the voxel probability value is subsequently updated according to the new point cloud result of each frame, wherein the updating method is: L new (n)=max(min(L old (n)+L(n),l max ),L min ) Where L new is the updated grid probability value, L old is the grid probability value before updating, p(n) is the probability value divided by the point cloud of the current frame, l min With l max Define a threshold for the probability.

4. A storage location detection device, characterized in that: include: An establishment module is used to establish a three-dimensional voxel probability map based on the relative positions of multiple time-of-flight ToF cameras in different orientations and the RGB images of the multiple ToF cameras, including: obtaining the relative position calibration results between the multiple time-of-flight ToF cameras in different orientations to stitch the point clouds within a preset range into a wide-angle point cloud; determining the camera self-motion between adjacent frames in the RGB image based on the feature point method to determine the motion change of the three-dimensional body between the previous and next frames; establishing the three-dimensional voxel probability map based on the local point cloud and the global position in the wide-angle point cloud, and updating the voxel probability value in the three-dimensional voxel probability map; A determination module, for determining the semantics of the preset obstacle according to the three-dimensional voxel probability map and the vertical height distribution characteristics of the preset obstacle, comprising: setting three height intervals for detection, wherein the three height intervals include at least one of the following: a ground detection interval, a bottom obstacle detection interval, and a high-level obstacle detection interval; performing through filtering on the three height intervals to obtain a voxel set within a range corresponding to the three height intervals; determining the distribution interval and geometric characteristics of the preset obstacle according to the voxel set corresponding to the three height intervals, and performing semantic judgment on the obstacle to determine the semantics of the preset obstacle; A processing module is used to extract the two-dimensional storage location of the dockable storage location according to the preset storage location model and the semantics of the determined preset obstacles, and determine the three-dimensional storage location corresponding to the two-dimensional storage location in combination with the three-dimensional voxel probability map, including: searching for the two-dimensional storage location representing the empty storage location according to the distribution result of the two-dimensional preset obstacles in the horizontal plane in the preset storage location model; based on the plane position of the two-dimensional storage location, growing toward the top surface of the garage with a preset length as a step length to form a three-dimensional bounding box; when the number of voxels of the obstacle contained in the three-dimensional bounding box exceeds a preset threshold, it is determined that the obstacle is touched and recorded as the maximum height value in the vertical direction of the storage location; the height value in the ground segmentation result is taken from the maximum height value as the ground height value, and the ground height value is used as the height limit value of the three-dimensional storage location; the two-dimensional storage location and the height limit value form the three-dimensional storage location.

5. The device according to claim 4, characterized in that The determination module comprises: A first processing unit, configured to acquire RGB images of adjacent frames through the multiple ToF cameras, and extract feature points in the RGB images of adjacent frames based on a scale-invariant feature transform (SIFT) algorithm; A matching unit, used for matching the extracted feature points according to a proximity algorithm; The second processing unit is used to calculate the change of the position of the adjacent cameras according to the matching result of the feature points to obtain a change matrix, and use the change matrix as the visual odometer.

6. The device according to claim 4, characterized in that The processing module is also used to update the voxel probability value according to the new point cloud result of each frame after the voxel has been assigned a value; wherein the updating method is: L new (n)=max(min(L old (n)+L(n),l max ),L min ) Where L new is the updated grid probability value, L old is the grid probability value before updating, p(n) is the probability value divided by the point cloud of the current frame, l min With l max Define a threshold for the probability.

7. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 3 when executed.

8. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 3.

Images