Omnidirectional environment sensing method for movable platform, and platform, medium and product

Abstract

An omnidirectional environment sensing method for a movable platform (1,500), and a platform, a medium and a product. The omnidirectional environment sensing method comprises: during the movement of a movable platform (1,500), collecting a first image by means of a forward-sensing device (10), and collecting second images by means of circumferential-sensing devices (20); determining first sensing information corresponding to the first image, and determining second sensing information corresponding to each second image; on the basis of the first sensing information and the second sensing information, determining a global occupancy map, wherein the global occupancy map comprises attribute information of each position point in an omnidirectional environment where the movable platform (1,500) is located; and on the basis of the global occupancy map, executing a corresponding control task (S104). A control task is executed on the basis of a global occupancy map, such that repeated computation of a surrounding environment is reduced, information is fully shared, computational load is reduced, computing resources are saved on, and the environment sensing capability, path planning efficiency and control task execution effect of the movable platform (1, 500) during movement can further be improved.

Description

Omnidirectional environmental sensing methods, platforms, media, and products for mobile platforms

[0001] This application claims priority to Chinese Patent Application No. 202410502880.6, filed on April 22, 2024, entitled “Omnidirectional Environmental Sensing Method, Platform, Medium and Product for Mobile Platform”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of automatic control technology, and in particular to an omnidirectional environmental sensing method, platform, medium and product for a mobile platform. Background Technology

[0003] Currently, sensing devices are installed at appropriate locations on mobile platforms to perceive the surrounding environment during platform movement. Different sensing algorithms are used for different control tasks during platform movement. For example, road elevation estimation requires sensing undulations on the road surface within a certain distance for suspension adjustment. Similarly, obstacle detection requires sensing obstacles within a certain distance for obstacle avoidance. Furthermore, 360° visualization requires visualizing the surrounding environment.

[0004] However, processing different control tasks separately means that different perception algorithms will repeatedly calculate parts of the surrounding environment, fail to fully share information, have a large amount of computation, and waste computing resources. Summary of the Invention

[0005] This application provides an omnidirectional environmental perception method, platform, medium, and product for a mobile platform, which reduces redundant calculations of the surrounding environment, fully shares information, reduces computational load, and saves computing resources.

[0006] Firstly, this application provides an omnidirectional environment perception method for a mobile platform, comprising:

[0007] During the movement of the mobile platform, a first image is acquired through a forward sensing device and a second image is acquired through a circumferential sensing device; the forward sensing device is used to acquire images of the environment in front of the mobile platform, and there are multiple circumferential sensing devices used to acquire images of the environment around the mobile platform.

[0008] First sensing information corresponding to the first image is determined, and second sensing information corresponding to each of the second images is determined; the first sensing information includes depth information and elevation information of each point in the target image related to the first image; the second sensing information includes depth information and elevation information of each point in the second image.

[0009] Based on the first and second perception information, a global occupancy map is determined; the global occupancy map includes attribute information of each location point in the omnidirectional environment in which the mobile platform is located; the attribute information includes obstacle information and elevation information.

[0010] The corresponding control tasks are performed based on the global occupancy map; the control tasks include at least one of obstacle detection, road surface elevation estimation, and 360° visualization.

[0011] In some embodiments, when the forward-looking sensing device is a forward-looking binocular camera, determining the first sensing information corresponding to the first image includes:

[0012] The first image is input into the trained first perception model, and the first perception model is used to determine the first perception information; wherein, the training process of the first perception model includes:

[0013] During the movement of the sample mobile platform, the pose information of the sample mobile platform is acquired, and binocular image samples are collected by a forward-looking binocular camera and environmental data around the sample mobile platform are collected by a point cloud sensor to obtain point cloud information.

[0014] A preset 3D reconstruction algorithm is used to generate training labels corresponding to the binocular image samples based on the point cloud information and the pose information; the training labels include the depth information and elevation information of each point in the fitted image corresponding to the binocular image samples;

[0015] Using the training labels corresponding to the binocular image samples as supervision data, the first perception model is iteratively trained based on the binocular image samples;

[0016] In response to meeting the preset training convergence condition, the first perception model that meets the training convergence condition is determined as the first perception model that has been trained.

[0017] In some embodiments, the step of generating training labels corresponding to the binocular image samples based on the point cloud information and the pose information using a preset 3D reconstruction algorithm includes:

[0018] Using a preset 3D reconstruction algorithm, perform the following operations:

[0019] The point cloud information of multiple consecutive frames is stitched together according to the pose information to obtain the first three-dimensional point cloud map;

[0020] The noise in the first 3D point cloud map is filtered out using a consistency loss function to obtain a second 3D point cloud map.

[0021] Training labels corresponding to the stereo image samples are generated based on the second 3D point cloud map.

[0022] In some embodiments, the location points are represented in cell form; determining the global occupancy map based on the first sensing information and the second sensing information includes:

[0023] Based on the depth information of each point in the first perception information, the obstacle information corresponding to the first image is determined, and based on the depth information of each point in the second perception information corresponding to each second image, the obstacle information corresponding to each second image is determined respectively.

[0024] For each cell in the global occupied map, based on the cell's position, obstacle information corresponding to the first image, and obstacle information corresponding to each of the second images, obstacle information of the cell is determined; and based on the cell's position, first perception information corresponding to the first image, and second perception information corresponding to each of the second images, elevation information of the cell is determined.

[0025] In some embodiments, determining the obstacle information of the cell based on the cell's position, the obstacle information corresponding to the first image, and the obstacle information corresponding to each of the second images includes:

[0026] If the cell is a forward cell, then the obstacle information of the cell is determined based on the obstacle information corresponding to the first image and the obstacle information corresponding to the second image at the forward position;

[0027] If the cell is a backward cell or a side cell, then the obstacle information of the cell is determined based on the obstacle information corresponding to the second image at the corresponding position.

[0028] In some embodiments, determining the elevation information of the cell based on the cell's position, the first sensing information corresponding to the first image, and the second sensing information corresponding to each of the second images includes:

[0029] If the cell is a forward cell, the first elevation information in the first sensing information is fused with the second elevation information in the second sensing information corresponding to the second image at the forward position to obtain the elevation information of the cell; both the first elevation information and the second elevation information are the elevation information of the point corresponding to the cell;

[0030] If the cell is a backward cell or a side cell, the elevation information of the point corresponding to the cell in the second perception information corresponding to the second image at the corresponding position is fused to obtain the elevation information of the cell.

[0031] In some embodiments, multiple global occupancy maps with different resolutions may be displayed at the same time.

[0032] The step of determining the global occupancy map based on the first sensing information and the second sensing information includes:

[0033] For each cell in the first global occupied map, if the cell is a forward cell, the attribute information of the cell is determined based on the obstacle information and elevation information corresponding to the first image; if the cell is a backward cell or a side cell, the attribute information of the cell is determined based on the obstacle information and elevation information corresponding to the second image at the corresponding position.

[0034] For each cell in the second global occupancy map, the attribute information of the cell is determined based on the obstacle information and elevation information corresponding to the second image at the corresponding position of the cell;

[0035] Among them, the resolution of the first global occupied map is higher than that of the second global occupied map, and the area of ​​the first global occupied map is smaller than that of the second global occupied map.

[0036] In some embodiments, when the control task is 360° visualization, executing the corresponding control task based on the global occupancy map includes:

[0037] Based on the obstacle information and elevation information of each location point in the global occupancy map, the environment around the mobile platform and the road surface under the chassis are visualized, and the obstacles and road surface elevations are highlighted.

[0038] In some embodiments, after determining the global occupancy map based on the first sensing information and the second sensing information, the method further includes at least one of the following:

[0039] The location information of the mobile platform at the current moment is stored in correspondence with the elevation information of each location point in the global occupancy map. When the platform enters the location indicated by the location information again, the road surface elevation estimation task is performed based on the elevation information of each location point in the global occupancy map stored in correspondence with the location information.

[0040] The road surface elevation sharing information is sent to other mobile platforms that are communicatively connected to the mobile platform; the road surface elevation sharing information includes the current location information and the elevation information of each location point in the global occupied map.

[0041] Secondly, this application provides a portable platform, including: a processor and a memory communicatively connected to the processor;

[0042] The memory stores computer-executed instructions;

[0043] The processor executes computer execution instructions stored in the memory to implement the omnidirectional environment perception method for a mobile platform as described in any of the first aspects.

[0044] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the omnidirectional environmental perception method of the mobile platform described in any of the first aspects.

[0045] Fourthly, this application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the omnidirectional environment perception method for a mobile platform as described in any of the first aspects.

[0046] The omnidirectional environmental perception method, platform, medium, and product for mobile platforms provided in this application include forward-facing and circumferential sensing devices on the mobile platform. During the movement of the mobile platform, images collected by the forward-facing and circumferential sensing devices can determine the corresponding perception information, thereby achieving omnidirectional perception of the environment surrounding the mobile platform. By integrating the perception information from the first and second images, a global occupancy map is determined, including attribute information of each location point, such as obstacle information and elevation information, thus reconstructing the environment in which the mobile platform is located. By combining the global occupancy map to perform control tasks, redundant calculations of the surrounding environment are reduced, information is fully shared, the amount of computation is reduced, and computing resources are saved. It can also improve the environmental perception capability, path planning efficiency, and control task execution effect of the mobile platform during movement. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] Figure 1 is a schematic diagram illustrating an application scenario according to an exemplary embodiment;

[0049] Figure 2 is a flowchart illustrating an omnidirectional environment perception method for a mobile platform according to an exemplary embodiment;

[0050] Figure 3 is a flowchart illustrating an omnidirectional environment perception method for a mobile platform according to another exemplary embodiment;

[0051] Figure 4 is a flowchart illustrating an omnidirectional environment perception method for a mobile platform according to yet another exemplary embodiment;

[0052] Figure 5 is a flowchart illustrating a training process for a first perception model according to an exemplary embodiment;

[0053] Figure 6 is a schematic diagram of the structure of an omnidirectional environmental sensing device for a mobile platform according to an exemplary embodiment;

[0054] Figure 7 is a schematic diagram of the structure of a mobile platform according to an exemplary embodiment.

[0055] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0057] The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. In the following descriptions of embodiments, "a plurality of" means two or more, unless otherwise explicitly defined.

[0058] Currently, sensing devices are installed at appropriate locations on mobile platforms to perceive the surrounding environment during platform movement. Different sensing algorithms are used for different control tasks during platform movement. Taking a mobile platform moving on a road surface as an example, control tasks such as road elevation estimation, obstacle detection, and 360° visualization are performed to assist movement. Road elevation estimation requires sensing undulations on the road surface within a certain distance for suspension adjustment. Obstacle detection requires sensing obstacles within a certain distance for obstacle avoidance. 360° visualization displays the surrounding environment.

[0059] Taking a vehicle as an example of a mobile platform, in a vehicle driving scenario, obstacle perception and road surface elevation estimation are two related tasks. Processing them separately would lead to redundant processing of road structure information and neglect the relationship between road surface undulations and obstacles. Obstacle perception and 360° visualization are also related tasks; processing them separately may result in distorted visualization images and inconsistent results. For example, road surface elevation estimation may be affected by obstacles ahead, leading to misidentification of outliers. Different control tasks have different requirements for perception characteristics, making it difficult to unify various perception schemes. For instance, road surface elevation estimation needs to perceive undulations within 10 meters ahead with an accuracy within 1 cm. Obstacle detection needs to detect obstacles within 120 meters ahead, but the accuracy requirement is not high. 360° visualization requires a complete reconstruction of the surrounding environment, including the chassis space, with centimeter-level accuracy. Therefore, it is difficult to handle these different requirements simultaneously with a single solution.

[0060] However, processing different control tasks separately means that different perception algorithms will repeatedly calculate parts of the surrounding environment, fail to fully share information, have a large amount of computation, and waste computing resources.

[0061] To address the above technical issues, this application proposes a unified environmental perception scheme that enables mobile platforms to reconstruct their surroundings while simultaneously meeting the requirements for obstacle detection, road surface elevation estimation, and 360° visualization. By jointly processing different control tasks, relevant information can be effectively utilized, improving the quality of perception results while reducing computational load.

[0062] In this application, a forward sensing device and a circumferential sensing device are provided on the mobile platform. During the movement of the mobile platform, the images collected by the forward sensing device and the circumferential sensing device can determine the corresponding sensing information, thereby perceiving the environment around the mobile platform in all directions. By integrating the sensing information of the first image and the second image, a global occupancy map is determined, which includes the attribute information of each location point, such as obstacle information and elevation information, thereby realizing the reconstruction of the environment in which the mobile platform is located. By combining the global occupancy map to perform control tasks, the redundant calculation of the surrounding environment is reduced, information is fully shared, the amount of calculation is reduced, and computing resources are saved. It can also improve the environmental perception capability, path planning efficiency and control task execution effect of the mobile platform during the movement process.

[0063] The omnidirectional environmental perception method for mobile platforms provided in this application is implemented by an omnidirectional environmental perception device for the mobile platform. This device is included in a device, which can be the mobile platform itself or a control terminal for controlling the mobile platform. The mobile platform includes, but is not limited to, intelligent vehicles, drones, unmanned vessels, and ground robots. The control terminal can include, but is not limited to, mobile phones, computers, remote controls, wearable devices (watches or bracelets), and controllers.

[0064] Figure 1 is a schematic diagram illustrating an application scenario according to an exemplary embodiment. As shown in Figure 1, the application scenario includes: a mobile platform 1.

[0065] A forward-facing sensing device 10 is installed on the mobile platform 1, and this forward-facing sensing device 10 is used to acquire images of the forward environment of the mobile platform 1. Exemplarily, the forward-facing sensing device 10 includes, but is not limited to, a camera or a point cloud sensor. The point cloud sensor includes, but is not limited to, a lidar sensor, a millimeter-wave radar sensor, and an ultrasonic radar sensor. Taking an intelligent vehicle as an example, it is equipped with a forward-facing binocular camera to acquire images of the forward environment. The forward environment refers to the environment located in the direction of movement of the mobile platform.

[0066] The mobile platform 1 is also equipped with multiple circumferential sensing devices 20, which are used to acquire images of the circumferential environment of the mobile platform 1. Exemplarily, the circumferential environment includes the forward, backward, left, and right environments. Accordingly, the circumferential sensing devices 20 include, but are not limited to, cameras or point cloud sensors. The number and installation location of the circumferential sensing devices 20 can be set according to actual needs, but must meet the condition of being able to completely acquire images of the circumferential environment of the mobile platform. For example, the number may be four, and they may be installed around the perimeter of the mobile platform. Taking a smart vehicle as an example, it may be equipped with four monocular cameras, respectively mounted on the left and right rearview mirrors and the front and rear bumpers, to acquire images of the circumferential environment. The monocular cameras may be fisheye monocular cameras.

[0067] The mobile platform 1 also includes a processing unit 30, which is communicatively connected to the forward sensing device 10 and each of the peripheral sensing devices 20. The processing unit 30 acquires images collected by the forward sensing device 10 and each of the peripheral sensing devices 20, and performs omnidirectional environmental perception based on the acquired images to control the movement of the mobile platform 1. Alternatively, the processing unit 30 can send the acquired images to a control terminal, enabling the control terminal to perform omnidirectional environmental perception based on the acquired images to control the movement of the mobile platform 1. Exemplarily, the processing unit 30 can be a processor.

[0068] In one application scenario, during the movement of the mobile platform 1, a first image of the forward environment is obtained by the forward sensing device 10, and a second image of the circumferential environment is obtained by the circumferential sensing device 20. The processing unit 30 determines the sensing information from the first and second images respectively, and then determines the global occupancy map to realize the reconstruction of the omnidirectional environment. Then, the control task is executed to assist the movement.

[0069] It should be noted that, for the sake of illustration, Figure 1 uses one forward sensing device and one circumferential sensing device as an example. Those skilled in the art will know that the number and installation location of the forward sensing device and the circumferential sensing device can be set according to actual needs.

[0070] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0071] Figure 2 is a flowchart illustrating an omnidirectional environment perception method for a mobile platform according to an exemplary embodiment. As shown in Figure 2, the omnidirectional environment perception method for a mobile platform provided in this embodiment includes the following steps S101-S104.

[0072] Step S101: During the movement of the mobile platform, a first image is acquired through a forward sensing device and a second image is acquired through a circumferential sensing device; the forward sensing device is used to acquire images of the environment in front of the mobile platform, and there are multiple circumferential sensing devices used to acquire images of the environment around the mobile platform.

[0073] In this embodiment, the mobile platform, while moving, controls the forward sensing device and the surrounding sensing device to acquire images in real time. Then, by executing the omnidirectional environmental perception method of the mobile platform provided in this embodiment, a global occupancy map is generated, thereby executing corresponding control tasks. In practical applications, the mobile platform, while moving, controls the forward sensing device and the surrounding sensing device to acquire images in real time.

[0074] In one application scenario, the forward-looking sensing device is a forward-looking binocular camera. The first image is a binocular image obtained by the forward-looking binocular camera by capturing the forward-looking environment, and this binocular image includes a left-eye image and a right-eye image. In another application scenario, the forward-looking sensing device is a point cloud sensor. The first image is a point cloud image obtained by the point cloud sensor by capturing the forward-looking environment. In yet another application scenario, the forward-looking sensing device is a monocular camera. The first image is a monocular image obtained by the monocular camera by capturing the forward-looking environment.

[0075] In one application scenario, the circumferential sensing device is a monocular camera. The second image is a monocular image obtained by the monocular camera by capturing the environment in the corresponding direction. For example, four monocular cameras may capture forward, backward, left, and right images of the environment, respectively. In another application scenario, the circumferential sensing device is a point cloud sensor. The second image is a point cloud image obtained by the point cloud sensor by capturing the environment in the corresponding direction. For example, four point cloud sensors may capture forward, backward, left, and right point cloud images of the environment, respectively. In yet another application scenario, the circumferential sensing device is a binocular camera. The second image is a binocular image obtained by the binocular camera by capturing the environment in the corresponding direction. This binocular image includes a left-eye image and a right-eye image. For example, four binocular cameras may capture forward, backward, left, and right binocular images of the environment, respectively.

[0076] It should be noted that in practical applications, the types of forward sensing devices and circumferential sensing devices installed on the same mobile platform are usually different. For example, when the forward sensing device is a forward-looking binocular camera, the circumferential sensing device is a monocular camera or a point cloud sensor; or, when the forward sensing device is a monocular camera, the circumferential sensing device is a binocular camera or a point cloud sensor; or, when the forward sensing device is a point cloud sensor, the circumferential sensing device is a binocular camera or a monocular camera.

[0077] Step S102: Determine the first perception information corresponding to the first image, and determine the second perception information corresponding to each second image; the first perception information includes the depth information and elevation information of each point in the target image related to the first image; the second perception information includes the depth information and elevation information of each point in the second image.

[0078] In this embodiment, after acquiring the first image, first perceptual information is determined based on the first image. The first perceptual information includes the depth and elevation information of each point in the target image related to the first image. When the first image is a left-eye image and a right-eye image acquired by a forward-looking binocular camera, the target image refers to the fitted image obtained by fitting the left-eye and right-eye images, and the points in the target image refer to the pixels in the fitted image. When the first image is a monocular image acquired by a monocular camera, the target image refers to the monocular image, and the points refer to the pixels in the monocular image. When the first image is a point cloud image obtained by a point cloud sensor by acquiring the forward-looking environment, the target image refers to the point cloud image, and the points in the target image refer to the three-dimensional spatial points in the point cloud image. Similarly, after acquiring the second image, for each second image, corresponding second perceptual information is determined based on the second image. The second perceptual information includes the depth and elevation information of each point in the second image.

[0079] In the first sensing information, the depth information of a point includes the distance between the object corresponding to that point and the forward sensing device in three-dimensional space. Similarly, in the second sensing information, the depth information of a point includes the distance between the object corresponding to that point and the corresponding circumferential sensing device in three-dimensional space. In the first sensing information, the elevation information of a point includes the height of the object corresponding to that point relative to a reference horizontal plane in three-dimensional space. Similarly, in the second sensing information, the elevation information of a point includes the height of the object corresponding to that point relative to a reference horizontal plane in three-dimensional space. In practical applications, the choice of the reference horizontal plane is related to the type of mobile platform. For example, when the mobile platform is an intelligent vehicle, the reference horizontal plane is the road surface; when the mobile platform is a ship, the reference horizontal plane is the water surface.

[0080] The methods for determining the first perceptual information corresponding to the first image include the following multiple cases.

[0081] In the first scenario, where the first image is the left and right eye images captured by a forward-looking binocular camera, a deep learning algorithm can be used to determine the first perceptual information. Accordingly, determining the first perceptual information corresponding to the first image can be achieved by inputting the first image into a trained first perceptual model and using the first perceptual model to determine the first perceptual information. The first perceptual model is a deep learning-based neural network model, and its input is the binocular image, while its output is the depth and elevation information of each pixel in the fitted image. The training process of the first perceptual model is illustrated in Figure 5 below and will not be elaborated further. For example, the principle of the first perceptual model is to use a stereo matching algorithm to find the matching relationship between corresponding pixels in the left and right eye images. By calculating the disparity between corresponding pixels in the left and right eye images—that is, the positional difference of the pixel in the two images—the depth information of the pixel can be determined. Once the depth information of the pixel is determined, the coordinates of the object corresponding to the pixel in three-dimensional space can be determined using the camera's intrinsic and extrinsic parameter matrices, combined with the principle of triangulation, thereby obtaining the elevation information of the pixel.

[0082] The second scenario involves a point cloud image obtained by a point cloud sensor from the forward-looking environment. In this case, the point cloud image includes point cloud data for each point in three-dimensional space, specifically including three-dimensional coordinates and reflection intensity. Accordingly, determining the first sensing information corresponding to the first image can be achieved by calculating the distance between each point in the point cloud image and the point cloud sensor to determine the depth information of each point, and by calculating the elevation information of each point based on a reference horizontal plane. For example, by projecting the point cloud data onto a reference horizontal plane, the height of each point relative to the reference horizontal plane can be determined.

[0083] The third scenario involves a monocular image obtained by a monocular camera capturing the forward-looking environment. Deep learning algorithms can also be used to determine the first perceptual information. Accordingly, determining the first perceptual information corresponding to the first image can be achieved by inputting the first image into a trained second perceptual model and using the second perceptual model to determine the corresponding first perceptual information. The second perceptual model is a deep learning-based neural network model, and its input is the monocular image, while its output is the depth and elevation information of each pixel in the monocular image. Therefore, the difference between the second and first perceptual models is that the first perceptual model is used to process binocular images, while the second perceptual model is used to process monocular images.

[0084] Similarly, the methods for determining the second perceptual information corresponding to the second image include the following various cases.

[0085] In the first scenario, where the second image is a monocular image captured by a monocular camera, a deep learning algorithm can be used to determine the second perceptual information. Accordingly, the method for determining the second perceptual information corresponding to the second image can be as follows: for each second image, input the second image into the trained second perceptual model, and use the second perceptual model to determine the corresponding second perceptual information. In this case, the method for determining the second perceptual information corresponding to the second image is the same as in the third scenario described above for determining the first perceptual information corresponding to the first image, and will not be elaborated further here.

[0086] The second scenario is when the second image is a point cloud image collected by a point cloud sensor. In this case, the method for determining the second sensing information corresponding to the second image is the same as the second scenario for determining the first sensing information corresponding to the first image, and will not be repeated here.

[0087] The third scenario involves a second image captured by a stereo camera. In this case, a deep learning algorithm can be used to determine the second perceptual information. The implementation method for determining the second perceptual information corresponding to the second image is as follows: for each second image, input the second image into a trained first perceptual model, and use the first perceptual model to determine the corresponding second perceptual information. In this case, the implementation method for determining the second perceptual information corresponding to the second image is the same as in the first scenario described above, and will not be repeated here.

[0088] Regarding the selection of the first and second perception models, please refer to the technical methods in the relevant technologies, which will not be elaborated here.

[0089] Step S103: Based on the first perception information and the second perception information, determine the global occupancy map; the global occupancy map includes the attribute information of each location point in the omnidirectional environment in which the mobile platform is located; the attribute information includes obstacle information and elevation information.

[0090] The global occupancy map can use different types of maps, including 3D grid maps, 2D grid maps, and other types. Each type of map has its advantages and disadvantages, and is suitable for different scenarios and needs. 3D grid maps: 3D grid maps provide a more accurate representation of the environment and are suitable for scenarios that require consideration of altitude, such as drone flight and indoor navigation. 2D grid maps: 2D grid maps are one of the most commonly used map types, suitable for autonomous driving and robotics applications. 2D grid maps are simple to use and can be built and updated quickly. Other map types: In addition to 3D and 2D grid maps, other map types are available, such as semantic maps. These map types can be selected based on actual needs.

[0091] In a 3D mesh map, a location point can be understood as a voxel, which is a combination of pixel, volume, and element, equivalent to a pixel in 3D space. In a 2D mesh map, a location point can be understood as a pixel. For example, location points are represented by coordinates, and the coordinate system can be a coordinate system with the center point of the movable platform or other reference point as its origin.

[0092] One implementation of this step is to fuse the first and second perception information to obtain a global occupancy map. The obstacle information of a location point indicates whether an obstacle exists at that point, and the elevation information of the location point indicates its height relative to a reference horizontal plane. For example, the attribute information of different location points in the first and second perception information is spliced ​​together, and the attribute information of the same location point is superimposed, weighted, or combined. For instance, if location point A has an obstacle in the first perception information but not in the second perception information, then the fused obstacle information for location point A indicates the presence of an obstacle; if location point B's height relative to the reference horizontal plane is 3cm in the first perception information and 2cm in the second perception information, then the fused elevation information for location point B indicates a height of 2.5cm relative to the reference horizontal plane.

[0093] In practical applications, the surrounding environment of a mobile platform changes in real time as it moves, necessitating timely updates to the global occupancy map. Accordingly, the mobile platform controls forward and circumferential sensing devices to acquire images at a certain frequency, thereby generating a new global occupancy map based on the newly acquired images, thus achieving the update of the global occupancy map. This frequency can be set as needed, for example, 5 times per second, 10 times per second, or 15 times per second, etc., and this embodiment does not limit it.

[0094] Each generated global occupancy map can be considered as a single frame of a map. Through real-time updates, multiple consecutive frames of global occupancy maps are obtained.

[0095] In some embodiments, the following step S104 is performed using the current frame global occupancy map. This current frame global occupancy map is obtained by fusing the first perception information and the second perception information at the current moment. The first perception information includes the forward environmental perception information at the current moment, while the second perception information includes the circumferential environmental perception information at the current moment. Therefore, the current frame global occupancy map incorporates the environmental perception information at the current moment. Here, "current moment" refers to the moment when the first image and the second image were acquired.

[0096] In other embodiments, the current frame global occupancy map incorporates environmental awareness information at the current moment, but lacks environmental awareness information from historical moments, which has limitations. Therefore, this embodiment also provides a method for updating the global occupancy map: by introducing temporal information to fuse historical frames with the current frame, the historical frames can influence the current frame. For example, using the pose information of the mobile platform, the current frame global occupancy map and the previous frame global occupancy map are fused to obtain a new global occupancy map, which is then used to perform the following step S104. Maps from different moments can undergo coordinate system transformation, alignment, and fusion. The fusion method can be a weighted average, an attention mechanism, or fusion through a trained convolutional neural network.

[0097] For example, the fusion operation is specifically implemented as follows: feature extraction is performed on the attribute information of each location point in the current frame global occupancy map and the previous frame global occupancy map to obtain feature representations; a temporal attention mechanism is used to perform weighted fusion on the feature representations of corresponding location points in the current frame global occupancy map and the previous frame global occupancy map to capture the temporal relationship and alignment features between them, and obtain the aligned feature representations, thereby obtaining a new global occupancy map.

[0098] Among them, the temporal attention mechanism is a deep learning algorithm that improves the attention to key information by applying attention weights to time series data. Its basic principle is to calculate the information related to the current time step by dynamically adjusting the weight of each time step in the input sequence.

[0099] The pose information includes position information and attitude information. Position information indicates the current location of the mobile platform, and attitude information indicates the current attitude of the mobile platform. In this embodiment, the pose information of the mobile platform is also acquired in real time during its movement. For example, the pose information can be determined by combining images captured by a camera, acceleration and angular velocity provided by an inertial measurement unit (IMU), velocity provided by a wheel speedometer, and 3D point cloud data provided by a lidar. Specifically, this determination can be achieved using sensor fusion: by integrating the acceleration and angular velocity provided by the IMU, the vehicle's velocity, displacement, and attitude angle can be obtained. Based on this, by integrating the velocity provided by the wheel speedometer, performing visual odometry on the images captured by the camera, and registering and extracting features from the 3D point cloud data provided by the lidar, the vehicle's position and attitude can be further determined.

[0100] This approach improves the continuity and accuracy of the global occupancy map through temporal fusion. By considering the pose and historical information of the mobile platform, the algorithm can better understand the state changes during the movement process.

[0101] Step S104: Execute corresponding control tasks based on the global occupancy map; the control tasks include at least one of obstacle detection, road surface elevation estimation, and 360° visualization.

[0102] The global occupancy map includes obstacle information and elevation information for each location point. Road elevation estimation requires sensing the undulations of the road surface within a certain distance ahead. Obstacle detection requires detecting obstacles within a certain distance ahead. 360° visualization requires a complete reconstruction of the surrounding environment, including the space of the chassis. Therefore, the global occupancy map can be applied to control tasks such as obstacle detection, road elevation estimation, and 360° visualization.

[0103] In this embodiment, a forward sensing device and a circumferential sensing device are provided on the mobile platform. During the movement of the mobile platform, the images collected by the forward sensing device and the circumferential sensing device can determine the corresponding sensing information, thereby perceiving the environment around the mobile platform in all directions. By integrating the sensing information of the first image and the second image, a global occupancy map is determined, which includes the attribute information of each location point, such as obstacle information and elevation information, thereby realizing the reconstruction of the environment in which the mobile platform is located. By combining the global occupancy map to perform control tasks, the redundant calculation of the surrounding environment is reduced, information is fully shared, the amount of calculation is reduced, and computing resources are saved. It can also improve the environmental perception capability, path planning efficiency and control task execution effect of the mobile platform during the movement process.

[0104] In some embodiments, the implementation of obstacle detection based on the global occupancy map in step S104 includes: identifying obstacles based on obstacle information at each location point in the global occupancy map, determining a movement path using the elevation information of each location point through a path planning algorithm, and controlling the movement of the movable platform according to the movement path to achieve obstacle avoidance. For example, the path planning algorithm can be a search-based path planning algorithm, a sampling-based path planning algorithm, or a path planning algorithm considering dynamics, etc.

[0105] Traditional obstacle detection solutions typically only provide information about obstacles ahead; if an obstacle is present, the vehicle is impassable, and if no obstacle is present, the vehicle is passable. In complex road conditions, such a binary classification is insufficient for making ideal decisions for downstream planning and control. The solution proposed in this application provides downstream elevation information during obstacle detection, adding further assurance to the judgment of whether a vehicle is passable or impassable. For example, if an 8cm high brick is present ahead of the vehicle, misjudging it as an obstacle and relying solely on the "obstacle" information will lead to accidental braking. However, if elevation information is available, downstream systems can make a judgment based on the elevation and the vehicle's chassis height, and, if appropriate, drive directly over the object. This also provides better comfort for users in assisted driving scenarios without causing a collision.

[0106] In some embodiments, road surface elevation estimation is applied in vehicle driving scenarios. For example, the implementation of road surface elevation estimation based on a global occupancy map includes: adjusting the vehicle's suspension system and / or adjusting the vehicle's speed and steering based on the elevation information of each location point in the global occupancy map. Specifically, when the road surface elevation changes, the vehicle's suspension system can adjust the suspension height according to this change. If the road surface elevation is high, the suspension system can adjust to a higher height to avoid collisions or bumps; if the road surface elevation is low, the suspension system can adjust to a lower height to maintain vehicle stability and driving comfort. When the vehicle is driving on a road surface with significant elevation changes, the driving system can adjust the vehicle's speed according to this change to ensure safe and stable driving. When a significant change in road surface elevation is detected during vehicle operation, such as a protruding object or pothole, the vehicle's steering angle can be adjusted to guide the tires away from the area of ​​significant elevation change. This method improves the vehicle's stability and comfort when traversing different road surface elevations by sensing changes in road surface elevation in real time and adjusting the vehicle's speed and steering in a timely manner to ensure the vehicle safely traverses undulations and slopes, thereby better adapting to different road conditions.

[0107] In some embodiments, when the mobile platform is a vehicle and a driver is present in the vehicle, 360° visualization can assist the driver in driving. Accordingly, the implementation of 360° visualization based on a global occupancy map includes: generating panoramic 360° visualization information based on obstacle information and elevation information at various locations in the global occupancy map; outputting this 360° visualization information to a display device, which can be the vehicle's central control screen or dashboard, so that the driver can observe the surrounding environment in real time. This method can improve the driver's perception of the surrounding environment, enhancing driving safety and comfort. Furthermore, it can be combined with a vehicle driver assistance system to provide real-time warnings and assistance functions, helping the driver drive the vehicle more safely. The 360° visualization information can be images or videos.

[0108] For example, based on obstacle information and elevation information at various locations in the global occupancy map, the surrounding environment of the mobile platform and the road surface under its chassis are visualized, highlighting obstacles and road elevations. During visualization, obstacle information can include not only whether an obstacle is present but also its type, allowing for different display styles (e.g., different obstacle 3D models). The visualization is implemented as follows: based on obstacle and elevation information at various locations in the global occupancy map, 3D modeling technology is used to model the surrounding environment of the mobile platform and the road surface under its chassis, obtaining a 3D model. This 3D model is then rendered to generate a visualized image or video, which is output to a display device. For obstacles, the corresponding 3D obstacle model can be selected based on the obstacle type, and the model size can be adjusted according to the elevation information. For example, the height of the obstacle is also displayed when showing it. Highlighting can be achieved by using a different color from the background, highlighting, etc.

[0109] In related technologies, 360° visualization solutions only consider the current surround view image information. However, in this application, on the one hand, by employing temporal fusion, the global occupancy map can reconstruct areas currently unobservable by the sensing device using past information, such as the space under the vehicle chassis. This provides 3D visualization of the road surface under the chassis, allowing global 3D reconstruction information to be reflected in the 360° visualization system. On the other hand, by combining obstacle information and elevation information for display, the location and height information of obstacles around the vehicle, obstacles under the vehicle chassis, and road surface elevation information are highlighted, helping the driver make better decisions.

[0110] In some embodiments, when the mobile platform is an autonomously moving unmanned device, 360° visualization can also be used to allow users to remotely view and control the mobile platform. Accordingly, the implementation of 360° visualization based on a global occupancy map includes: generating panoramic 360° visualization information based on obstacle information and elevation information at each location point in the global occupancy map; sending the 360° visualization information to a user device communicatively connected to the mobile platform, so that the user device displays the 360° visualization information, thereby enabling the user to observe the surrounding environment of the mobile platform in real time, and also enabling the user to control the mobile platform according to the environmental conditions. The user device can be a mobile phone, computer, remote control, wearable device (watch or bracelet), controller, etc. Exemplarily, the mobile platform and the user device are connected via a wireless network, such as Wi-Fi or Bluetooth.

[0111] In some embodiments, the global occupancy map is also called the global occupancy grid, in which location points are represented in the form of cells; accordingly, step S103 includes the following steps 1-2.

[0112] Step 1: Based on the depth information of each point in the first perception information, determine the obstacle information corresponding to the first image; and based on the depth information of each point in the second perception information corresponding to each second image, determine the obstacle information corresponding to each second image.

[0113] The obstacle information corresponding to the first image includes obstacle information for each point in the target image related to the first image. Similarly, the obstacle information corresponding to the second image includes obstacle information for each point in the second image. The obstacle information for a point is used to indicate whether an obstacle exists at that point.

[0114] For example, the method for determining obstacle information corresponding to the first image based on the depth information of each point in the first perception information can be as follows: for each point in the target image, the depth information of that point is compared with a preset depth value. If the depth information of that point is not less than the preset depth value, then the obstacle information of that point is determined to be non-existent; if the depth information of that point is less than the preset depth value, then the obstacle information of that point is determined to be present. The preset depth value can be set according to actual needs, and this embodiment does not limit it.

[0115] Similarly, for each second image, the method for determining the obstacle information corresponding to the second image based on the depth information of each point in the second perception information corresponding to the second image can be as follows: for each point in the second image, the depth information of the point is compared with a preset depth value. If the depth information of the point is not less than the preset depth value, the obstacle information of the point is determined to be non-existent; if the depth information of the point is less than the preset depth value, the obstacle information of the point is determined to be present.

[0116] Step 2: For each cell in the global occupied map, based on the cell's location, obstacle information corresponding to the first image, and obstacle information corresponding to each second image, determine the cell's obstacle information; and based on the cell's location, first perception information corresponding to the first image, and second perception information corresponding to each second image, determine the cell's elevation information.

[0117] In this embodiment, when determining the obstacle information of each cell in the global occupancy map, combining the cell's position, the obstacle information corresponding to the first image, and the obstacle information corresponding to each second image can more accurately determine whether there is an obstacle in each cell. At the same time, by combining the first perception information corresponding to the first image and the second perception information corresponding to each second image, the elevation information of the cell can be determined more accurately. Thus, by combining the perception results of the forward sensing device and the circumferential sensing device, the obstacle information and elevation information of the cell can be determined separately, resulting in high accuracy.

[0118] Regarding step 2, the process of determining the obstacle information of the cell will be explained first.

[0119] In some embodiments, the method for determining the obstacle information of a cell in step 2 based on the cell's position, the obstacle information corresponding to the first image, and the obstacle information corresponding to each second image includes the following two cases.

[0120] In the first case, if the cell is a forward cell, the obstacle information of the forward cell is determined based on the obstacle information corresponding to the first image and the obstacle information corresponding to the second image at the forward position.

[0121] Here, the forward cell refers to the cell located in the forward direction (movement direction) of the movable platform. Correspondingly, both the first image and the second image at the forward position are obtained by capturing the forward environment, thereby determining the obstacle information of the forward cell. For example, determining the obstacle information of the forward cell can be achieved as follows: if at least one of the obstacle information at the point corresponding to the forward cell in the target image, and the obstacle information at the point corresponding to the forward cell in the second image at the forward position, indicates the presence of an obstacle, then the obstacle information of the forward cell is determined to be that an obstacle exists; otherwise, the obstacle information of the forward cell is determined to be that no obstacle exists.

[0122] In the second case, if the cell is a backward cell or a side cell, the obstacle information of the corresponding cell is determined based on the obstacle information corresponding to the second image at the corresponding position.

[0123] Here, the backward cell refers to the cell located in the backward direction of the movable platform. Correspondingly, the second image at the backward position is obtained by collecting the backward environment, based on which the obstacle information of the backward cell can be determined.

[0124] For example, the obstacle information of the backward cell can be determined as follows: if the obstacle information of the point corresponding to the backward cell in the second image at the backward position indicates the existence of an obstacle, then the obstacle information of the backward cell is determined to be that an obstacle exists; otherwise, the obstacle information of the backward cell is determined to be that no obstacle exists.

[0125] Similarly, lateral cells refer to cells located in the lateral direction of the movable platform, specifically including left-hand and right-hand cells. Accordingly, the second image in the left-hand position is obtained by capturing the left-hand environment, thereby determining the obstacle information of the left-hand cell; similarly, the second image in the right-hand position is obtained by capturing the right-hand environment, thereby determining the obstacle information of the right-hand cell. The method for determining the obstacle information of lateral cells is the same as the method for determining the obstacle information of rear-hand cells, and will not be elaborated further here.

[0126] In this embodiment, obstacle information of a cell is determined by using obstacle information of the midpoint of the image in the corresponding direction, which has high accuracy.

[0127] Regarding step 2, the process of determining the elevation information of the cell will be explained below.

[0128] In some embodiments, the implementation of determining the cell elevation information based on the cell position, the first perception information corresponding to the first image, and the second perception information corresponding to each second image in step 2 includes the following two cases.

[0129] In the first scenario, if the cell is a forward cell, the first elevation information from the first perception information is fused with the second elevation information from the second perception information corresponding to the second image at the forward position to obtain the elevation information of the forward cell; both the first and second elevation information are the elevation information of the points corresponding to the forward cell. The fusion method can be set as needed, such as averaging or other more robust methods.

[0130] In the second scenario, if the cell is a backward or lateral cell, the elevation information of the point corresponding to that cell in the second perception information of the second image at that location is fused to obtain the elevation information of that cell. The fusion method can be set as needed, such as averaging or other more robust methods.

[0131] In this embodiment, the elevation information of a cell is determined by using obstacle information from points in the image in the corresponding direction, which has high accuracy.

[0132] Steps 1 and 2 above are one way to obtain the global occupancy map of the current frame by fusing the first and second perception information at the current moment. As can be seen from the description of step S103, a new global occupancy map can also be obtained by fusing historical frames with the current frame by introducing temporal information, which will not be elaborated here.

[0133] Steps 1 and 2 above combine the perception results from the forward-looking and circumferential perception devices to generate attribute information for each cell in the global occupancy map. However, when the first image is a binocular image captured by a forward-looking binocular camera and the second image is a monocular image captured by a monocular camera, the binocular camera has high accuracy but limited range, while the monocular camera has a wider range but lower accuracy. Therefore, the accuracy of the first perception information obtained from binocular stereo matching is higher than that of the second perception information. However, the area of ​​the monocular image is larger than that of the binocular image, which means that the second perception information contains more information than the first perception information. Therefore, multiple global occupancy maps with different resolutions can be generated so that downstream control tasks can select the corresponding global occupancy map according to actual accuracy requirements.

[0134] Accordingly, in some embodiments, multiple global occupancy maps with different resolutions exist simultaneously: a first global occupancy map and a second global occupancy map. Wherein, the resolution of the first global occupancy map is higher than that of the second global occupancy map, and the area of ​​the first global occupancy map is smaller than that of the second global occupancy map; then step S103 includes the following two cases.

[0135] In the first scenario, for each cell in the first global occupied map, if the cell is a forward cell, the attribute information of the cell is determined based on the obstacle information and elevation information corresponding to the first image; if the cell is a backward cell or a side cell, the attribute information of the cell is determined based on the obstacle information and elevation information corresponding to the second image at the corresponding position.

[0136] The attribute information includes obstacle information and elevation information. For example, determining the obstacle information of the preceding cell involves: if the obstacle information of the point corresponding to the preceding cell in the target image indicates the presence of an obstacle, then the obstacle information of the preceding cell is determined to be obstructed; otherwise, the obstacle information of the preceding cell is determined to be obstructed. Similarly, determining the elevation information of the preceding cell involves: fusing the elevation information of the point corresponding to the preceding cell in the target image to obtain the elevation information of the preceding cell.

[0137] For example, the method for determining obstacle information in a backward cell includes: if the obstacle information in the second image at the backward position indicates the presence of an obstacle at the point corresponding to the backward cell, then the obstacle information of the backward cell is determined to indicate the presence of an obstacle; otherwise, the obstacle information of the backward cell is determined to indicate the absence of an obstacle. The method for determining obstacle information in a lateral cell is similar to that for determining obstacle information in a backward cell, and will not be described again here.

[0138] Similarly, determining the elevation information of the backward cell involves fusing the elevation information of the corresponding point in the second image at the backward position with that of the backward cell to obtain the elevation information of the backward cell. The method for determining the elevation information of the lateral cell is similar to that for determining the elevation information of the backward cell, and will not be elaborated further here.

[0139] In the second scenario, for each cell in the second global map, the cell's attribute information is determined based on the obstacle information and elevation information corresponding to the second image at the cell's location.

[0140] For example, determining the obstacle information of the forward cell includes: if the obstacle information of the corresponding point in the second image at the forward position indicates the presence of an obstacle, then the obstacle information of the forward cell is determined to be present; otherwise, the obstacle information of the forward cell is determined to be absent. The methods for determining the obstacle information of the backward and lateral cells are similar to those for determining the obstacle information of the forward cell, and will not be repeated here.

[0141] Similarly, determining the elevation information of the forward cell involves fusing the elevation information of the corresponding point in the second image at the forward position with that of the forward cell to obtain the elevation information of the forward cell. The methods for determining the elevation information of the backward and lateral cells are similar to those for determining the elevation information of the forward cell, and will not be elaborated upon here.

[0142] In this embodiment, the attribute information of the forward cells in different global occupancy maps is determined using different perception results. One is determined by the perception result of the forward sensing device, and the other is determined by the perception result of the circumferential sensing device. The first perception information has higher accuracy, making the attribute information of the forward cells in the determined first global occupancy map more accurate, thus obtaining a high-resolution first global occupancy map. The attribute information of each cell in the second global occupancy map is determined by the second perception information. Therefore, the resolution of the second global occupancy map is lower than that of the first global occupancy map, but the second perception information contains more information, making the area of ​​the second global occupancy map larger and more comprehensive.

[0143] For example, obstacle detection requires long-distance detection but does not require high accuracy, so a second global occupancy map can be used to perform the obstacle detection task; road surface elevation estimation requires high accuracy but does not require high distance, so a first global occupancy map can be used to perform the road surface elevation estimation.

[0144] It should be noted that the above embodiments use two global occupancy maps with different resolutions as an example for illustration. In other embodiments, three or more global occupancy maps with different resolutions can be generated according to actual needs in order to better meet the accuracy requirements of downstream control tasks. The generation method can refer to the generation method of the first global occupancy map and the second global occupancy map, which will not be repeated here.

[0145] In some embodiments, for the same road surface, the degree of road surface undulation changes little in a short period of time. On the one hand, for the same road surface, the mobile platform can also use historical elevation information. For example, after step S103, the method further includes: storing the current position information of the mobile platform in correspondence with the elevation information of each position point in the global occupancy map, and when the platform re-enters the position indicated by the position information, performing a road surface elevation estimation task based on the elevation information of each position point in the global occupancy map stored in correspondence with the position information.

[0146] On the other hand, different mobile platforms can share elevation information. For example, after step S103, the method further includes: sending road surface elevation sharing information to other mobile platforms communicatively connected to the mobile platform; the road surface elevation sharing information includes the current location information and the elevation information of each location point in the global occupancy map. Specifically, for any mobile platform, after receiving the road surface elevation sharing information sent by other mobile platforms, it stores the road surface elevation sharing information, and during movement, it acquires its location information in real time, compares this location information with the location information in the road surface elevation sharing information, and if they match, acquires the elevation information of each location point in the global occupancy map corresponding to that location information, and performs a road surface elevation estimation task.

[0147] For example, when estimating road surface elevation, the road surface elevation can be estimated by combining the elevation information of each location point in the historically stored global occupancy map with the currently generated global occupancy map; alternatively, the elevation information of each location point in the historically stored global occupancy map can be used directly for road surface elevation estimation to save computing resources. Specifically, the combination method can be to merge the current elevation information of the same location point with historical elevation information, and the fusion method can be averaging or other methods.

[0148] By storing location information in correspondence with the elevation information of the location point, or by sharing elevation information with other mobile platforms, when the mobile platform re-enters the same location, it can also combine historical elevation information to estimate the road surface elevation, making the road surface elevation estimation results more accurate.

[0149] Based on this embodiment, considering the limited storage resources of the mobile platform, historical elevation information that has been stored for a long time can also be periodically cleared in order to save storage resources.

[0150] The following example uses a mobile platform as a vehicle, equipped with a forward-looking binocular camera and a panoramic monocular camera, with the panoramic monocular camera including multiple fisheye monocular cameras, to illustrate a specific omnidirectional perception process.

[0151] Referring to Figure 3, which is a flowchart illustrating an omnidirectional environment perception method for a mobile platform according to another exemplary embodiment, this embodiment relates to a specific omnidirectional environment perception process, as shown in Figure 3, which includes the following steps S201-S207.

[0152] Step S201: During the movement of the mobile platform, binocular images are acquired by a forward-looking binocular camera and monocular images are acquired by multiple fisheye monocular cameras respectively.

[0153] After step S201, steps S202 and S204 are executed respectively.

[0154] Step S202: Input the binocular images into the trained first perception model, and use the first perception model to determine the first perception information corresponding to the binocular images.

[0155] After step S202, step S203 is executed.

[0156] Step S203: Based on the depth information of each point in the first perception information, determine the obstacle information corresponding to the binocular image.

[0157] Step S204: For each monocular image, input the monocular image into the trained second perception model, and use the second perception model to determine the second perception information corresponding to the monocular image.

[0158] After step S204, step S205 is executed.

[0159] Step S205: Based on the depth information of each point in the second perception information corresponding to each monocular image, determine the obstacle information corresponding to each monocular image.

[0160] After steps S203 and S205, step S206 is executed.

[0161] Step S206: For each cell in the global occupied map, based on the cell's location, obstacle information corresponding to the binocular image, and obstacle information corresponding to each monocular image, determine the cell's obstacle information; and based on the cell's location, first perception information corresponding to the binocular image, and second perception information corresponding to each monocular image, determine the cell's elevation information.

[0162] Step S207: Based on the global occupancy map, perform obstacle detection, road surface elevation estimation, and 360° visualization respectively.

[0163] The implementation methods for each step are as described in the above embodiments, and will not be repeated here.

[0164] Based on the embodiment shown in Figure 3, and referring to Figure 4, binocular images are acquired by a forward-looking binocular camera on the vehicle. After stereo matching, forward depth and elevation information are obtained. Monocular images are acquired by a fisheye monocular camera on the vehicle. After monocular estimation, omnidirectional depth and elevation information are obtained. After temporal fusion and reconstruction, a global occupancy map is obtained. This map is used for obstacle detection: it can detect obstacles within 120 meters ahead with decimeter-level accuracy, and can also achieve more refined obstacle detection and backward obstacle detection. On the other hand, it is used for road surface pre-aiming, i.e., road surface elevation estimation: it can estimate road surface undulations within 20 meters ahead with centimeter-level accuracy, and can also achieve more stable obstacle removal and backward pre-aiming. Furthermore, it is used for 360° visualization: it can achieve omnidirectional environment reconstruction within 10 meters with centimeter-level accuracy, and can also visualize obstacles and road surface undulations.

[0165] As can be seen, this application proposes an omnidirectional perception system based on a forward-looking binocular camera and a panoramic monocular camera to meet the needs of different downstream control tasks. This omnidirectional perception system fuses the perception results from each camera into a unified global occupancy map. This map includes semantic and elevation elements, and can therefore be directly applied to control tasks such as obstacle detection, road elevation estimation, and 360° visualization. The generation of the global occupancy map utilizes the high precision of the forward-looking binocular camera, the omnidirectional perception capability of the panoramic monocular camera, and the temporal information during vehicle movement. The fusion of the perception results can provide richer and more accurate information for downstream applications.

[0166] The training process of the first perception model will be explained below.

[0167] In some embodiments, a first perception model is first trained to obtain a trained first perception model, and then the first perception model is deployed on a mobile platform or a control terminal for controlling the mobile platform to determine the first perception information using the first perception model.

[0168] Figure 5 is a flowchart illustrating a first perception model training process according to an exemplary embodiment. As shown in Figure 5, the training process of the first perception model includes the following steps S301-S304.

[0169] Step S301: During the movement of the sample movable platform, the pose information of the sample movable platform is obtained, and binocular image samples are collected by the forward-looking binocular camera and the surrounding environmental data of the sample movable platform are collected by the point cloud sensor to obtain point cloud information.

[0170] In this embodiment, sample data for training the first perception model is acquired during the movement of the sample mobile platform. The sample mobile platform can be a real mobile platform or a simulated mobile platform. If it is a simulated mobile platform, the movement process can be simulated using a virtual simulation environment and a simulated mobile platform model.

[0171] The pose information includes position information and attitude information. The position information represents the current location of the sample mobile platform, and the attitude information represents the current attitude of the sample mobile platform. The pose information can be obtained through sensor fusion. For the specific implementation method, please refer to the method for obtaining the pose information of the mobile platform in step S103 above, which will not be repeated here.

[0172] A forward-looking binocular camera is installed on the mobile sample platform. This camera is used to acquire images of the environment in front of the platform, resulting in binocular image samples. These binocular image samples include left and right eye image samples. A point cloud sensor is also installed on the mobile sample platform to acquire point cloud information by collecting environmental data around the platform.

[0173] Step S302: Using a preset 3D reconstruction algorithm, training labels corresponding to the binocular image samples are generated based on point cloud information and pose information.

[0174] The preset 3D reconstruction algorithm can be found in related technical documents and will not be elaborated here. The training labels include the depth and elevation information of each point in the fitted image corresponding to the binocular image sample.

[0175] In one embodiment, step S302 is implemented by: using a preset three-dimensional reconstruction algorithm to perform the following operations: stitching together point cloud information from multiple consecutive frames according to pose information to obtain a first three-dimensional point cloud map; using a consistency loss function to filter out noise in the first three-dimensional point cloud map to obtain a second three-dimensional point cloud map; and generating training labels corresponding to the binocular image samples based on the second three-dimensional point cloud map.

[0176] The point cloud information includes point cloud data for each point in three-dimensional space, specifically including information such as three-dimensional coordinates and reflection intensity. Correspondingly, the first three-dimensional point cloud map includes point cloud data for each point in the omnidirectional environment of the sample mobile platform, while the second three-dimensional point cloud map includes filtered point cloud data for each point in the omnidirectional environment of the sample mobile platform. The consistency loss function is used to determine the spatial dispersion of the point cloud and the repetitive features between multiple scan results. By removing points with high dispersion and inconsistent results from multiple scans, the noise of the point cloud is reduced, thereby improving accuracy.

[0177] For example, the implementation of generating training labels corresponding to binocular image samples based on the second 3D point cloud map can be as follows: by calculating the distance between each point in the second 3D point cloud map and the point cloud sensor, the depth information of each point can be determined, and the elevation information of each point can be calculated according to the reference horizontal plane. For example, by projecting the point cloud data onto the reference horizontal plane, the height of each point relative to the reference horizontal plane can be determined; the binocular image samples are fitted to obtain a fitted image, and each pixel in the fitted image is aligned with the points in the second 3D point cloud map; for each pixel in the fitted image, the depth information and elevation information of the point corresponding to that pixel in the second 3D point cloud map are determined as the depth information and elevation information of that pixel. Specifically, the alignment operation is implemented as follows: based on the intrinsic and extrinsic parameters of the forward-looking binocular camera, the pixels in the fitted image are converted into 3D coordinate points in the camera coordinate system, and then the 3D coordinate points in the camera coordinate system are converted into 3D coordinate points in the world coordinate system, and the converted 3D coordinate points are matched with the points in the second 3D point cloud map.

[0178] The accuracy of road surface elevation estimation in related technologies is highly susceptible to sensor noise. Especially in vehicle driving scenarios, varying environmental and lighting conditions cause image, depth, and pose data to be easily affected by sensor noise, leading to accumulated errors and hindering the achievement of high accuracy. This embodiment, however, establishes a consistency loss function to filter out noise in the point cloud reconstruction scene when generating sample data, generating a smooth, geometrically consistent, and high-precision 3D reconstruction result: a second 3D point cloud map. The training labels generated based on this map are therefore more accurate. Furthermore, by training a first perception model to determine the first perception information corresponding to the first image, the robustness upper limit of depth and height is improved, enabling the height information to achieve the required accuracy for road surface elevation estimation even under high noise conditions.

[0179] It should be noted that the amount of sample data required for training is large. Therefore, during the movement of the sample mobile platform, pose information, binocular image samples, and point cloud information at multiple time points can be obtained, thereby generating a large number of binocular image samples carrying training labels.

[0180] Step S303: Using the training labels corresponding to the binocular image samples as supervision data, the first perception model is iteratively trained based on the binocular image samples.

[0181] For example, in each iteration of training, the batch of binocular image samples required for this iteration of training is input into the first perception model. After internal data processing, the first perception model outputs prediction data, which includes the predicted depth and predicted height of each point in the fitted image corresponding to the binocular image sample. By determining the error between the prediction data and the training label, the model parameters of the first perception model are adjusted to obtain the first perception model after this iteration of training.

[0182] Step S304: In response to satisfying the preset training convergence condition, the first perception model that satisfies the training convergence condition is determined as the first perception model that has been trained.

[0183] The preset convergence conditions can be set as needed, and this embodiment does not limit them. For example, the preset convergence conditions could be that the number of iterations reaches a preset number or the model accuracy reaches a preset accuracy. If the first perception model after this iteration does not meet the preset training convergence conditions, the next iteration will be performed until the preset training convergence conditions are met.

[0184] In this embodiment, a training scheme for a first perception model is provided. This scheme uses a three-dimensional reconstruction method to obtain high-precision sample data, thereby the first perception model trained has good predictive ability.

[0185] Figure 6 is a schematic diagram illustrating the structure of an omnidirectional environmental sensing device for a mobile platform according to an exemplary embodiment. As shown in Figure 6, in this embodiment, the omnidirectional environmental sensing device 400 for the mobile platform is included in a device, which can be a mobile platform or a control terminal for controlling the mobile platform. The omnidirectional environmental sensing device 400 for the mobile platform includes:

[0186] The acquisition module 401 is used to acquire a first image through a forward sensing device and a second image through a circumferential sensing device during the movement of the mobile platform; the forward sensing device is used to acquire images of the environment in front of the mobile platform, and there are multiple circumferential sensing devices used to acquire images of the environment around the mobile platform.

[0187] The first determining module 402 is used to determine the first sensing information corresponding to the first image, and to determine the second sensing information corresponding to each second image; the first sensing information includes the depth information and elevation information of each point in the target image related to the first image; the second sensing information includes the depth information and elevation information of each point in the second image.

[0188] The second determining module 403 is used to determine a global occupancy map based on the first and second perception information; the global occupancy map includes attribute information of each location point in the omnidirectional environment in which the mobile platform is located; the attribute information includes obstacle information and elevation information.

[0189] The execution module 404 is used to perform corresponding control tasks based on the global occupancy map; the control tasks include at least one of obstacle detection, road surface elevation estimation and 360° visualization.

[0190] In some embodiments, when the forward-looking sensing device is a forward-looking binocular camera, the first determining module 402 is specifically used for:

[0191] The first image is input into the trained first perception model, and the first perception model is used to determine the first perception information;

[0192] Accordingly, a training module is also included, which includes:

[0193] The acquisition unit is used to acquire the pose information of the sample mobile platform during the movement of the sample mobile platform, and to acquire binocular image samples through a forward-looking binocular camera and collect environmental data around the sample mobile platform through a point cloud sensor to obtain point cloud information.

[0194] The generation unit is used to generate training labels corresponding to the binocular image samples based on point cloud information and pose information using a preset 3D reconstruction algorithm; the training labels include the depth information and elevation information of each point in the fitted image corresponding to the binocular image sample.

[0195] The training unit is used to iteratively train the first perception model based on the binocular image samples, using the training labels corresponding to the binocular image samples as supervision data.

[0196] The determining unit is used to determine the first perception model that meets the training convergence condition as the first perception model that has been trained in response to the satisfaction of the preset training convergence condition.

[0197] In some embodiments, the generating unit is specifically used for:

[0198] Using a preset 3D reconstruction algorithm, perform the following operations:

[0199] The point cloud information of multiple consecutive frames is stitched together according to the pose information to obtain the first three-dimensional point cloud map;

[0200] The noise in the first 3D point cloud map is filtered out using the consistency loss function to obtain the second 3D point cloud map.

[0201] Training labels are generated based on the second 3D point cloud map to correspond to the stereo image samples.

[0202] In some embodiments, the location point is represented in the form of a cell; the second determining module 403 includes:

[0203] The first determining unit is used to determine obstacle information corresponding to the first image based on the depth information of each point in the first sensing information, and to determine obstacle information corresponding to each second image based on the depth information of each point in the second sensing information corresponding to each second image.

[0204] The second determining unit is used to determine the obstacle information of each cell in the global occupied map based on the cell's position, obstacle information corresponding to the first image, and obstacle information corresponding to each second image, and to determine the cell's elevation information based on the cell's position, first perception information corresponding to the first image, and second perception information corresponding to each second image.

[0205] In some embodiments, the first determining unit is specifically used for:

[0206] If the cell is a forward cell, the obstacle information of the cell is determined based on the obstacle information corresponding to the first image and the obstacle information corresponding to the second image at the forward position;

[0207] If the cell is a backward cell or a side cell, the obstacle information of the cell is determined based on the obstacle information corresponding to the second image at the corresponding position.

[0208] In some embodiments, the second determining unit is specifically used for:

[0209] If the cell is a forward cell, the first elevation information in the first perception information is fused with the second elevation information in the second perception information corresponding to the second image at the forward position to obtain the cell's elevation information; both the first elevation information and the second elevation information are the elevation information of the point corresponding to the cell.

[0210] If the cell is a backward cell or a side cell, the elevation information of the point corresponding to the cell in the second perception information corresponding to the second image at the corresponding position is fused to obtain the elevation information of the cell.

[0211] In some embodiments, multiple global occupancy maps with different resolutions correspond to the same time; the second determining module 403 is used for:

[0212] For each cell in the first global occupied map, if the cell is a forward cell, the cell's attribute information is determined based on the obstacle information and elevation information corresponding to the first image; if the cell is a backward or lateral cell, the cell's attribute information is determined based on the obstacle information and elevation information corresponding to the second image at the corresponding location.

[0213] For each cell in the second global occupancy map, the cell's attribute information is determined based on the obstacle information and elevation information corresponding to the second image at the cell's location.

[0214] Among them, the resolution of the first global occupied map is higher than that of the second global occupied map, and the area of ​​the first global occupied map is smaller than that of the second global occupied map.

[0215] In some embodiments, when the control task is 360° visualization, the execution module 404 is specifically used to: visualize the environment around the mobile platform and the road surface under the chassis based on obstacle information and elevation information of each location point in the global occupancy map, and highlight obstacles and road surface elevations in the environment and on the road surface.

[0216] In some embodiments, at least one of the following modules is also included:

[0217] The storage module is used to store the location information of the mobile platform at the current moment and the elevation information of each location point in the global occupancy map. The execution module 404 is also used to perform a road surface elevation estimation task based on the elevation information of each location point in the global occupancy map stored in correspondence with the location information when the platform re-enters the location indicated by the location information.

[0218] The sending module is used to send road surface elevation sharing information to other mobile platforms that are communicatively connected to the mobile platform; the road surface elevation sharing information includes the current location information and the elevation information of each location point in the global occupied map.

[0219] The omnidirectional environmental sensing device for the mobile platform provided in this embodiment can execute the technical solution of the corresponding method embodiment. Its implementation principle and technical effect are similar to those of the corresponding method embodiment, and will not be described in detail here.

[0220] This application also provides a mobile platform. The mobile platform includes, but is not limited to, intelligent vehicles, drones, unmanned ships, and ground robots. Figure 7 is a schematic diagram of the structure of a mobile platform according to an exemplary embodiment. As shown in Figure 7, the mobile platform 500 includes a processor 501 and a memory 502 communicatively connected to the processor 501.

[0221] The memory 502 stores computer-executable instructions; the processor 501 executes the computer-executable instructions stored in the memory 502 to implement the omnidirectional environmental perception method for the mobile platform provided in this application.

[0222] In this embodiment, the memory 502 and the processor 501 are connected via a bus. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, a data bus, a control bus, etc.

[0223] It should be noted that, in addition to the processor and memory, the mobile platform may also include other functional components to achieve the corresponding functions, such as multimedia components, sensor components and communication components, etc. This embodiment does not limit this.

[0224] The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein. The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required.

[0225] In an exemplary embodiment, an electronic device is also provided, including a processor and a memory communicatively connected to the processor. The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the omnidirectional environmental perception method for a mobile platform provided in this application. The electronic device includes, but is not limited to, a computer, a mobile terminal, and a wearable device.

[0226] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores computer-executable instructions that, when executed by a processor, are used to implement the omnidirectional environmental perception method of the mobile platform provided in this application.

[0227] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, is used to implement the omnidirectional environmental perception method for the mobile platform provided in this application.

[0228] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0229] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0230] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0231] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0232] When an integrated unit / module is implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, ASIC, digital signal processor (DSP), programmable logic device (PLD), FPGA, controller, microcontroller, microprocessor, or other electronic component. Unless otherwise specified, the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a USB flash drive, random-access memory (RAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), enhanced dynamic random-access memory (EDRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), high-bandwidth memory (HBM), or hybrid memory cube (HMC) and other media capable of storing program code.

[0233] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0234] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0235] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for omnidirectional environmental perception on a mobile platform, characterized in that, include: During the movement of the mobile platform, a first image is acquired through a forward sensing device and a second image is acquired through a circumferential sensing device. The forward sensing device is used to collect images of the forward environment of the mobile platform, and the number of the circumferential sensing devices is multiple and used to collect images of the surrounding environment of the mobile platform. First sensing information corresponding to the first image is determined, and second sensing information corresponding to each of the second images is determined; the first sensing information includes depth information and elevation information of each point in the target image related to the first image; the second sensing information includes depth information and elevation information of each point in the second image. Based on the first and second sensing information, determine the global occupied map; The global occupancy map includes attribute information of each location point in the omnidirectional environment in which the mobile platform is located; the attribute information includes obstacle information and elevation information. The corresponding control tasks are performed based on the global occupancy map; the control tasks include at least one of obstacle detection, road surface elevation estimation, and 360° visualization.

2. The method according to claim 1, characterized in that, When the forward-looking sensing device is a forward-looking binocular camera, determining the first sensing information corresponding to the first image includes: The first image is input into the trained first perception model, and the first perception model is used to determine the first perception information; wherein, the training process of the first perception model includes: During the movement of the sample mobile platform, the pose information of the sample mobile platform is acquired, and binocular image samples are collected by a forward-looking binocular camera and environmental data around the sample mobile platform are collected by a point cloud sensor to obtain point cloud information. A preset 3D reconstruction algorithm is used to generate training labels corresponding to the binocular image samples based on the point cloud information and the pose information; the training labels include the depth information and elevation information of each point in the fitted image corresponding to the binocular image samples; Using the training labels corresponding to the binocular image samples as supervision data, the first perception model is iteratively trained based on the binocular image samples; In response to meeting the preset training convergence condition, the first perception model that meets the training convergence condition is determined as the first perception model that has been trained.

3. The method according to claim 1, characterized in that, The location points are represented in cell format; determining the global map occupancy based on the first and second sensing information includes: Based on the depth information of each point in the first perception information, the obstacle information corresponding to the first image is determined, and based on the depth information of each point in the second perception information corresponding to each second image, the obstacle information corresponding to each second image is determined respectively. For each cell in the global occupied map, based on the cell's position, obstacle information corresponding to the first image, and obstacle information corresponding to each of the second images, obstacle information of the cell is determined; and based on the cell's position, first perception information corresponding to the first image, and second perception information corresponding to each of the second images, elevation information of the cell is determined.

4. The method according to claim 3, characterized in that, Determining the obstacle information of the cell based on the cell's position, the obstacle information corresponding to the first image, and the obstacle information corresponding to each of the second images includes: If the cell is a forward cell, then the obstacle information of the cell is determined based on the obstacle information corresponding to the first image and the obstacle information corresponding to the second image at the forward position; If the cell is a backward cell or a side cell, then the obstacle information of the cell is determined based on the obstacle information corresponding to the second image at the corresponding position; and / or, Determining the elevation information of the cell based on its position, the first sensing information corresponding to the first image, and the second sensing information corresponding to each of the second images includes: If the cell is a forward cell, the first elevation information in the first sensing information is fused with the second elevation information in the second sensing information corresponding to the second image at the forward position to obtain the elevation information of the cell; both the first elevation information and the second elevation information are the elevation information of the point corresponding to the cell; If the cell is a backward cell or a side cell, the elevation information of the point corresponding to the cell in the second perception information corresponding to the second image at the corresponding position is fused to obtain the elevation information of the cell.

5. The method according to claim 3, characterized in that, Multiple global occupancy maps with different resolutions can be displayed simultaneously. The step of determining the global occupancy map based on the first sensing information and the second sensing information includes: For each cell in the first global occupied map, if the cell is a forward cell, the attribute information of the cell is determined based on the obstacle information and elevation information corresponding to the first image; if the cell is a backward cell or a side cell, the attribute information of the cell is determined based on the obstacle information and elevation information corresponding to the second image at the corresponding position. For each cell in the second global occupancy map, the attribute information of the cell is determined based on the obstacle information and elevation information corresponding to the second image at the corresponding position of the cell; Among them, the resolution of the first global occupied map is higher than that of the second global occupied map, and the area of ​​the first global occupied map is smaller than that of the second global occupied map.

6. The method according to claim 1, characterized in that, When the control task is 360° visualization, the execution of the corresponding control task based on the global occupancy map includes: Based on the obstacle information and elevation information of each location point in the global occupancy map, the environment around the mobile platform and the road surface under the chassis are visualized, and the obstacles and road surface elevations are highlighted.

7. The method according to any one of claims 1-6, characterized in that, After determining the global occupancy map based on the first and second perception information, the process further includes at least one of the following: The location information of the mobile platform at the current moment is stored in correspondence with the elevation information of each location point in the global occupancy map. When the platform enters the location indicated by the location information again, the road surface elevation estimation task is performed based on the elevation information of each location point in the global occupancy map stored in correspondence with the location information. The road surface elevation sharing information is sent to other mobile platforms that are communicatively connected to the mobile platform; the road surface elevation sharing information includes the current location information and the elevation information of each location point in the global occupied map.

8. An omnidirectional environmental sensing device for a mobile platform, characterized in that, include: The acquisition module is used to acquire a first image through a forward sensing device and a second image through a circumferential sensing device during the movement of the mobile platform. The forward sensing device is used to collect images of the forward environment of the mobile platform, and the number of the circumferential sensing devices is multiple and used to collect images of the surrounding environment of the mobile platform. A first determining module is configured to determine first perceptual information corresponding to the first image, and to determine second perceptual information corresponding to each of the second images; the first perceptual information includes depth information and elevation information of each point in the target image related to the first image; the second perceptual information includes depth information and elevation information of each point in the second image; The second determining module is used to determine the global occupied map based on the first sensing information and the second sensing information; The global occupancy map includes attribute information of each location point in the omnidirectional environment in which the mobile platform is located; the attribute information includes obstacle information and elevation information. An execution module is used to perform corresponding control tasks based on the global occupancy map; the control tasks include at least one of obstacle detection, road surface elevation estimation, and 360° visualization.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the omnidirectional environmental perception method for a mobile platform as provided in any one of claims 1-7.

10. The electronic device according to claim 9, characterized in that, The electronic device is a mobile platform.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the omnidirectional environment perception method for a mobile platform as described in any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it is used to implement the omnidirectional environment perception method for a mobile platform as described in any one of claims 1 to 7.

Images