Terrain Processing Method, Apparatus, Device, Storage Medium, Program Product

Abstract

The present application provides a terrain processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. The method includes: obtaining a depth map captured by a target device at a real-time moment; performing discretization processing on a local area where the target device is located at the real-time moment to obtain a plurality of real-time terrain units, and determining the terrain information of each real-time terrain unit at the real-time moment based on the depth map at the real-time moment; determining the terrain type of each real-time terrain unit at the real-time moment based on the terrain information of each real-time terrain unit at the real-time moment and the terrain information of each historical terrain unit at a historical moment; for each real-time terrain unit, performing fusion processing on the terrain information of the real-time terrain unit and the terrain information of the historical terrain unit at the same position at at least one historical moment to obtain the fused terrain information of the real-time terrain unit. Through the present application, it is possible to improve both the terrain recognition speed and the terrain recognition accuracy simultaneously.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular, to a terrain processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] A mobile robot is a machine device that automatically performs work. It can either be commanded by humans, run pre-programmed procedures, or act according to principles and guidelines formulated using artificial intelligence technology. Its task is to assist or replace human work, such as in the manufacturing industry, construction industry, or dangerous work.

[0003] In a motion scenario, the perception module of a mobile robot needs to face the challenge of continuous force impact, which can cause uncontrollable vibration of the mobile robot's sensors, resulting in incorrect terrain recognition near the mobile robot. The terrain recognition solutions in related technologies are difficult to eliminate the impact of vibration on terrain recognition. Summary of the Invention

[0004] Embodiments of the present application provide a terrain processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve both the terrain recognition speed and the terrain recognition accuracy.

[0005] The technical solution of the embodiments of the present application is implemented as follows:

[0006] Embodiments of the present application provide a terrain processing method, including:

[0007] Obtain a depth map captured by a target device at a real-time moment;

[0008] Perform discretization processing on the local area where the target device is located at the real-time moment to obtain a plurality of real-time terrain units, and based on the depth map at the real-time moment, determine the terrain information of each real-time terrain unit at the real-time moment;

[0009] Based on the terrain information of each real-time terrain unit at the real-time moment and the terrain information of each historical terrain unit at a historical moment, determine the terrain type of each real-time terrain unit at the real-time moment, where the historical moment is a moment before the real-time moment;

[0010] For each real-time terrain unit whose terrain type is a target terrain type, perform fusion processing on the terrain information of the real-time terrain unit and the terrain information of the historical terrain unit at the same position at at least one historical moment to obtain the fused terrain information of the real-time terrain unit.

[0011] Embodiments of the present application provide a terrain processing apparatus, including:

[0012] An acquisition module, configured to acquire a depth map captured by a target device at a real-time moment;

[0013] A discretization module, configured to discretize a local area where the target device is located at a real-time moment to obtain a plurality of real-time terrain units, and determine terrain information of each real-time terrain unit at the real-time moment based on the depth map at the real-time moment;

[0014] A terrain module, configured to determine a terrain type of each real-time terrain unit at the real-time moment based on the terrain information of each real-time terrain unit at the real-time moment and the terrain information of each historical terrain unit at a historical moment, where the historical moment is a moment before the real-time moment;

[0015] A fusion module, configured to perform a fusion process on the terrain information of a real-time terrain unit and the terrain information of a historical terrain unit at the same position at at least one historical moment for each real-time terrain unit whose terrain type is a target terrain type, to obtain the fused terrain information of the real-time terrain unit.

[0016] In the above solution, the acquisition module is further configured to: detect a depth map of a corresponding shooting area through a sensor of the target device, where the shooting area overlaps with the local area at the real-time moment, and the depth map includes depth values of each pixel.

[0017] In the above solution, the acquisition module is further configured to: after acquiring the depth map captured by the target device at the real-time moment, perform target plane recognition processing on the depth map to obtain a target depth map corresponding to the target plane in the depth map; the discretization module is further configured to: determine the terrain information of each real-time terrain unit at the real-time moment based on the target depth map at the real-time moment.

[0018] In the above solution, before discretizing the local area where the target device is located at the real-time moment to obtain a plurality of real-time terrain units, the discretization module is further configured to perform any one of the following processes: acquire a first area positively correlated with the moving speed of the target device; acquire a first area positively correlated with the volume of the target device; acquire a local area centered on the target device and meeting the first area.

[0019] In the above solution, the discrete module is further configured to perform any one of the following processes: obtaining the number of terrain units that is positively correlated with the computing speed of the target device; obtaining the number of terrain units that is positively correlated with the memory capacity of the target device; performing unit segmentation processing on the local area at the real-time moment based on the number of terrain units to obtain a plurality of the real-time terrain units.

[0020] In the above solution, the terrain module is further configured to: obtain a plurality of terrain information intervals; for each of the real-time terrain units, perform the following processes: obtaining a target terrain unit having the same position as the real-time terrain unit among at least one of the historical terrain units; determining the count number of the real-time terrain unit corresponding to each of the terrain information intervals based on the terrain information of the real-time terrain unit at the real-time moment and the terrain information of the target terrain unit at the historical moment; determining the terrain type of each of the real-time terrain units at the real-time moment based on the count numbers of the plurality of real-time terrain units corresponding to each of the terrain information intervals.

[0021] In the above solution, the terrain module is further configured to: perform self-state estimation processing through the sensor of the target device to obtain the real-time coordinate data of the target device corresponding to the world coordinate system at the real-time moment; obtain the historical coordinate data of the target device corresponding to the world coordinate system at each of the historical moments; for each of the historical moments, perform position calculation processing on each of the historical terrain units at the historical moment through the historical coordinate data at the historical moment to obtain the first position of each of the historical terrain units at the historical moment in the world coordinate system; perform position calculation processing on the real-time terrain unit through the real-time coordinate data to obtain the second position of the real-time terrain unit in the world coordinate system; and obtain the historical terrain unit whose first position is the same as the second position of the real-time terrain unit in each of the historical moments as the target terrain unit.

[0022] In the above solution, the terrain module is further configured to: for each of the terrain information intervals, obtain the number of terrain information that is within the terrain information interval from the terrain information of the real-time terrain unit and the terrain information of the target terrain units at a plurality of historical moments as the count number of the real-time terrain unit corresponding to the terrain information interval.

[0023] In the above solution, the terrain module is further configured to: for each of the real-time terrain units, obtain the terrain information interval for which the count number is greater than the number threshold as the target terrain information interval of the real-time terrain unit, and use the terrain type corresponding to the target terrain information interval as the terrain type of the real-time terrain unit at the real-time moment.

[0024] In the above solution, the terrain module is further configured to: when the area of the second region of the real-time terrain unit is less than the region area threshold, obtain at least one target terrain unit in the historical terrain units that has the same position as the real-time terrain unit; before obtaining at least one target terrain unit in the historical terrain units that has the same position as the real-time terrain unit, when the area of the second region of the real-time terrain unit is not less than the region area threshold, perform a plane fitting process on the real-time terrain unit to obtain a fitting plane in the real-time terrain unit, and update the fitting plane as the real-time terrain unit.

[0025] In the above solution, the fusion module is further configured to: obtain at least one target terrain unit in the historical terrain units that has the same position as the real-time terrain unit; perform a fusion process on the terrain information of the real-time terrain unit and the terrain information of the target terrain unit to obtain the fused terrain information of the real-time terrain unit.

[0026] In the above solution, the fusion module is further configured to: obtain the reciprocal of the distance between the real-time terrain unit and the target device at the real-time moment as the weight of the real-time terrain unit; obtain the reciprocal of the distance between each target terrain unit and the target device at the corresponding historical moment as the weight of each target terrain unit; based on the weight of the real-time terrain unit and the weights of each target terrain unit, perform a weighted summation process on the terrain information of the real-time terrain unit and the terrain information of the target terrain unit to obtain the fused terrain information of the real-time terrain unit.

[0027] An embodiment of the present application provides an electronic device, including:

[0028] A memory for storing computer-executable instructions;

[0029] A processor, configured to implement the terrain processing method provided by the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0030] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the terrain processing method provided by the embodiment of the present application when executed by a processor.

[0031] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions, where the computer program or computer-executable instructions, when executed by a processor, implement the terrain processing method provided by the embodiment of the present application.

[0032] The embodiments of the present application have the following beneficial effects:

[0033] In the embodiments of the present application, the local area where the target device is located is first discretized into terrain units, and the types of the terrain units are judged from a statistical perspective, effectively dealing with the incorrect depth maps caused by continuous force impacts or continuous vibrations. For each terrain unit of the target terrain type, the terrain information of each terrain unit at different moments is fused in the time domain process, and finally high-quality fused terrain information is obtained. Thus, real-time, accurate, and robust terrain information can be provided in a motion scenario. Description of the Drawings

[0034] Figure 1 is a schematic structural diagram of the terrain processing system provided by the embodiments of the present application;

[0035] Figure 2 is a schematic structural diagram of the electronic device provided by the embodiments of the present application;

[0036] Figures 3A - 3C is a schematic flowchart of the terrain processing method provided by the embodiments of the present application;

[0037] Figure 4 is a schematic framework diagram of the terrain processing method provided by the embodiments of the present application;

[0038] Figure 5 is a schematic diagram of the local airspace discretization of the terrain processing method provided by the embodiments of the present application;

[0039] Figure 6 is a schematic diagram of the time-domain processing of the terrain unit of the terrain processing method provided by the embodiments of the present application;

[0040] Figure 7 is a statistical distribution strategy diagram of the terrain processing method provided by the embodiments of the present application;

[0041] Figure 8 is a schematic framework diagram of the terrain processing method provided by the embodiments of the present application;

[0042] Figure 9 is a statistical distribution strategy diagram of the terrain processing method provided by the embodiments of the present application. Detailed Embodiments

[0043] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be construed as limitations on the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0044] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] In the following description, the terms "first", "second", and "third" are merely used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first", "second", and "third" may, where permitted, interchange their specific order or sequence so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are for the purpose of describing embodiments of this application only and are not intended to limit this application.

[0047] Before further elaborating on the embodiments of this application, the nouns and terms involved in the embodiments of this application are described. The nouns and terms involved in the embodiments of this application are subject to the following explanations.

[0048] 1) World coordinate system: The world coordinate system can describe the position of the camera in the real world and can also describe the position of the objects in the images captured by the camera in the real world. Generally, the x-axis of this world coordinate system points horizontally to the due east direction, the y-axis points horizontally to the due north direction, and the z-axis points vertically upward.

[0049] 2) Inertial measurement unit (IMU): An IMU is a device used to measure the three-axis attitude angle (or angular rate) and acceleration of an object. An IMU generally includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometers are used to detect the acceleration signals of the object on the three independent axes of the carrier coordinate system, and the gyroscopes are used to detect the angular velocity signals of the carrier relative to the navigation coordinate system. An IMU can measure the angular velocity and acceleration of an object in three-dimensional space and thereby determine the attitude of the object.

[0050] 3) Depth camera: The depth camera is the "eye" of the terminal and the robot. It can detect the depth distance of the captured space through this camera. By obtaining the distance of each point in the image from the camera and adding the two-dimensional coordinates of this point in the 2D image, the three-dimensional space coordinates of each point in the image can be obtained.

[0051] In related technologies, in a motion scenario, the perception module of a mobile robot needs to face the challenge of continuous force impact, which may cause uncontrollable vibrations in the sensors of the mobile robot, resulting in incorrect terrain recognition near the mobile robot. The terrain recognition solutions in related technologies are difficult to eliminate the impact of vibration on terrain recognition.

[0052] Embodiments of the present application provide a terrain processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can discretize the local area where the target device is located into terrain units, and perform type judgment on the terrain units from a statistical perspective, effectively coping with the incorrect depth map caused by continuous force impact or continuous vibration. For each terrain unit of the target terrain type, the terrain information of each terrain unit at different times is fused in the time domain process, and finally high-quality fused terrain information is obtained. Thus, real-time, accurate, and robust terrain information can be provided in a motion scenario. The following describes an exemplary application of the electronic device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application can be implemented as a laptop computer, a tablet computer, a desktop computer, a set-top box, a mobile device (such as a mobile phone, a vehicle-mounted terminal, a portable music player, a personal digital assistant, a dedicated messaging device, a portable game device, a smart voice interaction device, a smart home appliance, an aircraft).

[0053] See Figure 1 , Figure 1 FIG. is a schematic structural diagram of a terrain processing system provided in an embodiment of the present application. The terminal 400 is connected to the server 200 through the network 300, and the terminal 400 is also connected to the target device 500 through the network 300. The target device 500 may be a mobile robot, such as a legged robot, a tracked robot, etc. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0054] In some embodiments, the terrain processing method provided by the embodiments of the present application can be implemented in cooperation with a terminal and a server. The terrain processing method provided by the embodiments of the present application can be applied to a robot control APP. In response to a movement instruction triggered by a user on the terminal, the terminal 400 sends the movement instruction to the target device 500. During the movement of the target device 500, the depth map captured at the real-time moment (captured by the target device) and the local area where the target device is located at the real-time moment are sent to the server 200. The server 200 performs discretization processing on the local area to obtain a plurality of real-time terrain units, and based on the depth map at the real-time moment, determines the terrain information of each real-time terrain unit at the real-time moment. Based on the terrain information of each real-time terrain unit at the real-time moment and the terrain information of each historical terrain unit at the historical moment, determines the terrain type of each real-time terrain unit at the real-time moment. For each real-time terrain unit whose terrain type belongs to the target terrain type, performs fusion processing on the terrain information of the real-time terrain unit and the terrain information of the historical terrain unit at at least one historical moment at the same position to obtain the fused terrain information of the real-time terrain unit. When the target device 500 is an automatically controlled robot, the server 200 returns the fused terrain information of the real-time terrain unit to the target device 500. The target device 500 determines the movement route based on the fused terrain information of the real-time terrain unit. When the target device 500 is a robot controlled by the terminal 400, the server 200 returns the fused terrain information of the real-time terrain unit to the terminal 400. The terminal 400 determines the movement route based on the fused terrain information of the real-time terrain unit, and then sends the movement route to the target device 500. When the target device 500 is a robot controlled by the server 200, the server 200 determines the movement route based on the fused terrain information of the real-time terrain unit, and then sends the movement route to the target device 500.

[0055] The embodiments of the present application complete terrain recognition based on depth data. In a continuous force impact scenario, problems such as smearing and large errors may occur in the depth data. The embodiments of the present application minimize the impact of continuous force impact on terrain recognition through the discretization of the terrain units in the spatial domain scene and the statistical distribution for each terrain unit, etc.

[0056] In some embodiments, the terrain processing method provided by the embodiments of the present application can be implemented independently by a terminal or a server. When implemented independently by the terminal, in response to a movement instruction triggered by a user on the terminal, the terminal 400 sends a movement instruction to the target device 500. During the movement of the target device 500, the depth map captured at the real-time moment (captured by the target device) and the local area where the target device is located at the real-time moment are sent to the terminal 400. The terminal 400 performs discretization processing on the local area to obtain a plurality of real-time terrain units, and based on the depth map at the real-time moment, determines the terrain information of each real-time terrain unit at the real-time moment. Based on the terrain information of each real-time terrain unit at the real-time moment and the terrain information of each historical terrain unit at the historical moment, determines the terrain type of each real-time terrain unit at the real-time moment. For each real-time terrain unit whose terrain type belongs to the target terrain type, performs fusion processing on the terrain information of the real-time terrain unit and the terrain information of the historical terrain unit at the same position at at least one historical moment to obtain the fused terrain information of the real-time terrain unit. The terminal 400 determines a movement route based on the fused terrain information of the real-time terrain unit, and then sends the movement route to the target device 500.

[0057] In some embodiments, the terrain processing method provided by the embodiments of the present application can be implemented independently by a terminal or a server. When implemented independently by the terminal, the terminal 400 and the target device 500 can be the same device. During the movement of the target device 500, the target device 500 captures a depth map at the real-time moment and obtains the local area where it is located at the real-time moment. The target device 500 performs discretization processing on the local area to obtain a plurality of real-time terrain units, and based on the depth map at the real-time moment, determines the terrain information of each real-time terrain unit at the real-time moment. Based on the terrain information of each real-time terrain unit at the real-time moment and the terrain information of each historical terrain unit at the historical moment, determines the terrain type of each real-time terrain unit at the real-time moment. For each real-time terrain unit whose terrain type belongs to the target terrain type, performs fusion processing on the terrain information of the real-time terrain unit and the terrain information of the historical terrain unit at the same position at at least one historical moment to obtain the fused terrain information of the real-time terrain unit. The target device 500 determines a movement route based on the fused terrain information of the real-time terrain unit.

[0058] In some embodiments, the terminal or the server may implement the terrain processing method provided in the embodiments of the present application by running a computer program. For example, the computer program may be a native program or a software module in the operating system; it may be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a robot control APP; it may also be a small program, that is, a program that only needs to be downloaded to the browser environment to run; it may also be a small program that can be embedded in any APP. In short, the above computer program may be any form of application program, module or plug-in.

[0059] In some embodiments, the server 200 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected through wired or wireless communication methods, which are not limited in the embodiments of the present invention.

[0060] See Figure 2 , Figure 2 is a schematic structural diagram of an electronic device provided in the embodiments of the present application. Figure 2 The terminal 400 shown in the figure includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. Each component in the terminal 400 is coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 2 all kinds of buses are labeled as the bus system 440.

[0061] The processor 410 may be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0062] The user interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, and other input buttons and controls.

[0063] The memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard disk drives, optical disk drives, etc. The memory 450 optionally includes one or more storage devices that are physically located remote from the processor 410.

[0064] The memory 450 includes volatile memory or non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, the memory 450 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, which are illustrated below.

[0066] The operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and handling hardware-based tasks;

[0067] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wi-Fi (Wireless Fidelity), and USB (Universal Serial Bus), etc.;

[0068] The presentation module 453 is used to enable the presentation of information (such as a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with the user interface 430 (such as a display screen, speakers, etc.);

[0069] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one of the one or more input devices 432.

[0070] In some embodiments, the terrain processing device provided by the embodiments of the present application may be implemented in software. Figure 2 Shown is a terrain processing device 455 stored in a memory 450, which may be software in the form of a program and a plug-in, etc., including the following software modules: an acquisition module 4551, a discretization module 4552, a terrain module 4553, and a fusion module 4554. These modules are logical, so they can be combined arbitrarily or further split according to the functions implemented. The functions of each module will be described below.

[0071] The terrain processing method provided by the embodiments of the present application will be described in combination with the exemplary applications and implementations of the terminal provided by the embodiments of the present application.

[0072] See Figure 3A , Figure 3A is a schematic flowchart of the terrain processing method provided by the embodiments of the present application, which will be described in combination with Figure 3A the steps 101 to 104 shown.

[0073] In step 101, a depth map captured by the target device at a real-time moment is acquired.

[0074] As an example, during the movement of the target device, a depth map captured by the target device at a real-time moment is acquired. During the movement, it will be impacted by a continuous force. The target device is a movable robot, such as a legged robot, a tracked robot, etc. The target device is bound with sensors, and the depth map of the corresponding captured area is detected through the sensors of the target device. Among them, the captured area and the local area have an overlapping area at the real-time moment. The depth map includes the depth value of each pixel, and the sensor for capturing the depth map can be a depth camera.

[0075] In step 102, the local area where the target device is located at the real-time moment is discretized to obtain a plurality of real-time terrain units, and based on the depth map at the real-time moment, the terrain information of each real-time terrain unit at the real-time moment is determined.

[0076] As an example, the discretization process for the area is to divide the area into multiple units, and the division can be an equal-area division.

[0077] In some embodiments, after acquiring the depth map captured by the target device at the real-time moment, the target plane recognition process is performed on the depth map to obtain the target depth map corresponding to the target plane in the depth map; in step 102, based on the depth map at the real-time moment, the terrain information of each real-time terrain unit at the real-time moment can be determined through the following technical solution: based on the target depth map at the real-time moment, the terrain information of each real-time terrain unit corresponding to the target depth map at the real-time moment is determined.

[0078] As an example, based on the depth data, plane extraction is first performed, that is, all target planes in the local area are obtained. The target plane is the plane extracted by the algorithm. For example, if there are multiple plum blossom piles in the local area, the depth map at a certain moment can show these plum blossom piles at that moment, and the upper surface of each plum blossom pile can be extracted as a target plane, so as to obtain the facial depth map of the corresponding target plane in the depth map. In the subsequent step 102, based on the target depth map in the depth map at the real-time moment, the terrain information of each real-time terrain unit corresponding to the target depth map at the real-time moment can be determined. There is an overlap between the local area and the shooting area of the depth map. During the movement of the target device, any terrain unit can be captured by the depth camera multiple times. For example, a certain terrain unit is captured by the depth camera 10 times. Each time the terrain unit is captured by the depth camera, the terrain information of the terrain unit can be calculated using the depth values of the pixels of the corresponding terrain unit in the captured depth map. The terrain information includes terrain-related information such as height and plane normal.

[0079] In some embodiments, before performing the discretization process on the local area where the target device is located at the real-time moment in step 102 to obtain multiple real-time terrain units, any one of the following processes is performed: obtaining a first area positively correlated with the moving speed of the target device; obtaining a first area positively correlated with the volume of the target device; obtaining a local area centered on the target device and meeting the first area.

[0080] As an example, different discretization schemes can be adopted for different motion scenarios. First, the range of the local area needs to be determined. The range of the local area can be characterized by the first area. The first area is positively correlated with the moving speed of the target device and is also positively correlated with the volume of the target device. After determining the first area, a local area centered on the target device and meeting the first area can be obtained.

[0081] In some embodiments, in step 102, the discretization process on the local area where the target device is located at the real-time moment to obtain multiple real-time terrain units can be achieved through the following technical solutions: performing any one of the following processes: obtaining the number of terrain units positively correlated with the computing speed of the target device; obtaining the number of terrain units positively correlated with the memory capacity of the target device; performing unit segmentation processing on the local area at the real-time moment based on the number of terrain units to obtain multiple real-time terrain units.

[0082] As an example, different discretization schemes can be adopted for different target devices. First, the number of terrain units needs to be determined. The number of terrain units is positively correlated with the computing speed of the target device and also with the memory capacity of the target device. The more the number of terrain units, the higher the accuracy of terrain recognition. At the same time, the more the number of terrain units, the more computing amount will be brought, and a higher memory capacity is also required. After determining the number of terrain units according to the attributes of the target device, the local area is subjected to an average segmentation process based on the number of terrain units to obtain multiple real-time terrain units. The real-time terrain units are the terrain units obtained by discretization at the real-time moment.

[0083] In step 103, based on the terrain information of each real-time terrain unit at the real-time moment and the terrain information of each historical terrain unit at the historical moment, the terrain type of each real-time terrain unit at the real-time moment is determined.

[0084] As an example, in the time domain process, multiple shots will be taken by a depth camera. Assuming that the frame rate of the depth map is 30, a depth map will be output every 33 milliseconds. The real-time moment is the moment of the most recent shot of the depth map, and the historical moment is the shot moment before the real-time moment.

[0085] In some embodiments, refer to Figure 3B , Figure 3B is a schematic flowchart of the terrain processing method provided by the embodiment of the present application. In step 103, based on the terrain information of each real-time terrain unit at the real-time moment and the terrain information of each historical terrain unit at the historical moment, the terrain type of each real-time terrain unit at the real-time moment is determined, which can be implemented through Figure 3B shown in steps 1031 to 1032.

[0086] In step 1031, multiple terrain information intervals are obtained.

[0087] As an example, taking the terrain information as height for illustration, the terrain information interval is a height interval. For example, 10 cm to 12 cm is a height interval, and 12 cm to 14 cm is a height interval. The terrain information interval is determined according to the historical terrain information. First, the historical motion environment with a similarity higher than the similarity threshold to the current motion environment where the target device is located is obtained, the historical terrain information of the historical motion environment is collected, and the historical terrain information is equally divided into intervals to obtain multiple terrain information intervals. For example, the current motion environment is a primary school football field, and the historical motion environment with a similarity higher than the similarity threshold to the current motion environment where the target device is located is a known middle school football field. The historical terrain information obtained when the target device moves in the middle school football field is collected, and then the historical terrain information is equally divided into intervals to obtain multiple terrain information intervals. The number of terrain information intervals is positively correlated with the computing speed and memory capacity of the target device.

[0088] In step 1032, for each real-time terrain unit, the following processing is performed: Obtain a target terrain unit in at least one historical terrain unit that has the same position as the real-time terrain unit; Based on the terrain information of the real-time terrain unit at the real time and the terrain information of the target terrain unit at the historical time, determine the count number of the real-time terrain unit corresponding to each terrain information interval. Based on the count number of the real-time terrain unit corresponding to each terrain information interval, determine the terrain type of the real-time terrain unit at the real time.

[0089] In some embodiments, obtaining a target terrain unit in at least one historical terrain unit that has the same position as the real-time terrain unit in step 1032 can be achieved through the following technical solution: Perform self-state estimation processing through the sensors of the target device to obtain the real-time coordinate data of the target device corresponding to the world coordinate system at the real time; Obtain the historical coordinate data of the target device corresponding to the world coordinate system at each of the historical times; For each of the historical times, perform position calculation processing on each of the multiple historical terrain units at the historical time through the historical coordinate data at the historical time to obtain the first position of each of the historical terrain units at the historical time in the world coordinate system; Perform position calculation processing on the real-time terrain unit through the real-time coordinate data to obtain the second position of the real-time terrain unit in the world coordinate system. Obtain the historical terrain unit in each historical time where the first position is the same as the second position of the real-time terrain unit as the target terrain unit. Since the terrain unit is not a certain point, the position of the terrain unit is usually characterized by the position of the center of the terrain unit.

[0090] As an example, the robot performs self-state estimation through various types of sensors, such as cameras and IMUs, etc. The specific states include the position P (real-time coordinate data), velocity V, and orientation Q in the world coordinate system. Here, the camera usually refers to a color camera, and the position P (real-time coordinate data), velocity V, and orientation Q here are the results of state estimation; To obtain the position P (real-time coordinate data), velocity V, and orientation Q, a series of calculation tasks need to be completed based on sensor data. Here, the calculation tasks can be Bayesian filtering, Kalman filtering, etc. The method for obtaining the historical coordinate data of the target device corresponding to the world coordinate system at each historical time is the same as the method for obtaining the real-time coordinate data of the target device corresponding to the world coordinate system at the real time.

[0091] As an example, discretization is performed at each historical moment to obtain multiple historical terrain units corresponding to each historical moment. For a certain historical moment, based on the historical coordinate data of the target device at this historical moment, the historical coordinate data (the first position) of each historical terrain unit at this historical moment can be calculated. These historical coordinate data are all coordinates in the world coordinate system. For the real-time moment, based on the real-time coordinate data of the target device at this real-time moment, the real-time coordinate data (the second position) of each real-time terrain unit at this real-time moment can be calculated.

[0092] In some embodiments, in step 1032, based on the terrain information of the real-time terrain unit at the real-time moment and the terrain information of the target terrain unit at the historical moment, to determine the count number of the real-time terrain unit corresponding to each terrain information interval, the following technical solution can be implemented: For each terrain information interval, obtain the number of terrain information within the terrain information interval from the terrain information of the real-time terrain unit and the terrain information of the target terrain units at multiple historical moments, and use it as the count number of the real-time terrain unit corresponding to the terrain information interval.

[0093] As an example, taking the terrain information as height for illustration, the terrain information interval is the height interval. For example, 10 cm to 12 cm is a height interval, and 12 cm to 14 cm is a height interval. Suppose there are 10 historical moments. For example, for the height interval of 10 cm to 12 cm, obtain the number of heights within the height interval of 10 cm to 12 cm from the heights of the target terrain units at 10 historical moments and the height of the real-time terrain unit at the real-time moment, and use it as the count number corresponding to the height interval of 10 cm to 12 cm. At most, there is only one position of the historical terrain unit at each historical moment that is the same as the position of the real-time terrain unit. It is equivalent to statistically obtaining the number of terrain information falling within the height interval of 10 cm to 12 cm from the terrain information of at most 11 terrain units.

[0094] In some embodiments, in step 1032, based on the count number of the real-time terrain unit corresponding to each terrain information interval, to determine the terrain type of the real-time terrain unit at the real-time moment, the following technical solution can be implemented: Obtain the terrain information interval with the count number greater than the number threshold as the target terrain information interval of the real-time terrain unit, and use the terrain type corresponding to the target terrain information interval as the terrain type of the real-time terrain unit at the real-time moment.

[0095] As an example, after histogram statistics, the terrain type of each real-time terrain unit is judged respectively. Taking the terrain information as height for illustration, the terrain information interval is a height interval. For example, 10 cm to 12 cm is a height interval, and 12 cm to 14 cm is a height interval. For the real-time terrain unit A, the number threshold is 100. Among them, the count number corresponding to the height interval of 10 cm to 12 cm is 120. Then, the height interval of 10 cm to 12 cm is the target terrain information interval, and the terrain type corresponding to the height interval of 10 cm to 12 cm is used as the terrain type of the real-time terrain unit at the real-time moment.

[0096] In some embodiments, in step 1032, obtaining the target terrain unit having the same position as the real-time terrain unit in at least one historical terrain unit can be achieved through the following technical solution: when the second area of the real-time terrain unit is less than the area threshold, obtain the target terrain unit having the same position as the real-time terrain unit in at least one historical terrain unit; before obtaining the target terrain unit having the same position as the real-time terrain unit in at least one historical terrain unit, when the second area of the real-time terrain unit is not less than the area threshold, perform a plane fitting process on the real-time terrain unit to obtain the fitting plane in the real-time terrain unit, and update the fitting plane as the real-time terrain unit, so that subsequently, the position of the center of the fitting plane is used as the position of the real-time terrain unit.

[0097] As an example, refer to Figure 6 , for each real-time terrain unit, when the second area of the real-time terrain unit is not less than the area threshold, performing a plane fitting process on the real-time terrain unit is equivalent to obtaining a plane composed of three-dimensional points with a small depth gradient (the depth gradient is less than the depth gradient threshold) in a local spatial area. For example, the step surface of a staircase is very flat, and the depth gradient of the step surface is small, approaching zero, indicating that the ground is very flat; there is a step at the edge of the staircase, and the depth gradient here is very large. The plane composed of three-dimensional points with a small depth gradient can be expressed by a mathematical expression. Plane fitting can adopt the method of principal component analysis. If the division granularity of the terrain unit is small, such as the terrain unit is 1 cm * 1 cm, at this time, the plane fitting process can be skipped, that is, when the second area of the real-time terrain unit is less than the area threshold, directly obtain the target terrain unit having the same position as the real-time terrain unit in at least one historical terrain unit without performing plane fitting.

[0098] In step 104, for each real-time terrain unit whose terrain type belongs to the target terrain type, fuse the terrain information of the real-time terrain unit with the terrain information of the historical terrain unit at the same position at at least one historical moment to obtain the fused terrain information of the real-time terrain unit.

[0099] In some embodiments, referring to Figure 3C , Figure 3C is a schematic flowchart of a terrain processing method provided by an embodiment of the present application. In step 104, the terrain information of the real-time terrain unit is fused with the terrain information of the historical terrain unit at the same position at at least one historical moment to obtain the fused terrain information of the real-time terrain unit, which can be implemented through Figure 3C the steps 1041 to 1042 shown.

[0100] In step 1041, a target terrain unit having the same position as the real-time terrain unit is obtained from at least one historical terrain unit.

[0101] As an example, the self-state estimation process is performed on the target device through the sensor of the target device to obtain the real-time coordinate data of the target device corresponding to the world coordinate system at the real-time moment; the historical coordinate data of the target device corresponding to the world coordinate system at each historical moment is obtained; for each historical moment, the position calculation process is respectively performed on a plurality of historical terrain units at the historical moment through the historical coordinate data at the historical moment to obtain the first position of each historical terrain unit at the historical moment in the world coordinate system; the position calculation process is performed on the real-time terrain unit through the real-time coordinate data to obtain the second position of the real-time terrain unit in the world coordinate system. The historical terrain unit with the same first position and the second position of the real-time terrain unit in each historical moment is obtained as the target terrain unit.

[0102] In step 1042, the terrain information of the real-time terrain unit is fused with the terrain information of the target terrain unit to obtain the fused terrain information of the real-time terrain unit.

[0103] In some embodiments, in step 1042, the terrain information of the real-time terrain unit is fused with the terrain information of the target terrain unit to obtain the fused terrain information of the real-time terrain unit, which can be implemented through the following technical solutions: obtaining the reciprocal of the distance between the real-time terrain unit and the target device at the real-time moment as the weight of the real-time terrain unit; obtaining the reciprocal of the distance between each target terrain unit and the target device at the corresponding historical moment as the weight of each target terrain unit; based on the weight of the real-time terrain unit and the weight of each target terrain unit, performing a weighted summation process on the terrain information of the real-time terrain unit and the terrain information of the target terrain unit to obtain the fused terrain information of the real-time terrain unit.

[0104] As an example, the data after histogram statistics is subjected to time-domain weighted averaging. The corresponding weights can be the reciprocals of the distances between the terrain units (the real-time terrain unit and the historical terrain unit at the same position) and the depth sensor at each moment (historical moments and real-time moments) in the time domain. The statistical metrics for weighted averaging include terrain-related information such as the height and plane normal of the terrain units (the real-time terrain unit and the historical terrain unit at the same position) in the world coordinate system. The fused terrain information of the real-time terrain unit obtained is the high-quality discrete terrain information.

[0105] Through the embodiments of the present application, the local area where the target device is located is first discretized into terrain units, and the type of each terrain unit is judged from a statistical perspective, effectively coping with the incorrect depth map caused by continuous force impact or continuous vibration. For each terrain unit of the target terrain type, the terrain information of each terrain unit at different moments is fused in the time domain, and finally high-quality fused terrain information is obtained. Thus, real-time, accurate, and robust terrain information can be provided in a motion scenario.

[0106] Next, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0107] In some embodiments, the terrain processing method provided by the embodiments of the present application can be applied to a robot control APP. In response to a movement instruction triggered by a user on the terminal, the terminal sends a movement instruction to the target device. During the movement of the target device, the depth map captured at the real-time moment (captured by the target device) and the local area where the target device is located at the real-time moment are sent to the server. The server discretizes the local area to obtain a plurality of real-time terrain units, and based on the depth map at the real-time moment, determines the terrain information of each real-time terrain unit at the real-time moment. Based on the terrain information of each real-time terrain unit at the real-time moment and the terrain information of each historical terrain unit at the historical moment, determines the terrain type of each real-time terrain unit at the real-time moment. For each real-time terrain unit whose terrain type belongs to the target terrain type, the terrain information of the real-time terrain unit and the terrain information of the historical terrain unit at the same position at at least one historical moment are fused to obtain the fused terrain information of the real-time terrain unit. When the target device is an automatically controlled robot, the server returns the fused terrain information of the real-time terrain unit to the target device. The target device determines a movement route based on the fused terrain information of the real-time terrain unit. When the target device is a robot controlled by the terminal, the server returns the fused terrain information of the real-time terrain unit to the terminal. The terminal determines a movement route based on the fused terrain information of the real-time terrain unit, and then sends the movement route to the target device. When the target device is a robot controlled by the server, the server determines a movement route based on the fused terrain information of the real-time terrain unit, and then sends the movement route to the target device.

[0108] The embodiments of the present application are mainly applied to the terrain recognition module of a robot, especially a fast-moving legged robot, and can provide real-time, accurate and robust terrain information for the robot.

[0109] See Figure 4 , Figure 4 which is a schematic framework diagram of the terrain processing method provided by the embodiments of the present application. In step 501, depth data is acquired. In step 502, local spatial domain discretization processing is performed. In step 503, visual positioning processing is performed. In step 504, statistical distribution processing is performed on each discrete terrain unit by combining the visual positioning result and the depth data to obtain high-quality terrain information.

[0110] In some embodiments, the depth data can be from a depth camera or a lidar, etc. The depth data can reflect the distance information of the scene terrain relative to the sensor itself (depth camera or lidar) and has a real scale. In the fast-moving scenario of a legged robot, due to the relatively long exposure time of the depth camera or the too fast movement speed of the robot, problems such as motion blur will occur in the acquired depth data, resulting in relatively large errors.

[0111] In some embodiments, see Figure 5 , Figure 5 which is a schematic diagram of the local spatial domain discretization of the terrain processing method provided by the embodiments of the present application. The embodiments of the present application will discretize the local spatial area near the robot into regular terrain units. For example, the range of 5m * 5m around the robot. This is actually a three-dimensional space. For example, both the x-direction and the y-direction on the horizontal plane are 5m, and there is no restriction on the vertical z-direction, which can be understood as 100m or infinite. The x-direction and the y-direction here are more important and determine the size of the local area. The z-direction is just a value. So the local spatial area tends to be a two-dimensional projection. The area of each terrain unit depends on the specific scenario and can be 1 cm * 1 cm or 5 cm * 5 cm. The actual coordinates of these terrain units are in the world coordinate system under the robot state estimation, and all subsequent calculations are performed in the world coordinate system.

[0112] In some embodiments, the robot estimates its own state through various types of sensors, such as cameras and IMUs. The specific states include the position P, velocity V, and orientation Q in the world coordinate system. Here, the camera is a general term, including conventional color cameras and depth cameras, usually referring to color cameras. The position P, velocity V, and orientation Q here are the results of state estimation. To obtain the position P, velocity V, and orientation Q, a series of computational tasks need to be completed based on sensor data. There can be many types of sensors here, such as color cameras, depth cameras, IMUs, etc., which can be selectively used. The main role of the depth data in the embodiments of the present application is to perform terrain detection, which has no direct relation with the position P, velocity V, and orientation Q. It's just that during the terrain detection process, the position P, velocity V, and orientation Q are needed to ensure the unified position of terrain units and perform time-domain fusion based on terrain units at the same position.

[0113] In some embodiments, during the time domain process, state statistics and screening are performed on each terrain unit. Suppose the frame rate of the depth map is 30, then the result of visual positioning will be output every 33 milliseconds. During the movement of the robot, each terrain unit can be seen by the depth camera multiple times. For example, a certain terrain unit is seen by the depth camera 10 times. Each time the terrain unit is seen by the depth camera, the coordinates of this terrain unit can be transformed into the world coordinate system according to the visual positioning result (position P, velocity V, and orientation Q) at this time. In theory, the coordinates of a certain terrain unit in the world coordinate system when seen 10 times should be the same, but due to errors, the coordinates will be different and need to be optimized and fused. See Figure 6 , for each terrain unit, first, plane fitting processing is performed, which is equivalent to obtaining a plane composed of three-dimensional points with a small depth gradient in a local spatial area. For example, the step surface of a staircase is very flat, and the depth gradient of the step surface is small, close to zero, indicating that the ground is very flat; there is a step at the edge of the staircase, and the depth gradient here is very large. The plane composed of three-dimensional points with a small depth gradient can be expressed by a mathematical expression. Plane fitting can be carried out in the way of principal component analysis; secondly, time-domain histogram statistics are performed on the terrain unit. The statistical indicators can be information such as the height of the center of the terrain unit in the world coordinate system or the plane normal in the time domain process. Based on the threshold range, the histogram results are screened to obtain the statistical indicator corresponding to the data with the largest proportion, and at the same time, the data outside the threshold range is deleted. For example, the horizontal axis of the histogram result is the indicator interval, such as the height of the terrain unit. 10 cm to 12 cm is an indicator interval, and 12 cm to 14 cm is an indicator interval. The vertical axis is the number of times the statistical indicator of the terrain unit falls within this interval. See Figure 7 , Figure 7In the scene shown, there is a square wooden stake. Due to depth information errors or ghosting, the terrain from a certain frame of depth map or after temporal fusion may not be a square. However, through a statistical distribution strategy, only the square formed by the terrain units in the middle area is considered as the terrain T above the ground, and the mis-identified terrain units around it are classified as the ground G. The above-mentioned certain frame of depth map refers to the original data without fusion processing; the terrain after temporal fusion can be understood as having undergone at least one fusion process. Temporal fusion is carried out continuously. Here, the temporal fusion may be the temporal fusion at the middle moment, but the resulting temporal fusion result is not the final result. It can be set that the result obtained after a set number of temporal fusions is the final result.

[0114] Finally, temporal weighted average processing is performed. The data after histogram statistics is temporally weighted averaged, and the corresponding weights can be the reciprocals of the distances of the terrain units from the depth sensor during the temporal process. The statistical metrics for weighted average include terrain-related information such as the height and plane normal of the terrain units in the world coordinate system. Thus, high-quality discrete terrain information within the local airspace range is obtained.

[0115] If the division granularity of the terrain units is small, for example, the terrain unit is 1 cm * 1 cm, the plane fitting process can be skipped at this time, and only subsequent operations are performed based on terrain-related information such as the height and plane normal of the terrain units in the world coordinate system.

[0116] The above process is directly based on the original depth data. It can also be referred to Figure 8 , first perform plane extraction on the basis of the original depth data, that is, obtain all target planes in the entire scene. The effective planes are the planes extracted by the algorithm. For example, in a terrain scene with multiple plum blossom stakes, the depth map at a certain moment can show these plum blossom stakes at that moment, and the upper surfaces of each plum blossom stake can be extracted. As Figure 9 shown, two effective planes are first extracted in the scene. Here, ordinary terrains such as the ground can be filtered to save computing resources, and then repeat the Figure 4 steps shown. After temporal fusion, accurate and robust terrain information can be obtained.

[0117] Benefiting from the discretization method provided by the embodiments of the present application, the terrain fusion strategy of the embodiments of the present application can achieve a frame rate of more than 30 Hz or even higher. The embodiments of the present application mainly solve the problem of large terrain recognition errors in the fast-moving scenarios of robots. Through strategies such as scene discretization and statistical distribution, the terrain recognition errors caused by continuous force impacts or high-frequency vibrations are reduced as much as possible, enabling the robot to accurately and robustly perceive various terrains in the environment during fast movement and achieve the purpose of quickly traversing complex environments, which has practical significance for the robot to truly enter human habitats and other environments.

[0118] It is understandable that in the embodiments of the present application, data related to user information, etc. is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0119] Next, the exemplary structure of the terrain processing device 455 provided in the embodiments of the present application implemented as software modules will be continued. In some embodiments, as Figure 2 shown, the software modules stored in the terrain processing device 455 in the memory 450 may include: an acquisition module 4551 for acquiring a depth map obtained by the target device at a real-time moment; a discretization module 4552 for discretizing the local area where the target device is located at a real-time moment to obtain a plurality of real-time terrain units, and determining the terrain information of each real-time terrain unit at the real-time moment based on the depth map at the real-time moment; a terrain module 4553 for determining the terrain type of each real-time terrain unit at the real-time moment based on the terrain information of each real-time terrain unit at the real-time moment and the terrain information of each historical terrain unit at a historical moment, where the historical moment is a moment before the real-time moment; a fusion module 4554 for, for each real-time terrain unit whose terrain type belongs to the target terrain type, performing a fusion process on the terrain information of the real-time terrain unit and the terrain information of the historical terrain unit at the same position at at least one historical moment to obtain the fused terrain information of the real-time terrain unit.

[0120] In some embodiments, the acquisition module 4551 is further configured to: detect a depth map of a corresponding shooting area through a sensor of the electronic device, where the shooting area and the local area have an overlapping area at the real-time moment, and the depth map includes the depth value of each pixel.

[0121] In some embodiments, the acquisition module 4551 is further configured to: after acquiring the depth map obtained by the target device at a real-time moment, perform target plane recognition processing on the depth map to obtain a target depth map of the corresponding target plane in the depth map; the discretization module 4552 is further configured to: determine the terrain information of each real-time terrain unit corresponding to the target depth map at the real-time moment based on the target depth map at the real-time moment.

[0122] In some embodiments, before discretizing the local area where the target device is located at a real-time moment to obtain a plurality of real-time terrain units, the discretization module 4552 is further configured to perform any one of the following processes: acquire a first area positively correlated with the moving speed of the target device; acquire a first area positively correlated with the volume of the target device; acquire a local area centered on the target device and meeting the first area.

[0123] In some embodiments, the discrete module 4552 is further configured to perform any one of the following processes: obtaining the number of terrain units that is positively correlated with the computing speed of the target device; obtaining the number of terrain units that is positively correlated with the memory capacity of the target device; performing unit segmentation processing on the local area at the real-time moment based on the number of terrain units to obtain a plurality of real-time terrain units.

[0124] In some embodiments, the terrain module 4553 is further configured to: obtain a plurality of terrain information intervals; for each real-time terrain unit, perform the following processes: obtain a target terrain unit in at least one historical terrain unit that has the same position as the real-time terrain unit; based on the terrain information of the real-time terrain unit at the real-time moment and the terrain information of the target terrain unit at the historical moment, determine the count number of the real-time terrain unit corresponding to each terrain information interval; based on the count numbers of the plurality of real-time terrain units corresponding to each terrain information interval, determine the terrain type of each real-time terrain unit at the real-time moment.

[0125] In some embodiments, the terrain module 4553 is further configured to: perform self-state estimation processing through the sensors of the target device to obtain the real-time coordinate data of the target device corresponding to the world coordinate system at the real-time moment; perform position calculation processing on a plurality of historical terrain units at a plurality of historical moments respectively through the real-time coordinate data to obtain the first position of each historical terrain unit in the world coordinate system; perform position calculation processing on the real-time terrain unit through the real-time coordinate data to obtain the second position of the real-time terrain unit in the world coordinate system, and obtain the historical terrain unit whose first position is the same as the second position of the real-time terrain unit in each historical moment as the target terrain unit.

[0126] In some embodiments, the terrain module 4553 is further configured to: for each terrain information interval, obtain the number of terrain information that is within the terrain information interval from the terrain information of the real-time terrain unit and the terrain information of the target terrain units at a plurality of historical moments, as the count number of the real-time terrain unit corresponding to the terrain information interval.

[0127] In some embodiments, the terrain module 4553 is further configured to: for each real-time terrain unit, obtain the terrain information interval whose count number is greater than the number threshold as the target terrain information interval of the real-time terrain unit, and use the terrain type corresponding to the target terrain information interval as the terrain type of the real-time terrain unit at the real-time moment.

[0128] In some embodiments, the terrain module 4553 is further configured to: when the area of the second region of the real-time terrain unit is less than the region area threshold, obtain a target terrain unit in at least one historical terrain unit that has the same position as the real-time terrain unit; before obtaining the target terrain unit in at least one historical terrain unit that has the same position as the real-time terrain unit, when the area of the second region of the real-time terrain unit is not less than the region area threshold, perform a plane fitting process on the real-time terrain unit to obtain a fitting plane in the real-time terrain unit, and update the fitting plane as the real-time terrain unit.

[0129] In some embodiments, the fusion module 4554 is further configured to: obtain a target terrain unit in at least one historical terrain unit that has the same position as the real-time terrain unit; perform a fusion process on the terrain information of the real-time terrain unit and the terrain information of the target terrain unit to obtain the fused terrain information of the real-time terrain unit.

[0130] In some embodiments, the fusion module 4554 is further configured to: obtain the reciprocal of the distance between the real-time terrain unit and the target device at the real-time moment as the weight of the real-time terrain unit; obtain the reciprocal of the distance between each target terrain unit and the target device at the corresponding historical moment as the weight of each target terrain unit; based on the weight of the real-time terrain unit and the weights of each target terrain unit, perform a weighted summation process on the terrain information of the real-time terrain unit and the terrain information of the target terrain unit to obtain the fused terrain information of the real-time terrain unit.

[0131] An embodiment of the present application provides a computer program product, which includes a computer program or computer executable instructions, and the computer program or computer executable instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the terrain processing method described above in the embodiments of the present application.

[0132] An embodiment of the present application provides a computer-readable storage medium storing computer executable instructions, where the computer executable instructions are stored, and when the computer executable instructions are executed by a processor, the processor will be caused to execute the terrain processing method provided in the embodiments of the present application, for example, Figures 3A - 3C the terrain processing method shown.

[0133] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0134] In some embodiments, the computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as a stand-alone program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0135] As an example, the computer-executable instructions may or may not correspond to a file in a file system, may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, stored in a single file dedicated to the program in question, or, stored in multiple cooperating files (such as files that store one or more modules, subroutines, or portions of code).

[0136] As an example, the computer-executable instructions may be deployed to execute on one electronic device, or on multiple electronic devices located at one location, or, on multiple electronic devices distributed at multiple locations and interconnected by a communication network.

[0137] In summary, through the embodiments of the present application, the local area where the target device is located is first discretized into terrain units, and the type of the terrain unit is judged from a statistical perspective, effectively coping with the incorrect depth map caused by continuous force impact or continuous vibration. For each terrain unit of the target terrain type, the terrain information of each terrain unit at different times is fused in the time domain process, and finally high-quality fused terrain information is obtained. Thus, real-time, accurate, and robust terrain information can be provided in a motion scenario.

[0138] The above is only the embodiments of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are all included in the protection scope of the present application.

Claims

1. A terrain processing method, characterized in that, The method includes: Obtaining a depth map captured by a target device at a real-time moment; Performing discretization processing on a local area where the target device is located at the real-time moment to obtain a plurality of real-time terrain units, and determining terrain information of each real-time terrain unit at the real-time moment based on the depth map at the real-time moment; Obtaining a plurality of terrain information intervals, and for each real-time terrain unit, performing the following processing: obtaining a target terrain unit having the same position as the real-time terrain unit in at least one historical terrain unit; determining the count number of the real-time terrain unit corresponding to each terrain information interval based on the terrain information of the real-time terrain unit at the real-time moment and the terrain information of the target terrain unit at a historical moment; determining the terrain type of the real-time terrain unit at the real-time moment based on the count number of the real-time terrain unit corresponding to each terrain information interval, where the historical moment is a moment before the real-time moment; For each real-time terrain unit whose terrain type belongs to a target terrain type, obtaining the reciprocal of the distance between the real-time terrain unit and the target device at the real-time moment as the weight of the real-time terrain unit; obtaining the reciprocal of the distance between each target terrain unit and the target device at the corresponding historical moment as the weight of each target terrain unit; performing weighted summation processing on the terrain information of the real-time terrain unit and the terrain information of each target terrain unit based on the weight of the real-time terrain unit and the weight of each target terrain unit to obtain the fused terrain information of the real-time terrain unit.

2. The method according to claim 1, characterized in that, Before performing discretization processing on the local area where the target device is located at the real-time moment to obtain a plurality of real-time terrain units, it includes: Performing any one of the following processing: Obtaining a first area size positively correlated with the moving speed of the target device; Obtaining a first area size positively correlated with the volume of the target device; Obtaining a local area centered on the target device and meeting the first area size.

3. The method according to claim 1, characterized in that The discretization processing on the local area where the target device is located at the real-time moment to obtain a plurality of real-time terrain units includes: Performing any one of the following processing: Obtaining the number of terrain units positively correlated with the computing speed of the target device; Obtaining the number of terrain units positively correlated with the memory capacity of the target device; Performing unit segmentation processing on the local area at the real-time moment based on the number of terrain units to obtain a plurality of the real-time terrain units.

4. The method according to claim 1, wherein The obtaining of a target terrain unit having the same position as the real-time terrain unit in at least one historical terrain unit includes: Performing self-state estimation processing through a sensor of the target device to obtain real-time coordinate data of the target device corresponding to the world coordinate system at the real-time moment; Obtaining historical coordinate data of the target device corresponding to the world coordinate system at each historical moment; For each of the historical moments, position calculation processing is respectively performed on multiple historical terrain units at the historical moment through the historical coordinate data of the historical moment, to obtain the first position of each of the historical terrain units at the historical moment in the world coordinate system; Position calculation processing is performed on the real-time terrain unit through the real-time coordinate data, to obtain the second position of the real-time terrain unit in the world coordinate system; Historical terrain units in each of the historical moments, whose first positions are the same as the second position of the real-time terrain unit, are obtained as the target terrain units.

5. The method according to claim 1, wherein The determining the count number of the real-time terrain unit corresponding to each terrain information interval based on the terrain information of the real-time terrain unit at the real-time moment and the terrain information of the target terrain unit at the historical moment includes: For each terrain information interval, the number of terrain information within the terrain information interval is obtained from the terrain information of the real-time terrain unit and the terrain information of the target terrain units at multiple historical moments, as the count number of the real-time terrain unit corresponding to the terrain information interval.

6. The method according to claim 1, characterized in that, The determining the terrain type of the real-time terrain unit at the real-time moment based on the count number of the real-time terrain unit corresponding to each terrain information interval includes: The terrain information interval with a count number greater than the number threshold is obtained as the target terrain information interval of the real-time terrain unit, and the terrain type corresponding to the target terrain information interval is used as the terrain type of the real-time terrain unit at the real-time moment.

7. The method according to claim 1, characterized in that, The obtaining the target terrain unit having the same position as the real-time terrain unit in at least one historical terrain unit includes: When the second area of the real-time terrain unit is less than the area threshold, the target terrain unit having the same position as the real-time terrain unit in at least one of the historical terrain units is obtained; Before obtaining the target terrain unit having the same position as the real-time terrain unit in at least one of the historical terrain units, the method further includes: When the second area of the real-time terrain unit is not less than the area threshold, plane fitting processing is performed on the real-time terrain unit to obtain the fitting plane in the real-time terrain unit, and the fitting plane is updated as the real-time terrain unit.

8. The method according to claim 1, characterized in that, After obtaining the depth map captured by the target device at the real-time moment, the method further includes: Target plane recognition processing is performed on the depth map to obtain the target depth map corresponding to the target plane in the depth map; The determining the terrain information of each real-time terrain unit at the real-time moment based on the depth map at the real-time moment includes: Based on the target depth map at the real-time moment, the terrain information of each real-time terrain unit corresponding to the target depth map at the real-time moment is determined.

9. A terrain processing device, characterized in that, The apparatus includes: An acquisition module, configured to acquire the depth map captured by the target device at the real-time moment; A discrete module for discretizing the local area where the target device is located at a real-time moment to obtain a plurality of real-time terrain units, and determining the terrain information of each real-time terrain unit based on the depth map at the real-time moment; A terrain module for obtaining a plurality of terrain information intervals, and for each real-time terrain unit, performing the following processing: obtaining a target terrain unit in at least one historical terrain unit that has the same position as the real-time terrain unit; determining the count number of the real-time terrain unit corresponding to each terrain information interval based on the terrain information of the real-time terrain unit at the real-time moment and the terrain information of the target terrain unit at a historical moment; determining the terrain type of the real-time terrain unit at the real-time moment based on the count number of the real-time terrain unit corresponding to each terrain information interval, where the historical moment is a moment before the real-time moment; A fusion module for, for each real-time terrain unit whose terrain type is a target terrain type, obtaining the reciprocal of the distance between the real-time terrain unit and the target device at the real-time moment as the weight of the real-time terrain unit; obtaining the reciprocal of the distance between each target terrain unit and the target device at the corresponding historical moment as the weight of each target terrain unit; performing a weighted summation process on the terrain information of the real-time terrain unit and the terrain information of the target terrain unit based on the weight of the real-time terrain unit and the weight of each target terrain unit to obtain the fused terrain information of the real-time terrain unit.

10. An electronic device, characterized in that, The electronic device includes: A memory for storing computer-executable instructions; A processor for, when executing the computer-executable instructions stored in the memory, implementing the terrain processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by the processor, implement the terrain processing method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, The computer program or computer-executable instructions, when executed by the processor, implement the terrain processing method according to any one of claims 1 to 8.

Images