A method and apparatus for generating a depth map, an electronic device, and a storage medium

Abstract

A method and device for generating a depth map, an electronic device, and a storage medium are disclosed. The method comprises: obtaining panoramic image data and laser point cloud data of a target region, and dividing the panoramic image data into a plurality of pinhole image data; obtaining laser point cloud sub-data of each pinhole image data according to the laser point cloud data; and generating a depth map corresponding to each pinhole image data according to the laser point cloud sub-data of each pinhole image data, respectively. The scheme provided in the present application can break through the limitation of the traditional method of obtaining a depth map by a depth camera for measuring range, can generate a depth map of a large scene, and can save computing resources and improve the generation speed of the depth map.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for generating depth maps. Background Technology

[0002] In 3D computer graphics and computer vision, a depth map is an image or image channel in which each pixel value represents information about the distance of a point in a scene object to the camera, used to simulate or reconstruct 3D shapes.

[0003] Currently, depth maps are typically acquired directly using depth cameras. However, existing depth cameras usually have a measurement range of less than 10 meters, which is often insufficient for acquiring depth maps in larger scenes. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for generating depth maps, which can overcome the limitations of traditional depth map acquisition via depth cameras in terms of measurement range, generate depth maps for large scenes, save computing resources, and improve the generation speed of depth maps.

[0005] According to one aspect of the present invention, a method for generating a depth map is provided, comprising:

[0006] Acquire panoramic image data and laser point cloud data of the target area, and segment the panoramic image data into multiple pinhole image data;

[0007] Based on the laser point cloud data, obtain the laser point cloud sub-data for each pinhole image data;

[0008] Based on the laser point cloud sub-data of each pinhole image, a depth map corresponding to each pinhole image is generated.

[0009] Optionally, the panoramic image data can be segmented into multiple pinhole camera data, including:

[0010] Obtain the segmentation parameters, which include the width, height, and focal length of the pinhole image data;

[0011] Determine the rotation angle based on the width, height, and focal length of the pinhole image data;

[0012] Using the Z-axis of the panoramic sphere as the rotation axis, each rotation angle extracts a pinhole image from the panoramic data. After one rotation, multiple pinhole image data are obtained.

[0013] Optionally, for any pinhole image data, based on the laser point cloud data, obtain the laser point cloud sub-data of the pinhole image data, including:

[0014] The laser point cloud data is filtered once to obtain the first remaining point cloud data.

[0015] The first remaining point cloud data is filtered a second time to obtain the second remaining point cloud data;

[0016] The second remaining point cloud data is filtered three times to obtain the laser point cloud sub-data of the pinhole map data.

[0017] Optionally, the laser point cloud data can be filtered once to obtain the first remaining point cloud data, including:

[0018] Determine the timestamp of the panoramic image data to which the pinhole camera data belongs;

[0019] Based on the timestamp, the laser point cloud data is filtered once to obtain the first remaining point cloud data. The shooting time of the point cloud included in the first remaining point cloud data is all within the timestamp.

[0020] Optionally, the first remaining point cloud data is filtered a second time to obtain the second remaining point cloud data, including:

[0021] Calculate the position and orientation of the pinhole image data;

[0022] Based on the position and orientation of the pinhole image data, determine the origin and orientation of the view frustum, and generate the view frustum based on the origin, orientation, distance between near and far clipping planes and the camera field of view (FOV).

[0023] Based on the view frustum, the first remaining point cloud data is filtered a second time to obtain the second remaining point cloud data, all of which are located within the view frustum.

[0024] Optionally, the second remaining point cloud data is filtered three times to obtain the laser point cloud sub-data of the pinhole map data, including:

[0025] An octree grid is built on the second remaining point cloud data. The octree grid includes multiple subgrids, and each subgrid includes a portion of the point cloud of the second remaining point cloud data.

[0026] The target point cloud of each subgrid is determined from the partial point cloud of the second remaining point cloud data included in each subgrid. The target point cloud is the point cloud closest to the center of the subgrid. The target point clouds of all subgrids together constitute the laser point cloud subdata of the pinhole map data.

[0027] Optionally, for any pinhole map data, a depth map corresponding to the pinhole map data is generated based on the laser point cloud sub-data of the pinhole map data, including:

[0028] Create a blank plot, with the same dimensions as the pinhole plot data;

[0029] The laser point cloud sub-data of the pinhole map data is projected onto a blank map to generate a depth map corresponding to the pinhole map data.

[0030] According to another aspect of the present invention, a depth map generation apparatus is provided, comprising: a data acquisition module, a data processing module, and a depth map generation module;

[0031] The data acquisition module is used to acquire panoramic image data and laser point cloud data of the target area;

[0032] The data processing module is used to segment panoramic image data into multiple pinhole image data; and to obtain laser point cloud sub-data for each pinhole image data based on laser point cloud data;

[0033] The depth map generation module is used to generate a depth map corresponding to each pinhole image based on the laser point cloud sub-data of each pinhole image data.

[0034] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0035] At least one processor; and

[0036] A memory that is communicatively connected to at least one processor; wherein,

[0037] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the depth map generation method of any embodiment of the present invention.

[0038] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute a method for generating a depth map according to any embodiment of the present invention.

[0039] The technical solution of this invention acquires panoramic image data and laser point cloud data of the target area, divides the panoramic image data into multiple pinhole image data, obtains laser point cloud sub-data for each pinhole image data based on the laser point cloud data, and then generates a depth map corresponding to each pinhole image data based on the laser point cloud sub-data for each pinhole image data. This overcomes the limitations of traditional depth map acquisition via depth cameras in terms of measurement range, enabling the generation of depth maps for large scenes. Furthermore, dividing the panoramic image data into multiple pinhole image data and processing them separately saves computational resources and improves the speed of depth map generation.

[0040] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating a method for generating a depth map according to Embodiment 1 of the present invention;

[0043] Figure 2 This is a schematic diagram of panoramic image data provided in Embodiment 1 of the present invention;

[0044] Figure 3 This is a schematic diagram of laser point cloud data provided in Embodiment 1 of the present invention;

[0045] Figure 4 This is a method based on Embodiment 1 of the present invention. Figure 2 The diagram shows a portion of the pinhole camera data obtained by segmenting panoramic image data.

[0046] Figure 5 This is a schematic diagram of a viewing cone provided in Embodiment 1 of the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of a depth map generation device provided in Embodiment 2 of the present invention;

[0048] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0049] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0050] It should be noted that the terms "first," "second," "third," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0051] Example 1

[0052] Figure 1 This is a schematic flowchart illustrating a depth map generation method according to Embodiment 1 of the present invention. This embodiment is applicable to the generation of depth maps. The method can be executed by a depth map generation device, which can be implemented in hardware and / or software and can be configured in an electronic device (such as a computer or server). Figure 1 As shown, the method includes:

[0053] S110. Acquire panoramic image data and laser point cloud data of the target area, and divide the panoramic image data into multiple pinhole image data.

[0054] The target area can be any spatial area, such as a house, playground, warehouse, etc.

[0055] Typically, panoramic image data and laser point cloud data of a target area can be acquired using sensors mounted on backpack LiDAR devices. Panoramic image data (also known as 360° panoramic images) can represent the environment of the target area as comprehensively as possible through wide-angle representations and formats such as paintings, photographs, videos, and 3D models. Laser point cloud data is a dataset of spatial points, each containing three-dimensional coordinate information, namely X, Y, and Z elements; some may also include color information, reflection intensity information, and echo count information. For example... Figure 2 This is a schematic diagram of panoramic image data provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of laser point cloud data provided in Embodiment 1 of the present invention.

[0056] In this invention, the measurement range of the panoramic image data and laser point cloud data of the target area is determined by the performance of the lidar equipment, and can usually reach hundreds or even thousands of meters, thereby meeting the needs of generating depth maps for large scenes.

[0057] In one embodiment, the number of panoramic image data can be one frame or multiple frames. For each panoramic image data, the step of "segmenting the panoramic image data into multiple pinhole image data" must be performed. Specifically, the method for segmenting the panoramic image data into multiple pinhole image data can include the following three steps.

[0058] Step A1: Obtain the segmentation parameters, which include the width, height, and focal length of the pinhole image data.

[0059] Step A2: Determine the rotation angle based on the width, height, and focal length of the pinhole image data.

[0060] Step A3: Using the Z-axis of the panoramic sphere as the rotation axis, each rotation angle will extract a pinhole image data from the panoramic image data. After one rotation, multiple pinhole image data will be obtained.

[0061] Referring to steps A1-A3 above, the size of the rotation angle K determines the number of pinhole camera data points. For example, assuming the rotation angle K is 40°, one pinhole camera data point can be extracted from the panoramic image data for every 40° rotation. After one full rotation (360°), a total of 9 pinhole camera data points can be obtained. For example, Figure 4 A method based on Embodiment 1 of the present invention Figure 2 The diagram shows a partial pinhole camera image obtained by segmenting panoramic image data.

[0062] In this invention, panoramic image data is divided into multiple pinhole image data. These multiple pinhole image data can be processed in parallel to obtain laser point cloud sub-data of pinhole image data and generate corresponding depth maps, thereby improving the overall depth map generation speed.

[0063] S120. Based on the laser point cloud data, obtain the laser point cloud sub-data for each pinhole map data.

[0064] To save computing resources and improve the generation efficiency of depth maps corresponding to individual pinhole image data, this invention can also process laser point cloud data to obtain laser point cloud sub-data for each pinhole image data.

[0065] Specifically, the method for obtaining laser point cloud sub-data for each pinhole image based on laser point cloud data can include the following three steps.

[0066] Step B1: Filter the laser point cloud data once to obtain the first remaining point cloud data.

[0067] In step B1, the first filtering can also be called time filtering. The specific method can be: determining the timestamp of the panoramic image data to which the pinhole image data belongs; and filtering the laser point cloud data according to the timestamp to obtain the first remaining point cloud data. The shooting time of the point cloud included in the first remaining point cloud data is within the timestamp.

[0068] For example, assuming there are 5 frames of panoramic image data, the timestamp of the first frame refers to the time from when the first frame was captured to before the second frame was captured; the timestamp of the second frame refers to the time from when the first frame was captured to before the third frame; the timestamp of the third frame refers to the time from when the second frame was captured to before the fourth frame; the timestamp of the fourth frame refers to the time from when the third frame was captured to before the fifth frame; and the timestamp of the fifth frame refers to the time from when the fourth frame was captured to when the capture was completed. Each frame of panoramic image data is divided into 6 pinhole camera data points. After step S110 is completed, there are a total of 30 pinhole camera data points. The first filtering step mainly removes point clouds whose capture time is not within the timestamp of the panoramic image data to which the pinhole camera data belongs, based on the timestamp of the panoramic image data to which the pinhole camera data belongs. When the number of panoramic image data points is large, this step can effectively filter most of the point clouds, greatly saving computing resources.

[0069] Apart from the first and last panoramic data, step B1 can be represented by equation (1):

[0070]

[0071] The constraint conditions of equation (1) are as follows:

[0072] Where FilterPoints_K represents the first remaining point cloud data after filtering the pinhole image data of the Kth frame panoramic image data, pti.time represents the timestamp of the i-th point in the point cloud, Pano() represents the conversion interface from pinhole image data to panoramic image data, Pano(K).time represents the timestamp of the Kth frame panoramic image data, Pano(K).index represents the identifier (ID) of the Kth frame panoramic image data, and N represents the maximum ID of the panoramic image data. The ID range of the panoramic image data is [1, N].

[0073] Understandably, if the panoramic image data consists of only one frame, then all the pinhole image data obtained from segmentation belongs to this frame of panoramic image data. That is, the shooting time of all point clouds included in the laser point cloud data is within the timestamp of this frame of panoramic image data, and the point clouds will not be filtered out in one pass. In other words, if the panoramic image data consists of only one frame, then step B1 does not need to be executed.

[0074] Step B2: Perform secondary filtering on the first remaining point cloud data to obtain the second remaining point cloud data.

[0075] In step B2, the secondary filtering, also known as frustum filtering, can be performed as follows: calculate the position and orientation of the pinhole image data; determine the origin and orientation of the frustum based on the position and orientation of the pinhole image data, and generate the frustum based on the origin, orientation, distance between near and far clipping planes and the field of view (FOV); and perform secondary filtering on the first remaining point cloud data based on the frustum to obtain the second remaining point cloud data, wherein all points in the second remaining point cloud data are located within the frustum.

[0076] Specifically, the position of pinhole image data i is Twc_i = Tc, where Tc is the position of the panoramic image data to which pinhole image data i belongs in the world coordinate system; the pose of pinhole image data i is Rwc_i = Rwp * Rpc_i, where Rwp is the rotation matrix of the panoramic image data to which pinhole image data i belongs to the world coordinate system, and Rpc_i is the rotation matrix of pinhole image data i to the starting point of the segmentation of the panoramic image data to which it belongs (for example, taking a rotation angle of 40° as an example, the rotation of the first pinhole image data to the panoramic coordinate system to which it belongs is 40 degrees, and so on).

[0077] The distance between the near and far cutting surfaces can be set according to the actual situation. Typically, the range of the distance between the near and far cutting surfaces is [0.1, 200]. The camera's field of view (FOV) is determined by the sensors installed on the backpack-type LiDAR device.

[0078] Figure 5 This is a schematic diagram of a viewing cone provided in Embodiment 1 of the present invention. Figure 5 As shown, point E is the origin of the view frustum determined based on the position Twc_i and pose Rwc_i of the pinhole image data. The vertical distance between section 1 and section 2 is the distance between the near and far clipping planes. After the view frustum is established, the first remaining point cloud data is transformed to the camera coordinate system, and a second filtering is performed on the first remaining point cloud data to remove points outside the view frustum range. When there is a large amount of pinhole image data obtained from the segmentation of a single frame of panoramic image data, this step can effectively filter most of the point clouds, greatly saving computational resources.

[0079] Step B3: Filter the second remaining point cloud data three times to obtain the laser point cloud sub-data of the pinhole map data.

[0080] In step B3, the three-stage filtering, also known as voxel filtering, can be implemented as follows: an octree grid is built on the second remaining point cloud data. The octree grid includes multiple subgrids, each of which includes a portion of the point cloud data of the second remaining point cloud. The target point cloud of each subgrid is determined from the portion of the second remaining point cloud data included in each subgrid. The target point cloud is the point cloud closest to the center of the subgrid. The target point clouds of all subgrids together constitute the laser point cloud subdata of the pinhole map data.

[0081] The minimum scale of the octree mesh (i.e., the size of each sub-mesh) can be set according to the actual situation. Since the point clouds within the same sub-mesh are very close together, the three-stage filtering mainly uses the octree mesh to remove redundant point clouds in the same sub-mesh, further reducing the number of point clouds in the laser point cloud sub-data and improving computational efficiency.

[0082] S130. Generate a depth map corresponding to each pinhole image based on the laser point cloud sub-data of each pinhole image data.

[0083] Specifically, for any pinhole map data, the method for generating a depth map corresponding to the pinhole map data based on the laser point cloud sub-data of the pinhole map data can include the following two steps.

[0084] Step C1: Create a blank image. The size of the blank image is the same as the size of the pinhole image data.

[0085] Step C2: Project the laser point cloud sub-data of the pinhole map data onto the blank map to generate the depth map corresponding to the pinhole map data.

[0086] All point clouds included in the laser point cloud sub-data of the pinhole camera data must be projected. If the projected coordinates of a point cloud are within the range of the blank map, the location is marked on the blank map, and the depth information of that point cloud (i.e., the distance from the laser point cloud sub-data to the center of the image coordinates) is recorded; if the projected coordinates of a point cloud are not within the range of the blank map, that point cloud is discarded. This process continues until all point clouds included in the laser point cloud sub-data of the pinhole camera data have been projected, generating the depth map corresponding to the pinhole camera data.

[0087] In one embodiment, the depth information of the point cloud is Depth = Norm(pt3D - Tc_i), where pt3D represents the 3D coordinates of the point cloud, Tc_i represents the position of pinhole image data i, and Norm represents the modulus (i.e., calculating the point distance). The UV value is PROJECT_MATRIX * pt3D, where PROJECT_MATRIX represents the projection matrix (transforming from world coordinates to the pixel coordinates of pinhole image data i).

[0088] This invention provides a method for generating depth maps, comprising: acquiring panoramic image data and laser point cloud data of a target area, and segmenting the panoramic image data into multiple pinhole image data; acquiring laser point cloud sub-data for each pinhole image data based on the laser point cloud data; and generating a depth map corresponding to each pinhole image data based on the laser point cloud sub-data for each pinhole image data. By acquiring panoramic image data and laser point cloud data of a target area, segmenting the panoramic image data into multiple pinhole image data, acquiring laser point cloud sub-data for each pinhole image data based on the laser point cloud data, and then generating a depth map corresponding to each pinhole image data based on the laser point cloud sub-data for each pinhole image data, this method overcomes the limitations of traditional depth map acquisition via depth cameras in terms of measurement range, enabling the generation of depth maps for large scenes. Furthermore, segmenting the panoramic image data into multiple pinhole image data and processing them separately saves computational resources and improves the speed of depth map generation.

[0089] Example 2

[0090] Figure 6 This is a schematic diagram of a depth map generation device provided in Embodiment 2 of the present invention. Figure 6 As shown, the device includes: a data acquisition module 200, a data processing module 210, and a depth map generation module 220.

[0091] Data acquisition module 200 is used to acquire panoramic image data and laser point cloud data of the target area;

[0092] Data processing module 210 is used to segment panoramic image data into multiple pinhole image data; and to obtain laser point cloud sub-data for each pinhole image data based on laser point cloud data;

[0093] The depth map generation module 220 is used to generate a depth map corresponding to each pinhole image data based on the laser point cloud sub-data of each pinhole image data.

[0094] Optionally, the data processing module 210 is specifically used to obtain segmentation parameters, including the width, height, and focal length of the pinhole image data; determine the rotation angle based on the width, height, and focal length of the pinhole image data; and, with the Z-axis of the panoramic sphere as the rotation axis, segment one pinhole image data from the panoramic image data for each rotation angle, obtaining multiple pinhole image data after one rotation.

[0095] Optionally, the data processing module 210 is specifically used to perform a first filtering on the laser point cloud data to obtain the first remaining point cloud data; to perform a second filtering on the first remaining point cloud data to obtain the second remaining point cloud data; and to perform a third filtering on the second remaining point cloud data to obtain the laser point cloud sub-data of the pinhole map data.

[0096] Optionally, the data processing module 210 is specifically used to determine the timestamp of the panoramic image data to which the pinhole image data belongs; based on the timestamp, the laser point cloud data is filtered once to obtain the first remaining point cloud data, wherein the shooting time of the point cloud included in the first remaining point cloud data is within the timestamp.

[0097] Optionally, the data processing module 210 is specifically used to calculate the position and orientation of the pinhole image data; determine the origin and orientation of the view frustum based on the position and orientation of the pinhole image data, and generate the view frustum based on the origin, orientation, distance between near and far clipping planes and the camera field of view (FOV); and perform secondary filtering on the first remaining point cloud data based on the view frustum to obtain the second remaining point cloud data, wherein all point clouds included in the second remaining point cloud data are located within the view frustum.

[0098] Optionally, the data processing module 210 is specifically used to establish an octree grid on the second remaining point cloud data. The octree grid includes multiple sub-grids, and each sub-grid includes a portion of the point cloud of the second remaining point cloud data. The target point cloud of each sub-grid is determined from the portion of the second remaining point cloud data included in each sub-grid. The target point cloud is the point cloud closest to the center of the sub-grid. The target point clouds of all sub-grids together constitute the laser point cloud sub-data of the pinhole map data.

[0099] Optionally, the depth map generation module 220 is specifically used to create a blank map, the size of which is the same as the size of the pinhole map data; the laser point cloud sub-data of the pinhole map data is projected onto the blank map to generate the depth map corresponding to the pinhole map data.

[0100] The depth map generation apparatus provided in this embodiment of the invention can execute the depth map generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0101] Example 3

[0102] Figure 7A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0103] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0104] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0105] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as methods for generating depth maps.

[0106] In some embodiments, the depth map generation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the depth map generation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the depth map generation method by any other suitable means (e.g., by means of firmware).

[0107] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0109] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0111] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0112] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0113] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0114] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating a depth map, characterized in that, include: Acquire panoramic image data and laser point cloud data of the target area, and segment the panoramic image data into multiple pinhole image data; Based on the laser point cloud data, obtain the laser point cloud sub-data for each of the pinhole map data; Based on the laser point cloud sub-data of each pinhole image data, a depth map corresponding to each pinhole image data is generated; The step of segmenting the panoramic image data into multiple pinhole camera data includes: Obtain segmentation parameters, which include the width, height, and focal length of the pinhole image data; The rotation angle is determined based on the width, height, and focal length of the pinhole image data; Using the Z-axis of the panoramic sphere as the rotation axis, each rotation angle extracts one pinhole image from the panoramic image data, and multiple pinhole image data are obtained after one rotation.

2. The method according to claim 1, characterized in that, For any of the pinhole image data, obtaining the laser point cloud sub-data of the pinhole image data based on the laser point cloud data includes: The laser point cloud data is filtered once to obtain the first remaining point cloud data; The first remaining point cloud data is filtered a second time to obtain the second remaining point cloud data; The second remaining point cloud data is filtered three times to obtain the laser point cloud sub-data of the pinhole image data.

3. The method according to claim 2, characterized in that, The step of filtering the laser point cloud data to obtain the first remaining point cloud data includes: Determine the timestamp of the panoramic image data to which the pinhole image data belongs; Based on the timestamp, the laser point cloud data is filtered once to obtain the first remaining point cloud data, wherein the shooting time of the point cloud included in the first remaining point cloud data is within the timestamp.

4. The method according to claim 2, characterized in that, The step of performing secondary filtering on the first remaining point cloud data to obtain the second remaining point cloud data includes: Calculate the position and orientation of the pinhole image data; Based on the position and orientation of the pinhole image data, the origin and orientation of the view frustum are determined, and the view frustum is generated based on the origin, the orientation, the distance between the near and far clipping planes, and the camera field of view (FOV). Based on the view frustum, the first remaining point cloud data is filtered a second time to obtain the second remaining point cloud data, wherein all point clouds included in the second remaining point cloud data are located within the view frustum.

5. The method according to claim 2, characterized in that, The step of filtering the second remaining point cloud data three times to obtain the laser point cloud sub-data of the pinhole image data includes: An octree mesh is built on the second remaining point cloud data. The octree mesh includes multiple sub-mesh, and each sub-mesh includes a portion of the point cloud of the second remaining point cloud data. The target point cloud of each subgrid is determined from the partial point cloud of the second remaining point cloud data included in each subgrid. The target point cloud is the point cloud closest to the center of the subgrid. The target point clouds of all subgrids together constitute the laser point cloud subdata of the pinhole map data.

6. The method according to claim 1, characterized in that, For any of the pinhole map data, generating a depth map corresponding to the pinhole map data based on the laser point cloud sub-data of the pinhole map data includes: Create a blank image, the size of which is the same as the size of the pinhole image data; The laser point cloud sub-data of the pinhole map data is projected onto the blank map to generate a depth map corresponding to the pinhole map data.

7. A depth map generation apparatus, characterized in that, include: Data acquisition module, data processing module, and depth map generation module; The data acquisition module is used to acquire panoramic image data and laser point cloud data of the target area; The data processing module is used to segment the panoramic image data into multiple pinhole image data; and to obtain laser point cloud sub-data for each pinhole image data based on the laser point cloud data; The depth map generation module is used to generate a depth map corresponding to each pinhole image data based on the laser point cloud sub-data of each pinhole image data. The data processing module is specifically used to obtain segmentation parameters, including the width, height, and focal length of the pinhole image data; determine the rotation angle based on the width, height, and focal length of the pinhole image data; and, with the Z-axis of the panoramic sphere as the rotation axis, segment one pinhole image data from the panoramic image data for each rotation angle, obtaining multiple pinhole image data after one rotation.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method for generating a depth map according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for generating a depth map according to any one of claims 1-6.

Images