Method for detecting 3D objects and restoring 6DOF pose from panoramic video

Abstract

This application proposes a method for detecting three-dimensional objects from panoramic videos and restoring their six-degree-of-freedom poses, comprising: acquiring a panoramic video; extracting key frames from the panoramic video to obtain an equirectangular projection; and utilizing a pre-trained detector network model and the equirectangular projection to obtain the six-degree-of-freedom poses of the objects contained in the key frames. According to the technical solution of this application, a panoramic RGB video is used to detect objects therein and calculate the positions and angles of all objects in a unified three-dimensional space. This algorithm proposes an algorithm for calculating the poses of objects in panoramic videos, namely, the six-degree-of-freedom poses of objects.

Description

Technical Field

[0001] The present application relates to the field of smart retail, and more specifically, to a method and device for detecting three-dimensional objects from panoramic videos and restoring their six-degree-of-freedom posture. Background Art

[0002] Currently, object detection and pose calculation are of significant importance for scene spatial understanding, with applications spanning autonomous driving and navigation, robotic grasping and manipulation, augmented reality, and other fields. In smart retail, in particular, the detection and pose calculation of shelves (cases) and other in-store objects are the cornerstones of automated operations such as display layout and internal control inspections.

[0003] Existing technologies fuse image pixels and depth information, using neural network training to detect and estimate the pose of objects in a single image. Research has also used 3D point clouds or pixel information from a single image to achieve similar goals. Existing algorithms rely on sensor depth information, such as point clouds or RGBD data. The few algorithms that estimate the pose of 3D objects based on RGB pixels are applied to non-panoramic images, are not suitable for panoramas, and operate on single images. Currently, calculating the pose of objects in panoramic videos remains a mystery.

[0004] In the background technology section, the above information disclosed is only used to enhance the understanding of the background of the application and therefore it may contain information that does not constitute prior art information already known to a person of ordinary skill in the art. Summary of the Invention

[0005] The present application aims to provide a method and apparatus for detecting three-dimensional objects in panoramic videos and restoring their six-degree-of-freedom posture, which can calculate the posture of objects using panoramic videos.

[0006] According to one aspect of the present application, a method for detecting three-dimensional objects from a panoramic video and restoring their six-degree-of-freedom pose is proposed, comprising: obtaining a panoramic video; extracting key frames of the panoramic video to obtain an isometric rectangular projection; and obtaining the six-degree-of-freedom pose of the object contained in the key frames using a pre-trained detector network model and the isometric rectangular projection.

[0007] According to some embodiments, acquiring the panoramic video includes acquiring the panoramic video from a mobile panoramic camera, where the mobile panoramic camera includes a 360 camera or a dual fisheye panoramic camera.

[0008] According to some embodiments, extracting the key frames of the panoramic video includes extracting the key frames based on a capture area size of a capture space of the panoramic video.

[0009] According to some embodiments, the aforementioned method also includes pre-training the detector network model in the following manner: obtaining a panoramic video for training; extracting key frames of the panoramic video for training to obtain a training isometric rectangular projection image; using the training isometric rectangular projection image to reconstruct a point cloud top view; labeling objects on the training isometric rectangular projection image in combination with the point cloud top view; obtaining three-dimensional spatial information of the object; using the isometric rectangular projection image and the three-dimensional spatial information as training data to train the detection network and obtain the detector network model.

[0010] According to some embodiments, reconstructing a point cloud top view using the training equirectangular projection image includes reconstructing a collection space point cloud of the panoramic video by extracting key frames of the training panoramic video with the help of a SfM algorithm.

[0011] According to some embodiments, the three-dimensional space information includes three-dimensional bounding box information of the object, and the three-dimensional bounding box information includes a center point of the three-dimensional bounding box and an angle between a front normal vector of the three-dimensional bounding box and a camera visual axis.

[0012] According to some embodiments, the labeled object includes the plane position, width, and type of the labeled object.

[0013] According to some embodiments, acquiring the three-dimensional spatial information of the object includes obtaining the three-dimensional spatial information in a camera coordinate system according to the spatial position and size information of the object.

[0014] According to some embodiments, the detector network comprises CenterNet.

[0015] According to some embodiments, obtaining the six-degree-of-freedom pose of the object contained in the key frame includes: obtaining three-dimensional spatial information of the object using the detector network model; obtaining a camera pose corresponding to the key frame; and obtaining the six-degree-of-freedom pose of the object contained in the key frame based on the three-dimensional spatial information and the camera pose.

[0016] According to some embodiments, obtaining the camera pose corresponding to the key frame includes extracting the key frames of the panoramic video using a SfM algorithm and calculating the camera pose corresponding to each key frame.

[0017] According to some embodiments, the aforementioned method further comprises fusing the six-degree-of-freedom poses of the same object contained in multiple key frames to obtain a unified six-degree-of-freedom pose of the object.

[0018] According to some embodiments, the fusing of the six-degree-of-freedom poses of the same object contained in multiple key frames includes using camera parameters to project multiple groups of three-dimensional bounding boxes corresponding to the six-degree-of-freedom poses of the same object into the equirectangular projection image to obtain multiple groups of projection frames of the object; using the NMS algorithm on the multiple groups of projection frames to eliminate redundant projection frames of the multiple groups of projection frames to obtain optimal projection frames; and using the optimal projection frames to obtain a unified six-degree-of-freedom pose of the object.

[0019] According to some embodiments, the camera parameters include camera intrinsic parameters and camera extrinsic parameters; the camera intrinsic parameters are parameters unique to the camera itself and are related to the lens, and the camera extrinsic parameters are the camera pose of the key frame.

[0020] According to some embodiments, the aforementioned method further includes outputting a three-dimensional bounding box of the object contained in the panoramic video on a display interface.

[0021] According to one aspect of the present application, a device is proposed for detecting three-dimensional objects from a panoramic video and restoring a six-degree-of-freedom posture, comprising: a panoramic video acquisition module for acquiring a panoramic video from a panoramic camera; a key frame extraction module for extracting key frames of the panoramic video to obtain an equirectangular projection image; and a detector network module for using the equirectangular projection image to obtain the six-degree-of-freedom posture of the object contained in the key frame.

[0022] According to one aspect of the present application, an electronic device is proposed, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0023] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0024] According to some embodiments of the present application, panoramic RGB video is used to detect objects therein, and the positions and angles of all objects in a unified three-dimensional space are calculated. An algorithm for calculating the object pose, that is, the six-degree-of-freedom pose, of a panoramic video is proposed.

[0025] It should be understood that the foregoing general description and the following detailed description are merely illustrative and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects, features and advantages of the present application will become more apparent by describing exemplary embodiments thereof with reference to the accompanying drawings.

[0027] Figure 1A schematic diagram of a method framework for detecting three-dimensional objects and restoring six-degree-of-freedom postures from panoramic videos according to an example embodiment of the present application is shown.

[0028] Figure 2 A flowchart of a method for detecting three-dimensional objects from a panoramic video and restoring six-degree-of-freedom posture according to an example embodiment of the present application is shown.

[0029] Figure 3 A flowchart of another method for detecting three-dimensional objects from panoramic videos and restoring six-degree-of-freedom postures according to an exemplary embodiment of the present application is shown.

[0030] Figure 4 Flowchart of training a detector network model according to an exemplary embodiment of the present application.

[0031] Figure 5 A diagram shows a device for detecting three-dimensional objects from a panoramic video and restoring six-degree-of-freedom posture according to an example embodiment of the present application.

[0032] Figure 6 An isometric rectangular projection diagram of a key frame extracted according to an exemplary embodiment of the present application is shown.

[0033] Figure 7 An object annotation diagram according to an exemplary embodiment of the present application is shown.

[0034] Figure 8 A three-dimensional bounding box diagram of an object according to an exemplary embodiment of the present application is shown.

[0035] Figure 9 The reconstructed point cloud (white points) and the restored camera pose (black line) according to an exemplary embodiment of the present application are shown.

[0036] Figure 10 A top view of multiple sets of overlapping poses of an object according to an exemplary embodiment of the present application is shown.

[0037] Figure 11 A top view of the six-degree-of-freedom posture of an object according to an exemplary embodiment of the present application is shown.

[0038] Figure 12 A six-degree-of-freedom pose graph of an object according to an exemplary embodiment of the present application is shown.

[0039] Figure 13 A block diagram of an electronic device according to an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0041] The described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other modes, components, materials, devices or operations may be employed. In these cases, well-known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.

[0042] The block diagrams shown in the accompanying drawings do not necessarily correspond to physically independent entities. These functional entities or parts of these functional entities can be implemented in software, or in one or more hardware modules and / or programmable modules, or in different networks and / or processor devices and / or microcontrollers.

[0043] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0044] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0045] Object detection and pose calculation are crucial for understanding scene space. This is particularly true in smart retail. Detecting and calculating the pose of shelves (cases) and other in-store objects is the cornerstone of automated processes like display layout and internal control inspections. Object detection uses bounding boxes to identify the location of objects in an image and assigns them a category. Pose calculation involves calculating the position and angle of an object in three-dimensional space, also known as its six-degree-of-freedom pose.

[0046] This application is based on panoramic RGB video and uses deep learning to train a neural network model to detect objects in key frames and calculate the position and angle of all objects in the same three-dimensional space, that is, the six-degree-of-freedom position of the object.

[0047] The specific implementation according to the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0048] Figure 1 A schematic diagram of a framework of a method for detecting three-dimensional objects from a panoramic video and restoring six-degree-of-freedom posture according to an exemplary embodiment of the present application is shown.

[0049] According to the inventive concept of this application, it is necessary to first use the training data to train a convolutional neural network detector to obtain a detector network. After obtaining the detector network, the panoramic video of the acquisition space is framed to obtain equirectangular projections, which serve as inputs to the detector network and the SfM algorithm respectively. Using the three-dimensional spatial information of the object output by the detector network and the camera pose of each equirectangular projection output by the SfM algorithm, coordinate transformation is performed to output a non-unified object pose with redundant data. Finally, with the help of reprojection NMS, the redundant data is removed to obtain the unified object pose of the object in the acquisition space of the panoramic video.

[0050] According to the inventive concept of the present application, the detector network is obtained by training a convolutional neural network detector with training data. According to some embodiments of the present application, the training data can be obtained by:

[0051] First, a panoramic video for training is obtained. Frames are extracted from the panoramic video based on the size of the video's acquisition space, resulting in equirectangular projections of the keyframes. Using the SfM algorithm, a sequence of 2D images composed of the equirectangular projections of the extracted keyframes is used to estimate the camera pose corresponding to each keyframe, estimate the 3D structure of the panoramic video's acquisition space, and reconstruct a point cloud of the acquisition space. Using the reconstructed point cloud of the acquisition space, a top-down view of the reconstructed point cloud is obtained. For example, a virtual camera is added to the 3D space and adjusted based on the ground-to-ceiling orientation of the point cloud so that the point cloud fits perfectly within the camera's imaging plane. This is then rendered into an image. On the equirectangular projections of the keyframes extracted from each panoramic video, combined with the top-down view of the reconstructed point cloud, the object's planar position, width, and type are manually annotated. Combining the object's dimensional information (including height, thickness, and height above the ground, in addition to width) with the object's planar position in the world coordinate system, a coordinate transformation is performed to obtain the object's complete 3D spatial information in the camera coordinate system. The manually annotated keyframe isometric rectangular projection images and complete three-dimensional spatial information are used as training data to train a convolutional neural network detector and obtain the detector network.

[0052] Figure 2A flowchart of a method for detecting three-dimensional objects from a panoramic video and restoring six-degree-of-freedom posture according to an example embodiment of the present application is shown.

[0053] See also Figure 2 In step S201, a panoramic video is acquired. To achieve efficient spatial scene scanning for object detection and pose restoration, a 360° or dual fisheye panoramic camera can be used to capture panoramic video within the space. According to some exemplary embodiments of the present application, the panoramic video can be stored locally or in the cloud.

[0054] In S203, the key frames of the panoramic video are extracted to obtain an equirectangular projection. According to some exemplary embodiments of the present application, when extracting the key frames of the panoramic video, the key frames can be extracted in a certain proportion according to the size of the acquisition area. For example, 600 frames are extracted for every 80 square meters. For example, Figure 6 An isometric rectangular projection diagram of a key frame extracted using an exemplary embodiment of the present application is shown.

[0055] In S205, a pre-trained detector network model and an isometric rectangular projection are obtained. The training steps of the detector network model can be found in Figure 4 According to some embodiments of the present application, the method further includes using the SfM algorithm to calculate the camera pose corresponding to each frame using the training equirectangular projection sequence with visual motion information obtained in S203 and reconstruct the acquisition space point cloud map to estimate the three-dimensional structure of the panoramic video acquisition space, such as Figure 9 As shown in the figure, the white points are the reconstructed acquisition space point cloud, and the black line is the restored camera pose. The camera pose can be expressed as follows:

[0056] where R and T are the camera rotation and displacement extrinsic parameters, respectively.

[0057] In S207 , the six-degree-of-freedom pose of the object contained in the key frame is obtained using the pre-trained detector network model and the isometric rectangular projection image.

[0058] According to some embodiments of the present application, the equirectangular projection image extracted in step S203 and the camera pose corresponding to each frame calculated by the SfM algorithm are used as input, and the six-degree-of-freedom pose of the object contained in each key frame is obtained through a pre-trained detector network model, such as Figure 11 and Figure 12 shown.

[0059] Figure 3 A flowchart of another method for detecting three-dimensional objects from panoramic videos and restoring six-degree-of-freedom postures according to an exemplary embodiment of the present application is shown.

[0060] The steps of obtaining panoramic video S201 and extracting frames S203 are as described above and will not be described again here. Figure 2 The difference is that in S205, the detector network outputs not the six-degree-of-freedom pose of the object, but the three-dimensional spatial information of the object in each camera space.

[0061] According to some example embodiments of the present application, the detector network outputs three-dimensional spatial information of the object in each camera space by predicting the translation and rotation of the object in the camera space.

[0062] According to some other example embodiments of the present application, the detector network predicts the two-dimensional bounding box position of the object on the camera imaging plane, the distance from the camera, and the angle between the object's front normal vector and the camera's visual axis, and obtains the translation and rotation of the object in the camera space through camera projection geometric constraints, and outputs the three-dimensional spatial information of the object in each camera space.

[0063] At S2071, coordinate conversion is performed.

[0064] According to some embodiments of the present application, the equirectangular projection of the key frame extracted in S203 is used to calculate the camera pose corresponding to each frame; the three-dimensional spatial information of the object obtained in S205 and the camera pose corresponding to each frame are used to obtain the ununiform object pose of the object through the conversion relationship between the camera coordinate system and the world coordinate system, and the ununiform object pose is obtained through the conversion relationship between the camera coordinate system and the world coordinate system, such as Figure 10 shown.

[0065] In S2073 , the unified six-degree-of-freedom pose of the object is obtained by using reprojection NMS.

[0066] According to some embodiments of the present application, the six-degree-of-freedom poses with redundant data of the same object contained in multiple key frames, that is, the ununified object poses obtained in S2071, are fused to obtain the final unified six-degree-of-freedom pose of the object.

[0067] According to some embodiments of the present application, the six-degree-of-freedom pose with redundant data is deduplicated using the reprojection NMS algorithm to obtain the final unified six-degree-of-freedom pose of the object. The specific steps are as follows:

[0068] Using the camera parameters, multiple sets of 3D bounding boxes corresponding to the 6-DOF poses of the same object are projected onto the equirectangular projection to obtain multiple sets of projection boxes of the object, such as Figure 8 The 3D bounding box here is the 3D spatial information output by the detector network model. The reprojection NMS algorithm is used on the multiple projection frames to eliminate the redundant projection frames of the multiple projection frames and obtain the optimal projection frame. Using the optimal projection frame, the unified 6-DOF pose of the object is obtained, as shown in Figure 11 and Figure 12 The so-called camera parameters include camera intrinsic parameters and camera extrinsic parameters, where camera intrinsic parameters are parameters unique to the camera itself and are related to the lens, and camera extrinsic parameters are the camera pose of the key frame.

[0069] According to some embodiments of the present application, the detector network model outputs a three-dimensional bounding box of an object contained in the panoramic video on a display interface.

[0070] According to an example embodiment of the present application, an algorithm for calculating the object pose, that is, the six-degree-of-freedom pose, of a panoramic video is proposed by detecting objects in the panoramic RGB video and calculating the positions and angles of all objects in a unified three-dimensional space.

[0071] Figure 4 A flowchart of training a detector network model according to an exemplary embodiment of the present application is shown.

[0072] See also Figure 4 , the detector network model can be pre-trained as follows.

[0073] At S401, a training panoramic video is acquired. The training panoramic video can be stored locally or in the cloud. According to some embodiments of the present application, a 360 or dual fisheye panoramic camera can be used to capture the training panoramic video.

[0074] At S403, key frames of the training panoramic video are extracted to obtain a training equirectangular projection. When extracting key frames from the panoramic video, key frames may be extracted at a certain ratio based on the size of the captured area. For example, 600 frames are extracted for every 80 square meters.

[0075] In S405, the point cloud top view is reconstructed using the training equirectangular projection. According to some embodiments of the present application, the camera pose corresponding to each frame can be calculated using the training equirectangular projection sequence with visual motion information obtained in S403 using the SfM algorithm to reconstruct the spatial point cloud, and the point cloud top view is obtained using the obtained spatial point cloud. The camera pose corresponding to each frame can be expressed as follows:

[0076] where R and T are the camera rotation and displacement extrinsic parameters, respectively.

[0077] In S407 , objects are marked on the equirectangular projection of the key frames extracted from each panoramic video in combination with the reconstructed point cloud top view.

[0078] In S409 , the three-dimensional spatial information of the object is calculated.

[0079] According to some embodiments of the present application, the three-dimensional spatial information of the object is calculated by combining the size information of the object, including the height, thickness and height from the ground in addition to the width, and the plane position of the object in the world coordinate system.

[0080] like Figure 7 As shown in Figure 2. 3D spatial information is the position and orientation of an object in the camera coordinate system. Because the geometric shape of an object is complex, 3D spatial information is usually simplified to 3D bounding box information, such as Figure 8 According to some embodiments of the present application, the three-dimensional space information is the center point of the three-dimensional bounding box in the camera coordinate system and the angle between the front normal vector of the three-dimensional bounding box and the camera visual axis.

[0081] In S411, a convolutional neural network detector is trained using the equirectangular projection of each key frame and the three-dimensional spatial information of the objects contained therein as training data to obtain a detector network model. According to some embodiments of the present application, the convolutional neural network detector may be CenterNet.

[0082] Figure 5 A diagram shows a device for detecting three-dimensional objects from a panoramic video and restoring six-degree-of-freedom posture according to an example embodiment of the present application.

[0083] like Figure 5 As shown, the device for detecting three-dimensional objects from panoramic videos and restoring six-degree-of-freedom postures includes a panoramic video acquisition module 301 , a key frame extraction module 303 and a detector network module 305 .

[0084] According to an exemplary embodiment of the present application, a device for detecting three-dimensional objects from a panoramic video and restoring six-degree-of-freedom posture is proposed. Figure 5 , the device comprises:

[0085] The panoramic video acquisition module 501 is used to acquire the panoramic video from the panoramic camera.

[0086] The key frame extraction module 503 is used to extract the key frames of the panoramic video to obtain an equirectangular projection image.

[0087] The detector network module 505 is used to obtain the six-degree-of-freedom pose of the object contained in the key frame using the isometric rectangular projection.

[0088] The aforementioned device performs functions similar to the method provided above. For other functions, please refer to the above description and will not be repeated here.

[0089] Figure 13 A block diagram of an electronic device according to an exemplary embodiment of the present application is shown.

[0090] Refer to the following Figure 13hereinafter, an electronic device 200 according to this embodiment of the present application is described. Figure 13 The electronic device 200 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0091] like Figure 13 As shown, electronic device 200 is implemented as a general-purpose computing device. Components of electronic device 200 may include, but are not limited to, at least one processing unit 210, at least one storage unit 220, a bus 230 connecting various system components (including storage unit 220 and processing unit 210), a display unit 240, and the like.

[0092] The storage unit stores program codes, which can be executed by the processing unit 210, so that the processing unit 210 executes the methods described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 210 can execute the following Figure 2 The method shown in .

[0093] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 2201 and / or a cache memory unit 2202 , and may further include a read-only memory unit (ROM) 2203 .

[0094] The storage unit 220 may also include a program / utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0095] Bus 230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0096] The electronic device 200 can also communicate with one or more external devices 300 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 200, and / or any device that enables the electronic device 200 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 250. Furthermore, the electronic device 200 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 260. The network adapter 260 can communicate with other modules of the electronic device 200 via the bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 200, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0097] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software or by combining software with necessary hardware. The technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present application.

[0098] The software product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0099] Computer-readable storage media may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0100] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0101] The computer-readable medium carries one or more programs. When the one or more programs are executed by the device, the computer-readable medium implements the aforementioned functions.

[0102] Those skilled in the art will appreciate that the modules described above can be distributed in the device according to the description of the embodiment, or can be modified accordingly to be used in one or more devices that are different from the embodiment. The modules of the above embodiment can be combined into one module or further divided into multiple submodules.

[0103] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0104] According to an example embodiment of the present application, an algorithm for calculating the object pose, that is, the six-degree-of-freedom pose, of a panoramic video is proposed by detecting objects in the panoramic RGB video and calculating the positions and angles of all objects in a unified three-dimensional space.

[0105] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. At the same time, changes or modifications made by those skilled in the art based on the ideas of the present application, the specific implementation methods, and the scope of application of the present application, all fall within the scope of protection of the present application. In summary, the contents of this specification should not be construed as limiting the present application.

Claims

1. A method for detecting three-dimensional objects from panoramic videos and restoring their six-degree-of-freedom poses, characterized in that: include: Get panoramic video; Extracting key frames of the panoramic video to obtain an equirectangular projection image; Obtaining a six-degree-of-freedom pose of the object contained in the keyframe using a pre-trained detector network model and the isometric rectangular projection; The obtaining of the six-degree-of-freedom pose of the object contained in the key frame comprises: obtaining three-dimensional spatial information of the object using the detector network model, obtaining a camera pose corresponding to the key frame, and obtaining the six-degree-of-freedom pose of the object contained in the key frame based on the three-dimensional spatial information and the camera pose; The training data of the detector network model includes an equirectangular projection of a key frame of the panoramic video and three-dimensional spatial information of the object.

2. The method according to claim 1, characterized in that The obtaining of the panoramic video includes: The panoramic video is acquired from a mobile panoramic camera, where the mobile panoramic camera includes a 360 camera or a dual fisheye panoramic camera.

3. The method according to claim 1, characterized in that Extracting key frames of the panoramic video includes: Key frames are extracted according to the acquisition area size of the acquisition space of the panoramic video.

4. The method according to claim 1, wherein It also includes pre-training the detector network model as follows: Obtain panoramic videos for training; Extracting key frames of the training panoramic video to obtain a training equirectangular projection image; Reconstructing a point cloud top view using the training equirectangular projection image; Marking objects on the training equirectangular projection map in combination with the point cloud top view; Acquiring three-dimensional spatial information of the object; The isometric rectangular projection image and the three-dimensional space information are used as training data to train a detection network to obtain the detector network model.

5. The method according to claim 4, characterized in that Reconstructing a point cloud top view using the training equirectangular projection image includes: With the help of the SfM algorithm, the key frames of the training panoramic video are extracted to reconstruct the acquisition space point cloud of the panoramic video.

6. The method according to claim 4, characterized in that: The three-dimensional space information includes three-dimensional bounding box information of the object, and the three-dimensional bounding box information includes a center point of the three-dimensional bounding box and an angle between a front normal vector of the three-dimensional bounding box and a camera visual axis.

7. The method according to claim 4, characterized in that: The labeled object includes the plane position, width and type of the labeled object.

8. The method according to claim 4, characterized in that Acquiring three-dimensional spatial information of the object, including: The three-dimensional space information in the camera coordinate system is obtained according to the spatial position and size information of the object.

9. The method according to claim 4, wherein: The detector network includes CenterNet.

10. The method according to claim 9, characterized in that Obtaining the camera pose corresponding to the keyframe includes: With the help of the SfM algorithm, key frames of the panoramic video are extracted and the camera pose corresponding to each key frame is calculated.

11. The method according to claim 9, characterized in that Also includes: The six-degree-of-freedom poses of the same object contained in multiple key frames are fused to obtain a unified six-degree-of-freedom pose of the object.

12. The method according to claim 11, characterized in that The fusing of the six-degree-of-freedom poses of the same object contained in multiple key frames includes: Using camera parameters, projecting multiple sets of three-dimensional bounding boxes corresponding to the six-degree-of-freedom poses of the same object into the equirectangular projection to obtain multiple sets of projection boxes of the object; Using the NMS algorithm on the multiple groups of projection frames, eliminating redundant projection frames of the multiple groups of projection frames, and obtaining the optimal projection frame; The optimal projection frame is used to obtain a unified six-degree-of-freedom pose of the object.

13. The method according to claim 12, wherein: The camera parameters include camera intrinsic parameters and camera extrinsic parameters; The camera intrinsic parameters are parameters unique to the camera itself and are related to the lens, and the camera extrinsic parameters are the camera pose of the key frame.

14. The method according to claim 1, wherein Also includes: The three-dimensional bounding box of the object contained in the panoramic video is output on the display interface.

15. A device for detecting three-dimensional objects from panoramic videos and restoring their six-degree-of-freedom poses, characterized in that: include: A panoramic video acquisition module is used to acquire panoramic videos from a panoramic camera; A key frame extraction module, used to extract key frames of the panoramic video to obtain an equirectangular projection image; a detector network module, configured to obtain a six-degree-of-freedom pose of an object contained in the key frame using the isometric rectangular projection, wherein obtaining the six-degree-of-freedom pose of the object contained in the key frame comprises: obtaining three-dimensional spatial information of the object using the detector network model, obtaining a camera pose corresponding to the key frame, and obtaining the six-degree-of-freedom pose of the object contained in the key frame based on the three-dimensional spatial information and the camera pose; The training data of the detector network model includes an equirectangular projection of a key frame of the panoramic video and three-dimensional spatial information of the object.

16. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 14.

17. A computer-readable medium having a computer program stored thereon, characterized in that: include: When the program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images