Method for estimating pose graph and transformation matrix between cameras by recognizing markers on the ground in panoramic images

Abstract

The present application provides a method, a computer program and a computer system for 3D pose graph reconstruction. 3D reconstruction of indoor buildings for VR / AR applications (e.g. virtual tourism, digital museum and virtual house selling) can be performed based on estimating pose graph and transformation matrix between cameras by identifying markers on the ground in a panoramic image. Image data corresponding to one or more views of a first camera is received. One or more markers corresponding to a second camera are identified in the received image data. Based on the identified one or more markers, a pose graph corresponding to the one or more views of the first camera is constructed, the pose graph comprising at least edges.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 185,945 (filed May 7, 2021) and U.S. Patent Application No. 17 / 497,025 (filed October 8, 2021) issued by the United States Patent and Trademark Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the field of data processing, and more specifically to image processing. Background Technology

[0004] 3D reconstruction of interior buildings is an active research topic and has been applied in various industries such as real estate, construction, building restoration, and entertainment. 3D reconstruction utilizes computer vision and machine learning techniques, taking a single RGB image or a set of images from different viewpoints as input to generate a 3D geometric representation of the building in the scene. The development of depth sensors has made it more convenient and accurate to measure depth information directly from the scene; currently, widely used depth cameras include LiDAR and structured light.

[0005] 3D geometric representations are often in the form of so-called point clouds, which contain a set of 3D points in space, each containing 3D positional information and additional attributes such as color and reflectivity. Another popular 3D format is a textured mesh. In addition to individual 3D points, it also contains connectivity information between adjacent points, forming a set of facets (such as triangles), and texture information can also be attached to each facet. To capture large scenes, multiple images are acquired from different viewpoints. In this case, pose maps are important. Pose maps define the connectivity and visibility between different viewpoints. Summary of the Invention

[0006] Implementations relate to a method, system, and computer-readable medium for constructing a camera pose map. According to one aspect, a method for constructing a pose map is provided. The method may include receiving image data corresponding to one or more views of a first camera. One or more markers corresponding to a second camera are identified in the received image data. Based on the identified one or more markers, a pose map corresponding to the one or more views of the first camera is constructed, the pose map including at least edges.

[0007] According to another aspect, a computer system for constructing a pose graph is provided. The computer system may include one or more computer-readable non-transitory storage media configured to store computer program code; and one or more computer processors configured to access and operate according to the instructions of the computer program code, the computer program code including: receiving code configured to cause the one or more computer processors to receive image data of one or more views corresponding to a first camera; identification code configured to cause the one or more computer processors to identify one or more markers within the received image data corresponding to a second camera; and construction code configured to cause the one or more computer processors to construct a pose graph of the one or more views corresponding to the first camera based on the identified one or more markers, the pose graph including at least edges.

[0008] According to yet another aspect, a non-transitory computer-readable medium is provided, storing a computer program for constructing a pose diagram of a camera, the computer program being configured to cause one or more computer processors to implement the method described above for constructing a camera pose diagram. Attached Figure Description

[0009] These and other objects, features, and advantages will become apparent from the following detailed description of illustrative embodiments, which is taken in conjunction with the accompanying drawings. The various features in the drawings are not to scale, as they are illustrated for clarity of understanding by those skilled in the art in conjunction with the detailed description. In the drawings:

[0010] Figure 1 A networked computer environment according to at least one embodiment is illustrated;

[0011] Figure 2 It is a set of markers for pose diagram reconstruction according to at least one embodiment;

[0012] Figure 3 This is an operation flowchart illustrating the steps performed by a program for pose map reconstruction according to at least one embodiment;

[0013] Figure 4 It is based on at least one embodiment. Figure 1 A block diagram depicting the internal and external components of the server and computer;

[0014] Figure 5 It includes, according to at least one embodiment. Figure 1 A block diagram illustrating a cloud computing environment for a computer system; and

[0015] Figure 6 It is based on at least one embodiment. Figure 5 A block diagram illustrating the functional layers of an illustrative cloud computing environment. Detailed Implementation

[0016] This document discloses detailed embodiments of the claimed structures and methods; however, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which can be implemented in various forms. These structures and methods may be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this disclosure comprehensive and complete, and to fully convey the scope to those skilled in the art. Details of well-known features and techniques may be omitted in the description to avoid unnecessarily obscuring the presented embodiments.

[0017] The implementation generally relates to the field of data processing, and more specifically to image processing. The exemplary embodiments described below provide a system, method, and computer program for pose map reconstruction, etc. Thus, in some embodiments, the ability to estimate pose maps and transformation matrices between cameras by recognizing markers on the ground in a panoramic image allows for 3D reconstruction of interior buildings for VR / AR applications such as virtual tourism, digital museums, and virtual home sales, thereby enhancing the computing field.

[0018] As mentioned earlier, 3D geometric representation is often in the form of a point cloud, which contains a set of 3D points in space, each containing 3D positional information and additional attributes such as color and reflectivity. Another popular 3D format is a textured mesh. In addition to individual 3D points, it also contains connectivity information between adjacent points, forming a set of facets (such as triangles), and texture information can also be attached to each facet. To capture large scenes, multiple images are acquired from different viewpoints. In this case, a pose graph is important. The pose graph defines the connectivity and visibility between different viewpoints.

[0019] Existing approaches to this problem attempt to find correspondences and estimate similarities from image pairs, and then apply multi-view geometric techniques to estimate the transformation matrix. However, such algorithms can fail in many scenarios. For example, if the scene lacks features or has many repeating patterns, it is difficult to find the correct image correspondences among numerous images. Therefore, estimating pose maps and transformation matrices from multi-view panoramic images may be advantageous.

[0020] This document describes aspects with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0021] The exemplary embodiments described below provide a system, method, and computer program for estimating pose maps and transformation matrices based on images captured from multiple viewpoints. Reference is now made to... Figure 1 The functional block diagram of the networked computer environment illustrates a pose map reconstruction system 100 (hereinafter referred to as the "System"), which is used for 3D reconstruction of indoor buildings for VR / AR applications based on pose maps and transformation matrices between cameras estimated by recognizing markers on the ground in panoramic images. It should be understood that... Figure 1 This illustration only provides one implementation method and does not imply any limitation regarding the environment in which different implementation methods can be implemented. Many modifications can be made to the described environment based on design and implementation requirements.

[0022] System 100 may include computer 102 and server computer 114. Computer 102 may communicate with server computer 114 via communication network 110 (hereinafter referred to as the "network"). Computer 102 may include processor 104 and software program 108, which is stored on data storage device 106 and is capable of interfacing with a user and communicating with server computer 114. Reference will be made below. Figure 4 The computer 102 discussed may include internal component 800A and external component 900A, and the server computer 114 may include internal component 800B and external component 900B. For example, the computer 102 may be a mobile device, telephone, personal digital assistant, netbook, laptop computer, tablet computer, desktop computer, or any type of computing device capable of running programs, accessing networks, and accessing databases.

[0023] Server computer 114 can also operate in cloud computing service models, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS), as described below. Figure 5 and 6 The server computer 114, as discussed, can also be located in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud.

[0024] Server computer 114, which can be used to perform 3D reconstruction of indoor buildings for VR / AR applications based on pose maps and transformation matrices estimated between cameras by recognizing markers on the ground in panoramic images, is capable of running a pose map reconstruction program 116 (hereinafter referred to as the "program") that can interact with database 112. Reference is made below. Figure 3Further details explain the pose graph reconstruction procedure method. In one embodiment, computer 102 may operate as an input device including a user interface, while pose graph reconstruction program 116 may run primarily on server computer 114. In an alternative embodiment, pose graph reconstruction program 116 may run primarily on one or more computers 102, while server computer 114 may be used to process and store the data used by the pose graph reconstruction program 116 described above. It should be noted that pose graph reconstruction program 116 may be a standalone program or may be integrated into a larger pose graph reconstruction program.

[0025] However, it should be noted that in some cases, the processing of the pose graph reconstruction program 116 can be shared between computer 102 and server computer 114 in any proportion. In another embodiment, the pose graph reconstruction program 116 can operate on more than one computer, server computer, or some combination of computers and server computers; for example, multiple computers 102 communicate with a single server computer 114 via network 110. In another embodiment, for example, the pose graph reconstruction program 116 can operate on multiple server computers 114 by communicating across network 110 with multiple client computers. Alternatively, the pose graph reconstruction program 116 can operate on a network server, communicating with the server and multiple client computers via the network.

[0026] Network 110 may include wired connections, wireless connections, fiber optic connections, or combinations thereof. Typically, network 110 can be any combination of connections and protocols that support communication between computer 102 and server computer 114. For example, network 110 may include various types of networks, such as local area networks (LANs), wide area networks (WANs) such as the Internet, telecommunications networks such as the Public Switched Telephone Network (PSTN), wireless networks, public switched networks, satellite networks, cellular networks (e.g., fifth-generation (5G), Long Term Evolution (LTE), third-generation (3G), Code Division Multiple Access (CDMA), etc.), public land mobile networks (PLMNs), metropolitan area networks (MANs), private networks, self-organizing networks, intranets, fiber-optic networks, etc., and / or combinations of these or other types of networks.

[0027] Figure 1 The number and arrangement of devices and networks shown are provided as examples. In reality, with... Figure 1 Compared to the devices and / or networks shown, there may be additional devices and / or networks, fewer devices and / or networks, different devices and networks, or devices and / or networks with different arrangements. Furthermore, Figure 1 The two or more devices shown can be implemented within a single device, or Figure 1The single device shown can be implemented as multiple distributed devices. Additionally or alternatively, a group of devices in system 100 (e.g., one or more devices) can perform one or more functions, which are described as being performed by another group of devices in system 100.

[0028] Now for reference Figure 2 A set of markers 200 is depicted. In one or more embodiments, the pose map reconstruction system 100 ( Figure 1 These markers can be easily found in the image and can be distinguished by a unique ID. Each marker can be associated with an ID, through which the computer can automatically identify the ID number and connect it to the corresponding viewpoint in the pose diagram.

[0029] For example, in viewpoint "03", the pose map reconstruction system 100 ( Figure 1 The pose map reconstruction system 100 can automatically detect two visible adjacent viewpoints "02" and "04" using markers 200 on the floor. The pose map reconstruction system 100 can then construct a pose map with three viewpoints, where "03" has two edges connected to "02" and "04" respectively. The pose map reconstruction system 100 can construct a pose map by detecting all images of the scene. It is understood that if the computer cannot distinguish markers with different viewpoint IDs, this information can be collected (or corrected) through user input. In this case, the user may need to tell the computer which marker corresponds to which viewpoint.

[0030] The transformation matrix between connected views can be Transformation matrix The transformation matrix contains Rotation matrix and Translation vector This rotation matrix and translation vector can be associated with each pair of connected views. For example, the transformation matrix... (Depend on and The composition describes how the camera works from the viewpoint. Move to another viewpoint Viewpoint in 3D world coordinates The camera position is represented as ,Then .

[0031] For viewpoint and If they can see each other in the captured images, and if the camera height at each viewpoint is known, then the viewpoints can be calculated as follows. To viewpoint The transformation matrix, i.e., .

[0032] Assume that the camera heights at viewpoints i and j are Ci and Cj, respectively. Without loss of generality, assume that the panoramic images from different viewpoints have the same resolution W×H, where W=2H. Assume that a marker for viewpoint j is detected at position (uj,vj) in the panoramic image of viewpoint i, and a marker for viewpoint i is detected at position (ui,vi) in the panoramic image of viewpoint j.

[0033] Without loss of generality, assume the camera center is the origin of the world coordinate system in the Cartesian coordinate system, i.e., the camera coordinates are (0,0,0). Assume the camera is oriented towards the positive x-axis and the negative z-axis towards the floor plane, and the floor / ceiling plane is parallel to the xy plane. The horizontal vanishing line of the ground plane is located at the midpoint of the panoramic image.

[0034] Therefore, it can be viewed from the point of view. Calculate viewpoint in 3D coordinate system The relative positions are as follows:

[0035]

[0036] from arrive The distance is (4)

[0037] Similarly, it can be done from the viewpoint Calculate viewpoint in 3D coordinate system The relative positions are as follows:

[0038]

[0039] from arrive The distance is (8)

[0040] if If the value exceeds the threshold, it indicates an error. The computer may discard the erroneous image or display an error message.

[0041] To reduce systematic errors, in one implementation, the distance between the two viewpoints is determined by... and The average estimate, i.e. (9)

[0042] In order to change the coordinate system from the viewpoint Switch to viewpoint It can be derived from the translation vector. Translation coordinate system. In one implementation, by... Calculate the translation. In another embodiment, by... Calculate translation, without performing Adjustment. The coordinate system may be rotated by a certain yaw angle, which can be calculated as follows:

[0043]

[0044] Here, arctan2() is a function defined as follows:

[0045]

[0046] The corresponding rotation matrix is ​​as follows:

[0047]

[0048] Therefore, from the viewpoint To viewpoint The overall transformation equation can be written as follows:

[0049] (13)

[0050] in, and .

[0051] By identifying markers under each camera position on the ground, a pose map of the cameras can be constructed. For example, if the marker for camera B is visible in an image captured by camera A, an edge can be constructed between A and B in the pose map (assuming the pose map is undirected). If the marker for camera A is visible in an image captured by camera B, an edge can be constructed between A and B in the pose map (assuming the pose map is undirected). If the marker for camera B is visible in both an image captured by camera A and an image captured by camera B, an edge can be constructed between A and B in the pose map (assuming the pose map is undirected). By identifying markers under each camera position on the ground, this method can estimate the visibility between camera positions. If the marker for camera B is visible in an image captured by camera A, it can be estimated that camera B can see camera A, or that camera A and camera B can see each other. If the marker for camera A is visible in an image captured by camera B, it can be estimated that camera A can see camera B, or that camera A and camera B can see each other. If the marker for camera B is visible in an image captured by camera A, and the marker for camera A is visible in an image captured by camera B, then it can be estimated that cameras A and B can see each other. This method estimates the transformation matrix between cameras by identifying the markers below each camera position on the ground. The camera height (vertical distance from the floor) can be input from the user, estimated by the algorithm, or set by a default value. If the marker for camera B is visible in an image captured by camera A, and the marker for camera A is visible in an image captured by camera B, then the transformation matrix between A and B can be estimated.

[0052] Now for reference Figure 3 The flowchart illustrates the steps of a method 300 for 3D reconstruction of indoor buildings for VR / AR applications, based on estimating pose maps and transformation matrices between cameras by identifying markers on the ground in panoramic images.

[0053] In 302, method 300 may include receiving image data corresponding to one or more views.

[0054] In 304, method 300 may include identifying one or more markers corresponding to the second camera within the received image data.

[0055] In 306, method 300 may include constructing a pose map corresponding to one or more views of the first camera based on one or more identified markers.

[0056] Understandable. Figure 3 This is merely an illustration of one implementation method and does not imply any limitations on how different implementation methods may be carried out. Many modifications can be made to the described environment based on design and implementation requirements.

[0057] Figure 4 According to the illustrative implementation method Figure 1 Block diagram 400 depicts the internal and external components of a computer. It should be understood that... Figure 4 This illustration is provided for illustrative purposes only and does not imply any limitation regarding the environment in which different implementations may be carried out. Many modifications may be made to the described environment based on design and implementation requirements.

[0058] Computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) can include Figure 4 The diagram shows a corresponding set of internal components 800A, 800B and external components 900A, 900B. Each set of internal components 400 includes one or more processors 820, one or more computer-readable RAMs 822 and one or more computer-readable ROMs 824 on one or more buses 826, one or more operating systems 828, and one or more computer-readable tangible storage devices 830.

[0059] Processor 820 is implemented in hardware, firmware, or a combination of hardware and software. Processor 820 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or another type of processing component. In some embodiments, processor 820 includes one or more processors that can be programmed to perform functions. Bus 826 includes components that allow communication between internal components 800A, 800B.

[0060] Server computer 114 ( Figure 1 One or more operating systems 828 and software programs 108 on ) Figure 1 ) and pose graph reconstruction procedure 116 ( Figure 1 The data is stored on one or more of the corresponding computer-readable tangible storage devices 830 for execution by one or more of the corresponding processors 820 via one or more of the corresponding RAMs 822 (which typically include cache memory). Figure 4In the illustrated embodiment, each computer-readable tangible storage device 830 is a disk storage device of an internal hard disk drive. Alternatively, each computer-readable tangible storage device 830 is a semiconductor storage device, such as ROM 824, EPROM, flash memory, optical disc, magneto-optical disc, solid-state drive, optical disc (CD), digital universal disc (DVD), floppy disk, cassette tape, magnetic tape, and / or another type of non-transitory computer-readable tangible storage device capable of storing computer programs and digital information.

[0061] Each group of internal components 800A, 800B also includes an R / W drive or interface 832 for reading from and writing to one or more portable computer-readable tangible storage devices 936 (e.g., CD-ROM, DVD, Memory Stick, magnetic tape, disk, optical disc, or semiconductor storage device). Software programs, such as software program 108 ( Figure 1 ) and pose graph reconstruction procedure 116 ( Figure 1 It can be stored on one or more of the corresponding portable computer-readable tangible storage devices 936, read from and loaded into the corresponding hard disk drive via the corresponding R / W drive or interface 832.

[0062] Each set of internal components 800A, 800B also includes a network adapter or interface 836, such as a TCP / IP adapter card; a wireless Wi-Fi interface card; or a 3G, 4G, 5G wireless interface card or other wired or wireless communication links. The server computer 114 can be connected via a network (e.g., the Internet, a local area network, or other wide area network) and the corresponding network adapter or interface 836. Figure 1 Software program 108 on ) Figure 1 ) and pose graph reconstruction procedure 116 ( Figure 1 Download from external computer to computer 102 ( Figure 1 The network includes a network adapter or interface 836 and a server computer 114. Software program 108 and posture map reconstruction program 116 on server computer 114 are loaded into their respective hard drives from the network adapter or interface 836. The network may include copper wire, fiber optic, wireless transmission, routers, firewalls, switches, gateway computers, and / or side servers.

[0063] Each of the external component groups 900A and 900B may include a computer monitor 920, a keyboard 930, and a computer mouse 934. External components 900A and 900B may also include a touchscreen, a virtual keyboard, a touchpad, a pointing device, and other human-machine interface devices. Each of the internal component groups 800A and 800B also includes a device driver 840 for interfacing with the computer monitor 920, keyboard 930, and computer mouse 934. Device driver 840, R / W driver or interface 832, and network adapter or interface 836 include hardware and software (stored in storage device 830 and / or ROM 824).

[0064] It should be understood in advance that although this disclosure includes a detailed description of cloud computing, the implementations taught herein are not limited to cloud computing environments. Rather, some implementations can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0065] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources, such as networks, network bandwidth, servers, processing power, memory, storage, applications, virtual machines, and services, which can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0066] The characteristics are as follows:

[0067] On-demand self-service: Cloud consumers can automatically and unilaterally provide computing power, such as server time and network storage, as needed, without the need for manual interaction with service providers.

[0068] Extensive network access: Functionality can be accessed via the network and through standard mechanisms, which can facilitate the use of heterogeneous thin or thick client platforms such as mobile phones, laptops, and PDAs.

[0069] Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, dynamically allocating and reallocating different physical and virtual resources based on demand. There is a sense of location independence because consumers typically cannot control or know the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0070] Rapid and flexible: Features can be provided quickly and flexibly, in some cases automatically, to scale out quickly and release quickly to scale inward. For consumers, the available features often appear unlimited and can be purchased in any quantity at any time.

[0071] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0072] The service model is as follows:

[0073] Software as a Service (SaaS): The functionality provided to consumers is the use of a vendor's applications running on cloud infrastructure. These applications can be accessed from a variety of client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application functionality, with possible exceptions such as limited, user-specific application configuration settings.

[0074] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer onto cloud infrastructure. These applications are created using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they do control the deployed applications and the configuration of any possible application hosting environments.

[0075] Infrastructure as a Service (IaaS): This provides consumers with the capability to offer processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do control the operating system, storage, deployed applications, and may have limited control over selected network components (e.g., host firewalls).

[0076] The deployment model is as follows:

[0077] Private cloud: A cloud infrastructure that operates solely for an organization. It may be managed by the organization or a third party, and may exist internally or externally.

[0078] Community cloud: A cloud infrastructure shared by multiple organizations that supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by an organization or a third party and may exist internally or externally.

[0079] Public cloud: Cloud infrastructure available to the public or large industry groups, and cloud services are sold by the organizations that own it.

[0080] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community, or public) that remain independent entities but are bound together by standardized or proprietary technologies to enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0081] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure comprised of interconnected networks of nodes.

[0082] refer to Figure 5 The diagram illustrates an illustrative cloud computing environment 500. As shown, the cloud computing environment 500 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers, such as personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N, can communicate. The cloud computing nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private, community, public, or hybrid clouds as described above, or combinations thereof. This allows the cloud computing environment 500 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 5 The types of computing devices 54A-N shown are for illustrative purposes only, and cloud computing node 10 and cloud computing environment 500 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0083] refer to Figure 6 This demonstrates the 500-fold cloud computing environment ( Figure 5 This provides a set of functional abstraction layers, 600. It should be understood beforehand that... Figure 6 The components, layers, and functions shown are for illustrative purposes only, and the implementation is not limited thereto. As shown, the following layers and corresponding functions are provided:

[0084] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a server 62 based on a RISC (Reduced Instruction Set Computer) architecture; a server 63; a blade server 64; a storage device 65; and a network and network components 66. In some implementations, software components include network application server software 67 and database software 68.

[0085] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73 including virtual private network; virtual application and operating system 74; and virtual client 75.

[0086] In one example, management layer 80 may provide the following functionalities: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking when using resources in the cloud computing environment, as well as billing or invoicing the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to meet the required service level. Service Level Agreement (SLA) Planning and Implementation 85 provides pre-scheduling and procurement of cloud resources to align with future needs anticipated based on the SLA.

[0087] Workload layer 90 provides examples of functionalities that can leverage cloud computing environments. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and pose map reconstruction 96. Pose map reconstruction 96 can estimate pose maps and transformation matrices between cameras by identifying markers on the ground in panoramic images for 3D reconstruction of indoor buildings in VR / AR applications.

[0088] Some implementations may relate to systems, methods, and / or computer-readable media at any possible level of technical detail integration. A computer-readable medium may include a computer-readable non-transitory storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform operations.

[0089] The computer-readable storage medium can be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanical encoding devices (e.g., punched cards or raised structures in recesses storing instructions), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0090] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices or to external computers or external storage devices via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). This network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0091] The computer-readable program code / instructions used to perform the operations of this invention can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as C or similar languages. The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network (including local area network (LAN) or wide area network (WAN)) or can establish a connection with an external computer (e.g., via the Internet through an Internet service provider). In some embodiments, electronic circuitry (including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), and programmable logic arrays (PLAs)) can perform aspects or operations by utilizing the status information of the computer-readable program instructions to personalize the electronic circuitry.

[0092] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine so that the instructions, which execute via the processor of the computer or the processor of the other programmable data processing apparatus, create methods to perform the functions / actions specified in the flowcharts and / or block diagrams. The computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium containing the instructions includes an article of manufacture comprising instructions for performing the functions / actions specified in the flowcharts and / or block diagrams.

[0093] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus or other device implement the functions / actions specified in the flowchart and / or block diagram.

[0094] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. The method, computer system, and computer-readable medium may include additional blocks, fewer blocks, different blocks, or blocks arranged differently from those shown in the figures. In some alternative embodiments, the functions shown in the blocks may not occur in the order shown in the figures. For example, in practice, depending on the functions involved, two consecutively shown blocks may be executed simultaneously or substantially simultaneously, or sometimes these blocks may be executed in reverse order. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0095] Clearly, the systems and / or methods described herein can be implemented in various forms, including hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit these implementations. Therefore, this document describes the operation and behavior of the systems and / or methods without referring to any specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0096] Unless explicitly described as critical or necessary, no element, action, or instruction used herein should be construed as critical or necessary. Furthermore, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more.” If only one item is intended, the term “an” or similar language is used. Furthermore, as used herein, the terms “have,” “possess,” “have,” etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the term “based on” is intended to mean “at least partially based on.”

[0097] Descriptions of various aspects and embodiments have been presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Even though combinations of features are listed in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible embodiments. In fact, many of these features can be combined in ways not specifically stated in the claims and / or not disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the possible disclosure includes each dependent claim combined with every other claim in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, practical applications of technology found in the market, or technical improvements, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for constructing a camera pose diagram, characterized in that, Executed in the processor, including: Receive image data corresponding to one or more views of the first camera; Identify one or more markers corresponding to the second camera within the received image data; In response to one or more markers in the image data corresponding to the second camera in the one or more views of the first camera, it is determined that the second camera is visible from the first camera, and the first camera is visible from the second camera; In response to the second camera being visible from the first camera and the first camera being visible from the second camera, a transformation matrix between the first camera and the second camera is calculated based on the coordinates of each identified marker in the image data. The transformation matrix includes a rotation matrix between the first camera and the second camera, and a translation vector between the first camera and the second camera. The rotation matrix is ​​obtained by the following formula: ； Among them, the The transformation matrix is ​​the transformation matrix, the The yaw angle is the 3D coordinate system associated with the first camera, and the Obtained from the following formula: ； Among them, the For a predefined function, the Let be the relative ordinate of the second camera with respect to the first camera in the three-dimensional coordinate system. Let x be the relative x-coordinate of the second camera with respect to the first camera in the three-dimensional coordinate system. Let be the relative ordinate of the first camera with respect to the second camera in the three-dimensional coordinate system. Let x be the relative x-coordinate of the first camera with respect to the second camera in the three-dimensional coordinate system. Based on the movement relationship between the first camera and the second camera indicated by the transformation matrix, a pose graph corresponding to the one or more views of the first camera is constructed, the pose graph including at least edges.

2. The method according to claim 1, characterized in that, The nodes of the pose graph correspond to each of the one or more views.

3. The method according to claim 1, characterized in that, The edges of the pose graph are constructed based on the identified one or more markers.

4. The method according to claim 1, characterized in that, The edge is constructed between the first camera and the second camera in the one or more views corresponding to the first camera, based on the markers in the one or more views corresponding to the first camera.

5. The method according to any one of claims 1 to 4, characterized in that, Based on the transformations associated with the markers in the one or more views corresponding to the first camera and the one or more markers corresponding to the second camera, the transformation matrix between the first camera and the second camera is estimated from the one or more views.

6. The method according to claim 5, characterized in that, The transformation matrix is ​​associated with camera movement between the first camera and the second camera.

7. A computer system for constructing camera pose diagrams, characterized in that, The computer system includes: One or more computer-readable non-transitory storage media are configured to store computer program code; One or more computer processors are configured to access and operate in accordance with the instructions of the computer program code, the computer program code comprising: A receiving code is configured to cause the one or more computer processors to receive image data of one or more views corresponding to the first camera; The identification code is configured to enable the one or more computer processors to identify one or more markers in the received image data that correspond to the second camera; The code is configured to cause the one or more computer processors to determine, in response to one or more markers in image data corresponding to the one or more views of the first camera and corresponding to the second camera, that the second camera is visible from the first camera and the first camera is visible from the second camera; and in response to the second camera being visible from the first camera and the first camera being visible from the second camera, to calculate a transformation matrix between the first camera and the second camera based on the coordinates of each of the identified markers in the image data, wherein the transformation matrix includes: a rotation matrix between the first camera and the second camera, and a translation vector between the first camera and the second camera, the rotation matrix being obtained by the following formula: ； Among them, the The transformation matrix is ​​the transformation matrix, the The yaw angle is the 3D coordinate system associated with the first camera, and the Obtained from the following formula: ； Among them, the For a predefined function, the Let be the relative ordinate of the second camera with respect to the first camera in the three-dimensional coordinate system. Let x be the relative x-coordinate of the second camera with respect to the first camera in the three-dimensional coordinate system. Let be the relative ordinate of the first camera with respect to the second camera in the three-dimensional coordinate system. The relative horizontal coordinate of the first camera with respect to the second camera in the three-dimensional coordinate system is given; based on the movement relationship between the first camera and the second camera indicated by the transformation matrix, a pose diagram of the one or more views corresponding to the first camera is constructed, the pose diagram including at least edges.

8. The computer system according to claim 7, characterized in that, It also includes first determining code configured to cause the one or more computer processors to determine the visibility between the positions of the first camera and the second camera, wherein the second camera is determined to be visible from the first camera in response to the presence of a marker in one or more markers corresponding to the second camera in one or more views corresponding to the first camera.

9. The computer system according to claim 7, characterized in that, The nodes of the pose graph correspond to each of the one or more views.

10. The computer system according to claim 7, characterized in that, The edges of the pose graph are constructed based on the identified one or more markers.

11. The computer system according to claim 7, characterized in that, The edge is constructed between the first camera and the second camera in the one or more views corresponding to the first camera, based on the markers in the one or more views corresponding to the first camera.

12. The computer system according to any one of claims 7-11, characterized in that, It also includes estimation code configured to cause the one or more computer processors to estimate a transformation matrix between the first camera and the second camera from the one or more views based on a transformation associated with a marker in one or more markers corresponding to the second camera in the one or more views corresponding to the first camera.

13. The computer system according to claim 12, characterized in that, The transformation matrix is ​​associated with camera movement between the first camera and the second camera.

14. A non-transitory computer-readable medium storing a computer program for constructing a pose diagram of a camera, the computer program being configured to cause one or more computer processors to implement the method as claimed in any one of claims 1-6.

Images