Image display system and method

Abstract

Systems and methods for co-displaying an image and a map are disclosed. The method may include obtaining an image captured by a camera. The image may include an object in the environment around the camera. The method may further include obtaining a map of the surrounding environment. The map may at least include a location corresponding to the object and a location corresponding to the camera. The method may co-display the image and the map, wherein a representation of the image is aligned with the location corresponding to the object and the location corresponding to the camera in the map.

Description

Technical Field

[0001] This application generally relates to image display, and more particularly, to systems and methods for co-displaying images and maps. Background Art

[0002] Various techniques have been used to capture and display images covering a sufficiently large field of view (FOV).

[0003] In some embodiments, multiple cameras are applied to surveillance in large venues, such as amusement parks, squares, airports, to form a sufficiently large FOV. However, it may be difficult to effectively display the images captured by multiple cameras in a user-friendly manner.

[0004] In some embodiments, panoramic devices can be used to capture panoramic images with an FOV approaching or even exceeding 180 degrees. Virtual reality (VR) headsets can present panoramic images from different perspectives according to the movement of the headset. However, VR headsets are not suitable for multiple people to view simultaneously. Summary of the Invention

[0005] According to one aspect of the present application, a system is provided. The system may include at least one storage medium containing an instruction set, and at least one processor communicatively coupled to the at least one storage medium. When the instruction set is executed, the at least one processor may be directed to cause the system to obtain an image captured by a camera. The image may include a target in the environment around the camera. The at least one processor may also be directed to cause the system to obtain a map of the surrounding environment. The map may at least include the position corresponding to the target and the position corresponding to the camera. The at least one processor may also be directed to cause the system to co-display the image and the map, wherein the representation of the image is aligned with the position corresponding to the target and the position corresponding to the camera in the map.

[0006] In some embodiments, the image captured by the camera may be panoramic.

[0007] In some embodiments, to co-display the image and the map, the at least one processor may be directed to cause the system to project the content of the map onto a first imaging surface to form a representation of the map, and project the content of the image onto a second imaging surface to form a representation of the image. The at least one processor may also be directed to cause the system to co-display the representation of the map and the representation of the image.

[0008] In some embodiments, the second imaging surface may include a cylindrical surface.

[0009] In some embodiments, the representation of the map may be formed according to a first projection relationship associated with the map, and the representation of the image may be formed according to a second projection relationship associated with the image.

[0010] In some embodiments, the first projection relationship may include a first rotation angle associated with the orientation of the map.

[0011] In some embodiments, the second projection relationship may include a second rotation angle associated with the representation of the image on the second imaging surface.

[0012] In some embodiments, the at least one processor may also be oriented to cause the system to receive an instruction to adjust a common display orientation of the image and the map. The at least one processor may also be oriented to cause the system to adjust the first rotation angle associated with the orientation of the map and the second rotation angle associated with the representation of the image on the second imaging surface, and update the common display of the image and the map according to the adjusted first rotation angle and the adjusted second rotation angle.

[0013] In some embodiments, the first projection relationship may include a first scale associated with a scaling effect of the map.

[0014] In some embodiments, the second projection relationship may include a second scale associated with a scaling effect of the image.

[0015] In some embodiments, the at least one processor may also be oriented to cause the system to receive an instruction to adjust a field of view in a common display of the image and the map. The at least one processor may also be oriented to cause the system to adjust the first scale associated with the scaling effect of the map and the second scale associated with the scaling effect of the image, and update the common display of the image and the map according to the adjusted first scale and the adjusted second scale.

[0016] According to another aspect of the present application, a method is provided. The method may be implemented on a device having at least one storage medium and at least one processor, the at least one storage medium containing a set of instructions, and the at least one processor communicating with the at least one storage medium. The method may include obtaining an image captured by a camera. The image may include a target in the environment around the camera. The method may further include obtaining a map of the surrounding environment. The map may at least include a location corresponding to the target and a location corresponding to the camera. The method may further include co-displaying the image and the map, wherein the representation of the image is aligned with the location corresponding to the target and the location corresponding to the camera in the map.

[0017] Some additional features of the present application may be described in the following description. Through the study of the following description and the corresponding drawings, or the understanding of the production or operation of the embodiments, some additional features of the present application will be apparent to those skilled in the art. The features of the present application can be realized and achieved by the practice or use of the methods, means, and combinations of the various aspects of the specific embodiments described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present application will be further described by way of exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. The drawings are not drawn to scale. These embodiments are non-limiting exemplary embodiments, in which the same reference numerals in the various figures represent similar structures, where:

[0019] Figure 1 is a schematic diagram of an exemplary image display system according to some embodiments of the present application;

[0020] Figure 2 is a schematic diagram of exemplary hardware and / or software components of an exemplary computing device according to some embodiments of the present application;

[0021] Figure 3 is a schematic diagram of exemplary components of an exemplary user device according to some embodiments of the present application;

[0022] Figure 4 is a block diagram of an exemplary image processing device according to some embodiments of the present application;

[0023] Figure 5 is a flowchart of an exemplary process for jointly displaying a map and an image according to some embodiments of the present application;

[0024] Figure 6 is a flowchart of an exemplary process for jointly displaying a map and an image according to some embodiments of the present application;

[0025] Figure 7 is a flowchart of an exemplary process for adjusting the jointly displayed orientation according to some embodiments of the present application;

[0026] Figure 8 is a flowchart of an exemplary process for adjusting the field of view of the joint display according to some embodiments of the present application;

[0027] Figure 9 is an exemplary camera arrangement according to some embodiments of the present application;

[0028] Figure 10 is according to some embodiments of the present application Figure 9 an exemplary map of the surrounding environment of the building shown; and

[0029] Figure 11 is an exemplary co - display of an image and a map according to some embodiments of the present application. Detailed implementation manners

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application. For those of ordinary skill in the art, without creative efforts, the present application can also be applied to other similar scenarios based on these drawings. Unless obvious from the context or otherwise stated, the same reference numerals in the figures represent the same structures and operations.

[0031] As shown in the present application and the claims, unless the context clearly indicates an exception, words such as "a", "an", "one", and / or "the" are not specifically singular and may also include the plural. It should be further understood that the terms "including" and / or "comprising" used in the present application only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list, and other steps and elements may also be included.

[0032] According to some embodiments of the present application, some modules of the system may be referred to in various ways. However, any number of different modules may be used and operated in the client and / or the server. These modules are only for illustration and do not limit the scope of the present application. Different modules may be used in different aspects of the system and method.

[0033] According to some embodiments of the present application, flowcharts are used to show the operations performed by the system. It should be understood that the operations before or below do not necessarily need to be executed precisely in order. On the contrary, various steps may be executed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more of the above operations may be removed from these processes.

[0034] The technical solutions of the embodiments of the present application will be described with reference to the accompanying drawings as follows. Obviously, the described embodiments are not exhaustive or restrictive. Based on the embodiments proposed in the present application, other embodiments obtained by those of ordinary skill in the art without creative efforts are all within the scope of the present application.

[0035] In one aspect, the present application relates to systems and methods for co-displaying a map and an image. In some embodiments, an image can be captured by a camera. The image can include an object in the environment around the camera. The map can include information about the environment around the camera. The map can at least include a position corresponding to the object and a position corresponding to the camera. In some embodiments, in order to co-display the image and the map, the map can be oriented so that the representation of the image is aligned with the position corresponding to the object and the position corresponding to the camera in the map.

[0036] Figure 1 FIG. 4 is a schematic diagram of an exemplary image display system 100 shown according to some embodiments of the present application. The image display system 100 can be configured to co-display multiple pictures. In some embodiments, the multiple pictures can include one or more images captured by one or more cameras and a map indicating the environment around the one or more cameras. The co-display of the one or more images and the map can include presenting the content of the one or more images and the map simultaneously on one or more imaging surfaces. For example, the content of the one or more images can be displayed as a stereogram on a cylindrical imaging surface, while the map can be displayed on a planar imaging surface in a manner aligned with the stereogram. In some embodiments, the one or more images can include a combined image formed by stitching images captured by at least two cameras.

[0037] The image display system 100 can include at least two cameras 110, an image processing device 120, a network 130, one or more terminal devices 140, and a storage device 150. According to actual needs, the image display system 100 can also include other components.

[0038] The at least two cameras 110 (e.g., Figure 1 camera #1, camera #2, camera #3 shown in FIG. 4) can each generate an image or a video including at least two frames. As used herein, the image can be a photograph, a video frame, a picture, a spectrum, etc. or a combination thereof. The image can be in analog form (e.g., an electrical signal carrying image data) or digital form (e.g., a digital file including image data) or presented in analog form (e.g., an electrical signal carrying image data) or digital form (e.g., a digital file including image data). The at least two cameras 110 can sense light, waves, radiation, etc. to generate corresponding images.

[0039] In some embodiments, each of the at least two cameras 110 may be a panoramic camera. Exemplary panoramic cameras may include a stereocycler, a magic panoramic camera, a manual panoramic camera, a rotating panoramic camera, a fixed-lens camera, etc., or any combination thereof. A panoramic camera can generate a panoramic image. As used herein, a panoramic image may refer to an image that exhibits a FOV with an angle equal to or greater than a specific threshold. The threshold may be 90 degrees, 120 degrees, 140 degrees, 160 degrees, 180 degrees, etc. In some embodiments, the panoramic image may exhibit a FOV larger than that of the human eye. In some embodiments, the panoramic image may be formed by combining at least two sub-images captured by the panoramic camera. The at least two sub-images may have a horizontally elongated field of view.

[0040] The number of the at least two cameras 110 may be set according to actual needs. For example, if the FOV angle exhibited by the image generated by each camera 110 is close to 90 degrees, the number of cameras 110 may be four or more, such as six. If the FOV angle exhibited by the image generated by each camera 110 is close to 120 degrees, the number of cameras 110 may be three or more, such as four.

[0041] In some embodiments, some or all of the at least two cameras 110 may be included in an independent imaging device, such as an integrated camera, a sonar, a night vision device, a thermal imaging device, a medical imaging device, etc., or a combination thereof. For example, Figure 1 the cameras #1, #2, and #3 shown may be multi-lens multi-sensor cameras (or simply referred to as multi-lens cameras).

[0042] In some embodiments, the at least two cameras 110 may be independent of each other and installed at different positions. For example, the at least two cameras 110 may be installed at a plurality of positions distributed substantially along a circle. According to actual needs, the at least two cameras 110 may operate in a unified or different working state. For example, only a part of the at least two cameras 110 may be activated to capture images, while the remaining part of the at least two cameras 110 remains in a standby state.

[0043] The image processing device 120 can process the images (e.g., Image #1 to #3) generated by the at least two cameras 110. For example, the image processing device 120 can process multiple images to remove the overlapping regions between them. As another example, the image processing device 120 can generate a combined image based on the multiple images. As another example, the image processing device 120 can determine the way to display one or more of these images. Specifically, the image processing device 120 can have a display function of co-displaying one or more images and a map indicating the surrounding environment of the at least two cameras 110. To achieve the co-display, the image processing device 120 can project the content of the map onto a first imaging surface to form a representation of the map (e.g., a first projection image), and project the content of one or more images onto a second imaging surface to form a representation of the images (e.g., one or more second projection images). Then, the representation of the map and the representation of the one or more images can be co-displayed on the image processing device 120.

[0044] In some embodiments, the image processing device 120 can adjust the co-display according to instructions from the user. Exemplary instructions can include adjustments to the co-display direction, adjustments to the FOV in the co-display, etc. or a combination thereof. For example, the image processing device 120 can monitor the input device and identify user instructions by analyzing the movement of the input device (e.g., mouse, keyboard). Specifically, the image processing device 120 can identify an instruction to adjust the co-display direction through the drag-and-drop movement of the mouse, and adjust one or more parameters associated with a first rotation angle and / or one or more second rotation angles, where the first rotation angle is associated with the direction of the map and the second rotation angle is associated with the representation of the image. Additionally, the image processing device 120 can adjust the FOV in the co-display by adjusting a first ratio associated with the zoom effect of the map and a second ratio associated with the zoom effect of the image.

[0045] In some embodiments, the image processing device 120 can be implemented by a computing device as Figure 2 shown.

[0046] The image processing device 120 can communicate with the storage device 150 via one or more cables or networks 130. Before the images generated by the multiple cameras 110 are processed by the image processing device 120, the storage device 150 can store or cache the images. In some embodiments, the processed images can be stored or cached in the storage device 150 before being output to, for example, the image processing device 120 for display or the network 130 for transmission. In some embodiments, the image display system 100 can include multiple storage devices ( Figure 1(not shown in the figure) and a storage device 150. The multiple storage devices are used to store or cache the images generated by cameras #1, #2, and #3 respectively, and the storage device 150 is used to store or buffer the processed images generated by the image processing device 120.

[0047] The storage device 150 may be or may include a server, a mass storage device, a removable storage device, a volatile read-write memory, a read-only memory (ROM), a random access memory (RAM), etc., or any combination thereof. Exemplary mass storage devices may include magnetic disks, optical disks, solid state drives, etc. Exemplary removable storage devices may include flash drives, floppy disks, optical disks, memory cards, compact disks, magnetic tapes, etc. Exemplary volatile read-write memories may include random access memories (RAM). Exemplary RAM may include dynamic random access memory (DRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), static random access memory (SRAM), thyristor random access memory (T-RAM), and zero-capacitor random access memory (Z-RAM), etc. Exemplary ROM may include mask ROM (MROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), compact disk read-only memory (CD-ROM), and digital versatile disk read-only memory, etc.

[0048] In some embodiments, the storage device 150 may send data to the image processing device 120 using a burst transfer protocol. As used herein, the burst transfer protocol may refer to repeatedly transferring data stored in consecutive storage units of the storage device 150 without going through all the steps required to transfer each data segment in a separate transfer.

[0049] For example, when each data segment is not transmitted using the burst transfer protocol, the storage device 150 may first locate the storage unit storing the data segment and then access the data segment stored therein. Locating the storage unit may take multiple clock cycles (e.g., 2, 3), which may cause a delay in data transmission. By utilizing the burst transfer protocol, the storage device 150 only needs to locate the storage unit storing the starting data segment, transmit the starting data segment, and then repeatedly and automatically transmit the data segments stored in the adjacent storage units of the current storage unit (repeatedly beside the current storage unit without locating it) until a preset number (burst length) of storage units are accessed. Therefore, the time cost of transferring data stored in consecutive storage units can be reduced. For example, when it takes 3 clock cycles to locate the storage unit and 1 clock cycle to transmit the data segment stored in the storage unit, to transmit the data stored in 4 consecutive storage units, without using the above burst transfer protocol, a total of 16 clock cycles can be spent (4 clock cycles to transfer each data segment). However, in the above case, when the burst transfer protocol is adopted (while the burst length is more than 4), a total of 7 clock cycles can be spent (3 clock cycles to locate the storage unit storing the starting data segment and 1 clock cycle to transmit each data segment).

[0050] Generally, the image (or source image) received from any one of the at least two cameras 110 or each part thereof may be stored in consecutive storage units of the storage device 150. Such an image storage method may be referred to as the continuity of the image. By utilizing the burst transfer protocol, the time cost of sending at least two parts of the image to the image processing device 120 can be reduced.

[0051] It can be understood that the above burst transfer protocol is for illustrative purposes only and not restrictive. Different types of storage devices 150 may adopt different types of burst transfer protocols. In some embodiments, the storage device 150 may be a double data rate (DDR) memory, such as DDR synchronous dynamic random access memory (SDRAM), low power DDR SDRAM (LP-DDR).

[0052] In some embodiments, the storage device 150 may also be configured to store instructions (e.g., in the form of software, application, program) executed by the image processing device 120 (or one or more of its processors). When executed by the image processing device 120, the instructions may cause the image processing device 120 to execute the processes related to the co-display of images disclosed in the present application.

[0053] The image processing device 120 may receive images (e.g., source images) and maps generated by the at least two cameras 110 via one or more cables or networks 130. In some embodiments, the at least two cameras 110 and the image processing device 120 may form a multi-channel camera system. In some embodiments, the at least two cameras 110 and the image processing device 120 may be combined to form a single device, such as a multi-lens camera having the function of processing image data.

[0054] The network 130 may include any suitable network that can facilitate the transmission of data and signals between components of the image display system 100. For example, the network 130 may facilitate the transmission of images from the at least two cameras 110 to the image processing device 120 (or the storage device 150).

[0055] The network 130 may be and / or include one or more types of networks. For example, the network 130 may be and / or include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN), a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network, a Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), a frame relay network, a virtual private network (“VPN“), a satellite network, a Bluetooth TM network, a ZigBee TM network, a near field communication (NFC) network, etc. or any combination thereof. The network 130 may also include wired and / or wireless network access points, such as base stations, Internet exchange points, routers, hubs, gateways, switches, server computers, and / or any combination thereof.

[0056] The terminal device 140 may receive images or videos from the at least two cameras 110, the image processing device 120, and / or the storage device 150 via the network 130. In some embodiments, the terminal device 140 may also be a user interface for providing control of the at least two cameras 110 and / or the image processing device 120.

[0057] In some embodiments, the terminal device 140 may enable interaction between the user and the image processing device 120. For example, an administrator or an employee of a security company may send an instruction to the terminal device 140 via the terminal device 140, and the instruction is for jointly displaying an image (or a video) and a map indicating the physical location of the scene including the image (or the video). Additionally or alternatively, the administrator or the employee may submit a request to the image display system 100 via the terminal device 140 to search for or track a target object in the image (or the video).

[0058] The terminal device 140 may include a mobile computing device 141, a tablet computer 142, a laptop computer 143, a smart home device (not shown), a desktop computer (not shown), etc., or any combination thereof. In some embodiments, the mobile computing device 141 may include a wearable device, a mobile phone, a virtual reality device, an augmented reality device, a personal digital assistant (PDA), a navigation device, etc., or any combination thereof.

[0059] In some embodiments, the image display system 100 may implement a monitoring system. The source images generated by the at least two cameras 110 may be in the form of a video stream. The images displayed by the image processing device 120 may be a panoramic video stream. The processes provided in this application may reduce the time cost of generating the panoramic video stream and improve the real-time performance to meet the requirements of providing monitoring services. The image display system 100 may also be applied to other related fields, such as photography, film production, news, medical imaging, etc.

[0060] In some embodiments, the image display system 100 includes only one movable camera for generating source images. The movable camera may move along a preset movement trajectory and capture images at each preset shooting position along the preset movement trajectory. The obtained images may be used as source images for further processing.

[0061] In some embodiments, each preset shooting position along the preset movement trajectory may be associated with one or more preset imaging parameters (such as a shooting angle). When the movable camera moves to the preset shooting position, the camera may adopt the one or more preset imaging parameters associated with the preset shooting position to capture the corresponding source image.

[0062] For illustrative purposes, this application may be described on the premise that the source images are generated by a preset number of cameras. However, it can be understood that the source images may also be generated by a smaller number of cameras (including one or more movable cameras) or even just one movable camera.

[0063] Note that the above description of the image display system 100 is for illustrative purposes only and is not intended to limit the scope of this application. It can be understood that after learning the main concepts and mechanisms of this application, those of ordinary skill in the art can change the image display system 100 in a non-creative manner. The changes may include combining and / or splitting certain devices / components / modules / units, adding or removing optional devices / components / modules / units, changing the connection status of the devices / components / modules / units, applying the image display system 100 in related fields, etc., or any combination thereof. All such modifications are within the scope of this application.

[0064] Figure 2is a schematic diagram of exemplary hardware and / or software components of an exemplary computing device as shown in some embodiments of the present application. For example, the computing device 200 may be an image processing device 120, and / or an electronic device dedicated to video or image processing. As Figure 2 shown, the computing device 200 may include a processor 222, a memory 227, an input / output (I / O) 226, and a communication port 225.

[0065] The processor 222 (e.g., logic circuit) may execute computer instructions (e.g., program code) and perform functions according to the technologies described herein. For example, the processor 222 may include an interface circuit and a processing circuit inside. The interface circuit may be configured to receive an electronic signal from a bus ( Figure 2 not shown), where the electronic signal is an encoding of structured data and / or instructions for the processing circuit to process. The processing circuit may perform logical operation calculations and then determine conclusions, results, and / or instructions encoded as electronic signals. Then the interface circuit may send out the electronic signal from the processing circuit via the bus.

[0066] The computer instructions may include, for example, routines, programs, objects, components, data structures, procedures, modules, and functions for performing specific functions described herein. In some embodiments, the processor 222 may include one or more hardware processors, such as a microcontroller, a microprocessor, a microprocessor, a reduced instruction set computer (RISC), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physics processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0067] For illustration only, only one processor is described in the computing device 200. However, it should be noted that the computing device 200 in the present application may also include multiple processors. Therefore, the operations and / or method operations performed by one processor as described in the present application may also be performed jointly or separately by the multiple processors. For example, if the processor of the computing device 200 in the present application performs both operation A and operation B, it should be understood that operation A and operation B may also be performed jointly or separately by two or more different processors in the computing device 200 (e.g., the first processor performs operation A, the second processor performs operation B, or the first and second processors jointly perform operations A and B).

[0068] The memory 227 may store data / information obtained from any of the at least two cameras 110, the image processing device 120, the terminal device 140, the network 130, the storage device 150, and / or any other components of the image display system 100. In some embodiments, the memory 227 may include a mass storage, a removable memory, a volatile read-write memory, a read-only memory (ROM), etc., or any combination thereof. For example, the mass storage may include a magnetic disk, an optical disk, a solid state drive, etc. The removable memory may include a flash drive, a floppy disk, an optical disk, a memory card, a zip disk, a magnetic tape, etc. The volatile read-write memory may include a random access memory (RAM), a double data rate synchronous dynamic RAM (DDR SDRAM), a static RAM (SRAM), a thyristor RAM (T-RAM), and a zero capacitor RAM (Z-RAM), etc., where the RAM may include a dynamic RAM (DRAM). The ROM may include a mask ROM (MROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a CD-ROM, and a digital versatile disk ROM, etc. In some embodiments, the memory 227 may store one or more programs and / or instructions for executing the exemplary methods described in the present application.

[0069] The I / O 226 may input and / or output signals, data, information, etc. In some embodiments, the I / O 226 may include an input device and an output device. Examples of the input device may include a keyboard, a mouse, a touch screen, a microphone, etc., or a combination thereof. Examples of the output device may include a display device, a speaker, a printer, a projector, etc., or a combination thereof. Examples of the display device may include a liquid crystal display (LCD), a light emitting diode (LED), an LED-based display, a flat panel display, a curved screen, a television device, a cathode ray tube (CRT), a touch screen, etc., or a combination thereof.

[0070] The communication port 225 may be connected to a network (e.g., the network 130) to facilitate data communication. The communication port 225 may establish a connection between any of the at least two cameras 110, the image processing device 120, the terminal device 140, the network 130, the storage device 150, and / or any other components of the image display system 100. The connection may be a wired connection, a wireless connection, any other communication connection capable of realizing data sending and / or receiving, and / or any combination thereof. The wired connection may include, for example, a cable, an optical fiber cable, a telephone line, etc., or any combination thereof. The wireless connection may include, for example, Bluetooth TM link, Wi-Fi TM link, WiMAX TMLinks, WLAN links, ZigBee links, mobile network links (e.g., 3G, 4G, 5G), etc. or combinations thereof. In some embodiments, the communication port 225 may be and / or include a standardized communication port, e.g., RS232, RS485, etc. In some embodiments, the communication port 225 may be a specially designed communication port.

[0071] Figure 3 is a schematic diagram of exemplary components of an exemplary user device according to some embodiments of the present application. As Figure 3 shown, the user device 300 may include a communication platform 320, a display 310, a graphics processing unit (GPU) 330, a central processing unit (CPU) 340, an I / O port 350, a memory 360, and a storage 390. In some embodiments, any other suitable components, including but not limited to a system bus or a controller (not shown), may also be included in the user device 300. In some embodiments, a mobile operating system 370 (e.g., iOS TM , Android TM , WindowsPhone TM ) and one or more applications 380 may be loaded from the storage 390 into the memory 360 for execution by the processor 340. The user device 300 may be an implementation of the terminal device 140.

[0072] To implement the various modules, units, and their functions described in the present application, a computer hardware platform may be used as the hardware platform for one or more components described herein. A computer with user interface elements may be used to implement a personal computer (PC) or any other type of workstation or terminal device. If the computer is appropriately programmed, the computer may also be used as a server.

[0073] Figure 4 is a block diagram of an exemplary image processing device 120 according to some embodiments of the present application. The image processing device 120 may include an acquisition module 410, a map processing module 420, an image processing module 430, a display module 440, and a storage module 450. Without loss of generality, the image processing device 120 may include more or fewer components. For example, two modules may be combined into a single module, or one module may be split into two or more modules. As another example, one or more modules may reside on different computing devices (e.g., a desktop computer, a laptop computer, a mobile device, a tablet computer, a wearable computing device, etc. or combinations thereof). As another example, the image processing device 120 may be implemented on Figure 2 the computing device 200 shown.

[0074] Here and throughout this application, modules can be implemented in many different ways and in different combinations of hardware, software, or both. For example, the implementation of some or all of the modules can be a processing circuit, which can include part or all of an instruction processor, such as part or all of a central processing unit (CPU), a microcontroller, a microprocessor; or include an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, other electronic components; or be a circuit including discrete logic or other circuit components, including analog circuit components and / or digital circuit components; or any combination of the above. For example, the circuit can include discrete interconnected hardware components, or can be combined on a single integrated circuit die, distributed between multiple integrated circuit dies, or implemented in a multi-chip module (MCM) of multiple integrated circuit dies in a common package.

[0075] The obtaining module 410 can obtain information and / or data related to image display. For example, the obtaining module 410 can obtain an image captured by the camera from the shooting direction. The image can include an object in the environment around the camera. The object can be any object visible through the camera from the shooting direction (e.g., a building, a tree, a street, a river, etc.). In addition, the obtaining module 410 can obtain a map of the environment around the camera. In some embodiments, the map can be a planar map showing the environment around the camera in a two-dimensional (2D) manner.

[0076] In some embodiments, the obtaining module 410 can send the image and / or the map to other modules of the image processing device 120 for further processing. For example, the obtaining module 410 can send the map to the map processing module 420 for projecting the content of the map onto a first imaging surface to form a first projected image. As another example, the obtaining module 410 can send the image to the image processing module 430 for projecting the content of the image onto a second imaging surface to form a second projected image. As another example, the obtaining module 410 can send the image and / or the map to the storage module 450 to store it in a local database or a remote database.

[0077] The map processing module 420 can be configured to project the content of the map onto a first imaging surface to form a first projected image. The first projected image on the first imaging surface can be regarded as a representation of the map. In some embodiments, the map processing module 420 can generate the first projected image according to a first projection relationship associated with the map. The first projection relationship associated with the map can indicate the point-to-point relationship between the points (or pixels) of the map and the points (or pixels) of the first projected image. In some embodiments, the map processing module 420 can determine the first projection relationship according to the display parameters of the map. The display parameters of the map can include the first rotation angle of the map, the pitch angle of the map, the zoom ratio of the map, the vertical displacement of the map, etc.

[0078] The image processing module 430 can be configured to project the content of the image onto a second imaging surface to form a second projected image. The second projected image on the second imaging surface can be regarded as a representation of the image. In some embodiments, the second imaging surface can include a cylindrical surface. In some embodiments, the image processing module 430 can generate the second projected image according to a second projection relationship associated with the image. The second projection relationship associated with the image can indicate the point-to-point relationship between the points (or pixels) of the image and the points (or pixels) of the second projected image. In some embodiments, the image processing module 430 can determine the second projection relationship according to the display parameters of the image. The display parameters of the image can include the second rotation angle of the image, the pitch angle of the image, the zoom ratio of the image, the vertical displacement of the image, the radius of the cylinder, etc.

[0079] The display module 440 can be configured to co-display at least one image and the map. The co-display of the at least one image and the map indicates that the content of the at least one image and the content of the map can be displayed simultaneously. In some embodiments, the display module 440 can include different imaging surfaces. The content of the at least one image and the content of the map can be separately displayed on different imaging surfaces. Exemplary imaging surfaces can include a planar imaging surface, a cylindrical imaging surface, a spherical imaging surface, etc. or a combination thereof.

[0080] In some embodiments, in the co-display, the representation of the at least one image on the cylindrical imaging surface can match the representation of the map on the planar imaging surface. For example, the position information of the scene included in the at least one image can be indicated by the representation of the map in the co-display.

[0081] The storage module 450 can be configured to store data, instructions, and / or any other information related to image display. For example, the storage module 450 can store the image, the map, the display parameters of the image, the display parameters of the map, etc. or any combination thereof. In some embodiments, the storage module 450 can store data obtained from other components of the image processing device 120 and / or the camera 110. In some embodiments, the storage module 450 can store data and / or instructions that the image processing device 120 can execute or use to implement the exemplary methods described in this application. In some embodiments, the storage module 450 can include a mass storage device, a removable storage device, a volatile read-write memory, a read-only memory (ROM), etc. or a combination thereof. In some embodiments, as described elsewhere in this application, the storage module 450 can be implemented on a cloud platform. In some embodiments, the storage module 450 can be connected and / or communicate with one or more other components in the image processing device 120. One or more components in the image processing device 120 can access the data or instructions stored in the storage module 450.

[0082] It should be noted that the above description of the image processing device 120 is for illustrative purposes only and is not to be construed as the sole embodiment. Various changes and modifications can be made by those of ordinary skill in the art under the teachings of some embodiments of this application. However, these changes and modifications may not depart from the protection scope of some embodiments of this application.

[0083] Figure 5 is a flowchart of an exemplary process 500 for co-displaying a map and an image according to some embodiments of this application. In some embodiments, the process 500 can be implemented on the image display system 100 as shown in Figure 4 For example, the process 500 can be stored in a storage medium (e.g., the memory 227 of the storage device 150 or the computing device 200) in the form of instructions and invoked and / or executed by the image processing device 120. The operations in the process 500 presented below are for illustrative purposes. In some embodiments, the process 500 can be implemented with the aid of one or more additional operations not described and / or one or more operations not discussed. Additionally, the order of operations of the process 500 as shown in Figure 5 and described below is not intended to be limiting. It should be noted that although the co-display of a map and an image is described as an example, the co-display of a map and multiple images can be similarly implemented according to a similar process of the process 500.

[0084] In 502, an image processing device 120 (e.g., an acquisition module 410) may acquire an image captured by a camera from a shooting direction. The image may include an object in the environment around the camera. The object may be any object (e.g., a building, a tree, a street, a river, etc.) visible through the camera from the shooting direction. The shooting direction may refer to the direction in which the lens of the camera faces. In some embodiments, the image may be stored after being captured by the camera, and the image processing device 120 may acquire the image from a storage device (e.g., a storage device 150, a memory 227, or a memory 390). In some embodiments, after being captured by the camera, the image may be immediately sent to the image processing device 120 for processing.

[0085] In some embodiments, the camera may be one of at least two cameras 110 installed at different positions (e.g., at least two positions distributed substantially in a circle). Each of the at least two cameras 110 may correspond to a fixed shooting direction and capture an image from the fixed shooting direction. In some embodiments, the scenes included in two images captured by two adjacent cameras 110 may have an overlapping area. In order to better display the two images together, the overlapping area in the two images may be removed according to one or more algorithms.

[0086] In some embodiments, the camera may be a movable camera having at least two preset shooting directions. The camera may capture images from each preset shooting direction.

[0087] In 504, the image processing device 120 (e.g., the acquisition module 410) may acquire a map of the environment around the camera. The map may at least include the position corresponding to the object and the position corresponding to the camera.

[0088] For example, assuming that the object in the image is a landmark building in the environment around the camera, the representation of the landmark building in the map may be a certain area having a similar shape, a similar color, or a similar texture as the landmark building. The position corresponding to the object may be identified in this area.

[0089] As another example, assuming that the camera is difficult to be identified in the map, the position corresponding to the camera may be the position of a reference object related to the camera in the map. The reference object may be a structure on which the camera is installed, a road where the camera is located, etc. For example, assuming that the camera is installed on the roof of a white building, the position corresponding to the camera may be identified at the position of the representation of the white building in the map.

[0090] In some embodiments, the map may be a planar map showing the environment around the camera in a two-dimensional (2D) manner. Exemplary planar maps may include satellite maps, aerial maps, etc. or combinations thereof. The satellite maps may be collected by imaging satellites controlled by governments and enterprises around the world (e.g., Apple Maps, Google Maps). The aerial maps may be images taken by aircraft or other flying objects. Exemplary flying objects may include fixed-wing aircraft, helicopters, unmanned aerial vehicles (UAVs or "drones"), balloons, airships and airships, rockets, pigeons, kites, parachutes, independent telescopic telescopes, etc. or any combination thereof. In some embodiments, the map may be a three-dimensional (3D) stereogram showing the environment around the camera.

[0091] In some embodiments, information related to the map may be pre-stored in a memory (e.g., storage device 150, memory 227 of computing device 228, memory 390), and the image processing device 120 may obtain the map from the storage device. A detailed description of the information related to the map can be found elsewhere in this application (e.g., in Figure 10 and its description).

[0092] In 506, the image processing device 120 (e.g., display module 440) may co-display the image and the map. The co-display of the image and the map may mean that the content of the image and the content of the map can be displayed simultaneously.

[0093] In some embodiments, the display module 440 may include different imaging surfaces. The content of the image and the content of the map may be displayed on different imaging surfaces respectively. Exemplary imaging surfaces may include planar imaging surfaces, cylindrical imaging surfaces, spherical imaging surfaces, etc. or combinations thereof. For example, the display module 440 may project the content of the map onto a planar imaging surface to form a representation of the map (e.g., a first projection image), and project the content of the image onto a cylindrical imaging surface to form a representation of the image (e.g., a second projection image).

[0094] In the co - display, the representation of the image on the cylindrical imaging surface may match the representation of the map on the planar imaging surface. For example, when the representation of the image is displayed on the cylindrical imaging surface, the position information of the scene included in the image can be simultaneously shown through the map. Specifically, the representation of the image can be aligned with the position corresponding to the target and the position corresponding to the map, which means that the representation of the image on the cylindrical imaging surface can be orthogonal or substantially orthogonal to the line connecting the position corresponding to the target and the position corresponding to the map. As used herein, a surface being substantially orthogonal to a line can mean that the line and the surface can form an included angle that varies from 90 degrees. The amount of variation can be a value between 0 degrees and 30 degrees, such as 1 degree, 2 degrees, 5 degrees, 8 degrees, 10 degrees, 15 degrees, 20 degrees, 25 degrees, 28 degrees, etc. In some embodiments, when facing the representation of the image on the cylindrical imaging surface, the user can easily identify the position of the scene in the image included in the map because it is approximately the same as the user's perspective. A detailed description of the display of the image and the map can be found elsewhere in this application (e.g., in Figure 6 and its description).

[0095] It should be noted that the above description of process 500 is provided for illustrative purposes only and is not intended to be understood as the sole embodiment. Various changes and modifications can be made by those of ordinary skill in the art under the teachings of some embodiments of this application. In some embodiments, some operations can be reduced or added. However, these changes and modifications may not deviate from the protection scope of some embodiments of this application. For example, one or more other optional operations (e.g., a storage operation) can be added to the exemplary process 500. In the storage operation, the image processing device 120 can store one or more intermediate results in any storage device disclosed elsewhere in this application. As another example, at least two images captured by at least two cameras 110 can be displayed similarly to the map in a manner similar to process 500.

[0096] Figure 6 is a flowchart of an exemplary process 600 for co - displaying a map and an image as shown in some embodiments of this application. In some embodiments, as Figure 1 shown, process 600 can be implemented on the image display system 100. For example, process 600 can be stored in a storage medium (e.g., storage device 150 or memory 227) in the form of instructions and called and / or executed by the image processing device 120. The operations in process 600 shown below are for illustrative purposes. In some embodiments, process 600 can be completed with one or more additional operations not described and / or one or more operations not discussed. Additionally, as Figure 6The order of operations in process 600 as shown and described below is not intended to be restrictive. In some embodiments, operation 506 may be performed according to process 600.

[0097] In 602, the image processing device 120 (e.g., the map processing module 420) may project the content of the map onto a first imaging surface to form a first projected image. The first projected image on the first imaging surface may be regarded as a representation of the map.

[0098] For illustrative purposes, the first imaging surface may be a plane, and the first projected image may be a 2D image. The image processing device 120 may generate the first projected image according to a first projection relationship associated with the map. The first projection relationship associated with the map may indicate a point-to-point relationship between the points (or pixels) of the map and the points (or pixels) of the first projected image. In some embodiments, the first projection relationship may be represented as:

[0099] x map =y output *zoom*cos(pitch)*sin(yaw)+x output *zoom*cos(yaw) (1),

[0100] y map =y output *zoom*cos(pitch)*cos(yaw)-x output *zoom*sin(yaw)+ty map (2),

[0101] where, (x map , y map ) represents the coordinates of a pixel in the map, (x output , y output ) represents the coordinates of the corresponding pixel in the first projected image, yaw represents the first rotation angle of the map (also known as the heading angle of the map), pitch represents the pitch angle of the map, zoom represents the zoom ratio of the map, ty mapRepresents the vertical displacement coefficient of the map. The first rotation angle of the map may be related to the rotation angle of the map on the planar imaging surface relative to an axis passing through the center of the map and orthogonal to the planar imaging surface. With different first rotation angles, the map on the planar imaging surface can be oriented in different directions. For example, the map can be a north-facing map with north at the top of the map, or a south-facing map with south at the top of the map. The pitch angle of the map may be related to the angle between the observer's perspective and the plane of the planar imaging surface. The scaling ratio of the map may be a first ratio associated with the scaling effect of the map on the planar imaging surface. Using different map scaling ratios, the observer can view the map on the planar imaging surface with different FOVs. For example, the operation of magnifying the map on the planar imaging surface (e.g., by the user) can be achieved by increasing the value of the map's scaling ratio. The operation of reducing the map on the planar imaging surface (e.g., by the user) can be achieved by decreasing the value of the map's scaling ratio. The vertical displacement coefficient of the map may be related to the amount of displacement of the map in the vertical direction. For example, in the case where the user is looking down at the first projection image, a more careful observation of the first projection image can be achieved by increasing the vertical displacement coefficient of the map.

[0102] Based on equations (1) and (2), the image processing device 120 can project the map onto the planar imaging surface to form the first projection image. For simplicity, the first rotation angle of the map, the pitch angle of the map, the scaling ratio of the map, and the vertical displacement of the map may be referred to as the display parameters of the map. In some embodiments, the image processing device 120 can retrieve the information of the map (e.g., the coordinates of the pixels in the map) from a memory (e.g., storage device 150 or memory 227), and calculate the information of the first projection image on the planar imaging surface (e.g., the coordinates of the pixels in the first projection image) based on the information of the map and the display parameters of the map as described above. In some embodiments, the image processing device 120 can adjust one or more display parameters of the map according to a user command. For example, if the user sends a command to display more detailed structural information of the map on the imaging plane, the image processing device 120 can increase the value of the map's scaling ratio. If the user sends a command to change the direction of the map on the planar imaging surface, the image processing device 120 can adjust the first rotation angle of the map. In some embodiments, the user command can be sent by the user through the I / O interface of the image processing device 120. For example, the user command can be executed by a mouse, the scrolling operation of a mouse wheel, the touch operation of a finger, etc. or a combination thereof.

[0103] In some embodiments, the point-to-point relationship between the points (or pixels) of the map and the points (or pixels) of the first projection image can be calculated in advance and stored in a storage device (e.g., storage module 450) in the form of a mapping table. When needed, the image processing device 120 can retrieve the mapping table from the storage device and generate the first projection image based on the mapping table. Alternatively, as described in operation 504, after the map is obtained by the obtaining module 410, the image processing device 120 can project the content of the map onto the planar imaging surface to form the first projection image.

[0104] In 604, the image processing device 120 (e.g., image processing module 430) can project the content of the image onto a second imaging surface to form a second projection image. The second projection image on the second imaging surface can be regarded as a representation of the image.

[0105] For illustrative purposes, the second imaging surface can include a cylindrical surface. The image processing device 120 can generate the second projection image according to a second projection relationship associated with the image. The second projection relationship associated with the image can indicate the point-to-point relationship between the points (or pixels) of the image and the points (or pixels) of the second projection image.

[0106] In some embodiments, the second projection relationship can indicate the projection of each point (or pixel) in the image onto world coordinates. The world coordinates of the pixels in the image can be represented as:

[0107]

[0108]

[0109]

[0110] where (x input , y input ) represents the coordinates of the pixel in the image, (x w , y w , z w ) represents the coordinates of the pixel in world coordinates, h represents the height of the image, w represents the width of the image, yaw pano represents the second rotation angle of the image (also referred to as the heading angle of the image), pitch pano represents the pitch angle of the image, zoom pano represents the zoom ratio of the image, ty pano represents the vertical displacement coefficient of the image, r panoRepresents the radius of a cylinder associated with the second projection image (i.e., the cylinder forms an imaging surface). The second rotation angle of the image may be related to the rotation angle of the image (i.e., the second projection image) along the axis of the cylinder on the cylindrical imaging surface. In some embodiments, the second projection image may be displayed on a region of the cylindrical imaging surface, and the second rotation angle may affect the position of the displayed region on the cylindrical imaging surface. The pitch angle of the image may be related to the angle between the observer's viewing angle and the plane of the planar imaging surface. The scaling ratio of the image may be a second ratio associated with the scaling effect of the image on the cylindrical imaging surface. By using different image scaling ratios, the observer can view the image on the cylindrical imaging surface with different FOVs. For example, the operation of magnifying the image on the cylindrical imaging surface (e.g., by the user) can be achieved by increasing the value of the image scaling ratio. The operation of reducing the image on the cylindrical imaging surface (e.g., by the user) can be achieved by decreasing the value of the image scaling ratio. The vertical displacement coefficient of the image may be related to the amount of displacement of the image in the vertical direction. For example, in the case where the user is looking down at the second projection image, a more careful observation of the second projection image can be achieved by increasing the vertical displacement coefficient of the image.

[0111] The image processing device 120 may further determine the second projection image based on the world coordinates of the points (or pixels) of the image. The pixels in the second projection image may be represented as:

[0112]

[0113]

[0114] where (x output , y output ) represents the coordinates of the corresponding pixel in the second projection image.

[0115] Based on equations (3) to (7), the image processing device 120 can project the image onto the cylindrical imaging surface to form a second projected image. For the sake of brevity, the second rotation angle of the image, the pitch angle of the image, the scaling ratio of the image, the vertical displacement amount of the image, and the radius of the cylinder can be referred to as the display parameters of the image. In some embodiments, the image processing device 120 can retrieve information of the image (e.g., the coordinates of pixels in the image) from a memory (e.g., the storage device 150 or the memory 227), and calculate information of the second projected image on the cylindrical imaging surface image (e.g., the coordinates of pixels in the second projected image) based on the information of the image and the display parameters of the image described above. In some embodiments, the image processing device 120 can adjust one or more display parameters of the image according to a user command. For example, if the user sends a command to display more detailed structural information of the image on the cylindrical imaging surface, the image processing device 120 can increase the value of the scaling ratio of the image. If the user sends a command to change the display area of the image on the cylindrical imaging surface, the image processing device 120 can adjust the second rotation angle of the image. In some embodiments, the user command can be sent by the user through the I / O interface of the image processing device 120. For example, the user command can be executed through a click operation of a mouse, a scroll operation of a mouse wheel, a touch operation of a finger, etc. or a combination thereof.

[0116] In some embodiments, similar to the projection of the image onto the cylindrical imaging surface, the image processing device 120 can project multiple images simultaneously to generate multiple second projected images on the cylindrical imaging surface, and if necessary, splice all the second projected images together to form a combined projected image. As described elsewhere in this application, the multiple images can be captured by at least two cameras (e.g., the at least two cameras 110) installed at different positions at different shooting angles. Alternatively, the image processing device 120 can, for example, according to user input, project only a part of the multiple images captured by the multiple cameras simultaneously at a time.

[0117] In 606, the image processing device 120 (e.g., the display module 440) can co-display the first projected image and the second projected image.

[0118] In some embodiments, the image processing device 120 may coordinate the display parameters of the map with the display parameters of the image such that the second projection image is aligned with the map described in operation 506. For example, in a case where the map is a north-oriented map (i.e., a map with the north facing the top of the map) and the camera that captures the image has a shooting direction pointing north, the first rotation angle of the map and the second rotation angle of the image may be set to have the same value, whereby the second projection image may be aligned with the north direction in the map. In a case where the map is a north-oriented map and the camera that captures the image has a shooting direction pointing east, the first rotation angle of the map and the second rotation angle of the image may be set to have a 90-degree difference, whereby the second projection image may be aligned with the east direction in the map.

[0119] In some embodiments, the image processing device 120 may co-rotate and / or scale the first projection image and the second projection image according to a user's instruction to perform a rotation operation and / or a scaling operation on the co-display. For example, the image processing device 120 may co-rotate the first projection image and the second projection image by changing the values of the first rotation angle of the map and the second rotation angle of the image. A detailed description of the rotation of the co-display can be found elsewhere in this application (e.g., Figure 7 and its description). For another example, the image processing device 120 may co-enlarge and co-reduce the first projection image and the second projection image by changing the values of the first scale associated with the map and the second scale associated with the image. A detailed description of the scaling effect of the co-display can be found elsewhere in this application (e.g., in Figure 8 and its description).

[0120] In some embodiments, the image processing device 120 projects a plurality of images to form a plurality of second projection images and co-displays the first projection image and the plurality of second projection images. It should be noted that due to different shooting angles associated with the plurality of images, each of the plurality of images may have a different second rotation angle. For the sake of simplicity, N panoramic images captured by N cameras may be taken as an example. If the N cameras are evenly distributed along a circle, the second rotation angles of each panoramic image may also be evenly distributed within the range of 0 to 360 degrees. The angular interval between every two adjacent second rotation angles of two panoramic images may be set to 360 / N degrees. By doing so, the image processing device 120 may project each of the N panoramic images onto a unique part of a second imaging surface (e.g., a cylindrical surface) corresponding to a unique heading angle. For example, as Figure 9As shown, eight cameras can capture eight panoramic images at a time. The image processing device 120 can set the second rotation angles of the eight panoramic images to 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°, 270°, 315° respectively to form a pseudo 3D on the cylindrical surface (as Figure 11 shown). It should be noted that the second rotation angles of the eight panoramic images can be adjusted according to different application scenarios. Specifically, the second rotation angle can be set to any angle within the range of 0 to 360 degrees according to user input.

[0121] It should be noted that the above description of process 600 is provided for illustrative purposes only and is not intended to be understood as the sole embodiment. Various changes and modifications can be made by those of ordinary skill in the art under the guidance of some embodiments of this application. In some embodiments, some operations can be reduced or added. However, these changes and modifications may not deviate from the protection scope of some embodiments of this application. For example, one or more other optional operations (such as a storage operation) can be added to the exemplary process 600. In the storage operation, the image processing device 120 can store the first projection image in any storage device (such as storage device 150 or memory 227) disclosed elsewhere in this application.

[0122] Figure 7 is a flowchart of an exemplary process 700 for adjusting the common display direction shown in some embodiments of this application. In some embodiments, process 700 can be implemented on the image display system 100 as Figure 1 shown. For example, process 700 can be stored in a storage medium (such as storage device 150 or memory 227) in the form of instructions and called and / or executed by the image processing device 120. The operations in process 700 shown below are for illustrative purposes. In some embodiments, process 700 can be completed with the help of one or more additional operations not described and / or one or more operations not discussed. Additionally, as Figure 7 shown and described below, the operation sequence of process 700 may not be restrictive.

[0123] In 702, the image processing device 120 (such as display module 440) can receive an instruction to adjust the common display direction of the image and the map.

[0124] In some embodiments, the image processing device 120 may receive the instructions from a mouse, a keyboard, a touch screen, a microphone, etc., or any combination thereof. For example, the user may drag the co-display presented on the display device by a mouse, or click a button on the keyboard to rotate the co-display. The co-display direction to be adjusted may be determined by, for example, the direction of mouse movement, the travel distance of the mouse, the duration of the button being in the pressed state, the acceleration of the button being released or pressed, etc., or a combination thereof.

[0125] As another example, the instructions may be in the form of voice commands. The user may give a voice command "switch to north". The image processing device 120 may parse and convert the command into a computer-recognizable instruction to change the co-display of the image and the map, such that the image associated with the shooting direction pointing north may appear at a preset position on the second imaging surface. Or, the user may give a voice command "rotate". When receiving the command, the image processing device 120 may generate an instruction to rotate the co-display at a preset angular velocity until another instruction to stop rotating is received. The preset angular velocity may be the default setting of the image display system 100, or may be adjustable under different circumstances. For example, the preset angular velocity may be adjusted according to the input of the user's click operation on the mouse, the scrolling operation of the mouse wheel, etc., or any combination thereof.

[0126] In 704, the image processing device 120 (e.g., the display module 440) may adjust a first rotation angle associated with the direction of the map on the first imaging surface and a second rotation angle associated with the representation of the image on the second imaging surface. To achieve the adjustment, the image processing device 120 may determine an offset rotation angle based on the instructions. In addition, the image processing device 120 may calculate the target rotation angle of the map by adding the offset rotation angle to the current first rotation angle of the map, and calculate the target rotation angle of the image by adding the offset rotation information to the current second rotation angle of the image.

[0127] In 706, the image processing device 120 (e.g., the display module 440) may update the co-display of the image and the map according to the adjusted first rotation angle and the adjusted second rotation angle. In some embodiments, the image processing device 120 may specify the target rotation angle of the map as the adjusted first rotation angle, and specify the target rotation angle of the image as the adjusted second rotation angle. Then, the image processing device 120 may update the first projection image projected by the map according to the first projection relationship under the adjusted first rotation angle of the map, and update the second projection image projected by the image according to the second projection relationship under the adjusted second rotation angle of the image.

[0128] It should be noted that the above description of process 700 is provided for illustrative purposes only and is not intended to be understood as the sole embodiment. Various changes and modifications can be made by those of ordinary skill in the art under the teachings of some embodiments of the present application. In some embodiments, some operations can be reduced or added. However, these changes and modifications may not depart from the protection scope of some embodiments of the present application.

[0129] Figure 8 is a flowchart of an exemplary process 800 for adjusting the field of view of a common display as shown in some embodiments of the present application. In some embodiments, process 800 can be implemented on an image display system 100 as shown in Figure 1 FIG. For example, process 800 can be stored in a storage medium (e.g., storage device 150 or memory 227) in the form of instructions and called and / or executed by image processing device 120. The operations in process 800 shown below are for illustrative purposes. In some embodiments, process 800 can be completed with the aid of one or more additional operations not described and / or one or more operations not discussed. Additionally, as Figure 8 shown and described below, the order of operations in process 800 may not be intended to be restrictive.

[0130] In 802, the image processing device 120 (e.g., display module 440) can receive an instruction to adjust the FOV in the common display of an image and a map. The FOV in the common display of the image and the map can depend on the zoom effect of the image and the map. Similar to operation 702, the image processing device 120 can receive the instruction from a mouse, keyboard, touch screen, microphone, etc. or any combination thereof. For example, the user can use a preset zoom gesture (e.g., a multi-touch gesture) stored in one or more storage devices (e.g., storage device 150) of the image display system 100 to zoom in or out (i.e., expand or contract the field of view) the common display presented on the display device via the touch screen, or click a button on the keyboard to zoom in or out the common display. Exemplary zoom gestures can include moving two fingers away from or closer to each other on the touch screen.

[0131] In some embodiments, the adjustment of the FOV can be completed according to the center point. During the adjustment of the FOV, the position of the center point can remain unchanged on the first / second imaging surface. In some embodiments, the image processing device 120 can determine the center of the second projection image as the center point. In some embodiments, the image processing device 120 can determine a target based on the instruction and use the point (or pixel) related to the target as the center point.

[0132] In 804, the image processing device 120 (e.g., the display module 440) may adjust a first ratio associated with the zoom effect of the map and a second ratio associated with the zoom effect of the image. In some embodiments, the image processing device 120 may determine an offset ratio based on the instruction. Additionally, the image processing device 120 may calculate the target ratio of the map by, for example, adding the offset ratio to the first ratio, and calculate the target ratio of the image by, for example, adding the offset ratio to the second ratio.

[0133] In 806, the image processing device 120 (e.g., the display module 440) may update the co - display of the image and the map according to the adjusted first ratio and the adjusted second ratio. In some embodiments, the image processing device 120 may specify the target ratio of the map as the adjusted first ratio and the target ratio of the image as the adjusted second ratio. Then, the image processing device 120 may update the first projection image projected from the map according to the first projection relationship under the adjusted first ratio, and update the second projection image projected from the image according to the second projection relationship under the adjusted second ratio.

[0134] It should be noted that the above description of the process 700 is for illustrative purposes only and is not intended to be understood as the sole embodiment. For those of ordinary skill in the art, various changes and modifications can be made under the guidance of some embodiments of the present application. In some embodiments, some operations may be reduced or added. However, these changes and modifications may not deviate from the protection scope of some embodiments of the present application. For example, in addition to simultaneously zooming in or out the mapping of the map and the co - display of the image, the image processing device 120 may also separately control the zoom effects of the map and the image. For example, the image processing device 120 may only perform the operation of zooming in or out the second projection image on the second imaging surface, keeping the appearance of the first projection image projected from the map on the first imaging surface unchanged.

[0135] Figure 9 is an exemplary camera arrangement according to some embodiments of the present application. As Figure 9 shown, eight cameras (110 - 1, 110 - 2, 110 - 3, 110 - 4, 110 - 5, 110 - 6, 110 - 7, and 110 - 8) are installed on the roof of the white building. The eight cameras may be evenly arranged along a circle centered at the center of the white building. Each of the eight cameras has a unique shooting direction. For example, the shooting direction of camera 110 - 8 is towards the north (i.e., the top of the map). One or more images captured by the eight cameras may be projected onto a cylindrical imaging surface, while the map (e.g., Figure 10The map 1000 shown in [description] is projected onto a planar imaging surface to form a common display of the map and one or more images captured by the eight cameras.

[0136] Figure 10 Exemplary embodiments according to the present application illustrate Figure 9 An exemplary map 1000 of the surroundings of the white building shown. As Figure 10 shown, the white building is located at the center of the map 1000. The map depicts the surroundings of the white building and the eight cameras that are too small to be recognized. The map can depict the surroundings of the building within a predetermined distance. The predetermined distance can be the default setting of the image display system 100 or can be adjustable in different situations. For example, the predetermined distance can be adjusted according to user input. In some embodiments, the distance can be 0.5 km, 1 km, 1.5 km, 2 km, 5 km, 10 km, 50 km, 100 km, etc.

[0137] As described elsewhere in the present application, the map can be a satellite map or an aerial map. The satellite map can be collected by imaging satellites operated by governments and enterprises around the world (e.g., Apple Maps, Google Maps). The aerial map can be an image taken by an airplane or other flying object. Exemplary flying objects can include fixed-wing aircraft, helicopters, unmanned aerial vehicles (UAVs or "drones"), balloons, airships and airships, rockets, pigeons, kites, parachutes, independent telescopic telescopes, etc. or any combination thereof.

[0138] Figure 11 is a common display of exemplary images and maps according to some embodiments of the present application. The images can be taken separately by Figure 9 the eight cameras shown in [description]. As Figure 11 shown, eight projected images 1101, 1102, 1103, 1104, 1105, 1106, 1107, and 1108 are displayed on a cylindrical imaging surface. Each of the eight projected images can be generated by projecting the image captured by one of the eight cameras onto the cylindrical imaging surface. The map 1100, which can be regarded as the background, can be Figure 10 a projected image of the map 1000 shown in [description].

[0139] The basic concepts have been described above. Obviously, for those of ordinary skill in the art after reading this application, the above invention disclosure is only an example and does not constitute a limitation to the present application. Although not explicitly stated here, those of ordinary skill in the art may make various modifications, improvements, and corrections to the present application. Such modifications, improvements, and corrections are proposed in this application, so such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of the present application.

[0140] Meanwhile, this application uses specific terms to describe the embodiments of this application. For example, "one embodiment", "an embodiment", and "some embodiments" mean a certain feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment" or "one embodiment" or "an alternative embodiment" mentioned two or more times in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0141] In addition, those of ordinary skill in the art can understand that various aspects of this application can be illustrated and described by several patentable types or situations, including any new and useful process, machine, product, or composition of matter, or any new and useful improvement thereof. Therefore, various aspects of this application can be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of software and hardware, which are generally referred to as "modules", "units", "components", "devices", or "systems" in this article. In addition, various aspects of this application can take the form of a computer program product contained in one or more computer-readable media, having computer-readable program code thereon in the form of computer-readable program code.

[0142] A computer-readable signal medium may include a propagated data signal containing computer program code, for example, on a baseband or as part of a carrier wave. Such propagated signals can have various forms, including electromagnetic form, optical form, etc., or any suitable combination. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to communicate, propagate, or transmit a program for use. The program code located on the computer-readable signal medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, RF, etc., or any combination of the above media.

[0143] Computer program code for operating aspects of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., traditional procedural programming languages such as the "C" programming language, VisualBasic, Fortran2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages such as Python, Ruby, and Groovy, or other programming languages. The program code can run entirely on the user's computer, or as a stand-alone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can establish a connection with an external computer (e.g., by using the network of a network service provider) or be provided as a service in a cloud computing environment or as software as a service (SaaS).

[0144] In addition, unless clearly stated in the claims, the order of the processing elements and sequences, the use of numerical letters, or the use of other names in the present application are not used to limit the order of the processes and methods of the present application. Although some currently useful embodiments of the invention are discussed through various examples in the above disclosure, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that conform to the essence and scope of the embodiments of the present application. For example, although the implementation of the above various components can be embodied in a hardware device, it can also be implemented as a pure software solution, for example, an installation on an existing server or mobile device.

[0145] Similarly, it should be noted that, in order to simplify the description of the present application disclosure and thus help the understanding of one or more embodiments of the invention, in the previous description of the embodiments of the present application, multiple features are sometimes merged into one embodiment, drawing, or description thereof. However, this method of the present application should not be construed as reflecting an intention that the claimed subject matter requires more features than those expressly recited in each claim. In fact, the features of the embodiments are fewer than all the features of the individual embodiments disclosed above.

Claims

1. A system, comprising: at least one storage medium containing an instruction set; and at least one processor communicatively coupled to the at least one storage medium, wherein when the instruction set is executed, the at least one processor is directed to cause the system to perform the following operations: Obtain an image captured by a camera, the image including a target in the environment surrounding the camera; Obtain a map of the surrounding environment, the map including at least a location corresponding to the target and a location corresponding to the camera; Project the content of the map onto a first imaging surface to form a representation of the map, the first imaging surface being a plane; Project the content of the image onto a second imaging surface to form a representation of the image, the second imaging surface including a cylindrical surface; and Co-display the representation of the image and the representation of the map, wherein the representation of the image is aligned with the location corresponding to the target and the location corresponding to the camera in the map.

2. The system according to claim 1, wherein The image captured by the camera is panoramic.

3. The system according to claim 1, wherein The representation of the map is formed according to a first projection relationship associated with the map, and the representation of the image is formed according to a second projection relationship associated with the image.

4. The system according to claim 3, wherein The first projection relationship includes a first rotation angle associated with the orientation of the map.

5. The system according to claim 4, wherein The second projection relationship includes a second rotation angle associated with the representation of the image on the second imaging surface.

6. The system according to claim 5, wherein, The at least one processor is further directed to cause the system to perform the following operations: Receive an instruction to adjust the co-display orientation of the image and the map; Adjust the first rotation angle associated with the orientation of the map and the second rotation angle associated with the representation of the image on the second imaging surface; and Update the co-display of the image and the map according to the adjusted first rotation angle and the adjusted second rotation angle.

7. The system according to claim 3, wherein The first projection relationship includes a first scale associated with the scaling effect of the map.

8. The system according to claim 7, wherein, The second projection relationship includes a second scale associated with the scaling effect of the image.

9. The system according to claim 8, wherein, The at least one processor is further directed to cause the system to perform the following operations: Receive an instruction to adjust the field of view in the co-display of the image and the map; Adjust the first scale associated with the scaling effect of the map and the second scale associated with the scaling effect of the image; and Update the co-display of the image and the map according to the adjusted first scale and the adjusted second scale.

10. A method implemented on a device having at least one storage medium and at least one processor, the at least one storage medium containing an instruction set, the at least one processor communicatively coupled to the at least one storage medium, the method comprising: Obtain an image captured by a camera, the image including a target in the environment surrounding the camera; Obtain a map of the surrounding environment, the map including at least a location corresponding to the target and a location corresponding to the camera; Project the content of the map onto a first imaging surface to form a representation of the map, the first imaging surface being a plane; Project the content of the image onto a second imaging surface to form a representation of the image, the second imaging surface including a cylindrical surface; and Co-display the representation of the image and the representation of the map, wherein the representation of the image is aligned with the position corresponding to the target and the position corresponding to the camera in the map.

11. The method according to claim 10, wherein, The image captured by the camera is panoramic.

12. The method according to claim 10, wherein The representation of the map is formed according to a first projection relationship associated with the map, and the representation of the image is formed according to a second projection relationship associated with the image.

13. The method according to claim 12, wherein, The first projection relationship includes a first rotation angle associated with the orientation of the map.

14. The method according to claim 13, wherein, The second projection relationship includes a second rotation angle associated with the representation of the image on the second imaging surface.

15. The method according to claim 14, wherein, The method further includes: Receiving an instruction to adjust the co-display direction of the image and the map; Adjusting the first rotation angle associated with the orientation of the map and the second rotation angle associated with the representation of the image on the second imaging surface; and Updating the co-display of the image and the map according to the adjusted first rotation angle and the adjusted second rotation angle.

16. The method according to claim 12, wherein, The first projection relationship includes a first ratio associated with the scaling effect of the map.

17. The method according to claim 16, wherein, The second projection relationship includes a second ratio associated with the scaling effect of the image.

18. The method according to claim 17, wherein, The method further includes: Receiving an instruction to adjust the field of view in the co-display of the image and the map; Adjusting the first ratio associated with the scaling effect of the map and the second ratio associated with the scaling effect of the image; and Updating the co-display of the image and the map according to the adjusted first ratio and the adjusted second ratio.

Images