Space image display method and device and flat panel display equipment

Abstract

The invention provides a space image display method and device for a flat panel display device and the flat panel display device, and the method comprises the steps: obtaining a space image obtained through the shooting of a target scene by a binocular camera, and constructing a three-dimensional model of the target scene according to the space image; setting a virtual camera according to a preset initial pose; determining a first area matched with the visual angle of the virtual camera in the three-dimensional model according to the initial pose; and obtaining a first plane image according to the first area, and controlling the plane display device to render the first plane image.

Description

Technical Field

[0001] This disclosure relates to the field of flat panel display technology, and more specifically, to a spatial image display method, apparatus, and flat panel display device for use in a flat panel display device. Background Technology

[0002] With the rapid development of augmented reality (AR) technology, spatial photography and video playback devices are becoming increasingly popular, propelling spatial photography technology from the professional field to the mass consumer market and becoming an important trend in the development of current imaging technology. Compared to traditional monocular photography, spatial photography relies on the principle of binocular parallax to capture depth information of the shooting scene and rich content from different viewing angles, bringing users a more immersive and realistic imaging experience, making it a more advanced form of shooting.

[0003] Currently, the main medium for ordinary users to take spatial photos is the smartphone, and mainstream mobile phone brands have launched spatial video shooting functions that rely on dual cameras to capture parallax images.

[0004] However, existing mobile phones only select the image captured by one of the dual cameras as the preview content during and after shooting. This preview image is essentially a traditional 2D image and cannot present the depth information and multi-view content unique to spatial shooting. This means that users cannot intuitively judge whether the spatial shooting effect meets expectations on the shooting site. They can only rely on professional binocular display devices (such as AR glasses) to verify the effect after shooting. If the effect is not good, it is necessary to reshoot, which seriously affects shooting efficiency and user experience. Summary of the Invention

[0005] One object of this disclosure is to provide a new technical solution for displaying spatial images via a flat panel display device.

[0006] According to a first aspect of the present disclosure, a method for displaying spatial images in a flat panel display device is provided, comprising: Acquire spatial images of the target scene captured by a binocular camera, and construct a three-dimensional model of the target scene based on the spatial images; The virtual camera is set according to the preset initial pose; Based on the initial pose, determine the first region in the 3D model that matches the viewpoint of the virtual camera; A first planar image is obtained based on the first region, and the planar display device is controlled to render the first planar image.

[0007] Optionally, constructing a 3D model of the target scene based on the spatial imagery includes: Determine the point cloud data of the common viewing area of ​​the spatial image; The target scene is reconstructed in three dimensions based on the point cloud data to obtain a three-dimensional mesh of the target scene. The three-dimensional mesh is textured based on the spatial image to obtain the three-dimensional model with texture information.

[0008] Optionally, the method further includes: Acquire the inertial data of the flat panel display device; The initial pose is adjusted based on the inertial data to obtain the target pose of the virtual camera; Based on the target pose, a second region in the 3D model that matches the viewpoint of the virtual camera is redefined; A second planar image is obtained based on the second region, and the planar display device is controlled to render the second planar image.

[0009] Optionally, adjusting the initial pose based on the inertial data to obtain the target pose of the virtual camera includes: The first pose adjustment amount of the flat panel display device is determined based on the inertial data; The second pose adjustment amount of the virtual camera is determined based on the first pose adjustment amount; The target pose is obtained based on the initial pose and the second pose adjustment amount.

[0010] Optionally, the initial pose indicates that the virtual camera is located in the middle of the stereo camera and its pose is the same as that of the stereo camera.

[0011] Optionally, the spatial image is a spatial video or spatial image obtained by the binocular camera capturing the target scene.

[0012] Optionally, the spatial image is a spatial video, and the first planar image includes a first planar video frame that corresponds one-to-one with each spatial video frame in the spatial video. The method further includes: The first planar video frame in the first planar image that corresponds to the first spatial video frame of the spatial video is determined as the target planar video frame; The target planar video frame is stored as a thumbnail of the spatial video data and corresponding to the spatial video data.

[0013] According to a second aspect of this disclosure, a spatial image display apparatus for a flat panel display device is provided, comprising: The model building module is used to acquire spatial images of the target scene captured by the binocular camera, and to build a three-dimensional model of the target scene based on the spatial images. The camera setting module is used to set up the virtual camera according to the preset initial pose; A region determination module is used to determine a first region in the 3D model that matches the viewpoint of the virtual camera based on the initial pose; The image rendering module is used to obtain a first planar image based on the first region and control the planar display device to render the first planar image.

[0014] According to a third aspect of this disclosure, a flat panel display device is provided, including a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the method as described in the first aspect of this disclosure under the control of the computer program.

[0015] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect of this disclosure.

[0016] Through the embodiments of this disclosure, spatial images can be converted into two-dimensional first planar images and displayed on a flat panel display device. This allows for a preview of the complete effect of the spatial images, and the viewing angle of the presented image closely matches the shooting range of the spatial images. Furthermore, spatial image effects can be superimposed on the flat panel display, allowing users to intuitively perceive the depth and all-round content of the shooting scene. This enables users to quickly decide whether to reshoot, effectively avoiding the waste of time and resources caused by repeated shooting, and improving shooting efficiency and user experience.

[0017] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0019] Figure 1 This is a block diagram illustrating the hardware configuration of a flat panel display device that can implement embodiments of the present disclosure; Figure 2 This is a flowchart of a spatial image display method for a flat panel display device according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram of a virtual camera according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the viewpoint of a virtual camera according to an embodiment of the present disclosure; Figure 5This is a schematic diagram of the viewpoint of a virtual camera according to another embodiment of the present disclosure; Figure 6 This is a block diagram of a spatial image display apparatus for a flat panel display device according to an embodiment of the present disclosure; Figure 7 This is a block diagram of a flat panel display device according to an embodiment of the present disclosure. Detailed Implementation

[0020] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0021] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0022] Techniques, methods, and apparatus known to those skilled in the art in the relevant field may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

[0023] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0025] <Hardware Configuration> Figure 1 This is a block diagram illustrating the hardware configuration of a flat panel display device 1000 that can implement embodiments of the present disclosure.

[0026] The flat panel display device 1000 can be a portable computer, desktop computer, mobile phone, tablet computer, etc. For example... Figure 1As shown, the flat panel display device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc. The processor 1100 may be a CPU, a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface, a headphone jack, etc. The communication device 1400 may be capable of wired or wireless communication, specifically including Wi-Fi communication, Bluetooth communication, 2G / 3G / 4G / 5G communication, etc. The display device 1500 may be, for example, an LCD screen, a touch screen, etc. The input device 1600 may include, for example, a touch screen, a keyboard, motion input, a camera, etc. Users can input / output voice information through the speaker 1700 and the microphone 1800.

[0027] Figure 1 The flat panel display device illustrated is merely illustrative and in no way intended to limit this disclosure, its application, or use. In embodiments applied to this disclosure, the memory 1200 of the flat panel display device 1000 is used to store instructions for controlling the processor 1100 to operate to perform any of the methods provided in the embodiments of this disclosure. Those skilled in the art will understand that, although in Figure 1 The flat panel display device 1000 illustrates multiple devices; however, this disclosure may relate only to a portion of these devices. For example, the flat panel display device 1000 may only relate to the processor 1100 and the memory 1200. Those skilled in the art can design instructions based on the schemes disclosed herein. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0028] <Method Implementation> This disclosure provides a spatial image display method for a flat panel display device, which can be implemented by the flat panel display device. Specifically, the spatial image display method for a flat panel display device can be implemented by, for example... Figure 1 The flat panel display device 1000 shown is implemented.

[0029] Figure 2 This is a flowchart of a spatial image display method for a flat panel display device according to an embodiment of the present disclosure.

[0030] like Figure 2 As shown, the method includes the following steps S2100 to S2400: Step S2100: Obtain spatial images of the target scene captured by the binocular camera, and construct a three-dimensional model of the target scene based on the spatial images.

[0031] In one embodiment, the binocular camera can be mounted on a flat panel display device that implements the method of this embodiment, and the flat panel display device can acquire spatial images of the target scene captured by the binocular camera.

[0032] In another embodiment, the binocular camera may also be mounted on other devices, and the other devices may transmit the spatial images of the target scene captured by the binocular camera to the flat panel display device implementing the method of this embodiment.

[0033] In this embodiment, the binocular cameras are arranged in parallel on a fixed horizontal structure, that is, the left and right cameras are on the same horizontal plane and arranged in parallel, and their postures are the same.

[0034] In some embodiments, spatial images are spatial images or videos obtained by a binocular camera capturing a target scene.

[0035] In embodiments where the spatial image is a spatial picture, the spatial picture may include a pair of matched left-eye and right-eye pictures.

[0036] In an embodiment where the spatial image is spatial video, each spatial video frame may include a pair of matched left-eye and right-eye images.

[0037] In some embodiments, constructing a three-dimensional model of a target scene based on spatial imagery includes: determining point cloud data of the common viewing area of ​​the spatial imagery; performing three-dimensional reconstruction of the target scene based on the point cloud data to obtain a three-dimensional mesh of the target scene; and performing texture mapping processing on the three-dimensional mesh based on the spatial imagery to obtain a three-dimensional model with texture information.

[0038] Specifically, point cloud data of the shared viewing area of ​​the spatial image can be determined based on the camera parameters of a stereo camera. These camera parameters can include intrinsic and / or extrinsic parameters.

[0039] This embodiment allows for the construction of a 3D model of the target scene, facilitating the subsequent rendering of a 2D planar image on a flat panel display device.

[0040] Step S2200: Set up the virtual camera according to the preset initial pose.

[0041] In this embodiment, the pose may include position and attitude, and the initial pose may be set in advance according to the application scenario or specific requirements.

[0042] Wherein, position represents the coordinates of the virtual camera's optical center in the world coordinate system, and attitude represents the rotation angle and direction of the virtual camera coordinate system relative to the world coordinate system.

[0043] In some embodiments, the initial pose indicates that the virtual camera is positioned between the stereo cameras and its pose is the same as that of the stereo cameras.

[0044] In this embodiment, the relative position and orientation of the virtual camera B and the stereo cameras A1 and A2 can be as follows: Figure 3 As shown.

[0045] Step S2300: Determine the first region in the 3D model that matches the viewpoint of the virtual camera based on the initial pose.

[0046] In this embodiment, the intrinsic parameters of the virtual camera can be determined based on the intrinsic parameters of the stereo camera, or they can be determined jointly based on the intrinsic parameters of the stereo camera and the initial pose of the virtual camera.

[0047] Specifically, the virtual camera's optical center can be used as the vertex of the view frustum. Along the virtual camera's optical axis, near and far clipping plane distances can be set to filter out model areas that are too close or too far. Combined with the virtual camera's horizontal and vertical viewing angles, the four lateral boundaries of the view frustum are determined, ultimately constructing the virtual camera's view frustum space. Then, the 3D model vertices are transformed from the world coordinate system to the virtual camera coordinate system to facilitate determining whether vertices are within the view frustum's range and to filter out the core vertices that constitute the display area. The filtered 3D visible area is the first region.

[0048] Step S2400: Obtain a first planar image based on the first region, and control the planar display device to render the first planar image.

[0049] In this embodiment, the first region of the three-dimensional model can be projected onto the image plane of the virtual camera to obtain a two-dimensional first planar image.

[0050] In an embodiment where the spatial image is a spatial image, the resulting first planar image can be a planar image.

[0051] In an embodiment where the spatial image is spatial video, the first planar image obtained can be a planar video, and the i-th video frame in the planar video can be obtained based on the first display area corresponding to the three-dimensional model and the i-th spatial video frame.

[0052] In one example where the spatial image is a spatial video, it can be arranged in chronological order, thereby obtaining the corresponding first planar video frame in the first planar video based on each spatial video frame of the spatial video.

[0053] Specifically, the process can involve constructing the i-th 3D model based on the i-th spatial video frame, determining the i-th first region within the i-th 3D model that matches the virtual camera's viewpoint, and obtaining the i-th first planar video frame based on the i-th first region. If i is less than N, i is incremented by 1, and the i-th first planar video frame is redefined. If i equals N, the N obtained first planar video frames are sorted according to time to obtain the first planar image. Here, the initial value of i is 1, the maximum value of i is N, and N is the number of spatial video frames in the spatial video.

[0054] In this embodiment, the flat panel display device can be controlled to render the first planar image after it has been obtained. Alternatively, the flat panel display device can be controlled to render the latest obtained planar video frame after a planar video frame of the first planar image has been obtained.

[0055] Through the embodiments of this disclosure, spatial images can be converted into two-dimensional first planar images and displayed on a flat panel display device. This allows for a preview of the complete effect of the spatial images, and the viewing angle of the presented image closely matches the shooting range of the spatial images. Furthermore, spatial image effects can be superimposed on the flat panel display, allowing users to intuitively perceive the depth and all-round content of the shooting scene. This enables users to quickly decide whether to reshoot, effectively avoiding the waste of time and resources caused by repeated shooting, and improving shooting efficiency and user experience.

[0056] Furthermore, the embodiments disclosed herein do not require any additional professional display equipment. Clear viewing of spatial images can be achieved solely through a single screen of a flat panel display device, allowing users to view them instantly after shooting and review them at any time in daily life. This greatly reduces the viewing threshold for spatial images and enhances the practicality and ease of dissemination of spatial shooting content.

[0057] Furthermore, the embodiments disclosed herein can be fully adapted to the existing computing power of screen display devices equipped with binocular cameras, enabling real-time preview of spatial effects during spatial image capture, eliminating the need to wait for completion before verifying the effects. Users can adjust parameters such as shooting angle and distance during the shooting phase through real-time preview, accurately capturing spatial scenes that meet expectations. This not only simplifies the spatial shooting process but also reduces the learning curve for users, allowing them to easily complete high-quality spatial photography.

[0058] In an embodiment where the spatial image is a spatial video obtained by a binocular camera capturing a target scene, the first planar image is a planar video, and the first planar image includes a first planar video frame that corresponds one-to-one with each spatial video frame in the spatial video. The method further includes: determining the first planar video frame in the first planar image that corresponds to the first spatial video frame in the spatial video, and using it as the target planar video frame; and storing the target planar video frame as a thumbnail of the spatial video data corresponding to the spatial video data.

[0059] This embodiment makes it easier for users to subsequently search for the captured spatial video data.

[0060] In some embodiments, the method further includes: acquiring inertial data of a flat panel display device; adjusting an initial pose based on the inertial data to obtain a target pose of a virtual camera; redetermining a second region in a 3D model that matches the viewpoint of the virtual camera based on the target pose; obtaining a second planar image based on the second region; and controlling the flat panel display device to render the second planar image.

[0061] In this embodiment, the inertial data includes angular velocity data and / or acceleration data.

[0062] The target pose and / or orientation are different from the initial pose, and the first and second regions of the 3D model are also different.

[0063] This embodiment can change the virtual camera's perspective by adjusting its pose.

[0064] In one example, the target pose differs from the initial pose in both position and orientation, and the virtual camera's viewpoint matches the initial pose. A viewing angle that matches the target pose. It can be like Figure 4 As shown.

[0065] In another example, the target pose differs from the initial pose, and the virtual camera's viewpoint matches the initial pose. A viewing angle that matches the target pose. It can be like Figure 5 As shown.

[0066] In one embodiment, if the pose of the flat-panel display device changes during the display of the first planar image, rendering of the first planar video frame can be stopped, and the second planar video can be redefined and rendered. The second planar image may include second planar video frames that correspond one-to-one with each spatial video frame of the spatial video.

[0067] In some embodiments, during the display of the first planar image, if the pose of the planar display device changes and the first planar video frame corresponding to the j-th spatial video frame has not yet been displayed, the second planar video frames of the (j+1)-th and subsequent spatial video frames can be re-determined based on the target pose of the virtual camera, and the second planar video frames can be rendered sequentially. The second planar image may include second planar video frames that correspond one-to-one with the (j+1)-th and subsequent spatial video frames of the spatial video.

[0068] In some embodiments, adjusting the initial pose based on the inertial data to obtain the target pose of the virtual camera includes: adjusting the pose of the virtual camera to match the inertial data, so that the target pose matches the inertial data.

[0069] In this embodiment, a first mapping data that reflects the mapping relationship between inertial data and pose can be preset; based on the inertial data and the first mapping data, the pose corresponding to the inertial data is obtained as the target pose.

[0070] The first mapping data can be a first mapping function, where the dependent variable of the first mapping function is the pose and the independent variable is the inertial data. In this way, by substituting the inertial data into the first mapping function, the pose corresponding to the inertial data can be obtained as the target pose.

[0071] In some embodiments, adjusting the initial pose based on the inertial data to obtain the target pose of the virtual camera includes: determining a first pose adjustment amount of the flat panel display device based on the inertial data; determining a second pose adjustment amount of the virtual camera based on the first pose adjustment amount; and obtaining the target pose based on the initial pose and the second pose adjustment amount.

[0072] In this embodiment, the pose adjustment amount may include rotation angle and / or translation displacement.

[0073] In this embodiment, the rotation angle includes an angle value and a direction, with the direction indicated by a positive or negative sign. For example, the direction of the rotation angle can be clockwise or counterclockwise, and the positive and negative signs of the counterclockwise and clockwise rotation angles are different.

[0074] In this embodiment, the displacement includes distance and direction, and the direction of the displacement is indicated by positive and negative signs.

[0075] In this embodiment, the position represented by the target pose is different from the position represented by the initial pose, and / or the posture represented by the target pose is different from the posture represented by the initial pose.

[0076] In this embodiment, a second mapping data reflecting the mapping relationship between the pose adjustment amount of the flat panel display device and the pose adjustment amount of the virtual camera can be preset; based on the first pose adjustment amount and the second mapping data, a second pose adjustment amount corresponding to the first pose adjustment amount is obtained.

[0077] The second mapping data can be a second mapping function, a second lookup table, etc., and is not limited here.

[0078] For the second mapping function, the dependent variable is the pose adjustment amount of the virtual camera, and the independent variable is the pose adjustment amount of the flat panel display device. Thus, by substituting the first pose adjustment amount into the second mapping function, the second pose adjustment amount corresponding to the first pose adjustment amount can be obtained.

[0079] For the second lookup table, the second pose adjustment value corresponding to the first pose adjustment value can be found in the second lookup table. If the first pose adjustment value cannot be found directly in the second lookup table, the two values ​​adjacent to the first pose adjustment value can be found, and the second pose adjustment value corresponding to the first pose adjustment value can be obtained by interpolation based on these two values ​​and the second pose adjustment values ​​corresponding to these two values.

[0080] Through this embodiment, the pose of the virtual camera can be adjusted according to the inertial data of the flat panel display device to change the shooting angle of the virtual camera, thereby adjusting the corresponding display content of the flat image displayed by the flat panel display device and improving the user's viewing experience.

[0081] <Device Embodiment> This embodiment provides a spatial image display device for a flat panel display device. For example... Figure 6 As shown, the spatial image display device 6000 may include a model building module 6100, a camera setting module 6200, a region determination module 6300, and an image rendering module 6400.

[0082] The model building module 6100 is used to acquire spatial images of the target scene captured by the binocular camera, and to build a three-dimensional model of the target scene based on the spatial images.

[0083] The camera setup module 6200 is used to set up a virtual camera based on a pre-set initial pose.

[0084] The region determination module 6300 is used to determine a first region in the three-dimensional model that matches the viewpoint of the virtual camera based on the initial pose.

[0085] The image rendering module 6400 is used to obtain a first planar image based on the first region and control the planar display device to render the first planar image.

[0086] In some embodiments, constructing a 3D model of the target scene based on the spatial imagery includes: Determine the point cloud data of the common viewing area of ​​the spatial image; The target scene is reconstructed in three dimensions based on the point cloud data to obtain a three-dimensional mesh of the target scene. The three-dimensional mesh is textured based on the spatial image to obtain the three-dimensional model with texture information.

[0087] In some embodiments, the spatial image display device 6000 further includes: A module for acquiring inertial data of the flat panel display device; A module for adjusting the initial pose based on the inertial data to obtain the target pose of the virtual camera; A module for redetermining a second region in the 3D model that matches the viewpoint of the virtual camera based on the target pose; A module for obtaining a second planar image based on the second region and controlling the flat panel display device to render the second planar image.

[0088] In some embodiments, adjusting the initial pose based on the inertial data to obtain the target pose of the virtual camera includes: The first pose adjustment amount of the flat panel display device is determined based on the inertial data; The second pose adjustment amount of the virtual camera is determined based on the first pose adjustment amount; The target pose is obtained based on the initial pose and the second pose adjustment amount.

[0089] In some embodiments, the initial pose indicates that the virtual camera is positioned between the stereo cameras and in the same pose as the stereo cameras.

[0090] In some embodiments, the spatial image is a spatial video or spatial image obtained by the binocular camera capturing the target scene.

[0091] In some embodiments, the spatial image is a spatial video, the first planar image includes a first planar video frame that corresponds one-to-one with each spatial video frame in the spatial video, and the spatial image display device 6000 further includes: A module for determining the first planar video frame in the first planar image that corresponds to the first spatial video frame of the spatial video, as the target planar video frame; A module for storing the target planar video frame as a thumbnail of the spatial video data and corresponding to the spatial video data.

[0092] <Example of Flat Panel Display Device> This embodiment provides a flat panel display device, which in one aspect may include the aforementioned spatial image display device 6000.

[0093] On the other hand, such as Figure 5 As shown, the flat panel display device 5000 may include a processor 5100 and a memory 5200. The memory 5200 is used to store computer programs, and the processor 5100 is used to control the flat panel display device to execute the methods of any embodiment of this disclosure under the control of the computer programs.

[0094] <Example of a readable storage medium> This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the methods described in any of the method embodiments of this disclosure.

[0095] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0096] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0097] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0098] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of the present invention.

[0099] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0101] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0103] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A spatial image display method for a flat display device, characterized by, include: Acquire spatial images of the target scene captured by a binocular camera, and construct a three-dimensional model of the target scene based on the spatial images; The virtual camera is set according to the preset initial pose; Based on the initial pose, determine the first region in the 3D model that matches the viewpoint of the virtual camera; A first planar image is obtained based on the first region, and the planar display device is controlled to render the first planar image.

2. The method of claim 1, wherein, The step of constructing a 3D model of the target scene based on the spatial imagery includes: Determine the point cloud data of the common viewing area of ​​the spatial image; The target scene is reconstructed in three dimensions based on the point cloud data to obtain a three-dimensional mesh of the target scene. The three-dimensional mesh is textured based on the spatial image to obtain the three-dimensional model with texture information.

3. The method of claim 1, wherein, The method further includes: Acquire the inertial data of the flat panel display device; The initial pose is adjusted based on the inertial data to obtain the target pose of the virtual camera; Based on the target pose, a second region in the 3D model that matches the viewpoint of the virtual camera is redefined; A second planar image is obtained based on the second region, and the planar display device is controlled to render the second planar image.

4. The method of claim 3, wherein, The step of adjusting the initial pose based on the inertial data to obtain the target pose of the virtual camera includes: The first pose adjustment amount of the flat panel display device is determined based on the inertial data; The second pose adjustment amount of the virtual camera is determined based on the first pose adjustment amount; The target pose is obtained based on the initial pose and the second pose adjustment amount.

5. The method according to any one of claims 1 to 4, characterized in that, The initial pose indicates that the virtual camera is positioned between the stereo cameras and in the same pose as the stereo cameras.

6. The method according to any one of claims 1 to 4, characterized in that, The spatial image is a spatial video or spatial image obtained by the binocular camera capturing the target scene.

7. The method according to any one of claims 1 to 4, characterized in that, The spatial image is a spatial video, and the first planar image includes a first planar video frame that corresponds one-to-one with each spatial video frame in the spatial video. The method further includes: The first planar video frame in the first planar image that corresponds to the first spatial video frame of the spatial video is determined as the target planar video frame; The target planar video frame is stored as a thumbnail of the spatial video data and corresponding to the spatial video data.

8. A spatial image display device for a flat panel display equipment, characterized in that, include: The model building module is used to acquire spatial images of the target scene captured by the binocular camera, and to build a three-dimensional model of the target scene based on the spatial images. The camera setting module is used to set up the virtual camera according to the preset initial pose; A region determination module is used to determine a first region in the 3D model that matches the viewpoint of the virtual camera based on the initial pose; The image rendering module is used to obtain a first planar image based on the first region and control the planar display device to render the first planar image.

9. A flat panel display device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program, and the processor being used, under the control of the computer program, to execute the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images