System, method and medium for automatically detecting and inserting digital streams into 360-degree video
By automatically detecting and inserting high-resolution digital media streams in the 360-degree video system, the problem of poor imaging of 360-degree video in the high-density information area is solved, and the user-readable video display effect is achieved.
Patent Information
- Application Number
- CN201910565245.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-07-10
- Filing Date
- 2019-06-27
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-06-27
AI Technical Summary
360-degree videos are poorly imaged in certain areas containing high-density information, making it difficult for viewers to read the displayed information.
By acquiring 360-degree video using at least one camera and automatically detecting and inserting high-resolution digital media streams into 360-degree video using a processing unit, determining the insertion position using a mask and an image or geometric method based on the method to ensure that the media stream pair is paired with a pair of two or more. User-readable.
It effectively solves the problem of poor imaging of 360-degree video in high-density information areas, ensures that users can clearly read the displayed information, and enhances the readability of the video system.
Smart Images

Figure CN110708502B_ABST
Abstract
Description
Technical Field
[0001] The disclosed embodiments relate generally to augmented video systems and methods and, more particularly, to systems and methods for automatically detecting and inserting digital streams into 360-degree video. Background Art
[0002] 360-degree video (also called immersive video or spherical video) is a video recording obtained using an omnidirectional camera or a collection of multiple cameras pointing in different directions, where the view in each direction is recorded simultaneously. During the playback of a 360-degree video on a conventional flat-panel display, the viewer can be provided with control over the viewing direction (such as a panoramic view). 360-degree videos can also be played on displays or projectors set in a portion of a cylinder or sphere. For remote participation in conferences or other group gatherings (such as trade shows, museums, and art exhibitions), the aforementioned 360-degree videos are a significant improvement because they provide remote viewers with complete control over what the viewer can view.
[0003] One drawback of the technology, however, is that 360-degree video does not image certain areas that contain dense information (such as presentation slides projected onto a screen or whiteboard) well. In many cases, glare and the distance to the area of interest make it impossible for the viewer to read the displayed information.
[0004] Therefore, in view of the above-mentioned and other shortcomings of conventional 360-degree video technology, a need exists for new and improved systems and methods that will automatically detect a high-resolution digital video stream and insert the digital stream into a 360-degree video so that it is readable by a user. Summary of the invention
[0005] Implementations described herein are directed to systems and methods that substantially obviate one or more of the above-identified and other problems associated with conventional systems and methods for 360 degree video.
[0006] According to one aspect of the embodiments described herein, a system is provided, comprising: at least one camera that acquires a 360-degree video of an environment; and a processing unit that is configured to identify at least one insertion candidate within the acquired 360-degree video and insert media into the at least one identified insertion candidate.
[0007] In one or more implementations, the insertion candidates are regions within the 360-degree video.
[0008] In one or more implementations, the area within the 360-degree video is a surface.
[0009] In one or more implementations, the area within the 360 degree video is a display screen.
[0010] In one or more implementations, the area within the 360-degree video is a whiteboard.
[0011] In one or more implementations, the media is an image.
[0012] In one or more implementations, the media is a video stream.
[0013] In one or more implementations, the resolution of the media is higher than the resolution of the 360-degree video.
[0014] In one or more implementations, the inserted media is cropped based on detected occlusions of the insertion candidate.
[0015] In one or more embodiments, the inserted media is cropped using a mask.
[0016] In one or more implementations, insertion candidates are identified using the content of the 360-degree video.
[0017] In one or more implementations, the insertion candidate is identified based on a position of at least one camera relative to the position of the insertion candidate.
[0018] In one or more implementations, the media to be inserted into the identified at least one insertion candidate is selected based on facial recognition of a person located in the 360-degree video near the identified at least one insertion candidate.
[0019] In one or more implementations, insertion candidates are identified using object recognition within the 360-degree video.
[0020] In one or more implementations, insertion candidates are additionally identified based on user input.
[0021] In one or more implementations, the media to be inserted into the identified at least one insertion candidate is selected based on a location of the identified at least one insertion candidate within the 360-degree video.
[0022] According to another aspect of the embodiment described herein, a method is provided, comprising the following steps: acquiring a 360-degree video of an environment using at least one camera; and identifying at least one insertion candidate within the acquired 360-degree video using a processing unit, and inserting media into at least one of the identified insertion candidates.
[0023] According to another aspect of the embodiments described herein, there is provided a tangible computer-readable medium containing a set of instructions for implementing a method, the method comprising the steps of: acquiring a 360-degree video of an environment using at least one camera; and identifying, using a processing unit, at least one insertion candidate within the acquired 360-degree video and inserting media into the at least one identified insertion candidate.
[0024] Other aspects related to the present invention will be explained in part in the following description, and in part will be obvious from the description, or can be learned by practicing the present invention. Various aspects of the present invention can be realized and obtained by the combination of the components and various components and aspects particularly pointed out in the following detailed description and the appended claims.
[0025] It should be understood that the foregoing and following descriptions are exemplary and illustrative only and are not intended to limit the claimed invention or its applications in any way. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The accompanying drawings, which are included in and constitute a part of this specification, illustrate embodiments of the present invention and, together with the specification, explain and illustrate the principles of the present invention. Specifically:
[0027] Figure 1 An exemplary setup for 360-degree video is illustrated.
[0028] Figure 2 An exemplary operational sequence of an embodiment of a system for automatically detecting and inserting a digital stream into a 360-degree video as described herein is illustrated.
[0029] Figure 3 Some exemplary areas in a 360-degree video (such as a laptop's LCD screen) that are not directly visible to the camera are illustrated.
[0030] Figure 4 An exemplary implementation of a computer system 400 is illustrated that may be used to implement the techniques described herein. DETAILED DESCRIPTION
[0031] In the following detailed description, reference will be made to the accompanying drawings, in which identical functional components are represented by identical reference numerals. The above-mentioned accompanying drawings illustrate specific embodiments and implementations that conform to the principles of the present invention by way of example and not by way of limitation. These implementations are described in sufficient detail to enable those skilled in the art to practice the present invention, and it will be appreciated that other implementations may be utilized, and structural changes and / or replacements may be made to various components without departing from the scope and spirit of the present invention. Therefore, the following detailed description should not be construed as limiting. In addition, the various embodiments of the present invention described may be implemented in the form of software running on a general-purpose computer, in the form of dedicated hardware, or in a combination of software and hardware.
[0032] According to one aspect of the embodiments described herein, systems and methods are provided for automatically detecting and inserting digital streams into 360-degree videos. Various examples of such user interest areas include, but are not limited to, content displayed on various electronic displays or written on electronic paper (electronic ink), content projected on various surfaces using electronic projectors, content of paper documents appearing in 360-degree videos, and / or content written on whiteboards (blackboards) within 360-degree videos. As will be clear to those of ordinary skill in the art, these user interest areas are increasingly "digital born," meaning that a computer is driving the display. For some content (such as a whiteboard, paper document, or painting in a museum), a participant (or manager) can take a picture of the area, again stored somewhere in digital form and available for download.
[0033] In one or more embodiments, these digital streams with content of interest to the user are obtained and then inserted onto the 360-degree view generated from the original 360-degree video feed, allowing the user to view them in their native high resolution. It should be noted that while this description focuses on 360-degree field of view images, the methods described herein can be applied to other views with less than a full 360-degree panoramic field of view.
[0034] In one or more embodiments, to achieve this, the system semi-automatically determines where each digital stream should be inserted into the 360-degree video. To provide realistic insertions, embodiments of the system also apply masks to the digital streams to avoid occluding objects, such as people that may move between the 360-degree camera and the area of interest (e.g., a presenter moving in front of the projected content).
[0035] Various embodiments use two methods to find the area and determine where to place the insert: image-based; and geometry-based. With the image-based method, no information about the actual arrangement of the camera or area is needed (they are simply determined and manipulated by their position within the image from the 360-degree camera). Alternatively, the geometry-based method uses the pose of the camera and the location of the area in the world to determine their corresponding position in the image. The image-based method is simpler and may be suitable for situations where the camera and the area of interest are fixed, such as a permanently placed 360-degree camera and electronic (LCD, LED, plasma) display in a conference room.
[0036] In another alternative embodiment, when the camera moves, the area moves, or auxiliary tools such as other cameras or sensors are used to determine the location of the region of interest or occluders, a geometry-based method may be needed. The aforementioned image-based method (especially for the conference room scene) is described first, and then the geometry-based method is discussed.
[0037] In one or more embodiments, the first step is to extract a set of candidate faces and objects (collectively referred to as “regions of interest”) from the 360-degree video feed, such as Figure 1 As shown, it includes three planes 101, 102, 103, five faces 104, 105, 106, 107, 108, and a laptop 109 associated with one of the faces. In various embodiments, the aforementioned automatic detection can use face and person detection and detected flickering screens to determine their corresponding positions in the video.
[0038] Figure 2 An exemplary operational sequence 200 of an embodiment of a system for automatically detecting and inserting digital streams into a 360-degree video as described herein is illustrated. First, at step 201, a 360-degree video is created using one or more cameras. At step 202, an area in the 360-degree video is detected.
[0039] At step 203, candidate regions are identified. In various embodiments, at step 203, a user (end user or "room curator") may be enabled to manually adjust the position of the aforementioned region of interest. It should be noted that for most rooms, since the projector screen or whiteboard is unlikely to be moved over a large range, no further adjustment of the above-mentioned surfaces is required. At step 205, embodiments of the system then collect a set of digital streams available to the system, including but not limited to: a VNC video stream (video+audio, HTML5 solution) running from a podium computer, an image or real-time video from a mobile device imaging a whiteboard, a WebRTC (Web Real-Time Communication) stream of a screen shared during a meeting along with the names of its participants, or any other video stream in a format known in the art.
[0040] In various embodiments, using image matching, the system associates the digital stream with the previously identified region of interest (see step 204), thereby generating stream / region pairs 206. Furthermore, the end user of the application or the system administrator can modify the automatically generated default mapping. For example, in Figure 1 In the digital image of the whiteboard, it may be difficult to distinguish which whiteboard should be mapped to the digital image of the whiteboard.
[0041] Once the location of the region of interest is found and mapped to a digital stream, the digital stream is added to the 3D rendered 360-degree video feed. However, because it is possible for people in the room to occlude the region of interest (e.g., a presenter moves in front of the projected content), embodiments of the system include an occlusion detector that calculates a mask of the digital stream before inserting it into the final rendering, see step 207. In various embodiments, masking can be accomplished by:
[0042] 1. Mask out regions with large differences (e.g., Figure 1 , whiteboard).
[0043] 2. Detect occluded shapes in front of the digital stream by using the Mask R-CNN method described in He, Kaiming, et al. "Mask R-CNN." Computer Vision (ICCV), 2017 IEEE International Conference on. IEEE, 2017. Figure 3The method is illustrated in , where a laptop stream is rendered over an area of interest detected as a laptop, but if the user moves in front of the laptop during a meeting, the laptop may disappear. In one or more alternative embodiments, other methods may be used instead of the aforementioned Mask R-CNN method, including but not limited to: any object detector that can, for example, determine whether a person is standing; any human pose estimator that can be used to obtain the same information; or any other semantic segmentation network, such as DeepLab available from Google. Finally, in step 208, the final inset rendering is performed.
[0044] It should be noted that some areas may not be visible to the camera, such as the laptop screen shared by a participant during the meeting via WebRTC: the 360 camera can only see the back of the laptop, such as Figure 3 The element 301 shown makes it impossible to use image differences to calculate the mask. Instead, in one embodiment, the system relies on a person detector and segmentation method (e.g., Mask R-CNN) to detect that a person is standing in front of a specific area.
[0045] In one or more embodiments, if the face detector also has face recognition, the system can insert the shared view into the 360 video on the laptop closest to the recognized face. In the event that the face is not easily recognized, in one embodiment, the participant can manually assign themselves to the area, such as by clicking on their face in the 360 video feed.
[0046] Below, the method based on geometry for enhancing video is described in detail. Another method for determining the area where the insert should be placed is to determine the world position and arrangement of the 360-degree camera, as well as the world position and layout of the area of interest. Then, the world coordinates (such as the vertices of the rectangle) defining the area can be projected into the image coordinates. In the case where the camera or object in the scene is moving, or when other position information sources are used to provide information for the mask, this may be necessary. For example, the Kinect system (known in the art and widely commercially available) with an element (skeleton) tracker can determine whether the host is standing in front of the display. If the Kinect is placed very close to the 360-degree camera, then the image-based method is sufficient, but if the Kinect is in other positions, then the world position of the tracked host must be projected into the 360-degree image coordinates to determine whether there is occlusion. This can also be used when using certain tracking technologies such as structured coded light to determine the position of a device such as a laptop.
[0047] For the case where the camera is moving, the camera position can be determined by identifying the locations of fiducials in the world, such as on a placed QR code as is known in the art, or at landmarks on a registered image. This can be done in real time. For some applications (such as a virtually recorded museum tour), the camera pose can be determined by structure from motion methods as is known in the art, such as provided in OpenSFM, which do not run in real time.
[0048] Exemplary Implementation of a Computer System
[0049] Figure 4 An exemplary embodiment of a computer system 400 that can be used to implement the techniques described herein is illustrated. In one or more embodiments, the computer 400 can be implemented in the form factor of a mobile computing device known to those skilled in the art. In an alternative embodiment, the computer 400 can be implemented based on a laptop or notebook computer. However, in an alternative embodiment, the computer 400 can be a dedicated computing system.
[0050] The computer 400 may include a data bus 404 or other interconnection or communication mechanism for transmitting information throughout and between the various hardware components of the computer 400, and a central processing unit (CPU or simply processor) 401 coupled to the data bus 404 to process information and perform other computing and control tasks. The computer 400 also includes a memory 412, such as a random access memory (RAM) or other dynamic storage device, coupled to the data bus 404 for storing various information and instructions to be executed by the processor 401. The memory 412 may also include a permanent storage device, such as a magnetic disk, an optical disk, a solid-state flash memory device, or other non-volatile solid-state storage device.
[0051] In one or more embodiments, the memory 412 may also be used to store temporary variables or other intermediate information during the execution of instructions by the processor 401. Optionally, the computer 400 may also include a read-only memory (ROM or EPROM) 402 or other static storage device coupled to the data bus 404 to store static information and instructions for the processor 401, such as firmware required for the operation of the computer 400, a basic input and output system (BIOS), and various configuration parameters of the computer 400.
[0052] In one or more embodiments, the computer 400 may additionally include two or more cameras 409, 410 for generating 360-degree videos. In addition, the computer 400 may include multiple microphones 411 for obtaining audio associated with the captured 360-degree videos.
[0053] In one or more embodiments, the computer 400 may additionally include a communication interface, such as a network interface 405 coupled to the data bus 404. The network interface 405 may be configured to establish a connection between the computer 400 and the Internet 424 using at least one of a WIFI interface 407 and a cellular network (GSM or CDMA) adapter 408. The network interface 405 may be configured to provide two-way data communication between the computer 400 and the Internet 424. The WIFI interface 407 may operate in accordance with 802.11a, 802.11b, 802.11g and / or 802.11n protocols and Bluetooth protocols known to those of ordinary skill in the art. In an exemplary implementation, the WIFI interface 407 and the cellular network (GSM or CDMA) adapter 408 send and receive electrical or electromagnetic signals carrying digital data streams representing various types of information.
[0054] In one or more embodiments, the Internet 424 provides data communication to other network resources through one or more subnetworks usually. Therefore, computer 400 can access various network resources anywhere on the Internet 424, such as remote media servers, web servers, other content servers and other network data storage resources. In one or more embodiments, computer 400 is configured to send and receive messages, media and other data, including application code, by means of network interface 405 through the multiple networks including the Internet 424. In this Internet example, when computer 400 acts as a network client, it can be the application request code or data executed in computer 400. Similarly, it can send various data or computer code to other network resources.
[0055] In one or more embodiments, the functions described herein are implemented by computer 400 in response to processor 401 executing one or more sequences of one or more instructions contained in memory 412. Such instructions can be read into memory 412 from another computer-readable medium. Executing the sequence of instructions contained in memory 412 causes processor 401 to perform the various processing steps described herein. In alternative embodiments, hard-wired circuits may be used instead of or in combination with software instructions to implement embodiments of the present invention. Therefore, embodiments of the present invention are not limited to any specific combination of hardware circuits and software.
[0056] The term "computer-readable medium" as used herein refers to any medium that participates in providing instructions to the processor 401 for execution. The computer-readable medium is only one example of a machine-readable medium that can carry instructions for implementing any of the methods and / or techniques described herein. Such a medium can take a variety of forms, including but not limited to non-volatile media and volatile media.
[0057] Common forms of non-transitory computer-readable media include, for example, floppy disks, floppy disks, hard disks, tapes or any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAMs, PROMs, EPROMs, FLASH-EPROMs, flash drives, memory cards, any other memory chips or cassettes, or any other media from which a computer can read. Various forms of computer-readable media may involve one or more sequences of one or more instructions transmitted to processor 401 for execution. For example, the instruction may initially be carried on a disk from a remote computer. Alternatively, the remote computer may load the instruction into its dynamic memory and send the instruction via the Internet 424. Specifically, the computer instruction may be downloaded from the aforementioned remote computer to the memory 412 of the computer 400 via the Internet 424 using a variety of network data communication protocols known in the art.
[0058] In one or more embodiments, the memory 412 of the computer 400 may store any of the following software programs, applications, and / or modules:
[0059] 1. Operating system (OS) 413, which may be a mobile operating system for implementing basic system services and managing various hardware components of computer 400. Exemplary implementations of operating system 413 are well known to those skilled in the art and may include any mobile operating system now known or later developed. In addition, a network communication module 414 may be provided that can utilize network interface 405 to implement network communication.
[0060] 2. Software module 415 may, for example, include a set of software modules executed by processor 401 of computer 400, which enables computer 400 to perform certain predetermined functions, such as recording 360-degree video using cameras 409 and 410 (module 416) and performing automatic detection of digital streams and inserting them into 360-degree video (module 417).
[0061] 3. The data storage unit 418 can be used to store various parameters and thresholds 419, for example.
[0062] Finally, it should be understood that the processes and techniques described herein are not inherently related to any particular device, but can be implemented by any suitable combination of components. Moreover, various types of general-purpose devices can be used according to the teachings described herein. It can also be proved that it is advantageous to construct a special device to perform the method steps described herein. The present invention has been described in conjunction with specific embodiments, which are intended to be illustrative rather than restrictive in all respects. It should be clear to those skilled in the art that many different combinations of hardware, software and firmware will be suitable for practicing the present invention. For example, the described software can be implemented in various programming or scripting languages, such as assembler, C / C++, Objective-C, perl, shell, PHP, Java, and any programming or scripting language now known or developed later.
[0063] In addition, other implementations of the present invention will be apparent to those skilled in the art in view of the specification and practice of the invention disclosed herein. Various aspects and / or components of the described embodiments may be used alone or in any combination in a system and method for automatically detecting and inserting data streams into a 360-degree video. The present specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present invention is indicated by the appended claims.
Claims
1. A system for automatically detecting and inserting digital streams into 360-degree video, the system comprising: a. at least one camera, wherein the at least one camera acquires a 360-degree video of the environment; as well as b. a processing unit that identifies at least one insertion candidate within the acquired 360-degree video and inserts the media into the at least one identified insertion candidate, The insertion candidate is a region within the 360-degree video. wherein, when the at least one camera is moving, the position of the at least one camera is determined by identifying the position of a reference point of the at least one camera in world coordinates, and The insertion candidate is identified based on the position of the at least one camera relative to the position of the insertion candidate.
2. The system according to claim 1, wherein: The area within the 360-degree video is a surface.
3. The system according to claim 1, wherein: The area within the 360-degree video is the display screen.
4. The system according to claim 1, wherein: The area within the 360-degree video is a whiteboard.
5. The system according to claim 1, wherein: The media is an image.
6. The system according to claim 1, wherein: The media is a video stream.
7. The system according to claim 1, wherein: The resolution of the media is higher than the resolution of the 360-degree video.
8. The system according to claim 1, wherein: The inserted media is cropped based on the detected occlusion of the insertion candidate.
9. The system according to claim 8, wherein: The inserted media is cut using a mask.
10. A method for automatically detecting and inserting a digital stream into a 360-degree video, the method comprising the steps of: a. Use at least one camera to obtain a 360-degree video of the environment; as well as b. using a processing unit to identify at least one insertion candidate within the acquired 360-degree video, and inserting the media into the at least one identified insertion candidate, The insertion candidate is a region within the 360-degree video. wherein, when the at least one camera is moving, the position of the at least one camera is determined by identifying the position of a reference point of the at least one camera in world coordinates, and The insertion candidate is identified based on the position of the at least one camera relative to the position of the insertion candidate.
11. The method according to claim 10, wherein: The area within the 360-degree video is a surface.
12. A tangible computer readable medium comprising a set of instructions for implementing a method comprising the steps of: a. Use at least one camera to obtain a 360-degree video of the environment; as well as b. using a processing unit to identify at least one insertion candidate within the acquired 360-degree video, and inserting the media into the at least one identified insertion candidate, The insertion candidate is a region within the 360-degree video. wherein, when the at least one camera is moving, the position of the at least one camera is determined by identifying the position of a reference point of the at least one camera in world coordinates, and The insertion candidate is identified based on the position of the at least one camera relative to the position of the insertion candidate.
Citation Information
Patent Citations
Video processing method and apparatus, terminal device and storage medium
CN107909022A
Method for sending intersting digital video selected zone by selected acuity
CN1510927A
Image processing system, image processing method, electronic apparatus, and image processing program
JP2016119567A
Mark processing unit and program
JP2017037434A
System and methods for generating automatic and user-controllable movies of presentations on small devices
US20090113278A1
