Panorama generation using mobile camera

Abstract

To provide a computer mounting method for selecting and stitching an image frame, and a computing device.SOLUTION: The method performs identifying one or more regions of interest in a plurality of image frames acquired by a camera, selecting a set of base frames such that the one or more identified regions of interest are completely contained within at least one selected base frame based on respective quality measures associated with each image frame of the plurality of image frames, and combining the selected sets of base frames to form a composite image.SELECTED DRAWING: Figure 2

Description

[Background technology]

[0001] background In image processing, "image stitching" is the process of combining multiple individual image frames into a composite image, e.g., a panoramic image. While many approaches exist, most stitching algorithms rely on individual image frames that contain at least multiple overlapping regions. Such stitching algorithms generally identify distinctive features within the overlapping regions and then match the features to establish correspondence between the individual image frames. The stitching algorithm then generally blends corresponding image frames in the overlapping regions to create the final composite image. Summary of the Invention

[0002] overview An example embodiment includes a computing device that performs image stitching. The computing device may include a base frame selection module operable to select one or more base frames from a plurality of image frames. The computing device may also include a stitching module operable to stitch together the selected one or more base frames. Using these two modules, the computing device can create composite images, such as panoramic images, and then display these composite images to a user.

[0003] In a first aspect, a computer-implemented method is provided. The method comprises a computing device acquiring a plurality of image frames. The method further comprises the computing device identifying one or more regions of interest in one or more image frames of the plurality of image frames. The method further comprises the computing device selecting a set of base frames based on a respective quality measure associated with each image frame of the plurality of image frames, wherein each identified region of interest of the identified one or more regions of interest is entirely contained within at least one base frame of the set of selected base frames. The method further comprises the computing device stitching together the set of selected base frames to create a composite image.

[0004] In a second aspect, a computing device is provided. The computing device may include one or more processors. The computing device may further include a non-transitory data storage storing at least computer-readable instructions that, when executed by the one or more processors, cause the computing device to perform the operations. The operations may include acquiring a plurality of image frames. The operations may further include identifying one or more regions of interest in one or more image frames of the plurality of image frames. The operations may further include selecting a set of base frames based on a respective quality measure associated with each image frame of the plurality of image frames, each identified region of interest of the identified one or more regions of interest being entirely contained within at least one base frame of the selected set of base frames. The operations may further include stitching together the set of selected base frames to create a composite image.

[0005] In a third aspect, an article of manufacture is provided. The article of manufacture may include non-transitory data storage storing at least computer readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations. The operations may include acquiring a plurality of image frames. The operations may further include acquiring a plurality of image frames. The operations may include identifying one or more regions of interest in one or more image frames of the plurality of image frames. The operations may further include selecting a set of base frames based on a respective quality metric associated with each image frame of the plurality of image frames, wherein each identified region of interest of the identified one or more regions of interest is contained entirely within at least one base frame of the selected set of base frames. The operations may further include stitching together the set of selected base frames to create a composite image.

[0006] In a fourth aspect, a system is provided. The system may comprise means for acquiring a plurality of image frames. The system may further comprise means for identifying one or more regions of interest within one or more image frames of the plurality of image frames. The system may further comprise means for selecting a set of base frames based on a respective quality metric associated with each image frame of the plurality of image frames, each identified region of interest of the identified one or more regions of interest being entirely contained within at least one base frame of the selected set of base frames. The system may further comprise means for stitching together the set of selected base frames to create a composite image.

[0007] Other aspects, embodiments, and implementations will become apparent to those of ordinary skill in the art from a reading of the following detailed description, where appropriate in conjunction with the accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 illustrates a computing device according to an example embodiment. [Diagram 2]FIG. 2 illustrates an overview of the operation of the base frame selection module and the stitching module, according to an example embodiment. [Diagram 3] FIG. 13 illustrates the operation of a base frame selection module, according to an example embodiment. [Figure 4] FIG. 2 illustrates an example of an image frame subset, according to an example embodiment. [Diagram 5] FIG. 4 illustrates the operation of a stitching module, according to an example embodiment. [Figure 6] 1A-1C are diagrams illustrating examples of image frame projections, according to example embodiments. [Figure 7] 1A-1C are diagrams illustrating examples of seams according to example embodiments. [Figure 8] FIG. 1 illustrates a method according to an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Detailed Description Example methods, devices, and systems are described herein. And the word "exemplary" is used herein to mean "serving as an example, instance, or It should be understood that the term "example" or "exemplary" is used to mean "serving as an example." Any embodiment or feature described herein as "example" or "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or features, unless so indicated. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0010] Accordingly, the example embodiments described herein are not meant to be limiting, as it will be readily understood that the aspects of the present disclosure, as generally described and illustrated in the Figures herein, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0011] Throughout this description, the articles "a" or "an" are used to introduce elements of example embodiments. Any reference to "a" or "an" means "at least "at least one" and any reference to "the" does not Unless otherwise specified or clearly indicated otherwise by context, the term "at least one" refers to "the at least one of." The use of the conjunction "or" in a stated list of at least two terms is intended to refer to any one of the listed terms or any combination of the listed terms. It is to show.

[0012] The use of ordinal numbers such as "first," "second," and "third" is not intended to indicate a particular order of those elements, but rather to distinguish one element from another. For purposes of this description, the terms "multiple" and "a plurality of" refer to "two or more" or "more than one."

[0013] Furthermore, unless the context suggests otherwise, features shown in each figure may be used in combination with one another. Thus, it should be understood that the drawings should generally be viewed as aspects of components of one or more overall embodiments, and that not all illustrated features are necessary for each embodiment. In the drawings, like symbols typically identify like components, unless the context dictates otherwise. Furthermore, unless otherwise noted, the figures are not drawn to scale and are used for illustrative purposes only. Furthermore, the figures are merely representative, and not all components are shown. For example, additional structures or limiting components may not be shown.

[0014] Additionally, any recitation of elements, blocks, or steps in the specification or claims is for clarity purposes and therefore should not be construed as requiring or implying that those elements, blocks, or steps follow a particular arrangement or be performed in a particular order.

[0015] I. Overview Some example image stitching processes include four stages: base frame selection, feature detection, alignment, and blending. The base frame selection stage includes selecting one or more base frames from a set of candidate base frames. The feature detection stage includes identifying corresponding features in the selected one or more base frames. The alignment stage includes transforming at least a portion of the selected one or more base frames to align the identified features. And the blending stage includes combining the aligned frames into a single composite image.

[0016] Many image stitching processes include a base frame selection stage that does not attempt to distinguish between the object of interest and background objects. As a result, such image stitching processes may select a base frame that includes a low-quality representation of the object of interest therein, i.e., the object of interest is blurred, underexposed, and / or distorted in some way. This problem may be detrimental in terms of image fidelity and may reduce the overall quality of the composite image created from these image stitching processes. In particular, even if the distortions to the object of interest are relatively small, these distortions may be particularly noticeable on the composite image and may represent a significant degradation in image quality.

[0017] Many image stitching processes include a blending stage that does not attempt to distinguish between objects of interest and background objects. As a result, when blending two image frames, such image stitching processes often place seams directly on the objects of interest, thereby introducing artifacts and / or other distortions on those objects of interest. This problem can be detrimental in terms of image fidelity and can reduce the overall quality of the composite image created from these image stitching processes.

[0018] This disclosure provides an image stitching process that can help address these issues. More specifically, an example image stitching process intelligently stitches a base frame by considering the quality of an object of interest in a set of candidate base frames. The example image stitching process may also penalize seams that are placed on the object of interest during the blending stage. Advantageously, the disclosed image stitching process enables the creation of a composite image that includes a high quality object of interest.

[0019] The disclosed process may be implemented by a computing device, such as a mobile device, a server device, or another type of computing device. The computing device may include a base frame selection module operable to receive a plurality of image frames and responsively identify a region of interest within the plurality of image frames. The region of interest may correspond to a region that includes an object of interest, such as a human face, a building, a vehicle, or an animal, among other possibilities. After identifying the region of interest, the base frame selection module may select a set of base frames from the plurality of image frames. In particular, the selection may be such that each identified region of interest is completely contained within at least one base frame of the set of selected base frames.

[0020] The computing device may also include a stitching module operable to receive the set of base frames selected by the base frame selection module and stitch the set of base frames together to create a composite image. While performing stitching, the stitching module may implement a seam finding process that adds a computational bias to seams located in a region of interest within the set of base frames. In some examples, this computational bias includes adding a penalty term to any seams that include pixels from the region of interest.

[0021] In some examples, the disclosed process is performed by the same device that captured the one or more image frames. For example, the base frame selection module and the stitching module can be installed on a computing device. Then, after the computing device captures one or more image frames, the base frame selection module can be invoked to select a set of base frames from the one or more image frames. The stitching module can then be invoked to create a composite image from the set of base frames. The composite image can be displayed, communicated, stored, and / or otherwise utilized, for example, printed on paper. In other examples, the base frame selection and / or stitching process can be performed by a device that is separate but communicatively coupled to the device that captured the one or more image frames.

[0022] In some examples, the frames may be stitched together from a continuous image stream (e.g., a video stream). The image stream may be captured by a front-facing camera (e.g., user-facing) of the computing device, a rear-facing camera (e.g., non-user-facing) of the computing device, or another camera of the computing device. In some cases, the continuous image stream may be captured using multiple cameras of the computing device, e.g., a front-facing camera and a rear-facing camera.

[0023] In some examples, the composite image may be created with minimal or no user input. For example, the composite image may be created without requiring a user to identify an area of ​​interest, an object of interest, or other aspects of the image frames. Furthermore, the composite image may be created without requiring a user to capture one or more image frames using a particular gesture (e.g., horizontal scanning of a scene with a computing device). Automated image stitching applications may benefit from not requiring such user input. However, variations of the processes described herein that involve one or more types of user input are contemplated as well.

[0024] In some examples, the computing device may select a base frame by using a machine learning model that is trained based on a base frame selection decision by the computing device. For example, after the computing device has made several (e.g., 4-10) base frame selection decisions using the base frame selection module described herein, the computing device may train a machine learning model using the base frame selection decisions. After training is complete, the computing device may intelligently select a base frame using the trained machine learning model in combination with the described base frame selection module. Other methods of selecting a base frame are also possible.

[0025] For embodiments involving using a machine learning model to select a base frame, a computing device interacting with a server device, or otherwise sharing a base frame or synthetic image with other computing devices, a user may be provided with selectable controls for both whether and when the systems, programs, devices, or features described herein enable collection of user information (e.g., information about the user's social network, social actions, or activities, occupation, user preferences, or current location of the user), and whether the user is sent content or communications from the server. Additionally, certain data may be treated in one or more ways before being stored or used such that personally identifiable information is removed. For example, the user's identity may be treated such that personally identifiable information cannot be determined about the user, or the user's geographic location may be generalized where location information is obtained (e.g., to the city, zip code, or state level, etc.) such that the user's specific location cannot be determined. Thus, a user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0026] These and other aspects, advantages, and alternatives will become apparent to those reading the following description, with reference to the accompanying drawings, as appropriate. Moreover, it should be understood that the discussion in this summary and elsewhere herein is provided by way of example only, and that numerous variations are possible.

[0027] II. Examples of Computing Devices 1 illustrates a computing device 100 according to an example embodiment. The computing device 100 may be an example of a computing device that can select base frames from a plurality of image frames and then stitch the selected base frames together to create a composite image. The computing device 100 may take various forms, such as a server device, a mobile device, a camera device, or some other form of device.

[0028] As shown in FIG. 1, computing device 100 may include camera 110. Camera 110 may include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light in one or more image frames. That is, camera 110 may create an image frame(s) of the captured light. The one or more image frames may be one or more still image frames and / or one or more image frames utilized in a video image (e.g., a continuous stream of image frames). Camera 110 may capture light and / or electromagnetic radiation emitted as visible light, infrared, ultraviolet, and / or as one or more other frequencies of light.

[0029] Camera 110 may be configured as a front-facing camera (e.g., facing toward the user) and / or a rear-facing camera (e.g., not facing toward the user) of computing device 100. In some implementations, camera 110 may capture image frames at a preconfigured frame rate. That is, every X seconds, camera 110 may capture an image frame. Examples of frame rates include 24 frames per second (FPS), 30 FPS, or 50 FPS, among other possibilities.

[0030] In some examples, the camera 110 may be oriented at a particular rotation angle and may capture an image frame at that rotation angle. In some implementations, the rotation angle is a horizontal angle, i.e., the rotation angle may be a horizontal rotation of the camera 110 from an initial pointing direction. In other implementations, the rotation angle is a vertical angle, i.e., the rotation angle may be a vertical rotation of the camera 110 from an initial pointing direction. In example embodiments, the initial pointing direction may correspond to the pointing direction of the camera 110 when capturing the first image frame in the stream of image frames.

[0031] In example embodiments, each image frame captured by camera 110 may be associated with a quality measure. This quality measure may be a quantitative metric calculated based on the motion blur of the captured image frame, the overall focus of the captured image frame, and / or the exposure of the captured image frame, among other possibilities. In some implementations, the quality measure of a captured image frame may be computationally biased to give greater weight to pixels located within a region of interest located within the captured image frame. For example, the quality measure of an image frame having an underexposed region of interest but properly exposed background objects may be lower than the quality measure of an image frame having a properly exposed region of interest but properly exposed background objects.

[0032] The display component 112 may be configured to provide output signals to a user via one or more screens (including touch screens), cathode ray tubes (CRTs), liquid crystal displays (LCDs), light emitting diodes (LEDs), displays using digital light processing (DLP) technology, and / or other similar technologies. The display component 112 may also be configured to generate audible output, such as via speakers, speaker jacks, audio output ports, audio output devices, earphones, and / or other similar devices. The display component 112 may further be configured with one or more tactile components capable of generating tactile output, such as vibrations and / or other outputs detectable by touch and / or physical contact with the computing device 100.

[0033] The network interface 114 may serve as an interface between the computing device 100 and other computing devices. The network interface 114 may include one or more wireless and / or wired interfaces configurable to communicate over a network. The wireless interface may include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth transceiver, a Zigbee transceiver, a Wi-Fi transceiver, a WiMAX transceiver, and / or other similar types of wireless transceivers configurable to communicate over a wireless network. The wired interface may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or similar transceivers configurable to communicate over a twisted pair of wires, a coaxial cable, an optical fiber link, or a similar physical connection to a wired network.

[0034] In some embodiments, the network interface 114 is a trusted, secure For each communication described herein, information to facilitate reliable communication (e.g., guaranteed message delivery) may be provided, possibly as part of the message header and / or footer (e.g., packet / message sequence information, encapsulation header and / or footer, size / time information, and transmission verification information such as Cyclic Redundancy Check (CRC) and / or parity check values). Communications may be encrypted using encryption protocols such as the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), the Rivest-Shamir-Adelman (RSA) algorithm, D One or more encryption methods, such as, but not limited to, the Iffie-Hellman algorithm, a secure sockets protocol such as SSL (Secure Sockets Layer) or TLS (Transport Layer Security), and / or the Digital Signature Algorithm (DSA). Cryptographic protocols and / or algorithms may be used to secure (e.g., encode or encrypt) and / or decrypt / decode. Other cryptographic protocols and / or algorithms may be used to secure (and then decrypt / decode) communications, as well as or in addition to those enumerated herein.

[0035] The power source(s) 116 are configurable to provide power to various components of the computing device 100. The power source(s) 116 may include a hydraulic system, an electrical system, a battery, or other types of power sources. Some components of the computing device 100 may each connect to a different power source, may be powered by the same power source, or may be powered by multiple power sources. The power source(s) 116 may be charged using various types of charging, such as a wired connection to an external power source, wireless charging, combustion, or other examples.

[0036] The sensor(s) 118 can be configured to measure conditions in and provide data about the environment of the computing device 100. For example, the sensor(s) 118 can be configured to measure conditions in and provide data about the environment of the computing device 100. For example, the sensor(s) 118 can be configured to measure conditions in and provide data about the environment of the computing device 100. For example, the sensor(s) 118 can be configured to measure conditions in and provide data about the environment of the computing device 100. For example, the sensor(s) 118 can be configured to measure conditions in and provide data about the environment of the computing device 100. For example, the sensor(s) 118 can be configured to measure conditions in and provide data about the environment of the computing device 100. For example, the sensor(s) 118 can be configured to measure conditions in and provide data about the environment of the computing device 100. The sensors 118 may include one or more of: (iii) environmental sensors for obtaining data indicative of the computing device 100's environment, such as, but not limited to, infrared sensors, optical sensors, light sensors, biosensors, capacitive sensors, touch sensors, temperature sensors, wireless sensors, radio sensors, movement sensors, microphones, sound sensors, ultrasonic sensors, and / or smoke sensors; and (iv) force sensors for measuring one or more forces (e.g., inertial forces and / or G-forces) acting on the surroundings of the computing device 100, such as, but not limited to, one or more sensors measuring force in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor for identifying a ZMP and / or a ZMP location. Many other examples of sensors 118 are possible.

[0037] The base frame selection module 120 may be a software application or subsystem within the computing device 100 that is operable to receive one or more image frames and, in response, select a base frame from the one or more image frames. In some implementations, the base frame selection module 120 may receive one or more image frames from the camera 110. In other implementations, the base frame selection module 120 may receive one or more image frames from the camera 110. The selection module 120 may receive one or more image frames from another computing device via the network interface 114. After selecting a base frame, the base frame selection module 120 may send the selected base frame to the stitching module 130.

[0038] The stitching module 130 may be a software application or subsystem within the computing device 100 operable to receive the base frames selected by the base frame selection module 120 and stitch the base frames together to create one composite image, such as a panoramic image. The composite image created by the stitching module 130 may be displayed to a user via the display 112 or communicated to a separate computing device via the network interface 114.

[0039] To conceptually illustrate an example operation of the base frame selection module 120 and the stitching module 130, Figure 2 is provided. In particular, Figure 2 illustrates how the base frame selection module 120 can receive candidate image frames 200 from the camera 110 and select a base frame 230 from the candidate image frames 200. After selection, the base frame selection module 120 can provide the base frames 230 to the stitching module 130, which can stitch the base frames 230 together to create a composite image 240.

[0040] As shown, candidate image frames 200 include five separate image frames: image frame 210, image frame 212, image frame 214, image frame 216, and image frame 218. These five separate image frames include three regions of interest: region of interest 220, region of interest 222, and region of interest 224. Each of these three regions of interest may be either (i) completely contained in the image frame, (ii) partially contained in the image frame, or (iii) not contained in the image frame. For example, region of interest 220 is completely contained in image frame 212, partially contained in image frames 210 and 214, and not contained in image frames 216 and 218. Similarly, region of interest 222 is completely contained in image frame 214, partially contained in image frames 212 and 216, and not contained in image frames 210 and 218. Furthermore, region of interest 224 is completely contained within both image frames 216 and 218 , partially contained within image frame 214 , and not contained within image frames 210 and 212 .

[0041] In accordance with the above description, the base frame selection module 120 may select base frames from the candidate image frames 200 such that each identified region of interest of the candidate image frames 200 is completely contained within at least one of the base frames 230. For example, as shown in FIG. 2, regions of interest 220, 222, and 224 are completely contained within at least one of the base frames 230. Specifically, region of interest 220 is completely contained within image frame 212, region of interest 222 is completely contained within image frame 214, and region of interest 224 is completely contained within image frame 218.

[0042] 2 are used for illustrative purposes and are not intended to be limiting with respect to the embodiments herein. In practice, the candidate image frames 200 and the base frame 230 may include a smaller number of frames or a larger number of frames, including hundreds or thousands of frames.

[0043] 1, the computing device 100 also includes a controller 140. The controller 140 may include at least one of a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). Generally, the controller 140 may include one or more processors 142 and a memory 144. The processor(s) 142 may include a general-purpose processor or a special-purpose processor (e.g., a digital signal processor, etc.). The processor(s) 142 may be configured to execute computer-readable program instructions stored in the memory 144.

[0044] Memory 144 may include or take the form of one or more computer-readable storage media that may be read or accessed by processor(s) 142. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices, that may be integrated in whole or in part with at least one of the one or more processors 142. In some embodiments, memory 144 may be implemented using one physical device (e.g., one optical, magnetic, organic, or other memory or disk storage unit), while in other embodiments, memory 144 may be implemented using two or more physical devices.

[0045] As mentioned above, the memory 144 may include computer readable program instructions related to the operation of the computing device 100. Thus, the memory 144 may include program instructions for performing or facilitating some or all of the functionality described herein. The memory 144 may store the base frame selection module 120 and / or the stitching module 130. In some embodiments, the controller 140 may perform various operations by the processor(s) 142 executing instructions stored in the memory 144.

[0046] For example, controller 140 may instruct camera 110 to capture one or more image frames according to one or more image capture characteristics. The image capture characteristics may include, among other possibilities, a desired aperture, a desired exposure time, and / or a desired image sensor light sensitivity (e.g., ISO sensitivity). As another example, controller 140 may instruct camera 110 to adjust its focal length according to one or more configuration characteristics. The configuration characteristics may include, among other possibilities, a desired focal length, a desired magnification, and / or a desired angle of view.

[0047] The controller 140 may be configured to perform other operations. For example, the controller 140 may perform the operations of the base frame selection module 120 and the stitching module 130 to create a composite image from image frames captured by the camera 110. The controller 140 may then cause the display 112 to display the composite image or the network interface 114 to transmit the composite image to a remote computing device, among other possibilities.

[0048] III. Method example 3 illustrates a method 300 according to an example embodiment. The method 300 may be implemented to select one or more base frames from a plurality of image frames. The selected base frames may be provided to the stitching module 130 or may be used for other purposes. The method 300 may be performed by various components of the computing device 100, such as the base frame selection module 120 and / or other components. For brevity, an example implementation of the method 300 is described herein using the base frame selection module 120. However, it should be understood that the disclosed principles may be applied in other scenarios with other components.

[0049] The method 300 may begin at block 310, where the base frame selection module 120 receives N image frames. In accordance with the above description, the N image frames may be The N image frames may be image frames captured by camera 110. Alternatively and / or additionally, the N image frames may be image frames communicated to computing device 100 from a remote computing device, such as a server device operating on a remote network.

[0050] Upon receiving the N image frames, the base frame selection module 120 may identify one or more regions of interest within the N image frames. In some cases, this may involve the base frame selection module 120 determining a unique identifier for each of the one or more regions of interest. For example, upon receiving the N image frames at block 310, the base frame selection module 120 may invoke an object detection module to detect objects of interest within the N image frames. The base frame selection module 120 may then assign unique identifiers to the detected objects and may store the unique identifiers as metadata with the N image frames. Alternatively, the base frame selection module 120 may determine the unique identifier at another time. For example, while executing block 340 (as described further below), the base frame selection module 120 may invoke an object detection module to detect objects of interest within the N image frames. k and image frame N x The base frame selection module 120 then selects an object of interest from image frame N x and image frame N k A unique identifier can be assigned to each

[0051] Further, at block 310, the base frame selection module 120 may order the N image frames by rotation angle, thereby generating an ordered set of N image frames. To do this, the base frame selection module 120 may evaluate metadata associated with each image frame and then determine, based on the metadata, the rotation angle of the camera that captured the image frame when capturing the image frame. In some implementations, the base frame selection module 120 may order the N image frames in ascending order. That is, for any given image frame in the ordering, the next image frame in the ordering will have a rotation angle equal to or greater than the magnitude of the rotation angle of the given image frame. In other implementations, the base frame selection module 120 may order the N image frames in descending order.

[0052] After ordering the N image frames by rotation angle (or while ordering the N image frames by rotation angle), the base frame selection module 120 may set the value of a variable K to one.

[0053] In block 320, the base frame selection module 120 may set the value of the variable X to 1. Next, the base frame selection module 120 may select the image frame N x can be added to the set of selected base frames, image frame N x corresponds to the Xth image frame from an ordered set of N image frames.

[0054] In block 330, the base frame selection module 120 may increment the value of K by 1. That is, the base frame selection module 120 may set the variable K=K+1. In block 340, the base frame selection module 120 may then select the image frame N x is image frame N kand determining whether the image frame N includes a unique identifier that is different from k corresponds to the Kth image frame from the ordered set of N image frames. In accordance with the above description, the base frame selection module 120 uses the stored metadata to select the Kth image frame from the ordered set of N image frames. x is image frame N k In another example, the base frame selection module 120 may call an object detection module to determine whether the image includes a unique identifier different from the image. Frame N x is image frame N k In either case, it can be established whether image frame N contains a unique identifier different from that of image frame N. x is image frame N k If the base frame selection module 120 determines that image frame N includes a unique identifier different from the base frame N, then the method 300 may proceed to block 350. x is image frame N k If it determines that the unique identifier does not differ from the unique identifier, the method 300 may return to block 330 .

[0055] As an illustrative example related to blocks 330 and 340, FIG. 4 shows an example scenario 400 including image frame 412, image frame 414, image frame 416, and image frame 422. In scenario 400, image frames 412, 414, 416, and 422 each have an index within a set of N ordered image frames. That is, image frame 412 has a position of 1, image frame 414 has a position of 2, image frame 416 has a position of 3, and image frame 422 has a position of 4. Additionally, each of image frames 412, 414, 416, and 422 is shown to include at least one unique identifier. That is, image frames 412, 414, and 416 include unique identifier 402, and image frame 422 includes unique identifiers 402 and 404.

[0056] During scenario 400, base frame selection module 120 may designate image frame 412 as the starting image frame. Base frame selection module 120 may then evaluate image frame 414 to determine that image frame 414 has the same unique identifier as image frame 412. Upon making this determination, base frame selection module 120 may iterate through image frame 414 and evaluate image frame 416. Similarly, base frame selection module 120 may determine that image frame 416 has the same unique identifier as image frame 414. Upon making this determination, base frame selection module 120 may iterate through image frame 416 and then evaluate image frame 422. At this point, base frame selection module 120 may determine that image frame 422 includes a different unique identifier than image frame 416 and may therefore stop its iteration. The image frames repeated during scenario 400 (e.g., image frames 412, 414, and 416) may be considered herein to be a subset of image frame 410.

[0057] Referring again to FIG. 3, in block 350, the base frame selection module 120 selects image frame N x and image frame N k-1 The quality measure associated with each image frame between image frame N can be evaluated. k-1 corresponds to the (K-1)th image frame from the ordered set of N image frames. After performing the evaluation, the base frame selection module 120 selects the image frame N x and image frame N k-1 Alternatively, the base frame selection module 120 may select the image frame associated with the highest quality measure between image frames N x and image frame N k-1may select all image frames having a threshold high quality metric (e.g., having an associated quality metric greater than X) between X and X. In either scenario, the base frame selection module 120 may add the selected image frame(s) to a set of selected base frames.

[0058] An example related to block 350 is shown in scenario 400 of Figure 4. In particular, image frames 412, 414, 416, and 422 are each shown to have an associated quality measure. That is, image frame 412 has an associated quality measure of 7, image frame 414 has an associated quality measure of 8, image frame 416 has an associated quality measure of 5, and image frame 422 has an associated quality measure of 5. During scenario 400, the base frame selection module 120 selects an associated quality measure for each image frame that it iterates through. In other words, the base frame selection module 120 may evaluate a quality metric for each image frame in the image frame subset 410. Thus, because image frame 414 has the highest quality metric among the frames in the image frame subset 410, the base frame selection module 120 may select image frame 414 for use as a base frame and will refrain from selecting image frames 412 and 416 for use as base frames.

[0059] Referring again to FIG. 3, in block 360, the base frame selection module 120 may set the value of the variable X equal to the value of the variable K.

[0060] At block 370, the base frame selection module 120 may determine whether the value of the variable K is less than N (i.e., the number of image frames received at block 310). If the base frame selection module 120 determines that the value of the variable K is less than N, the method 300 may return to block 330. Otherwise, if the base frame selection module 120 determines that the value of the variable K is greater than or equal to N, the method 300 may proceed to block 380.

[0061] At block 380, the base frame selection module 120 may provide the set of selected base frames determined from blocks 310-370. In some cases, the base frame selection module 120 may provide the set of selected base frames to the stitching module 130. In other cases, the base frame selection module 120 may provide the set of selected base frames to a remote computing device.

[0062] The blocks presented in Figure 3 are used for example purposes and are not intended to be limiting with respect to the embodiments herein. The operation of the base frame selection module 120 may be highly configurable and may include more, fewer, or different blocks than those shown in the method 300. In some cases, one or more blocks may be customized or may deviate in other aspects from the above described examples.

[0063] 5 illustrates a method 500 according to an example embodiment. The method 500 can be implemented to stitch together one or more base frames to create a composite image. The method 500 can be performed by various components of the computing device 100, such as the stitching module 130 and / or other components. For brevity, the example implementation of the method 500 is described herein using the stitching module 130. However, it should be understood that the disclosed principles can be applied in other scenarios with other components.

[0064] Method 500 may begin at block 510, where stitching module 130 receives N base frames. In accordance with the above description, the N base frames may be base frames selected by base frame selection module 120. Alternatively and / or additionally, the N base frames may be base frames communicated to computing device 100 from a remote computing device, such as a server device operating on a remote network.

[0065] After receiving the N base frames, the stitching module 130 may perform feature and keypoint detection on each of the N base frames. More specifically, for each base frame, the stitching module 130 may detect a set of local features that describe points of interest (e.g., keypoints) within the base frame. Various techniques can be used to efficiently detect keypoints, including scale invariant feature transform (SIFT), speeded up robust features (SURF), KAZE, and oriented FAST and rotational BRIEF (ORB), among other possibilities. Once the keypoints and their associated descriptions are obtained, the stitching module 130 can match keypoints from different base frames to determine pairs of overlapping base frames, e.g., base frames that include at least a number of overlapping regions. Various approaches can be used to efficiently match keypoints, including cascade hashing, k-nearest neighbor based approaches, and brute force matchers, among other possibilities.

[0066] In block 520, the stitching module 130 may select an initial pair of base frames from the pairs of overlapping base frames determined in block 510. In some implementations, the stitching module 130 may select the pair of base frames with the most keypoint matches to be the initial pair. In other implementations, the stitching module 130 may select the pair of base frames with the highest synthesis quality metric to be the initial pair. Other implementations are possible. After selecting the initial pair of base frames, the stitching module 130 may apply triangulation to determine three-dimensional (3D) coordinates of keypoints in the initial pair of base frames. A direct linear triangulation approach, among other possibilities, may be used. Triangulation can be performed using a variety of approaches, including a linear triangulation approach, a midpoint triangulation approach, and a non-linear triangulation approach.

[0067] In block 530, the stitching module 130 may incrementally add base frames to the initial 3D coordinates of block 520. More specifically, for each newly added base frame, the stitching module 130 may evaluate the correspondence between key points of the new base frame and key points of the previously added base frame, and then apply triangulation to determine the 3D coordinates of the new key points. Furthermore, for each newly added base frame, the stitching module 130 may apply bundle adjustment to reduce inaccuracies and generate optimal values ​​of the 3D coordinates. The operations of block 530 may be repeated until all N base frames have been evaluated.

[0068] As an example related to block 530, FIG. 6 illustrates an example scenario 600 including base frames 610, 612, and 614. FIG. 6 illustrates how base frames 610 and 612 share a 3D coordinate 630, how base frames 610, 612, and 614 share a 3D coordinate 632, and how base frames 612 and 614 share a 3D coordinate 634. FIG. 6 also illustrates how base frame 610 has a 3D coordinate 620 that is not shared with any other base frame, and how base frame 614 has a 3D coordinate 622 that is not shared with any other base frame. Additionally, FIG. 6 illustrates how 3D coordinates 620, 622, 630, 632, and 634 can all be projected to form a composite image 640.

[0069] 5, in block 540, stitching module 130 may project the 3D coordinates calculated in block 530 onto a panoramic coordinate system. The panoramic coordinate system may be selected with respect to one of the N base frames. Once the 3D coordinates are mapped onto the panoramic coordinate system, stitching module 130 may blend pixels from the N base images onto the panoramic coordinate system. In some embodiments, Blending may include a seam finding process that places seams in overlapping regions between pairs of base frames such that the transition from one base frame to another is smooth and less visible. In some embodiments, the seam finding process includes adding a computational bias to seams that include pixels from one or more identified regions of interest. For example, the computational bias may include adding a penalty term to any seams that include pixels from one or more identified regions of interest.

[0070] As an example related to block 540, FIG. 7 shows two example composite images, composite image 710 and composite image 720. Both composite images 710 and 720 include a region of interest 712 and a region of interest 714. Note that in composite image 710, a portion of seam 716 is located on region of interest 712 and a portion of seam 718 is located on region of interest 714. In accordance with the above discussion, such positioning of seams 716 and 718 may result in undesirable artifacts appearing in regions of interest 712 and 714. In contrast, note that in composite image 720, seam 726 is not located on region of interest 712 and seam 728 is not located on region of interest 714. This is a result of the penalty terms discussed above and may result in a composite image that includes a higher quality region of interest.

[0071] 5, in block 550, stitching module 130 may locate all overlapping regions in the panoramic projection from block 540 and then compute an optical flow field for each of these overlapping regions. In some embodiments, the optical flow field is computed by dividing each overlapping region into a grid of non-overlapping cells and representing the flow of a pixel within a cell as a bilinear combination of the flows at the four corners of the cell that contains it.

[0072] After computing the optical flow field, stitching module 130 can apply the optical flow field to simultaneously align all of the corresponding 3D coordinates from the overlapping regions of blocks 530. Stitching module 130 can then reproject the 3D coordinates onto the panoramic coordinate system to create the final composite image.

[0073] At block 560, the stitching module 130 may provide the composite image determined at block 550. In some cases, the stitching module 130 may provide the composite image to the display 112, which may display the composite image to a user. In other cases, the stitching module 130 may provide the composite image to a remote computing device via the network interface 114.

[0074] IV. Example of operation 8 illustrates a method 800 according to an example embodiment. The method 800 may comprise various blocks or steps. The blocks or steps may be performed individually or in combination. The blocks or steps may be performed in any order and / or sequentially or in parallel. Furthermore, blocks or steps may be omitted or added to the method 800. The blocks of the method 800 may be performed by various elements of the computing device 100 as illustrated and described with reference to FIG. 1.

[0075] Block 810 may include acquiring a plurality of image frames. In some embodiments, the plurality of image frames are captured in one continuous stream by a camera device. Further, in some embodiments, the plurality of image frames are captured using a front camera of the camera device.

[0076] Block 820 may include identifying one or more regions of interest in one or more image frames of the plurality of image frames, in some embodiments, each of the one or more regions of interest corresponds to a region that includes a face.

[0077] Block 830 may include selecting a set of base frames based on a respective quality metric associated with each image frame of the plurality of image frames, wherein each identified region of interest of the identified one or more regions of interest is entirely contained within at least one base frame of the selected set of base frames.

[0078] Block 840 may include stitching together the set of selected base frames to create a composite image.

[0079] In some embodiments, the respective quality measure associated with each image frame of the plurality of image frames is a metric based on at least one of motion blur of the image frame, focus of the image frame, or exposure of the image frame. Further, in some embodiments, the respective quality measure is computationally biased to give greater weight to pixels located within the identified one or more regions of interest.

[0080] In some embodiments, selecting the set of base frames includes determining a plurality of subsets of image frames from the plurality of image frames, each of the subsets including image frames that include the same one or more regions of interest, and the selecting further includes selecting a base frame from each of the subsets based on a respective quality measure associated with each image frame in the subset.

[0081] In some embodiments, selecting a base frame from each of the subsets comprises selecting, from among the image frames of the subset, the image frame having the highest quality measure.

[0082] In some embodiments, identifying the one or more regions of interest includes determining a unique identifier for each of the one or more regions of interest, and each of the subsets includes one or more image frames that include the same unique identifier.

[0083] In some embodiments, each image frame of the plurality of image frames is captured by the camera device at a respective rotation angle. In such embodiments, determining the subset of image frames includes ordering the plurality of image frames based on the rotation angle, designating a starting image frame from the plurality of image frames, and repeating the plurality of image frames according to the ordering starting with the starting image frame until a next image frame to be repeated is an image frame having at least one unique identifier different from the starting image frame. In such embodiments, the subset of image frames are the repeated image frames.

[0084] In some embodiments, the rotation angle includes a measure based on the horizontal angle of the camera device. In some embodiments, designating the starting image frame includes designating the first image frame from the ordering.

[0085] In some embodiments, the subset of image frames is a first subset of image frames. In such embodiments, determining the second subset of image frames includes designating a second starting image frame from the plurality of image frames and determining, starting with the second starting image frame, that the next image frame to be repeated is different from the second starting image frame. and repeating the plurality of image frames according to the ordering until an image frame having at least one unique identifier that matches the first subset of image frames is obtained. In such an embodiment, the second subset of image frames are the repeated image frames starting from the second starting image frame.

[0086] In some embodiments, designating the second starting image frame includes designating an image frame having at least one unique face identifier different from the starting image frame.

[0087] In some embodiments, the stitching occurs after each image frame from the plurality of image frames has been repeated at least once.

[0088] In some embodiments, stitching includes a seam finding process that includes adding a computational bias to seams that include pixels from one or more of the identified regions of interest. In such embodiments, the computational bias may include adding a penalty term to any seams that include pixels from one or more of the identified regions of interest.

[0089] Some embodiments include an optical flow seam repair step that includes determining overlapping regions of a selected set of base frames, computing a respective optical flow field for each of the overlapping regions, and applying the computed optical flow fields to align the overlapping regions.

[0090] The particular arrangement shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or less of each element shown in a given figure. Further, some of the illustrated elements may be combined or omitted. Additionally, illustrative embodiments may include elements not shown in the figures.

[0091] The steps or blocks representing the processing of information may correspond to circuitry configurable to perform a particular logical function of a method or technique described herein. Alternatively or additionally, the steps or blocks representing the processing of information may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more instructions executable by a processor to perform a particular logical function or action in a method or technique. The program code and / or associated data may be stored in any type of computer-readable medium, such as a storage device, including a disk, hard drive, or other storage medium.

[0092] The computer readable medium may also include non-transitory computer readable media, such as a register memory, a processor cache, and a computer readable medium that stores data for a short period of time, such as a random access memory (RAM). The computer readable medium may also include a non-transitory computer readable medium that stores program code and / or data for a longer period of time. Thus, the computer readable medium may include secondary or persistent long-term storage, such as, for example, a read-only memory (ROM), an optical or magnetic disk, a compact disk read-only memory (CD-ROM), etc. The computer readable medium may also be any other volatile or non-volatile storage system. The computer readable medium may be considered a computer readable storage medium, or, for example, a tangible storage device.

[0093] Various examples and embodiments have been disclosed; however, other examples and embodiments will be apparent to those of ordinary skill in the art. The various disclosed examples and embodiments are intended to be illustrative and not limiting. It is not intended to be exhaustive, and the true scope is indicated by the following claims.

Claims

1. 1. A computer-implemented method, comprising: A computing device acquiring a plurality of image frames; the computing device identifying one or more regions of interest in one or more image frames of the plurality of image frames; and selecting a set of base frames based on a respective quality metric associated with each image frame of the plurality of image frames, wherein each identified region of interest of the one or more identified regions of interest is entirely contained within at least one base frame of the selected set of base frames, the method further comprising: The method comprises the computing device stitching together the selected set of base frames to create a composite image.

2. The computer-implemented method of claim 1 , wherein each of the one or more regions of interest corresponds to a region that includes a face.

3. 2. The computer-implemented method of claim 1, wherein the respective quality measures associated with each image frame of the plurality of image frames include a metric based on at least one of motion blur of the image frame, focus of the image frame, or exposure of the image frame.

4. The computer-implemented method of claim 1 , wherein the respective quality measures are computationally biased to give greater weight to pixels that are located within the identified one or more regions of interest.

5. Selecting the set of base frames comprises: determining a plurality of subsets of image frames from the plurality of image frames, each of the subsets including image frames that include the same one or more regions of interest, the selecting further comprising: The computer-implemented method of claim 1 , comprising selecting a base frame from each of the subsets based on the respective quality measures associated with each image frame in the subset.

6. The computer-implemented method of claim 5 , wherein selecting the base frame from each of the subsets comprises selecting the image frame from among the image frames of the subsets that has the highest quality measure.

7. 6. The computer-implemented method of claim 5, wherein identifying the one or more regions of interest comprises determining a unique identifier for each of the one or more regions of interest, and each of the subsets comprises one or more image frames that include a same unique identifier.

8. Each image frame of the plurality of image frames is captured by a camera device at a respective rotational angle, and determining the subset of image frames includes: ordering the plurality of image frames based on a rotation angle; designating a start image frame from the plurality of image frames; and repeating the plurality of image frames according to the ordering, starting with the starting image frame, until a next image frame to be repeated is an image frame having at least one unique identifier different from the starting image frame, wherein the subset of image frames includes the repeated image frames. How to wear it.

9. The computer-implemented method of claim 8 , wherein the rotation angle comprises a measure based on a horizontal angle of the camera device.

10. The computer-implemented method of claim 8 , wherein specifying the starting image frame includes specifying a first image frame from the ordering.

11. The subset of image frames is a first subset of image frames, and determining a second subset of image frames includes: designating a second start image frame from the plurality of image frames; beginning with the second starting image frame and repeating the plurality of image frames in the ordering until a next image frame to be repeated is an image frame having at least one unique identifier different from the second starting image frame; The computer-implemented method of claim 8 , wherein the second subset of image frames includes the image frames repeated starting from the second starting image frame.

12. The computer-implemented method of claim 11 , wherein designating the second starting image frame includes designating the image frame having at least one unique face identifier different from the starting image frame.

13. The computer-implemented method of claim 8 , wherein the stitching occurs after each image frame from the plurality of image frames has been repeated at least once.

14. The computer-implemented method of claim 1 , wherein the stitching comprises a seam-finding process that includes adding a computational bias to seams that include pixels from the identified one or more regions of interest.

15. The computer-implemented method of claim 14 , wherein the computational biasing comprises adding a penalty term to any seam that includes pixels from one or more identified regions of interest.

16. The method further includes an optical flow seam repair step, the optical flow seam repair step comprising: determining an overlapping region of the selected set of base frames; calculating a respective optical flow field for each of the overlapping regions; and applying the calculated optical flow field to align the overlapping regions.

17. The computer-implemented method of claim 1 , wherein the multiple image frames are captured in a continuous stream by a camera device.

18. The computer-implemented method of claim 1 , wherein the plurality of image frames are captured using a front-facing camera of a camera device.

19. 1. A computing device comprising: one or more processors; a non-transitory data storage device storing at least computer readable instructions that, when executed by the one or more processors, cause the computing device to perform operations; and a message, the operation comprising: acquiring a plurality of image frames; identifying one or more regions of interest in one or more image frames of the plurality of image frames; and selecting a set of base frames based on a respective quality metric associated with each image frame of the plurality of image frames, wherein each identified region of interest of the one or more identified regions of interest is entirely contained within at least one base frame of the selected set of base frames, the operations further comprising: stitching the selected set of base frames together to create a composite image.

20. 1. An article of manufacture comprising non-transitory data storage storing at least computer readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations, the operations including: acquiring a plurality of image frames; identifying one or more regions of interest in one or more image frames of the plurality of image frames; and selecting a set of base frames based on a respective quality metric associated with each image frame of the plurality of image frames, wherein each identified region of interest of the one or more identified regions of interest is entirely contained within at least one base frame of the selected set of base frames, the operations further comprising: stitching together a selected set of said base frames to create a composite image.

Images