System and method for generating astronomical delayed video on user device

Abstract

A method (1300) and system (201) for generating astronomical delay video on a user device is disclosed. First, at least one current frame comprising one or more stars is captured via a user equipment. Further, one or more regions having a degree of exposure and focus greater than or less than a predefined threshold are determined, and a set of subsequent frames including the star is captured. Exposure and focus of one or more regions in a subsequent frame are corrected during capture based on the degree of exposure and focus of one or more regions within a current frame. Further, fusion weights for the current frame and subsequent frames are determined based on the one or more cloud categories and the associated cloud density scores. Finally, an astronomical delayed video is generated by fusing at least one current frame and a set of subsequent frames based on the determined fusion weights.

Description

Technical Field

[0001] Embodiments disclosed herein relate to astrophotography and astrovideography, and more particularly, to a system and method for generating astronomical time-lapse videos in real time on a user device. Background Art

[0002] Astrophotography and astrovideography involve the recording of celestial bodies and related events through images and videos. Astrophotography and astrovideography have recently become increasingly popular not only among professionals but also among the general public. For example, users can capture the dynamic movement of celestial objects, such as planetary migrations, solar eclipses, and star trails. Today, electronic devices and cameras (such as smartphone cameras) have various configurations, such as exposure time, focus, and hyperlapse, to capture this dynamic movement.

[0003] Traditionally, due to the slow relative motion of celestial bodies, a camera's hyperlapse mode is often used to capture video. Furthermore, the exposure time is set relatively high. To properly capture the video, the camera must remain stationary to avoid undesirable trajectories caused by the high exposure time. A tripod is often used to support the associated equipment used to capture the video. Initially, an electronic device with a camera can capture a scene using specific values, such as exposure, aperture, and focus. The camera can then automatically capture multiple images in conjunction with a time interval schedule.

[0004] Next, to improve the quality of the video output, post-processing tasks are typically performed on the captured video. Post-processing tasks include object masking (to remove blur from celestial object paths), removal of undesired objects, celestial object redrawing, frame blending, contrast adjustment, sharpness adjustment, etc. In many cases, post-processing can be done manually using processing tools.

[0005] Conventional technologies have several drawbacks. For example, they do not allow for dynamic adaptation to changes in the scene being captured. Exposure and focus cannot be adjusted during capture. Furthermore, conventional technologies do not include effective real-time processing and rely heavily on post-processing of the captured video. Post-processing of the video output is not only undesirable and time-consuming, but also requires proficiency in editing tools.

[0006] Therefore, it is desirable to address the above-mentioned shortcomings or other deficiencies, or at least provide useful alternatives to overcome the above-mentioned shortcomings. Summary of the Invention Solution to the problem

[0007] This summary is provided to introduce in a simplified form a selection of concepts that are further described in the detailed description of the present disclosure. This summary is not intended to identify the basic inventive concept of the present disclosure nor to determine the scope of the present disclosure.

[0008] Disclosed herein is a method for generating an astronomical time-lapse video in a user device. The method includes capturing at least one current frame via the user device, wherein the at least one current frame includes one or more stars. Furthermore, the method includes determining, within the at least one current frame, one or more regions having an exposure and focus greater than or less than a predefined threshold. Furthermore, the method includes capturing a set of subsequent frames including the one or more stars, wherein, during capture, the exposure and focus of the corresponding one or more regions in the set of subsequent frames are corrected based on the exposure and focus of the one or more regions within the at least one current frame. Furthermore, the method includes determining a fusion weight for the at least one current frame and a set of subsequent frames based on one or more cloud categories and a cloud density score associated with each of the one or more cloud categories. Furthermore, the method includes generating an astronomical time-lapse video by fusing the at least one current frame with the set of subsequent frames based on the determined fusion weight.

[0009] Also disclosed herein is a system for generating an astronomical time-lapse video in a user device. The system includes a memory and at least one processor coupled to the memory. The at least one processor is configured to capture at least one current frame via the user device, wherein the at least one current frame includes one or more celestial bodies. Furthermore, the at least one processor is configured to determine, within the at least one current frame, one or more regions having exposure and focus levels greater than or less than a predefined threshold. Furthermore, the at least one processor is configured to capture a set of subsequent frames including the one or more celestial bodies, wherein, during capture, the exposure and focus of the corresponding one or more regions in the set of subsequent frames are corrected based on the exposure and focus levels of the one or more regions within the at least one current frame. Furthermore, the at least one processor is configured to determine fusion weights for the at least one current frame and a set of subsequent frames based on one or more cloud categories and a cloud density score associated with each of the one or more cloud categories. Furthermore, the at least one processor is configured to generate the astronomical time-lapse video by fusing the at least one current frame with the set of subsequent frames based on the determined fusion weights.

[0010] To further illustrate the advantages and features of the present disclosure, a more detailed description of the present disclosure will be presented by reference to specific embodiments of the present disclosure as shown in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the present disclosure and, therefore, should not be considered as limiting the scope of the present disclosure. The present disclosure will be described and illustrated with additional features and details in the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] These and other features, aspects, and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings, wherein like reference numerals refer to like parts throughout, and wherein:

[0012] Figure 1An exemplary overview of an environment including a user device and a scene that a user wishes to capture using the user device according to an embodiment of the present disclosure is shown;

[0013] Figure 2 A schematic block diagram of a system and a user equipment according to an embodiment of the present disclosure is shown;

[0014] Figure 3 shows a block diagram depicting modules for generating astronomical time-lapse videos according to an embodiment of the present disclosure;

[0015] Figure 4 A block diagram illustrating an operation flow of a depiction module according to an embodiment of the present disclosure;

[0016] Figure 5 is a block diagram illustrating an exemplary implementation of a transformation module according to an embodiment of the present disclosure;

[0017] Figure 6 is a diagram illustrating the appearance of stars based on proximity to true north in an exemplary captured frame according to an embodiment of the present disclosure;

[0018] Figure 7 is a schematic diagram illustrating an exemplary distribution of trajectory length weights of stars in an exemplary captured frame according to an embodiment of the present disclosure;

[0019] Figure 8 is a schematic diagram illustrating an exemplary star trajectory correction based on star orbit estimation according to an embodiment of the present disclosure;

[0020] Figure 9 is a schematic diagram illustrating exemplary segmentation performed by a segmentation module according to an embodiment of the present disclosure;

[0021] Figure 10A is a block diagram illustrating a focus and exposure determination module according to an embodiment of the present disclosure;

[0022] Figure 10B is a schematic diagram showing division of preferred type areas by depicting focus and exposure determination modules according to an embodiment of the present disclosure;

[0023] Figure 11 is a schematic diagram illustrating image quality improvement of one or more regions on consecutive frames according to an embodiment of the present disclosure;

[0024] Figure 12A is a schematic diagram illustrating an exemplary implementation of a customized alpha blending technique for long star trails according to an embodiment of the present disclosure;

[0025] Figure 12Bis a schematic diagram illustrating an exemplary implementation of a customized alpha blending technique for short star trails according to an embodiment of the present disclosure; and

[0026] Figure 13 An exemplary process flow of a method for generating astronomical time-lapse videos on a user device according to an embodiment of the present disclosure is shown.

[0027] Furthermore, the skilled artisan will appreciate that the elements in the accompanying drawings are illustrated for simplicity and may not necessarily be drawn to scale. For example, a flow chart illustrates the most important steps involved in a method to help improve understanding of various aspects of the present disclosure. Furthermore, with respect to the configuration of an apparatus, one or more components of the apparatus may be represented in the accompanying drawings by conventional symbols, and the accompanying drawings may show only those specific details that are relevant to understanding the embodiments of the present disclosure, so as not to obscure the drawings with details that would be readily understood by a person of ordinary skill in the art having the benefit of the description herein. DETAILED DESCRIPTION

[0028] To facilitate an understanding of the principles of the present disclosure, reference will now be made to various embodiments, and specific language will be used to describe these embodiments. It should be understood at the outset that although illustrative implementations of embodiments of the present disclosure are shown below, the present disclosure may be implemented using any number of techniques, whether currently known or currently existing. The present disclosure is not necessarily limited to the illustrative implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the scope of the present disclosure.

[0029] Those skilled in the art will understand that both the foregoing general description and the following detailed description are illustrative of the present disclosure and are not intended to limit the present disclosure.

[0030] Throughout this specification, reference to "on one hand," "on the other hand," or similar language indicates that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases "in an embodiment," "in another embodiment," and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0031] It should be understood that as used herein, terms such as "comprises," "comprising," "having," etc. are intended to indicate that one or more of the listed features or elements are within the defined elements, but the elements are not necessarily limited to the listed features and elements, and additional features and elements may be within the meaning of the defined elements. Conversely, terms such as "consisting of" are intended to exclude unlisted features and elements.

[0032] The embodiments herein and their various features and advantageous details are more fully described with reference to the non-limiting embodiments shown in the accompanying drawings and described in detail in the following description. Descriptions of well-known components and processing techniques are omitted so as not to unnecessarily obscure the embodiments herein. In addition, since some embodiments can be combined with one or more other embodiments to form new embodiments, the various embodiments described herein are not mutually exclusive. As used herein, the term "or" refers to a non-exclusive or unless otherwise indicated. The examples used herein are merely intended to help understand how the embodiments herein can be practiced and to further enable those skilled in the art to practice the embodiments herein. Therefore, the examples should not be interpreted as limiting the scope of the embodiments herein.

[0033] According to the tradition of the art, embodiments can be described and illustrated in terms of blocks that perform one or more functions described. These blocks (which may be referred to herein as units or modules, etc.) are physically implemented by analog or digital circuits (e.g., logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.) and can optionally be driven by firmware and software. The circuits can be implemented, for example, in one or more semiconductor chips, or on substrate supports such as printed circuit boards. The circuits that constitute the blocks can be implemented by dedicated hardware, by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware that performs some functions of the block and a processor that performs other functions of the block. Without departing from the scope of this disclosure, each block of the embodiment can be physically divided into two or more interactive and discrete blocks. Similarly, without departing from the scope of this disclosure, the blocks of the embodiment can be physically combined into more complex blocks.

[0034] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. Therefore, the present disclosure should be interpreted as being able to extend to any changes, equivalents and substitutes other than those specifically described in the accompanying drawings. Although the terms first, second, etc. can be used in this article to describe various elements, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0035] The present disclosure is directed to methods and systems for generating astronomical time-lapse videos on a user device.

[0036] Figure 1An exemplary overview of an environment 100 including a user device 101 and a scene 103 that a user wishes to capture using the user device 101 is shown according to an embodiment of the present disclosure. The scene 103 may refer to any real view intended to be captured via the user device 101. In some embodiments, the scene 103 may be associated with the sky (e.g., the night sky). The scene 103 may include one or more objects (e.g., but not limited to, stars). In some embodiments, the scene 103 may also include additional objects (e.g., birds, aircraft, etc.). In some embodiments, a tripod ( Figure 1 ) supports the user device while capturing scene 103.

[0037] User equipment 101 may include a system ( Figure 1 For example, the system can communicate with a user device 101. The user device 101 can be associated with a user and can include, but is not limited to, a smartphone, a tablet, a professional camera (e.g., a digital single-lens reflex (DSLR) camera), and any other electronic device including a camera configured to help a user capture images and videos of a scene 103.

[0038] refer to Figure 2 , which shows a schematic block diagram 200 of a system 201 and a user device 101 according to an embodiment of the present disclosure. The system 201 can be integrated within the user device 101. In some embodiments, the system 201 can be a standalone entity located at a remote location and connected to the user device 101 via any suitable network. For example, the system 201 can be implemented on a physical server (not shown) or in a cloud-based architecture and communicatively coupled to the user device 101. In some embodiments, the system 201 can be implemented in a distributed manner, in which one or more components of the system 201 can be implemented within the user device 101, while one or more components of the system 201 can be implemented within a cloud-based server or a physical server.

[0039] The system 201 may be configured to be able to perform at least reference Figures 3 to 13 One or more operations are described in detail to generate an astronomical time-lapse video via a user device.

[0040] like Figure 2As shown, system 201 may include a processor 203, memory 205, storage 207, camera 209, module 213, and sensor 215. Camera 209 may also include a hardware abstraction layer (HAL) 211. Camera 209 may be a single camera or a group of cameras configured to capture scene 103. User device 101 may include a user interface that enables a user to view scene 103 captured by camera 209. In some embodiments, camera 209 may be associated with a camera application within user device 101. In some embodiments, camera 209 may capture scene 103 in response to user input. User input may be, in non-limiting embodiments, a shutter button via a physical key or soft key available on the user device 101 interface. Camera 209 may also be associated with one or more adjustable configurations or settings (e.g., exposure, focus, hyperlapse, shutter speed, aperture, etc.).

[0041] In some embodiments, system 201 may also include an input / output (I / O) interface and transceiver (not shown). In an exemplary embodiment, processor 203 may be operably coupled to each of the I / O interface, module 213, transceiver, memory 205, and storage device 207. In one embodiment, processor 203 may include at least one data processor for performing processing in a virtual storage area network. Processor 203 may include a specialized processing unit (e.g., an integrated system (bus) controller, a memory management control unit, a floating point unit, a graphics processing unit, a digital signal processing unit, etc.). In one embodiment, processor 203 may include a central processing unit (CPU), a graphics processing unit (GPU), or both. Processor 203 may be one or more general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, servers, networks, digital circuits, analog circuits, combinations thereof, or other currently known or later developed devices for analyzing and processing data. Processor 203 may execute software programs (e.g., code manually generated (i.e., programmed) to perform desired operations).

[0042] The processor 203 may be configured to communicate with one or more input / output (I / O) devices via an I / O interface. In some embodiments, the processor 203 may communicate with the user equipment 101 using the I / O interface. In some embodiments, the I / O interface may be implemented within the user equipment 101. The I / O interface may utilize communication technologies such as Code Division Multiple Access (CDMA), High Speed ​​Packet Access (HSPA+), Global System for Mobile Communications (GSM), Long Term Evolution (LTE), WiMax, and the like.

[0043] System 201 can communicate with one or more I / O devices using an I / O interface. For example, an input device can be an antenna, a microphone, a touch screen, a touchpad, a storage device, a transceiver, a video device / source, etc. An output device can be a printer, a fax machine, a video display (e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), a plasma, a plasma display panel (PDP), an organic light emitting diode display (OLED), etc.), an audio speaker, etc. The transceiver can be configured to receive signals from user device 101 and / or send signals to user device 101.

[0044] Processor 203 may be configured to communicate with a communications network via a network interface. In an embodiment, the network interface may be an I / O interface. The network interface may connect to a communications network to enable connection between system 201 and user device 101 and / or the external environment. The network interface may utilize connection protocols including, but not limited to, direct connection, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 802.11a / b / g / n / x, and the like. Communications networks may include, but are not limited to, direct interconnection, a local area network (LAN), a wide area network (WAN), a wireless network (e.g., a wireless network using the Wireless Application Protocol), the Internet, and the like. System 201 may communicate with other devices using the network interface and the communications network. The network interface may utilize connection protocols including, but not limited to, direct connection, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 802.11a / b / g / n / x, and the like.

[0045] In some embodiments, the memory 205 may be communicatively coupled to the processor 203. The memory 205 may be configured to store data and instructions executable by the at least one processor 203. In one embodiment, the memory 205 may be located within the user device 101. In another embodiment, the memory 205 may be located within the system 201 remote from the user device 101. In yet another embodiment, the memory 205 may communicate with the processor 203 via a bus within the system 201. In yet another embodiment, the memory 205 may be remote from the processor 203 and may communicate with the at least one processor 203 via a network.

[0046] Memory 205 may include, but is not limited to, non-transitory computer-readable storage media (e.g., various types of volatile and non-volatile storage media, including, but not limited to, random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media, etc.). In one example, memory 205 may include cache or random access memory for at least one processor 203. In alternative examples, memory 205 is separate from at least one processor 203, such as a processor's cache memory, system memory, or other memory. Memory 205 may be an external storage device or database for storing data. Memory 205 may be operable to store instructions executable by at least one processor 203. The functions, actions, or tasks shown or described in the figures may be performed by at least one processor 203 executing the instructions stored in memory 205. These functions, actions, or tasks are independent of a particular type of instruction set, storage medium, processor, or processing strategy, and may be performed by software, hardware, integrated circuits, firmware, microcode, etc., operating alone or in combination. Similarly, processing strategies may include multi-processing, multi-tasking, parallel processing, etc.

[0047] In some embodiments, module 213 may be included in memory 205. Memory 205 may also include a database for storing data. Module 213 may include a set of instructions that can be executed to cause system 201 (specifically, processor 203 of system 201) to perform any one or more of the methods / processes disclosed herein. Module 213 may be configured to use the data stored in the database to perform the steps of the present disclosure. For example, module 213 may be configured to execute Figure 4 In an embodiment, each module 213 may be a hardware unit that may be external to the memory 205. In addition, the memory 205 may include an operating system for executing one or more tasks of the system 201 (such as performed by a general-purpose operating system).

[0048] Furthermore, the present disclosure contemplates a computer-readable medium that includes instructions or receives and executes instructions in response to a propagated signal. Furthermore, instructions may be sent or received over a network via a communication port or interface or using a bus (not shown). The communication port or interface may be part of the processor 203 or a separate component. The communication port may be created in software or may be a physical connection in hardware. The communication port may be configured to connect to a network, external media, a display, or any other component in the system, or a combination thereof. The connection to the network may be a physical connection (e.g., a wired Ethernet connection) or may be established via wireless technology. Similarly, additional connections to other components of the system 201 may be physical or may be established via wireless technology. Alternatively, the network may be directly connected to the bus. For the sake of brevity, the architecture and standard operation of the operating system, memory 205, processor 203, transceiver, and I / O interface will not be discussed in detail.

[0049] One or more sensors 215 may include, but are not limited to, an inertial measurement unit (IMU) and a global positioning system (GPS) sensor. An IMU may refer to a combination of an accelerometer and a gyroscope that measures the motion and orientation of a device in three-dimensional space. According to embodiments of the present disclosure, data obtained from one or more sensors 215 may be utilized by module 213 of user device 101.

[0050] Module 213 may include a set of instructions that can be executed to enable user device 101 to generate an astronomical time-lapse video. Figure 3 and Figure 4 Module 213 and the module's operating procedures are described in detail.

[0051] Figure 3 Shown is a block diagram 300 of the depiction module 213 according to an embodiment of the present disclosure. Figure 4 A block diagram 400 depicting the operational flow 400 of the depiction module 213 is shown, and therefore for brevity and ease of reference, the diagram will be referred to as Figure 3 In addition, for ease of description and understanding, the reference numerals used for the same components remain the same throughout this disclosure.

[0052] The module 213 may include a transformation module 301 , a segmentation module 303 , a focus and exposure determination module 305 , an automatic IQ correction module 307 , a classification module 309 , a fusion weight determination module 311 and a star trajectory fusion module 313 .

[0053] Initially, user device 101 captures a frame including one or more stars via camera 209. In embodiments, capturing a frame may mean obtaining a frame by capturing one or more stars via camera 209. In embodiments, the captured frame includes at least one current frame and a set of subsequent frames. As described above, camera 209 includes HAL 211, which adjusts camera settings (e.g., focus, exposure, and white balance) based on the methods provided herein to capture optimal quality frames. The captured frame is then used as input by transformation module 301 and segmentation module 303.

[0054] In an embodiment, the transformation module 301 is configured to estimate star orbits and perform star shape correction and trajectory adjustment for one or more stars in the captured frame (as shown in step 401). According to an embodiment of the present disclosure, the transformation module 301 estimates star orbits (as shown in step 417) and performs star shape correction and trajectory adjustment based on the proximity of the pointing direction of the user device 101 to true north. The proximity to true north is used to determine a true north proximity factor. According to an embodiment of the present disclosure, the true north proximity factor corresponds to a factor that determines how close or far away one or more stars are from true north.

[0055] In an alternative embodiment, the transformation module 301 can estimate the star orbit and perform star shape correction and track length correction based on the proximity of the pointing direction of the user device 101 to the true south. In this case, a true south proximity factor is determined. The true south proximity factor can correspond to a factor that determines how close or far away one or more stars are from the true south. Figure 5 The transformation module 301 is described in more detail.

[0056] In an embodiment, the segmentation module 303 may be configured to segment each frame (received at step 401) into one or more regions, and to classify each region into one or more objects (e.g., but not limited to, sky, mountain, house, water, river, tree, etc.). Furthermore, based on the classification of the one or more segmented regions, the segmentation module 303 may identify a preferred type region and one or more non-preferred type regions from the one or more segmented regions. In an embodiment, the preferred type region may correspond to an area of ​​interest for capturing and generating astronomical time-lapse video. For example, the sky may be considered a preferred type region. The identified preferred type region and one or more non-preferred type regions may be collectively referred to as segmentation information.

[0057] The segmentation module 303 forwards the captured frames and segmentation information to the focus and exposure determination module 305 (as shown at step 403). Figure 9 The segmentation module 303 is described in more detail.

[0058] In an embodiment, the focus and exposure determination module 305 may be configured to determine one or more regions where the exposure and focus levels are greater than or less than a predefined threshold. For example, the focus and exposure determination module 305 may be configured to determine one or more regions where the exposure levels are greater than or less than a predefined threshold. For example, the focus and exposure determination module 305 may be configured to determine one or more regions corresponding to the subject for adjusting focus when the size of the area occupied by the subject to be focused is greater than or less than a predefined threshold. For example, the focus and exposure determination module 305 may be configured to determine one or more regions where the exposure levels and the area occupied by the subject to be focused are greater than or less than a predefined threshold. Specifically, the one or more regions are determined from the preferred type regions and may correspond to regions that can improve the image quality (IQ) of subsequently captured frames. The focus and exposure determination module 305 divides the preferred type regions into a plurality of blocks and compares the brightness of each block with the average brightness of the plurality of blocks to determine the exposure and focus levels.

[0059] Based on this comparison, the focus and exposure determination module 305 determines one or more regions corresponding to one or more blocks having exposure and focus levels greater than or less than a predefined threshold. In an embodiment, the focus and exposure determination module 305 makes this determination based on a true north proximity factor determined by and received from the transformation module (as shown at step 409). The determined one or more regions may be marked for IQ improvement and provided as input to the automatic IQ correction module 307 (as shown at step 405). This will be discussed below in conjunction with Figure 10A and Figure 10B The focus and exposure determination module 305 is described in more detail.

[0060] In an embodiment, the automatic IQ correction module 307 may be configured to determine a gain value that may be applied to improve the IQ of one or more regions in a subsequently captured frame. The automatic IQ correction module 307 provides the determined gain value to the HAL 211 (as shown at step 407), which causes the camera 209 to capture a set of subsequent frames including one or more celestial bodies, so that the exposure and focus of the corresponding one or more regions in the set of subsequent frames are corrected based on the exposure and focus of the one or more regions determined in at least one current frame during the capture. This will be described below in conjunction with Figure 11 The automatic IQ correction module 307 is described in more detail.

[0061] In an embodiment, the focus and exposure determination module 305 is configured to forward the captured frame and segmentation information received from the segmentation module 303 to the classification module 309 (as shown at step 411 ).

[0062] In an embodiment, the classification module 309 is configured to classify each block of the captured frame into one or more cloud categories. A cloud category can be a category used to categorize the state of the sky based on the degree of cloud coverage. Furthermore, each block of the captured frame is classified based on the degree of cloud coverage, and each block of the captured frame corresponds to one of the cloud categories (e.g., but not limited to, clear sky, partly cloudy, and overcast). Furthermore, the classification module 309 determines a cloud density score for each classified cloud category. The cloud density score can be a value representing the density of clouds in the frame. For example, the cloud density score can be determined based on the degree of cloud aggregation within the frame and / or the amount of area occupied by clouds within the frame. The cloud category and cloud density score are then provided as inputs to the fusion weight determination module 311. The classification module 309 will be described in more detail below.

[0063] In an embodiment, the fusion weight determination module 311 is configured to determine a fusion weight for each frame based on the classified cloud category and confidence score, for use in star trail fusion performed by the star trail fusion module 313. The fusion weight determination module 311 will be described in more detail below.

[0064] In an embodiment, the star trail fusion module 313 generates an astronomical time-lapse video by fusing at least one current frame with a set of subsequent frames based on the determined fusion weights received from the fusion weight determination module 311 at step 415, the corrected star shapes received from the transformation module 301 at step 417, and the estimated star orbits. The star trail fusion module 313 will be described in more detail below.

[0065] In an embodiment, system 201 may include an image format converter. The image format converter may be configured to convert an image from a first format (e.g., NV12 / NV21 YuV interleaved (YUV4-2-0)) to a second format (e.g., YUV 444, red, green, and blue (RGB)) to obtain more data per pixel, thereby improving image quality. The image format converter generates a video stream output that can be updated in an encoder (for video recording) and a display (for video preview and recording). The video stream output can be sent in the format expected by the display encoder.

[0066] Now combine Figures 5 to 12B Each of modules 213 is described in more detail.

[0067] Figure 5FIG5 is a block diagram 500 illustrating an exemplary implementation of the transformation module 301 according to an embodiment of the present disclosure. The transformation module 301 can be configured to generate star orbits and perform star shape correction and trajectory adjustment based on the celestial positions of one or more stars in a captured frame. The transformation module 301 can utilize sensor data to determine the pointing direction of the user device 101, represented by a quaternion. For example, the pointing direction of the user device 101 can include the pointing direction of the camera 209 of the user device 101. The quaternion represents a rotation about a predefined set of coordinate axes applied to the user device 101 that orients the camera 209 in a specific direction. Furthermore, the transformation module 301 can determine the celestial coordinates of one or more stars within the field of view (FOV) of the camera 209. The determined celestial coordinates are then used to estimate the star orbits and perform star shape correction. The transformation module 301 also includes submodules such as a device positioning module 501 and a position-based star detection module 503.

[0068] The device positioning module 501 can be configured to determine a representation of the pointing direction of the user device 101 in the form of a quaternion and detect whether there is global motion of the camera 209 relative to the scene or environment being captured. In an embodiment, the global motion of the camera 209 on the user device 101 refers to the overall movement and orientation of the camera 209 in three-dimensional space relative to a fixed reference point (e.g., the surface of the Earth or an external object). Global motion can include translation (i.e., movement in space) and rotation (i.e., change in orientation) of the camera 209.

[0069] In the device positioning module 501, sensor data is fused using predefined filters. Fusion is required to synchronize sensor data with respect to timestamps and to correct inaccuracies in readings from one sensor based on readings from another sensor. For example, readings from a gyroscope are susceptible to inaccuracies due to drift of the user device 101. These inaccuracies can be corrected using readings from an accelerometer. In an exemplary embodiment, a Kalman filter can be used to fuse the sensor data.

[0070] Afterwards, exponential smoothing of the filter output is performed and the pointing direction of the user device 101 in quaternion form is obtained using the following reference axes:

[0071] x-axis: perpendicular to the y-axis and z-axis and pointing roughly east;

[0072] y-axis: magnetic north (based on the heading direction of the magnetometer); and

[0073] z-axis: perpendicular to the ground (same as the accelerometer z-axis).

[0074] The obtained quaternion is then rotated and returned so that the y-axis points to true north instead of magnetic north. The rotation is performed by applying a standard magnetic model (e.g., World Magnetic Model (WMM)-2020) to the GPS coordinates of the user device 101 based on readings from the GPS sensor.

[0075] Additionally, the device positioning module 501 may be configured to detect the presence of global motion by analyzing sensor fusion data and GPS coordinates.

[0076] The position-based star detection module 503 can be configured to determine the celestial coordinates of one or more stars in the captured frame in a manner discussed below. In addition, the position-based star detection module 503 can be configured to determine a celestial coordinate transformation map from pixels to celestial coordinates, and vice versa.

[0077] The position-based star detection module 503 may be configured to determine the celestial coordinates of each pixel based on a predefined application programming interface (API) using the celestial coordinates of the center pixel of each frame and the FOV of the camera 209 .

[0078] First, the altitude-azimuth (Alt-az) coordinates of the center pixel of each frame are calculated based on the quaternion input from the device positioning module 501. According to an embodiment of the present disclosure, the altitude represents the angle between the pointing direction of the user device 101 and the horizontal line (xy plane), and the azimuth represents the angle between the pointing direction of the user device 101 and the true north pole (y axis). These values ​​are obtained directly from the quaternion. Afterwards, the right ascension-declination (RA-dec) coordinate conversion is performed based on the following equations (1), (2), and (3):

[0079] … (1)

[0080] (2)

[0081] (3), Where, H: local hour angle, Right Ascension, Local sidereal time, declination, A: azimuth, h: altitude, latitude.

[0082] Next, a set of predefined APIs based on the Flexible Image Transport System (FITS)-World Coordinate System (WCS) is obtained. FITS-WCS is a well-established standard for representing pixels of sky images as celestial coordinates. For each frame, this set of predefined APIs uses as input the RA-dec coordinates of the corresponding center pixel, the field of view (FOV) of camera 209, and a predefined star magnitude threshold to provide a celestial coordinate transformation map for one or more stars within the FOV of camera 209 and having a magnitude greater than the predefined star magnitude threshold. The predefined star magnitude threshold refers to the absolute magnitude, i.e., the brightness of a star from a distance of 10 parsecs from the observer, which can be calculated using the apparent magnitude (i.e., the brightness as observed from Earth) and the distance from the observer to the star in parsecs.

[0083] Therefore, assuming that T is the transformation map returned by the set of predefined APIs, (x, y) are pixel coordinates, and (Ra, Dec) are celestial coordinates for pixel (x, y), then T(x, y) = (Ra, Dec) and T'(Ra, Dec) = (x, y) are returned by the position-based star detection module 503.

[0084] The determined celestial coordinates of the one or more celestial bodies are used to correct a celestial body shape associated with the one or more celestial bodies and determine a trajectory length weight. In addition, the determined celestial coordinates are used together with a celestial coordinate transformation map to estimate a celestial body orbit associated with the one or more celestial bodies.

[0085] In embodiments of the present disclosure, two problems related to the positions of stars in astronomical time-lapse videos are solved.

[0086] In the first problem, when viewing stars in different parts of the sky, some stars appear to move faster than others. This shift occurs when stars closer to true north and true south appear to move slower than stars farther from true north and true south. Due to the long exposures used to capture astronomical time-lapse videos, stars farther from true north and true south appear to be elongated. To address and mitigate the first problem, embodiments of the present disclosure describe the following steps:

[0087] Step 1: Detect stars that are far from the true north and true south based on their celestial positions. For the detection of stars that are far from the true north and true south, the following equation (4) is used to determine the angular separation between each star (whose celestial coordinates have been determined) and the North Star (i.e., the North Star):

[0088] …(4),

[0089] in

[0090] ( ) = the celestial coordinates of the star; and

[0091] ( ) = (2h41m39s, +89°15'51"), which represents the coordinates of the North Star

[0092] Step 2: Determine a True North proximity factor for each celestial body based on the determined angular separation. The True North proximity factor is represented by a value between 0.0 and 1.0 and indicates how close or far away each celestial body is from True North. A True North proximity factor of 0.0 indicates no angular separation (i.e., at the North Star), while a True North proximity factor of 1.0 indicates an angular separation of 90 degrees (i.e., away from the North Star).

[0093] Step 3: The proximity factor is sent to the automatic IQ correction module 307 via the focus and exposure determination module 305 so that subsequently captured frames have exposure and focus adjusted based on the true north proximity factor. Sending the proximity factor to the automatic IQ correction module 307 enables the stretching to be reduced to some extent.

[0094] Step 4: Perform intra-frame elongation correction. Since the ideal star shape is circular, but the elongated star shape is elliptical, the star pixels and star elongation pixels can be corrected as follows:

[0095] If (x, y) represents the center pixel of a star, then the intensities of pixels (x+i, y), (xi, y), (x, y+i), and (x, yi) are determined, where i = 1, 2, 3, and so on. If the intensity of any of these pixels is below a predefined tunable threshold, then the i-th value of that pixel is taken as the radius of the star. For other nearby pixels outside this radius, the pixel's intensity is reduced to suppress elongation, or the pixel is replaced with a nearby background pixel.

[0096] In the second problem, when the captured frames are merged with stars at different distances from true north and true south, the trail length sometimes appears longer and sometimes shorter. Figure 6 Schematic diagram 600 illustrates the appearance of stars based on their proximity to true north in an exemplary captured frame 601. As shown, stars closer to true north have small trails (as indicated by block 603), while stars further from true north have long trails (as indicated by block 605). This inconsistency in trail length occurs because the static fusion weights used during the fusion of the captured frames do not account for the star's velocity. To address and mitigate the second issue, a trail length weight is assigned to each star in the corresponding frame based on the determined angular separation.

[0097] Figure 7FIG7 is a diagram 700 showing an exemplary assignment of trajectory length weights for stars in an exemplary captured frame 601 according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the trajectory length weights to be used during fusion may be assigned as follows:

[0098] For angular intervals between 0 and 35 degrees, a high trajectory length weight of 0.99 may be assigned;

[0099] For angular intervals between 35 and 60 degrees, a medium track length weight of 0.80 may be assigned; and

[0100] For angular intervals between 60 and 90 degrees, a low track length weight of 0.60 may be assigned.

[0101] According to an embodiment of the present disclosure, a high track length weight causes stars to remain longer in subsequent fused frames, resulting in increased track length. Conversely, a low track length weight has the opposite effect, thereby achieving a uniform track length. In addition, by correcting the star shape and determining the track length weight, the celestial coordinates determined by the position-based star detection module 503 are transformed into the celestial coordinates. Figure 1 It is used to estimate the orbits of stars associated with one or more stars. Figure 8 Describe star orbit estimation in more detail.

[0102] Figure 8 is a diagram 800 illustrating an exemplary star trajectory correction based on star trajectory estimation according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the celestial coordinates of one or more stars, a celestial coordinate transformation map, and information associated with the presence of global motion are used to estimate the stellar orbits of one or more stars in a captured frame. In an embodiment, the information associated with the presence of global motion and the stellar orbit estimates enable a determination to be made whether to skip fusion of a particular frame and draw a trajectory using the estimated stellar orbits, or to reset fusion based on global motion detection. Embodiments of the present disclosure address two main issues associated with the presence of global motion during the capture of astronomical time-lapse videos. These two main issues are described as follows:

[0103] Due to the long exposure, a given frame itself may have undesirable light trails.

[0104] If a frame with global motion is used for fusion, the resulting star trails may appear uneven, as shown by blocks 801a-801c in exemplary frame 801 with global motion, which illustrates a portion of the background problem. According to an embodiment of the present disclosure, the background problem shown in frame 801 is corrected by star trail correction, as shown by blocks 803a-803c in frame 803.

[0105] Therefore, frames captured during global motion are skipped during fusion.

[0106] When the global motion of the camera 209 is complete, the two most common situations that arise are described below:

[0107] Case 1: After the global motion is complete, camera 209 may be pointing to a different part of the sky. Alternatively, camera 209 may be pointing to the same part of the sky, but at a different orientation than before the global motion. In this case, a complete reset of the trajectory may be required because using the previous trajectory may result in randomly overlapping trajectories in different directions, leading to undesirable star trajectory output.

[0108] Case 2: After the global motion is complete, camera 209 may be pointing at the same part of the sky in the same orientation as before the global motion. In this case, because the new trajectory is a continuation of the old trajectory, a complete reset of the trajectory may not be necessary. However, due to the global motion, a few frames may be skipped, resulting in gaps between the old and new trajectories. Such gaps can be filled by drawing trajectories using the estimated star orbits.

[0109] The process for estimating star orbits stores the celestial coordinates of all stars in the captured frames to detect global motion in subsequent frames and determine whether to skip fusion or reset fusion after the global motion is completed. The process for estimating star orbits includes the following steps:

[0110] Step 1: Calculate the (Alt-az) coordinates of all stars based on the previously stored celestial coordinates. First, use the following equation (5) to calculate the local sidereal time ( ).based on , the coordinates are calculated based on equations (6) and (7) mentioned below.

[0111] (5), where

[0112] : Local sidereal time

[0113] JD: Julian Days, which is the number of days since January 1, 2000

[0114] L: longitude

[0115] UT: Universal Time

[0116]

[0117] …(7), where H: local hour angle, : right ascension, : local sidereal time, : declination, A: azimuth, h: altitude, :latitude

[0118] Step 2: Use the process described in step 1 to calculate the (Alt-az) coordinates of all stars based on the current celestial coordinate input.

[0119] Step 3: Compare the (Alt-az) coordinates obtained in the two steps above. When the compared coordinates do not match, this indicates the presence of global motion, as determined by image-based global motion detection. This indication, along with the input from sensor-based global motion detection, accurately detects the presence of global motion.

[0120] In an embodiment, if the presence of global motion is detected, a signal is generated to skip fusion of the current frame.In an alternative embodiment, if the presence of global motion is not detected in the current frame but is detected in a previous frame, both of the aforementioned cases are checked.

[0121] Step 4a: Calculate the (Alt-az) coordinates of all stars based on their celestial coordinates stored just before the global motion.

[0122] Step 4b: Compare the (Alt-az) coordinates calculated at step 4a with the (Alt-az) coordinates calculated in step 2.

[0123] Step 4c: If the (Alt-az) coordinates compared at step 4b do not match, it corresponds to case 1 and generates a signal for resending fusion. Alternatively, if the (Alt-az) coordinates compared at step 4b match, it corresponds to case 2 and fusion continues.

[0124] Step 5: Using the celestial coordinate transformation map input (T, T') obtained from the position-based star detection module 503 and the star's celestial coordinates (Ra, Dec), the pixel position (x, y) of each star is obtained. The pixel position of each star can then be used to draw the track when fusion is skipped or when color tracks are to be generated.

[0125] Thus, the transformation module 301 takes the captured frames and sensor data as input and produces star shape corrected frames, estimates of star orbits, and determinations of true north proximity factors. In addition to the transformation module 301, the captured frame input is also used by the segmentation module 303, which will be discussed below in conjunction with Figure 9 Provide a description.

[0126] Figure 9900 is a schematic diagram illustrating exemplary segmentation performed by segmentation module 303 according to an embodiment of the present disclosure. In an embodiment, segmentation module 303 may be configured to perform frame segmentation on at least one current frame and at least one of a set of subsequent frames by segmenting the corresponding frame into one or more regions and classifying each region into one or more objects. Segmentation may be performed using a predetermined fully convolutional network. In an embodiment, the one or more objects may correspond to the type of region depicted in the corresponding frame. For example, the one or more objects may include, but are not limited to, the sky, a mountain, a river, a lighthouse, a bird, an airplane, and a rock. Figure 9 An exemplary frame 901 is shown segmented into one or more objects (eg, sky, water, mountains, and trees).

[0127] In an embodiment, the segmentation module 303 determines segmentation information including identification of a preferred type region and one or more non-preferred type regions from the one or more segmented regions based on the classification of the one or more segmented regions. For example, the sky may be identified as a preferred type region, and mountains, rivers, lighthouses, birds, airplanes, and rocks may be identified as non-preferred type regions. Additionally, each pixel in the corresponding frame may be identified as part of an object category. This information may be later used by the automatic IQ correction module 307. Furthermore, the segmentation information and the captured frames are forwarded to the focus and exposure determination module 305, which will be discussed below in conjunction with the image processing module 306. Figure 10A and Figure 10B Provide a description.

[0128] Figure 10A is a block diagram 1000a illustrating the focus and exposure determination module 305 according to an embodiment of the present disclosure. Figure 10B 1000b illustrates the division of preferred type regions by the focus and exposure determination module 305 according to an embodiment of the present disclosure. In an embodiment, the focus and exposure determination module 305 may be configured to determine one or more regions in a captured frame where the exposure and focus levels are greater than or less than a predefined threshold. For example, the focus and exposure determination module 305 may be configured to determine one or more regions where the exposure levels are greater than or less than a predefined threshold. For example, the focus and exposure determination module 305 may be configured to determine one or more regions corresponding to the subject for adjusting focus when the size of the area occupied by the subject to be focused is greater than or less than a predefined threshold. For example, the focus and exposure determination module 305 may be configured to determine one or more regions where the exposure levels and the extent of the area occupied by the subject to be focused are greater than or less than a predefined threshold. In an embodiment, the exposure and focus levels may be determined based on the frame segmentation and a true north proximity factor associated with one or more celestial bodies in the captured frame.

[0129] To determine one or more regions, the focus and exposure determination module 305 divides the preferred type region into multiple blocks and compares the brightness of each block with the average brightness of the multiple blocks to obtain exposure and focus levels. Finally, one or more regions are determined corresponding to one or more blocks whose exposure and focus levels are greater than or less than a predefined threshold.

[0130] In the focus and exposure determination module 305, the preferred type area is divided into a plurality of blocks. In an exemplary embodiment, when the sky (i.e., the preferred type area) occupies less than 50% of the area of ​​the captured frame, the preferred type area can be divided into a finer grid of blocks, such as Figure 10B As shown, the width and height of each block are less than 200 pixels. In another exemplary embodiment, when the sky occupies more than 50% of the area of ​​the captured frame, the preferred type area can be divided into a coarse block grid so that the width and height of each block exceeds 200 pixels.

[0131] The individual blocks are then determined to be underexposed or overexposed by comparing the brightness of each block to the average brightness of the multiple blocks and determining one or more regions corresponding to one or more blocks having exposure and focus levels greater than or less than a predefined threshold. In an exemplary embodiment, if the brightness of the individual block is less than the average brightness of the multiple blocks by a predefined threshold (e.g., 20%), the individual block is determined to be underexposed. In another exemplary embodiment, if the brightness of the individual block is greater than the average brightness of the multiple blocks by a predefined threshold (e.g., 20%), the individual block is determined to be overexposed.

[0132] In an alternative embodiment, if a non-preferred type of area, including an object such as a mountain or a tree, occupies a significant portion (e.g., greater than 70% of the frame), the object is considered important in the context of the scene. In such a scenario, the focus can be shifted to the object instead of the sky. In another scenario where the object moves out of the frame, such as in the case of a car, the focus can be switched back to the sky.

[0133] The focus and exposure determination module 305 can calculate the optimal exposure and focus frame by frame in the preferred type area. Since the same scene may have different exposures in the preferred type area (sky) as the astronomical time-lapse video progresses, the frame-by-frame calculation is performed.

[0134] In embodiments, when the brightness variance of a block relative to the average brightness of a frame is high, the exposure and focus values ​​of the block may be corrected. In other scenarios, the exposure and focus values ​​(and therefore the exposure time) may be modified. For example, when stars are farther from true north and true south, they may appear to be moving faster than usual, which may result from a long exposure. To partially counteract this stretching effect, the exposure time may be reduced based on a factor related to the proximity to true north.

[0135] In one embodiment, the output of the focus and exposure determination module 305 (ie, the one or more identified regions) is provided to the automatic IQ correction module 307 .

[0136] In the automatic IQ correction module 307, for each block of one or more identified regions, a gain value is calculated based on the deviation from the average brightness of the plurality of blocks. The gain value may be calculated as follows:

[0137] For underexposed blocks, the gain value = 1 + (average brightness of multiple blocks - brightness of underexposed blocks) / 256.

[0138] For an overexposed block, the gain value = 1 - (average brightness of multiple blocks - brightness of underexposed blocks) / 256.

[0139] The determined gain value is sent to the HAL 211 to improve exposure and focus of the identified one or more regions in subsequent frames.

[0140] In embodiments, the weights for each block can be dynamically adjusted on a frame-by-frame basis. According to embodiments of the present disclosure, weights refer to thresholds associated with exposure and focus set for each block to achieve improved image quality. For example, certain areas in a frame (e.g., trees and mountains) may appear overexposed depending on the time of day and the direction of the light source. According to embodiments of the present disclosure, weighted metering techniques can be used to dynamically adjust the exposure of these areas, ensuring that they appear less bright and closer to the average brightness of multiple blocks.

[0141] In a typical working scenario, when the sky occupies a large portion of the frame, the focus is maintained at infinity. However, if an object occupies a large portion of the frame (greater than 70%), the focus is moved from infinity to the finite distance where the object is located. A custom weighted metering technique is used, and the average exposure is calculated for a fixed block across the entire frame. Based on the output of the semantic segmentation module 303, when the sky is determined to be a preferred type of region, the custom weighted metering technique is applied to the sky region.

[0142] Figure 11 1100 is a diagram illustrating IQ improvement for one or more regions across consecutive frames according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the exposure in each block is considered and the entire region is iteratively enhanced across consecutive frames by increasing the average exposure by a predetermined value (e.g., approximately 30%) compared to the entire frame. In this case, a custom weighted metering technique can be implemented to enhance the exposure value of the sky region, thereby iteratively achieving a better exposure.

[0143] In another embodiment, the output of the focus and exposure determination module 305 forwards the captured frame and segmentation information received from the segmentation module 303 to the classification module 309 .

[0144] The classification module 309 can be configured to classify each block of the corresponding frame into one or more cloud categories (e.g., clear sky, partly cloudy, and overcast). Furthermore, the classification module 309 can determine a cloud density score for each cloud category. The cloud density score can be a value representing the density of clouds in the frame. For example, the cloud density score can be determined based on the degree of cloud aggregation within the frame and / or the amount of area occupied by clouds within the frame.

[0145] To identify different cloud categories from the input frame, the classification module 309 converts the input frame into a feature vector representation using a rectified linear unit activation function. The classification module 309 then concatenates the generated feature vectors into a final feature vector and classifies the input frame into clear sky, partly cloudy, or overcast. In an embodiment, the classification module 309 may be implemented using a predetermined fully convolutional neural network. The network may be a cloud network model, which is an artificial intelligence model trained to classify each block in the input frame into multiple cloud categories. The network can predict a cloud category for all blocks in the input frame. Furthermore, for each predicted category, the network can also determine a cloud density score. For example, when the cloud network model is applied to a frame, the cloud network model outputs information indicating the frame classification, such as clear sky, partly cloudy, or overcast, and a confidence score. For example, the input frame is converted into a feature vector representation using a rectified linear unit activation function, the generated feature vectors are concatenated into a final feature vector, and the input frame is classified into clear sky, partly cloudy, or overcast. Furthermore, the cloud network model network can predict a cloud category for all blocks in the input frame. For each predicted category, the cloud network model determines a cloud density score. For example, an input frame is fed into a cloud network model that classifies the input frame into categories such as clear sky, partly cloudy, and overcast. For each dynamically partitioned tile in the input frame, the cloud network model determines a cloud density score for all predicted categories. Based on the cloud category and cloud density score, the fusion weight determination module 311 determines a fusion weight for the corresponding frame.

[0146] The fusion weight determination module 311 can be based on a multivariate linear regression neural network model that predicts customized fusion weights for star trajectory fusion. The multivariate linear regression neural network model is used to predict customized fusion weights for an input frame. Based on the input frame features and cloud density score, the model determines customized fusion weights for each dynamically partitioned tile in the frame. The steps followed by the fusion weight determination module 311 to determine customized fusion weights are described below:

[0147] Step 1: Initially, the input frame is dynamically divided into multiple blocks. These input features are denoted as X1, X2, ..., Xn.

[0148] Step 2: Using the cloud category, determine the cloud density score for each block of the frame. These density scores are denoted as W1, W2, ..., Wn.

[0149] Step 3: Train a multivariate linear regression neural network model based on the customized fusion weights and known training data for various input images.

[0150] Step 4: The input training images contain a mixture of various images with clear sky, partly cloudy, cloudy sky, mountains with clouds, trees with clouds, etc.

[0151] Step 5: Input customized fusion weight data based on the manually annotated data for the various training images mentioned above.

[0152] Step 6: The model analyzes the relationship between multiple input image features and cloud density scores to predict customized fusion weights.

[0153] Step 7: Predict the custom fusion weights by training the model based on the following equation: Y = W0 + W1X1 + W2X2 + … + WnXn, where W0 is the default starting weight (0.6) and will be a constant weight when the weights (W1 … Wn) are zero (assuming the sky is completely cloudless).

[0154] According to an embodiment of the present disclosure, an astronomical time-lapse video can be generated by fusing at least one current frame with a set of subsequent frames based on determined fusion weights, corrected star shapes, and estimated star orbits by the star trail fusion module 313. The star trail fusion module 313 includes a night fusion mode and a color fusion mode, and generates the astronomical time-lapse video in the night fusion mode or the color fusion mode based on input received from the user.

[0155] To generate an astronomical time-lapse video in night fusion mode, the star trail fusion module 313 performs the following steps:

[0156] Step 1: First, denote the shape-corrected video frame (current frame) as F1, F2, F3, ..., Fn, and denote the previous frame as P1, P2, P3, ..., Pn.

[0157] Step 2: Initially, the current frame F1 can be encoded directly to the output buffer and treated as the previous frame P1 for the next frame.

[0158] Step 3: Custom alpha blend the next frame F2 with the output of the previous frame P1, and treat the output of the current frame F2 as P2.

[0159] Step 4: Custom alpha blend the next frame F3 with the output of the previous frame P2, and treat the output of the current frame F3 as P3.

[0160] Step 5: Encode all output frames P1, P2, P3, ..., Pn into the output buffer, convert the buffer format and save it as a video file.

[0161] To generate an astronomical time-lapse video in color fusion mode, the star trajectory fusion module 313 performs the following steps:

[0162] Step 1: First, denote the shape-corrected video frame (current frame) as F1, F2, F3, ..., Fn, and denote the previous frame as P1, P2, P3, ..., Pn.

[0163] Step 2: Initially, a color map is applied to the current frame F1 based on the estimated star orbits and encoded directly to the output buffer and treated as the previous frame P1 for the next frame.

[0164] Step 3: For the next frame F2, the color map is applied and the custom alpha is blended with the output of the previous frame P1 and the output of the current frame F2 is considered as P2.

[0165] Step 4: For the next frame F3, the color map is applied and the custom alpha is blended with the output of the previous frame P2 and the output of the current frame F3 is considered as P3.

[0166] Step 5: Encode all output frames P1, P2, P3, ..., Pn into the output buffer, convert the buffer format and save it as a video file.

[0167] In order to apply the color map to the determined star orbit for star trajectory color fusion, the orbit data determined by the star orbit estimation module outputs a matrix in the form of a binary matrix, in which the value for the new star position is 1 and the value for other star positions is 0. Thereafter, the star orbit binary matrix is ​​used to identify the input frame source pixel position X. Thereafter, a random color map or random color value is applied to the source pixel position X. In an embodiment, a color map frame can be generated by multiplying the input pixel brightness by the random color value and normalizing it within the range (0, 255), so that the output (Y(x, y)) = normalized (input (X(x, y)) × random color value). In the example, the random color value is as follows:

[0168] Color value for green (0, 255, 0)

[0169] The color value for red is (255, 0, 0)

[0170] Color value for blue (0, 0, 255)

[0171] The above operation can be performed for all determined pixel positions. Custom alpha blending can be applied after this process.

[0172] Figure 12A is a diagram 1200a illustrating an exemplary implementation of a custom alpha blending technique for generating long star trails according to an embodiment of the present disclosure. Figure 12B 1200b is a diagram illustrating an exemplary implementation of a customized alpha blending technique for generating short star trails according to an embodiment of the present disclosure. The customized alpha blending method blends captured frames, including at least one current frame and a set of subsequently captured frames, with determined customized blending weights and trail length weights. Based on user input, shape-corrected video frames or star color map frames are blended with the determined weights. Based on the blending mode determined from the star track estimation module, the following blending operations can be triggered:

[0173] If blending is skipped: any frames with skipped global motion will be ignored while blending.

[0174] If Reset Fusion: Resets if a specific star position does not return to its original position after global motion is detected.

[0175] Otherwise: perform fusion.

[0176] Typically, the mixing operation is performed based on equation (8):

[0177] g(x)=alpha×(f0(x))+beta×(f1(x))+gamma……(8),

[0178] Where, g(x) = output frame, f0(x) = previous frame, f1(x) = current frame,

[0179] Among them, alpha, beta (=1-alpha), gamma = constant weight values,

[0180] Where x∈frame 1 to n

[0181] The constant weight value used in equation (8) is usually set to a constant weight that does not change according to the local characteristics of the frame. The constant weight value is used for all pixels in the frame. However, the embodiment of the present disclosure describes determining a dynamically changing fusion weight based on the following two aspects:

[0182] Customized fusion weights — based on local image features

[0183] Track length weighting - based on angular separation

[0184] Therefore, according to an embodiment of the present disclosure, the modified equation (9) for performing the custom mixing operation becomes:

[0185] g(x,k)=alpha(x,k)×f0(x,k)+beta(x,k)×f1(x,k)+gamma(x,k)……(9)

[0186] Among them, alpha(x, k): the average of the custom fusion weight and the trajectory length weight,

[0187] β(x,k): (1-α(x,k)),

[0188] gamma(x,k): the mean of (alpha(x,k), beta(x,k)),

[0189] x∈frames 1 to n, and k∈dynamic blocks in frames 1 to m.

[0190] In some embodiments, alpha(x, k) may be a combination of a custom fusion weight and a trajectory length weight.

[0191] The following combination Figure 13 A method 1300 for generating an astronomical time-lapse video in a user device is described.

[0192] Figure 13 An exemplary process flow of a method 1300 for generating an astronomical time-lapse video in a user device (e.g., user device 101) according to an embodiment of the present disclosure is shown. In one embodiment, the steps of method 1300 may be performed by system 201, for example, by processor 203 in conjunction with module 209 and memory 205 of system 201.

[0193] At step 1301, at least one current frame is captured via a user device. The at least one current frame includes one or more stars. The at least one current frame is obtained by capturing the one or more stars via a camera of the user device.

[0194] At step 1303, one or more regions within the at least one current frame are determined to have exposure and focus levels greater than or less than predefined thresholds. The one or more regions within the at least one current frame are determined based on the exposure and focus levels. In some embodiments, the exposure and focus levels are determined based on frame segmentation and a true north proximity factor associated with one or more stars within the at least one current frame.

[0195] In some embodiments, exposure and focus levels are determined based on frame segmentation and a true north proximity factor associated with one or more celestial bodies in at least one current frame. In some embodiments, method 1300 may further include performing frame segmentation for at least one current frame and at least one of a set of subsequent frames. Frame segmentation may include segmenting the corresponding frame into one or more regions and classifying each region into one or more objects. In some embodiments, method 1300 may further include identifying a preferred type region and one or more non-preferred type regions from the one or more segmented regions. In addition, method 1300 may include dividing the preferred type region into a plurality of blocks. In addition, method 1300 may include comparing the brightness of each block with the average brightness of the plurality of blocks to obtain exposure and focus levels. In addition, method 1300 may include determining, based on the comparison, one or more regions corresponding to one or more blocks having exposure and focus levels greater than or less than a predefined threshold.

[0196] At step 1305, a set of subsequent frames including the one or more celestial bodies is captured. The set of subsequent frames is obtained by capturing the one or more celestial bodies via a camera of a user device. Exposure and focus of the one or more regions in the set of subsequent frames are corrected based on exposure and focus levels of the corresponding one or more regions within at least one current frame during capture.

[0197] At step 1307 , fusion weights of the at least one current frame and the set of subsequent frames are determined based on one or more cloud categories and a cloud density score associated with each of the one or more cloud categories.

[0198] In some embodiments, method 1300 may include classifying each block of the corresponding frame into one or more cloud categories. Furthermore, method 1300 may include determining a cloud density score for each cloud category. Furthermore, method 1300 may include determining a fusion weight for the corresponding frame based on the cloud classification and the cloud density score.

[0199] At step 1309, an astronomical time-lapse video is generated by fusing the at least one current frame with a set of subsequent frames based on the determined fusion weights. In some embodiments, to generate the astronomical time-lapse video, method 1300 may further include correcting star shapes associated with one or more celestial bodies and estimating star orbits associated with the one or more celestial bodies. Furthermore, method 1300 may include generating the astronomical time-lapse video by fusing the at least one current frame with a set of subsequent frames based on the determined fusion weights, the corrected star shapes, and the estimated star orbits.

[0200] In some embodiments, to correct star shapes, method 1300 may include correcting the shape of at least one of the one or more stars in at least one current frame and a set of subsequent frames to obtain a plurality of star shape-corrected frames. Furthermore, method 1300 may include assigning a trajectory length weight to each star in the corresponding frame based on an angular interval. Furthermore, method 1300 may include estimating a star orbit for each star in the corresponding frame based on celestial positioning information of the one or more stars. Furthermore, method 1300 may include obtaining a pixel position of each star in the corresponding frame based on the determined celestial positioning information of each star and a transformation map associated with the celestial positioning information of the one or more stars.

[0201] In some embodiments, method 1300 may include correcting the star trajectory of at least one of the one or more star bodies based on a corresponding star trajectory of the one or more star bodies to obtain a plurality of star trajectory corrected frames. In some embodiments, method 1300 may include blending at least one current frame with a set of subsequent frames using a custom alpha blending technique based on the plurality of star shape corrected frames, the corresponding trajectory length weights, the plurality of star trajectory corrected frames, and the determined blending weights.

[0202] In some embodiments, method 1300 may include providing information associated with one or more regions having exposure and focus greater than or less than a threshold exposure and focus level to a camera hardware abstraction layer (HAL) of a user device. Furthermore, method 1300 may include correcting the exposure and focus levels of the one or more regions in a set of subsequent frames captured by the camera.

[0203] Although shown and described in a specific order Figure 13 However, according to various embodiments, these steps may occur in a varying order. Figure 13 The detailed description of each step is covered in Figure 1-12B The relevant description is omitted here for the sake of brevity.

[0204] In an additional embodiment of the present disclosure, a user device and method for capturing astronomical time-lapse videos in real time are provided. The user device can be a mobile device equipped with at least one camera and a configurable camera pipeline. The mobile device can include a mobile phone, a tablet computer, etc. The user device can also include different sensors with different exposure capabilities. The sensors can be used to capture wide-angle images and telephoto images based on the exposure capabilities. The different sensors can include a GPS sensor. The GPS sensor data can be used to determine longitude and latitude data. The longitude and latitude data can be used to detect true north and true south, thereby maintaining the shape of the fused object. The user device can also include an inertial sensor for eliminating erroneous motion. Eliminating erroneous motion enables the generation of accurate star tracks.

[0205] A method for capturing astronomy time-lapse video in real time may include capturing multiple astronomy time-lapse images of the night sky with the click of a button. The astronomy time-lapse images may be captured in a fusion mode or without fusion mode. The speed at which the images are captured may be adjusted based on the requirements of the use case. Furthermore, the astronomy time-lapse recordings may be captured in different resolutions, such as, but not limited to, full high definition (FHD) or ultra high definition (UHD).

[0206] With at least the foregoing in mind, the present subject matter provides at least the following advantages:

[0207] The methods described in the embodiments of this document enable the user device to capture astronomical time-lapse videos by clicking a button.

[0208] Furthermore, the methods described in the embodiments herein are capable of handling multi-stage processing in real-time with robustness to parameters including capture time, brightness, global motion, and the like.

[0209] Furthermore, the method described in the embodiments herein provides a real-time pipeline primarily for automatic IQ improvement, fusion weight determination, and star trajectory fusion.

[0210] Furthermore, the methods described in embodiments herein provide an adaptive IQ configuration for astronomical time-lapse mode, which sets exposure time, ISO (ie, the sensitivity of the camera's image sensor to light), and focus based on per-frame semantic analysis.

[0211] Furthermore, the methods described in the embodiments herein provide a classification model that controls fusion weights based on object detection and masking parameters.

[0212] Furthermore, the methods described in the various embodiments herein provide techniques for converting device coordinates into celestial coordinates using a combination of inertial sensors and GPS for accurate positioning of celestial objects for re-rendering. Furthermore, the use of inertial sensors and GPS also enables per-frame skip / reset fusion based on global motion detection.

[0213] Furthermore, the methods described in embodiments herein can provide additional controls to the user to select custom blending parameters such as trajectory color.

[0214] Although specific language has been used to describe this subject matter, it is not intended to cause any limitations resulting therefrom. It will be apparent to those skilled in the art that various practical modifications may be made to the method to implement the inventive concept as taught herein. The accompanying drawings and the foregoing description provide examples of embodiments. It will be understood by those skilled in the art that one or more of the elements described may be well combined into a single functional element. Alternatively, certain elements may be divided into multiple functional elements. Elements from one embodiment may be added to another embodiment.

Claims

1. A method for generating an astronomical time-lapse video in a user device, the method comprising: obtaining at least one current frame by capturing one or more celestial bodies via a camera of the user device, wherein the at least one current frame includes the one or more celestial bodies; determining the one or more regions based on exposure and focus levels corresponding to the one or more regions within the at least one current frame; obtaining a set of subsequent frames including the one or more celestial bodies by capturing the one or more celestial bodies via the camera, wherein during the capturing, exposure and focus corresponding to one or more regions within the set of subsequent frames are corrected based on the exposure and focus levels corresponding to the one or more regions within the at least one current frame; determining a fusion weight for the at least one current frame and the set of subsequent frames based on one or more cloud categories and a cloud density score associated with each of the one or more cloud categories; and An astronomical time-lapse video is generated by fusing the at least one current frame with the set of subsequent frames based on the determined fusion weights.

2. The method according to claim 1, wherein The exposure and focus levels are determined based on frame segmentation and true north proximity factors associated with the one or more stars in the at least one current frame.

3. The method according to claim 2, further comprising: The frame segmentation is performed on at least one of the at least one current frame or the set of subsequent frames by segmenting the corresponding frame into one or more regions and classifying each region into one or more objects.

4. The method according to claim 3, wherein: Determining the one or more regions includes: identifying a preferred type region and one or more non-preferred type regions from the one or more segmented regions; Dividing the preferred type area into a plurality of blocks; comparing the brightness of each block with the average brightness of the plurality of blocks to obtain the exposure and focus levels; and Based on the comparison, the one or more regions corresponding to the one or more blocks having the exposure and focus levels greater than or less than a predefined threshold are determined.

5. The method according to claim 4, comprising: classifying each block of the corresponding frame into one or more cloud categories; determining the cloud density score for each cloud category; as well as The fusion weight is determined for the corresponding frame based on the cloud category and the cloud density score.

6. The method according to claim 1, wherein Generating the astronomical time-lapse video includes: Correcting a star shape associated with the one or more stars; estimating stellar orbits associated with the one or more stellar bodies; and The astronomical time-lapse video is generated by fusing the at least one current frame with the set of subsequent frames based on the determined fusion weights, the corrected star shapes, and the estimated star orbits.

7. The method according to claim 6, wherein: Correcting the star shape includes: Correcting the shape of at least one of the one or more stars in the at least one current frame and the set of subsequent frames to obtain a plurality of star shape-corrected frames; Assigning a track length weight to each star in the corresponding frame based on the angular interval; For each star in the corresponding frame, estimating a star orbit based on the celestial positioning information of the one or more stars; and The pixel position of each star in the corresponding frame is obtained based on the determined celestial positioning information of each star and a transformation map associated with the celestial positioning information of the one or more stars.

8. The method according to claim 7, comprising: The star trajectory of at least one of the one or more star bodies is corrected based on the corresponding star trajectory of the one or more star body orbits to obtain a plurality of star trajectory corrected frames.

9. The method according to claim 6, comprising: Receive input to generate color astronomical time-lapse video; as well as A color astronomical time-lapse is generated by fusing the at least one current frame and the set of subsequent frames together with a predetermined color map based on the determined fusion weights, the corrected star shapes, and the estimated star orbits, wherein the predetermined color map is applied to corresponding pixel positions of the one or more stars in the plurality of frames.

10. The method of claim 7, wherein the fusing comprises: The at least one current frame and the set of subsequent frames are blended using a custom alpha blending technique based on the plurality of star shape corrected frames, the corresponding track length weights, the plurality of star track corrected frames, and the determined blending weights.

11. The method according to claim 1 , comprising: providing, to a camera hardware abstraction layer (HAL) of the user device, information associated with one or more regions where exposure and focus are greater than or less than a threshold degree of exposure and focus; as well as Exposure and focus levels of the determined one or more regions in a set of subsequent frames captured by the camera are corrected.

12. A system for generating an astronomical time-lapse video in a user device, the system comprising: Memory, which stores instructions; at least one processor; wherein the instructions, when executed by the at least one processor, cause the system to: obtaining at least one current frame by capturing one or more celestial bodies via a camera of the user device, wherein the at least one current frame includes the one or more celestial bodies; determining one or more regions within the at least one current frame based on exposure and focus levels; obtaining a set of subsequent frames including the one or more celestial bodies by capturing the one or more celestial bodies via the camera, wherein exposure and focus of the corresponding one or more regions within the set of subsequent frames are corrected during the capturing based on the exposure and focus levels corresponding to the one or more regions within the at least one current frame; determining a fusion weight for the at least one current frame and the set of subsequent frames based on one or more cloud categories and a cloud density score associated with each of the one or more cloud categories; and An astronomical time-lapse video is generated by fusing the at least one current frame with the set of subsequent frames based on the determined fusion weights.

13. The system according to claim 12, wherein: The exposure and focus levels are determined based on frame segmentation and a true north proximity factor associated with the one or more stars in the at least one current frame.

14. The system according to claim 13, wherein: The instructions, when executed by the at least one processor, cause the system to: The frame segmentation is performed on at least one of the at least one current frame or the set of subsequent frames by segmenting the corresponding frame into one or more regions and classifying each region into one or more objects.

15. The system according to claim 14, wherein: To determine the one or more regions, the instructions, when executed by the at least one processor, cause the system to: identifying a preferred type region and one or more non-preferred type regions from the one or more segmented regions; Dividing the preferred type area into a plurality of blocks; comparing the brightness of each block with the average brightness of the plurality of blocks to obtain the exposure and focus levels; as well as Based on the comparison, the one or more regions corresponding to the one or more blocks having the exposure and focus levels greater than or less than a predefined threshold are determined.

Images