Systems and methods for synthetic augmentation of cameras using neural networks

Abstract

A method of augmenting camera devices with neural networks to control framerate, time synchronization, image stitching and resolution, the method comprising: creating synthetic camera frames by using neural networks to interpolate between actual frame captures; utilizing frame interpolation to align camera images when the frames are misaligned in time, with additional hardware; retroactively processing recorded sensor data to achieve time synchronization from multiple camera sensors; stitching temporally misaligned camera recordings together to create spherical or panoramic images with vision pipelines augmented by neural networks; and enhance images by utilizing neural networks to adjust optimize resolution.

Description

[0001]This application claims priority under 35 USC § 119(e) to U.S. Provisional Application No. 63 / 125,136, filed on Dec. 14, 2020, the contents of which is hereby incorporated by reference in its entirety.BACKGROUNDField[0002]Methods and apparatus for enhancing camera performance using Neural Networks are provided. More specifically, cameras may leverage an synthetic artificial intelligence (AI) pipeline to improve frame rate, resolution, high dynamic range, removal of image artifacts with image inpainting, and filters for privacy. Neural Networks may also leverage orthogonal sensors to conduct hypothesis testing. Optimized cross-dimensional sensor platforms may assist Neural Networks to enhance accuracy. Deployment of synthetic AI pipeline using Neural Networks may be executed in the cloud, on-device, or on-chip with dedicated hardware.Related Art[0003]Related art performant cameras are bulky, cost-prohibitive, unnecessarily complex, and have limitations in bandwidth, resolution ...

AI Technical Summary

Benefits of technology

The patent text describes a method to improve camera performance using neural networks. The method can use synthetic camera feeds generated by neural networks to match the performance of state-of-the-art cameras with commodity sensors. The method can also leverage neural networks to create privacy enabled data captures from camera sensors. Furthermore, the method can include using neural networks for frame interpolation to align and synchronize camera images, stitching together misaligned camera recordings, and adjusting image quality through resolution enhancement. Additionally, the patent text describes the use of hardware accelerated neural networks for removing personal identifiable information from camera videos and improving or denoising camera data through inpainting, optimizing image quality in low light conditions, and inferring sections of the image which are occluded by objects.

Problems solved by technology

Related art performant cameras are bulky, cost-prohibitive, unnecessarily complex, and have limitations in bandwidth, resolution and high latency, due to serialization.

These limitations of the related art cameras restrict the capabilities of the camera, and further negatively impact the performance of AI pipelines with respect to the camera data.

Images