Ai-based vision system with digital zoom for autonomous navigation to space

Abstract

The invention relates to an AI-based vision system (110), called a smart camera (SC), for autonomous navigation of a satellite to some a priori known but non-cooperative target object in space. The SC uses images acquired by a single high-resolution vision sensor (112) installed on the satellite. The vision sensor supports a digital zooming function (128) through configurable image cropping within the sensor, which allows to dynamically adapt the region of interest (ROI) in the sensor image thereby eliminating needless pixel operations. The acquired visual information is processed in real-time by quantized AI models (130) trained to predict the relative position and attitude (132) (jointly referred to as the pose) of the target object as seen by the satellite. The pose predictions are used for autonomous steering of the satellite towards the target object by appropriate guidance, navigation, and control (GNC) algorithms (142).

Description

[0001] Al-based vision system with digital zoom for autonomous navigation to space targets

[0002] TECHNICAL FIELD

[0003] This invention relates generally to the field of visual servoing systems for on-orbit services (OOS) such as active space debris removal (ADR). More specifically, this invention relates to a smart camera (SC) using at least one artificial intelligence (Al) model to provide target position and orientation information for autonomous navigation to this target in space. The invention further relates to a closed-loop simulation environment for developing and testing real-time implementations of such a SC.

[0004] STATE OF THE ART

[0005] Vision systems that must deal with objects at different distances have to cope with the trade-off between image resolution and field of view. To overcome this problem either zoom lenses

[0001] or multiple lenses with different focal lengths [2] are usually deployed.

[0006] Al machine learning techniques have been used successfully to model complex problems without the need for an exact mathematically description. Therefore, they are widely adopted in vision applications for detection and tracking of objects. One prerequisite for their application is the existence of a large amount of labelled data for training of the Al models i.e., many images or image sequences in case of vision systems. If such data is not available or difficult to obtain, synthetic image generation is often used [3],

[0007] Earth-based evaluation and testing of visual servoing algorithms and their real-time implementations for space applications is inherently difficult and causes excessive costs when using physical mock-ups and robotic platforms. Therefore, simulation environments are sometimes used for this purpose [4,5], Purpose of the invention

[0008] For autonomous navigation to non-cooperative space targets, it is important to provide precise position and orientation information of the target object to the GNC unit of the satellite. Due to latency requirements in mid- to near-range operations this information must be obtained from on-board sensors and cannot be provided through earth-based communication. When using vision sensors for this purpose, pixel resolution of objects at different distances must be kept at the same level by zooming. Digital zoom techniques are preferred for spaceflight applications because the mechanics of zoom lenses must withstand extreme temperatures, vacuum, and radiation conditions, which makes them difficult to build, bulky and expensive.

[0009] To provide sufficient and consistent training data poses a particular challenge for space vision applications. The accuracy of Al models trained on synthetic data often suffers from a domain gap between rendered imagery and real-world environment. For visual servoing operations with interdependent movement of camera sensor and target object, the domain gap effect is intensified if this movement is not reflected in the training data. In case of autonomous satellite navigation to space targets this movement is governed by the physics of space law and the GNC algorithm deployed. Image sequences obtained in consideration of the interdependent movement of camera sensor and target object are therefore preferable for the training of such tracking models.

[0010] Earth-based evaluation and testing of visual servoing systems for OOS application by means of simulation is desirable for reasons of cost and flexibility. However, both important properties of the space environment and the real-time implementation of certain elements of the visual servoing system should be considered. Therefore, a simulation environment with space-qualifiable hardware-in-the-loop is of interest.

[0011] SHORT DESCRIPTION OF THE INVENTION

[0012] The invention relates to an Al-based smart camera system for autonomous navigation of a satellite to some a priori known but non-cooperative target object in space. The smart camera system uses images acquired by a single high-resolution vision sensor that is closely coupled to some real-time processing unit through a data interface and a configuration interface. The vision sensor supports digital zooming by in-sensor image cropping to a rectangular region of interest (ROI). The current ROI is dynamically determined within the real-time processing unit based on Al-model predictions and configured through the configuration interface of the sensor. The zooming function is realized purely electronically and avoids any zoom lens mechanics.

[0013] The invention also provides a solution for the efficient generation of training data for the Al-models in consideration of the interdependent movement of camera sensor and target object. For this, astro-dynamic simulations of GNC algorithms are coupled with a photo-realistic image rendering tool and integrated with the smart camera system into a common closed-loop environment. This way the invention provides an efficient way to generate photo-realistic Al training data that allows to reduce the domain gap effect between simulation and target environment without expensive target mock-up installations and their robot-based movement.

[0014] By replacing the true target pose information obtained from astro-dynamic simulation with estimations from the trained and quantized Al models implemented in the smart camera system, the invention also enables the evaluation of different GNC algorithms for a given OOS task and associated Al pose prediction model under real-time processing constraints. The evaluation can be performed with respect to system parameters like mission success rate and fuel consumption. Once GNC algorithms and Al-models are fixed, the invention enables final testing of the real-time processing implementation.

[0015] LIST OF TECHNICAL DRAWINGS

[0016] Two technical drawings depicted in FIG. 1 and FIG. 2 present the invention.

[0017] - FIG. 1 Al-based vision testbed according to the invention

[0018] - FIG. 2 Region-of-interest (ROI), bounding box, and target object for digital zooming LIST OF REFERENCE NUMERALS

[0019] 100 Al-vision testbed for hardware-in-the-loop simulation

[0020] 110 Smart Camera (SC) system

[0021] 112 High-resolution vision sensor with fixed focal-length lens

[0022] 114 Data interface of vision sensor 112 to real-time processing unit 118

[0023] 116 Configuration interface of vision sensor 112 for image cropping

[0024] 118 Real-time processing unit

[0025] 120 Selector for target image input to the frame buffer 122 Position A: true target image 152

[0026] Position B: target image from data interface 114 of vision sensor 112

[0027] 122 Frame buffer for storing full-scale target images to be used for initial detection and cropped images to be used for tracking

[0028] 124 Method for initial detection of target object in full-scale image implemented in real-time processing unit 118

[0029] 126 Target position information from initial detection

[0030] 128 Method for digital zoom control using either the estimated target position 126 or 132 to configure the cropping area via configuration interface 116

[0031] 130 Quantized Al tracking models obtained from 162 by applying quantization method 164

[0032] 132 Estimated target pose from quantized Al tracking models 130

[0033] 140 Space Mission Simulator (SMS)

[0034] 142 GNC unit calculating forces to be generated by the satellite propulsion system according to some mission strategy and control algorithms given the current in-orbit constellation

[0035] 144 Astro-dynamic simulation environment calculating the new universe state (in-orbit constellation of satellite and target object) given the current universe state and the calculated satellite propulsion forces Image rendering environment using a 3D-model of the target object to generate a photo-realistic image of the target object as seen by the satellite in the current universe state Satellite propulsion forces to be generated according to the selected GNC strategy True target pose (relative target position and attitude) according to current universe state True target image as seen by satellite in current universe state Selector for the target pose input to the GNC algorithm 142 Position A’: true target pose 150

[0036] Position B’: estimated target pose 132 from Smart Camera 110 Al training environment using training data 152 and labels 150 generated by running the SMS 140 with selector 154 in A’ Al models to be trained for prediction of the target pose in the current universe state given a single or a sequence of target images Quantization of network weights of the trained Al tracking models for their real-time implementation in processing unit 118 Image projector oriented towards the projection screen 190 Projection screen reflecting the projected image sequence Sequence Controller providing time synchronization signals 182 for an event-driven simulation Timing control signals for an event-driven simulation Full-scale image pixel array corresponding to native resolution of vision sensor 112 Region of Interest (ROI) computed by digital zoom method 128 and configured to vision sensor 112 via configuration interface 116 Bounding box of target object estimated by initial detection method 124 (initialization mode) or tracking models 130 (tracking mode) Target object sensed by vision sensor 112 DETAILED DESCRIPTION OF THE INVENTION

[0037] The invention relates to an Al-based vision system (110), called a smart camera (SC), and a corresponding closed-loop simulation environment (100), called the Al- vision testbed.

[0038] In its minimum configuration the Al-vision testbed (100) combines SC (110) with a Space Mission Simulator (SMS) (140) and an Al training environment (160), see FIG. 1. This minimum configuration enables hardware-in-the-loop simulation.

[0039] In a further preferred embodiments of this invention the simulation environment (100) additionally comprises an image projector (170) and a projection screen (180) to enable camera-in-the-loop simulation.

[0040] In another preferred embodiments of this invention the simulation environment (100) additionally comprises a sequence controller (190) to synchronize the operation of the SC (110) and SMS (140) for synchronized camera-in-the-loop simulation.

[0041] The purpose of the SC (110) is to enable autonomous navigation of a satellite to some a priori known but non-cooperative target object during spaceflight by providing information about the current target position and orientation obtained through a monocular camera system to the guidance, navigation, and control (GNC) unit of the satellite.

[0042] The purpose of the Al-vision testbed (100) is to enable: a) the efficient generation of training data for the Al models used in the SC with minimum domain gap between simulation and target environments. b) verification of the real-time implementation of the SC system under realistic operating conditions. c) the performance evaluation of different combinations of GNC strategies and algorithms used with the SC.

[0043] In its minimum configuration the SC comprises a single high-resolution vision sensor (112) with a fixed focal-length lens, and a real-time processing unit (118) such as a Field Programmable Gate Array (FPGA) or System-on-Chip (SoC).

[0044] The vision sensor (112) is tightly coupled to the processing unit (118) through a parallel or serial data interface (114) and a configuration interface (116), meaning that vision sensor and processing unit typically reside on the same assembly unit e.g., a printed circuit board (PCB). The lens used for the sensor has fixed focal length. Zooming is only supported through cropping the data from the pixel array within the sensor to some rectangular region of interest (ROI). The position and size of the ROI can be configured dynamically by setting appropriate configuration registers of the vision sensor (112) through the configuration interface (116).

[0045] The vision sensor only transfers pixels within the configured ROI over the data interface (114) to the processing unit (118). During target object tracking this will minimize the pixel transfer time from the sensor (112) to the processing unit (118) and avoid needless pixel processing operations in (118).

[0046] The real-time processing unit (118) provides a frame buffer (122) with input selector (120), such that image data from either data interface (114) position (B) or from an external SC interface position (A) is written to the buffer.

[0047] The SC operates in either initialization mode or tracking mode, and the frame buffer stores two different types of images depending on the current SC operation mode. If the SC is in initialization mode, the buffer stores full-scale target images which are used for initial detection of the target object. If the SC is in tracking mode, the frame buffer stores images cropped to the current ROI which are used for estimating the pose (position and attitude) (132) of the target object.

[0048] In a preferred embodiment of this invention, the frame buffer is only used for storing cropped images in tracking mode and processing full-scale images in initialization mode without frame buffering. In this embodiment the memory capacity required for the frame buffer (122) is reduced from full-scale images to the maximum cropped image size.

[0049] In initialization mode target object detection is performed on full-scale images (200) by a method (124) implemented in real-time processing unit (118). Method (124) uses either a pixel-based approach such as foreground-background separation, or an Al-model such as a Convolutional Neural Network (CNN) detector trained using the Space Mission Simulator (SMS) (140) and an Al training environment (160). Method (124) provides target position information (126) in form of bounding box coordinates to the digital zooming method (128).

[0050] After successful initial detection of the target object, digital zooming (128) calculates an initial rectangular region of interest ROlo (210) from the coordinates of the bounding box (220) of the target object (230) provided from initial detection (124), such that ROlo completely contains the bounding box with a certain target intersection-over-union IOLIT with IOLIT < 1 , see FIG. 2. The digital zooming method (128) configures vision sensor (112) through the configuration interface (116) for the initial ROlo i.e., the ROI active in the vision sensor is set to ROIA = ROIo. Subsequently, the SC changes from initialization mode to tracking mode.

[0051] In tracking mode, the digital zooming method (128) continuously updates the current lollk between the active ROIA and the estimated bounding box coordinates of the target object provided by tracking models (130). If lollk is not within a certain interval (IOLIT- AL ^ lollk < IOUT + H) the current ROIk is calculated such that the target IOUT is obtained with the current bounding box coordinates, and ROIA = ROIk is set for vision sensor (112).

[0052] The Al tracking models (130) continuously estimate the pose (position and attitude) of the target from cropped images stored in frame buffer (122). The pose estimation is either performed with a feed-forward neural network such as a CNN, or with a recurrent neural network (RNN).

[0053] A preferred embodiment of this invention uses a two-stage process for pose estimation with dedicated Al models for each stage. In the first stage, a detection model estimates the target position and the corresponding bounding box coordinates are provided for loU and ROI calculation to digital zooming method (128). The pixels within this bounding box are used by a dedicated Al model to estimate the target attitude.

[0054] In a preferred embodiment of this invention the neural network models used in (130) always work on the same nominal input image size which equals a pre-defined maximum size for the active ROIA (in case of position estimation) and the target bounding box (in case of attitude estimation), respectively.

[0055] All Al models deployed in processing unit (118) are trained in some Al training environment (160) for their corresponding task. This applies to tracking models (130) used for pose estimation and any Al model used for initial detection (124).

[0056] The SMS (140) is used to generate synthetic training data for all Al models deployed in processing unit (118). The SMS (140) comprises a GNC simulation unit (142), an astro-dynamic simulation environment (144), and a phot-realistic image rendering environment (146).

[0057] Given the current in-orbit constellation of satellite and target object, the GNC unit (142) calculates the forces (148) to be generated by the satellite propulsion system to realize a new in-orbit constellation according to some mission strategy and control algorithms. The astro-dynamic simulation environment (144) takes the calculated forces (148) and determines the resulting target pose (position and orientation) (150) as seen from the satellite in the new in-orbit constellation. Pose information (150) is input to the rendering environment (146) to generate a photo-realistic image (152) of the target as seen from the satellite by rendering a 3D model of the target object in earth orbit.

[0058] The current in-orbit constellation provided to the GNC unit is either based on the true target pose (150) determined from astro-dynamic simulations - selector (154) in position (A’) or on the target pose (132) estimated by the quantized Al models (130) - selector (154) in position (B’).

[0059] SMS (140) is run standalone with selector (154) in position (A’) to generate initial training data for Al models. For these runs relevant navigation manoeuvres are simulated with selected control algorithms over a set of randomized parameters for initial constellation, lighting, and image background conditions. These conditions are selected according to possible scenarios during the mission such that the resulting target images (152) and associated ground-truth labels (150) form a training dataset with minimum domain gap to the space environment and the application task the SC is being designed for.

[0060] The network weights of the trained Al models are quantized into a form appropriate for real-time implementation in processing unit (118). A preferred embodiment of this invention jointly performs hardware-aware model training and network quantization.

[0061] After deploying the quantized Al models (130) in processing unit (118), the SC (110) and SMS (140) are combined in the Al-vision testbed (100) modelling a visual servoing system. The Al-vision testbed (100) enables three forms of closed-loop simulations with increasing complexity and serving different purposes.

[0062] The first form of closed-loop simulation supported by the Al-vision testbed is hardware-in-the-loop simulation. This form comprises SC (110) with selector (120) in position A and SMS (140) with selector (154) in position (B’). Hardware-in-the- loop simulation is used to verify the correct implementation of the quantized Al tracking models (130), by comparing each estimated target pose (132) with the corresponding true target pose (150). Initial detection and digital zooming are either skipped from the simulation sequence or performed within the SMS (140).

[0063] The second form of closed-loop simulation supported by the Al-vision testbed is camera-in-the-loop simulation. This form additionally comprises an image projector (170) and a projection screen (180) for reflecting the image. Camera-in-the-loop simulation additionally enables the verification and optimization of the initial detection method and digital zooming function implemented in the SC (110). This form of closed-loop simulation also enables to evaluate the effect of using estimated pose information instead of true pose information on performance metrics of the navigation tasks. Optical effects of the vision sensor and its lens, as well as different GNC algorithms and strategies can be evaluated.

[0064] In a preferred embodiment of this invention, true target images (152) and the associated ground truth labels (150) generated with the second form of closed-loop simulation are used to augment the initial training data for any of the Al-models used in the SC (110). This way, Al-models and overall system performance can be iteratively optimized with respect to relevant OOS mission parameters.

[0065] The third form of closed-loop simulation supported by the Al-vision testbed is synchronized camera-in-the-loop simulation. This form additionally comprises a sequence controller (190) to synchronize the operation of the SC (110) and SMS (140) by means of timing control signals (192). This form additionally enables an event-driven simulation that emulates real-time operations of the overall visual servoing system without a real-time implementation of the SMS (140). The effect of different frame rates of the SC (110) on overall system performance can be evaluated.

[0066] REFERENCES

[0067] This list of references cited by the applicant is for the reader’s convenience only.

[0068] [1] US10642271 B1 Vehicle guidance camera with zoom lens

[0069] [2] US6650360B1 Camera guidance system

[0070] [3] WQ2023014369A1 Synthetic dataset creation for object detection and classification with deep learning

[0071] [4] CN101726296A Vision measurement, path planning and GNC integrated simulation system for space robot

[0072] [5] CN114565742A Small celestial body surface dynamic simulation and landing visual simulation system and method

Claims

CLAIMS1. A smart camera (SC) system (110) comprising a high-resolution vision sensor (112) with fixed focal-length lens coupled through a data interface (114) and a configuration interface (116) to a real-time processing unit (118) which dependent on selector (120) stores input images from either data interface (114) position (B) or from an external SC interface position (A) into a frame buffer (122).

2. A system according to claim 1 with quantized Al models (130) implemented in the same real-time processing unit (118) predicting the target pose (relative position and attitude) (132) from cropped images read from the frame buffer (122).

3. A system according to claim 2 using a two-stage pose estimation process with dedicated Al models (132) for position estimation in the first stage and attitude estimation in stage two using only pixels within the estimated bounding box from stage one as input.

4. A system according to claim 1 and a method (124) implemented in the same realtime processing unit (118) for initial detection of the target object position (126) from full-scale images received from input selector (120).

5. A system according to claims 2 or 3 and according to claim 4 and a method (128) for digital zooming using either the target position information (126) or (132) to calculate the appropriate ROI and configuring the corresponding cropping area in the vision sensor (112) through the configuration interface (116).

6. A space mission simulator (SMS) (140) comprising an astro-dynamic simulation environment (144) that takes propulsion forces (148) calculated by some GNC algorithm (142) from the previous in-orbit constellation of satellite and target object - Selector (154) in position (A’) - to determine the new in-orbit constellation (true target pose) (150) after applying such forces and a rendering environment (146) to generate target image or image sequence (152) as seen from the satellite in the current orbit constellation.

7. A system according to claim 6 and an Al training environment (160) using the generated target images and / or image sequences (152) and associated target pose labels (150) for training Al tracking models.

8. A system according to claims 2 or 3 and according to claim 7 and a method (164) to quantize the network weights of the Al tracking models trained in environment(160) such that model inference can be performed on real-time processing unit (118).

9. A system according to claim 2 or 3 with selector (120) in position (A) and a system according to claim 6 with selector (154) in position (B’), performing inference on the quantized Al tracking models (130) to estimate the target pose (132) from target images (152) generated by the SMS (140), with the predicted target pose (132) being used by the SMS (140) to calculate the next true target image (152), thus SC (110) and SMS (140) forming a closed-loop system for hardware-in-the- loop simulation.

10. A system according to claim 5 with selector (120) in position (B) and an image projector (170) projecting the target image on some projection screen (190) from which vision sensor (112) of the SC (110) receives the image and provides it to the quantized Al models (130) via frame buffer (122) for estimating the target pose (132), thus forming a closed-loop system for camera-in-the-loop simulation.11 . A system according to claim 10 and a sequence control unit (180) providing timing control signals (182) for synchronization of the SC (110) and SMS (140) during camera-in-the-loop simulation.

Images