Systems and methods for training predictive systems

Abstract

To provide a system and a method for training a prediction system capable of improving depth perception in low-light.SOLUTION: In one embodiment, the method includes computing, in a first training stage, losses associated with predicting a depth map for a synthetic image of a low-light scene, wherein the losses include a pose loss, a flow loss, and a supervised loss. The method also includes adjusting, according to the losses, a style model and a depth model. The method also includes training, in a second training stage, the depth model using a synthetic representation of a low-light image. The method also includes providing the depth model.SELECTED DRAWING: Figure 3B

Description

[Technical Field]

[0001] The subject matter described herein, in general, concerns training systems, and more specifically, training predictive systems for depth perception in low light conditions. [Background technology]

[0002] The system can use sensor data to facilitate depth recognition of objects in an image. For example, a vehicle can be equipped with sensors that facilitate the estimation of the depth of other vehicles, obstacles, pedestrians, and additional aspects of the surrounding environment from an image. In various implementations, a vehicle can be equipped with optical and LiDAR sensors, where the LiDAR sensor uses light to scan the surrounding environment, and the logic associated with the LiDAR analyzes the acquired information to facilitate the detection of the presence of objects and other features of the surrounding environment. In further examples, additional sensors such as cameras can be implemented to acquire information about the environment from which information for the vehicle or any computing device is derived by the system. For example, this sensor data can be useful in various situations to improve the perception of the surrounding environment so that systems such as automated driving systems can accurately plan and drive the vehicle.

[0003] Furthermore, to reduce complexity, the system can use a monocular camera to facilitate predicting the depth of objects in an image without relying on data from a LiDAR sensor, stereo camera, or other sensors. For example, a depth model can use images from a monocular camera to generate a depth map. For problem-free operation, the system can train the depth model against different lighting conditions. However, for low-light conditions, the computation may encounter difficulties, especially when training the system for depth perception. [Overview of the project]

[0004] In one embodiment, the example system and method relate to training a system for low-light conditions to improve depth perception. In various implementations, systems that use images from a monocular camera to predict depth may have difficulty generating depth maps in low light conditions, such as at night. Furthermore, combining information from a monocular camera with other sensor data to increase the clarity of depth maps for low-light images can increase complexity. Therefore, in one embodiment, a prediction system can train a machine learning (ML) architecture by partially using a composite image of a low-light environment to generate a depth map. Here, the first training stage can use a composite image relating to a simulated model associated with the low-light environment. In one approach, the prediction system can be trained by using a style model to transform the composite image of the low-light environment into a daytime representation as input to a depth model for generating a depth map. In particular, the clarity of the depth map can be increased by calculating the loss due to the movement of objects between image frames using pause and flow losses for the first training stage to improve the semantic consistency of the network. For example, the prediction system can train style and depth models by tuning parameters using pause and flow losses. In this way, the prediction system improves depth map generation in low-light environments by adjusting daytime processing and losses in the synthetic domain during the first training stage.

[0005] Furthermore, the prediction system can use a second training stage to transform low-light images of real scenes into a synthetic representation using an adversarial network. Here, the second training stage can use a synthetic representation in a simulated scene of a low-light environment for depth estimation. Thus, the prediction system can process the synthetic representation using the style and depth models trained in the first training stage for the second training stage to generate depth maps. Additionally, the prediction system can perform training in the second stage according to the loss resulting from transforming low-light images from real scenes into a synthetic daytime representation and generating depth maps. In this way, the prediction system can increase the clarity of depth maps for low-light environments due to improved semantics during training by adjusting for pause and flow losses. The improved semantics can also increase the clarity of depth maps generated by the prediction system from sequences of images captured by a monocular camera, thereby benefiting applications that rely on higher-resolution depth maps.

[0006] In one embodiment, it is disclosed that a prediction system is trained to improve depth perception in low light. The prediction system includes a processor and memory, the memory of which, when executed by the processor, stores instructions for the processor to calculate losses associated with predicting a depth map for a composite image of a low-light scene in a first training stage, wherein the losses include a pose loss, a flow loss, and a supervised loss. The instructions also include instructions for tuning a style model and a depth model according to the losses. The instructions also include instructions for training a depth model using a composite representation of the low-light image in a second training stage. The instructions also include instructions for providing the depth model.

[0007] In one embodiment, a non-temporal computer-readable medium is disclosed for training a predictive system to improve depth perception in low light, and which, when executed by a processor, causes the processor to perform one or more functions. The instructions store instructions for calculating losses associated with predicting a depth map for a composite image of a low-light scene in a first training stage, wherein the losses include a pause loss, a flow loss, and a supervised loss. The instructions also include instructions for tuning a style model and a depth model according to the losses. The instructions also include instructions for training a depth model using a composite representation of the low-light image in a second training stage. The instructions also include instructions for providing the depth model.

[0008] In one embodiment, a method is disclosed for training a predictive system to improve depth perception in low light. In one embodiment, the method includes, in a first training stage, calculating losses associated with predicting a depth map for a composite image of a low-light scene, wherein the losses include a pose loss, a flow loss, and a supervised loss. The method also includes tuning a style model and a depth model according to the losses. The method also includes, in a second training stage, training a depth model using a composite representation of the low-light image. The method also includes providing the depth model. [Brief explanation of the drawing]

[0009] The accompanying drawings incorporated into and constituting part of the specification illustrate various systems, methods, and other embodiments of the disclosure. It will be recognized that the boundaries of elements illustrated in the drawings (e.g., boxes, groups of boxes, or other shapes) represent one embodiment of the boundary. In some embodiments, one element can be designed as multiple elements, or multiple elements can be designed as one element. In some embodiments, an element shown as an internal component of another element can be realized as an external component, and vice versa. Furthermore, elements may not be drawn scaled or enlarged at a constant ratio.

[0010] [Figure 1] This figure illustrates one embodiment of a vehicle capable of implementing the system and method disclosed herein. [Figure 2] This figure illustrates one embodiment of a prediction system for estimating the depth of objects in an image in a low-light environment. [Figure 3A] Figure 2 illustrates an example of a training phase embodiment for the prediction system shown in Figure 2. [Figure 3B] Figure 2 illustrates an example of a training phase embodiment for the prediction system shown in Figure 2. [Figure 4] This figure shows an example of a discriminator model used with a prediction system. [Figure 5] This figure illustrates one embodiment of an encoder / decoder network that estimates the depth of objects in an image in a low-light environment. [Figure 6A] This figure illustrates one embodiment of a method associated with training a prediction system for estimating the depth of objects in an image, using two stages. [Figure 6B] This figure illustrates one embodiment of a method associated with training a prediction system for estimating the depth of objects in an image, using two stages. [Figure 7]FIG. showing an example of a depth map for a low - light environment generated by a system trained using two stages. DETAILED DESCRIPTION OF THE INVENTION

[0011] Systems, methods, and other embodiments associated with training a prediction system to improve depth recognition in low light are disclosed herein. A prediction system for a depth map can be trained using two stages by relying on semantic consistency, thereby preventing dilution of the generated content between a style model and a depth model. In particular, the prediction system can use a pose and flow model as a supervisor in a first training stage when generating a corresponding depth map of a synthetic image for a daytime representation and a simulated scene. The daytime representation can be associated with a synthetic image from a low - light environment. In terms of lighting level, low light can represent a lighting environment where a system using a monocular camera, a red / green / blue (RGB) camera, an optical camera, machine vision, etc. may have difficulty identifying object features due to reduced color intensity.

[0012] Furthermore, the prediction system can use a pose model to enhance the style model by adjusting the movement of the camera and a flow model to enhance the depth model by considering the movement of objects in the first training stage. To reduce complexity, the first training stage can operate in a synthetic domain by using image data simulated for a traffic scene, reducing reliance on actual images, paired images, etc., and simplifying the operation. On the other hand, the second training stage can use actual images from a camera taken in an actual low - light environment to complete the training.

[0013] Regarding the architecture of the first training stage, the style model can perform style transfer by predicting how the features of an object appear, such as due to color, shading, lighting conditions, etc. Thus, the daytime representation can have the same content as the synthetic image while changing the lighting or style to increase richness. The depth model can be a network that generates a depth map from the daytime representation, such as a generative adversarial network (GAN). And the prediction system can calculate various losses and supervised losses for the first training stage through loss minimization. For example, the pose loss can be minimized when the transformation of the synthetic image reaches a threshold. On the other hand, the prediction system can minimize the flow loss associated with the dynamic movement of objects between the daytime representation and the associated depth map. Thus, the prediction system can use the loss to adjust the model parameters in the first training stage until the criterion is met, such as by using additional synthetic images related to the traffic scene when the criterion is not satisfied.

[0014] Regarding the architecture of the second training stage, the prediction system can continue training by using a generative adversarial network to transform the actual image into a synthetic representation. For example, a low-light image of an actual traffic scene can be transformed into a synthetic representation to train the network for the actual environment. Further, the second training stage can inherit the weights from the first training stage and process the synthetic representation using the style and depth models to calculate losses for further training. For example, the prediction system can calculate style transfer and super vision losses to meet a criterion (e.g., minimum loss) for predicting the depth map associated with the actual image. Thus, the processing in the second training stage can continue until the model parameters meet the criterion for the loss, such as by using additional actual images related to the traffic scene when the criterion is not satisfied. In this way, the two-stage system improves domain training and synthetic image processing for more accurate depth maps at low light levels by reducing the image correspondence or correlation between the stages.

[0015] Referring to Figure 1, an example of vehicle 100 is shown. As used herein, “vehicle” refers to any form of motorized transport device. In one or more embodiments, vehicle 100 is an automobile. Here, the configuration is described in relation to an automobile, but it will be understood that the embodiments are not limited to automobiles. In some embodiments, the predictive system 170 can be used for consumer electronics (CE), mobile devices, robotic devices, drones, etc. In some embodiments, vehicle 100 may be any form of robotic device or motorized transport device that includes sensors to recognize the characteristics of the surrounding environment and thus benefits from the functions considered herein in conjunction with training of the predictive system for improving depth recognition in low light.

[0016] Vehicle 100 also includes various elements. In various embodiments, it will be understood that vehicle 100 may have fewer elements than those shown in Figure 1. Vehicle 100 may have any combination of the various elements shown in Figure 1. Furthermore, vehicle 100 may have additional elements in addition to those shown in Figure 1. In some configurations, vehicle 100 can be realized without one or more of the elements shown in Figure 1. Although the various elements are shown in Figure 1 as being located inside vehicle 100, it will be understood that one or more of these elements may be located outside vehicle 100. Furthermore, the elements shown may be physically separated by a considerable distance. For example, as considered, one or more disclosed components may be realized inside the vehicle, while further components of the system may be realized in a cloud computing environment or other system far away from vehicle 100.

[0017] Some possible elements of vehicle 100 are shown in Figure 1 and described in subsequent figures. However, descriptions of many of the elements in Figure 1 are provided after the discussion of Figures 2-7 for the sake of brevity in this description. Additionally, for the sake of brevity and clarity of the examples, it will be noted that, where appropriate, reference numbers are repeated in different figures to indicate corresponding or similar elements. Additionally, the discussion has described the gist of numerous specific details in order to provide a complete understanding of the embodiments described herein. However, those skilled in the art will understand that the embodiments described herein can be put into practice using various combinations of these elements. In any case, vehicle 100 includes a predictive system 170 implemented to perform methods and other functions with respect to improving the training of a predictive system for depth perception in low light conditions, as disclosed herein. As will be discussed in more detail below, in various implementations, the predictive system 170 can be partially implemented inside CE devices, mobile devices, robots, drones, vehicles, etc., or can be implemented as a cloud-based service. For example, in one approach, functions associated with at least one module of the prediction system 170 are implemented within the vehicle 100, while further functions are implemented within a cloud-based computing system.

[0018] Referring to Figure 2, one embodiment of the prediction system 170 of Figure 1 is further illustrated. The prediction system 170 is shown as including the processor 110 of the vehicle 100 in Figure 1. However, in other examples, the processor 110 may be part of a CE device, mobile device, robot, drone, etc. Thus, the processor 110 may be part of the prediction system 170, the prediction system 170 may include a processor separate from the processor 110 of the vehicle 100, or the prediction system 170 may access the processor 110 through a data bus or other communication path. In one embodiment, the prediction system 170 includes a memory 210 that stores a prediction module 220. The memory 210 may be random access memory (RAM), read-only memory (ROM), a hard disk drive, flash memory, or other suitable memory for storing the prediction module 220. The prediction module 220 is, for example, a computer-readable instruction, which, when executed by the processor 110, causes the processor 110 to perform the various functions disclosed herein.

[0019] The prediction system 170 illustrated in Figure 2 is generally an abstraction of a prediction system 170 that can be implemented between a CE device, mobile device, robot, drone, vehicle, etc., and a cloud computing environment. Furthermore, the prediction module 220 generally includes instructions that function to control the processor 110 to receive data inputs from one or more sensors of the vehicle 100. In one embodiment, the inputs are observations of one or more objects in the environment adjacent to the vehicle 100, and / or other aspects of the surroundings.

[0020] Accordingly, in one embodiment, the prediction module 220 controls each sensor to provide data input in the form of sensor data 240. Additionally, although the prediction module 220 is considered to control various sensors to provide sensor data 240, in one or more embodiments, the prediction module 220 may employ other techniques for acquiring sensor data 240, which may be active or passive. For example, the prediction module 220 may take various approaches to fusing data from multiple sensors when providing sensor data 240 and / or sensor data acquired via a wireless communication link.

[0021] Furthermore, in one embodiment, the prediction system 170 includes a data storage device 230. In one embodiment, the data storage device 230 is a database. In one embodiment, the database is an electronic data structure stored in memory 210, or another data storage device, and consists of routines that can be executed by the processor 110 to analyze the stored data, provide the stored data, organize the stored data, and so on. Thus, in one embodiment, the data storage device 230 stores data that the prediction module 220 uses when performing various functions. In one embodiment, the data storage device 230 includes, for example, sensor data 240 along with metadata that characterizes various aspects of the sensor data 240.

[0022] Furthermore, in various embodiments, the data storage device 230 further includes simulation data 250 and image data 260. The simulation data 250 may include images of the environment from various driving scenarios of the vehicle 100. The environment may include buildings, other vehicles, sidewalks, streetlights, guide rails, etc. In addition, the data storage device 230 may include image data 260 captured by the vehicle 100's camera and intensity and color information relating to the image data 260. In one approach, the prediction module 220 may take a sequence of images from the scene to calculate a depth map that follows changes between frames.

[0023] To train the prediction module 220, the prediction system 170 can utilize a training phase. As described below, in the first training phase, the prediction system 170 can calculate the loss by predicting the depth map for a composite image of a night scene. In one approach, the overall loss may include the pause loss, the flow loss, and the supervised loss. The prediction system 170 can then adjust the parameters of the depth model and repeat the training process until convergence occurs or until a criterion for the parameters is met. For example, the criterion may be that the loss meets a threshold or magnitude for the resolution associated with the depth map in a given application. The second training phase completes and refines the training using actual images converted to daytime representations so that the network can operate in a real environment.

[0024] Looking at Figures 3A and 3B, systems 300 and 302 illustrate embodiments of the training phase for the prediction system 170 in Figure 2. In 300, the first training phase 310 can transform a sequence of synthetic images from low-light, nighttime, or severe weather scenes to amplified light in order to train the prediction system 170. In various embodiments, the server can use data to train the prediction system 170 offline. As described below, the first training phase 310 can operate in a synthetic domain to simplify its operation by reducing reliance on real images, image-to-image, real ground truth, etc. For example, operation in a synthetic domain can utilize a daytime representation of a current traffic scene in a synthetic nighttime scene rather than a real scene.

[0025] Regarding the use of simulated data for synthetic domains, the simulator can generate synthetic images of real-world environments such as cities, suburbs, and major highways. In one approach, the simulator can run offline on a server or network for training, thereby reducing reliance on computing resources in vehicle 100.

[0026] To understand illumination levels, low illuminance can represent lighting environments where systems using monocular cameras, RGB cameras, optical cameras, machine vision, etc., may have difficulty identifying object features due to reduced color intensity. For example, driving vehicle 100 at night, or using prediction system 170 in darkness, indoors, or at dusk, may be environments with low illuminance. Increased light can represent brighter lighting conditions where systems using monocular cameras, RGB cameras, optical cameras, machine vision, etc., can reliably identify object features from color intensity. For example, driving vehicle 100 during the day, or using prediction system 170 in sunlight, outdoors, or in the morning, may be environments with increased light. Of course, referring to daytime and nighttime to define illumination levels can generally be subjective due to variations in weather and other factors affecting illumination. Lux (lx (unit of illuminance)) can represent illumination levels as a unit of luminance per square meter. Therefore, during the day, increased light, brighter lighting, etc., generally refers to situations where the ambient light has enough lumens (e.g., 30k lux to 100k lux) to distinguish the features of an object in terms of depth processing. Conversely, at night, in low light, indoors, etc., generally refers to situations where the ambient light does not have enough lumens (e.g., 1 lux to 60 lux) to distinguish the features of an object.

[0027] Furthermore, the first training stage 310 is Style Model. n-d Using 330, a sequence of composite images can be converted into a sequence representing daytime. In particular, Style Model n-d 330 can perform style or domain transfer by predicting how object features will appear, such as color, shadows, shading, and lighting conditions. Therefore, daytime representations have the same content, while changing lighting or style to increase richness for depth processing.

[0028] Regarding the architecture, in various implementations, the Style Model n-d 330 can be a GAN or neural network that predicts color intensity to identify object features in a low-light scene. A GAN can include a pair of neural networks competing in a zero-sum game for training. The output of a GAN can be new data generated by similar statistics to the training data. Here, the GAN can transform real or synthetic images by predicting how object features will appear, such as by color, shadows, shading, lighting conditions, etc.

[0029] Furthermore, GANs can achieve indirect learning through adaptive discriminators. Figure 4 shows an example of a discriminator model 400 for a prediction system 170. The discriminator model 400 can work in conjunction with cyclic or conditional GANs (cGANs) for classifying data. In one approach, generators A2B can transform a real image A taken in low light into a daytime representation B. Discriminator A processes the daytime representation B for backpropagation and subsequent transformations by generators B2A to generate cyclic image A. Cyclic image A can represent a reconstructed low-light image of the real image A to determine the reversibility of generators A2B. Discriminator B then compares the loss of the real image A to the cyclic image A to adjust the weights for training. In this approach, the system can train a cGAN by misleading discriminator B instead of minimizing the distance to a specific object in the image, so that the model learns unsupervised. In discriminator model 400, this process continues until the GAN converges to complete training.

[0030] Furthermore, GANs trained on real or synthetic images can generate new images that appear real to humans while possessing realistic characteristics, albeit with fundamental changes. Here, the GAN can generate new synthetic images with enhanced, controlled light characteristics compared to realistic renderings. The rendered images can preserve the semantic relationships of content and distance in low-light environments, allowing the prediction system to generate depth maps.

[0031] Once the sequence of composite images is transformed, the prediction system 170 can calculate the pose loss by using a pose model to compare the low-light composite image with the daytime representation. The pose model can be a neural network that estimates the time, position, location, angle, etc., of the image sensor or camera relative to objects in the image. For the vehicle 100, the pose loss may change due to the camera's movement from the road or environmental conditions. Therefore, the pose model can consider and adjust the camera model according to the pose loss between the sequence of composite images and the daytime representation. In one approach, the pose model can be a convolutional neural network (CNN), a PoseNet model, etc., that estimates the pose of a human or animal in the composite image. Furthermore, the pose loss is calculated using a Style Model. n-d The transformation of the composite image by 330 can be minimized when it reaches a threshold for the image sequence.

[0032] Once the prediction system 170 computes a sequence of daytime representations, the depth model 320 can predict a depth map from the sequence. In one approach, the depth model 320 can be a GAN or cGAN that predicts depth according to a given input image. Thus, the cGAN can regulate noise while generating an image for a depth map whose content is substantially similar to the input synthetic daytime representation. The output of the cGAN can be a depth map in grayscale, where the intensity of each pixel represents the distance between the object and the image sensor or camera.

[0033] Regarding the architecture of the depth model 320, Figure 5 illustrates one embodiment of an encoder / decoder network 500 for estimating the depth of objects in an image in a low-light environment. In one approach, the encoder / decoder network 500 can be a diabolic-shaped deep neural network (DNN) that uses multiple layers to transform the intensity space of the original pixels in the synthetic daytime representation. In particular, the encoder / decoder network 500 can non-linearly transform the original pixels into other spaces that enhance the intrinsic properties of the synthetic daytime representation.

[0034] Furthermore, encoder layers 3, 64, 128, 256, and 512 can utilize 3x3 two-dimensional (2D) convolutional layers and normalized linear units pre-trained across different layers. In one approach, encoder layers 64, 128, 256, and 512 can utilize maximum pooling operations within the layer to output prominent features of the previous feature map. On the decoder side, decoder layers 1, 64, 128, 256, and 512 can utilize 3x3 two-dimensional convolutional layers and normalized linear units within the layer. Additionally, decoder layers 32, 64, 128, 256, and 512 can utilize convolutional transpose in two dimensions, stride of 2, and normalized linear units. Thus, the output of the encoder / decoder network 500 can be a feature map having a group of matrices with numerical elements for generating a depth map.

[0035] Regarding additional losses, the prediction system 170 can calculate the flow loss using a flow model on the depth map generated by the depth model 320. The flow model can be a neural network that determines the flow loss for the dynamic movement of objects between the daytime representation and the sequence of depth maps to fit the depth model. For example, the flow model can use edge or boundary information to identify the dynamic movement of objects. In one approach, the flow model can be a network such as a CNN, optical flow model, or FlowNet model that stacks two consecutively adjacent input images to estimate the movement of objects. Furthermore, the depth model 320 can be fitted using the flow model for smoother and slower changes in the depth map.

[0036] In the above description, the pause and flow models can be auxiliary tasks or constraints for improving the training of the prediction system 170. In particular, simultaneously adjusting the pause and flow losses can provide semantic consistency for various forms of training the depth model 320, and thus increase the depth accuracy for objects in composite images, especially low-light images. Semantic relationships can provide important information for a complete understanding of the scene structure. For example, the pause model can provide the prediction system 170 with observation points and origins that improve depth estimation. At the same time, optical flow is another semantic relationship of the scene for understanding the dynamic movement of objects across a sequence of images. In other words, systems that rely on semantic consistency for training can prevent dilution of generated content between models during the training phase by using the pause and flow models as supervision.

[0037] Regarding further training for depth prediction, the prediction system 170 can calculate the loss L1 in the first training stage 310 to minimize the loss. L1 = a1 × L sup1 +a2×L pose +a3×L flow+a4×L Style Modeln-d +a5×L Depth Model Equation (1) The variables a1 to a5 in Equation (1) can be hyperparameters used to control the learning process in the first training stage 310. In addition to the loss described above, the prediction system 170 compares the depth map predicted by the depth model 320 with the ground truth generated according to the lidar simulation, thereby obtaining the supervised loss L sup1 can be calculated. For example, the supervised loss can represent the photometric loss per pixel between the predicted depth map and the ground truth depth map.

[0038] Furthermore, in Equation (1), L Style Modeln-d and L Depth Model can be the forward or nominal losses of the style and depth models respectively, used together with the pose loss L pose and the flow loss L flow to calculate the overall loss from generating the depth map. For example, L pose can be the difference in x, y, z coordinates between the low-light synthetic image and the daytime representation. Additionally, L flow can represent the per-pixel difference between the daytime representation and the depth map. In one approach, the prediction system 170 performs backpropagation of errors to adjust the parameters, weights, or internal coefficients of the Style Model n-d 330 or the depth model 320 until the L1 loss meets the criteria for predicting the depth map.

[0039] Looking at the second training stage 350 here, the prediction system 170 can use the actual image converted from the synthetic image by StyleGAN R-S 340. For example, StyleGAN R-S340 can convert real images from low-light conditions in the training dataset into synthetic images while preserving content, shading, shadows, etc. Figure 3B shows a two-stage system 302 having a second training stage 350 along with a first training stage 310. The second training stage 350 can process real images into synthetic daytime representations and inherits weights from the first training stage 310 to generate depth maps. Furthermore, a model of hidden variables for style distribution can also be set between the two stages to improve depth prediction. Hidden, or latent, variables can be data that is not observed but is part of the trained network. In one approach, the second training stage 350 can use real images from a monocular camera of vehicle 100 taken in a low-light environment for depth estimation, while the first training stage 310 uses simulated image data for the current traffic scene. In this way, the two-stage system 302 improves domain training and synthetic image processing for more accurate depth maps in low light by reducing image correspondence or correlation between stages.

[0040] For training to the second training stage 350, the prediction system 170 can calculate style transfer and supervision loss to minimize and satisfy the criteria for predicting depth maps associated with actual images. The prediction system 170 uses actual images from the second training stage 350 for actual realizations in the vehicle 100. In one approach, the prediction system 170 can compare the predicted depth map with a point cloud generated by a LiDAR simulation for a given night scene. In this way, the second training stage can utilize semi-supervision to improve depth prediction.

[0041] Regarding the second-stage loss, the prediction system 170 can calculate the total loss L2 in the second training stage 350 as follows in order to minimize the loss. L2 = a1 × L sup2 +a2×L StyleGANR-S Equation (2) In equation (2), variables a1 and a2 can be hyperparameters used to control the learning process in the second training stage 350. Here, L sup2 This can be a supervised loss calculated by comparing the depth map predicted by the depth model with the ground truth generated according to the lidar simulation in the second training stage 350. For example, the supervised loss can represent the photometric loss per pixel between the predicted depth map and the ground truth depth map. Furthermore, the loss L StyleGANR-S This can represent a forward or nominal loss associated with converting a real image to a composite image. For example, this loss can represent color intensity, scaling, and other information lost by converting a real image taken in low-light conditions to a composite image. Thus, the prediction system 170 can process real images for a second training stage until the L2 loss meets the criteria for predicting the depth map.

[0042] Looking at Figures 6A and 6B, a flowchart of Method 600 associated with improving depth prediction for objects in low-light images is illustrated. Method 600 is considered in terms of the prediction system 170 shown in Figures 1 and 2. Although Method 600 is considered in combination with the prediction system 170, it should be recognized that Method 600 is not limited to being implemented within the prediction system 170, and that the prediction system 170 is just one example of a system that can implement Method 600. For a brief explanation, in various implementations, Method 600 may include two training stages. The first training stage converts a composite image of a low-light situation associated with a traffic scene into an augmented light image for training the prediction system 170. The first training stage can operate in a composite domain to simplify its operation by reducing reliance on real images, image-to-image, real ground truth, etc., by using simulated image data for the traffic scene. Furthermore, the second training stage can use actual images from the training dataset or from the vehicle 100's monocular camera, captured in low-light environments, to complete the training. In this way, the two-stage system improves domain training and composite image processing for more accurate depth maps in low light by reducing image correspondence or correlation between stages.

[0043] In 610, the prediction system 170 converts the composite images to daytime representations using the style model from the first training phase. As described above, the prediction system 170 can convert a sequence of composite images from a low-light environment to a sequence of daytime representations using the style model. With respect to illumination levels, low light can represent an illumination environment in which systems using monocular cameras, GRB cameras, optical cameras, machine vision, etc., may have difficulty identifying object features due to reduced color intensity. Also, as described above, the style model can be a GAN that performs style or domain transfer by predicting how object features will appear, such as color, shadows, shading, and lighting conditions. Thus, daytime representations can have the same content while changing the lighting or style to increase richness for depth processing.

[0044] In 620, the prediction system 170 uses a depth model to output a depth map of the daytime representation. In one approach, once the prediction system 170 computes a sequence of daytime representations, the depth model can predict a depth map from the sequence. The depth map may be a grayscale where the intensity of each pixel represents the distance between an object and the image sensor or camera. In one approach, the depth model can be a GAN or cGAN that predicts depth according to a conditional input image. Thus, the cGAN can regulate noise while generating an image for a depth map whose content is substantially similar to a synthetic daytime input.

[0045] In 630, the prediction system 170 calculates the style model loss using a pose model and the depth model loss using a flow model during the first training phase. Details of the loss functions used by the prediction system 170 are described above. The pose model can be a network (e.g., CNN) that estimates the time, position, location, angle, etc., of an object in an acquired image from an image sensor or camera. In one approach, the pose loss can be minimized when the transformation of the composite image by the style model reaches a threshold. Furthermore, the flow model can be a network (e.g., CNN) that determines the flow loss for the dynamic movement of an object between a sequence of daytime representations and a sequence of depth maps. Thus, the depth model can be adapted using the flow model to smoother and slower changes in the sequence of depth maps, thereby improving the network's performance.

[0046] In 640, the prediction system 170 adjusts the parameters for training according to the loss. In one approach, the pause and flow models can be an auxiliary task for improving training in the synthetic domain. In particular, adjusting the pause and flow losses simultaneously can provide semantic consistency across forms for training the depth model, thereby improving the accuracy of depth estimates for objects in the synthetic image. This consistency can particularly improve the processing of low-light images where objects are not very prominent. For example, the pause model can provide the prediction system 170 with observation points and origins that improve depth estimation. As described above, optical flow is another semantic relationship for understanding the dynamic movement of objects across a sequence of images. Therefore, for training, systems that rely on semantic consistency can prevent dilution of the generated content between models during training by using the pause and flow models as supervision.

[0047] At step 650, the prediction system 170 adjusts the parameters of the depth model and continues training until a criterion for the model parameters or weights converges or is met. For example, the criterion could be that the loss meets a threshold or magnitude for resolution. Therefore, when the criterion is not met, the prediction system 170 can use additional composite images of the traffic scene and perform further training.

[0048] In 660, the prediction system 170 proceeds to the second training stage by using an adversarial network to convert real-world images into a composite representation after completing the first training stage. For example, low-light or nighttime images of a real-world scene can be converted into a composite representation. The prediction system 170 can be trained with real-world images in the second training stage for actual realizations in vehicle 100. As described above, the second training stage can inherit weights from the first training stage for the conversion in order to improve and complete the training of the network using real-world images. In one approach, the adversarial network can be a GAN, and the nighttime images can be real-world images corresponding to traffic scenes in the first training stage.

[0049] In 670, the prediction system 170 processes the composite representation using daytime style and depth models. In connection with the processing, the prediction system 170 can calculate style transfer and supervision loss to minimize and satisfy criteria for predicting depth maps associated with actual images. In one approach, the prediction system 170 can compare the predicted depth map with a point cloud generated by a LiDAR simulation for a given nighttime scene, in order to avoid relying on actual LiDAR measurements from the vehicle 100.

[0050] In step 680, the prediction system 170 uses loss by processing real images for training. Specifically, the prediction system 170 can use style transfer and supervised loss by generating depth maps of synthetic daytime images associated with real images. In step 690, processing in the second training stage continues until the parameters meet a certain criterion. For example, the criterion may be that the loss from the adversarial network and the depth loss meet a threshold for estimating a depth map for a certain time of day associated with the vehicle 100. Furthermore, if the criterion is not met, the prediction system 170 can use additional real images of the traffic scene and perform a further second training stage. In this way, the two-stage system improves domain training and synthetic image processing for more accurate depth maps in low light by reducing image correspondence or correlation between stages.

[0051] Figure 7 shows an example of a depth map for a low-light environment generated by a system trained using two stages. After training remotely or on a server, the vehicle 100 can have the predictive system 170 pre-installed on its electronic control unit as a recognition module for real-time navigation, safety control, automation, etc. The predictive system 170 can use low-light or nighttime images 710 of a real scene, such as from a monocular camera, to generate a daytime representation 720 using style transformation. In one approach, the predictive system 170 can use a GAN for style transformation. The GAN or cGAN of the predictive system 170 can use the daytime representation 720 to predict a depth map 730. Subsequently, the predictive system 170 can provide the depth map to other systems in the vehicle 100 for navigation, safety control, automation, etc.

[0052] Here, Figure 1 is examined in sufficient detail as an example environment in which the system and method disclosed herein can operate. However, in various implementations, the system and method can operate in any device such as CE devices, mobile devices, robots, drones, vehicles, cloud-based services, etc., as described above. In some examples, vehicle 100 is configured to selectively switch between different modes of operation / control according to the instructions of one or more modules / systems of vehicle 100. In one approach, the modes include 0, no automation, 1, driver assistance, 2, partial automation, 3, conditional automation, 4, high automation, and 5, full automation. In one or more configurations, vehicle 100 can be configured to operate in a subset of the possible modes.

[0053] In one or more embodiments, the vehicle 100 is an automated or autonomous vehicle. As used herein, “autonomous vehicle” means a vehicle that can operate in an autonomous mode (e.g., Category 5, fully automated). “Automated mode” or “autonomous mode” means driving and / or steering the vehicle 100 along a route using one or more computing systems for controlling the vehicle 100 with minimal or no input from a human driver. In one or more embodiments, the vehicle 100 is highly automated or fully automated. In one embodiment, the vehicle 100 is configured in one or more semi-autonomous operating modes in which one or more computing systems perform part of the driving and / or steering of the vehicle along a route, and the vehicle operator (i.e., driver) provides input to the vehicle for performing part of the driving and / or steering of the vehicle 100 along a route.

[0054] Vehicle 100 includes one or more processors 110. In one or more configurations, a processor 110 may be the main processor of vehicle 100. For example, a processor 110 may be an ECU, an application-specific integrated circuit (ASIC), a microprocessor, etc. Vehicle 100 may include one or more data storage devices 115 for storing one or more types of data. The data storage devices 115 may include volatile and / or non-volatile memory. Examples of suitable data storage devices 115 include RAM, flash memory, ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, and hard drives. The data storage devices 115 may be components of a processor 110, or a data storage device 115 may be functionally connected to a processor 110 for use by the processor 110. The term “functionally connected” as used throughout this description may include direct or indirect connections, including connections without direct physical contact.

[0055] In one or more configurations, one or more data storage devices 115 may include map data 116. Map data 116 may include maps of one or more geographical areas. In some examples, map data 116 may include information or data about roads, traffic control devices, road signs, structures, features, and / or landmarks in one or more geographical areas. Map data 116 can be any appropriate shape. In some examples, map data 116 may include an aerial view of the area. In some examples, map data 116 may include a ground map of the area, including a 360-degree ground map. Map data 116 may include measurements, dimensions, distances, and / or information for one or more items included in map data 116, and / or for other items included in map data 116. Map data 116 may include a digital map having information about the relative positions of roads.

[0056] In one or more arrangements, map data 116 may include one or more terrain maps 117. A terrain map 117 may include information about the topography, roads, ground surface, and / or other features of one or more geographical regions. A terrain map 117 may include elevation data in one or more geographical regions. A terrain map 117 may define one or more ground surfaces, which may include paved roads, unpaved roads, land, and other things that define the ground surface.

[0057] In one or more placements, the map data 116 may include one or more static obstacle maps 118. A static obstacle map 118 may include information about one or more static obstacles located within one or more geographical areas. A “static obstacle” is a physical object whose position does not change or substantially change over time, and / or whose size does not change or substantially change over time. Examples of static obstacles include trees, buildings, curbs, walls, fences, median strips, utility poles, statues, monuments, signs, benches, furniture, mailboxes, large stones, and slopes. Static obstacles can be objects that extend above the ground. One or more static obstacles included in a static obstacle map 118 may have location data, size data, dimension data, material data, and / or other data associated with them. A static obstacle map 118 may include measurements, dimensions, distances, and / or information for one or more static obstacles. A static obstacle map 118 may be of high quality and / or highly detailed. The static obstacle map 118 can be updated to reflect changes within the area described in the map.

[0058] One or more data storage devices 115 may contain sensor data 119. In this context, “sensor data” means any information about the sensors equipped on the vehicle 100, including the function and other information about such sensors. The vehicle 100 may include a sensor system 120, as described below. The sensor data 119 may relate to one or more sensors of the sensor system 120. For example, in one or more configurations, the sensor data 119 may include information about one or more lidar sensors 124 of the sensor system 120.

[0059] In some examples, at least a portion of the map data 116 and / or sensor data 119 may be stored in one or more data storage devices 115 located on the vehicle 100. Alternatively, or additionally, at least a portion of the map data 116 and / or sensor data 119 may be stored in one or more data storage devices 115 located away from the vehicle 100.

[0060] As described above, the vehicle 100 may include a sensor system 120. The sensor system 120 may include one or more sensors. "Sensor" means a device capable of detecting and / or sensing something. In at least one embodiment, one or more sensors detect and / or sense in real time. As used herein, the term "real time" means a level of processing response that the user or system perceives as sufficiently immediate in response to a particular process or decision being performed, or a level of processing response that allows a processor to remain responsive to some external process.

[0061] In an arrangement where the sensor system 120 includes multiple sensors, the sensors can function independently of each other, or two or more sensors can function in combination. The sensor system 120 and / or one or more sensors can be functionally connected to the processor 110, the data storage device 115, and / or other elements of the vehicle 100. The sensor system 120 can generate observational results about a part of the environment of the vehicle 100 (e.g., nearby vehicles).

[0062] The sensor system 120 may include any suitable type of sensor. Various examples of different types of sensors are described herein. However, it will be understood that embodiments are not limited to the specific sensors described. The sensor system 120 may include one or more vehicle sensors 121. The vehicle sensors 121 can detect information about the vehicle 100 itself. In one or more configurations, the vehicle sensors 121 may be configured to detect changes in the position and orientation of the vehicle 100, for example, based on inertial acceleration. In one or more configurations, the vehicle sensors 121 may include one or more accelerometers, one or more gyroscopes, inertial measuring units (IMUs), dead reckoning systems, global navigation satellite systems (GNSS), global positioning systems (GPS), navigation systems 147, and / or other suitable sensors. The vehicle sensors 121 may be configured to detect one or more characteristics of the vehicle 100 and / or how the vehicle 100 is operating. In one or more configurations, the vehicle sensors 121 may include a speedometer that determines the current speed of the vehicle 100.

[0063] Alternatively, or additionally, the sensor system 120 may include one or more environmental sensors 122 configured to acquire data about the environment surrounding the vehicle 100 in which the vehicle 100 is operating. "Environmental data" includes data about the external environment in which the vehicle is located, or about one or more parts thereof. For example, one or more environmental sensors 122 may be configured to sense obstacles in at least part of the external environment of the vehicle 100, and / or data about such obstacles. Such obstacles may be static and / or dynamic objects. One or more environmental sensors 122 may be configured to detect other things in the external environment of the vehicle 100, such as lane markings, signs, traffic lights, traffic signs, lane lines, crosswalks, curbs near the vehicle 100, and objects outside the road.

[0064] Various examples of sensors of the sensor system 120 are described here. The example sensors may be part of one or more environmental sensors 122 and / or one or more vehicle sensors 121. However, it will be understood that the embodiments are not limited to the specific sensors described.

[0065] For example, in one or more configurations, the sensor system 120 may include one or more of the following: a radar sensor 123, a lidar sensor 124, a sonar sensor 125, a weather sensor, a tactile sensor, a position sensor, and / or one or more cameras 126. In one or more configurations, one or more cameras 126 may be high dynamic range (HDR) cameras, stereo cameras, or infrared (IR) cameras.

[0066] Vehicle 100 may include an input system 130. The "input system" includes components or arrangements or groups thereof that enable various entities to input data into the machine. The input system 130 can receive input from a person riding in the vehicle. Vehicle 100 may also include an output system 135. The "output system" includes one or more components that facilitate the presentation of data to a person riding in the vehicle.

[0067] Vehicle 100 may include one or more vehicle systems 140. Various examples of one or more vehicle systems 140 are shown in Figure 1. However, vehicle 100 may include more, fewer, or different vehicle systems. Although special vehicle systems are defined separately, it should be recognized that any system, or parts thereof, can be combined or separated within vehicle 100 via hardware and / or software. Vehicle 100 may include a propulsion system 141, a braking system 142, a steering system 143, a throttle system 144, a transmission system 145, a signaling system 146, and / or a navigation system 147. Any of these systems may include one or more devices, components, and / or combinations thereof, which are currently known or may be developed in the future.

[0068] The navigation system 147 may include one or more currently known or to be developed devices, applications, and / or combinations thereof configured to determine the geographical location of the vehicle 100 and / or determine a driving route for the vehicle 100. The navigation system 147 may include one or more mapping applications for determining a driving route for the vehicle 100. The navigation system 147 may include a global positioning system, a local positioning system, or a geolocation system.

[0069] The processor 110, the predictive system 170, and / or the autonomous driving module 160 can be functionally connected to communicate with various vehicle systems 140 and / or their individual components. For example, returning to Figure 1, the processor 110 and / or the autonomous driving module 160 can be in a state of communication to send information to and / or receive information from various vehicle systems 140 in order to control the movement of the vehicle 100. The processor 110, the predictive system 170, and / or the autonomous driving module 160 can control some or all of the vehicle systems 140 and can therefore be partially or fully autonomous as defined by the Society of Automotive Engineers (SAE) levels 0 through 5.

[0070] The processor 110, the prediction system 170, and / or the autonomous driving module 160 can be functionally connected to communicate with various vehicle systems 140 and / or their individual components. For example, returning to Figure 1, the processor 110, the prediction system 170, and / or the autonomous driving module 160 can be in a state of communication to send information to and / or receive information from various vehicle systems 140 in order to control the movement of the vehicle 100. The processor 110, the prediction system 170, and / or the autonomous driving module 160 can control some or all of the vehicle systems 140.

[0071] The processor 110, the predictive system 170, and / or the autonomous driving module 160 can operate to control the driving and steering of the vehicle 100 by controlling one or more of the vehicle system 140 and / or its components. For example, when operating in autonomous mode, the processor 110, the predictive system 170, and / or the autonomous driving module 160 can control the direction and / or speed of the vehicle 100. The processor 110, the predictive system 170, and / or the autonomous driving module 160 can cause the vehicle 100 to accelerate, decelerate, and / or change direction. As used herein, “cause” or “make to” means to cause, compel, force, compel, direct, command, instruct, and / or enable to happen, or to cause, compel, compel, direct, command, instruct, and / or enable to become such an event or action, either directly or indirectly, or to at least be in a state where such an event or action could happen.

[0072] The vehicle 100 may include one or more actuators 150. The actuators 150 may be elements or combinations of elements that can operate to modify one or more of the vehicle system 140 or its components in response to receiving signals or inputs from the processor 110 and / or the autonomous driving module 160. For example, one or more actuators 150 may include, to name a few, motors, pneumatic actuators, hydraulic pistons, relays, solenoids, and / or piezoelectric actuators.

[0073] Vehicle 100 may include one or more modules, at least some of which are described herein. A module may be implemented as computer-readable program code, which, when executed by processor 110, implements one or more of the various processes described herein. One or more modules may be components of processor 110, or one or more modules may run on and / or be distributed across other processing systems to which processor 110 is functionally connected. A module may include instructions (e.g., program logic) that can be executed by one or more processors 110. Alternatively, or additionally, one or more data storage devices 115 may include such instructions.

[0074] In one or more configurations, one or more of the modules described herein may include artificial intelligence elements, such as neural networks, fuzzy logic, or other ML algorithms. Furthermore, in one or more configurations, one or more modules may be distributed among multiple modules described herein. In one or more configurations, two or more of the modules described herein may be combined into a single module.

[0075] Vehicle 100 may include one or more autonomous driving modules 160. The autonomous driving modules 160 can be configured to receive data from sensor systems 120 and / or other types of systems that can take in information about the vehicle 100 and / or the external environment of the vehicle 100. In one or more configurations, the autonomous driving modules 160 can use such data to generate one or more driving scene models. The autonomous driving modules 160 can determine the position and speed of the vehicle 100. The autonomous driving modules 160 can determine the position of obstacles, obstacles, or other environmental features, including traffic signs, trees, shrubs, nearby vehicles, pedestrians, etc.

[0076] The autonomous driving module 160 can be configured to receive and / or determine positional information of the vehicle 100 relative to obstacles in the external environment for use by one or more of the processors 110 and / or modules described herein, in order to estimate the position and orientation of the vehicle 100, which is the position of the vehicle 100 in global coordinates, based on signals from multiple satellites, or for use in determining the current state of the vehicle 100, or when creating a map, or when determining the position of the vehicle 100 relative to map data, based on any other data and / or signals that can be used to determine the position of the vehicle 100 relative to its environment.

[0077] The autonomous driving module 160 can be configured, independently or in combination with the predictive system 170, to determine the driving path, the current autonomous driving maneuvers for the vehicle 100, future autonomous driving maneuvers, and / or modifications to the current autonomous driving maneuvers, based on data from any other suitable source, such as data acquired by the sensor system 120, a driving scene model, and / or decisions from the sensor data 240. “Driving maneuvers” means one or more actions that affect the movement of the vehicle. Some possible examples of driving maneuvers include acceleration, deceleration, braking, turning, moving the vehicle 100 sideways, changing lanes, merging into lanes, and / or reversing. The autonomous driving module 160 can be configured to implement the determined driving maneuvers. The autonomous driving module 160 can be made to implement such autonomous driving maneuvers directly or indirectly. As used herein, “to cause” or “to make something happen” means, either directly or indirectly, to cause an event or action to occur, to command it to occur, to instruct it to occur, and / or to make it happen, or to cause, command, instruct, and / or make it happen so that such an event or action can at least be in a state where it can occur. The autonomous driving module 160 can be configured to perform various vehicle functions, to transmit data to the vehicle 100 or one or more of its systems (e.g., one or more of the vehicle systems 140), to receive data from them, to interact with them, and / or to control them.

[0078] Detailed embodiments are disclosed herein. However, it should be understood that the disclosed embodiments are intended to be illustrative. Accordingly, the details of the specific structures and functions disclosed herein should not be construed as restrictive, but merely as grounds for the claims and as representative grounds for teaching a person skilled in the art to employ various aspects herein in substantially any appropriately detailed structure. Furthermore, the terms and phrases used herein are not intended to be restrictive, but are intended to provide an understandable description of possible implementations. Various embodiments are shown in Figures 1–7, but embodiments are not limited to the exemplified structures or applications.

[0079] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, a block in a flowchart or block diagram can represent a module, segment, or portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions described in a block may occur in a different order than that shown in the diagram. For example, depending on the functions involved, two blocks shown consecutively may actually be executed substantially simultaneously, or blocks may be executed in reverse order.

[0080] The systems, components, and / or processes described herein can be implemented in hardware or a combination of hardware and software, and can be implemented centrally in a single processing system or distributed across several interconnected processing systems with different elements. Any type of processing system or other device adapted to perform the methods described herein is suitable. A typical combination of hardware and software may be a processing system having computer-readable program code that, when deployed and executed, controls the processing system to perform the methods described herein.

[0081] Systems, components, and / or processes can also be embedded in computer-readable storage devices, such as computer program products or other data program storage devices, that are machine-readable and substantially contain a program of machine-executable instructions for performing the methods and processes described herein. These elements, too, possess features that enable implementations of the methods described herein and, when deployed in a processing system, can be embedded in application products capable of performing these methods.

[0082] Furthermore, the arrangements described herein can take the form of a computer program product contained in one or more computer-readable media having, for example, stored computer-readable program code. Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The phrase "computer-readable storage medium" means a non-temporary storage medium. A computer-readable storage medium can be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof, but is not limited to the following. More specific examples (not an exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disk drives (HDDs), solid-state drives (SSDs), ROMs, EPROMs or flash memory, portable compact disk read-only memory (CD-ROMs), digital multipurpose discs (DVDs), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium that contains or can store programs for use by, or in conjunction with, instruction execution systems, apparatus, or devices.

[0083] Modules used herein generally include routines, programs, objects, components, data structures, etc., that perform a special task or realize a special data type. In a further embodiment, memory generally includes a well-known module. The memory associated with the module may be a buffer or cache embedded in a processor, RAM, ROM, flash memory, or other suitable electronic storage medium. In a further embodiment, the modules considered in this disclosure may be implemented as an ASIC, as a hardware component of a system-on-a-chip (SoC), as a programmable logic array (PLA), or as other suitable hardware component embedded with a defined set of configurations (e.g., instructions) for performing the disclosed functions.

[0084] The program code contained in the computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical fiber, cable, radio frequency (RF), or any suitable combination thereof. The computer program code for performing the operations for this configuration is Java®. TMProgram code can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and traditional procedural programming languages ​​such as the C programming language or similar languages. The program code can run entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can connect to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or can connect to an external computer (for example, through the internet using an Internet service provider).

[0085] The term “one” as used herein is defined as one or more. The term “multiple” as used herein is defined as two or more. The term “other” as used herein is defined as at least the second or subsequent. The terms “contains” and / or “have” as used herein are defined as “equipped” (i.e., open language). The phrase “at least one of ~ and ~” as used herein means any and all combinations of one or more of the items listed together, including them. For example, the phrase “at least one of A, B, and C” includes A, B, C, or any combination thereof (e.g., AB, AC, BC, or ABC).

[0086] The embodiments described herein can be embodied in other forms without departing from their spirit or essential attributes. Therefore, to illustrate their scope, one should refer to the claims below, rather than the aforementioned specification. The inventions disclosed herein include the following embodiments: [Aspect 1] A prediction system for depth perception, Processor and A memory that stores instructions, and when an instruction is executed by the processor, the processor receives In the first training phase, the model is made to calculate losses, including pause loss, flow loss, and supervised loss, which are associated with predicting depth maps for composite images of low-light scenes. The style model and depth model are adjusted according to the aforementioned loss. In the second training phase, the depth model is trained using a composite representation of low-light images. To provide the aforementioned depth model, Memory and A prediction system equipped with the following features. [Aspect 2] The prediction system according to embodiment 1, further comprising instructions to convert the composite image to a daytime representation using the style model, and to output other depth maps from the daytime representation using the depth model to determine the supervised loss. [Aspect 3] The prediction system according to embodiment 2, wherein the instructions for adjusting the style model and the depth model further include instructions for adjusting the movement of an object between the daytime representation and the previous representation using the flow loss. [Aspect 4] The prediction system according to embodiment 1, wherein the instructions for calculating the loss further include instructions for determining the pose loss of the style model using a pose model and the flow loss of the depth model using a flow model. [Aspect 5] The prediction system according to embodiment 1, further comprising instructions in the second training stage to use an adversarial network to convert the low-light image into the composite representation and to use the composite representation to form other depth maps according to the style model and the depth model. [Aspect 6] The prediction system according to embodiment 5, wherein the instructions for training the depth model further include instructions for training the adversarial network according to the semi-supervised loss of the depth map. [Aspect 7] The prediction system according to embodiment 1, further comprising instructions for converting the composite representation to a daytime representation using the style model in order to determine the depth prediction for the second training stage. [Aspect 8] The prediction system according to embodiment 1, wherein the instructions for adjusting the style model and the depth model further include instructions for adjusting the style model for motion using a pose model for a series of composite images associated with the low-light scene. [Aspect 9] A non-temporary computer-readable medium containing instructions for training a prediction system, wherein the instructions, when executed by a processor, are directed to the processor. In the first training phase, the model is made to calculate losses, including pause loss, flow loss, and supervised loss, which are associated with predicting depth maps for composite images of low-light scenes. The style model and depth model are adjusted according to the aforementioned loss. In the second training phase, the depth model is trained using a composite representation of low-light images. To provide the aforementioned depth model, Non-temporary computer-readable media. [Aspect 10] A non-temporal computer-readable medium according to embodiment 9, further comprising instructions for converting the composite image to a daytime representation using the style model, and for outputting other depth maps from the daytime representation using the depth model to determine the supervised loss. [Aspect 11] The non-temporary computer-readable medium according to embodiment 10, wherein the instructions for adjusting the style model and the depth model further include instructions for adjusting the movement of an object between the daytime representation and the previous representation using the flow loss. [Aspect 12] The non-temporal computer-readable medium according to aspect 9, wherein the instructions for calculating the losses further include instructions for determining the pose loss of the style model using a pose model and the flow loss of the depth model using a flow model. [Aspect 13] In the first training phase, the calculation of losses, including pause loss, flow loss, and supervised loss, is associated with predicting a depth map for a composite image of a low-light scene. Adjusting the style model and depth model according to the aforementioned loss, In the second training phase, the depth model is trained using a composite representation of low-light images, To provide the aforementioned depth model, A method that includes [Aspect 14] Using the aforementioned style model, the composite image is converted to a daytime representation, To determine the supervised loss, the depth model is used to output other depth maps from the daytime representation, The method according to embodiment 13, further comprising the following: [Aspect 15] The method according to embodiment 14, wherein adjusting the style model and the depth model further includes using the flow loss to adjust the movement of an object between the daytime representation and the previous representation. [Aspect 16] The method according to embodiment 13, wherein calculating the loss further includes determining the pose loss of the style model using a pose model and the flow loss of the depth model using a flow model. [Aspect 17] The method according to embodiment 13, further comprising using an adversarial network to convert the low-light image into the composite representation and forming other depth maps using the composite representation according to the style model and the depth model. [Aspect 18] The method according to embodiment 17, wherein training the depth model further comprises training the adversarial network according to the semi-supervised loss of the depth map. [Aspect 19] The method according to embodiment 13, further comprising using the style model to convert the composite representation to a daytime representation in order to determine the depth prediction for the second training stage. [Aspect 20] The method according to aspect 13, wherein adjusting the style model and the depth model further includes adjusting the style model for motion using a pose model for a sequential composite image associated with the low-light scene.

Claims

1. A prediction system for depth perception, Processor and A memory that stores instructions, and when an instruction is executed by the processor, the processor receives In the first training phase, the simulated model associated with a low-light environment is made to predict a depth map for a composite image of a low-light scene generated, and is made to calculate losses including pose loss related to the time, position, location, and angle of the image sensor or camera for objects in the composite image, flow loss related to the dynamic movement of the objects, and supervised loss which is the photometric loss per pixel between the depth map and the ground truth depth map. By adjusting the parameters using the aforementioned loss, the style model and depth model are adjusted. In the second training phase, the depth model is trained by adjusting the parameters using a composite representation of low-light images converted from actual images. To provide the aforementioned depth model, Memory and A prediction system equipped with the following features.

2. The prediction system according to claim 1, further comprising instructions to convert the composite image to a daytime representation using the style model, and to output another depth map from the daytime representation using the depth model to determine the supervised loss.

3. The prediction system according to claim 2, wherein the instructions for adjusting the style model and the depth model further include instructions for adjusting the movement of the object between the daytime representation and the previous representation using the flow loss.

4. Using the style model, the composite image is converted to a daytime representation, The prediction system according to claim 1, wherein the instructions for calculating the loss further include instructions for inputting the composite image of the low-light scene into a pose model, using the pose model to determine the pose loss of the style model, and inputting the daytime representation into a flow model, using the flow model to determine the flow loss of the depth model.

5. The prediction system according to claim 1, further comprising, in the second training stage, instructions for using an adversarial network to convert the low-light image into the composite representation and for using the composite representation to form other depth maps according to the style model and the depth model.

6. The prediction system according to claim 5, wherein the instructions for training the depth model further include instructions for training the adversarial network according to the semi-supervised loss of the depth map.

7. The prediction system according to claim 1, further comprising instructions for converting the synthetic representation to a daytime representation using the style model in order to determine the depth prediction for the second training stage.

8. The prediction system according to claim 1, wherein the instructions for adjusting the style model and the depth model further include instructions for adjusting the style model for motion using a pose model for the sequential composite images associated with the low-light scene.

9. A non-temporary computer-readable medium containing instructions for training a prediction system, wherein the instructions, when executed by a processor, are directed to the processor. In the first training phase, the simulated model associated with a low-light environment is made to predict a depth map for a composite image of a low-light scene generated, and is made to calculate losses including pose loss related to the time, position, location, and angle of the image sensor or camera for objects in the composite image, flow loss related to the dynamic movement of the objects, and supervised loss which is the photometric loss per pixel between the depth map and the ground truth depth map. By adjusting the parameters using the aforementioned loss, the style model and depth model are adjusted. In the second training phase, the depth model is trained by adjusting the parameters using a composite representation of low-light images converted from actual images. To provide the aforementioned depth model, Non-temporary computer-readable media.

10. The non-temporal computer-readable medium according to claim 9, further comprising instructions for converting the composite image to a daytime representation using the style model, and for outputting another depth map from the daytime representation using the depth model to determine the supervised loss.

11. The non-temporary computer-readable medium according to claim 10, wherein the instructions for adjusting the style model and the depth model further include instructions for adjusting the movement of the object between the daytime representation and the previous representation using the flow loss.

12. Using the style model, the composite image is converted to a daytime representation, The non-temporal computer-readable medium according to claim 9, wherein the instructions for calculating the loss further include instructions for inputting the composite image of the low-light scene into a pose model and using the pose model to determine the pose loss of the style model, and for inputting the daytime representation into a flow model and using the flow model to determine the flow loss of the depth model.

13. In the first training phase, the system calculates losses including pose loss related to the time, position, location, and angle of an image sensor or camera relative to an object in a composite image, flow loss related to the dynamic movement of the object, and supervised loss which is the photometric loss per pixel between the depth map and the ground truth depth map, all associated with predicting a depth map for a composite image of a low-light scene generated in a simulated model associated with a low-light environment. By adjusting the parameters using the aforementioned loss, the style model and depth model are adjusted. In the second training phase, the depth model is trained by adjusting the parameters using a composite representation of low-light images converted from actual images. To provide the aforementioned depth model, A method that includes this.

14. Converting the composite image to a daytime representation using the aforementioned style model, To determine the supervised loss, the depth model is used to output another depth map from the daytime representation, The method according to claim 13, further comprising:

15. The method according to claim 14, wherein adjusting the style model and the depth model further includes using the flow loss to adjust the movement of the object between the daytime representation and the previous representation.

16. Using the style model, the composite image is converted to a daytime representation, The method according to claim 13, wherein calculating the loss further comprises inputting a composite image of the low-light scene into a pose model and using the pose model to determine the pose loss of the style model, and inputting the daytime representation into a flow model and using the flow model to determine the flow loss of the depth model.

17. The method according to claim 13, further comprising, in the second training stage, using an adversarial network to convert the low-light image into the composite representation, and using the composite representation to form other depth maps according to the style model and the depth model.

18. The method according to claim 17, wherein training the depth model further comprises training the adversarial network according to the semi-supervised loss of the depth map.

19. The method according to claim 13, further comprising using the style model to convert the composite representation to a daytime representation in order to determine the depth prediction for the second training stage.

20. The method according to claim 13, wherein adjusting the style model and the depth model further includes adjusting the style model for motion using a pose model for the sequential composite images associated with the low-light scene.

Images