Hazard classification training method, hazard classification method, assisted or automatic vehicle driving system

Abstract

The present invention relates to a hazard classification training method, a hazard classification method, and an assisted or automatic vehicle driving system. The hazard classification training method comprises: training a first deep neural network for general object recognition in a general image; training a second deep neural network for specific object recognition in an image related to a specific application; training a third deep neural network for specific scene flow prediction in an image sequence related to the specific application; training a fourth deep neural network for locating potential dangerous areas in an image or image sequence related to the specific application using at least one human-trained hazard labeling method; training a fifth deep neural network for invisible specific object anticipation and / or visible specific object prediction in an image or image sequence related to the specific application; and determining at least one dangerous pixel in the image or image sequence using an end-to-end deep neural network that is a transfer learning sequence of five deep neural networks.

Description

Technical Field

[0001] The present invention generally relates to machine learning techniques, and more specifically, to a hazard classification training method, a hazard classification method, an assisted or automated vehicle driving system, and more specifically, to a deep neural network (DNN), such as a deep convolutional neural network (CNN) for processing spatial information and a recurrent neural network (RNN) / long short-term memory (LSTM) for processing temporal information. In particular, the present invention relates to a hazard classification training method for learning how to detect hazards in a training image or a training image sequence, and a trained hazard classification method for detecting hazards in a real-time image or a real-time image sequence.

[0002] This approach is particularly useful in the field of human-assisted or autonomous vehicles that use sensors for obstacle detection and avoidance in order to safely navigate their environment. Background Art

[0003] The publication by Chen et al., entitled “Atomic Scenes for Scalable Traffic Scene Recognition in Monocular Videos” discloses a system consisting of four main parts: object recognition, traffic scene recognition, prediction of driving conditions associated with specific objects (e.g., a sudden stop of the vehicle ahead), and motion planning to propose a safe path.

[0004] ( http: / / www.nec-labs.com / uploads / images / Department-Images / MediaAnalytics / pa pers / wacv16_atomicscenes.pdf ).

[0005] The drawback of this system is that it is essentially a rule-based system that is broken down into atomic and high-level levels. This system cannot cope with driving situations in the real world that have never been seen before and cannot be explained by rules or combinations of rules. Although this system attributes a predetermined level of danger to a specific object that is primarily responsible for a potential collision, it is unable to attribute different levels of danger to all objects in the driving scene, especially in scenarios where no actor is in serious danger. While this atomic scenario approach can provide sufficient hazard prediction accuracy in 90% of everyday driving scenarios, it cannot cope with rare and never-before-seen situations, which are the main reason for human failure.

[0006] The publication by Bojarski et al. titled “End to End Learning for Self-Driving Cars” ( https: / / arxiv.org / pdf / 1604.07316.pdf ) discloses a trained convolutional neural network (CNN) for mapping raw pixels from a single front-facing camera directly to steering commands.

[0007] The drawbacks of such systems are that they automatically learn internal representations of the necessary processing steps, such as detecting useful road features using only the human steering angle as a training signal, and activation maps that indicate the importance of these features and their locations for making vehicle control decisions. However, if no hazard-related labels are provided, such systems cannot learn a general set of hazard-related features on their own without human supervision.

[0008] The publication by Rummelhard et al. entitled “Probabilistic Grid-based Collision Risk Prediction for Driving Application” ( https: / / hal.inria.fr / hal-01011808 / document ) discloses a new grid-based collision risk prediction method.

[0009] Such systems are unable to cope with the estimation of future states associated with objects that are not visible in the scene, or such systems are unable to effectively estimate the hazard level associated with visible static objects.

[0010] Zeng et al.’s publication entitled “Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localization” ( https: / / arxiv.org / pdf / 1705.06560.pdf ) discloses an agent-centric approach to study accident anticipation and risk area localization tasks.

[0011] Such systems cannot cope with sparse labels and cannot estimate the danger level without risk events. Such systems require annotation of individual objects and cannot support the assessment of different risk levels for multiple potentially dangerous objects simultaneously.

[0012] The publication titled “Visual Forecasting by Imitating Dynamics in Natural Sequences” by Zeng et al. introduces a general framework for visual prediction that directly imitates visual sequences without additional supervision ( http: / / ai.stanford.edu / ~dahuang / papers / iccv17-vfid.pdf ).

[0013] The disadvantage of this approach is that it can only provide semantic keywords as prediction results, and it cannot generate dense pixel-level maps required for fine-scale hazard classification tasks.

[0014] The publication by Lee et al. titled “DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents” introduces a deep stochastic IOC1 RNN encoder / decoder framework for the task of distant future prediction for multiple interacting agents in dynamic scenes ( http: / / www.nec-labs.com / uploads / images / Department-Images / MediaAnalytics / pape rs / cvpr17_futureprediction.pdf ).

[0015] A drawback of this system is that only visible moving objects with past trajectories of significant duration available are used in order to generate future predictions. This system cannot be used to predict the future state of static or invisible objects in the scene, so if such predictions are used as the main input, the hazard level map will be incomplete.

[0016] The publication by Vondrick et al. entitled “Generating the Future with Adversarial Transformers” ( http: / / carlvondrick.com / transformer.pdf ) disclose a learning model for generating near-future images in videos.

[0017] The paper does not mention the possibility of generating future videos and classifying objects in this video to obtain a hazard map. The described method does not solve the problem of attributing different hazard levels to different objects and how to attribute hazard levels to all pixels including those on unrecognizable objects.

[0018] The publication by Vondrick et al. entitled “Predicting Motivations of Actions by Leveraging Text” ( http: / / carlvondrick.com / intention.pdf ) discloses understanding human behavior as a key problem in computer vision.

[0019] The paper does not mention the possibility of predicting object momentum to obtain a hazard map. In any case, such a model is limited to recognizable objects. Moreover, since the label considers objects at the bounding box level, the location of complex objects such as roads will not be determined with high accuracy.

[0020] The publication by Vu et al. titled “Predicting Actions from Static Scenes” aims to discover action-scene correlations for a large number of scene categories and use this correlation for action prediction ( http: / / www.di.ens.fr / willow / research / actionsfromscenes / paper / eccv14_ actionsfro mscenes.pdf ).

[0021] The paper does not mention the possibility of predicting actions to obtain a hazard map. The model only gives semantic keywords as prediction results, so it cannot be used to generate dense pixel-level maps required for fine-scale trajectory control.

[0022] Titled "CAD2RL: Real Single-Image Flight Without a Single Real Image (CAD 2 Sadeghi et al.'s publication (RL: Real Single Image Flight Without a Single Ground Truth Image) https: / / arxiv.org / pdf / 1611.04201.pdf ) proposed a learning method CAD 2 RL, a learning method that can be used to perform collision-free indoor flight in the real world while being fully trained on 3D CAD models.

[0023] The disadvantage of such a system is that it outputs a free space probability map based only on visible object detection and cannot account for hazards associated with the future state of the objects.

[0024] Ballan et al.’s publication entitled “Knowledge Transfer for Scene-specific Motion Prediction” ( https: / / arxiv.org / pdf / 1603.06987v2.pdf ) introduces a dynamic Bayesian network for trajectory prediction using scene-specific knowledge.

[0025] The disadvantage of this approach is that it relies on long-term data collection of scenes with moving objects. This approach cannot cope with sparse labels and dynamic camera motion used to collect training data. It takes a very long time to acquire training data for different situations and cannot cope with static objects or rare events. Summary of the invention

[0026] The present invention aims to solve the above-mentioned shortcomings of the prior art, and more specifically, firstly, the present invention aims to propose a hazard classification training method capable of detecting hazards in images or image sequences related to any specific application. The present invention also aims to propose a hazard classification training method, using a trained method to detect hazards in real-time images or image sequences related to specific applications and evaluate the hazard level, that is, each hazard is located and has an equal / consistent level for each application.

[0027] A first aspect of the present invention relates to a method for training risk classification in an image or image sequence related to a specific application, the method comprising the following steps:

[0028] Training a first deep neural network (DNN-A) for generic object recognition in generic images;

[0029] training a second deep neural network (DNN-B) for specific object recognition within the image associated with the specific application;

[0030] training a third deep neural network (DNN-C) for scene-specific flow prediction within an image sequence associated with a specific application;

[0031] training a fourth deep neural network (DNN-E) for locating potentially hazardous areas within an image or sequence of images relevant to a particular application using at least one human-trained labeling method for capturing how humans react and / or analyze an environment; and

[0032] Using a human-trained hazard labeling method, an end-to-end deep neural network (E2E DNN-4) is used to determine at least one hazard pixel within an image or image sequence related to a specific application. The end-to-end deep neural network (E2EDNN-4) is a transfer learning sequence of four deep neural networks, and the end-to-end deep neural network (E2E DNN-4) is followed by at least one end-to-end specific layer, preferably at least one top layer (E2E TL).

[0033] This hazard classification training method enables training an end-to-end deep neural network to determine at least one dangerous pixel in an image or image sequence related to a specific application, preferably representing the most dangerous pixel in the image or image sequence. Such an achievement can be achieved due to the different training steps including general and specific image recognition, specific scene flow prediction and potential danger area location. In fact, the image recognition step allows the training method to learn how to classify any object, especially any object related to a specific application. The scene flow prediction step ensures the learning of how objects move in the environment. The potential danger area location step trains the method of learning how humans react and / or analyze the environment (for example, in driving scenarios of automotive applications), where some of these scenarios may contain potential dangers and human reactions will be specific to the danger. Therefore, we learn how to distinguish potentially dangerous driving scenarios by using human reactions as labels. Finally, the end-to-end deep neural network (E2E DNN-4) also includes one or more end-to-end top layers, which receive trained features for each previous knowledge. These top layers also learn specific features of their own hazard classification steps and reuse features learned from previous steps. This end-to-end deep neural network is trained to learn a hazard classification method by replicating the way humans detect hazardous areas in situations represented by images or image sequences, and assesses hazard levels by collecting human-based hazard level labels in a systematic manner by asking humans to label hazards according to the same level definition of hazard protocol for all images or image sequences.

[0034] This approach takes rare events into account: i.e., by introducing multiple knowledge transfers, it significantly reduces the probability of rare events, since at each knowledge level a rare event may be a common event at another knowledge level, and therefore, rare events are limited to the very few cases where the event is rare in each combination of knowledge levels.

[0035] Furthermore, using this training method, each deep neural network (DNN) can be trained independently using the same DNN architecture by transferring features from one network to another.

[0036] Advantageously, for the determination step, the training method uses an end-to-end deep neural network (E2EDNN-ABCE), which is a transfer learning sequence of the first, second, third and fourth deep neural networks, followed by at least one end-to-end top layer (E2E TL).

[0037] In fact, the number of training samples / labels is very important for the first DNN, thus representing structured knowledge, and for each of the second, third, and fourth DNNs, the number of training samples / labels decreases significantly until unstructured knowledge is represented. Therefore, this training sequence of each individual DNN benefits from the previous more structured knowledge.

[0038] Advantageously, the risk classification training method further comprises the step of training a fifth deep neural network (DNN-D) for anticipation of invisible specific objects or regions and / or prediction of visible specific objects or regions within an image or image sequence associated with a specific application; and wherein, for the determination step, an end-to-end deep neural network (E2E DNN-5) is used which is a transfer learning sequence of five deep neural networks. Preferably, for the determination step, the training method uses an end-to-end deep neural network (E2E DNN-ABCDE) which is a transfer learning sequence of the first, second, third, fifth and fourth deep neural networks, followed by at least one end-to-end top layer (E2E TL).

[0039] Training on unseen objects / regions helps to: predict / anticipate that an object / region will change its state or anticipate the future appearance of an object / region at a specific location within a specific duration, especially when labels are missing at the object level but labels are used at the pixel or region level. More generally, prediction / anticipation involves at least the predicted / anticipated location, duration of appearance, direction of motion, type of shape deformation, selection of target objects to which the current object / region may move, and any other possible state changes and combinations thereof. In specific cases where specific objects are limited by the training dataset (specific to the application of interest), the objects used in the prediction and anticipation steps and their trajectories of change from the current state to the future state or various potential states can still be automatically discovered in an unsupervised manner.

[0040] The training of DNN-D further brings attention features of invisible objects / regions and / or visible specific objects or regions to predict / anticipate the location, state, etc. in an image or image sequence.

[0041] Training the fifth individual network between the third and fourth individual networks is perfectly suited for the sequence in order to benefit from the previous more structured knowledge.

[0042] Advantageously, the risk classification training method further comprises the following steps: training an end-to-end multi-task learning deep neural network (E2E MTL DNN-1), wherein the end-to-end multi-task learning deep neural network (E2E MTL DNN-1) comprises the end-to-end deep neural network (E2E DNN-5) representing a knowledge block using a transfer learning sequence of five deep neural networks and combined knowledge of at least another deep neural network representing at least another knowledge block among the following: (i) any one of the first, second, third, fourth or fifth deep neural networks (DNN-1), (ii) any one of the end-to-end deep neural networks (E2E DNN-2), wherein any one of the end-to-end deep neural networks (E2E DNN-2) is a transfer learning sequence of two of the first, second, third, fourth or fifth deep neural networks, (iii) any one of the end-to-end deep neural networks (E2E DNN-3), wherein any one of the end-to-end deep neural networks (E2E DNN-4) is a transfer learning sequence of two of the first, second, third, fourth or fifth deep neural networks. DNN-3) is a transfer learning sequence of three deep neural networks in the first, second, third, fourth or fifth deep neural network, and (iv) any end-to-end deep neural network (E2EDNN-4), wherein any end-to-end deep neural network (E2E DNN-4) is a transfer learning sequence of four deep neural networks in the first, second, third, fourth or fifth deep neural network, and wherein, for the determining step, the combined knowledge of the end-to-end multi-task learning deep neural network (E2E MTL DNN-1) is used, followed by at least one end-to-end top layer (E2E TL).

[0043] More preferably, the risk classification training method further comprises the following steps: training an end-to-end multi-task learning deep neural network (E2E MTL DNN-2), the end-to-end multi-task learning deep neural network (E2EMTL DNN-2) comprising the end-to-end deep neural network (E2E DNN-ABCDE) representing a knowledge block using a transfer learning sequence of the first, second, third, fifth and fourth deep neural networks and a combination of knowledge of at least another deep neural network representing at least another knowledge block among the following: (i) the first deep neural network (DNN-A), (ii) an end-to-end deep neural network (E2EDNN-AB), the end-to-end deep neural network (E2E DNN-AB) being a transfer learning sequence of the first and second deep neural networks, (iii) an end-to-end deep neural network (E2E DNN-ABC), the end-to-end deep neural network (E2E DNN-ABC) being a transfer learning sequence of the first, second and third deep neural networks, and (iv) an end-to-end deep neural network (E2E DNN-ABCD), the end-to-end deep neural network (E2E DNN-ABCD) is a transfer learning sequence of the first, second, third and fifth deep neural networks, and wherein, for the determining step, the combined knowledge of the end-to-end multi-task learning deep neural network (E2E MTL DNN-2) is used, followed by at least one end-to-end top layer (E2ETL).

[0044] The advantages of this training method are: each knowledge block is trained independently using the same DNN architecture by transferring features from one knowledge block to another, and all knowledge blocks are combined into a single end-to-end multi-task DNN with shared access to high-level features, thereby learning pixel-level hazard classification based on the combination of multiple knowledge blocks.

[0045] Advantageously, all pixels in both generic and specific objects are marked, all objects within a specific scene flow are marked, and all potentially hazardous areas are marked.

[0046] According to another object, a second aspect of the present invention relates to a method for hazard classification within an image or image sequence related to a specific application, comprising the following steps: providing an image or image sequence; drawing a hazard classification map using a hazard classification training method trained according to the first aspect, wherein the hazard classification map displays any hazard level within the image or image sequence.

[0047] This approach combines sparse / dense labels from multiple human individuals; thus, the final trained end-to-end deep neural network recombines all these individual human experiences into a single multi-expert experience, making it more powerful than the experience of a single human. For example, if some hazard labels come from a person who always drives in snowy conditions, and other labels come from people who drive in dry conditions, the trained combined end-to-end network will perform at least as well as both drivers in both cases, while humans will only perform well when they have significant experience. In this way, the end-to-end network recombines and integrates the experience of all drivers in all driving situations. Therefore, this approach provides a combined end-to-end deep neural network that takes a single image or a time series of images as input and generates a hazard graded image map as output, taking into account human hazard perception in all situations and enabling estimation of hazard levels in unrestricted scenarios. Therefore, this provides a high level of autonomy for the trained method.

[0048] Advantageously, the risk classification map is a risk classified pixel level map representing the risk level of each pixel within said image or sequence of images.

[0049] Likewise, this approach provides an end-to-end deep neural network that takes a single image or a time series of images as input and generates a pixel-level hazard graded image map as output. Furthermore, the pixel-level hazard map is not limited by the quality of the sensor that segments the image or image sequence. Additionally, the hazard graded pixel level map provides a hazard level (hazard grade) relative to the current situation for each image pixel (which is uniform for similar scenarios). Having more labels and having denser labels can improve training accuracy but also increase data processing and labeling time as well as training time. The proposed approach has proven to be very effective in estimating hazard levels by using only sparse labels, as the combined end-to-end deep neural network inherits the ability to be trained using only sparse labels and generates dense hazard graded maps in real-time driving scenarios.

[0050] Advantageously, the specific application is an assisted or automated vehicle driving system and the hazard classification method further comprises the step of capturing an image or a sequence of images of the vehicle's surroundings using an image sensor unit mounted on the vehicle.

[0051] For this application, a hazard-graded pixel level map provides, for each image pixel, a hazard level for the current driving situation (uniform for similar driving scenarios). Such a map is a key element of highly assisted / autonomous vehicle control systems that require safe and efficient path planning for trajectory control, especially in rare / unique driving scenarios.

[0052] Alternatively, the specific application is a control system for a sensitive area, and the hazard classification method further comprises the step of capturing an image or a sequence of images using at least one video surveillance device installed in or around the sensitive area.

[0053] These are, for example, intensive care areas in hospitals, high-security areas in prisons or high-security areas in nuclear power plants.

[0054] According to another object, the present invention also relates to an assisted or automatic vehicle driving system, comprising: at least one image sensor unit for capturing images or image sequences of the vehicle's surroundings; a processing unit for calculating a hazard classification map according to the method of the second aspect; and a control unit for controlling the vehicle driving system according to the hazard classification map.

[0055] Advantageously, the processing unit is also used to evaluate the danger level of the danger classification map by comparing it with a predetermined danger threshold; and wherein the vehicle driving system also includes at least one display unit for displaying a danger warning; and a driving unit for planning a safe driving path to reduce the danger level; and wherein the control unit is used to control the display unit and / or the driving unit when the danger level is higher than the predetermined danger threshold.

[0056] Such application to assisted or automated vehicle driving systems increases the safety of the driving system by taking into account any potential hazards in the environment in the form of a hazard ranking map that shows potential hazard areas that a human might perceive, including, for example, upcoming invisible objects.

[0057] According to another object, the present invention also relates to a vehicle network, comprising a plurality of vehicles, wherein the plurality of vehicles are equipped with an assisted or automatic vehicle driving system according to the third aspect, wherein the driving system of each vehicle further comprises a navigation unit implementing an adaptive navigation high-definition map, wherein the adaptive navigation high-definition map comprises a hazard layer based on a hazard classification map of at least two vehicles of the vehicle network.

[0058] This vehicle network increases the safety of each vehicle’s driving system thanks to the hazard-graded maps of other vehicles in the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Other features and advantages of the present invention will appear more clearly from the following detailed description of a particular non-limiting example of the invention as illustrated in the accompanying drawings, in which:

[0060] Figure 1 Shows the risk classification training method according to two preferred embodiments of the present invention;

[0061] Figure 2A general flow chart representing transfer knowledge learning;

[0062] Figure 3 It shows a risk classification method according to a preferred embodiment of the present invention;

[0063] Figure 4 A general flow chart representing knowledge combination;

[0064] Figure 5 Designates a vehicle equipped with the necessary units for implementing the method according to the invention. DETAILED DESCRIPTION

[0065] Before describing the different embodiments of the invention in more detail, here are some general statements that will help understand the rest of the description which primarily describes the application of self-driving cars. However, it will be understood that the invention can be perfectly transferred to other specific applications of control systems with sensitive areas, such as intensive care areas within hospitals, high security areas in prisons, or high security areas in nuclear power plants, etc.

[0066] One aspect of the present invention is to introduce a hazard graded pixel level map that provides, for each image pixel, a hazard level (i.e., grade) about the current driving situation (which is uniform for similar driving scenarios). Such a map is a key element of highly assisted / autonomous vehicle control systems that need to achieve safe and efficient path planning for trajectory control, especially in rare / unique driving scenarios.

[0067] Figure 1 Two preferred embodiments of the risk classification training method according to the present invention are shown.

[0068] Before implementing this hazard classification method into a vehicle driving system, training this algorithm is key. For simplicity, Figure 1 The training methods represented in this paper involve applications in assisted or autonomous vehicles.

[0069] According to a first variant, the risk stratification training method comprises the following steps.

[0070] S1: Training a first deep neural network (DNN-A), the first deep neural network (DNN-A) is used for general object recognition in a general image.

[0071] S2: Training a second deep neural network (DNN-B), which is used for specific object recognition within the image related to the specific application.

[0072] S3a: Training a third deep neural network (DNN-C) for scene-specific flow prediction within an image sequence related to a specific application.

[0073] S4: Training a fourth deep neural network (DNN-E) using at least one labeling method for capturing how humans react and / or analyze the environment in a driving scenario or in an offline reproduction or simulation of a driving scenario, wherein the fourth deep neural network (DNN-E) is used to locate potential hazardous areas within an image or image sequence related to a specific application.

[0074] S5: Using at least one human-trained danger labeling method, using an end-to-end deep neural network (E2EDNN-ABCE-) to determine at least one dangerous pixel within an image or image sequence related to a specific application, the end-to-end deep neural network (E2E DNN-4) is a transfer learning sequence of the first, second, third and fourth deep neural networks, and the end-to-end deep neural network (E2E DNN-ABCE-) is followed by one or more end-to-end top layers (E2E TL).

[0075] According to a second variant, the risk stratification training method further comprises a step S3b between steps S3a and S4.

[0076] S3b: Training a fifth deep neural network (DNN-D) for anticipation of invisible specific objects or regions and / or prediction of visible specific objects or regions within an image or image sequence related to a specific application.

[0077] Therefore, step S5 applies as follows.

[0078] S5: Using at least one human driving hazard labeling method, determine at least one dangerous pixel within an image or image sequence related to a specific application using the following end-to-end deep neural network (E2EDNN-ABCDE), wherein the end-to-end deep neural network is a transfer learning sequence of a first, second, third, fifth, and fourth deep neural networks, and the end-to-end deep neural network is followed by one or more end-to-end top layers (E2E TL).

[0079] More specifically, the training of each neural network is done by providing datasets and labels corresponding to those datasets.

[0080] Training the first deep neural network (DNN-A) for generic object recognition is accomplished by the following steps.

[0081] providing generic images containing generic objects (dataset-A) and generic object labels (label-A); and

[0082] At least one class-general feature space (Conv-A) and at least one class-general decision space (FC-A) are output for classifying general objects of the general image into general object labels.

[0083] The goal of this first network, DNN-A, is to learn what makes one object different from another. This training provides robust low-level and high-level visual features. For transfer learning, it is able to generate robust and unique responses for each object category.

[0084] Training the second deep neural network (DNN-B) for specific object recognition is completed by the following steps.

[0085] Provide a specific image containing at least one specific object (dataset-B) and a specific object label (Label-B);

[0086] At least one class-specific feature space (Conv-B) and at least one class-specific decision space (FC-B) are output for classifying a specific object of a specific image into a specific object label.

[0087] The goal of this second network, DNN-B, is to learn to segment images into pixel-level categories. This training provides knowledge of the categories and locations of specific objects in the driving scene. For transfer learning, it transfers the segmentation of images to repeatable regions of semantic categories and driving environment priors.

[0088] Training the third deep neural network (DNN-C) for scene-specific flow prediction is done through the following steps.

[0089] Providing a specific image sequence (dataset-B) containing at least one specific object and a specific motion or scene flow label (label-C);

[0090] At least one class-specific feature space (Conv-C) and at least one class-specific decision space (FC-C) are output for predicting a specific scene flow of a specific image sequence into a specific scene flow label.

[0091] The goal of this third network, DNN-C, is to learn how to estimate dense optical low-depth maps. This training provides knowledge of the direction, speed, and distance of objects. For transfer learning, it transfers understanding of observable driving scenes that is independent of sensor locations.

[0092] The training of the fifth deep neural network (DNN-D) for invisible specific object / action or area prediction or visible specific object / action or area prediction is completed by the following steps.

[0093] Providing a specific image or a specific image sequence (dataset-D) containing at least one specific object and a specific invisible object label (label-D);

[0094] Outputting at least one class-specific feature space (Conv-D) and at least one class-specific decision space (FC-D) for anticipating / predicting the occurrence of invisible or visible specific objects / actions or regions on a specific image or a specific image sequence.

[0095] The goal of this fifth network DNN-D is to learn how to predict the class of an object, its behavior and location that are mainly dependent on the environment. This training provides knowledge of the potential classes and locations of objects in the driving scene. For transfer learning, it transfers the understanding of predicting driving scenes that is independent of sensor location. Object / action predictions for the future state of a specific object required to focus on potentially dangerous image areas when the object is not seen in the current driving scene lead to classification and localization.

[0096] Training the fourth deep neural network (DNN-E) for specific eye tracking localization of potential danger areas is completed through the following steps.

[0097] Providing a specific image sequence containing at least one visible or invisible specific object (Dataset-E) and a specific hazard mark label (Label-E) marked by a human expert;

[0098] - Output at least one class-specific feature space (Conv-E) and at least one class-specific decision space (FC-E) for locating potential dangerous areas in a specific image sequence as taught by human experts.

[0099] The goal of this fourth network DNN-E is to replicate the driver's experience, focusing only on a few image regions. This training enables the distinction of important / noticeable objects or regions of each driving scene. For transfer learning, it transfers estimates of the perceptibility level of each object or region of the driving scene. In particular, eye tracking localization is needed to filter out image regions that are perceptible to humans. In particular, automotive eye tracking localization is important for further filtering image regions that are potentially important for driving.

[0100] The sequence of transfer learning through the first, second, third, fifth and fourth deep neural networks, followed by the training of the end-to-end deep neural network (E2EDNN-ABCDE) completed by one or more end-to-end top layers (E2E TL) is completed by the following steps.

[0101] Providing a specific image sequence containing at least one visible or invisible specific object (Dataset-E2E) and a specific hazard mark label (Label-E2E) marked by a human expert;

[0102] At least one dangerous pixel is determined.

[0103] The goal of this end-to-end network is to estimate the collision risk level with each object or area within the driving scene. This training provides knowledge of the hazard classification map. For transfer learning, it transfers the pixel-level hazard classification map based on a single image as input to other time series deep neural networks (LSTM, RNN, etc.). The specific knowledge of hazard level classification and localization is important for filtering image areas that are important for hazard estimation in driving-important areas.

[0104] Figure 2 A general flow chart showing transfer knowledge learning applicable to DNN AE. It can be easily understood from the figure that network A has the most structured knowledge and obtains more training samples and labels than network B, followed by network B obtaining more training samples and labels than network C, followed by network C obtaining more training samples and labels than network D, which in turn obtains more training samples and labels than network E, which has the most unstructured knowledge.

[0105] The training of each individual model or network A to E benefits from the more structured previous knowledge.

[0106] For automotive applications, the selected dataset for training purposes may be, for example, the following dataset.

[0107] Dataset-A: ImageNet;

[0108] Dataset-B: Cytiscapes;

[0109] Dataset-C: KITTI

[0110] Dataset-D: KITTI

[0111] Dataset-E: CAT2000 or DriveEye

[0112] E2E Dataset: Cytiscapes

[0113] After training, the goal is to embed this approach directly into the driving systems of assisted / autonomous vehicles for automotive applications.

[0114] Figure 3 A risk classification method according to a preferred embodiment of the present invention is shown.

[0115] The deep neural networks A to E and E2E differ from the training method only in that they are trained. Therefore, knowledge transfer is accomplished by a trained end-to-end DNN-ABCDE-TL that receives an input image of a driving situation and outputs a pixel-level hazard map where each pixel value corresponds to a hazard level.

[0116] We demonstrate that by training an end-to-end deep neural network on training data with only one labeled pixel per image, and performing a sequence of transfer learning steps, it is possible to learn to successfully combine object recognition knowledge (learned in the transfer step) with hazard knowledge (learned in the current step, on very sparse labeled data); and that using a sequence of transfer learning steps, it is possible to obtain a dense hazard classification map by training on very sparse (1 pixel per image) training data, without having to label all hazard-bearing pixels - a substantial speedup in labeling time.

[0117] We demonstrate that an end-to-end deep neural network is able to integrate multiple experiences (annotated by the same person, but with various images labeled over a duration of several weeks) into a single solution; while only one object is annotated per image, the network indicates high hazard levels for multiple objects in the scene; and that combining data from multiple experts should also be feasible.

[0118] Figure 4 A generalized flow chart showing the combination of knowledge according to a preferred embodiment of the present invention.

[0119] It can be easily understood from this figure that by using multi-task training of DNNs, one can train the hazard classification map not only by accessing the features extracted by the end-to-end convolutional part of the network, but also by accessing the features extracted by each transfer learning block in the deconvolutional part (i.e., A, AB, ABC, ABCD, and ABCDE). This will enable benchmarking the importance of each transfer block to hazard classification by removing the transfer blocks one by one and comparing the multi-task hazard classification accuracy. This should also enable visualization of activation maps that show the location and strength of the connections between the multi-task hazard classification and the individual transfer learning maps. The higher the connection strength, the more the hazard classification depends on the transfer learning map for each specific image pixel. This enables understanding: for each image region, where does the hazard classification get information from to make decisions, and which combination of transfer learning modules is strongest in which case. This information is very important for continuously improving hazard classification training methods and hazard classification methods.

[0120] Furthermore, in this way, the training process can be reduced to training an individual deep neural network using specific or general knowledge and training an end-to-end deep neural network to classify the dangerousness of pixels in an image or image sequence. Similarly, the training process can be extended to any reasonable number of knowledge transfer sequences, followed by the combined knowledge of an end-to-end multi-task learning deep neural network (E2E MTL DNN-1), followed by at least one end-to-end top layer (E2E TL).

[0121] Figure 5 A vehicle is shown equipped with at least one camera 200 directed toward the road ahead or the vehicle environment to capture a video or a continuous series of images (image sequence). The vehicle 100 further comprises a processing unit and electronic control unit 300 , a display unit, and an autonomous driving unit 400 , 410 .

[0122] The processing unit is configured to evaluate the danger level of the danger classification diagram compared with a predetermined danger threshold; and the control unit is configured to control the display unit and / or the driving unit when the danger level is higher than the predetermined danger threshold, so that the display unit can display a danger warning and / or the driving unit can plan a safe driving path to reduce the danger level.

[0123] It will be understood that various modifications and / or improvements obvious to those skilled in the art may be made to the different embodiments of the invention described in this specification without departing from the scope of the invention as defined by the accompanying claims.

Claims

1. A method for training hazard classification in images or image sequences related to a specific application, wherein the specific application is an assisted or automatic vehicle driving system or a control system of a sensitive area, the method comprising the following steps: Training a first deep neural network DNN-A, wherein the first deep neural network DNN-A is used for general object recognition in a general image; Training a second deep neural network DNN-B, the second deep neural network DNN-B is used for specific object recognition in the image related to the specific application; Training a third deep neural network DNN-C, the third deep neural network DNN-C being used for specific scene flow prediction within an image sequence associated with the specific application; training a fourth deep neural network DNN-E using at least one labeling method for capturing how humans react and / or analyze the environment in the image or image sequence related to the specific application, the fourth deep neural network being used for locating potential dangerous areas within the image or image sequence; Using at least one human-trained danger labeling method, at least one danger pixel within an image or image sequence related to the specific application is determined using an end-to-end deep neural network E2E DNN-4, wherein the end-to-end deep neural network E2E DNN-4 is a transfer learning sequence of the first deep neural network, the second deep neural network, the third deep neural network, and the fourth deep neural network, followed by at least one top-level E2E TL.

2. The risk classification training method according to claim 1 further comprises the following steps: training a fifth deep neural network DNN-D for anticipation of invisible specific objects / actions or regions and / or prediction of visible specific objects or regions within an image or image sequence related to the specific application; and Wherein, for the determining step, an end-to-end deep neural network E2E DNN-5 is used, and the end-to-end deep neural network E2E DNN-5 is a transfer learning sequence of the first deep neural network, the second deep neural network, the third deep neural network, the fifth deep neural network and the fourth deep neural network, followed by the at least one top-level E2E TL.

3. The risk classification training method according to claim 2 further comprises the following steps: Training an end-to-end multi-task learning deep neural network E2E MTL DNN-1, the end-to-end multi-task learning deep neural network E2E MTL DNN-1 comprising a combination of the end-to-end deep neural network E2E DNN-5 using a transfer learning sequence of the first deep neural network, the second deep neural network, the third deep neural network, the fifth deep neural network, and the fourth deep neural network and at least another deep neural network of: Any one of the first deep neural network, the second deep neural network, the third deep neural network, the fourth deep neural network or the fifth deep neural network DNN-1, Any end-to-end deep neural network E2E DNN-2 as a transfer learning sequence of two deep neural networks among the first deep neural network, the second deep neural network, the third deep neural network, the fourth deep neural network or the fifth deep neural network, Any end-to-end deep neural network E2E DNN-3 that is a transfer learning sequence of three deep neural networks among the first deep neural network, the second deep neural network, the third deep neural network, the fourth deep neural network, or the fifth deep neural network, and Any end-to-end deep neural network E2E DNN-4 that is a transfer learning sequence of four deep neural networks among the first deep neural network, the second deep neural network, the third deep neural network, the fourth deep neural network, or the fifth deep neural network, and Wherein, for the determining step, a combination of the end-to-end multi-task learning deep neural network E2E MTL DNN-1 is used, followed by the at least one top layer E2E TL.

4. The risk classification training method according to claim 2 further comprises the following steps: Training an end-to-end multi-task learning deep neural network E2E MTL DNN-2, the end-to-end multi-task learning deep neural network E2E MTL DNN-2 comprising a combination of the end-to-end deep neural network E2E DNN-5 using a transfer learning sequence of the first deep neural network, the second deep neural network, the third deep neural network, the fifth deep neural network, and the fourth deep neural network and at least another deep neural network among the following: The first deep neural network DNN-A, An end-to-end deep neural network E2E DNN-AB as a transfer learning sequence of the first deep neural network and the second deep neural network, an end-to-end deep neural network E2E DNN-ABC as a transfer learning sequence of the first deep neural network, the second deep neural network and the third deep neural network, and an end-to-end deep neural network E2E DNN-ABCD as a transfer learning sequence of the first deep neural network, the second deep neural network, the third deep neural network and the fifth deep neural network, and Wherein, for the determining step, a combination of the end-to-end multi-task learning deep neural network E2E MTL DNN-2 is used, followed by the at least one top layer E2E TL.

5. The risk classification training method according to any one of claims 1 to 4, wherein: labeling pixels in the general object and the specific object, marking objects within said particular scene stream, Mark potential danger areas.

6. A method for stratifying danger in an image or image sequence related to a specific application, the specific application being an assisted or automatic vehicle driving system or a control system for a sensitive area, the method comprising the following steps: providing an image or a sequence of images; A hazard grading map is drawn using the trained hazard grading training method according to any one of claims 1 to 5, wherein the hazard grading map displays any hazard level within the image or image sequence.

7. The risk classification method according to claim 6, wherein: The hazard grading map is a hazard graded pixel level map representing the hazard level of each pixel in the image or image sequence.

8. The risk classification method according to claim 6 or 7, wherein: The specific application is an assisted or automated vehicle driving system, and wherein the risk classification method further comprises the following steps: An image or a sequence of images of the vehicle's surroundings is captured using an image sensor unit mounted on the vehicle.

9. The risk classification method according to claim 6 or 7, wherein: The specific application is a control system for sensitive areas, and wherein the hazard classification method further comprises the following steps: An image or sequence of images is captured using at least one video surveillance device installed in or around the sensitive area.

10. An assisted or automated vehicle driving system comprising: at least one image sensor unit for capturing an image or a sequence of images of the vehicle's surroundings; A processing unit, used for calculating a hazard grading diagram according to the hazard grading method according to claim 6 or 7; as well as A control unit is used to control the vehicle driving system based on the hazard classification map.

11. The assisted or automated vehicle driving system of claim 10, wherein: The processing unit is further configured to compare with a predetermined danger threshold to evaluate the danger level of the danger classification diagram; and wherein the vehicle driving system further comprises at least one of the following: A display unit for displaying a hazard warning; and Driving unit, used to plan safe driving paths to reduce the risk level; And wherein, the control unit is used to control the display unit and / or the driving unit when the danger level is higher than the predetermined danger threshold.

12. A vehicle network comprising a plurality of vehicles, the plurality of vehicles being equipped with the assisted or automatic vehicle driving system according to claim 10 or 11, wherein: The driving system of each vehicle further includes a navigation unit implemented with an adaptive navigation high definition map, wherein the adaptive navigation high definition map includes hazard layers based on a hazard ranking map of at least two vehicles of the vehicle network.

Images