SYSTEM AND METHOD FOR ESTIMATE AN EGO LANE

Abstract

This paper presents a system (103) and a method (500) for estimating the ego lane of a vehicle (102). The method (500) comprises receiving a plurality of road signatures of the vehicle (102) at each timestamp of a plurality of timestamps. Furthermore, the method comprises applying an encoding technique to the plurality of road signatures to generate a plurality of coded road signatures corresponding to the current timestamp of the plurality of timestamps and one or more previous timestamps. In addition, the method comprises grouping a plurality of coded road signatures and adding one or more dummy lane encoding data to generate context-aware sequence data. Finally, the method comprises performing the aforementioned steps for the remaining timestamps of the plurality of timestamps to generate the remaining context-aware sequence data.Finally, the procedure includes estimating a vehicle's ego lane based on contextual sequence data and remaining contextual sequence data.

Description

PREAMBLE TO THE DESCRIPTION:

[0001] The following description explains the invention and how it is to be implemented in more detail. TECHNICAL AREA

[0002] The present subject matter relates generally to an advanced driver assistance system (ADAS) for a vehicle and in particular, but not exclusively, to a method and system for estimating a vehicle's ego lane. BACKGROUND OF THE REVELATION

[0003] Generally, a vehicle's ego lane is estimated using vehicle observations and map information. Ego lane localization techniques can be used to estimate the ego vehicle's position on the road. When estimating or detecting the ego lane, the vehicle localization method can also detect the vehicle's current lane and its surrounding boundaries, enabling autonomous vehicles to safely change lanes and turn into adjacent streets. Ego lane estimation is essential for automated vehicles to navigate roads such as highways, city streets, and so on.

[0004] Conventional methods for estimating ego lanes rely exclusively on state-space filters, Bayesian filters, or heuristic systems. State-space filters and Bayesian filters often depend on hand-crafted features (e.g., image features or properties derived from information within the image itself using various techniques) and rule-based systems (e.g., state-space filters can identify elements based on categories and parameter values). Using such filters for ego lane estimation becomes challenging when there are different driving scenarios, varying infrastructure, and sensor inaccuracies in the autonomous vehicle.

[0005] In particular, heuristic systems such as state-space filters cannot adapt to changing road conditions (including changes in lane markings), changing weather conditions, and / or changing road types. Therefore, such problems degrade the performance of autonomous vehicles. In other words, under dynamic road conditions, the performance of state-space filters or heuristic systems is inefficient.

[0006] Furthermore, heuristic-based systems require constant adjustment of hyperparameters when infrastructure variations occur (e.g., accuracy fluctuations due to differing road infrastructure). These hyperparameters must be modified based on information about the ego-vehicle's geography. Modifying or adjusting hyperparameters reduces the system's efficiency. Finally, if the sensors are inaccurate—that is, if the sensors connected to the ego-vehicle fail to correctly receive or collect data at certain times—the sensor may provide inaccurate data to the system for lane estimation. Such inaccuracies in the received data lead to an inaccurate estimate of the ego-vehicle's lane. For example, suppose the ego-vehicle has an associated sensor (e.g., a camera). This sensor may be blocked or obscured by another vehicle (e.g., a truck).Accordingly, the ego vehicle cannot change lanes because it has no visibility from the edge of the road. Consequently, it will be difficult to determine the vehicle's location or position.

[0007] Estimating the ego lane for the ego vehicle is known. For example, German patent DE102022004774A1 discloses methods for determining the lane of an ego vehicle based on lane attributes. This German patent makes the lane estimation method extremely error-resistant and suppresses overestimated boundary distances. However, the German patent uses the state-space filter to determine the ego vehicle's lane, which is inefficient for the reasons mentioned above. US patent US20120150437A1 discloses methods for accurately estimating the position of vehicles on sub-lanes. In this US application, the vehicle position is determined with sub-lane accuracy by combining inputs from a GPS, a camera, radar, a lane-level digital map, and a vehicle dynamics sensor. Accordingly, the US application estimates the vehicle's lane by tuning the hyperparameters.However, the aforementioned state of the art uses rule-based systems or state-space filters and does not provide an effective solution for estimating the lane for the ego vehicle in order to locate the ego vehicle or to determine the position of the ego vehicle.

[0008] In light of the above, it is necessary to estimate the vehicle's ego track in order to efficiently locate the vehicle's position in relation to the lanes on the road. SUMMARY OF DISCLOSURE

[0009] This document describes a method for estimating the ego lane of a vehicle. The method comprises receiving a plurality of road signatures of the vehicle at each timestamp of a plurality of timestamps from an image acquisition device associated with the vehicle, wherein each road signature includes a lane marking type and corresponding distance information. The method further comprises applying an encoding technique to the plurality of lane signatures to generate a plurality of coded lane signatures corresponding to a current timestamp of the plurality of timestamps and one or more previous timestamps, wherein each coded lane signature includes vector data corresponding to each lane marking type.Furthermore, the procedure includes grouping the multiple coded road signatures corresponding to the one or more previous timestamps and the current timestamps, and adding one or more dummy lane coding data to generate context-aware sequence data for the timestamp. The procedure then includes performing the aforementioned steps of applying and grouping to the remaining timestamps of the multitude of timestamps to generate the remaining context-aware sequence data. Finally, the procedure includes estimating the vehicle's ego lane based on the context-aware sequence data and the remaining context-aware sequence data.

[0010] This document describes a device for estimating a vehicle's ego lane. The device comprises a processor and a memory. The memory is communicatively coupled to the processor, and the processor is configured to receive a plurality of lane signatures of the vehicle at each timestamp of a plurality of timestamps from an image acquisition device attached to the vehicle, wherein each lane signature includes a lane marker type and corresponding distance information. Furthermore, the processor is configured to apply an encoding procedure to the multiple lane signatures to generate multiple encoded lane signatures corresponding to a current timestamp of the plurality of timestamps and one or more previous timestamps, wherein each encoded lane signature includes vector data corresponding to each lane type.Furthermore, the processor is configured to group the multiple coded lane signatures corresponding to the one or more previous timestamps and the current timestamp, and to add one or more dummy lane coding data to generate context-aware sequence data for the timestamp. The processor is then configured to perform the aforementioned steps for the remaining timestamps of the plurality of timestamps to generate the remaining context-aware sequence data. Finally, the processor is configured to estimate the vehicle's ego lane based on the context-aware sequence data and the remaining context-aware sequence data.

[0011] The foregoing summary serves only for illustration and is in no way intended to be limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become clear by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which form part of this disclosure, illustrate exemplary embodiments and, together with the description, explain the disclosed principles. In the figures, the leftmost digit(s) of a reference number indicates the figure in which the reference number first appears. The same numbers are used throughout the figures to identify identical features and components. Some embodiments of systems and / or methods according to the embodiments of the present subject matter are now described only by way of example and with reference to the accompanying figures, in which: Fig. Figure 1a shows a conceptual diagram 100a with examples of street signatures 101 for a time frame / stamp according to some embodiments of the present disclosure. Fig. Figure 1b shows a block diagram 100b of a system 103 for estimating the ego lane, comprising a device 107 for estimating the ego lane of a vehicle 102, according to some embodiments of the present disclosure. Fig. Figure 2a shows a flowchart 200a for encoding a lane marking type according to some embodiments of the present disclosure. Fig. Figure 2b shows a flowchart 200b for generating context-related sequence data in accordance with some embodiments of the present disclosure. Fig. Figure 2c shows a flowchart 200c for generating coded lane-road signature sequence data in accordance with some embodiments of the present disclosure. Fig. Figure 2d shows a nested OnexD hot coding technique 200d according to some embodiments of the present disclosure. The Fig. Figures 3a-3e show examples of the encoding of different lane marking types in some embodiments of the present disclosure. Fig. Figure 4 shows a flowchart illustrating a method 400 for estimating the ego lane of vehicle 102 in accordance with some embodiments of the present disclosure.

[0013] The person skilled in the art should know that all block diagrams contained herein represent conceptual views of systems embodying the principles of the subject matter at hand. Likewise, it will be understood that all flowcharts, process diagrams, state transition diagrams, pseudocodes, and the like represent various processes, essentially represented in a computer-readable medium and executable by a computer or processor, regardless of whether such a computer or processor is explicitly depicted. DETAILED DESCRIPTION

[0014] In this document, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Each embodiment or implementation of the subject matter described herein as "exemplary" is not necessarily to be understood as preferred or advantageous over other embodiments.

[0015] While the disclosure is open to various modifications and alternative forms, one specific embodiment has been shown by way of example in the drawings and is described in detail below. It should be understood, however, that the disclosure is not intended to be limited to the specific forms disclosed, but rather, on the contrary, to cover all modifications, equivalents, and alternatives that fall within the scope of the disclosure.

[0016] The expressions “includes,” “comprises,” “includes,” or other variations thereof are intended to cover non-exclusive inclusion, so that a setup, device, or process that includes a list of components or steps may include not only those components or steps but may also include other components or steps not expressly listed or belonging to such setup, device, or process. In other words, one or more elements in a system or device introduced by “includes ... a” do not, without further limitations, exclude the existence of other or additional elements in the system or process.

[0017] The following detailed description of embodiments of the disclosure refers to the accompanying drawings, which form part of this document and illustrate specific embodiments in which the disclosure can be put into practice. These embodiments are described in sufficient detail to enable the person skilled in the art to implement the disclosure in practice, and it is understood that other embodiments may be used and modifications may be made without departing from the scope of this disclosure. The following description is therefore not to be understood in a restrictive sense.

[0018] Fig. Figure 1a shows a conceptual diagram 100a, which represents an example of street signatures 101 for a time frame / stamp in accordance with some embodiments of the present disclosure. The in Fig. Example 1a shows, for instance, street signatures 101 for a time frame / stamp with a left index = 2 and a right index = 1 (a total of 4 lanes).

[0019] Fig. Figure 1a shows a vehicle 102 (alternatively referred to as an "ego vehicle") from a bird's-eye view, driving on the road, specifically in a lane. The road may include a variety of lane markings. Each road signature includes a lane type 101 and corresponding distance information. As in Fig. As shown in Figure 1a, a position of the ego-vehicle 102 on the lane is designated (C). When the ego-vehicle 102 is traveling on the lane, there can be a number of lanes to the left of the ego-vehicle 102 and to the right of the ego-vehicle 102. However, the present disclosure is not limited to this, and there can be various permutations and combinations of lane-marking type for the vehicle. For example, there can be no lanes to the left of the ego-vehicle 102 and a number of "n" lanes to the right of the ego-vehicle 102. Likewise, there can be no lanes to the right of the ego-vehicle 102 and "n" lanes to the left of the ego-vehicle 102. In another example, there might be a lane type on which Ego-Vehicle 102 is driving, and a lane type to the right of Ego-Vehicle 102. In another example, there might be no lanes on either side of Ego-Vehicle 102. Fig. 1a The Ego vehicle 102 can drive on the lane, e.g. on lane 3. rd from the left side of a road edge or on lane 2 nd from the right side of the roadway. The present disclosure is not limited to this. The Ego-Vehicle 102 can drive in any lane of the road and then estimate the type of lane marking and the corresponding distance.

[0020] The Ego Vehicle 102 can be a vehicle equipped with a range of sensors that attempt to perceive its surroundings. In other words, the Ego Vehicle 102 can be a vehicle controlled by an autonomous driving system. As mentioned earlier, the road can include a variety of road signatures and a relative distance (e.g., the distance traveled by the Ego Vehicle between time windows) of lane marking types from the Ego Vehicle 102. As in Fig. As shown in Figure 1a, the multitude of road markings can, for example, without restriction, include a lane marking type Road Boundary (RB), a lane marking type Dashed White (DW), a lane marking type Dashed Yellow (DY), a lane marking type Solid White (SW), a lane marking type Solid Yellow (SY), and the position of the ego vehicle (C). The multitude of road markings can be independent of one another. In some examples, there may be a larger number of lane marking types on the right or left side of the ego vehicle 102, which are shown in Fig. 1a are not mentioned. The position of the ego vehicle can use the default distance of its position. In one example, the default distance can be a zero value (0). The present disclosure is not limited to this.

[0021] The Ego vehicle 102 may include a sensor (e.g., a camera) that captures images of the lane marking types and generates the multitude of road signatures. Finally, the lane types and the distance of each lane type are estimated for multiple timeframes, e.g., every 30–50 milliseconds. This disclosure is not limited to this.

[0022] As in Fig. As shown in Figure 1a, the road signature for the first time window includes, for example, lane marking type RB and its corresponding distance is estimated at 9.19, lane marking type SW and its corresponding distance is estimated at 8.67, lane marking type DW and its corresponding distance is estimated at 4.74, lane marking type DW and its corresponding distance is estimated at 1.31, "C" is the position of ego vehicle 102, lane marking type DW (right side of ego vehicle 102) and its corresponding distance are estimated at -2.39, lane marking type DW and its corresponding distance are estimated at -6.35, lane marking type SW and its corresponding distance are estimated at -9.55, and finally lane marking type RB and its corresponding distance are estimated at -9.55. The estimation of the lane marking type and its corresponding distance is described in the section on Fig. 1a explained. In some examples, the estimation can be performed for the first timeframe. In one example, the distance between the lane marking types to the left of the ego vehicle can be 102 positive values, and the distance between the lane marking types to the right of the ego vehicle can be 102 negative values. The final output data or output string for the first timeframe can be (RB, 9.19) (SW, 8.67) (DW, 4.74) (DW, 1.31) C (DW, -2.39) (SW, -6.35), (RB, -9.55). The person skilled in the art will understand that the procedure for estimating the lane marking type and the corresponding distance is repeated for all timeframes that are eligible for estimating the ego lane. The output data, estimated frame by frame, can be stored in a buffer. The stored output data (which can also be called historical data) can be used to account for inaccuracies, obscurations, and blockages on the road.

[0023] Fig. Figure 1b shows a block diagram 100b of a system for estimating the ego lane 103, which includes a device 107 for estimating the ego lane of an ego vehicle 102 according to some embodiments of the present disclosure.

[0024] As in Fig. As shown in Figure 1b, the ego lane estimation system 103 can comprise an image acquisition device 105, a device 107, and an ego lane estimation network (ELE) 115. For example, the ego lane estimation system 103 can comprise any system configured to estimate the ego lane of the ego vehicle 102 using a plurality of road signatures 101 received by the image acquisition device 105 and the ELE network 115. The device 107 can comprise an interface 109, a memory 111, and a processor 113. In one example, the device 107 can communicate with the image acquisition device 105 via a wireless communication network and receive input data (e.g., the plurality of road signatures) to estimate the ego lane of the ego vehicle 102.In another example, the device 107 can communicate with the image acquisition device 105 via the interface 109 using wired communication and receive input data to estimate the ego lane of the ego vehicle 102.

[0025] In some examples, the image capture device 105 may be mounted on the front of the ego vehicle 102. However, the positioning of the device 105 is not limited to this, and it can be mounted at various locations on the vehicle 102 to capture images. For example, the device 105 may be a sensor or a front-facing camera that captures images of the front view of the ego lane vehicle 102. The image capture device 105 may be connected to a road signature feature extraction network 105a to generate the road signature and the associated spacing of that specific road signature. Feature extraction is a process of converting raw information from the captured image into numerical features that can be processed while preserving the information in the data set.In one example, the road feature extraction network 105a can be a convolutional neural network that can be used to extract the road signature and its associated distance from the captured image. However, the present disclosure is not limited to this and can use any feature extraction network to extract the road signature and its associated distance for a given road signature. Based on the above, the image capture device 105 can extract the road signature and distance for that given road signature in different timeframes.

[0026] First, the device 107 can receive a plurality of street signatures 101 of the ego vehicle 102 at each timestamp of a plurality of timestamps from an image acquisition device 105 connected to the ego vehicle 102. As explained above, the image acquisition device 105 includes the street signature feature extraction network 105a, which can generate the street signature and the associated distance of that particular street signature. This can be performed for each timestamp or time frame (e.g., 30–50 milliseconds) to generate multiple street signatures and their associated distances.

[0027] After receiving the multiple road signatures 101 from the ego vehicle 102, the device 107 can apply an encoding procedure to the multiple road signatures to generate multiple coded road signatures corresponding to a current timestamp of the multiple timestamps and one or more previous timestamps. Each coded road signature comprises vector data corresponding to the individual lane marking types. The encoding technique can be a nested OnexD hot coding technique. The nested OnexD hot coding technique can convert the input data (lane markings and distance information) into corresponding numerical values.

[0028] In particular, the device 107 can create a one-hot coding format for at least one lane marking type using a one-hot coding technique. The one-hot coding technique can be a machine learning technique that converts categorical data into numerical data. The one-hot coding technique can generate binary columns for each category, where the value "1" indicates the presence of that category and "0" elsewhere. The one-hot coding format can include a vector that specifies the value "1" for the selected at least one lane marking type and the value "0" for the other lane marking types. Fig. 3a-3c, for example, represents the vector of the one-hot coding format with the values ​​"1" and "0".

[0029] After the creation of the one-hot coding format, the device 107 can replace the vector value "1" in the one-hot coding format with the associated distance information of the selected at least one lane marking type. For example, as in Fig. As shown in Figure 1a, if the lane marking type is RB, then the corresponding distance 9.19 is selected and the value "1" can be replaced by the actual distance 9.19. In the next step, the device 107 can convert the value "0" of the other lane marking types into a positive or negative predefined value. As explained in the paragraphs above, the value "0", for example, is a standard distance for the Ego vehicle 102. The values ​​"0" can be converted into positive or negative infinity values ​​(∞), as shown in Fig. Figure 3a shows that adding infinity values ​​in one-hot coding format to other lane marker types can indicate that the lane marker type with infinity values ​​does not exist. Positive infinity could represent the values ​​to the left of Ego-Vehicle 102, and negative infinity could represent the values ​​to the right of Ego-Vehicle 102. Adding infinity values ​​to other lane marker types in hot-coding format can be a theoretical practice. These infinity values ​​cannot be provided to the neural ELE network because the ELE network cannot process mathematical functions (such as infinity values, etc.) since the ELE network typically processes vector or tensor data (primary data structure). To provide the neural network with the one-hot coding format, the other lane marker types can be replaced with predefined values ​​for practical purposes.For practical purposes, the lane width can be approximately 60 m on both sides of the road. The specified positive or negative value can be assigned to other lane marking types with the value infinity. Accordingly, the selected lane marking type (LR) can be highlighted in the vector of the one-hot coding format. In an example, the specified values ​​can be +1000 or -1000. The value 1000 can indicate the distance in meters from the Ego vehicle 102. However, the present disclosure is not limited to this and can use any range of the safest distance for practical purposes.

[0030] After replacing predetermined positive and negative values, the device can add 107 random variables to the one-hot coding format of the at least one lane marking type to stabilize the neural network's training process and prevent the optimization of model parameters from getting stuck at local minima. When the one-hot coding format is provided to the neural network, it can get stuck, or the optimization of the model parameters can get stuck. To avoid this, the device can add 107 random variables or stabilizing values ​​using a stabilization module. In other words, the stabilization module can decide which random variables can be added to the one-hot coding format. Accordingly, the values ​​+1000 or -1000 can be replaced by other values, such as 1001.24, 999.46, 9998.79, 1000.33, 1000.89, as shown in Fig. Figure 3a shows that the stabilizing values ​​for the other lane marking types can be replaced using a probability distribution or a Gaussian distribution, the mean of which can be "0" and the standard deviation of which can be determined by a ratio between the distance of the lane marking type (D) of the Ego vehicle 102 and the total road width (RW). The formula for calculating the standard deviation can be as follows: Standard deviation = D / RW

[0031] The random variables or stabilization values ​​can be calculated using a formula: X∼G(0,C1∗tan h(C2∗(D / RW)))

[0032] X can be a random variable derived from the Gaussian distribution (G) with a mean of 0. C1 and C2 can be constant, and tan h can be a hyperbolic tangent function to match the hyperparameters.

[0033] After adding random variables or stabilizing values ​​to the one-hot coding format, the device 107 can generate at least one coded road signature in the form of a vector for at least one lane marking type. In the next step, the device 107 can repeat the process of generating the one-hot coding format for all other lane marking types. Similarly, the value 1 can be replaced by the respective spacing of the other lane marking types, and the value 0 is converted into the positive and negative predetermined values, and then the other coded road signatures are generated. In other words, each lane from the multitude of lane markings is coded using the coded procedure. For example, a vector of size 6 can be generated for the lane marking type RB. Similarly, a vector of size 6 can be generated for each of the other lane marking types.After generating each coded lane signature for each lane type, the device 107 can generate a plurality of coded lane signatures by grouping the generated coded lane signatures based on their generated order.

[0034] After applying the coding technique to generate the multiple coded lane signatures corresponding to the multiple lane signatures, the device 107 can group each of the individual coded lane signatures corresponding to the one or more previous timestamps and the current timestamps, and the device 107 can add one or more dummy lane coding data (as in conjunction with Fig. (described in 2d) to generate context-aware sequence data. To generate the context-aware sequence data, the device 107 can collect historically coded road signatures for a predefined distance in front of and behind the vehicle. In one example, the predefined distance can be 15 meters (m) in front of the ego-vehicle 102 and 15 m behind the ego-vehicle 102. The present disclosure is not limited to this. Subsequently, the device 107 can collect the real-time coded road signature and combine the historically coded road signatures with the real-time coded road signatures. In the next step, the device 107 can establish a threshold of frames for the combined multitude of coded road signatures and compare the combined coded road signatures with the threshold frames.If the frames of the combined coded road signatures are at most or equal to the threshold frames, the device 107 can group the combined coded road signatures up to the set threshold frames. Alternatively, if the frames of the combined coded road signatures are at least equal to or less than the threshold frames, the device 107 can group all frames associated with the combined coded road signatures. The device 107 can then determine continuity in frame selection based on the threshold frames and the combined coded road signatures, and add dummy frames to maintain continuity if there is a discontinuity. The process of adding the dummy coding data to the grouped coded signatures is described in [reference to relevant document]. Fig. 2d explained. Device 107 can generate context-sensitive sequence data based on the continuity of selected frames and the added dummy frames. The generated context-sensitive sequence data can be for a single timestamp. Device 107 can repeat the techniques described above to generate the remaining context-sensitive sequence data for different timestamps.

[0035] Subsequently, the device 107 can estimate the ego lane of the ego vehicle 102 based on the context-sensitive sequence data and the remaining context-sensitive sequence data. In particular, the device 107 can provide context-dependent sequence data as input for the ELE network. In one example, the ELE network 115 can be a long-term memory (LSTM) neural network. The LSTM can contain a recurrent neural network (RNN), which can be used in deep learning to process and predict data sequences. The LSTM can enable a neural network to learn long-term dependencies between different timestamps and sequence data.

[0036] The neural network can receive context-sensitive sequence data as input and prevent overfitting of the context-sensitive sequence data on dropout layers of the ELE network. The linear layer associated with the ELE network can map the selected time frame onto a low-dimensional space. The ELE network can determine nonlinearity in the time frame and predict a left and right index of the ego lane of the ego vehicle 102 based on the time frame mapping. The activation function of the parametrically equalized linear unit (PreLU) layer can determine the nonlinearity in the time frame after the time frame selection. The ELE network can estimate the ego lane of the ego vehicle 102 based on the predicted left and right index of the ego lane of the ego vehicle 102. In some embodiments, the ELE network can be located outside the device 107.In some examples, the ELE network can be part of device 107.

[0037] In one example, the in Fig. The device 107 shown in Figure 1b comprises an interface 109, a memory 111, and a central processing unit (also referred to as "CPUs" or "one or more processors") 113. In some embodiments, the memory 111 stores instructions and is coupled to the one or more processors 113. The memory 111 stores the instructions that can be executed by the one or more processors 113. The one or more processors 113 may include at least one data processor for executing program components for handling user- or system-generated requests. The one or more processors 113 may execute one or more functions of the device 107 for estimating an ego lane of the vehicle. The interface 109 may be coupled to the one or more processors 113, through which a plurality of road signatures are received.For example, one or more processors 113 can communicate with an image capture device 105 to receive the multitude of street signatures, as in . Fig. 1b shown.

[0038] In one embodiment, the one or more processors 113 may comprise one or more modules or hardware units, such as a receiver unit 117, an application unit 119, a grouping unit 121, an execution unit 123, and an estimating unit 125, but not only these. In some embodiments, the one or more modules or units may be software modules that can be stored in memory 111. The one or more modules or dedicated hardware units may be configured to perform the various operations of this disclosure to estimate the ego-trajectory of the ego-vehicle.

[0039] Fig. 2a shows a flowchart 200a for the coding of a variety of lane marking types, in accordance with some embodiments of the present disclosure.

[0040] In block 202, the device 107, which includes a receiver unit 117, can receive the type of lane marking (e.g., RB, DW, etc.) and the associated distance from the image acquisition device 105. In block 204, the device 107, which includes an application unit 119, can apply an encoding technique to code the type of lane marking. The encoding technique can be a one-hot encoding technique, as explained in the preceding sections. The value "0" can be assigned to the other lane marking types, and the value "1" can be assigned to the selected lane marking type. In block 206, the device 107, which includes a replacement unit, can replace the value "1" in a hot encoding with the lane marking distance using Ego. In block 208, the device 107, which includes a conversion unit, can convert the “0” of the other lane marking type in hot-coding format to infinity.In another example, the device 107, with a conversion unit in block 210, can convert the infinity values ​​of the other lane marking types into the values ​​+1000 and -1000. This is because the value "0" can indicate the absence of all other lane marking types (except the lane marking type itself). The device 107, which includes an adding unit, can add random variables to ±1000 values ​​in a hot coding process and generate at least one coded road signature for the corresponding lane type. The device 107, with the execution unit 123, can repeat the steps described in blocks 202-214 for all lane markings in the road signature and generate further coded road signatures.In block 216, the device 107, which includes a grouping unit 121, can sequentially group encodings or generated coded street signatures (nested OnexD encoding) and add dummy encodings (coded street signatures), and stack the multitude of coded street signatures, as specified in block 218. In block 220, the device with the grouping unit 121 can group all coded street signatures. Finally, the device 107, which includes a collection unit, can collect coded street signatures for past timestamps and current timestamps, as shown in block 222. The device 107, which includes a generation unit, can generate at least one context-dependent sequence file, as specified in block 224.The process can be repeated for different lane marking types at different timestamps and can generate different context-dependent sequence data in relation to the different lane marking types.

[0041] Fig. Figure 2b shows a flowchart 200b for generating context-related sequence data in accordance with some embodiments of the present disclosure.

[0042] In block 226, the device 107, which includes a capture unit, can capture frames or vectors (historical or temporal data) of combined coded road signatures. In block 228, the capture unit can capture still images of coded road signatures at a distance of 15 m in front of the ego vehicle and at a distance of 15 m behind the ego vehicle 102. The step in block 228 can be performed to capture data on the movement of the ego vehicle 102 across the road. In each time frame, the lane type and distance data are captured. Between two consecutive time frames, the ego vehicle 102 can move forward spatially. Accordingly, all images required to move the vehicle 102 from 30 m behind its current position to its current position are captured. The ego vehicle 102 can, by default, look 15 m ahead. If the number "N" of collected images is greater than or equal to the threshold (e.g.,(120 images), the Ego vehicle 102 can move at different speeds, so the different images may be required to cover a distance of 30 m. Accordingly, the limit of 120 images can be considered a practical upper limit.), the device 107 can only stack or group the most recent images of the coded street signatures, as specified in Block 230. If the number of frames "N" of the collected frames is less than the threshold, the device 107 can stack all frames of the combined coded street signatures, as specified in Block 232. Following the steps mentioned in Blocks 230 or 232, the device 107 can only select images until continuity is maintained, as specified in Block 234. Maintaining continuity is primarily based on the data provided by the sensor (e.g., the image capture device 105).If the system detects that the data supplied by the sensor is incorrect or inaccurate, continuity can be broken. Continuity in image selection can only be maintained until the sensor supplies correct or accurate data.

[0043] In block 236, the device 107, which includes the adding unit, can add dummy frame encodings to increase the total number of frames or the number of frames up to the threshold. For example, if more than 120 images are needed, the device 107 can capture the sequence up to 15 m in front of and x m behind it until 120 images are covered. Here, x can be greater and / or less than 15 m. If fewer than 120 images are needed, the device 107, which includes an adding unit, can add dummy frame encodings to generate a sequence of 120 images. Continuity is maintained over time; all successive timeframes can be considered valid. Alternatively, the continuous data stream can be considered, with all data from the point of discontinuity onward being discarded. The generated sequence of 120 individual images (feature vector) can be viewed as context-dependent sequence data.Accordingly, the context-dependent sequence data can be generated as specified in Block 238.

[0044] Fig. Figure 2c shows a flowchart 200c for generating coded lane-road signature sequence data in accordance with some embodiments of the present disclosure.

[0045] In block 240, the device 107, which includes a capture unit, can capture current and previous images, e.g., i, j, vf[i], and vf[j]. For example, "i" can be a current image, vf[i] can be a current video frame number, "j" can be a previous image, and vf[j] can be a previous video frame number. In block 242, the device 107 can verify or check whether the difference between the current image and the previous image is equal to the difference between the numbers of the current video frame and the numbers of the previous image. The device 107 also checks whether the distance is less than or equal to 30 m (e.g., 15 m in front of the vehicle and 15 m behind the vehicle). If the condition in block 242 is "true," the device 107 can stop sequence generation as specified in block 244. If the condition in block 242 is not "true", the process can continue in block 252.In block 252, the device can add 107 sequences using the adding unit connected to the device 107.

[0046] In block 246, the device 107 can check whether the number of frames equals the threshold of 120. If the number of frames equals the threshold frames, the process can be stopped as specified in block 250. If the number of frames does not equal the threshold frames, the device 107 can terminate the generation of the sequence, as specified in block 248. The sequence of images can be generated based on the distance traveled by the ego vehicle 102. Each sequence can include images where the cumulative distance between successive images is up to 30 meters. The present disclosure is not limited to this. Each generated sequence can contain "N" images. Finally, continuity issues during the merging of the data sets can be checked by performing an integrity check using the video frame numbers.

[0047] Fig. Figure 2d shows a nested OnexD hot coding technique according to some embodiments of the present disclosure.

[0048] As in Fig. Figure 2d shows the coding convention (RB, DW, DY, C, SW, SY) 253. For each lane marking (with lane type and spacing), the device 107 can generate an individual coded lane marking. For example, if (RB,9.19) is the road signature mentioned in Block 254, the coded road signature (9.19, 1001.24, 999.46, 998.79, 1000.33, 1000.89) 255 can be generated. Similarly, for the (SW, 8.67) mentioned in Block 254, the coded road signature (1000.14, 999.91, 1001.79, 1002.03, 8.67, 998.25) 256 can be generated. Furthermore, (DW, 4.74) can be the street signature mentioned in block 254, and the encrypted street signature (-1000.78, -998.25, -1000.27, -1001.79, -6.35, -99.22) 257 can be generated. Similarly, C can be the street signature mentioned in block 254, and the encoded street signature [1000.54, 999.78, 998.26, -100.5, -1002.31) 258 can be generated. Additionally, (RB, -9.55) The road signature mentioned in block 254, the coded road signature [-9.55, -1001.24, -999.46, -998.79, -1000.33, -1000.89) 259, can be generated. Similarly, corresponding coded road signatures 260 can be generated for all road signatures. The stabilization module (in . Fig. (2d not shown) Add left dummy codes, e.g. [1000+n, 1000+n, 1000+n, 1000+n, 1000+n, 1000+n] and right dummy codes [-1000+n, -1000+n, -1000+n, -1000+n, -1000+n, -1000+n]. The 'n' can be randomly selected from Gauss (0, C1*tanh(C2*(D / RW)).

[0049] The next step, as described in block 261 of Fig. As shown in Figure 2d, the device 107 can stack or group all coded lane markings and add coded dummy lane markings. For example, the type of lane marking on both sides of the "C" is variable. However, the neural network may require fixed input data. The fixed input data can be created. The maximum upper limit for the lane marking types can be assumed for the right side of the "C" and the left side of the "C". For example, the upper limit can be assumed to be 8 for each side of the "C". Accordingly, the road may contain 8 lane marking types on the left side of the ego vehicle 102 and 8 lane marking types on the right side of the ego vehicle 102, which is located in lane "C" of the ego vehicle 102. The present disclosure is not limited to this. The lane may contain any number of lane marking types to the right of "C" or to the left of "C".In the example above, the total number of lane marking types on the road can be 17 (8+8+1). The size of the coded road signature is "6". This can mean that 6 dimensions are required for each type of lane marking, i.e., in block 262, a feature vector with 6*17=102 dimensions can be generated.

[0050] To add fictitious coded lane markings, the device 107 can check if there are fewer lane marking types; if so, the device 107 can consider all lane marking types and fill in the remaining lane marking types with the fictitious lane markings. In the Fig. In the example shown in Figure 1a, there are 4 lane marking types on the left and 3 on the right side of "C", therefore 8-4 = 4 coded dummy lane markings can be added to the left of (RB,9.19). Similarly, 8-3 = 5 fictitious coded lane markings can be added to the right of (RB,-9.55).

[0051] The Fig. Figures 3a to 3c show examples of the coding of different lane marking types in some embodiments of the present disclosure.

[0052] As in Fig. Figure 3a shows the coding convention (RB, DW, DY, C, SW, SY) 253. First, in block 302, device 107 can select a road signature with lane type RB and the associated spacing of 9.19. In the next step, the one-hot coding format can be generated. The one-hot coding format can have a size of 6. In block 303, device 107 can apply a one-hot coding procedure to the road signature and generate the one-hot coding format [1,0,0,0,0,0,0]. Since RB is first in the one-hot coding format, it is replaced by the value 1 using the one-hot coding procedure. The other lane marking types can be 0. In the next step, the device 107 can replace the value “1” in the one-hot coding format with the corresponding distance of the selected lane marking type (RB) using the OnexD technique. For example, [9.19,0,0,0,0,0].In block 304, the value "1" is replaced by the distance 9.19, and the remaining lane marking types, such as DW, DY, C, SW, and SY, are not changed. These lane marking types can display the value "0". However, the value "0" can indicate the default distance of the Ego vehicle 102. In block 305, device 107 can replace the value "0" for the other lane marking types with positive or negative infinity [9.19, ∞, ∞, ∞, ∞, ∞] on either side of the "C". The positive infinities can be used, for example, to replace the "0" to the left of the "C". The negative infinities can be used to replace the "0" on the right side of the "C". In some examples, the positive and negative infinities can indicate that the other lane marking types are not present. Replacing "0" with infinities can be a mathematical operation.For practical purposes, the device 107 in block 306 can replace the infinities with predefined values, e.g., +1000 and -1000. For example, the infinities [9.19, ∞, ∞, ∞, ∞, ∞] are replaced by +1000 [9.19, 1000, 1000, 1000, 1000, 1000]. The road width can be approximately 30 m on both sides; therefore, +1000 and -1000 can be used instead of the infinities to mark an RB-type lane.

[0053] With the single hot encoding format containing +1000 and -1000 values, the neural network algorithm or optimization can get stuck when processed within the neural network. To overcome this problem, Device 107 can add stabilizing values ​​using the stabilization module. Accordingly, Device 107 can generate the vector data or the encoded road signature [9.19, 1001.24, 999.46, 998.79, 1000.33, 1000.89] 307 for at least one lane marking type (RB) in Block 307.

[0054] As in Fig. Figure 3b shows the coding convention (RB, DW, DY, C, SW, SY) 253. First, in block 309, the device 107 can select a road signature with the lane type "SW" and the associated spacing -6.35. In the next step, the one-hot coding format can be generated. The one-hot coding format can have a size of "6". In block 310, the device 107 can apply a one-hot coding procedure to the road signature and generate the one-hot coding format [0,0,0,0,1,0]. Since "SW" is in the fifth position in the one-hot coding format, "SW" is replaced by the value "1" using the one-hot coding procedure. The other lane marking types can be 0. In the next step, the device 107 can replace the value “1” in the one-hot coding format with the corresponding distance of the selected lane marking type (SW) using the OnexD technique. For example: [0, 0, 0, 0, -6,35, 0].In block 311, the value "1" is replaced by the distance -6.35, and the remaining lane marking types, such as RB, DW, DY, C, and SY, must not be changed. These lane marking types can display the value "0." However, "0" can indicate the default distance of the Ego vehicle 102. In block 312, device 107 can replace the value "0" for the other lane marking types with positive or negative infinity [∞, ∞, ∞, ∞, -6.35, ∞] on either side of the "C." The positive infinities can be used, for example, to replace the "0" to the left of the "C." The negative infinities can be used to replace the "0" to the right of the "C." In some examples, the positive and negative infinities can indicate that the other lane marking types are not present. Replacing "0" with infinities can be a mathematical operation.For practical purposes, device 107 in block 313 can replace the infinities with predefined values, e.g., +1000 and -1000. For example, the infinities [∞, ∞, ∞, ∞, -6.35, ∞] are replaced by -1000 [-1000,-1000, -1000,-1000,-6.35,-1000]. The road width can be approximately 30 m on both sides; therefore, +1000 and -1000 can be used as replacements for the infinities to mark a lane of type "SW".

[0055] With the single hot encoding format with +1000 values ​​and -1000 values, the algorithm or optimization of the neural network can get stuck at local minima. To overcome this problem, the device 107 can add stabilization values ​​using the stabilization module. Accordingly, the device 107 can generate the vector data or the encoded road signature [1000.78,-998.25, -1000.27,-1000.79,-6.35,-999.22] for at least one lane marking type (SW) in block 314.

[0056] As in Fig. As shown in Figure 3c, the device 107 can first select a lane signature with lane type "C" and the associated spacing of 0 in block 316. In the next step, the one-hot coding format can be generated. The one-hot coding format can have a size of 6. In block 317, the device 107 can apply a one-hot coding procedure to the road signature and generate the one-hot coding format [0,0,0,1,0,0]. Since the "C" is in the fourth position in the one-hot coding format, the "C" is replaced by the value "1" using the one-hot coding procedure. The other lane marking types can be 0. In the next step, the device 107 can replace the value “1” in the one-hot coding format with the corresponding distance of the selected lane marking type (C) using the OnexD technique. For example: [0, 0, 0, 0,-0,0].In block 318, the value "1" is replaced by the distance "0", and the remaining lane marking types, such as RB, DW, DY, SW, and SY, must not be changed. These lane marking types can display the value "0". However, "0" can indicate the default distance of the Ego vehicle 102. In block 319, device 107 can replace the value "0" for the other lane marking types with positive or negative infinity [∞, ∞, ∞, ∞, ∞] on either side of the "C". The positive infinities can be used, for example, to replace the "0" to the left of the "C". The negative infinities can be used to replace the "0" on the right side of the "C". In some examples, the positive infinity and negative infinity can indicate that the other lane marking types are not present. Replacing "0" with infinities can be a mathematical operation.For practical purposes, the device 107 in block 320 can replace the infinities with the specified values, for example, +1000 and -1000, as indicated in block 320. For example, the infinities [∞, ∞, ∞,0, ∞, ∞] are replaced by -1000 [-1000,-1000,-1000,0,-1000,-1000]. The road widths can be approximately 30 m on both sides; therefore, +1000 and -1000 can be used to replace the infinities and mark a lane of type "SW".

[0057] With the single hot encoding format containing +1000 and -1000 values, the neural network algorithm or optimization can get stuck when processed within the neural network. To overcome this problem, Device 107 can add stabilization values ​​using the stabilization module. Accordingly, Device 107 in Block 321 can generate the vector data or the encoded road signature [-1000.54, -998.78, 998.26,-0,-1001.50,-1002.31] for at least one lane marking type (C).

[0058] In one example, the coded street signatures are grouped, lane coding data is added, and context-sensitive sequence data is generated. This context-sensitive sequence data is then passed to the ELE network. The ELE network can be, for example, an LSTM neural network. The ELE network can contain multiple dropout layers and a linear layer. The LSTM-based neural network can accept sequential data as input and pass it to the dropout layers. The dropout layers prevent overfitting of the input data. The linear layer can map the selected Nth frame features to a low-dimensional space. The Parametric Rectified Linear Unit (PReLU) activation function introduces nonlinearity. The dual linear heads can predict the index for both the left and right lanes.

[0059] Fig. Figure 4 shows a flowchart illustrating a method 400 for estimating the ego lane of vehicle 102 in accordance with some embodiments of the present disclosure.

[0060] As in Fig. As shown in Figure 4, Procedure 400 can comprise one or more steps. Procedure 400 can be described in the general context of computer-executable instructions. Computer-executable instructions can generally include routines, programs, objects, components, data structures, procedures, modules, and functions that perform specific functions or implement specific abstract data types.

[0061] The order in which Method 400 is described is not to be understood as a restriction, and any number of the described method blocks can be combined in any order to perform the method. Furthermore, individual blocks can be omitted from the method without affecting the scope of the subject matter described herein. In addition, the method can be implemented in any suitable hardware, software, firmware, or a combination thereof.

[0062] In Block 402, Procedure 400 comprises receiving a plurality of road signatures of the vehicle at each timestamp from an image acquisition device connected to the vehicle. Each road signature includes a lane type and corresponding distance information. The operations of Block 402 can be performed by Processor 113 (in particular by the receiving unit 117) of Fig. 2 will be carried out.

[0063] In Block 404, Procedure 400 involves applying a coding technique to the plurality of lane markings to generate a plurality of coded lane markings corresponding to a current timestamp of the plurality of timestamps and one or more previous timestamps. Each coded road signature includes vector data corresponding to each lane marking type. The operations of Block 404 can be performed by Processor 113 (specifically, by Application Unit 119) of Fig. 2 will be carried out.

[0064] In block 406, procedure 400 includes grouping the multiple coded street signatures corresponding to the one or more previous timestamps and the current timestamp, and adding one or more dummy lane coding data to generate context-dependent sequence data for the timestamp. The operations of block 406 can be performed by processor 113 (specifically by grouping unit 121) of Fig. 2 will be carried out.

[0065] In block 408, procedure 400 comprises performing the steps mentioned in blocks 402-406 for the remaining timestamps of the plurality of timestamps in order to generate remaining context-dependent sequence data. The operations of block 408 can be performed by processor 113 (specifically by execution unit 123) of Fig. 2 will be carried out.

[0066] In block 410, procedure 400 comprises the estimation of the vehicle's ego lane based on the context-sensitive sequence data and the remaining context-sensitive sequence data. The operations of block 410 can be performed by processor 113 (specifically by estimation unit 125) of Fig. 2 will be carried out.

[0067] The terms “an embodiment”, “elaboration”, “elaborations”, “the embodiment”, “the embodiments”, “one or more embodiments”, “some embodiments”, and “an embodiment” mean “one or more (but not all) embodiments of the invention(s)”, unless expressly stated otherwise. The terms “including”, “comprising”, “with”, and variations thereof mean “including, but not limited to”, unless expressly stated otherwise. The listing of items does not imply that all or individual items are mutually exclusive unless expressly stated otherwise. The terms “a”, “an”, and “the” mean “one or more”, unless expressly stated otherwise.

[0068] The description of an embodiment with several interconnected components does not mean that all of these components are required. Rather, a multitude of optional components are described to illustrate the wide variety of possible embodiments of the invention.

[0069] When a single device or article is described here, it is clear that more than one device / article (regardless of whether they interact) may be used instead of a single device / article. When more than one device or article is described here (regardless of whether they interact), it is clear that a single device or article may be used instead of the multiple devices or articles, or that a different number of devices or articles may be used instead of the specified number of devices or programs. The functionality and / or features of a device may alternatively be embodied by one or more other devices that are not explicitly described as having this functionality / features.Therefore, other embodiments of the invention need not include the device itself.

[0070] Finally, the language used in the description was chosen primarily for readability and guidance purposes, and not to define or describe the subject matter of the invention. It is therefore intended that the scope of the invention is not limited by this detailed description, but rather by all claims based on an application thereunder. Accordingly, the embodiments of the present invention are intended to illustrate, but not to limit, the scope of the invention set forth in the following claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 102022004774A1

[0007] US 20120150437A1

[0007]

Images