METHOD AND DEVICE FOR CONTROLLING THE TERRAIN MODE USING A ROAD CONDITION ASSESSMENT MODEL BASED ON DEEP LEARNING

Abstract

Method for automatically controlling a vehicle's driving state (118) in a terrain mode optimized for a road surface, the method comprising the following steps: Obtaining road surface data from a camera (104) attached to the vehicle (118) and designed to capture an image of the road surface; Inputting the road surface data into a pre-trained, deep learning-based learning model that is trained to estimate a road friction coefficient according to the road surface; Determine, based on the estimated road friction coefficient, a terrain mode optimized for the road surface, and generate optimal terrain mode information; and Controlling an in-vehicle module based on optimal terrain mode information, characterized by the fact that Estimating the coefficient of road friction involves the following steps: Measuring a wheel speed and wheel torque value of a wheel attached to the vehicle (118) using an electronic stability control (ESC); and Estimating the road friction coefficient based on the measured wheel speed and the measured wheel torque value.

Description

CROSS-REFERENCE TO RELATED REGISTRATIONS

[0001] The present application is based on and claims priority of Korean patent application No. KR 10 2022 0 027 327 A, filed on August 26, 2020, which is incorporated in full by reference into the present application. BACKGROUND OF THE INVENTION 1. Field of the invention

[0002] In some embodiments, the present disclosure relates to a method and a device for controlling a terrain mode using a deep learning-based road condition assessment model. In particular, the present disclosure relates to a method and a device for controlling a terrain mode using a pre-trained deep learning-based road condition assessment model and road surface data obtained from a camera. 2. Discussion of the state of the art

[0003] The statements made in this section merely provide background information in connection with the present disclosure and do not necessarily represent the state of the art.

[0004] A vehicle's performance changes in response to different road surface conditions, such as a wet road, a dirt road or off-road track, a sandy and muddy road, and a snow-covered road. Vehicle manufacturers have conducted research and development of electronic control units installed in vehicles, enabling them to actively respond to different road surface conditions and improve performance.

[0005] The result is a commercially available terrain mode control technology that allows the driver to select a specific terrain mode according to the given surface condition of the road being traveled, thereby controlling the vehicle's drive system, shift time, and braking system.

[0006] The terrain mode control technology is implemented by a process that involves a driver visually determining the road surface condition, selecting a specific terrain mode using a jog shuttle selector, and changing the operating state of the corresponding in-vehicle modules according to the selected terrain mode. The jog shuttle selector is one of several different types of in-vehicle physical buttons that can be installed during the vehicle manufacturing process, but which are interchangeable with other button types.

[0007] Conventional terrain mode control technology requires the driver to manually determine the road surface condition with the naked eye, requires an additional action by the driver to change the terrain mode, and requires a special device, such as a physical button or software button to be provided in the vehicle, resulting in a web of limitations.

[0008] Currently, vehicle-related technologies are evolving towards the realization of an autonomous vehicle that is self-driving, while conventional terrain mode control technology requires human intervention at various stages, which adversely affects the implementation of advanced autonomous driving technology.

[0009] As a basic technology for implementing Level 4 or higher autonomous driving, a technology is therefore required in which the vehicle can actively intervene in the control of the terrain mode and control the assistance function of automatic driving. DE 10 2018 203 807 A1 discloses a method for automatically controlling a vehicle's driving state in a terrain mode optimized for a road surface, with the features of the preamble of claim 1. The subsequently published DE 10 2020 121 406 A1 discloses a device for classifying a road surface and a system for controlling a vehicle's terrain mode. BRIEF OVERVIEW OF THE INVENTION

[0010] According to at least one embodiment, the present disclosure provides a method for automatically controlling a vehicle's driving state in a terrain mode optimized for a road surface, comprising the steps of obtaining road surface data from one or more cameras mounted on the vehicle by means of a captured image of the road surface traveled by the vehicle, estimating a road friction coefficient corresponding to the road surface by inputting the road surface data into a pre-trained, deep learning-based learning model, determining terrain modes optimized for road surfaces traveled by the vehicle by using the road friction coefficient, generating information about the optimal terrain mode, and controlling one or more vehicle-internal modules based on the information about the optimal terrain mode.Estimating the road friction coefficient comprises the following steps: measuring a wheel speed and wheel torque value of a wheel fitted to the vehicle (118) using an electronic stability control (ESC); and estimating the road friction coefficient based on the measured wheel speed and wheel torque value.

[0011] According to another embodiment, the present disclosure provides a method for assisting a vehicle in controlling a terrain mode, comprising the steps of obtaining road surface data from one or more cameras mounted on the vehicle by means of a captured image of the road surface traveled by the vehicle, estimating a road friction coefficient corresponding to the road surface by inputting the road surface data into a pre-trained, deep learning-based learning model, determining terrain modes which are optimized for road surfaces traveled by the vehicle by using the road friction coefficient, and providing information about the optimal terrain mode.wherein estimating the road friction coefficient comprises the following steps: measuring a wheel speed and a wheel torque value of a wheel fitted to the vehicle (118) using an electronic stability control (ESC); and estimating the road friction coefficient based on the measured wheel speed and the measured wheel torque value.

[0012] According to a further embodiment, the present disclosure provides a method for assisting a vehicle in controlling autonomous driving, comprising the steps of obtaining road surface data from one or more cameras mounted on the vehicle by means of a captured image of the road surface traveled by the vehicle, estimating a road friction coefficient according to the road surface by inputting the road surface data into a pre-trained, deep learning-based learning model, determining a driving threshold using the road friction coefficient, and controlling one or more vehicle-internal modules based on the driving threshold.

[0013] According to a further embodiment, the present disclosure provides a device for automatically controlling a vehicle's terrain mode, comprising an estimating unit, a terrain mode determination unit, and a terrain mode control unit. The estimating unit estimates a road surface condition, has a pre-trained, deep learning-based learning model, and is configured to obtain road surface data from one or more cameras mounted on the vehicle by capturing an image of the road surface traversed by the vehicle, and to input a road friction coefficient corresponding to the road surface by feeding the road surface data into the deep learning-based model. The terrain mode determination unit is configured to determine, using the road friction coefficient, terrain modes that are optimized for the road surfaces traversed by the vehicle.The terrain mode control unit is configured to control one or more in-vehicle modules using information about the terrain modes. The estimation unit is further configured to estimate the road friction coefficient by measuring a wheel speed and a wheel torque value of a wheel fitted to the vehicle (118) using electronic stability control (ESC); and estimating the road friction coefficient based on the measured wheel speed and the measured wheel torque value.

[0014] According to a further embodiment, the present disclosure provides a computer program stored on a computer-readable medium for carrying out steps which are provided in the methods for automatically controlling a driving state of the vehicle in terrain modes optimized for road surfaces. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1A is a block diagram of an automatic terrain mode control device according to at least one embodiment of the present disclosure. Fig. Figure 1B is a diagram illustrating components relating to a method for supporting the control of autonomous driving of a vehicle according to at least one embodiment of the present disclosure. Fig. Figure 2 is a block diagram of a learning unit based on deep learning according to at least one embodiment of the present disclosure. Fig. Figure 3 is a flowchart of a training procedure for a learning model based on deep learning according to at least one embodiment of the present disclosure. Fig. Figure 4A is a flowchart of a method for automatically controlling a terrain mode according to at least one embodiment of the present disclosure. Fig. Figure 4B is a flowchart of a method for supporting the control of autonomous driving according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] According to at least one embodiment, the present disclosure aims to provide a technique that enables a vehicle to autonomously determine the road surface condition without driver intervention by using a pre-trained, deep learning-based learning model and to change the vehicle's driving state in response to different road surface conditions.

[0016] According to another embodiment, the present disclosure aims to assist a vehicle in carrying out terrain mode control by determining road surface conditions even in the usual and non-critical driving situations that do not lead to intervention of the electronic stability control (ESC), and by determining terrain modes optimized for the road surfaces and subsequently providing the terrain mode information to the driver.

[0017] According to a further embodiment, the present disclosure aims to provide a device for terrain mode control in order to save the vehicle manufacturer the need to install separate physical buttons in the vehicle to assist the driver in handling the vehicle, thereby saving vehicle manufacturing costs and ensuring free space in the vehicle to provide more comfort to the driver.

[0018] Some exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings. In the following description, identical reference numerals preferably denote identical elements, even if the elements are shown in different drawings. Furthermore, for the sake of clarity and brevity, a detailed description of known functions and configurations included herein is omitted in the following description of some embodiments.

[0019] Furthermore, alphanumeric codes such as first, second, i), ii), a), b), etc., are used in component numbering solely to distinguish one component from another, and not to imply or suggest the substances, order, or sequence of the components. Throughout the description, if parts "contain" or "have" one component, they may also have other components and do not exclude them, unless explicitly stated otherwise. Terms such as "unit," "module," and the like denote units for processing at least one function or operation, which may be implemented by hardware, software, or a combination thereof.

[0020] Recent functional improvements exist for estimating physical information from image data based on a deep learning method. The present disclosure, in several embodiments, provides a new way of using such an information estimation function as a basis for enabling a vehicle to determine road conditions and autonomously control its driving.

[0021] More precisely, the present disclosure, according to several embodiments, provides a device and a method comprising the steps of calculating road friction coefficient information by means of a road surface condition estimation unit using a deep learning-based learning model and controlling the engine power, vehicle direction, braking speed, braking point, and other factors without driver intervention. Furthermore, the proposed uses of the present disclosure are specified.

[0022] Although some embodiments of the present disclosure describe a driving state control method in an autonomously driving or semi-autonomously driving vehicle, the embodiments relate to numerous different means of transport equipped with a camera and an electronic control unit (ECU).

[0023] The following description of the present disclosure, with reference to the associated drawings, serves to describe exemplary embodiments of the present disclosure, rather than to present the only embodiments in which the present disclosure can be implemented.

[0024] Fig. Figure 1A is a block diagram of an automatic terrain mode control device 100 according to at least one embodiment of the present disclosure.

[0025] The automatic terrain mode control device 100 according to this embodiment comprises a road surface estimation unit 102, which is generally formed from three units of a data collection unit (DCU) 106, and / or a road friction coefficient estimation unit 110 and / or a learning unit 112 and / or a terrain mode determination unit 114 and / or a terrain mode control unit 116.

[0026] The in Fig. The automatic terrain mode control device 100 shown in Figure 1 A corresponds to at least one embodiment of the present disclosure and all of the features described therein. Fig. The blocks shown in Figure 1A are not essential blocks. In another embodiment, some of the blocks included in the automatic terrain mode control device 100 can be added, modified, or omitted. For example, provided that a communication unit (not shown) capable of communicating with the driver is added between the terrain mode determination unit 114 and the terrain mode control unit 116, the automatic terrain mode control device 100 can operate as a terrain mode assist device to provide the driver with terrain mode information optimized for the road surface conditions of the road being traveled and to control the respective in-vehicle modules according to the terrain mode selected by the driver in response.

[0027] The automatic terrain mode control device 100 receives road surface data, estimates the road friction coefficients, determines suitable terrain modes for the road surfaces, and makes a change in the driving state in response to different terrains. The terrain modes relate to driving states that are preset by a vehicle manufacturer in the respective in-vehicle terrain mode controllers after calculating reference values ​​optimized for different road surfaces and using these calculated reference values ​​to preset the driving states. When the driver or the system selects a specific terrain mode from among several available options, each in-vehicle terrain mode controller can change the driving state to a preset reference value. The individual components contained in the automatic terrain mode control device 100 are described below.

[0028] The road surface condition estimator 102 sends road friction coefficient information to the terrain mode determination unit 114, which is processed using road surface data obtained from a camera 104 and a deep learning-based learning model. The road surface condition estimator 102 can be an electronic control unit that trains the deep learning-based learning module.

[0029] The camera 104 is mounted on one side of the vehicle to provide road surface data, such as an image or video recording of the vehicle's circumference. The data acquisition unit 106 can receive road surface data from the camera 104 and includes a data storage unit 108 for storing the received road surface data. In another embodiment, the data acquisition unit 106 can, in addition to collecting the road surface data from the camera 104, receive and store road surface data from a radar or lidar (not shown in each case) mounted on one side of the vehicle to emit a signal to an object in the vicinity of the vehicle and to analyze the signal reflected by the object.

[0030] The road friction coefficient estimating unit 110 estimates road friction coefficient information using road surface data according to a feature relating to road conditions, temporarily stores the estimated road friction coefficient information in a memory such as a random access memory (RAM), and then transmits the road friction coefficient to the terrain mode determination unit 114 via a controller area network bus (CAN).

[0031] The learning unit 112 receives road surface data from the camera, radar, or lidar and generates labeling data by labeling the road friction coefficient in the road surface data. The learning unit 112 then trains the deep learning-based learning model using this labeling data. In at least one embodiment of the present disclosure, the learning unit 112 is implemented as a separate stand-alone device connected to the road surface condition estimation unit 102.

[0032] The terrain mode determination unit 114 determines a terrain mode optimized for the surface of the road being traveled based on the road friction coefficient received from the road surface condition estimator unit 102. For example, if the terrain mode determination unit 114 determines, as a result of comparing the road surface coefficient provided by the road friction coefficient estimator unit with the road friction coefficients preset for various road surfaces, that the surface of the road being traveled is a wet road, it can send the terrain mode information to the terrain mode controller 116 to change the driving condition to a preset low engine power setting.

[0033] The Terrain Mode Control Unit 116 uses the terrain mode as a basis for controlling various components in the drive and braking systems, such as the engine control unit (ECU), electronic stability control (ESC), and transmission control unit (TCU), which are not shown. For example, the ECU can modify the engine power output to a preset assigned engine value according to the terrain mode selected by the Terrain Mode Determination Unit 114. The ESC operates a controller for a preset traction control system (TCS) according to the terrain mode selected by the Terrain Mode Determination Unit 114. The TCU can control the vehicle's transmission at a preset shift point according to the terrain mode selected by the Terrain Mode Determination Unit 114.

[0034] Fig. Figure 1B is a diagram illustrating components relating to a method for supporting the control of autonomous driving of a vehicle according to at least one embodiment of the present disclosure.

[0035] The vehicle 118, equipped with an autonomous driving control unit 120, estimates the road friction coefficient using road surface data obtained from the camera 104 and uses the autonomous driving control unit 120 to control the respective in-vehicle modules. The operation of a vehicle 118 equipped with an autonomous driving control unit is described below with reference to Fig. 4B described.

[0036] Fig. Figure 2 is a block diagram of a learning unit based on deep learning according to at least one embodiment of the present disclosure.

[0037] The following refers to Fig. 2 the learning unit 112 is described according to at least one embodiment of the present disclosure.

[0038] The learning unit 112 according to at least one embodiment comprises a data acquisition unit 202 and / or a labeling data generator 204 and / or a training unit 206.

[0039] The data acquisition unit 202 collects training data to train the deep learning-based learning model.

[0040] The data acquisition unit 202 receives the road friction coefficient, which was collected for the road surface traveled on, from the road friction coefficient estimation unit 110. In another embodiment, the data acquisition unit 202 receives from the ESC a road friction coefficient that was calculated based on a wheel speed and a wheel torque value of a wheel attached to the vehicle.

[0041] The data acquisition unit 202 collects the road surface data from camera 104, which correspond to road friction coefficients obtained by the road friction coefficient estimation unit 110 in a road friction coefficient acquisition stage, which is carried out as part of a training data collection process to train the deep learning-based learning model.

[0042] The road surface data are images of road surface areas in front of the vehicle, captured by camera 104, where there is a certain distance between the road surface area in the image and the vehicle. Based on this fact, according to at least one embodiment, the data acquisition unit 202 takes into account the distance between the road surface area in the image and the vehicle, for example, the vehicle's tire, when collecting the road surface data from camera 104, in order to enable the collection of accurate road surface data corresponding to the relevant road friction coefficient.

[0043] On the other hand, in at least one embodiment of the present disclosure, the data acquisition unit 202 selectively collects, together with the road friction coefficients, the road surface data corresponding to the relevant road friction coefficient only when event information is collected from the road friction coefficient estimating unit 110. In other words, the data acquisition unit 202 can react to event information when it is detected in the data transmitted by the road friction coefficient estimating unit 110 in order to determine the road friction coefficient as highly reliable, to extract the road friction coefficient accordingly, and to collect the road surface data corresponding to the extracted road friction coefficient.The data acquisition unit 202 can specify and obtain a value between 20 ms and 1 s for the road friction coefficient in the data in order to accurately extract the road friction coefficient. The data acquisition unit 202 can skip data from the first 1 s and the last 1 s, which may contain event information.

[0044] On the other hand, in at least one other embodiment of the present disclosure, the event information is generated by the road friction coefficient estimator 110 exclusively when a predefined event situation occurs in conjunction with the learning of road friction coefficients. In this case, this predefined event situation can be determined depending on the prediction accuracy of the road friction coefficient calculated by the road friction coefficient estimator 110 when the corresponding event situation occurs. For example, the road friction coefficient estimator 110

[0045] Set an event situation based on a situation in which the prediction accuracy of the calculated road friction coefficient is equal to or higher than a preset threshold of prediction accuracy.

[0046] The data acquisition unit 202 can perform preprocessing of the collected road surface data. For example, the data acquisition unit 202 can detect a road surface area within the collected road surface data and extract and provide image data that corresponds exclusively to the detected road surface area. This simplifies the subsequent operation in the training process by reducing the pixel area to be computed.

[0047] The identification data generator 204 serves to obtain the road surface data and the road friction coefficient from the data acquisition unit 202 and to use this as a basis for generating training data to train a learning model.

[0048] In at least one embodiment of the present disclosure, the marking data generator 204 generates marking data based on the road surface data and the road friction coefficient obtained from the data acquisition unit 202, and provides the generated marking data as training data.

[0049] For example, the label generator can generate 204 label data by labeling the road friction coefficient in the collected road surface data.

[0050] Training unit 206 uses the labeling data obtained from labeling data generator 204 as training data to train the deep learning-based learning model. A specific procedure for training a learning model using training unit 206 based on the training data is now common practice in the relevant technical field, and a detailed description of this procedure is therefore unnecessary.

[0051] The training unit 206 according to the present embodiment operates such that the learning model is trained only on the images from camera 104 to estimate road friction coefficient information. In particular, the learning model learns and stores the road friction coefficient measured by the road friction coefficient estimation unit 110 for each road surface data element with different feature information.

[0052] The road friction coefficient estimating unit 110 has a learning model that was pre-trained according to the operation of the learning unit 112 and undergoes a learning process using its pre-trained learning model to estimate a road friction coefficient according to the road travelled by the vehicle.

[0053] The road friction coefficient estimating unit 110 estimates the road friction coefficient according to the road surface by applying the road surface data obtained from the data collection unit 106 to the training model.

[0054] The road friction coefficient estimator 110 applies the road surface data to the training model to extract suitable training data for the road surface data with the same or similar feature information as the road surface data, and estimates the road friction coefficient based on the extracted training data. The road friction coefficient estimator 110 can estimate the current road friction coefficient information from information about the previous road friction coefficient that is labeled in the preceding road surface data with the same or similar feature information as the road surface data collected at the current time.

[0055] Fig. Figure 3 is a flowchart of a training procedure for estimating a road friction coefficient according to at least one embodiment of the present disclosure.

[0056] The data acquisition unit 202 receives the road friction coefficient collected for the road surface traveled by the vehicle from the road friction coefficient estimation unit 110 (S300).

[0057] The data acquisition unit 202 checks whether event information is being collected along with the road friction coefficient (S302). In step S302, the data acquisition unit 202 can only receive event information calculated by the road friction coefficient estimator unit 110 if a predefined event situation occurs in connection with the learning of the road friction coefficient. This event information indicates that the collected road friction coefficient is relevant data for learning.

[0058] Upon confirmation of event information collection in step S302, the data acquisition unit 202 collects road surface data from camera 104, according to the road friction coefficient (S304) obtained in step S300. In step S304, the data acquisition unit 202 renders the precise road surface data to be collected according to the current road surface coefficient, taking into account the current distance between the road surface area in the image and the vehicle.

[0059] The data acquisition unit 202 can identify a road surface area from the collected road surface data and extract road surface data that corresponds exclusively to the identified road surface area.

[0060] The labeling data generator 204 labels the road friction coefficient from step S300 in the road surface data collected in step S304 to generate labeling data (S306).

[0061] The training unit 206 trains the deep learning-based model using the labeling data generated in step S306 as training data (S308).

[0062] Fig. Figure 4A is a flowchart of a method for automatically controlling a terrain mode according to at least one embodiment of the present disclosure.

[0063] The data collection unit 106 collects road surface data from a camera attached to the vehicle from a captured image of the road surface traveled by the vehicle (S410).

[0064] The road friction coefficient estimating unit 110 estimates the road friction coefficient information by feeding the road surface data collected by the data collection unit 106 into a pre-trained learning model (S412).

[0065] The road friction coefficient estimation unit 110 sends the road friction coefficient estimated in step S412 via a CAN bus to the terrain mode determination unit 114.

[0066] The terrain mode determination unit 114 compares the road friction coefficient with the road friction coefficients preset for the respective road conditions and determines the road surface condition of the road traveled by the vehicle and a terrain mode optimized for the road surface (S414).

[0067] The terrain mode determination unit 114 sends terrain mode information about the road surface traveled by the vehicle to the terrain mode control unit 116.

[0068] The terrain mode control unit 116 changes the driving condition to one of the reference values ​​preset for the respective terrain modes based on the terrain mode information (S416).

[0069] In this process, the step of changing the driving state is a procedure in which an electronic control unit such as an ESC, ECU or TCU controls the respective internal vehicle modules such as a TCS controller, an engine and a transmission.

[0070] Fig. Figure 4B is a flowchart of a method for supporting the control of autonomous driving according to at least one embodiment of the present disclosure.

[0071] The data collection unit 106 receives road surface data from a camera attached to the vehicle from a captured image of the road surface traveled by the vehicle (S420).

[0072] The road friction coefficient estimating unit 110 estimates the road friction coefficient information by feeding the road surface data obtained from the data collection unit 106 into a pre-trained learning model (S422).

[0073] The road friction coefficient estimation unit 110 sends the road friction coefficient estimated in step S422 via a CAN bus to the autonomous driving control unit.

[0074] Unit 120, used for controlling autonomous driving, compares road friction coefficient information with preset road friction coefficient information for specific road conditions to determine the road surface condition of the road being traveled and establishes a vehicle driving threshold (S424). This driving threshold refers to thresholds for engine power, transmission shift points, braking points, etc., which are necessary to improve fuel efficiency or ensure driving safety based on the road surface condition. For example, on a road surface with 20 mm or more of accumulated snow, the engine power threshold might be 50 hundredths of the vehicle's maximum achievable speed.

[0075] Unit 120 for controlling autonomous driving controls the respective in-vehicle modules based on the driving threshold for the road surface traveled by the vehicle (S426).

[0076] On the other hand, Unit 120 for autonomous driving control can improve autonomous driving performance by autonomously changing the vehicle's driving state without driver intervention, using the road friction coefficient. For example, Unit 120 for autonomous driving control could be a shift pattern control device with a smart cruise control system or a brake control device with a collision avoidance system.

[0077] The Smart Cruise control system refers to a control system that responds to the driver setting a driving speed in order to adapt the vehicle's driving state by maintaining a constant driving speed according to the driving speed set by the driver and accelerating or decelerating the vehicle according to the traffic conditions for the vehicles driving on the relevant road surface.

[0078] The shift pattern control device changes the vehicle's speed in response to traffic conditions and road surface conditions by changing the engine power and the shift pattern of a transmission via an engine control unit and a transmission control unit.

[0079] The collision avoidance system refers to a system that reacts to the presence of a risk of collision between the vehicle and a vehicle ahead, that is, when the distance between the vehicles is measured as being less than a preset threshold for a notification to the driver about the danger or for autonomous control of the braking system by the vehicle, thereby avoiding vehicle collisions.

[0080] The brake control device reduces the vehicle speed by controlling a braking start speed and a braking time in order to avoid a collision between the moving vehicle and a vehicle ahead.

[0081] Although the steps in Figures 3, 4A, and 4B are described as being carried out sequentially, they merely instantiate the technical concept of some embodiments of the present disclosure. A person skilled in the art in the relevant field can therefore make various modifications, additions, and substitutions in the implementation of the present disclosure by applying the steps shown in the Fig. 3, Fig. 4A and Fig. The sequence described in 4B is changed, or by modifying one or more of the steps in the Fig. 3, Fig. 4A and Fig. 4B is carried out in parallel without departing from the basic idea and the nature of at least one embodiment of the present disclosure, and thus the steps in the Fig. 3, Fig. 4A and Fig. 4B is not limited to the chronological sequence shown.

[0082] The operation of the automatic terrain mode control device 100 can be implemented as a computer program and recorded on a computer-readable recording medium. The computer-readable recording medium storing the computer program for implementing the operation of the automatic terrain mode control device 100 comprises any type of recording device on which data readable by a computer system can be recorded. The computer-readable recording medium can comprise non-volatile media such as a ROM, RAM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, optical / magnetic disk, optical data storage devices, and the like, as well as volatile media such as a carrier wave (for example, transmission over the Internet) and a data transmission medium.Furthermore, the computer-readable recording medium can be distributed across computer systems connected via a network, allowing the computer-readable codes to be stored and executed in a distributed mode.

[0083] Various implementations of the systems and techniques described herein can be realized using digital electronic circuits, integrated circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include those that are realized in one or more computer programs executable on a programmable system.The programmable system comprises at least one programmable processor connected for receiving and sending data and instructions to and from a data storage system (including volatile memory, non-volatile memory, or any other type of storage system or a combination thereof), at least one input device, and at least one output device, wherein the programmable processor may be a special-purpose processor or a general-purpose processor. For example, the programmable system may be a server, a network device, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a personal data assistant (PDA), a cloud computing system, and a mobile device.Computer programs, also known as programs, software, software applications or code, contain instructions for a programmable processor and are stored on a "computer-readable recording medium".

[0084] As previously described, according to the present disclosure, one or more cameras mounted on a vehicle receive road surface data, and the road surface condition estimation unit estimates information about a road friction coefficient based on the road surface by inputting the road surface data into the pre-trained, deep learning-based learning model. This saves the driver from manually checking the road surface conditions with the naked eye, as the vehicle is able to autonomously determine the road surface conditions and provide the driver with information about the terrain modes based on the road surface conditions to improve driver comfort.

[0085] According to the present disclosure, the terrain mode determination unit receives road friction coefficient information from the road surface condition estimation unit and autonomously determines terrain modes optimized for the road surfaces traveled by the vehicle. Furthermore, the terrain mode control unit autonomously manages the driving state to control one or more in-vehicle modules, including an ESC, an engine control unit (ECU), and a transmission control unit (TCU), thus relieving the driver of additional manual actions when switching between terrain modes based on information about the optimized terrain modes.

[0086] Furthermore, the present disclosure eliminates the need to install physical buttons to assist the driver in operating the vehicle, resulting in reduced vehicle manufacturing costs and free space gained inside the vehicle, thus increasing the degree of freedom for the vehicle's interior design.

[0087] Although exemplary embodiments of the present disclosure have been described for illustrative purposes, it is apparent to the person skilled in the art that various modifications, additions, and substitutions are possible without deviating from the concept and scope of the claimed invention. Therefore, for the sake of brevity and clarity, exemplary embodiments of the present disclosure have been described. The scope of the technical concept of the present embodiments is not limited by the illustrations. A person skilled in the art therefore understands that the scope of the claimed invention is not to be limited by the embodiments explicitly described above, but rather by the claims and their equivalents.

Claims

[1] Method for automatically controlling a vehicle's driving state (118) in a terrain mode optimized for a road surface, the method comprising the following steps: Obtaining road surface data from a camera (104) attached to the vehicle (118) and designed to capture an image of the road surface; Inputting the road surface data into a pre-trained, deep learning-based learning model that is trained to estimate a road friction coefficient according to the road surface; Determine, based on the estimated road friction coefficient, a terrain mode optimized for the road surface, and generate optimal terrain mode information; and Controlling an in-vehicle module based on optimal terrain mode information, characterized by , that Estimating the coefficient of road friction involves the following steps: Measuring a wheel speed and wheel torque value of a wheel attached to the vehicle (118) using an electronic stability control (ESC); and Estimating the road friction coefficient based on the measured wheel speed and the measured wheel torque value. [2] Method according to claim 1, wherein obtaining the road surface data comprises obtaining additional road surface data using a RADAR or LIDAR unit attached to the vehicle (118). [3] Method according to claim 1, further comprising the steps: Generating labeling data based on road surface data and the estimated road friction coefficient; and Training the deep learning-based learning model using the labeling data as training data. [4] The method of claim 1, wherein: the vehicle's internal module features electronic stability control (ESC), and the control of the vehicle's internal module to cause the ESC to operate a controller of a traction control system (TCS) preset for terrain mode. [5] The method of claim 1, wherein: the vehicle's internal module includes an engine control unit (ECU), and controlling the vehicle's internal module to cause the ECU to change a preset engine mapping value for terrain mode. [6] The method of claim 1, wherein: the vehicle's internal module includes a transmission control unit (TCU), and controlling the vehicle's internal module to instruct the TCU to change a preset shift time for shifting a transmission for terrain mode. [7] Method for assisting a vehicle (118) in controlling a terrain mode, the method comprising the following steps: Obtaining road surface data from a camera (104) attached to the vehicle (118) and designed to capture an image of the road surface traveled by the vehicle (118); Inputting the road surface data into a pre-trained, deep learning-based learning model that is trained to estimate a road friction coefficient according to the road surface; Determine, based on the estimated coefficient of road friction, a terrain mode optimized for the road surface; and Providing optimal terrain mode information based on the optimized terrain mode characterized by , that Estimating the coefficient of road friction involves the following steps: Measuring a wheel speed and wheel torque value of a wheel attached to the vehicle (118) using an electronic stability control (ESC); and Estimating the road friction coefficient based on the measured wheel speed and the measured wheel torque value. [8] Method according to claim 7, wherein obtaining the road surface data comprises obtaining additional road surface data using a RADAR or LIDAR unit attached to the vehicle (118). [9] Method according to claim 7, wherein the provision of the optimal terrain mode information comprises the provision of the optimal terrain mode information either visually via a display attached to the vehicle (118) or audibly via a loudspeaker attached to the vehicle (118). [10] Method according to claim 7, further comprising the steps: Generating labeling data based on road surface data and the estimated road friction coefficient; and Training the deep learning-based learning model using the labeling data as training data. [11] Method for assisting a vehicle (118) in controlling autonomous driving, the method comprising the following steps: Obtaining road surface data from a camera attached to the vehicle (118) and designed to capture an image of the road surface traveled on by the vehicle; Inputting the road surface data into a pre-trained, deep learning-based learning model, which is trained to estimate a road friction coefficient according to the road surface; and Determining a vehicle speed threshold (118) based on the estimated road friction coefficient, and controlling an in-vehicle module based on the speed threshold. [12] Method according to claim 11, wherein the control of the vehicle-internal module comprises changing a shift pattern of a transmission of the vehicle (118) based on the driving threshold value. [13] Method according to claim 11, wherein the control of the vehicle-internal module comprises changing a braking start speed and a braking time of a braking system of the vehicle (118) based on the driving threshold value. [14] Device for automatically controlling a terrain mode of a vehicle (118), wherein the device comprises: a unit of estimation that is trained to: Straßenoberflächendaten von einer Kamera (104) zu erhalten, welche an dem Fahrzeug (118) angebracht ist und dazu ausgebildet ist, ein Bild der von dem Fahrzeug (118) befahrenen Straßenoberfläche aufzunehmen; und die Straßenoberflächendaten in ein vortrainiertes, auf Deep Learning basierendes Lernmodell einzugeben, das dazu ausgebildet ist, einen Straßenreibungskoeffizienten entsprechend der Straßenoberfläche zu schätzen; eine Terrainmodusbestimmungseinheit (114), die dazu ausgebildet ist, einen für die Straßenoberfläche optimierten Terrainmodus basierend auf dem geschätzten Straßenreibungskoeffizienten zu bestimmen; und a terrain mode control unit (116) which is trained to control an in-vehicle module based on the specified terrain mode, the estimating unit is further designed to estimate the road friction coefficient by Measuring a wheel speed and wheel torque value of a wheel attached to the vehicle (118) using an electronic stability control (ESC); and Estimating the road friction coefficient based on the measured wheel speed and the measured wheel torque value. [15] Nichtflüchtiges computerlesbares Medium, das Befehle aufweist, welche bei Ausführung durch einen Prozessor, den Prozessor veranlassen, ein System zu steuern, um das Verfahren nach Anspruch 1 durchzuführen.

Images