Driving assistance system and method

By introducing a driving support system in autonomous driving vehicles and generating control signaling using camera systems and processors, the flexible assist and autonomous driving function selection of the vehicle is achieved, solving the problems of updating and closing the system restrictions in the existing technology, and improving the User experience and driving efficiency.

CN112272632BActive Publication Date: 2024-12-06SAMSUNG ELECTRO MECHANICS CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN201980037861.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-04-23
Filing Date
2019-11-12
Publication Date
2024-12-06
Estimated Expiration
2039-11-12

AI Technical Summary

Technical Problem

The driving program updates and assisted information services of existing autonomous driving vehicles are restricted, and users cannot experience the complete autonomous driving and assisted driving functions, and the manufacturer's closed system limits the autonomous driving level of the vehicle.

Method used

A driving support system is designed, including a camera system, a vehicle controller and a communication port, interacting with the vehicle through a driving support terminal and a mobile device, generating and providing control signaling to control the auxiliary and autonomous driving operations of the vehicle, and performing using a processor. Driving program to achieve different levels of driving assistance and autonomous operation.

Benefits of technology

It realizes the flexibility of the vehicle and the selective provision of advanced driving assistance and autonomous driving functions, exceeds the limitations of the manufacturer's closed system and improves the user experience and vehicle driving efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112272632B_ABST
    Figure CN112272632B_ABST
Patent Text Reader

Abstract

A driving assistance system and method are provided. The driving assistance system is used to connect to a separate vehicle interface and includes: a camera system that generates image information about the surrounding environment of the vehicle; a vehicle controller that generates driving information about the vehicle; and a communication port. The driving assistance system includes a driving assistance terminal and / or a mobile device connected to the communication port, acquires image information and / or driving information by using the communication port, and executes at least a part of a driving program for the driving assistance terminal and / or the mobile device based on the image and / or driving information to control the vehicle for one or more of assisted driving and / or autonomous operations of the vehicle through a control signal generated by the driving assistance terminal or the mobile device and provided to the vehicle.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a driving support system and method. Background Art

[0002] Autonomous or self-driving vehicles or semi-autonomous or assisted driving vehicles are increasingly used to achieve driving efficiency and driving convenience and reduce traffic accidents. In order to achieve such autonomous / self-driving or semi-autonomous / assisted driving operations, there are various problems related to the control of such vehicles that are being solved in various implementations from the following: simply providing information about the driving experience to the driver, providing driving assistance functions (wherein one or more of braking, acceleration and / or steering can be selectively controlled) while requiring the driver to supervise the driving assistance functions (wherein such driving assistance functions may include lane departure warning or avoidance, collision avoidance or automatic emergency braking or blind spot warning), while other implementations allow the vehicle to additionally or alternatively perform the driving of the vehicle, and the passenger only needs to drive according to the request of the vehicle, or there are more fully automatic implementations (wherein the vehicle will not require the passenger to perform any driving function, or even the vehicle does not include a steering wheel and an accelerator pedal and a brake pedal). Such vehicle implementations can provide such information services, driving assistance, various capabilities of semi-autonomous, autonomous and fully autonomous driving through various driving programs in the vehicle to achieve such different levels of information, assistance and / or autonomy. As a result, many automotive, non-automotive, and non-manufacturing companies and organizations have each developed various driving programs for such varying levels of information, assistance, and / or autonomy through advanced driver assistance systems (ADAS).

[0003] Furthermore, while vehicle manufacturers have manufactured or have indicated that they will manufacture automobiles with engineering configurations that have the design capability to provide information or other feedback and to control acceleration, braking, steering, etc. based entirely on electronic control signals, these manufacturers have made it too slow for the vehicles to achieve these capabilities through these various driving programs (e.g., no manufacturer has produced a commercially available vehicle with an on-board autonomous driving program in the vehicle to achieve fully autonomous driving). Therefore, when these vehicles are released without such autonomous driving programs or without various assistance or information programs, although such vehicles may have physical capabilities for such automation and assistance / information implementation through various control signals, users may not be able to experience fully autonomous driving and / or experience various assistance and / or information services when using such vehicles, or may be limited by the closed system or proprietary driving program of the automobile manufacturer (which can only implement limited autonomous and / or assistance / information services about the vehicle), for example, although the vehicle has the engineering or manufacturing capabilities to provide such assistance and / or information services and control of all driving components based on, for example, various control signals, the manufacturer's vehicle autonomous driving program can only provide limited driving condition information to the driver when the vehicle can provide many other information / assistance functions and some, more or fully autonomous control of vehicle driving through alternative programming. In addition, such manufacturers may generally not provide updates to such driving programs after the vehicle is manufactured, such as when additional or alternative information-based, assistance-based and / or autonomous functions become available. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one general aspect, a driving support system for interacting with a separate vehicle, the vehicle comprising a camera system that generates image information of the vehicle's surroundings, a vehicle controller that generates driving information of the vehicle, and a communication port, the driving support system comprising: a driving support terminal, the driving support terminal comprising one or more processors, the one or more processors being configured to: connect to the communication port, obtain the image information and / or the driving information using the communication port, execute at least a portion of a driving program of the driving support terminal based on the image information and / or the driving information, so as to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle through control signaling generated by the driving support terminal and provided to the vehicle.

[0006] The driving program of the driving support terminal may include a driving assistance program and / or an autonomous driving program.

[0007] The vehicle controller may be configured to execute a vehicle driving program stored in a memory of the vehicle to control the vehicle, the vehicle driving program including a vehicle driving assistance program and / or a vehicle autonomous driving program.

[0008] The one or more processors of the driving support terminal may also be configured to determine which of the driving assistance program of the driving program of the driving support terminal and the vehicle driving assistance program of the vehicle driving program is the ADAS highest level driving assistance program, and / or determine which of the autonomous driving program of the driving program of the driving support terminal and the vehicle autonomous driving program of the vehicle driving program is the ADAS highest level autonomous driving program, and the one or more processors are configured to execute the ADAS highest level driving assistance program and / or the ADAS highest level autonomous driving program to control the vehicle for the one or more assisted driving and / or autonomous operations of the vehicle.

[0009] The one or more processors of the driving support terminal may also be configured to: perform a first determination to determine which functions of the driving program of the driving support terminal have a higher functional level than the corresponding functions of the vehicle driving program, perform a second determination to determine which functions of the driving program of the driving support terminal are not provided by the vehicle driving program, and / or perform a third determination to determine which functions of the vehicle driving program are not provided by the driving program of the driving support terminal, and the one or more processors are configured to control the driving program functions implemented by the vehicle by selectively implementing the driving program of the driving support terminal and the vehicle driving program under the control of the driving support terminal based on any one or any combination of the first determination, the second determination and the third determination, so as to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle.

[0010] The driving support system may also include a mobile device having one or more processors, the one or more processors of the mobile device being configured to execute at least a part of a mobile device driving program (including a mobile device driving assistance program and / or a mobile device autonomous driving program) based on the image information and / or the driving information to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle, and the one or more processors of the mobile device or the one or more processors of the driving support terminal may also be configured to determine which of the driving assistance program of the driving program of the driving support terminal, the mobile device driving assistance program, and the vehicle driving assistance program of the vehicle driving program is the ADAS highest level driving assistance program, and / or determine which of the autonomous driving program of the driving program of the driving support terminal, the mobile device autonomous driving program, and the autonomous driving program of the vehicle driving program is the ADAS highest level autonomous driving program, and be configured to execute the ADAS highest level driving assistance program and / or the ADAS highest level autonomous driving program to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle.

[0011] The driving support system may also include a mobile device having one or more processors, the one or more processors of the mobile device being configured to execute at least a portion of a mobile device driving program (including a mobile device driving assistance program and / or a mobile device autonomous driving program) based on the image information and / or the driving information to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle, and the one or more processors of the mobile device or the one or more processors of the driving support terminal may also be configured to perform a first determination to determine which functions of the driving program of the driving support terminal and which functions of the mobile device driving program have a higher priority than the vehicle driving program. A higher functional level than the corresponding function of the program, a second determination is performed to determine which functions of the driving program of the driving support terminal and which functions of the mobile device driving program are not provided by the vehicle driving program, and / or a third determination is performed to determine which functions of the vehicle driving program are not provided by the driving program of the driving support terminal and the mobile device driving program, and is configured to control the driving program functions implemented by the vehicle by selectively implementing the driving program of the driving support terminal, the mobile device driving program and the vehicle driving program under the control of the driving support terminal or the mobile device based on any one or any combination of the first determination, the second determination and the third determination.

[0012] When the driving program of the driving support terminal, the mobile device driving program and the vehicle driving program are selectively implemented based on any one or any combination of the first determination, the second determination and the third determination, it may include selecting between corresponding functions of the driving program of the driving support terminal and the mobile device driving program, wherein the one or more processors of the mobile device or the one or more processors of the driving support terminal may be configured to select a function with a predetermined greater functionality from these corresponding functions.

[0013] The one or more processors of the driving support terminal may also be configured to: perform a first determination to determine which functions of the driving program of the driving support terminal and which functions of a mobile device driving program of a mobile device communicating with the driving support terminal have a higher functional level than the corresponding functions of the vehicle driving program, perform a second determination to determine which functions of the driving program of the driving support terminal and which functions of the mobile device driving program are not provided by the vehicle driving program, and / or a third determination to determine which functions of the vehicle driving program are not provided by the driving program of the driving support terminal and the mobile device driving program, and be configured to control the driving program functions implemented by the vehicle by selectively implementing the driving program of the driving support terminal, the mobile device driving program and the vehicle driving program under the control of the driving program of the driving support terminal based on any one or any combination of the first determination, the second determination and the third determination, so as to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle.

[0014] The driving support system may also include a mobile device having one or more processors and a memory storing a mobile device driving program including a mobile device driving assistance program and / or a mobile device autonomous driving program.

[0015] When the driving program of the driving support terminal, the mobile device driving program and the vehicle driving program are selectively implemented based on any one or any combination of the first determination, the second determination and the third determination, it may include selecting between corresponding functions of the driving program of the driving support terminal and the mobile device driving program, wherein one or more processors of the driving support terminal may be configured to select a function with a predetermined greater function from these corresponding functions.

[0016] One or more processors of the driving support terminal may also be configured to determine which of the driving assistance program of the driving program of the driving support terminal, the mobile device driving assistance program of the driving program of the mobile device communicating with the driving support terminal, and the vehicle driving assistance program of the vehicle driving program is the ADAS highest level driving assistance program, and / or determine which of the autonomous driving program of the driving program of the driving support terminal, the mobile device autonomous driving program in the driving program of the mobile device, and the autonomous driving program in the vehicle driving program is the ADAS highest level autonomous driving program, and be configured to execute the ADAS highest level driving assistance program and / or the ADAS highest level autonomous driving program to control the vehicle for the one or more assisted driving and / or autonomous operations of the vehicle.

[0017] The driving support system may also include a mobile device having one or more processors and a memory storing a mobile device driving program including a mobile device driving assistance program and a mobile device autonomous driving program.

[0018] In order to generate the control signaling, the driving assistance program may be configured to generate driving support control signaling and driving support information, wherein the driving support control signaling is a signaling for assisting in controlling the assist driving operation of the vehicle to display the driving support information.

[0019] The driving support information of the vehicle may include information indicating at least one of a steering operation, a braking operation, and an acceleration operation of a driver controlling the vehicle.

[0020] The operations instructing the driver may include those determined by the driving program: obstacle detection warnings and / or instructing the driver to perform corresponding obstacle avoidance steering and / or braking; providing rear collision warnings and / or instructing the driver to perform corresponding rear collision avoidance braking; providing lane departure warnings, instructing the driver to perform corresponding lane departure remedies or maintain steering, and / or instructing the driver to suggest a lane change.

[0021] In order to generate the control signaling, the driving assistance program may be configured to generate a driving support control signaling, wherein the driving support control signaling may be a signaling for controlling the assisted driving operation of the vehicle to assist in controlling at least one of the steering operation, braking operation and acceleration operation of the vehicle.

[0022] The signaling for controlling the assisted driving operation of the vehicle to assist in controlling at least one of the steering operation, braking operation and acceleration operation of the vehicle may include: controlling at least one of the steering operation, braking operation and acceleration operation of the vehicle for detected obstacle assisted avoidance, detected rear collision assisted avoidance, predicted and / or detected lane departure assisted prevention or remediation, and / or assisted lane change determined by the driving program.

[0023] The driving support control signaling may be a signaling for controlling the assisted driving operation of the vehicle to display driving support information generated by the driving assistance program, the driving support information indicating information about at least one of a steering operation, a braking operation, and an acceleration operation of the assisted control of the vehicle.

[0024] The autonomous driving program may be configured to generate driving control signaling, wherein the driving control signaling may be signaling for controlling autonomous driving operations of the vehicle to control at least one of a steering operation, a braking operation, and an acceleration operation of the vehicle without auxiliary control by a driver.

[0025] The operation of generating the driving control signaling may include: performing any one or any combination of two or more of occluded object prediction, lane change determination, pedestrian recognition, road change recognition and road sign recognition; and generating the driving control signaling based on the results of the execution.

[0026] The one or more processors of the driving support terminal may include one or more artificial intelligence processors, and the one or more artificial intelligence processors are configured to: load one or more artificial intelligence models from the memory of the driving support system and implement the one or more artificial intelligence models to perform any one or any combination of two or more of the obscured object prediction, the lane change determination, the pedestrian recognition, the road change recognition and the road sign recognition, and generate the driving control signaling based on the results of the execution.

[0027] The one or more artificial intelligence processors may include one or more neuromorphic processors, and the one or more artificial intelligence models may be respective neural network models.

[0028] The one or more processors may be configured to perform post-processing on the obtained image information and / or other image information obtained by the driving support terminal model, and the one or more processors may include one or more artificial intelligence processors, wherein the execution of at least a portion of the driving program may include executing one or more neural network models for providing post-processed obtained image information and / or post-processed other image information.

[0029] The one or more processors may be configured to perform post-processing on the obtained image information and / or other image information obtained by the driving support terminal model, and the one or more processors may include one or more artificial intelligence processors, wherein the one or more processors may be configured to implement a navigation program by executing one or more neural network models using the post-processed obtained image information and / or the post-processed other image information, wherein the one or more neural network models are executed using the one or more artificial intelligence processors that provide the post-processed obtained image information and / or the post-processed other image information.

[0030] The one or more processors may be configured to connect to the communication port to obtain the driving information, and may be configured to obtain other image information from another camera separated from the vehicle and communicating with the driving support terminal, and may execute at least a portion of the driving program based on the obtained driving information and the other image information.

[0031] The one or more processors may be configured to generate an image of a periphery of the vehicle from the other image information and to provide the image of the periphery of the vehicle to the vehicle for display by the vehicle.

[0032] The driving support terminal may include an OBD type connection terminal, and the driving support terminal is connected to the communication port using the OBD type connection terminal.

[0033] The image information may be obtained from the vehicle, and / or other image information may be obtained from a camera separate from the vehicle, and the one or more processors may be further configured to use the image information and / or the other image information and the driving information to implement at least a portion of a navigation program.

[0034] The driving support terminal may be connected to the communication port using a Type-C USB connection terminal.

[0035] The driving support terminal may include: a memory storing the driving assistance program and / or the autonomous driving program; and the one or more processors may include a neural processing unit generating driving support information by executing at least a portion of the driving program.

[0036] The neural processing unit may include: an image receiver, into which an image obtained from the camera system is input; an image preprocessor, configured to extract a region of an image of interest from the image input to the image receiver; a target recognition neural network model, implemented to recognize a target in the image of interest; and a driving support information generator, configured to execute at least a portion of the driving assistance program and / or the autonomous driving program for the recognized target.

[0037] The driving support system may also include a mobile device having one or more processors, the one or more processors being configured to execute at least a portion of a mobile device driving program (including a mobile device driving assistance program and / or a mobile device autonomous driving program) based on the image information and / or the driving information to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle, wherein the one or more processors of the mobile device or the one or more processors of the driving support terminal may also be configured to determine which of the driving assistance programs of the driving support terminal and the mobile device driving assistance program is the ADAS highest level driving assistance program, and / or determine which of the driving program of the driving support terminal and the mobile device autonomous driving program is the ADAS highest level autonomous driving program, and be configured to execute the ADAS highest level driving assistance program and / or the ADAS highest level autonomous driving program to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle.

[0038] The driving support system may also include a mobile device having one or more processors, wherein the one or more processors are configured to execute at least a portion of a mobile device driving program (including a mobile device driving assistance program and / or a mobile device autonomous driving program) based on the image information and / or the driving information to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle, and the one or more processors of the mobile device or the one or more processors of the driving support terminal may also be configured to perform a first determination to determine which of the functions of the driving program of the driving support terminal and the functions of the mobile device driving program have higher functionality. energy level, performs a second determination to determine which functions of the driving program of the driving support terminal are not provided by the mobile device driving program, and / or performs a third determination to determine which functions of the driving program of the mobile device are not provided by the driving program of the driving support terminal, and is configured to control the implemented driving program functions of the vehicle by selectively implementing the driving program of the driving support terminal and the mobile device driving program based on any one or any combination of the first determination, the second determination and the third determination under the control of the mobile device or the driving support terminal, so as to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle.

[0039] In one general aspect, a driving support method for interacting with a vehicle, the vehicle comprising a camera system that generates image information of the vehicle's surroundings, a vehicle controller that generates driving information of the vehicle, and a communication port, the driving support method comprising: connecting to the communication port via a device separate from the vehicle to obtain the image information and / or the driving information using the communication port; executing at least a portion of a device driving program based on the image information and / or the driving information by the device, the device driving program comprising a driving assistance program and / or an autonomous driving program of a driving support terminal or a mobile device used as the device, respectively, and controlling the vehicle for one or more assisted driving and / or autonomous operations of the vehicle through control signals generated by the driving support terminal or the mobile device; and providing the generated control signals to the vehicle to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle.

[0040] The method may also include: determining which of the driving assistance program that implements the device driving program and any vehicle driving assistance program configured as a driving program implemented by the vehicle when the control signaling is generated by the driving support terminal or the mobile device, and / or determining which of the autonomous driving program that implements the device driving program and any vehicle autonomous driving program configured as a driving program implemented by the vehicle when the control signaling is generated by the driving support terminal or the mobile device.

[0041] The method may further include determining which respective functions of the device driving program and any vehicle driving program configured to be implemented by the vehicle are implemented for use in generating the control signaling by the driving support terminal or the mobile device.

[0042] In one general aspect, a driving support system for interacting with an individual vehicle includes a camera system that generates image information of the vehicle's surroundings, a vehicle controller that generates driving information of the vehicle, and a communication port, the driving support system comprising: a smart phone, the smart phone including one or more processors, the one or more processors being configured to execute at least a portion of a mobile device driving program based on the image information and / or the driving information, the mobile device driving program including a mobile device driving assistance program and / or a mobile device autonomous driving program and stored in a memory included in the smart phone, and to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle through control signaling generated by the smart phone and provided to the vehicle.

[0043] The vehicle controller may be configured to execute a vehicle driving program for controlling the vehicle, the vehicle driving program including a vehicle driving assistance program and / or a vehicle autonomous driving program stored in a memory of the vehicle.

[0044] The one or more processors of the smart phone may also be configured to determine which of the mobile device driving assistance program and the vehicle driving assistance program is the ADAS highest level driving assistance program, and / or determine which of the mobile device autonomous driving program and the vehicle autonomous driving program is the ADAS highest level autonomous driving program, and the one or more processors are configured to execute the determined ADAS highest level driving assistance program and / or the determined ADAS highest level autonomous driving program to control the vehicle for the one or more assisted driving and / or autonomous operations of the vehicle.

[0045] The one or more processors of the smart phone may also be configured to: perform a first determination to determine which functions of the mobile device driving program have a higher functional level than corresponding functions of the vehicle driving program; perform a second determination to determine which functions of the mobile device driving program are not provided by the vehicle driving program; and / or perform a third determination to determine which functions of the vehicle driving program are not provided by the mobile device driving program, and the one or more processors are configured to control the driving program functions implemented by the vehicle by selectively implementing the mobile device driving program and / or the vehicle driving program under the control of the smart phone based on any one or any combination of the first determination, the second determination and the third determination, so as to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle.

[0046] In order to generate the control signaling, the mobile device driving assistance program may be configured to generate driving support information to control the assisted driving operation of the vehicle to display the driving support information.

[0047] The driving support information of the vehicle may include information indicating at least one of a steering operation, a braking operation, and an acceleration operation of a driver controlling the vehicle.

[0048] The operations instructing the driver may include those determined by the mobile device driving program: obstacle detection warnings and / or instructing the driver to perform corresponding obstacle avoidance steering and / or braking; providing rear collision warnings, instructing the driver to perform corresponding rear collision avoidance braking; providing lane departure warnings, instructing the driver to perform corresponding lane departure remedies or maintain steering; and / or instructing the driver to suggest a lane change.

[0049] In order to generate the control signaling by the smart phone, the mobile device driving assistance program can be configured to generate a driving support control signaling, wherein the driving support control signaling is a signaling for controlling the assisted driving operation of the vehicle to assist in controlling at least one of the steering operation, braking operation and acceleration operation of the vehicle.

[0050] Signaling for controlling the assisted driving operation of the vehicle to control at least one of the steering operation, braking operation and acceleration operation of the vehicle may include: assisting in controlling at least one of the steering operation, braking operation and acceleration operation of the vehicle for detected obstacle assisted avoidance, detected rear collision assisted avoidance, predicted and / or detected lane departure assisted prevention or remediation and / or assisted lane change determined by the mobile device driving program.

[0051] The driving support control signaling may be a signaling for controlling the assisted driving operation of the vehicle to display driving support information generated by the mobile device driving assistance program, the driving support information indicating information about at least one of the steering operation, braking operation and acceleration operation of the assisted control of the vehicle.

[0052] In order to generate the control signaling by the smart phone, the mobile device autonomous driving program can be configured to generate driving control signaling, wherein the driving control signaling can be a signaling for controlling the autonomous driving operation of the vehicle to control at least one of the steering operation, braking operation and acceleration operation of the vehicle without auxiliary control of the driver.

[0053] The operation of generating the driving control signaling may include performing any one or any combination of two or more of blocked object prediction, lane change determination, pedestrian recognition, road change recognition, and road sign recognition, and generating the driving control signaling based on the results of the execution.

[0054] The one or more processors of the smart phone may include one or more artificial intelligence processors, and the one or more artificial intelligence processors are configured to load one or more artificial intelligence models from the memory of the smart phone and implement the one or more artificial intelligence models to perform any one or any combination of two or more of the occluded object prediction, the lane change determination, the pedestrian recognition, the road change recognition, and the road sign recognition, and generate the driving control signaling based on the results of the execution.

[0055] The one or more artificial intelligence processors may include one or more neuromorphic processors, and the one or more artificial intelligence models may be respective neural network models.

[0056] The one or more processors may be configured to connect to the communication port to obtain the driving information and obtain other image information from at least one camera separate from the vehicle and in communication with the smart phone, and the one or more processors may be configured to execute at least a portion of the mobile device driving program based at least on the obtained driving information and the other image information.

[0057] The at least one camera may include a camera included in the smartphone.

[0058] The one or more processors may be configured to generate an image of a periphery of the vehicle from the other image information and to provide the generated image to the vehicle for display by the vehicle.

[0059] In one general aspect, a driving support system for interacting with a separate vehicle, the vehicle including a camera system that generates image information of the vehicle's surroundings, a vehicle controller that generates driving information of the vehicle, and a communication port, the driving support system including: a mobile device having one or more processors, the one or more processors being configured to execute at least a portion of a mobile device driving program (including a mobile device driving assistance program and / or a mobile device autonomous driving program and stored in a memory of the mobile device) based on the image information and / or the driving information to perform one or more assisted driving and / or automatic driving operations on the vehicle through control signaling generated by the mobile device; and a driving support terminal configured to be connected to the communication port and including communication hardware, the communication hardware being configured to obtain the image information and / or the driving information using the communication port, transmit the obtained image information and / or the driving information to the mobile device, and transmit the generated control signaling to the vehicle, wherein the mobile device is remotely connected to the driving support terminal and is separated from the driving support terminal, and includes computing functions different from providing driving support.

[0060] The vehicle controller may be configured to execute a vehicle driving program stored in a memory of the vehicle for controlling the vehicle, the vehicle driving program including a vehicle driving assistance program and / or a vehicle autonomous driving program.

[0061] One or more processors of the mobile device may also be configured to determine which of the mobile device driving assistance program and the vehicle driving assistance program is the ADAS highest level driving assistance program, and / or determine which of the mobile device autonomous driving program and the vehicle autonomous driving program is the ADAS highest level autonomous driving program, and be configured to execute the determined ADAS highest level driving assistance program and / or the determined ADAS highest level autonomous driving program to control the vehicle for the one or more assisted driving and / or autonomous operations of the vehicle.

[0062] One or more processors of the mobile device may also be configured to perform a first determination to determine which functions of the mobile device driver have a higher function level than corresponding functions of the vehicle driver, perform a second determination to determine which functions of the mobile device driver are not provided by the vehicle driver, and / or perform a third determination to determine which functions of the vehicle driver are not provided by the mobile device driver, and be configured to control (through control of the mobile device) the implementation of the driving program functions of the vehicle by selectively implementing the mobile device driving program and the vehicle driving program based on any one or any combination of the first determination, the second determination and the third determination to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle.

[0063] The mobile device may be a smartphone.

[0064] The driving support terminal may further include a wireless power transmission module and a physical support, wherein the wireless power transmission module may be configured within the physical support to provide wireless power transmission to a mobile device, and the mobile device may also have a mobile wireless power transmission module configured to at least receive wirelessly transmitted power.

[0065] The driving support terminal may include one or more processors, which are configured to execute at least a portion of a driving support terminal driving program based on images and / or driving information to control the vehicle for one or more assisted driving and / or autonomous operations of the vehicle, wherein the driving support terminal driving program includes a driving support terminal driving assistance program and / or a driving support terminal autonomous driving program.

[0066] The driving support terminal may also be configured to receive control signaling from the mobile device and provide the control signaling to the communication port via the vehicle's controller area network (CAN) to provide the control signaling to the vehicle controller for controlling the vehicle for one or more assisted driving and / or autonomous operations of the vehicle.

[0067] In order to generate the control signaling, the mobile device driving assistance program may be configured to generate driving support information to assist in controlling the assisted driving operation of the vehicle to display the driving support information.

[0068] The driving support information of the vehicle may include information indicating that a driver controls at least one of steering, braking, and acceleration operations of the vehicle.

[0069] The instructing the driver may include, as determined by the driving program: obstacle detection warnings and / or instructing the driver to perform corresponding obstacle avoidance steering and / or braking, providing rear collision warnings, instructing the driver to perform corresponding rear obstacle avoidance braking, providing lane departure warnings, instructing the driver to perform corresponding lane departure remedies or maintain steering, and / or instructing the driver to suggest a lane change.

[0070] In order to generate control signaling by the mobile device, the mobile device driving assistance program can be configured to generate driving support control signaling, wherein the driving support control signaling can be a signaling for controlling the assisted driving operation of the vehicle to assistively control at least one of the steering, braking and acceleration operations of the vehicle.

[0071] Signaling for controlling an assisted driving operation of the vehicle to control at least one of the vehicle's steering, braking, and acceleration operations may include, determined by a driving program, assisting in controlling at least one of the vehicle's steering, braking, and acceleration operations for detected obstacle avoidance assistance, detected rear collision avoidance assistance, predicted and / or detected lane departure assistance prevention or remediation, and / or assisted lane change assistance.

[0072] The driving support control signaling may be a signaling for controlling an assisted driving operation of a vehicle to display driving support information generated by a mobile device driving assistance program, the driving support information indicating information on at least one of steering, braking, and acceleration operations of assisted control of the vehicle.

[0073] In order to generate control signaling by the mobile device, the mobile device autonomous driving program can be configured to generate driving control signaling, wherein the driving control signaling is a signaling for controlling the autonomous driving operation of the vehicle to control at least one of the steering, braking and acceleration operations of the vehicle without auxiliary control of the driver.

[0074] The generating of the driving control signaling may include performing any one or any combination of two or more of occluded object prediction, lane change determination, pedestrian recognition, road change recognition, and road sign recognition, and generating the driving control signaling based on the results of the performing.

[0075] The one or more processors of the mobile device may include one or more artificial intelligence processors configured to load one or more artificial intelligence models from the memory of the mobile device and implement one or more artificial intelligence models to perform any one or any combination of two or more of occluded object prediction, lane change determination, pedestrian recognition, road change recognition, road sign recognition, and generate driving control signaling based on the results of the execution.

[0076] The one or more artificial intelligence processors may include one or more neuromorphic processors, and the one or more artificial intelligence models may be corresponding neural network models.

[0077] One or more processors may be configured to generate mobile device image information including an image of the vehicle's surroundings based on image information obtained by the mobile device or other image information, and to execute at least a portion of a mobile device driving program based on the generated mobile device image information and / or to execute a navigation program based on the generated mobile device image information.

[0078] Other information may be obtained from a camera included in the mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 is a block diagram illustrating a driving support system according to one or more embodiments.

[0080] Figure 2 is a block diagram illustrating a driving support system according to one or more embodiments.

[0081] Figure 3 is a block diagram illustrating a driving support system according to one or more embodiments.

[0082] Figure 4 and Figure 5 is a block diagram illustrating a driving support system according to one or more embodiments.

[0083] Figure 6 is a block diagram illustrating a driving support system according to one or more embodiments.

[0084] Figure 7 is a block diagram illustrating a driving support terminal according to one or more embodiments.

[0085] Figure 8 is a block diagram illustrating a driving support terminal according to one or more embodiments.

[0086] Fig. 9 is a diagram illustrating a connection relationship between a driving support terminal and a controller of a vehicle according to one or more embodiments.

[0087] Fig.10 is a perspective view illustrating a driving support terminal according to one or more embodiments.

[0088] Fig.11 is a block diagram illustrating an implementation of a driving program using an NPU according to one or more embodiments.

[0089] Fig.122 are a sectional view and a plan view respectively showing a driving support terminal according to one or more embodiments.

[0090] Fig.13 is a block diagram illustrating a driving support system according to one or more embodiments.

[0091] Fig.14 and Fig.15 is a diagram illustrating a calibration operation performed by the image processor of the present disclosure.

[0092] Fig.16 and Fig.17 is a diagram illustrating an operation of controlling a camera unit performed by an artificial intelligence processor of the present disclosure.

[0093] Throughout the drawings and detailed description, the same reference numerals denote the same elements. For clarity, illustration, and convenience, the drawings may not necessarily be drawn to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. DETAILED DESCRIPTION

[0094] The following detailed description is provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the method, device and / or system described herein will be apparent. For example, except for the operations that must occur in a specific order, the order of operations described herein is merely an example and is not limited to the order of operations set forth herein, but can be changed obviously after understanding the disclosure of the present application. In addition, in order to increase clarity and brevity, the description of features known in the art may be omitted.

[0095] The features described herein can be implemented in different forms and will not be interpreted as being limited to the examples described herein. More specifically, the examples described herein have been provided only to illustrate some of the many possible ways of implementing the methods, devices and / or systems described herein that will be apparent after understanding the disclosure of the present application. In addition, it will be understood that, although various examples of this article are described by different diagrams, the various examples of this article are not mutually exclusive. For example, without departing from the spirit and scope of the present disclosure, the structures, shapes and sizes described about these examples can be implemented in any other examples. In addition, without departing from the spirit and scope of the present disclosure, examples include examples in which the position or arrangement of elements are variously modified.

[0096] In addition, although the terms "first" or "second" may be used to describe various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another. For example, within the scope of the concept of the present disclosure, a "first" component may be referred to as a "second" component, or similarly, a "second" component may be referred to as a "first" component.

[0097] As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form.For example, as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form.It should also be understood that, unless the context and the understanding of the present disclosure indicate otherwise, when the terms "comprise", "include" and / or "have" are used in this specification, the stated features, integers, operations, elements, components and / or their combinations / groupings are listed in the example embodiments, but it is not excluded that one or more other features, integers, operations, elements, components and / or their combinations / groupings are present or added in alternative embodiments, nor is it excluded that these stated features, integers, operations, elements, components and / or their combinations / groupings are lacking in other alternative embodiments.In addition, the use of the term "may" herein about examples or embodiments (for example, what can be included or what is realized about examples or embodiments) means that there is at least one example or embodiment including or realizing such features, but all examples and embodiments are not limited thereto.

[0098] Unless otherwise defined herein, all terms (including technical or scientific terms) used herein have the same meaning as those generally understood by those of ordinary skill in the art after understanding the present disclosure. Unless otherwise defined herein, terms defined in commonly used dictionaries should be interpreted as having a meaning that matches the contextual meaning in the art and the present disclosure, and should not be interpreted according to an idealized or overly formal meaning.

[0099] Figure 1 is a block diagram illustrating a driving support system according to one or more embodiments.

[0100] Reference Figure 1 , the driving support system may include a camera unit 110 , a vehicle control unit 120 , a telematics control unit 130 , a communication port 140 , and a driving support terminal 200 .

[0101] The camera unit 110, the vehicle control unit 120, the telematics control unit 130, and the communication port 140 may be components in the vehicle 100 (e.g., components installed during the manufacture of the vehicle 100). The driving support terminal 200 may be, for example, a hardware component that is selectively connected to the vehicle later, for example, the driving support terminal 200 may provide a connection or a connection and physical support for an additional device (e.g., a connection and / or physical support of an example of a mobile device 300 discussed below), so that the additional device can communicate with the driving support terminal 200 and / or the vehicle control unit 120. Here, the vehicle control unit 120, the telematics control unit 130, the communication port 140, and the driving support terminal 200 are all hardware devices or components, for example, each of which may be implemented as hardware only, hardware configured by executing instructions based on calculations (e.g., one or more processors), or a combination of hardware and such hardware configured by executing instructions based on calculations, for example, as further explained below, wherein such instructions may be instructions that, when executed by such one or more processors, configure one or more processors to implement any one, any combination, or all of the operations or methods of the various devices or components described herein. In addition, although the devices or components are individually identified, there are examples in which their respective functions are implemented together by a single device or component, or examples in which the respective functions are differently combined in any combination in two or more alternative devices or components.

[0102] The camera unit 110 may be mounted on the vehicle and may image an external area around the vehicle. For example, the camera unit 110 may include one or more cameras and may generate an image of the vehicle's surroundings. For example, the camera unit 110 may include four cameras and may image the front area, the side area, and the rear area of ​​the vehicle and may generate an image of the vehicle's surroundings. The camera unit 110 may provide the generated image to the communication port 140. In addition, the camera unit 110 may also be represented as one or more cameras and one or more processors, which may be configured to perform pre-processing on the captured image, such as reducing noise, standardizing the captured image, or otherwise performing image processing on the captured image or image information so that it can be used by, for example, the communication port 140 or the vehicle control unit 120 or can be transmitted to the communication port 140 or the vehicle control unit 120, and, for example, in processing the captured image or image information into an image for input to the driving support terminal 200 (and / or Figures 4 to 6In an example of the form or format of a model implemented by one or more artificial intelligence (AI) components or processors of the mobile device 300), for example, a neuromorphic processing unit / processor included in the driving support terminal 200 (and / or the mobile device 300) or one or more processors (e.g., CPU and / or GPU) represented by the driving support terminal 200 (or a processor represented by the mobile device 300) is configured to implement such an artificial intelligence model, such as a trained neural network system, which, as a non-limiting example, may be trained to extract features or recognize objects or targets from provided image information, such as by a series of convolutional layers followed by a series of feed-forward layers, and may be further configured to generate a control signal output based on the extracted features, such as by further training layers of the neural network system. In other examples, such as the following for Figures 2 to 3 As discussed, one or more of such image pre-processing operations may also or alternatively be performed by the data processing module 150 .

[0103] Therefore, the vehicle control unit 120 may include an electronic control unit 121 and a body control module 122 , and may control overall operations of the vehicle.

[0104] The electronic control unit 121 may generate driving information (including travel information and operation information of the vehicle), and may control the engine of the vehicle according to the generated travel information and operation information.

[0105] The vehicle may include one or more sensors that sense the driving state and operating state of the vehicle, such as speed sensors, steering sensors, brake sensors, acceleration sensors, roll sensors, various drive component position or value indication sensors, and temperature sensors (which may generate temperature information considered by the electronic control unit 121 to adjust acceleration and / or braking in near or below freezing conditions) or moisture or humidity sensors (which generate corresponding information considered by the electronic control unit 121 to similarly adjust acceleration and / or braking in a determined wet or slippery road environment). The electronic control unit 121 may generate driving information and operating information of the vehicle through these example sensed value outputs of one or more sensors. The electronic control unit 121 may include multiple management modules (i.e., processors or other computing hardware), and sensed values ​​output from sensors of the same type may be managed through an example single management module. For example, the multiple cameras of the camera unit 110 may be sensors for imaging the surroundings of the vehicle, in which case the electronic control unit 121 may manage the images of the multiple cameras of the camera unit 110 through a first management module, while information or signals from environmental sensors (such as a temperature sensor or a humidity sensor) may be managed by a second management module, wherein each management module is configured to process the sensor information received by each and to perform various processing on the sensor data as described above to provide to various other components of the vehicle (such as, to an instrument cluster to provide information notifications only to the driver) and / or to the above-mentioned artificial intelligence components and / or models implemented by, for example, the electronic control unit 121.

[0106] The electronic control unit 121 may provide the generated driving information and operating information and other information about control signaling (which the electronic control unit 121 may be executing or that the electronic control unit 121 is configured to execute to control the vehicle) to the communication port 140, including, as non-limiting examples, any corresponding control signaling for (for) information, driving assistance information and control, and / or other autonomous driving control. Thus, the electronic control unit 121 may make available or provide (and / or be requested by the driving support terminal 200 and / or the mobile device 300) current information of the driving program included in the vehicle 100, information of auxiliary and / or autonomous features or functions (e.g., information originally embedded in the electronic control unit 121 at the time of manufacture) for comparison, such as by the driving support terminal 200 and / or the mobile device 300, with information of the driving program included in the vehicle 100, information of auxiliary and / or autonomous features or functions, and either or both of the driving support terminal 200 and / or the mobile device 300 may determine whether to supplement or replace the existing driving program of the electronic control unit 121, for example, by utilizing the driving program implemented by the driving support terminal 200 and / or the mobile device 300 or a selected function thereof.

[0107] For example, since different vehicles have different relative positions, heights, and fields of view (FOV) of cameras installed during vehicle manufacturing, in order to allow the driving program of the driving support terminal 200 and / or the mobile device 300 to appropriately record the received image information for information, auxiliary, and autonomous driving functions, program information or other information may similarly make the communication port 140 available or provide the communication port 140 with positioning or alignment recording information for reporting the various positions of the camera or camera system of the camera unit 110 and / or the configuration or performance of the camera or camera system of the camera unit 110, and / or their respective fields of view (FOV). The driving support terminal 200 and / or the mobile device 300 may also be configured to communicate with a remote server to obtain such various vehicle-specific information so as to appropriately anticipate and use the information provided by the various cameras of the camera unit 110 of the vehicle 100. For example, the driving support terminal 200 and / or the mobile device 300 may also consider additional variables, such as sensed tire pressure or user-indicated non-OEM tire brand and model and size, or other post-manufacturing suspension modifications or other change information input by the user or detectable by the driving support terminal 200 and / or the mobile device 300 using information from the camera unit 110. In an example including the mobile device 300 and the driving support terminal 200, the driving support terminal 200 and / or the mobile device 300 may also have a predetermined relative positional relationship, for example based on the physical support configuration of the driving support terminal 200 and the positioning of the mobile device 300 in contact with the driving support terminal 200. Here, the user may input the manufacturer and model of the mobile device 300 in the user interface of the driving support terminal 200 and / or the mobile device 300, or the driving support terminal 200 may only request the manufacturer and model from the mobile device 300, or the mobile device 300 may similarly self-identify the manufacturer and model to record the position of one or more cameras of the mobile device 300 as additional cameras for consideration by the driving support terminal 200 and / or the driving program of the mobile device 300. Information of any housing or shell of the mobile device 300 may also be determined or similarly input by the user. In such an example, as a non-limiting example, the driving support terminal 200 may determine its position relative to the vehicle 100 based on information received from one or more cameras of the mobile device 300 and information from the camera unit 110, and / or the user may input such positioning information into the user interface of the driving support terminal 200 and / or the mobile device 300 for recording the position of the camera of the mobile device 300 (or any alternative or additional supplementary camera) relative to the vehicle 100 and the positions of other cameras of the camera unit 110.

[0108] The communication port 140 may include an on-board diagnostic (OBD) port. The OBD port may be arranged in or near the console of the vehicle and may be accessed by a passenger or driver to connect to an external device (e.g., a typical code reader used by a mechanic to diagnose mechanical or electrical faults or problems in various components of the vehicle). The communication port 140 may communicate with the external device connected to the communication port 140 and the components of the vehicle through a controller area network (CAN) communication, or may provide a network interface between the external device connected to the communication port 140 and the corresponding CAN network of the components of the vehicle shown to be connected, so that the external device can use the CAN network to directly receive information from these components and / or send information to these components. In an example, the CAN network can provide multiple serial communications using the same communication channel, so that multiple lines or communication channels previously required between components in the vehicle for providing or transmitting different information may no longer be necessary because different information can be provided using the same communication channel of the example. As an example, the communication port 140 can be connected to all of the camera unit 110, the electronic control unit 121, and the telematics control unit 130 through the CAN network.

[0109] The body control module 122 may control elements included in the body of the vehicle. For example, the body control module 122 may control wipers, lights, electric seats, sunroofs, air conditioning systems, and the like.

[0110] The telematics control unit 130 may include a wireless communication module and may perform wireless communication with one or more external devices present outside the vehicle. For example, as a non-limiting example, the telematics control unit 130 may be configured to perform at least one of cellular communication, Wi-Fi communication, Bluetooth communication, or other local, near-field, or far-field communication. The telematics control unit 130 may receive external environmental information of the vehicle via cellular communication, and may provide the received external environmental information to the driving support terminal 200 via Wi-Fi communication, or the driving support terminal 200 (or the mobile device 300 in the example) may request and receive the received external environmental information. The telematics control unit 130 may also be configured to provide an alternative communication channel (e.g., an alternative OBD port) to the communication port 140, so that the driving support terminal 200 (and / or the mobile device 300) may interact with the electronic control unit 121 or other components of the vehicle via the CAN network.

[0111] Accordingly, the driving support terminal 200 can be connected to the communication port 140, and can obtain the driving information and operation information of the vehicle generated in the electronic control unit 121 and the images captured by the camera unit 110 or the image information results of the above-mentioned pre-processing of these captured images. The driving support terminal 200 can also serve as a communication intermediary or translation / conversion between the above-mentioned mobile device 300 and the communication port 140.

[0112] The driving support terminal 200 may include a first connection terminal (Type_OBD) configured to be paired with the communication port 140. As an example, the first connection terminal (Type_OBD) may include an OBD type connection terminal.

[0113] The driving support terminal 200 may also or alternatively receive external environment information from the telematics control unit 130, and access the CAN network through the communication channel between the telematics control unit 130 and the communication port 140. For example, the driving support terminal 200 and the telematics control unit 130 may receive data from each other and send data to each other through their respective Wi-Fi communication modules. In addition, the driving support terminal 200 may also include, for example, a communication module that provides cellular or other wide area network radio communications, and request external environment information from a remote server, or in the following example, request external environment information from an example mobile device 300, wherein the mobile device 300 includes such a communication module. In this article, the term module refers to a hardware component. As another example, the functions of the telematics control unit 130 may be implemented by the corresponding hardware of the communication port 140, and the driving support terminal 200 may be directly connected to the communication port 140 through such WiFi communication.

[0114] In the examples of this article, the driving support terminal 200 can be connected to the communication port 140, and can obtain the driving information and operation information of the vehicle, and / or can obtain the image information generated in the camera unit 110 or the corresponding pre-processed image information, and it is noted that the examples are not limited to this. In any of the various examples of this article, a Wi-Fi communication module can be set in the camera unit 110 (for example, in the corresponding management module or in each camera) and the communication port 140. In this case, the driving support terminal 200 can perform wireless communication with the camera unit 110 and the communication port 140, and can obtain the driving information and operation information of the vehicle and such various image information. As described above, the camera unit 110 is represented as one or more cameras and / or camera systems, and it is noted that, as a non-limiting example, the driving support system may also include, in some examples, one or more cameras of the mobile device 300 of the above example, which may be additionally supported by the vehicle 100, selectively mounted to the vehicle 100, or otherwise mounted in or on the vehicle 100 and wiredly connected to the driving support terminal 200 or the mobile device 300 or wirelessly connected to the driving support terminal 200 and / or the mobile device 300. For example, as a non-limiting example, such an additional camera may be connected to the driving support terminal 200 or the mobile device 300 using a USB cable and connector of the driving support terminal 200 or the mobile device 300 or through a proprietary connector of the mobile device 300.

[0115] The driving support terminal 200 is configured to store at least one driving program. In various examples, the driving support terminal 200 includes driving programs that provide different levels of information, driving assistance and / or autonomy, and / or programs that provide different information, assistance and / or autonomous functions within the same level of autonomy, and the driving support terminal 200 may differently consider various available image information obtained and / or exported from the camera unit 110, the corresponding management module, the electronic control unit 121, or otherwise available through the CAN network (for example, accessed through the communication port 140), as well as other above-mentioned example sensor information, driving information and operation information of the vehicle obtained from the CAN network (for example, through the communication port 140), and the above-mentioned external environment information that can be obtained from the telematics control unit 130 to generate various driving support information and / or driving control signaling for example levels of autonomous driving control. The driving support terminal 200 may therefore provide / send such information and / or driving control signaling to the communication port 140, which may be connected to a CAN network interface to forward various different information to the vehicle's central information display unit, instrument cluster, and / or provide information to the driver via a sound generator for auditory indications, and may similarly be connected to a CAN network interface to forward various driving control signaling to the electronic control unit 121 or corresponding controlled components, such as by sending a brake control signaling to an electronic brake controller to activate (or reduce application) the brakes according to a controlled intensity, sending an accelerator control signaling to an electronic accelerator controller to control the accelerator or otherwise control the throttle, and sending a steering control signaling to an electronic steering controller or electric power steering (EPS) to control the degree and / or speed of the change in steering or maintain the current steering control. In the electric vehicle example, the driving control signaling may be transmitted to separate controllers for different wheel motors, brakes, and steering systems. Moreover, in such an electric vehicle example, a corresponding stored driving program may perform such informational, auxiliary and / or other autonomous control functions based on additional information from one or more battery modules to, for example, maximize battery life or charge of one or more battery modules.Alternatively or additionally, the mobile device 300 discussed below may similarly store such a driving program and generate such information and / or driving control signaling, and provide / send such information and / or driving control signaling to the communication port 140 or the telematics control unit 130 directly or by using the driving support terminal 200 as an intermediary or converter with the vehicle, for example, wherein the mobile device 300 provides / sends an encoding of instructions to the driving support terminal 200 to generate corresponding information and / or control signaling, or the mobile device 300 generates such information and / or control signaling and provides / sends such information and / or control signaling to the driving support terminal 200, and the driving support terminal 200 provides / sends such information and / or control signaling to the communication port 140. For example, in an example where the mobile device 300 provides the driving support terminal 200 with the encoding of the instruction to generate information and / or control signaling, the driving support terminal 200 receives the encoding of the instruction (e.g., in the communication protocol between the driving support terminal 200 and the mobile device 300), and generates the information and / or control signaling in a form and format compatible with the CAN communication protocol of the CAN network, and provides / sends the information and / or control signaling to the communication port 140. As a non-limiting example, the mobile device 300 may alternatively generate and forward the information and / or control signaling in the communication protocol between the driving support terminal 200 and the mobile device, and the driving support terminal 200 only converts the information and / or control signaling into a form and format compatible with the CAN network, and provides / sends the information and / or control signaling to the communication port 140.

[0116] Therefore, the driving support terminal 200 can provide various advanced driving support functions to the vehicle, for example, according to a driving program stored in the driving support terminal 200 (and / or stored in the mobile device 300), thereby providing, for example, an advanced driver assistance system (ADAS).

[0117] The driving support terminal 200 in any example herein may provide such advanced driving support functions at a predetermined level specified in the ADAS standard. For example, according to the 0 to 5 levels of SAEJ3016 defined by SAE International, which was originally established as the Society of Automotive Engineers, during SAE 0 to 2 levels, whenever multiple driver support features are implemented or used, even when the driver's foot is off the pedal and the driver is not steering, the driver will drive, because the driver must continuously supervise the support features and must steer, brake or accelerate as needed to maintain safety, in contrast, under SAE 3-5 levels, when the driver support features are implemented or used, the driver does not drive even if the driver is sitting in the "driver's seat", wherein, under SAE 3 level, the driver must drive when the vehicle or driver support feature requests, while under SAE4 and 5 levels, the autonomous driving feature will not require the driver to take over the driving. As a non-limiting example, starting from SAE 0 level, the example driver support features that can be used include automatic emergency braking, blind spot warning, and lane departure warning, and these features are limited to providing warnings and instant assistance. In the case of SAE Level 1, the driver support features that may be used may also include lane centering or adaptive cruise control, where these features provide steering or braking / acceleration support to the driver, in contrast, in the case of SAE Level 2, the autonomous driving features also include both lane centering and adaptive cruise control, and these features provide steering and braking / acceleration support to the driver. Therefore, in the case of SAE Level 0-2, the vehicle provides driver support features, in contrast, in the case of SAE Level 3-5, the vehicle is considered to provide autonomous driving features. For example, in the case of SAE Level 3, as a non-limiting example, example autonomous driving features may include congestion assistance drivers, and again as a non-limiting example, the autonomous driving features of SAE Level 4 include local driverless taxis and even example vehicles that may or may not be equipped with pedals / steering wheels. In the case of SAE Levels 3 and 4, the autonomous driving features may drive the vehicle under limited conditions, and the autonomous driving features will not be operated unless all required conditions are met. The autonomous driving features of SAE Level 5 may include all of the autonomous driving features of SAE Level 4, but the autonomous driving features may drive everywhere under all conditions. Another way to depict such SAE levels is with the respective names of the levels, such as SAE Level 0 is referred to as “No Automation”, SAE Level 1 is referred to as “Driver Assistance”, SAE Level 2 is referred to as “Partial Automation”, SAE Level 3 is referred to as “Conditional Automation”, SAE Level 4 is referred to as “High Automation”, and SAE Level 5 is referred to as “Full Automation”. While the example ADAS standard example above is SAE J3016 from the SAE, the examples are not limited thereto.However, to simplify the following explanation, these various available ADAS standards will be simply collectively referred to as ADAS standards.

[0118] Therefore, the driving support terminal 200 may be configured to store the driving support terminal 200 (or Figures 4 to 6 ) to provide advanced driving support functions.

[0119] In addition, in different examples, the driving support terminal 200 (and / or the mobile device 300) may have different storage capabilities and processing capabilities, wherein one such example is that the driving support terminal 200 is pre-restricted to implement or execute a driving program that provides an advanced driving support function corresponding to the second level or lower level of ADAS, wherein the processing module of the driving support terminal 200 may include a high bandwidth memory (HBM) of 8GB / 512 bits, and may have a size of 65×65mm, a processing speed of 10 peta floating point operations per second (10TFLOPS), and a power consumption level of 4 watts or lower. As described above, the example driving support terminal 200 (and / or the mobile device 300) may be pre-restricted to a driving program for providing advanced driving support functions of level 2 and below (i.e., not more than level 2), which may include, as non-limiting examples, a forward collision warning (FCW) function, a lane departure warning (LDW) function, a lane keeping assist (LKA) function, a blind spot warning (BSW) function, an adaptive cruise control (ACC) function, a driving road recognition function, a traffic sign recognition function, etc.

[0120] In another example, the driving support terminal 200 (and / or the mobile device 300) may have storage capacity and processing capacity corresponding to the pre-limitation of the driving support terminal 200 (and / or the mobile device 300) to implement or execute a driving program that provides a third level or higher advanced driving support function, wherein the processing module of the driving support terminal 200 may include a 32GB / 1024-bit high bandwidth memory (HBM), and may have a size of 180×120mm, a processing speed of 100 peta floating point operations per second (10TFLOPS), and a power consumption level of 6.6 watts or less. As described above, the example driving support terminal 200 (and / or the mobile device 300) may be pre-limited to provide a driving program of a third level or higher advanced driving support function specified in the ADAS standard, which may include an obscured object prediction function, a lane change function, a pedestrian recognition function, a road change recognition function, a road sign recognition function, etc. Optionally, in the example, in the case of this exemplary storage and processing configuration, or in the case of an exemplary storage and processing configuration with greater storage capacity and processing capacity, the driving support terminal 200 (and / or the mobile device 300) may not be pre-restricted as to which driving program can be executed or implemented, and thus any driving program corresponding to any one of levels zero to five of the ADAS standard may be implemented.

[0121] Accordingly, the driving support terminal 200 may include a driving program, and the driving program may include a driving assistance program and / or an autonomous driving program. As an example, in the case where the driving support terminal 200 (or the mobile device 300) is pre-restricted to provide a second level or lower level of advanced driving support functions of the ADAS standard due to the lower storage capacity and processing capacity of the above example, for example, the driving support terminal 200 (and / or the mobile device 300) may only include a driving assistance program.

[0122] Alternatively, in the case where the driving support terminal 200 has a large storage capacity and processing capacity and is pre-restricted to provide advanced driving support functions of the third level or higher of the ADAS standard, or in the case where the driving support terminal 200 (and / or the mobile device 300) is not pre-restricted to any specific level of the ADAS standard, the driving program that can be implemented by the driving support terminal 200 may include one or both of the driving assistance program and the autonomous driving program. In the following description, for ease of description, an example will be described in which the driving support terminal 200 has such a large storage capacity and processing capacity and is able to implement the functions of both the driving assistance program and the autonomous driving program.

[0123] In an example, the electronic control unit 121 may include such a memory and processor configuration having an example smaller storage capacity and processing capability for providing advanced driving support functions of the second level or lower level of the ADAS standard, while the driving support terminal 200 (and / or the mobile device 300) has a larger storage capacity and processing capability of at least the third level or higher of the ADAS standard, and may include either or both of a driving assistance program and an autonomous driving program. The driving support terminal 200 (and / or the mobile device 300) may request or otherwise obtain information on the driving information capabilities, assistance capabilities, and control capabilities of the electronic control unit 121, and may selectively (e.g., based on user control) or automatically replace or supplement the information functions, assistance functions, and / or control functions of the electronic control unit 121 based on a comparison of the functions of the driving program of the driving support terminal 200 (or the mobile device 300) with the functions of the driving program of the electronic control unit 121, and if any functions of the driving program of the driving support terminal 200 (or the mobile device 300) have a higher ADAS level or provide additional or more functions or features than the corresponding functions or features of the driving program of the electronic control unit 121, those functions of the electronic control unit 121 may be replaced by the information and / or control signaling of the driving support terminal 200 (or the mobile device 300). Optionally, if the driving program of the electronic control unit 121 provides the same information or auxiliary functions as the driving program of the driving support terminal 200 (and / or the mobile device 300), but does not provide an autonomous driving program and the driving program of the driving support terminal 200 (and / or the mobile device 300) includes an autonomous driving program, the autonomous driving program of the driving program of the driving support terminal 200 (or the mobile device 300) can be used to supplement the driving program of the electronic control unit 121.

[0124] Therefore, the driving support terminal 200 (and / or the mobile device 300) may apply a driving program including a driving assistance program and an autonomous driving program and may generate various driving support information and / or driving control signaling, wherein the driving assistance program and the autonomous driving program respectively consider the image information provided by the camera unit 110 (or the above-mentioned pre-processed image information such as from the corresponding management module), the driving information and operation information of the vehicle obtained from the communication port 140, and the external environment information obtained from the telematics control unit 130. As further discussed below, either or both of the driving assistance program and the autonomous driving program may include one or more AI models such as implemented by a neural network or implemented by other machine learning. In addition, in the example, the driving assistance program and the autonomous driving program may share artificial intelligence processing or models, such as, where an example trained neural network object detection model (which is trained to output information considered by both the driving assistance program and the autonomous driving program for different respective functions, or trained to output respective different information considered by two different programs) is used to perform object detection for considered image information, such as, where the function provided by the driving assistance program can be performed based on a first feature or corresponding resultant aspect, probability, classification, etc. of the object in the image, and the autonomous driving program can be performed based on a second feature or corresponding resultant aspect, probability, classification, etc. of the object in the image. In the example trained neural network object detection model, depending on the purpose of the neural network model during model training and the function to be performed based on the result of such a neural network model, the image information can be input to the first layer of the neural network, and then input to a series of convolutional neural network layers for feature extraction, followed by multiple feedforward neural network layers or other loops, bidirectional loops, long short-term memory (LSTM) layers, etc. Such a neural network model can also be used for multiple functions, and multiple such neural network models are used together for a single function. Although examples are discussed above for artificial intelligence models with various neural network configurations, other neural network configurations and other machine learning models are also available, and it is noted that the examples are not limited to the examples in this article.

[0125] For example, the driving support terminal 200 (or the mobile device 300) can sense peripheral obstacles by implementing a corresponding artificial intelligence model. As a non-limiting example, the artificial intelligence model can identify the distance from the sensed peripheral obstacles). When the identified distance is less than the reference distance indicated by the artificial intelligence model, the driving support terminal 200 can provide a collision warning. In another example, the artificial intelligence model can also be trained to generate a control signal in this case to automatically implement emergency braking, or such signaling can be generated based on the determination for issuing a collision warning. In another example, when the identified distance is less than a shorter reference distance and / or based on other considerations, the artificial intelligence model can alternatively be trained to generate a control signal for such emergency braking. The driving support terminal 200 can, for example, use the same or another artificial intelligence model to sense whether the vehicle has deviated from the lane, and when the vehicle has deviated from the lane, the driving support terminal 200 can determine to provide a deviation warning, or can automatically issue a deviation warning through the determination of the artificial intelligence model. Similarly, the artificial intelligence model can also simultaneously issue an assisted driving control signal or be trained to issue an assisted driving control signal when a larger lane deviation is determined. The driving support terminal 200 can identify pedestrians around the vehicle through such an object detection artificial intelligence model or another artificial intelligence model, which can be trained to predict the future movement route of the detected pedestrians, such as through a cyclic or bidirectionally connected layer or LSTM layer or a similar functional component of the artificial intelligence model, to support the user's driving, and similarly to the above, the artificial intelligence model can be trained for obstacle avoidance (i.e., to avoid the predicted future pedestrians) and correspondingly issue control signaling to control steering, acceleration and / or braking to avoid the predicted future pedestrians, or the avoidance can be controlled in other ways by the driving program of the driving support terminal 200. The driving support terminal 200 can also similarly support the functions of keeping lanes and changing lanes through the same or corresponding artificial intelligence model, and can control the electronic control unit 121 to control the appropriate driving components of the vehicle accordingly by issuing driving control signaling to the electronic control unit 121, or by sending the driving control signaling directly to the controlled driving components of the vehicle, to support driving at intersections without lanes, ordinary roads, off-road roads, etc.

[0126] The driving support terminal 200 (and / or the mobile device 300) may use the vehicle's operating information to determine a fault indication of the vehicle, and may suggest causes and solutions to the fault to the user. In addition, the driving support terminal 200 (and / or the mobile device 300) may control the body control module 122 based on external environmental information, such as an example in which the body control module 122 may control the operation of the air conditioning system of the vehicle 100 based on the internal temperature being greater than a first threshold and the external temperature being greater than a second threshold.

[0127] In addition to the processor and memory included, the driving support terminal 200 (and / or the mobile device 300) is represented by a display and a user interface, which can be controlled to display the preferred location on the current driving route of the vehicle according to predetermined user information, and can provide the analysis of the price and reservation service of the preferred location to the information display system of the vehicle 100 and / or control the display of the analysis of the price and reservation service of the preferred location. Above, various examples of the capabilities and operations of the driving support terminal 200 and the mobile device 300 are discussed with respect to the driving programs that can be stored in the driving support terminal 200 and the mobile device 300. Similar to the discussion of the supplement and / or replacement of the driving program of the electronic control unit 121, when the driving support terminal 200 physically supports the mobile device 300 and communicates with the mobile device 300, for example, through wireless charging communication, Bluetooth, Wi-Fi, or USB standard of USB wiring or other proprietary connection therebetween, the highest ADAS level function and / or additional or more feature settings between the driving assistance program and / or autonomous driving program in the electronic control unit 121, the driving support terminal 200, and the mobile device 300 may be selected, for example, by the user or automatically, and for example, the highest / maximum information and auxiliary functions and / or autonomous driving control functions selected in any selected and / or otherwise automatically selected may be ultimately provided, and the information and auxiliary functions and / or autonomous driving control functions not selected may not be provided. For example, the individual scores of the functions in the driving programs may be predetermined, so that a function in one driving program having a score higher than the corresponding function in another driving program may be considered to have the highest or maximum function. In addition, as described above, although the driving support terminal 200 may have a processor configuration with smaller storage capacity and processing capability that is pre-limited to the example of ADAS zero to second levels, the mobile device 300 may have a processor configuration with larger storage capacity and processing capability and may be pre-limited to ADAS third to fifth levels, so that different storage / processing configurations can be executed in parallel or simultaneously with their respective driving programs, so that all levels of driving assistance and autonomous control can be provided, for example, using the driving support terminal 200 that implements the assisted driving program and the mobile device 300 that implements the autonomous driving program, wherein the corresponding information and control signaling are provided from the driving support terminal 200 (or directly from the mobile device 300 or from the mobile device 300 through the driving support terminal 200) to the communication port 140, and then directly provided to the corresponding components via the CAN network or directly provided to the electronic control unit 121, so that the provided information and assisted and autonomous driving control are supplied through the driving support terminal 200 and the mobile device 300.As another example, in the case where the driving support terminal 200 only acts as an intermediary and translator, for example, to receive the above-mentioned coded control information and convert it into a form and format compatible with the CAN network for communication with the electronic control unit 121 or for directly controlling the components of the vehicle, or in the case where the driving support terminal 200 only acts as an intermediary to receive such control signaling already in the form and format of the CAN network from the mobile device 300 and pass it to the communication port 140 for communication with the electronic control unit 121 or for directly controlling the components of the vehicle, when the driving support terminal 200 does not store a driving program or only stores a driving program with low information or auxiliary functions, then only a driving assistance program and an autonomous driving program with higher or greater functions of the driving program of the mobile device 300 may be implemented. Similarly, in the example, the driving support terminal 200 may not include a processing capability for implementing any driving program, and may only perform an intermediary function and a charging function with the mobile device 300, for example, by wireless charging.

[0128] Figure 2 is a block diagram illustrating a driving support system according to one or more embodiments.

[0129] Figure 2 The driving support system in Figure 1 The driving support system shown in , therefore, overlapping descriptions will not be repeated, and differences will be described.

[0130] Reference Figure 2 The driving support system may further include a data processing module 150. Figure 2 In the embodiment of the present invention, the data processing module 150 may be a hardware component included in the vehicle 100 .

[0131] The camera unit 110 may provide the obtained image to the data processing module 150. As an example, the camera unit 110 may provide the generated image signal to the data processing module 150 through a low voltage differential signaling (LVDS) interface.

[0132] The data processing module 150 may be connected to the communication port 140 and may obtain driving information and operation information. As an example, the communication port 140 and the data processing module 150 may be connected to each other via a CAN network, and it is noted that although the examples herein discuss such connections between components in a vehicle provided by the CAN network utilizing the CAN communication protocol, the examples are not limited thereto. For example, the data processing module 150 may process an image provided from the camera unit 110 using the driving information and the operation information.

[0133] For example, when the data processing module 150 is implemented as a navigation device used in a vehicle, the data processing module 150 may include a navigation program, and the data processing module 150 may apply the navigation program to the image provided from the camera unit 110 and the driving information provided from the electronic control unit 121 to generate post-processing data. The data processing module 150 may output the post-processing data via a display and a speaker of the vehicle.

[0134] As another example, when the data processing module 150 includes a driving assistance program, the data processing module 150 may apply the driving assistance program to an image provided from the camera unit 110 and travel information provided from the electronic control unit 121 , and may generate post-processing data.

[0135] The data processing module 150 may make or provide the generated post-processed data to the communication port 140 and make or provide the images provided from the camera unit 110 to the communication port 140 .

[0136] The driving support terminal 200 may thus obtain or receive images and / or post-processing data via the communication port 140 .

[0137] The driving support terminal 200 may determine whether to apply a driving program stored in the driving support terminal 200 , for example, based on the post-processing data generated in the data processing module 150 .

[0138] For example, when only the navigation program of the vehicle 100 is applied to the post-processed data generated in the data processing module 150 and the driving assistance program is not applied, the driving support terminal 200 may apply the driving program of the driving support terminal 200 (e.g., the corresponding driving assistance program and autonomous driving program of the driving program) to the received data.

[0139] However, when the vehicle (e.g., through the electronic control unit 121) applies the driving assistance program to the post-processed data provided from the data processing module 150, the driving support terminal 200 may apply only the autonomous driving program among the driving programs of the driving support terminal 200 to the received data. In other words, when the vehicle applies the driving assistance program to the post-processed data, the driving support terminal 200 in the example may apply only the autonomous driving program to the received data without applying the driving assistance program, so that system resources can be used efficiently.

[0140] Even when the driving assistance program is applied to the post-processed data, the driving support terminal 200 may compare the driving assistance program of the data processing module 150 with the driving assistance program of the driving support terminal 200, and when the function of the driving assistance program of the driving support terminal 200 is improved or an additional function is provided on the basis of the driving assistance program of the data processing module 150, the driving support terminal 200 may apply the driving assistance program to the received data (e.g., a received image or processed image data, which is independent of the image data to which the driving assistance program of the data processing module 150 is applied). In such an example, the driving support terminal 200 may also predetermine such a difference in function, and provide instructions or control signaling to the data processing module 150 so as not to implement the driving assistance program of the data processing module 150 but only forward the image and / or image data otherwise processed by the data processing module 150.

[0141] Figure 3 is a block diagram illustrating a driving support system according to one or more embodiments.

[0142] Figure 3 The driving support system in Figure 2 The driving support system shown in , therefore, overlapping descriptions will not be repeated, and differences will be described.

[0143] Figure 2 An example is shown in which the driving support terminal 200 is directly connected to the communication port 140. However, referring to Figure 3 The driving support terminal 200 can be directly connected to the data processing module 150 and can obtain images obtained in the camera unit 110, driving information and operation information of the vehicle generated in the electronic control unit 121, and post-processing data generated in the data processing module 150.

[0144] The driving support terminal 200 and the data processing module 150 may be interconnected to each other via a second connection terminal (Type_C). The driving support terminal 200 and the data processing module 150 may each include a second connection terminal (Type_C). As an example, the driving support terminal 200 and the data processing module 150 may be interconnected to each other via a Type-C USB connection terminal.

[0145] Here, although no discussion has been made about Figure 2 and Figure 3 The above-mentioned mobile device 300, but the discussion is similarly applicable to the driving program of the mobile device 300, and also to the above-mentioned (and following) examples of collaborative considerations of the driving support terminal 200 connected to the mobile device 300, wherein either or both of the driving support terminal 200 and the mobile device 300 may store and be respectively configured to implement their respective driving programs or selective components and functions thereof.

[0146] Figure 4 and Figure 5 is a block diagram illustrating a driving support system according to one or more embodiments.

[0147] Figure 4 and Figure 5 The driving support system in Figures 1 to 3 The driving support system shown in , therefore, the repeated description will not be repeated, and the differences will be described.

[0148] Reference Figure 4 , in different examples, Figure 1 The camera unit 110 in the example or Figure 2 and Figure 3 The camera unit 110 and the data processing module 150 in the examples may be implemented additionally or alternatively by a mobile device 300 (including a smart phone or a tablet computer). For example, the mobile device 300 and the communication port 140 may be interconnected with each other via wireless communications such as Wi-Fi communications, but the examples are not limited thereto. In any of the examples herein, the mobile device 300 and the communication port 140 may also be similarly wirelessly connected to each other via the above-mentioned example controller area network (CAN) communications. Therefore, the functions of the camera unit 110 or the camera unit 110 and the data processing module 150 may also or additionally be performed by a camera and a processor employed in the mobile device 300.

[0149] Reference Figure 4 , the driving support terminal 200 can be directly connected to the mobile device 300 via the second connection terminal (Type_C). Figure 5 , the driving support terminal 200 may be directly connected to the communication port 140 via the first connection terminal (Type_OBD).

[0150] Figure 6 is a block diagram illustrating a driving support system according to one or more embodiments.

[0151] Figure 6 The driving support system in Figure 4 and Figure 5 Therefore, the overlapping description will not be repeated, and the differences will be described. Figures 1 to 3 The discussion of the driving program for the mobile device 300 and the driving support terminal 200 also applies to Figure 6 Driving support system.

[0152] Reference Figure 6 , Figure 6 The mobile device 300 in the example of FIG. Figure 4 and Figure 5The functions and functionality of the driving support terminal 200 shown in FIG. Figure 4 and Figure 5The functions and functionality of the driving support terminal 200 shown in (such as the implementation of the driving program of the driving support terminal 200 or the mobile device 300) can be executed by one or more processors adopted in the mobile device 300. For example, in different examples, one or more processors may have the above-mentioned different smaller storage and processing power configurations of respective ADAS levels zero to second or the above-mentioned different higher storage and processing power configurations of respective ADAS levels three to five, or one or more processors may have a larger storage and processing power configuration than these discussions, and are not limited to specific driving assistance or autonomous driving function groupings of ADAS levels. In addition, in this article, the driving program functions and functionality may be implemented by the CPU, NPU, GPU and / or other processors of the mobile device 300, such as the following examples: any one or any combination of such processors loads corresponding instructions and / or artificial intelligence models (e.g., parameters stored as one or more neural networks), and such as using a specific processor of the mobile device 300 and an operating system of the mobile device 300, using a software development kit (SDK) and / or an application programming interface (API) that enables the example artificial intelligence model to run on the mobile device 300. For example, the artificial intelligence model may be input into a model conversion tool to convert the artificial intelligence model into a deep learning container (DLC) format, and then optimized using an optimization tool that generates, for example, a deep learning container format file that can be executed by an artificial intelligence application. As a more specific example, the model conversion tool may be part of a neural processing engine (NPE) SDK (e.g., a Snapdragon neural processing engine (SNPE)), which, while operating with the accelerated runtime of one or more processors of the mobile device 300, may convert such an artificial intelligence model into a form and file format that can be executed by the NPE of the mobile device 300, for example, through one or more Qualcomm Snapdragon processors in the SNPE example. This conversion may be performed by the mobile device 300, or by a remote device or server, and stored in the mobile device 300 in the converted format for selective execution according to driving program instructions stored and executed by the mobile device 300. As another example, in this document, the example artificial intelligence model can be generated in any of a variety of formats (such as CaffeEmit, CNTK, CoreML, Keras, MXNet, ONNX, PyTorch, TensorFlow, and iOS formats), stored in any of a variety of formats, or converted to / from any of a variety of formats for execution by the processor of the mobile device 300.In another or additional example, the device driver level may be implemented by an operating system of the mobile device 300, wherein the artificial intelligence model is selectively executed by any one or any combination of such processors by controlling the device driver level, such as in an example of implementing Huawei HiAi engine or heterogeneous computing system (HiHCS). For example, in an example, the mobile device 300 may be interconnected with both the telematics control unit 130 and the communication port 140 via wireless communication (such as Wi-Fi communication), and may obtain information of the vehicle 100. Here is directed to. Figure 6 The discussed available configurations and capabilities of the mobile device 300 discussed herein also apply to all references herein to such mobile device 300 in the various examples discussed with respect to any or all other figures, and such further or alternative discussions of available configurations and capabilities discussed with respect to any or all other figures also apply to Figure 6 A mobile device 300 is provided.

[0153] Figure 7 is a block diagram illustrating a driving support terminal according to one or more embodiments.

[0154] Figure 7 The driving support terminal 200 in the example is a terminal that can provide advanced driving support functions such as ADAS level 2 or lower, and the driving program of the driving support terminal 200 may include a driving assistance program, for example, it can provide or support forward collision warning (FCW) function, lane departure warning (LDW) function, lane keeping assist (LKA) function, blind spot warning (BSW) function, adaptive cruise control (ACC) function, driving road recognition function, traffic sign recognition function, etc.

[0155] Reference Figure 7 As a non-limiting example, the example of the driving support terminal 200 herein may include an artificial intelligence processor 210 , a storage unit 220 , a communication module 230 , and a communication terminal 240 , and may also include a power terminal 250 .

[0156] The artificial intelligence processor 210 may include a central processing unit (CPU) 211, a neural processing unit (NPU) 212, and an interface 213. For example, the NPU 212 may be a neuromorphic processor. The CPU 211, the NPU 212, and the interface 213 may be electrically connected to each other. The CPU 211 and the NPU 212 of the artificial intelligence processor 210 may be connected to the communication module 230, the communication terminal 240, and the power terminal 250 through the interface 213.

[0157] The storage unit 220 may store a driving program including a driving assistance program. The storage unit 220 may include a plurality of memories 221 and 222 storing the driving assistance program. The CPU 211 and the NPU 212 may be connected to a plurality of different memories 221 and 222, respectively. Therefore, when executing the driving assistance program stored in the memories 221 and 222, the CPU 211 and the NPU 212 may, for example, generate driving support information. In an example, the plurality of memories 221 and 222 may all be implemented as high bandwidth memories (HBM).

[0158] The communication module 230 may include a Wi-Fi communication module and may perform wireless communication with one or more external devices. For example, the communication module 230 may perform wireless communication with the telematics control unit 130 of the vehicle and with the mobile device 300 in any example herein. In any of the various examples herein, the Wi-Fi communication module may also be included in the camera unit 110 and the communication port 140, and in this case, the communication module 230 may perform wireless communication with the camera unit 110 and the communication port 140.

[0159] The communication terminal 240 may be used as a path for receiving data from the vehicle and the mobile device 300 and transmitting data to the vehicle and the mobile device 300. For example, the communication terminal 240 may have the shape of a Type C USB connection terminal and an OBD type connection terminal. Similar to the above, in the example, the mobile device may include a similar artificial intelligence processor 210, a storage unit 220, a communication module 230, and a communication terminal.

[0160] The power terminal 250 may be connected to the battery of the vehicle, for example, through an OBD connection or in other ways, and may provide power provided from the battery of the vehicle to each element of the driving support terminal 200. In a different example, the driving support terminal 200 may further include a power management module (PMIC) 260 that regulates the voltage of the power provided from the vehicle. The driving support terminal 200 may further include a transmitting coil 270 that wirelessly provides the power provided from the battery of the vehicle to an external device. In this case, the driving support terminal 200 may be implemented as a stand or a stand (e.g., as shown in FIG. 1 ). Fig.10 ) to support Figure 4 and Figure 5 and wirelessly transmits power to the mobile device 300, for example, by inductive or resonant coupling, and the mobile device 300 is similarly configured to receive the wirelessly transmitted power by complementary inductive or resonant coupling.

[0161] In an example, the driving support terminal 200 may include a power terminal 250 so that the driving support terminal 200 receives power from a battery of the vehicle, but examples are not limited thereto. In any of the various examples herein, the driving support terminal 200 may further include a receiving coil and may also receive power wirelessly.

[0162] Figure 8 is a block diagram illustrating a driving support terminal according to one or more embodiments.

[0163] Figure 8 The driving support terminal 200 in the example may be a driving support terminal that provides advanced driving support functions of the third level or higher specified in the ADAS standard, and the driving program of the driving support terminal 200 may include a driving assistance program and an autonomous driving program, or may only include an autonomous driving program. In the example, an obscured object prediction function, a lane change function, a pedestrian recognition function, a road change recognition function, a road sign recognition function, etc. may be provided or supported. Figure 8 The driving support terminal 200 in Figure 7 The driving support terminal 200 shown in FIG. 2A and 2B , therefore, overlapping descriptions will not be repeated and differences will be described.

[0164] Reference Figure 8 , Figure 8 The storage unit 220 may include Figure 7 The number of memories provided in the storage unit 220 shown in FIG. 2 is greater than the number of memories provided in FIG. Figure 7 In the case where the storage unit 220 includes two memories, Figure 8 The storage unit 220 shown in FIG. 2 may include four memories. In the storage unit 220, the CPU 211 and the NPU 212 may correspond to a plurality of memories different from each other, and may guide the driving assistance program and the autonomous driving program.

[0165] Reference Figure 8 , Figure 7 The driving support terminal 200 in the embodiment may include a single communication terminal 240, and Figure 8 The driving support terminal 200 in may include a plurality of communication terminals 240a, 240b, 240c, and 240d.

[0166] The first communication terminal 240a may be a C-type USB connection terminal and an OBD type connection terminal. The second communication terminal 240b may be a connection terminal of a CAN network, the third communication terminal 240c may be a connection terminal of a Gigabit Multimedia Serial Link (GMSL), and the fourth communication terminal 240d may be a communication terminal of Ethernet. Similar to the above, in the example, the mobile device 300 may include a similar artificial intelligence processor 210, a storage unit 220, a communication module 230, and a communication terminal.

[0167] Compare Figure 7 and Figure 8 The driving support terminal 200 in Figure 7 The driving support terminal 200 described herein can provide advanced driving support functions of the second level or lower level specified in the ADAS standard, while Figure 8 The driving support terminal 200 in the example in can provide advanced driving support functions of the third level or higher specified in the ADAS standard. Figure 7 and Figure 8 The driving support terminals 200 shown in FIG. 1 may be different from each other, and may be the same terminal having a storage and processing configuration or two separate driving support storage and processing configurations for providing respective functions.

[0168] exist Figure 7 In the example of the driving support terminal 200 in , the CPU may have the specifications of ARM Cortex A72x2 and A53x4@1.5GHz, and the NPU may have a processing speed of 10 peta floating point operations per second (TFLOPS). In such an example, the storage unit 220 may include a high-bandwidth memory of 8GB / 512 bits, and as another example, the storage unit 220 may include a 32GB NAND flash memory. For example, the Wi-Fi communication module of the communication module 230 may provide a speed of at least 2.2Gbps. For example, the communication module 230 may have specifications of the IEEE 802.11ac / ax standard, 4×4MU-MIMO and 1024QAM. In such an example, the driving support terminal 200 may have a power consumption level of 4 watts or less and may have a size of 65×65mm.

[0169] exist Figure 8In the example of the driving support terminal 200 in , the CPU may have specifications of ARM CortexA72x2 and A53x4@3GHz, and the NPU may have a processing speed of 100 peta floating point operations per second (TFLOPS). In such an example, the storage unit 220 may include a 32GB / 1024-bit high bandwidth memory (HBM). As another example, the storage unit 220 may include a 64GB NAND flash memory. For example, the Wi-Fi communication module of the communication module 230 may provide a speed of at least 1Gbps. The communication module 230 may have specifications of the IEEE 802.11ac / ax standard, 4×4MU-MIMO and 1024QAM. The example driving support terminal 200 may have a power consumption level of 6.6 watts and may have a size of 180×120mm.

[0170] Fig. 9 is a diagram illustrating an example connection relationship between a driving support terminal and a controller of a vehicle according to one or more embodiments.

[0171] Reference Fig. 9 , the driving support terminal 200 may be connected to a controller (e.g., electronic control unit 121) of the vehicle via the third communication terminal 240c. The driving support terminal 200 may be connected to the electronic control unit 121 via the third communication terminal 240c, and may receive images of the front area, side area, and rear area of ​​the vehicle imaged by the camera unit 110. In this case, the electronic control unit 121 may include a driving program that provides an advanced driving support function of the driving support terminal 200.

[0172] For example, in order for the driving support terminal 200 to provide the third level or higher advanced driving support functions specified in the ADAS standard without delay, according to the ADAS standard, high-performance data processing speed and high-performance data receiving and sending speed may be expected or required, so a plurality of ports for collecting various data may be arranged. However, according to the exemplary processing, storage and communication specifications, the example of the small driving support terminal 200 may not fully meet the above-mentioned high-performance aspects, so as a non-limiting example, such a small driving support terminal may mainly implement the ADAS zero to second level functions of the above-mentioned example.

[0173] In an example, the driving program may be embedded in the electronic control unit 121 , the vehicle may not include the driving program, or the driving program may include either or both of a driving assistance program and an autonomous driving program.

[0174] In an example, instructions for assisting the interaction of the driving support terminal 200 and / or the mobile device 300 with the electronic control unit 121 may be uploaded, installed or embedded in the electronic control unit 121. For example, such instructions may assist the electronic control unit 121 in receiving and understanding information and / or control signaling provided from the driving support terminal 200 and / or the mobile device 300 through the communication port 140 or otherwise provided through the CAN network, wherein such understanding of the information or control signaling may include the driving support terminal 200 forwarding the information or control signaling to the appropriate controlled vehicle component, or the electronic control unit 121 copying and forwarding it to the appropriate vehicle component (just as if the electronic control unit 121 has implemented its own driving program), for example, thereby, for example, audibly and / or visually controlling the provision or display of information from the driving support terminal 200 and / or the mobile device 300, and / or thereby controlling the provision of other driving assistance or different levels of autonomous driving control from the driving support terminal 200 and / or the mobile device 300. The instructions may also include adding coding or controlling, modifying or deleting current coding of the electronic control unit 121 for such reception and understanding of the information or control signaling by the electronic control unit 121, or for controlled passivity of the electronic control unit 121 to only forward the same information or control signaling to the corresponding component of the vehicle, or for controlling the electronic control unit 121 so as not to interrupt or interfere with the same control or control signaling provided directly to the corresponding component by the driving support terminal 200 or the mobile device 300, for example, by not issuing information or control signaling according to the driving program of the electronic control unit 121 (except for information or control signaling from the driving support terminal 200 and / or the mobile device 300), or otherwise preventing the information or control signaling from displaying (or audibly reproducing) the received information to the user / driver or controlling the corresponding component using the control signaling. As described above, the instruction may also include adding code or controlling, modifying or deleting the current code of the electronic control unit 121 to control the cooperation of the current assisted driving program or autonomous driving program, or controlling the cooperation of such program of the driving program of the electronic control unit 121 with the selection function of the assisted driving program and / or autonomous driving program, or controlling the cooperation of the driving support terminal 200 and / or the mobile device 300. Such uploading, installing or embedding may be automatically processed when the driving support terminal 200 and / or the mobile device 300 is initially connected to the communication port 140 or the corresponding application or program of the driving support terminal 200 or the mobile device 300 is executed to start or continue to provide the driving assistance / control function, or by user selection or authorization to upload, install or embed the instruction to the electronic control unit 121 (for example, through an example user interface of the driving support terminal 200 or the mobile device 300).Additionally or alternatively, a separate control module may be inserted into the vehicle to perform conversion or switching operations between the electronic control unit 121 and the information and control signaling from the driving support terminal 200 or the mobile device 300. The instructions uploaded, installed, or embedded in the electronic control unit 121 or the instructions providing the conversion or switching operations through a separate control module may also provide similar interface functions to provide vehicle and sensed environment information to the driving support terminal 200 and / or the mobile device 300 to generate such control by the driving support terminal 200 and the mobile device 300. In addition, in the example, the electronic control unit 121 may be replaced with the same or compatible electronic control unit 121 that already includes the above instructions, so there is no need to upload, install, or embed the above instructions.

[0175] Fig.11 is a block diagram illustrating an implementation of a driving program using an NPU according to one or more embodiments.

[0176] The NPU 212 may implement an image receiver 212a, an image preprocessor 212b, an object recognition model 212c, and a driving support information generator 212d, and thus may recognize an object in an image obtained in the camera unit 110 and generate driving support information specific to the recognized object.

[0177] An image obtained in the camera unit 110 may be input to the image receiver 212 a .

[0178] The image preprocessor 212b may perform image processing to specify a region of interest (ROI) within an input image, and may generate an image of interest therefrom. As an example, the image preprocessor 212b may extract edge components of the input image, such as by high-frequency filtering, to detect different regions of interest and may generate corresponding images of interest. For example, the image preprocessor 212b may perform standardization on the generated image of interest to, for example, have a predetermined size (such as a size that one or more subsequent artificial intelligence models have been trained on). The image preprocessor 212b may further perform calibration on the brightness of the standardized image of interest. As an example, a zero-center method may be used to calibrate the brightness value of the standardized image of interest. The image preprocessor 212b may provide the standardized and calibrated image of interest to the target recognition model 212c, for example, the target recognition model 212c may be represented as a neural network system having multiple convolutional layers, followed by an example feedforward, RNN or other classification training layer.

[0179] The object recognition model 212c may implement a neural network system by training the stored parameters (e.g., previously trained parameters) of the various layers of the neural network system for a specific purpose image of an example object through deep learning (such as loss-based back propagation), so that when the parameters are loaded from the memory, the processing element of the NRU 212 analyzes the pre-processed image of interest through multiple neural network layers, and the NRU 212 can recognize the object in the input image. As an example, the object recognition model 212c may repeatedly implement the neural network system to recognize the object in each received frame and / or recognize the object for multiple detected ROIs of the input image, and can recognize the object with very high accuracy. Multiple such neural network systems (where each system is trained to recognize a respective object) may be executed in parallel, or the neural network system may be configured to perform classification of the input image and recognize various trained objects. For example, the objects may include traffic signals, lanes, crosswalks, road signs, etc., such that the neural network system or each neural network system is trained using various training images until the neural network system or each neural network system recognizes the correct object within a predetermined accuracy or predetermined inaccuracy.

[0180] The driving support information generator 212d may implement a driving program or a component of a driving program. For example, the driving support information generator 212d may implement at least one of a driving assistance program and an autonomous driving program of a driving program, read the target identified in the target recognition model 212c from the memory, and may generate driving support information specific to the identified target (e.g., specific to the position of the identified target and the relationship between the identified target and other objects, etc.). Here, for each corresponding discussion, the described vehicle refers to a vehicle having an engineering capability to implement at least the corresponding described driving assistance function, and in the example of discussing the implementation of a specific level of autonomous driving program or a specific driving assistance function, the corresponding vehicle is a vehicle having an engineering capability for such a corresponding function. Optionally, although the vehicle is a vehicle having at least one engineering capability to implement at least one driving assistance function, the example includes that the driving support terminal 200 and / or the mobile device 300 determines the function of the vehicle and does not enable the function of the driving program of the driving support terminal 200 and / or the mobile device 300 that the vehicle cannot perform, or the corresponding function of the driving program of the driving support terminal 200 and / or the mobile device 300 that the vehicle cannot perform may be disabled or not implemented.

[0181] Fig.12 FIG. 1 is a cross-sectional view and FIG. 1 is a plan view, respectively, showing a driving support terminal according to one or more embodiments.

[0182] Reference Fig.12As a non-limiting example, the driving support terminal 200 in the example may include a substrate 280 , an artificial intelligence processor 210 , a storage unit 220 , a communication module 230 , a communication terminal 240 , a power management module 260 , and a housing 290 .

[0183] The artificial intelligence processor 210, the communication module 230, the communication terminal 240, and the power management module 260 may be arranged on the substrate 280. The communication module 230 and the power management module 260 may be arranged on one surface of the substrate 280, and the artificial intelligence processor 210 may be arranged on another surface of the substrate 280. The communication terminal 240 may be arranged on the artificial intelligence processor 210. As a non-limiting example, the exterior of the driving support terminal 200 may be formed by the housing 290, and the communication terminal 240 may extend and protrude in one direction to be connected to an external device.

[0184] The artificial intelligence processor 210 may include a driving program or may load a driving program from a memory, the driving program may include an assistance program and / or an autonomous driving program, and the driving program may include loading training parameters of corresponding artificial intelligence models implemented by each program or selectively loading these parameters as needed, and may generate driving support information and driving assistance and autonomous driving control using data provided through a communication terminal, for example, from a communication port of the vehicle and / or from an additional camera (such as a mobile device) and from other cameras connected to the mobile device 300 (such as through a USB connector of the mobile device 300). As a non-limiting example, the artificial intelligence processor 210 may include a central processing unit (CPU) and a neural processing unit (NPU), and may be integrated with the storage unit 220.

[0185] The communication module 230 may include a Wi-Fi communication module and may be configured to perform wireless communication with an external device. Figure 2 The telematics control unit 130 and one or more cameras and the mobile device 300 shown in the figure perform wireless communication. When the power management module 260 is connected to the vehicle, the power management module 260 can adjust the voltage provided from the battery of the vehicle. The communication terminal 240 can be directly connected to an external device and can be used as a path for sending and receiving data, for example, directly connected to a communication port of the vehicle and / or connected to the mobile device 300.

[0186] exist Fig.12 In the embodiment, the communication terminal 240 may have the shape of a Type-C USB connection terminal, but as described above, the communication terminal 240 may have the shape of an OBD type connection terminal.

[0187] Fig.13 is a block diagram illustrating a driving support system according to an example embodiment, Fig.14and Fig.15 is a diagram showing a calibration operation performed by the image processor of the present disclosure, and Fig.16 and Fig.17 is a diagram illustrating an operation of controlling a camera unit performed by an artificial intelligence processor of the present disclosure.

[0188] because Fig.13 The driving support system in Figures 1 to 12 The driving support system in FIG. 1 will not be described repeatedly, and the differences will be mainly described.

[0189] Reference Fig.13 , Figure 1 The camera unit 110 in the example embodiment shown in FIG. Figure 2 and Figure 3 The camera unit 110 and the data processing module 150 in the example embodiment shown in FIG. 1 may alternatively be implemented by a mobile device 300 including a smartphone or a tablet. Therefore, the functions of the camera unit 110 or the camera unit 110 and the data processing module 150 may be performed by a camera unit and a controller employed in the mobile device 300.

[0190] Reference Fig.13 , the mobile device 300 may include a controller 310 , a communication unit 320 , a camera unit 330 , a display unit 340 , a power supply unit 350 , and a power switch 360 .

[0191] The controller 310 may be electrically or functionally connected to other blocks of the mobile device 300, and may control the overall operation of the mobile device 300 and the signal flow between internal blocks of the mobile device 300, and may process data. The controller 310 may include a central processing unit (CPU), an application processor, a graphics processing unit (GPU), and the like.

[0192] The communication unit 320 may be interconnected with the external cloud 400 via a network. The network may refer to a communication network formed using a predetermined communication method. The predetermined communication method may include all communication methods, such as communication via a predetermined communication standard, a predetermined frequency band, a predetermined protocol, or a predetermined channel. For example, the communication method may include a communication method via Bluetooth, BLE, Wi-Fi, Zigbee, 3G, 4G, 5G, and ultrasound, and may include near field communication, long distance communication, wireless communication, and wired communication.

[0193] Meanwhile, the vehicle 100 can be interconnected with the mobile device 300 through wireless communication. For example, the communication port 140 (see Figures 1 to 3 ) can be interconnected with the communication unit 320 of the mobile device 300 through wireless communication such as Wi-Fi communication. As another example, the communication port 140 (see Figures 1 to 3) may be wirelessly connected to the communication unit 320 of the mobile device 300 via a controller area network (CAN) communication.

[0194] The camera unit 330 may be mounted on the front surface and / or the rear surface of the mobile device 300, and may image an external area of ​​the mobile device 300. The camera unit 330 may provide the controller 310 with a generated image.

[0195] The display unit 340 may be provided on the front surface of the mobile device 300, and may output information according to a user input on the screen. For example, the display unit 340 may be integrated with a touch screen device for receiving a user touch input.

[0196] The power supply unit 350 may be electrically connected to other blocks of the mobile device 300, and may supply power required to drive the other blocks. For example, the power supply unit 350 may include a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell. The power supply unit 350 may be integrally provided in the mobile device 300, or may be detachably provided with the mobile device 300.

[0197] The power switch 360 may supply the power provided from the power supply unit 350 to the driving support terminal 200. The controller 310 may detect whether the driving support terminal 200 is connected to the mobile device 300, and may control the power supply operation of the power switch 360. For example, when the driving support terminal 200 is connected to the mobile device 300, the controller 310 may control the power switch 360 to be turned on and power may be supplied to the driving support terminal 200, and for example, when the driving support terminal 200 is not connected to the mobile device 300, the controller 310 may control the power switch 360 to be turned off and power may be blocked from being supplied to the driving support terminal 200.

[0198] Reference Fig.13 As a non-limiting example, the driving support terminal 200 may include an artificial intelligence processor 210, an image processor 205, a model storage unit 215, a learning processor 225, and a power management module 260. The driving support terminal 200 may be directly connected to the mobile device 300 through the second connection terminal Type_C.

[0199] The power management module (PMIC) 260 may adjust the voltage of power supplied from the mobile device 300. The power management module 260 may be electrically connected to other blocks of the driving support terminal 200, and may supply power required to drive the other blocks.

[0200] The image processor 235 may pre-process an image provided from the camera unit 330 of the mobile device 300. The image processor 235 may receive an image from the camera unit 330 of the mobile device 300 when the driving support terminal 200 is interconnected with the mobile device 300 through the second connection terminal (Type_C).

[0201] According to an example embodiment, the controller 310 of the mobile device 300 may mainly pre-process the image provided from the camera unit 330, and the image processor 235 may secondarily pre-process the mainly pre-processed image. To this end, the controller 310 of the mobile device 300 may include a separate image processor. The image processor 235 may correspond to Fig.11 The image preprocessor 212b shown in FIG.

[0202] As an example, the image processor 235 may perform image processing to specify a region of interest (ROI) within an image provided from the camera unit 330, and may generate an image of interest therefrom. As an example, the image processor 235 may detect different regions of interest by extracting edge components of an input image using high-frequency filtering, and may generate images of interest corresponding to different regions of interest. For example, the image processor 235 may perform normalization on the generated regions of interest so that one or more subsequent artificial intelligence models have a predetermined size equal to the training size. The image processor 235 may also perform additional calibration on the brightness of the standardized image of interest. When the image processor 235 calibrates the brightness of the standardized image of interest, the brightness may be appropriately reduced. Fig.14 The brightness value of the image of interest shown in FIG. (a) is improved so that the object visibility can be improved as follows Fig.14 As an example, the brightness value of the standardized image of interest can be calibrated using a zero-center method.

[0203] At the same time, the image processor 235 can perform additional calibration by rotating the standardized image of interest. Fig.15 When performing calibration using the image of interest shown in FIG. Fig.15 The horizontal side and the vertical side of the image of interest shown in FIG. 2 (b) can be changed to be set in the horizontal direction and the vertical direction.

[0204] The image processor 235 may provide the standardized and calibrated image of interest to the artificial intelligence processor 210. The artificial intelligence processor 210 may use the image of interest provided from the image processor 235 as input data.

[0205] The artificial intelligence processor 210 may be electrically or functionally connected to other blocks of the driving support terminal 200 , and may control overall operations of the driving support terminal 200 and signal flows between internal blocks of the driving support terminal 200 , and may process data.

[0206] The artificial intelligence processor 210 may include an artificial neural network processing unit.

[0207] Artificial neural networks are information processing systems that model the operating principles of biological neurons and the relationships between neurons, and are connected in a layered structure with multiple neurons called nodes or processing elements. Artificial neural networks are models used for machine learning, statistical learning algorithms inspired by biological neural networks (central nervous systems of animals, especially the brain) in machine learning and cognitive science.

[0208] Specifically, an artificial neural network refers to an overall model that has problem-solving capabilities by changing the strength of synaptic binding through learning by artificial neurons (nodes) that form a network by combining synapses. A neural network may include multiple layers, and each layer may include multiple neurons. In addition, an artificial neural network may include synapses connecting neurons.

[0209] In general, an artificial neural network can be defined by an activation function that generates an output value from three factors: the connection pattern between neurons in different layers, the learning process that updates the weights of the connections, and the weighted sum of the inputs received from the previous layer.

[0210] The artificial neural network may include network models such as a deep neural network (DNN), a recurrent neural network (RNN), a bidirectional recurrent deep neural network (BRDNN), a multilayer perceptron (MLP), and a convolutional neural network (CNN), but is not limited thereto.

[0211] Artificial neural networks are divided into single-layer neural networks and multi-layer neural networks according to the number of layers.

[0212] The artificial neural network can be trained using training data. Here, learning refers to the process of determining the parameters of the artificial neural network using the training data in order to achieve the purpose of classifying, regressing or clustering the input data. As representative examples of the parameters of the artificial neural network, there may be weights applied to synapses or biases applied to neurons.

[0213] The artificial intelligence processor 210 may receive input data and training data for model training.

[0214] The artificial intelligence processor 210 may apply input data provided from the image processor 205 to the training model stored in the model storage unit 215 and may infer output data.

[0215] Meanwhile, the artificial intelligence processor 210 may control at least one block provided in the mobile device 300 according to input data corresponding to the image of interest provided from the image processor 205 .

[0216] As an example, Fig.16 As shown in FIG. 1 (a), when the image of interest includes a flickering phenomenon, the shutter speed of the camera unit 330 can be changed by the controller 310, so that Fig.16 The flickering phenomenon included in the image of interest is eliminated as shown in Figure (b).

[0217] As another example, Fig.17 As shown in FIG. 1 (a), when the screen brightness of the image of interest is dark, the aperture value of the camera unit 330 can be changed by the controller 310, so that Fig.17 As shown in Figure (b), the screen brightness of the image of interest is appropriately changed.

[0218] The learning processor 225 may train (or learn) the artificial neural network stored in the model storage unit 215 using the training data provided from the artificial intelligence processor 210 .

[0219] The artificial intelligence processor 210 may transmit the training data provided from the communication unit 320 of the mobile device 300 to the learning processor 225. The communication unit 320 may download the training data from the cloud 400. When the driving support terminal 200 is interconnected with the mobile device 300 through the second connection terminal (Type_C), the artificial intelligence processor 210 may download the training data through the communication unit 320 of the mobile device 300. That is, when the driving support terminal 200 is interconnected with the mobile device 300 through the second connection terminal Type_C, the training data may be updated.

[0220] The driving support terminal 200 may be directly connected to the mobile device 300 through the second connection terminal (Type_C).

[0221] The artificial intelligence processor 210 may preprocess the input data and the training data, and may generate preprocessed input data and preprocessed training data. For example, the preprocessing of the input data performed by the artificial intelligence processor 210 may refer to extracting input features from the input data.

[0222] The model storage unit 215 may store an artificial neural network. The artificial neural network stored in the model storage unit 215 may include a plurality of hidden layers. However, the artificial neural network in the example embodiment is not limited thereto. The artificial neural network may be implemented by hardware, software, or a combination of hardware and software. When part or all of the artificial neural network is implemented by software, one or more command words configuring the artificial neural network may be stored in the memory.

[0223] The artificial neural network stored in the model storage unit 215 may be learned by the learning processor 225. The model storage unit 215 may store a model trained or learned by the learning processor 225. When the model is updated by learning, the model storage unit 215 may store the updated model. The model storage unit 215 may classify the learning model into a plurality of versions according to a learning time point or a learning progress, and may store the model when necessary.

[0224] As an example, the model storage unit 215 may include Fig.11 The target recognition model 212c shown in .

[0225] The object recognition model 212c may implement a neural network system by training the image for a specific purpose of an example object using stored parameters of the various layers of the neural network system (such as previously trained parameters) through deep learning (such as loss-based back propagation). Thus, the object recognition model 212c may process the components of the NRU 212 by uploading parameters from the memory to analyze the pre-processed image of interest through multiple neural network layers so that the NRU 212 may recognize the object in the input image. As an example, the object recognition model 212c may repeatedly implement the neural network system to recognize the object for each received frame and / or for multiple detected ROIs of the input image, and may recognize the object with very high accuracy. The multiple neural network systems may be trained for each system to recognize the corresponding object, and the multiple neural network systems may be executed in parallel, or the neural network system may be configured to classify the input image and may recognize various trained objects. For example, the object may include a traffic signal, a lane, a crosswalk, a road sign, etc., so that the neural network system or each neural network system may be trained using various training images until the neural network system or each neural network system recognizes the accurate object within a predetermined accuracy or a predetermined inaccuracy.

[0226] at the same time, Fig.11 The driving support information generator 212 d in may be implemented by the artificial intelligence processor 210 .

[0227] The driving support information generator 212d may implement a driving program or a component of the driving program. For example, the driving support information generator 212d may implement at least one of a driving assistance program and an autonomous driving program of the driving program read from the memory for the target identified in the target recognition model 212c, and may generate driving support information specified for the identified target, for example, driving support information specified for the position of the identified target and the relationship between the identified target and other objects.

[0228] The learning processor 225 may train (or learn) the artificial neural network stored in the model storage unit 215 using the training data.

[0229] The learning processor 225 may acquire the training data provided from the artificial intelligence processor 210, and may learn the artificial neural network stored in the model storage unit 215. For example, the learning processor 225 may determine the optimized model parameters of the artificial neural network by repeatedly training the artificial neural network using various well-known learning techniques. The artificial neural network whose parameters are determined by training using the training data may be referred to as a training model or a trained model.

[0230] This article is aimed at Figures 1 to 12The camera, camera unit, camera unit 110, electronic control unit, vehicle control unit 120, electronic control unit 121, body control module 122, controller, communication port, communication port 140, telematics control unit 130, data processing module, preprocessor, data processing module 150, module, terminal, management module, driving support terminal, driving support terminal 200, mobile device, mobile device 300, mobile phone, communication module, communication module 230, communication terminal 240, communication terminals 240a-240d, power terminal 250, transmitting coil 270, PMIC 260, storage unit 220, memory, memory 221 and 222, connector, neuromorphic processor, processor, NPU, NPU 212, CPU, CPU 211, interface, interface 213, artificial intelligence processor, artificial intelligence processor 210 and other devices, terminals, modules, units, devices and other components described in FIG. (b) are hardware components and are implemented by hardware components. Examples of suitable hardware components that can be used to perform the operations described in this application include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application can be implemented by computing hardware, such as by one or more processors or computers. A processor or computer can be implemented by one or more processing elements (such as, logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other devices or combinations of devices configured to respond and execute instructions in a limited manner to achieve the desired results). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as, operating system (OS) and one or more software applications running on OS) to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or the processor or computer may include multiple processing elements or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller.One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, examples of which include a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0231] about Figures 1 to 11 The methods shown and discussed and performing the operations described in this application are performed by computing hardware (e.g., by one or more processors or computers implemented as described above), which executes instructions or software to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors or a processor and a controller, and one or more other operations may be performed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may perform a single operation or two or more operations.

[0232] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform methods as described above may be written as computer programs, code segments, instructions, or any combination thereof that are used to individually or collectively instruct or configure one or more processors or computers to operate as machines or special-purpose computers to perform operations performed by hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include higher-level codes that are executed by one or more processors or computers using an interpreter. Instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding description of the algorithms disclosed in the specification that are used to perform operations performed by hardware components and methods as described above.

[0233] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods and any associated data, data files, and data structures described above may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card-type storage (such as MultiMediaCard micro or card (e.g., Secure Digital (SD) or Extreme Digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner, and provide instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0234] Although the present disclosure includes specific examples, it will be apparent after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein will be understood only as descriptive, and not for limiting purposes. The description of the features or aspects in each example will be understood to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described system, architecture, device, or circuit are replaced or added in a different manner and / or by other components or their equivalents, suitable results may be obtained. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents will be interpreted as included in the present disclosure.

Claims

1. A driving support system comprising: Vehicles, and a driving support terminal configured to interact with the vehicle, The vehicle includes: a camera system configured to generate first image information of the periphery of the vehicle, a vehicle controller configured to generate driving information of the vehicle, and a communication port, and Wherein, the driving support terminal includes one or more processors, and the one or more processors are configured to: connected to the communication port, obtaining the first image information and the driving information from a camera system of the vehicle using the communication port, acquiring second image information from another camera separate from the vehicle, and At least a portion of the driving program of the driving support terminal is executed based on at least one of the first image information from the camera system and the second image information obtained from the other camera and the driving information to control the vehicle based on control signaling generated by the driving support terminal and provided to the vehicle for one or more assisted driving operations and autonomous operations of the vehicle.

2. The driving support system according to claim 1, wherein: The driving program of the driving support terminal includes one or more of a driving assistance program and an autonomous driving program.

3. The driving support system according to claim 2, wherein: The vehicle controller is configured to execute a vehicle driving program stored in a memory of the vehicle to control the vehicle, the vehicle driving program including one or more of a vehicle driving assistance program and a vehicle autonomous driving program.

4. The driving support system according to claim 3, wherein: The one or more processors of the driving support terminal are also configured to: determine which of the driving assistance program of the driving program of the driving support terminal and the vehicle driving assistance program of the vehicle driving program is the driving assistance program of the highest level of an advanced driver assistance system, and / or determine which of the autonomous driving program of the driving program of the driving support terminal and the vehicle autonomous driving program of the vehicle driving program is the autonomous driving program of the highest level of an advanced driver assistance system, and the one or more processors are configured to execute the driving assistance program of the highest level of an advanced driver assistance system and / or the autonomous driving program of the highest level of an advanced driver assistance system to control the vehicle for the one or more assisted driving operations and / or autonomous operations of the vehicle.

5. The driving support system according to claim 3, wherein: The one or more processors of the driving support terminal are also configured to: perform a first determination to determine which functions of the driving program of the driving support terminal have a higher functional level than the corresponding functions of the vehicle driving program, perform a second determination to determine which functions of the driving program of the driving support terminal are not provided by the vehicle driving program, and / or perform a third determination to determine which functions of the vehicle driving program are not provided by the driving program of the driving support terminal, and the one or more processors are configured to control the driving program functions implemented by the vehicle by selectively implementing the driving program of the driving support terminal and the vehicle driving program under the control of the driving support terminal based on any one or any combination of the first determination, the second determination and the third determination, so as to control the vehicle for one or more assisted driving operations and / or autonomous operations of the vehicle.

6. The driving support system according to claim 2, wherein: In order to generate the control signaling, the driving assistance program is configured to generate driving support control signaling and driving support information, wherein the driving support control signaling is a signaling for assisting in controlling the assist driving operation of the vehicle to display the driving support information.

7. The driving support system according to claim 6, wherein: The driving support information of the vehicle includes information instructing a driver to control at least one of a steering operation, a braking operation, and an acceleration operation of the vehicle.

8. The driving support system according to claim 7, wherein: The operations instructing the driver include those determined by the driving program: obstacle detection warning and / or instructing the driver to perform corresponding obstacle avoidance steering and / or braking; providing rear collision warning and / or instructing the driver to perform corresponding rear collision avoidance braking; providing lane departure warning, instructing the driver to perform corresponding lane departure remedies or maintain steering, and / or indicating a recommended lane change to the driver.

9. The driving support system according to claim 2, wherein: In order to generate the control signaling, the driving assistance program is configured to generate driving support control signaling, wherein the driving support control signaling is a signaling for controlling the auxiliary driving operation of the vehicle to assist in controlling at least one of the steering operation, braking operation and acceleration operation of the vehicle.

10. The driving support system according to claim 9, wherein: The signaling for controlling the assisted driving operation of the vehicle to assist in controlling at least one of the steering operation, braking operation and acceleration operation of the vehicle includes: controlling at least one of the steering operation, braking operation and acceleration operation of the vehicle for detected obstacle assisted avoidance, detected rear collision assisted avoidance, predicted and / or detected lane departure assisted prevention or remediation, and / or assisted lane change determined by the driving program.

11. The driving support system according to claim 10, wherein: The driving support control signaling is signaling for controlling the assist driving operation of the vehicle to display driving support information generated by the driving assist program, the driving support information indicating information on at least one of a steering operation, a braking operation, and an acceleration operation of assist control of the vehicle.

12. The driving support system according to claim 2, wherein: The autonomous driving program is configured to generate driving control signaling, wherein the driving control signaling is signaling for controlling autonomous driving operation of the vehicle to control at least one of a steering operation, a braking operation, and an acceleration operation of the vehicle without auxiliary control by a driver.

13. The driving support system according to claim 12, wherein: The operation of generating the driving control signaling includes: performing any one or any combination of two or more of blocked object prediction, lane change determination, pedestrian recognition, road change recognition and road sign recognition; and generating the driving control signaling based on the results of the execution.

14. The driving support system according to claim 13, wherein: The one or more processors of the driving support terminal include one or more artificial intelligence processors, and the one or more artificial intelligence processors are configured to: load one or more artificial intelligence models from the memory included in the driving support system and implement the one or more artificial intelligence models to perform any one or any combination of two or more of the obscured object prediction, the lane change determination, the pedestrian recognition, the road change recognition and the road sign recognition, and generate the driving control signaling based on the result of the execution.

15. The driving support system according to claim 14, wherein: The one or more artificial intelligence processors include one or more neuromorphic processors, and the one or more artificial intelligence models are respective neural network models.

16. The driving support system according to claim 1, wherein: The one or more processors are configured to perform post-processing on the obtained image information and / or other image information obtained by the driving support terminal model, and the one or more processors include one or more artificial intelligence processors, wherein executing at least a portion of the driving program includes executing one or more neural network models for providing post-processed obtained image information and / or post-processed other image information.

17. The driving support system according to claim 1, wherein: The one or more processors are configured to perform post-processing on the obtained image information and / or other image information obtained by the driving support terminal model, and the one or more processors include one or more artificial intelligence processors, wherein the one or more processors are configured to implement a navigation program by executing one or more neural network models using the post-processed obtained image information and / or the post-processed other image information, wherein the one or more neural network models are executed using the one or more artificial intelligence processors that provide the post-processed obtained image information and / or the post-processed other image information.

18. The driving support system according to claim 1, wherein: The one or more processors are configured to connect to the communication port to obtain the driving information, and are configured to obtain the second image information from the other camera separated from the vehicle and communicating with the driving support terminal, and execute at least a part of the driving program based on the obtained driving information and the second image information.

19. The driving support system according to claim 18, wherein: The one or more processors are configured to generate an image of a periphery of the vehicle from the second image information and to provide the image of the periphery of the vehicle to the vehicle for display by the vehicle.

20. The driving support system according to claim 1, wherein: The driving support terminal includes an on-board diagnostic type connection terminal, and the driving support terminal is connected to the communication port using the on-board diagnostic type connection terminal.

21. The driving support system according to claim 1, wherein: The first image information is obtained from the vehicle, and / or the second image information is obtained from the other camera separated from the vehicle, and the one or more processors are also configured to use the first image information and / or the second image information and the driving information to implement at least part of a navigation program.

22. The driving support system according to claim 21, wherein: The driving support terminal is connected to the communication port using a Type-C USB connection terminal.

23. The driving support system according to claim 2, wherein: The driving support terminal includes: a memory storing the driving assistance program and / or the autonomous driving program; and The one or more processors include a neural processing unit configured to generate driving support information by executing the at least a portion of the driving program.

24. The driving support system according to claim 23, wherein: The neural processing unit comprises: an image receiver into which an image obtained from the camera system is input; an image preprocessor configured to extract a region of an image of interest from an image input to the image receiver; an object recognition neural network model implemented to recognize an object in the image of interest; and A driving support information generator is configured to execute at least a part of the driving assistance program and / or the autonomous driving program for the identified target.

Citation Information

Patent Citations

  • Method for Driver Assistance System of a Vehicle

    US20150094899A1

  • Autonomous ready vehicle

    US20160167653A1