Apparatus and method for controlling wiper based on image

Abstract

The image-based wiper control device of the present invention is characterized by comprising: a front camera that captures an image of the front; a wiper driving unit that drives a wiper; and a control module that receives an image captured by the front camera and adjusts the signal interval of a control signal input to the wiper driving unit through a pre-trained deep learning network to variably control the operating speed of the wiper.

Description

Technology Field

[0001] The present invention relates to an image-based wiper control device and method, and more specifically, to an image-based wiper control device and method that variably controls the operating speed of a wiper based on a front camera image. Background Technology

[0002] Vehicle rain sensors use LEDs (Light Emitting Diodes) to transmit infrared light to the windshield and use photodetectors to determine if there is a reflected signal.

[0003] When there are no raindrops, the vehicle rain sensor does not receive a signal because it moves in the opposite direction to the photodetector according to the law of reflection (the angle of incidence and the angle of reflection are equal).

[0004] However, if raindrops fall on the windshield or are falling in the atmosphere, the light transmitted from the infrared LED undergoes scattering and is reflected toward the photodetector, allowing a signal to be detected. As the density of raindrops increases, the probability of reception and the magnitude of the signal reflected by scattering also increase, making it possible to determine the amount of rainfall.

[0005] The background technology of the present invention is disclosed in Korean Published Patent Application No. 10-2021-0015194 (February 10, 2021), titled ‘Rain sensor for a vehicle, wiper system using the same, and wiper control method.’ The problem to be solved

[0006] The present invention was devised to improve upon the aforementioned problems, and an objective according to one aspect of the present invention is to provide an image-based wiper control device and method that variably controls the operating speed of a wiper based on a front camera image. means of solving the problem

[0007] An image-based wiper control device according to one aspect of the present invention is characterized by comprising: a front camera that captures an image of the front; a wiper driving unit that drives a wiper; and a control module that receives an image captured by the front camera and adjusts the signal interval of a control signal input to the wiper driving unit through a pre-trained deep learning network to variably control the operating speed of the wiper.

[0008] The control module of the present invention is characterized by comprising: a learning unit that inputs an image captured by the front camera into a pre-trained deep learning network and outputs a signal interval of a control signal input to the wiper driving unit to drive the wiper; and a control unit that inputs the control signal to the wiper driving unit according to the signal interval output from the learning unit.

[0009] The learning unit of the present invention is characterized by performing a network learning process that infers the signal interval of the control signal using previously stored learning data and updates deep learning network parameters so that the signal interval of the control signal and the previously stored correct answer data are identical.

[0010] The learning unit of the present invention is characterized by using the image before the point in time when the signal interval of the ETACS control signal output from the ETACS (Electronics, Time, Alarm, Control, System) changes and the image after the set time as the learning data.

[0011] The learning unit of the present invention is characterized by using the signal interval at the point in time when the signal interval of the ETACS control signal output from the ETACS changes as correct answer data.

[0012] A video-based wiper control method according to one aspect of the present invention is characterized by comprising: a step in which a front camera captures a front image; and a step in which a control module receives the image captured by the front camera and adjusts the signal interval of a control signal input to a wiper drive unit through a previously trained deep learning network to variably control the operating speed of the wiper.

[0013] The step of variablely controlling the operating speed of the wiper according to the present invention is characterized in that the control module inputs an image captured by the front camera into a learned deep learning network and outputs a signal interval of a control signal input to the wiper drive unit; and inputs a wiper relay control signal to the wiper drive unit according to the signal interval of the control signal.

[0014] The present invention is characterized by further including the step of performing a network learning process that infers the signal interval of a wiper relay control signal using previously stored training data and updates deep learning network parameters so that the inferred signal interval is identical to the previously stored correct answer data.

[0015] The step of performing the network learning process of the present invention is characterized by using the image before the point in time when the signal interval of the ETACS control signal output from the ETACS (Electronics, Time, Alarm, Control, System) changes and the image after the set time as the learning data.

[0016] The step of performing the network learning process of the present invention is characterized by using the signal interval at the point in time when the signal interval of the ETACS control signal output from the ETACS changes as correct data. Effects of the invention

[0017] An image-based wiper control device and method according to one aspect of the present invention recognizes a front camera image using a deep learning network and controls the speed of the wiper according to rainfall, thereby reducing the cost of the rain sensor and wiring.

[0018] An image-based wiper control device and method according to another aspect of the present invention can secure windshield space equal to the size of the rain sensor, thereby improving recognition performance by placing multiple sensors in optimal positions. Brief explanation of the drawing

[0019] FIG. 1 is a block diagram of an image-based wiper control device according to one embodiment of the present invention. FIG. 2 is a diagram showing a deep learning network according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating the process of creating a learning database according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating a deep learning network learning process according to an embodiment of the present invention. FIG. 5 is a flowchart of an image-based wiper control method according to an embodiment of the present invention. Specific details for implementing the invention

[0020] Hereinafter, an image-based wiper control device and method according to an embodiment of the present invention will be described in detail with reference to the attached drawings. In this process, the thickness of lines or the size of components shown in the drawings may be exaggerated for clarity and convenience of explanation. Furthermore, the terms described below are defined considering their functions in the present invention, and these may vary depending on the intention or convention of the user or operator. Therefore, the definitions of these terms should be based on the content throughout this specification.

[0021] FIG. 1 is a block diagram of an image-based wiper control device according to an embodiment of the present invention, FIG. 2 is a diagram showing a deep learning network according to an embodiment of the present invention, FIG. 3 is a flowchart illustrating a learning database creation process according to an embodiment of the present invention, FIG. 4 is a flowchart illustrating a deep learning network learning process according to an embodiment of the present invention, and FIG. 5 is a flowchart of an image-based wiper control method according to an embodiment of the present invention.

[0022] Referring to FIG. 1, an image-based wiper control device according to one embodiment of the present invention includes a front camera (10), a control module (20), and a wiper driving unit (30).

[0023] The front camera (10) captures the front of the vehicle and obtains an image of the front of the vehicle.

[0024] The wiper drive unit (30) drives the vehicle's wiper according to a control signal input from the control module (20).

[0025] The wiper drive unit (30) may be a wiper drive motor for driving the wiper, and the control signal may be a wiper relay control signal for controlling the wiper drive unit (30).

[0026] In this case, the wiper relay control signal is a PWM (Pulse Width Generation) signal, and the operating speed of the wiper is controlled according to the signal interval (wiper speed step) of the wiper relay control signal.

[0027] The wiper drive unit (30) can drive the rotation speed of the wiper in multiple steps, for example, 5 steps, according to the signal interval of the control signal input from the control module (20).

[0028] The control module (20) controls the wiper drive unit (30) through a pre-trained deep learning network using an image captured by the front camera (10).

[0029] That is, the control module (20) inputs a series of images from the front camera (10) into a deep learning network combined with a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN) to obtain the interval of the wiper relay control signal, thereby variably controlling the operation speed of the wiper.

[0030] The control module (20) includes a learning unit (21) and a control unit (22).

[0031] The learning unit (21) inputs an image captured by the front camera (10) into a pre-trained deep learning network and outputs the signal interval of the control signal input to the wiper driving unit (30) to drive the wiper.

[0032] The deep learning network is pre-trained under supervised learning to output the signal interval of the control signal.

[0033] To supervise a deep learning network, ground truth data consisting of images and control signals is required.

[0034] Accordingly, the learning unit (21) monitors the existing ETACS (Electronic, Time, Alarm, Control, System) control signal and matches the images before and after the point in time when the wiper speed level changes, and the signal interval, and stores them in the database.

[0035] The learning unit (21) reads an image from the generated database and inputs it into the deep learning network, and then updates the network parameters so that the interval between the output value of the deep learning network and the control signal of the database becomes equal.

[0036] The learning unit (21) stores the wiper operation signal and the front camera (10) image in a buffer.

[0037] Typically, the wiper is operated by controlling the voltage supplied to the wiper drive unit (30) in the ETACS using a relay.

[0038] The signal interval of the control signal changes according to the signal input from the rain sensor. Accordingly, the learning unit (21) monitors the ETACS control signal output from the existing ETACS and measures the signal interval.

[0039] The ETACS control signal is a wiper relay control signal input from the ETACS to the wiper drive unit (30).

[0040] The learning unit (21) stores the image (image stored in the buffer) before the point in time when the signal interval changes (change in wiper operation speed) and the image after a set time, for example, 1 second, as learning data.

[0041] At this time, the learning unit (21) uses the image before the point in time when the signal interval of the ETACS control signal output from the ETACS changes and the image after a set time from that point in time as input data, and uses the signal interval as correct answer data.

[0042] That is, as illustrated in FIG. 3, the learning unit (21) stores the signal interval and reception time of the ETACS control signal input from the ETACS to the wiper driving unit (30) in a buffer (S110).

[0043] Next, the video captured by the front camera (10) is received and the video is stored in a buffer along with the time of receipt (S120).

[0044] Next, the learning unit (21) determines whether the signal interval of the ETACS control signal has changed (S130).

[0045] If the signal interval of the ETACS control signal has changed as a result of the judgment in step S130, the learning unit (21) extracts the image before the point in time when the signal interval of the ETACS control signal changed, the image at the point in time after a set time has elapsed from that point in time, and the signal interval from the buffer (S140).

[0046] Next, the learning unit (21) matches the signal interval having the closest reception time to the time when the video was received (S150), and stores the video and reception time at the time when a set time has elapsed from the time when the video was received in the learning database (S160).

[0047] In this way, as the learning database is stored, the learning unit (21) infers the signal interval of the control signal using the learning data for the rain detection wiper operation, and performs a network learning process to update the deep learning network parameters so that the signal interval of the control signal and the previously stored correct answer data are the same.

[0048] That is, the learning unit (21) reads images from a database and inputs them into a CNN to extract features, and uses an RNN to update deep learning network parameters so that the signal interval of the control signal for a series of images becomes equal to the signal interval of the control signal stored in the database.

[0049] Referring to FIG. 2, the learning unit (21) receives a series of images from the front camera (10) and extracts features for each image in the base network. The base network uses layers up to an output stride of 16 in the classification network and performs learning by loading values ​​learned in ILSVRC 2012 as pretrained weights.

[0050] Features extracted from the base network pass through global average pooling to retain only the spatial average values, and are input into an LSTM to estimate the signal interval (wiper speed step) for consecutive images. Since the result of the LSTM is classification (wiper speed step), the wiper signal interval mapped to each image is read from the database and the cross-entropy loss is calculated. Then, the learning unit (21) updates the network parameters (weights) using the stochastic gradient descent (SGD) method.

[0051] That is, as illustrated in FIG. 4, the learning unit (21) reads an image from a database and inputs it into a deep learning network (S210).

[0052] The deep learning network compares the signal interval matched to the input image with the ground truth data in the aforementioned database, and updates the deep learning network parameters so that the signal interval matched to the input image and the ground truth data stored are identical.

[0053] After the learning is completed in this way, when an image is input from the front camera (10), the deep learning network of the learning unit (21) outputs the signal interval of the control signal as an LSTM result.

[0054] The control unit (22) inputs a control signal corresponding to the signal interval to the wiper driving unit (30) according to the signal interval input from the learning unit (21), thereby causing the wiper driving unit (30) to operate at the corresponding operating speed.

[0055] That is, as illustrated in FIG. 5, the front camera (10) captures an image, and the control module (20) receives the image captured by the front camera (10) (S310).

[0056] The deep learning network of the learning unit (21) receives an image captured by the front camera (10) and controls the signal interval of the control signal input to the wiper driving unit (30) as described above (S320).

[0057] Accordingly, the wiper drive unit (30) is controlled to a wiper speed corresponding to the current rainfall value (S330).

[0058] In this way, the image-based wiper control device and method according to one embodiment of the present invention recognizes a front camera image using a deep learning network and controls the speed of the wiper according to the amount of rainfall, thereby reducing the cost of the rain sensor and wiring.

[0059] In addition, the image-based wiper control device and method according to one embodiment of the present invention can secure windshield space equal to the size of the rain sensor, so recognition performance can be improved by placing multiple sensors in optimal positions.

[0060] The implementations described herein may be implemented, for example, as methods or processes, devices, software programs, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., devices or programs). Devices may be implemented in appropriate hardware, software, and firmware, etc. Methods may be implemented in devices such as processors, which generally refer to processing devices including, for example, computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices such as computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end-users.

[0061] Although the present invention has been described with reference to the embodiments illustrated in the drawings, this is merely illustrative and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the claims below. Explanation of the symbols

[0062] 10: Front camera 20: Control module 21: Learning unit 22: Control unit 30: Wiper drive unit

Claims

Claim 1 An image-based wiper control device comprising: a front camera for capturing an image of the front; a wiper driving unit for driving a wiper; and a control module that receives an image captured by the front camera and variably controls the operating speed of the wiper by adjusting the signal interval of a control signal input to the wiper driving unit through a pre-trained deep learning network, wherein the control module includes a learning unit that inputs the image captured by the front camera into a pre-trained deep learning network and outputs the signal interval of a control signal input to the wiper driving unit to drive the wiper, and wherein the learning unit infers the signal interval of the control signal using pre-stored learning data and performs a network learning process that updates deep learning network parameters so that the signal interval of the control signal and the pre-stored correct answer data are identical. Claim 2 An image-based wiper control device according to claim 1, wherein the control module further comprises a control unit that inputs the control signal to the wiper driving unit according to the signal interval output from the learning unit. Claim 3 delete Claim 4 An image-based wiper control device according to claim 1, wherein the learning unit uses an image before the point in time when the signal interval of an ETACS control signal output from an ETACS (Electronics, Time, Alarm, Control, System) changes and an image after a set time as the learning data. Claim 5 An image-based wiper control device according to claim 1, characterized in that the learning unit uses the signal interval at the point in time when the signal interval of the ETACS control signal output from the ETACS changes as correct answer data. Claim 6 A video-based wiper control method comprising: a step in which a front camera captures a front image; and a step in which a control module receives the image captured by the front camera and variably controls the operation speed of a wiper by adjusting the signal interval of a control signal input to a wiper drive unit through a previously trained deep learning network, wherein the step of variably controlling the operation speed of the wiper further comprises a step in which the control module inputs the image captured by the front camera into a trained deep learning network and outputs the signal interval of a control signal input to the wiper drive unit, and further comprises a step of performing a network learning process in which the signal interval of a wiper relay control signal is inferred using previously stored training data, and the deep learning network parameters are updated so that the inferred signal interval is identical to the previously stored correct answer data. Claim 7 A video-based wiper control method according to claim 6, wherein the step of variablely controlling the operating speed of the wiper further includes the step of inputting a wiper relay control signal to the wiper driving unit according to the signal interval of the control signal. Claim 8 delete Claim 9 In claim 6, the step of performing the network learning process is characterized by using an image before the point in time when the signal interval of an ETACS control signal output from an ETACS (Electronics, Time, Alarm, Control, System) changes and an image after a set time as the learning data. Claim 10 A video-based wiper control method according to claim 6, wherein the step of performing the network learning process is characterized by using the signal interval at the point in time when the signal interval of the ETACS control signal output from the ETACS changes as ground truth data.

Images