Video processing device and method for surroundings of vehicle

Abstract

To provide a video processing device and method for the surroundings of a vehicle.SOLUTION: A video processing device for the surroundings of a vehicle includes: a camera module disposed in the vehicle and including a plurality of cameras each having a wide-angle lens mounted thereon; and an electronic device which processes videos obtained via the camera module. The electronic device includes: a communication circuit; a video input unit which receives the videos of the surroundings of the vehicle transmitted by the communication circuit; a video conversion unit which corrects the videos of the surroundings of the vehicle received by the video input unit; and a control unit for controlling the communication circuit, the video input unit, and the video conversion unit. The video conversion unit may be controlled by the control unit so that a circularly warped overview screen captured by the camera module is corrected to a plane image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a vehicle peripheral video processing apparatus and method, and more particularly, to a vehicle peripheral video processing apparatus and method capable of correcting and outputting a video of a vehicle periphery in a circularly distorted form captured by a wide-angle lens mounted on a side mirror of a vehicle into a planar image.

Background Art

[0002] Side view mirrors disposed on the left and right sides of a vehicle are composed of mirrors for checking peripheral situations such as changing lanes or checking surrounding vehicles and pedestrians.

[0003] However, side view mirrors of vehicles have a demerit that a blind spot is generated due to a narrow viewing angle.

[0004] To improve such demerits, the function of a Digital Side Mirror (DSM) that omits conventional side view mirrors disposed on a vehicle and uses a camera and a display instead has been developed.

[0005] A Digital Side Mirror (DSM) has a camera installed on the left and right sides of a vehicle instead of a side view mirror, and since the video captured through the camera can be confirmed via a display installed inside the vehicle, a clear and wide-angle rear view can be obtained.

[0006] However, due to the characteristic of providing a rear view with a camera and a monitor, a Digital Side Mirror (DSM) has no way to secure a rear view when a system error occurs.

[0007] Therefore, recently, various technologies have been developed that can conveniently check the blind spot of a vehicle's side mirror.

Summary of the Invention

Problems to be Solved by the Invention

[0008] One of the various problems of the present invention is to provide a vehicle peripheral video processing apparatus and method capable of substituting functions performed by a plurality of cameras by photographing the periphery of the vehicle with a camera equipped with a wide-angle lens having a wide viewing angle.

[0009] One of the various problems of the present invention is to provide a vehicle peripheral video processing apparatus and method capable of correcting and outputting an overview screen in a circularly bent form to a planar image so that the state of the periphery of the vehicle can be easily confirmed.

[0010] One of the various problems of the present invention is to provide a vehicle peripheral video processing apparatus and method using a wide-angle lens that can output an area suitable for a driving mode by dividing and correcting a photographed video of the vehicle periphery by area.

[0011] One of the various problems of the present invention is to provide a vehicle peripheral video processing apparatus and method capable of increasing a monitoring function by confirming the state of the periphery of the vehicle using a camera equipped with a wide-angle lens having a wide viewing angle.

Means for Solving the Problems

[0012] A vehicle peripheral video processing apparatus, comprising: a camera module installed in a vehicle and including a plurality of cameras equipped with wide-angle lenses; and an electronic device that processes video acquired via the camera module, wherein the electronic device includes a communication circuit, a video input unit that receives transmission of video of the periphery of the vehicle photographed via the communication circuit, a video conversion unit that corrects the video of the periphery of the vehicle received by the video input unit, and a control unit for controlling the communication circuit, the video input unit, and the video conversion unit, and the video conversion unit can be controlled by the control unit so that an overview screen in a circularly bent form photographed by the camera module is corrected to a planar image.

[0013] The camera module may include a first camera installed on the left side mirror of the vehicle, a second camera installed on the right side mirror of the vehicle, a third camera that captures the front area of the vehicle, and a fourth camera that captures the rear area of the vehicle.

[0014] The video conversion unit can be controlled by the control unit to classify and divide the received video of the vehicle surroundings according to the shooting area of the vehicle into multiple parts.

[0015] The video of the vehicle surroundings is divided into the front area, rear area, left side area, and right side area of the vehicle based on the mounting point of the wide-angle lens installed on the side mirror of the vehicle, and the left side area and the right side area can be further divided into upper and lower end areas respectively.

[0016] The video conversion unit can be controlled to correct a part of the divided multiple videos of the vehicle surroundings into a planar image.

[0017] The video conversion unit can be controlled to selectively merge a part of the divided multiple videos of the vehicle surroundings and correct them into one planar image.

[0018] It further includes a sensor unit for sensing the internal and external environments of the vehicle, and the sensor unit can be controlled by the control unit to sense the driving mode of the vehicle.

[0019] The driving mode may include a forward mode, a reverse mode, a left turn mode, and a right turn mode.

[0020] The video conversion unit can be controlled to selectively merge a part of the multiple videos of the vehicle surroundings divided according to the driving mode of the vehicle and correct them into one planar image.

[0021] When the driving mode is the forward mode, the video conversion unit can be controlled by the control unit to merge the videos of the vehicle surroundings captured by the first camera, the second camera, and the fourth camera, respectively, and correct them into one planar image.

[0022] When the driving mode is the reverse mode, the video conversion unit can be controlled to merge the videos of the vehicle surroundings captured by the first camera, the second camera, and the fourth camera, respectively, correct them into one planar image, and extract and merge only the lower end regions of the left side region and the right side region, respectively.

[0023] When the driving mode is the left turn mode, the video conversion unit can be controlled to correct the video of the vehicle surroundings captured by the first camera into one planar image.

[0024] When the driving mode is the right turn mode, the video conversion unit can be controlled to correct the video of the vehicle surroundings captured by the second camera into one planar image.

[0025] A method for processing a video of the vehicle surroundings, the video processing method including: sensing a driving mode of the vehicle; using a camera module equipped with a wide-angle lens to capture the surroundings of the vehicle; receiving the captured video of the vehicle surroundings; correcting the received video of the vehicle surroundings; and outputting the corrected video of the vehicle surroundings, wherein the corrected video of the vehicle surroundings may be a video or an image in which an overview screen in a form that is curved in a circular shape is corrected into a planar image.

[0026] The camera module includes a first camera and a second camera respectively installed on the left and right side mirrors of the vehicle, a third camera, and a fourth camera, and the captured video of the vehicle surroundings can include the videos captured by the first camera and the second camera, the third camera, and the fourth camera, respectively.

[0027] The correction step can be controlled to selectively merge a part of the images of the vehicle surroundings divided into a plurality of parts after dividing the received images of the vehicle surroundings according to the driving mode of the vehicle, and correct them into one planar image.

[0028] When the driving mode is the forward mode, the correction step can be controlled to merge the images of the vehicle surroundings taken by the first camera, the second camera, and the fourth camera respectively, and correct them into one planar image.

[0029] When the driving mode is the reverse mode, the correction step can be controlled to merge the images of the vehicle surroundings taken by the first camera, the second camera, and the fourth camera respectively, correct them into one planar image, and only the lower end regions of the images taken by the first and second cameras can be selectively extracted and merged.

[0030] On the other hand, a computer-readable recording medium according to an embodiment of the present invention for achieving the above object can record a program for executing a method for processing images of the vehicle surroundings.

[0031] In addition, a computer program according to an embodiment of the present invention for achieving the above object can include program codes for executing a method for processing images of the vehicle surroundings.

Effect of the Invention

[0032] The apparatus and method for processing images of the vehicle surroundings according to an exemplary embodiment of the present invention photograph the vehicle surroundings at a wide angle of view with a camera equipped with a wide-angle lens, and by substituting the functions performed by a plurality of cameras, an economic effect can be obtained from the reduction in weight and cost due to the reduction in the number of cameras.

[0033] The vehicle surrounding video processing apparatus and method according to an exemplary embodiment of the present invention can easily confirm the state of the vehicle surroundings by correcting and outputting an overview screen in a circularly distorted form, which is captured by a wide-angle lens, into a planar image.

[0034] The vehicle surrounding video processing apparatus and method according to an exemplary embodiment of the present invention can increase convenience by dividing and correcting the captured video by region so that a region suitable for the driving mode is output.

[0035] The vehicle surrounding video processing apparatus and method according to an exemplary embodiment of the present invention can increase the monitoring function because the state of the vehicle surroundings can be confirmed with a wide viewing angle of 360° and no blind spots are generated.

Brief Description of the Drawings

[0036]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0037] Hereinafter, specific embodiments of the present invention will be described. The following detailed description is provided to assist in a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely an example and the present invention is not limited thereto.

[0038] In describing embodiments of the present invention, when it is determined that a specific description of a known technique related to the present invention may obscure the gist of the present invention, the detailed description thereof will be omitted. Further, the terms described below are terms defined in consideration of the functions in the present invention, and these may vary depending on the intention or convention of the user, operator, etc. Therefore, the definition should be made based on the contents throughout this specification. The terms used in the detailed description are merely for describing the embodiments of the present invention and should never be restrictive. Unless otherwise clearly stated, the singular form of the expression includes the meaning of the plural form. In this description, expressions such as "including" or "comprising" are for indicating a certain characteristic, number, step, operation, element, part thereof, or combination thereof, and should not be construed so as to exclude the existence or possibility of one or more other characteristics, numbers, steps, operations, elements, part thereof, or combination thereof other than those described.

[0039] Also, in describing the components of the embodiments of the present invention, terms such as first, second, A, B, (a), (b), etc. can be used. These terms are for distinguishing the components from other components, and the essence, procedure, order, etc. of the components are not limited by these terms.

[0040] FIG. 1 is a configuration diagram for explaining a vehicle peripheral video processing apparatus 1 according to an exemplary embodiment of the present invention.

[0041] Referring to FIG. 1, the vehicle peripheral video processing apparatus 1 can include a camera module 10 and an electronic device 20. The video processing apparatus 1 can be mounted on a vehicle. In some embodiments, at least one other component may be added to the video processing apparatus 10.

[0042] The camera module 10 can include first and second cameras 110 and 120 equipped with wide-angle lenses, a third camera 130 that captures the front area of the vehicle, and a fourth camera 140 that captures the rear area of the vehicle.

[0043] The first camera 110 and the second camera 120 can obtain a wide angle of view by using an ultra-wide angle lens such as a fish-eye lens. The first camera 110 is installed on the left side mirror of the vehicle and can photograph the entire left side with an angle of view of about 180°. The second camera 120 is installed on the right side mirror of the vehicle and can photograph the entire right side with an angle of view of about 180°. Depending on the attached positions, the first camera 110 and the second camera 120 can photograph the entire 180° region in front of the shooting direction.

[0044] The third camera 130 may be a front camera that photographs the front region of the vehicle, and the fourth camera 140 may be a rear camera that photographs the rear region of the vehicle. The third camera 130 can photograph the front of the vehicle with an angle of view of about 180°, and the fourth camera 140 can photograph the rear of the vehicle with an angle of view of about 180°. However, it is not limited thereto, and the shooting angles of view of the third camera 130 and the fourth camera 140 may be narrower than the angles of view of the first camera 110 and the second camera 120.

[0045] The camera module 10 can include at least one of one or more lenses, an image sensor, a flash, and an image signal processor (ISP). Some of the lenses can have the same lens attributes (e.g., angle of view, focal length, autofocus, f number, or optical zoom). The lenses can include a light source lens or a telephoto lens. The image sensor can obtain an image corresponding to a subject by converting light emitted or reflected from the subject and transmitted through one or more lenses into an electrical signal. For example, the image sensor can be one image sensor selected from image sensors with different attributes, such as an RGB sensor, a BW (black and white) sensor, an IF sensor, or a UV sensor, multiple image sensors with the same attribute, or multiple image sensors with different attributes. Each image sensor can be realized using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor. The flash can include one or more light emitting diodes (e.g., RGB (red - green - blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp.

[0046] Based on the video obtained by the camera module 10, the electronic device 20 can perform video processing.

[0047] The electronic device 20 can include a communication circuit 210, a video input unit 220, a control unit 230, a video conversion unit 240, a memory 250, and a sensor unit 260.

[0048] Referring to FIG. 1, the communication circuit 210, the video input unit 220, the control unit 230, and the video conversion unit 240 can be electrically and / or operatively coupled to each other by electronic components and / or electrical components such as the communication bus 205. Hereinafter, the operative coupling of the electronic components may mean that a direct or indirect connection between the first electronic component and the second electronic component is established, either wired or wirelessly, such that the second electronic component is controlled by the first electronic component. Although illustrated based on different blocks, the embodiments are not limited thereto, and a part of the electronic components in FIG. 1 (e.g., at least a part of the control unit 230 and the video conversion unit 240) can be included in a single integrated circuit such as an SoC (system on a chip). The type and / or number of the electronic components included in the electronic device 20 are not limited to those illustrated in FIG. 1. For example, the electronic device 20 may include only a part of the electronic components illustrated in FIG. 1.

[0049] The communication circuit 210 of the electronic device 20 can receive images of the vehicle surroundings captured by the first and second cameras 110 and 120, the third camera 130, and the fourth camera 140. Here, the camera module 10 and the electronic device 20 can be communicably connected by wire or wirelessly. For example, the camera module 10 and the electronic device 20 can be connected via a communication cable. For example, the camera module 10 and the electronic device 20 can communicate in an analog manner. The analog manner can include, for example, AHD (analogue high definition). For example, the camera module 10 and the electronic device 20 can communicate in a digital manner. The digital manner can include a serial transmission manner. In such a case, the camera module 10 can include a serializer, and the electronic device 20 can include a deserializer. For example, the camera module 10 and the electronic device 20 can be connected via in-vehicle network communication (e.g., CAN communication). For example, the camera module 10 and the electronic device 20 can include various communication chips.

[0050] The video input unit 220 of the electronic device 20 can receive the video transmitted by the communication circuit 210 from the first to fourth cameras 110, 120, 130, and 140. The video input unit 220 can receive the transmission of an overview video in a form that bends circularly from the first and second cameras 110 and 120, and can receive the transmission of a video including a planar image from the third camera 130 and the fourth camera 140. Among all the original videos received by the video input unit 220 from the camera module 10, the videos of the areas not output to the display 40 described later can be stored in the memory 250 by the control unit 230. The videos stored in the memory 250 can then be corrected by the video conversion unit 240 and output as needed.

[0051] The control unit 230 can include a circuit (e.g., a processing circuit) for processing data. The circuit for processing data can include, for example, an ALU (arithmetic and logic unit), an FPU (floating point unit), an FPGA (field programmable gate array), a CPU (central processing unit), a GPU (graphic processing unit), an NPU (neural processing unit), and / or an AP (application processor). For example, the number of control units 230 may be one or more. The processing circuit of the control unit 230 may be referred to as, or be referenced as, a core circuit (or a core). For example, the control unit 230 can have a multi-core processor structure that includes multiple core circuits, such as a dual core, a quad core, a hexa core, or an octa core. The functions and / or operations described with reference to the present disclosure can be performed individually and / or collectively by one or more processing circuits included in the processor.

[0052] The video conversion unit 240 can be controlled by the control unit 230 such that the video input unit 220 corrects the overview screen in a circularly distorted form received from the first and second cameras 110 and 120 into a planar image, and synthesizes the corrected planar image with the planar images received from the third camera 130 and the fourth camera 140. The process of the video conversion unit 240 correcting the image will be described later.

[0053] The memory 250 can include a circuit for storing data input to or output from the control unit 230. The memory 250 can include, for example, a volatile memory such as a RAM (random-access memory) and / or a non-volatile memory such as a ROM (read-only memory). The non-volatile memory may be referred to as storage. The volatile memory can include, for example, at least one of a DRAM (dynamic RAM), an SRAM (static RAM), a Cache RAM, and a PSRAM (pseudo SRAM). The non-volatile memory can include, for example, at least one of a PROM (programmable ROM), an EPROM (erasable PROM), an EEPROM (electrically erasable PROM), a flash memory, a hard disk, a compact disk, an SSD (solid state drive), and an eMMC (embedded multi media Card). The memory 250 can include one or more recording media (e.g., the above-described volatile memory and / or non-volatile memory) located in a distributed manner in the electronic device 20.

[0054] The sensor unit 260 can sense the internal environment and the external environment of the vehicle. Specifically, it can sense the internal environment including information such as gear shifting of the vehicle and the moving direction of the vehicle, and the external environment including information such as the speed and size of an object approaching from outside the vehicle. The sensor unit 260 can collect the internal environment and external environment information and transmit it to the control unit 230 to control the driving mode related to driving or parking of the vehicle.

[0055] FIG. 2 is a flowchart showing the operation of the electronic device 20 that converts the captured video into a planar video and / or an image.

[0056] Referring to FIG. 2, in the S1 operation, the electronic device 20 can receive an image of the vehicle surrounding environment from the camera module 10 via the communication circuit 210. For example, the first camera 110 and the second camera 120 can be included in the vehicle and arranged to face the surrounding environment of the vehicle. For example, the surrounding environment can be described as the actual environment. For example, the surrounding environment can include the vehicle. For example, the surrounding environment can include external objects (e.g., pedestrians, etc.). For example, the first camera 110 and the second camera 120 can acquire an image of the surrounding environment. For example, the first camera 110 and the second camera 120 can acquire a video of the surrounding environment. The first camera 110 and the second camera 120 can transmit the acquired images and / or videos to the video input unit 220.

[0057] In the S2 operation, the electronic device 20 can select an image and / or a video for correcting the image and / or the video received from the camera module 10. For example, the selected video for performing video correction may be the video acquired via the first camera 110 and the second camera 120. The content regarding video correction will be described in detail and illustrated with reference to FIGS. 3 and 4.

[0058] FIGS. 3 and 4 are diagrams for explaining that a video conversion unit according to an exemplary embodiment of the present invention converts a video taken using a wide-angle lens into a planar image.

[0059] In the S3 operation (see FIG. 2), the video conversion unit 240 can convert the hemispherical overview screen into a flat image video by using the Dewarping technology. The Dewarping technology can be composed of the processes of distortion modeling, coordinate transformation, video reconstruction, and video output. First, the distortion characteristics generated by the wide-angle lens can be analyzed and mathematically modeled. Generally, since the distortion at the edge of the image is more severe, a lens distortion equation for correcting this can be utilized. Each pixel of the video captured by the wide-angle lens is located at a distorted coordinate, so it is necessary to convert this to a coordinate on the plane, and using the spherical coordinate system, the position of each pixel can be converted to a rectangular coordinate system on the plane. Here, the position of each pixel can be readjusted to remove the distortion, and after the conversion, for pixels that are otherwise empty, the hue value can be estimated and a smooth image video can be generated.

[0060] Referring to FIGS. 3 and 4, FIGS. 3(a) and 4(a) are the original images received by the video input unit 220. The original of the video captured by the wide-angle lens can be transmitted as a hemispherical distorted image. The video conversion unit 240 can utilize the Dewarping technology to divide such a distorted image by region and correct it to a plane. FIGS. 3(b), (c), (d) and FIGS. 4(b), (c), (d) are the images obtained by converting the front side, rear side, and lower side end faces of the received original image to a plane in sequence. The hemispherical distorted video captured by the wide-angle lens can be corrected to a flat image. Here, the corrected flat image can be transmitted to the display 40. The user (e.g., the driver) can view the images of the front side, rear side, and lower side end faces through the display 40. However, it is not limited to this, and the video corrected to a plane can perform a compositing operation and be transmitted to the display 40.

[0061] In the S4 operation, the images corrected to a plane can be synthesized and combined into a single planar image as shown in FIGS. 3(e) and 4(e). The combined planar image can be transmitted to the display 40. A screen can be displayed via the display 40. For example, the screen can include an image of the surrounding area of the vehicle provided when the vehicle is traveling or parked. For example, the screen can include content that warns the user (e.g., the driver) of a collision risk. For example, the screen can include text that indicates danger.

[0062] FIG. 5 is a diagram for explaining the mounting position of the camera module 10 and the division of the imaging area of the captured image according to an exemplary embodiment of the present invention.

[0063] Referring to FIG. 5, the video conversion unit 240 can be controlled to classify and divide the video of the surrounding area of the vehicle received by the video input unit 220 according to the imaging area of the vehicle into a plurality of parts. The video of the surrounding area of the vehicle can be divided into areas of the front, rear, left side, and right side of the vehicle based on both side mirrors of the vehicle on which the first and second cameras 110 and 120 including wide-angle lenses are mounted.

[0064] Specifically, the first camera 110 mounted on the left side mirror can capture the left side of the vehicle at an angle of view of about 180° (areas a and b), and the second camera 120 mounted on the right side mirror can capture the right side of the vehicle at an angle of view of about 180° (areas d and e). In addition, the front area (area f) can be captured by the third camera 130, and the rear area (area c) can be captured by the fourth camera 140.

[0065] In addition, the front area of the vehicle can include an area that combines the front part of the vehicle (areas a, f, e) based on the locations where the first and second cameras 110 and 120 are installed, and the rear area of the vehicle can include an area that combines the rear part of the vehicle (areas b, c, d) based on the locations where the first and second cameras 110 and 120 are installed. Therefore, the left and right sides can be subdivided and divided into a left front area (area a), a left rear area (area b), a right front area (area e), and a right rear area (area d) with respect to the first and second cameras 110 and 120.

[0066] In addition, the left side area and the right side area photographed by the first and second cameras 110 and 120 can be further divided into an upper end area and a lower end area respectively. Based on the side mirrors where the first and second cameras 110 and 120 are installed, the area vertically above can be divided into the upper end area, and the area vertically below can be divided into the lower end area.

[0067] As an embodiment, the left side area photographed by the first camera 110 can be divided into a left front area (area a) and a left rear area (area b), and each can be further divided into an upper end and a lower end area, and can be divided into a left front upper end area (area a-2), a left front lower end area (area a-1), a left rear upper end area (area b-2), and a left rear lower end area (area b-1).

[0068] Although not shown in the drawings, the right side area photographed by the second camera 120 can also be further divided into an upper end and a lower end area, and can be classified into a right front upper end area, a right front lower end area, a right rear upper end area, and a right rear lower end area.

[0069] On the other hand, a part of the images of the periphery of the divided vehicle can be selected as an image for image correction. After the selected image is corrected for image (distortion) using the dewarping technique, the corrected image and the planar image can be synthesized and corrected into one planar image.

[0070] In other embodiments, a part of the divided images around the vehicle can be selectively merged and corrected into one planar image.

[0071] In one embodiment, when receiving images of the left rear area (area b), rear area (area c), and right rear area (d) of the vehicle, the images of the left rear area (area b) and right rear area (d) of the vehicle that need to be corrected into the planar image, after correcting them into the planar image, can be combined with the image of the rear area (area c) and output to the planar image.

[0072] In other embodiments, by selecting and merging the images of the left rear area (area b), rear area (area c), and right rear area (area d) of the vehicle and correcting them into the planar image, the image of the rear area of the vehicle can be output as a smooth single image.

[0073] As yet another embodiment, when receiving images of the lower left rear area, lower rear area, and lower right rear area of the vehicle, the images of the lower left rear area and lower right area of the vehicle that need to be corrected into the planar image, after correcting them into the planar image, can be combined with the image of the lower rear area and output to the planar image.

[0074] As yet another embodiment, by selecting and merging the images of the lower left rear area, lower rear area, and lower right rear area of the vehicle and correcting them into the planar image, the lower rear area of the vehicle can be output.

[0075] As yet another embodiment, in order to check the blind spot area of the front part of the vehicle, when receiving images of the lower left front area, lower front area, and lower right rear area of the vehicle, the images of the lower left front area and lower right rear area that need to be corrected into the planar image, after correcting them into the planar image, can be combined with the image of the lower front area and output to the planar image.

[0076] As yet another embodiment, in order to check the blind spot area of the front part of the vehicle, images of the lower left front area, the lower front area, and the lower right rear area of the vehicle are selected and merged, and corrected into a planar image, so that the lower front area of the vehicle can be output.

[0077] FIG. 6 is a simplified block diagram of the control unit 230 of the electronic device 20 and the modules included in the vehicle according to an exemplary embodiment of the present invention.

[0078] Referring to FIG. 6, the vehicle can include a plurality of modules. For example, it can include an alarm device 30 and a display 40. Here, the vehicle can also include a camera module 10 among the plurality of modules. However, it is not limited thereto.

[0079] The plurality of modules can be executed by the control unit 230 of the electronic device 20.

[0080] The alarm device 30 can receive alarm information from the control unit 230 of the electronic device 20. For example, the alarm device 30 can use a buzzer, a speaker, and an LED display lamp to transmit the alarm information to the user (e.g., the driver) together with an image. Here, the alarm information may be danger information regarding an external object.

[0081] The display 40 can be used to display the image corrected by the electronic device 20. The control unit 230 of the electronic device 20 can transmit the corrected image to the display 40. The user (e.g., the driver) can select and view the images divided according to the driving mode by touching the screen on the display 40.

[0082] FIGS. 7 and 8 are diagrams for explaining the areas photographed by the camera equipped with a wide-angle lens according to an exemplary embodiment of the present invention according to the driving mode.

[0083] The control unit 230 of the electronic device 20 can receive information on the internal and external environments of the vehicle from the sensor unit 260. When the sensor unit 260 senses the driving mode, the video conversion unit 240 can be controlled to selectively merge a part of the video around the vehicle divided into a plurality of parts according to the driving mode of the vehicle and correct it into one planar image. The driving mode can include a forward mode 510, a reverse mode 520, a left turn mode, and a right turn mode.

[0084] When the driving mode is the forward mode 510, the video conversion unit 240 can be controlled by the control unit 230 to merge the videos around the vehicle taken by the first camera 110, the second camera 120, and the fourth camera 140 respectively so as to correct them into one planar image, so that the rear area of the vehicle can be checked.

[0085] In one embodiment, referring to FIGS. 7 and 8, when the vehicle is running, the left and right sides can be checked by the videos taken by the first and second cameras 110 and 120. Also, when only the rear area of the vehicle is to be checked, the video conversion unit 240 can be controlled according to the user's selection to merge the videos of the left rear area, the rear area, and the right rear area and output them as one corrected image.

[0086] When the driving mode is the reverse mode 520, the video conversion unit 240 merges the videos around the vehicle taken by the first camera 110, the second camera 120, and the fourth camera 140 respectively so as to correct them into one planar image, so that the lower end areas on the left and right sides and the rear area of the vehicle can be checked. However, only the lower end areas of the left side area and the right side area are extracted and merged respectively.

[0087] As an embodiment, referring to FIG. 8, when the vehicle is parked or moving backward, the video conversion unit 240 can be controlled to merge the images of the lower end area and the rear area among the areas photographed by the first and second cameras 110 and 120, and output them as a single corrected image. Also, when it is desired to check the entire rear area of the vehicle, the video conversion unit 240 can be controlled to merge the images of the left rear area, the rear area, and the right rear area according to the user's selection and output them as a single corrected image.

[0088] When the driving mode is the left turn mode, the video conversion unit 240 can be controlled to correct the image of the periphery of the vehicle photographed by the first camera 110 into a planar image so that the left side area of the vehicle can be checked.

[0089] When the driving mode is the right turn mode, the video conversion unit 240 can be controlled to correct the image of the periphery of the vehicle photographed by the second camera 120 into a planar image so that the right side area of the vehicle can be checked.

[0090] FIG. 9 is a flowchart for explaining a method of processing an image of the periphery of a vehicle photographed using a wide-angle lens according to an exemplary embodiment of the present invention.

[0091] Referring to FIG. 9, a method of processing an image of the periphery of a vehicle photographed using a wide-angle lens can be performed by a driving mode sensing step (S10), a vehicle peripheral photographing step (S20), a video receiving step (S30), a video correcting step (S40), and a display output step (S50).

[0092] In the driving mode sensing step (S10), the sensor unit 260 can sense the forward mode 510, the reverse mode 520, the left turn mode, and the right turn mode of the vehicle, and the camera module 10 can be activated so that an appropriate area can be photographed according to each driving mode.

[0093] The surrounding shooting step (S20) of the vehicle can shoot the surroundings of the vehicle using the camera module 10 equipped with a wide-angle lens. In the forward mode 510 and the reverse mode 520, the first camera 110, the second camera 120, and the fourth camera 140 can shoot. In the left turn mode, the first camera 110 can shoot, and in the right turn mode, the second camera 120 can shoot.

[0094] The video reception step (S30) can receive the video of the surroundings of the vehicle taken. The video input unit (20) can receive an overview screen in a circularly curved form from the first and second cameras 110 and 120 equipped with wide-angle lenses.

[0095] The video correction step (S40) can correct the received video according to the driving mode. The video conversion unit 240 can be controlled to divide the received video of the surroundings of the vehicle into a plurality of parts, then selectively merge a part of the video of the surroundings of the vehicle divided into a plurality of parts, and correct it into one planar image.

[0096] In one embodiment, when the driving mode is the forward mode 510, the video corresponding to the rear area among the videos taken by the first and second cameras 110 and 120 and the video of the fourth camera 140 can be merged and corrected into one planar image.

[0097] In still another embodiment, when the driving mode is the reverse mode 520, the video corresponding to the lower end area of the rear area among the videos taken by the first and second cameras 110 and 120 and the video of the fourth camera 140 can also be merged and corrected into one planar image.

[0098] The display output step (S50) can output the corrected video of the surroundings of the vehicle to the LCD screen. The video of the surroundings of the vehicle can be output together with alarm devices such as a buzzer, a speaker, and an LED display lamp, and the video can be stored in another storage device.

[0099] As described above, referring to FIGS. 1 to 8, a vehicle surrounding video processing apparatus 1 and method according to an exemplary embodiment of the present invention captures the surrounding of the vehicle with a wide-angle lens at a wide angle of view, and by substituting the functions performed by a plurality of cameras, an economic effect can be obtained from a reduction in weight and cost due to a reduction in the number of cameras.

[0100] A vehicle surrounding video processing apparatus 1 and method according to an exemplary embodiment of the present invention can easily confirm the state of the vehicle surroundings by correcting and outputting an overview screen in a circularly distorted form into a planar image, which is captured with a wide-angle lens.

[0101] A vehicle surrounding video processing apparatus 1 and method according to an exemplary embodiment of the present invention can increase convenience by dividing and correcting the captured video by region so that a region suitable for the driving mode is output.

[0102] A vehicle surrounding video processing apparatus 1 and method according to an exemplary embodiment of the present invention can increase the monitoring function because the state of the vehicle surroundings can be confirmed at a wide angle of view of 360° and no blind spot area is generated.

[0103] Hereinafter, an autonomous driving system, an autonomous driving mobile body, and a user device using various information, data, videos, etc. collected by a driving video recording system 1000 according to the present invention will be described more specifically with reference to FIGS. 10 to 13.

[0104] FIG. 10 is a diagram illustrating an example of a block diagram showing an autonomous driving system of a vehicle according to an embodiment.

[0105] The autonomous driving system 800 of the vehicle according to FIG. 10 may be a deep learning network including a sensor 803, an image pre-processor 805, a deep learning network 807, an artificial intelligence (AI) processor 809, a vehicle control module 811, a network interface 813, and a communication unit 815. In various embodiments, each element can be connected via various interfaces. For example, the sensor data sensed and output by the sensor 803 can be fed to the image pre-processor 805. The sensor data processed by the image pre-processor 805 can be fed to the deep learning network 807 executed by the AI processor 809. The output of the deep learning network 807 executed by the AI processor 809 can be fed to the vehicle control module 811. The intermediate result of the deep learning network 807 executed by the AI processor 809 can be fed to the AI processor 809. In various embodiments, the network interface 813 transmits autonomous driving route information and / or autonomous driving control commands for the autonomous driving of the vehicle to the internal block configuration by communicating with the electronic devices in the vehicle. In one embodiment, the network interface 813 can be used to transmit the sensor data acquired by the sensor 803 to an external server. In some embodiments, the autonomous driving control system 800 can appropriately include additional or fewer components. For example, in some embodiments, the image pre-processor 805 may be an optional component. According to another example, a post-processing component (not shown) can be included in the autonomous driving control system 800 to perform post-processing on the output of the deep learning network 807 before the output is provided to the vehicle control module 811.

[0106] In some embodiments, sensor 803 can include one or more sensors. In various embodiments, sensor 803 can be attached at different locations of the vehicle. Sensor 803 can face one or more different directions. For example, sensor 803 can be attached to the front, sides, rear, and / or roof of the vehicle so as to face directions such as forward-facing, rear-facing, side-facing, etc. In some embodiments, sensor 803 can be an image sensor such as high dynamic range cameras. In some embodiments, sensor 803 includes non-visual sensors. In some embodiments, sensor 803 includes, in addition to the image sensor, RADAR (Radio Detection And Ranging), LiDAR (Light Detection And Ranging), and / or ultrasonic sensors. In some embodiments, sensor 803 is not mounted on a vehicle having vehicle control module 811. For example, sensor 803 can be included as part of a deep learning system for capturing sensor data and can be attached to the environment or road and / or mounted on a surrounding vehicle.

[0107] In some embodiments, an Image pre-processor 805 can be used to pre-process the sensor data of sensor 803. For example, the Image pre-processor 805 can be used to split the sensor data with one or more components, and / or to post-process one or more components, for pre-processing the sensor data. In some embodiments, the Image pre-processor 805 may be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the Image pre-processor 805 may be a tone-mapper processor for processing high dynamic range data. In some embodiments, the Image pre-processor 805 may be a component of the AI processor 809.

[0108] In some embodiments, a Deep learning network 807 may be a deep learning network for realizing control instructions for controlling an autonomous vehicle. For example, the Deep learning network 807 may be an artificial neural network such as a convolutional neural network (CNN) trained using sensor data, and the output of the Deep learning network 807 is provided to the vehicle control module 811.

[0109] In some embodiments, the artificial intelligence (AI) processor 809 may be a hardware processor for running the deep learning network 807. In some embodiments, the AI processor 809 is a specialized AI processor for performing inference by a convolutional neural network (CNN) on sensor data. In some embodiments, the AI processor 809 can be optimized for the bit depth of the sensor data. In some embodiments, the AI processor 809 can be optimized for deep learning operations such as neural network operations including convolution, inner product, vector, and / or matrix operations. In some embodiments, the AI processor 809 can be implemented by a plurality of graphics processing units (GPUs) that can effectively perform parallel processing.

[0110] In various embodiments, the AI processor 809 can be coupled via an input / output interface to a memory configured to provide an AI processor having instruction words that induce the AI processor 809 to perform deep learning analysis on sensor data received from the sensor 803 during execution of the AI processor 809 and to determine machine learning results used to at least partially autonomously operate the vehicle. In some embodiments, a Vehicle Control Module 811 can process instructions for vehicle control output from the artificial intelligence (AI) processor 809 and can be used to translate the output of the AI processor 809 into instruction words for controlling each module of the vehicle in order to control various modules of the vehicle. In some embodiments, the vehicle control module 811 is used to control a vehicle for autonomous driving. In some embodiments, the vehicle control module 811 can adjust the steering and / or speed of the vehicle. For example, the vehicle control module 811 can be used to control the driving of the vehicle such as decelerating, accelerating, steering, changing lanes, maintaining lanes, etc. In some embodiments, the vehicle control module 811 can generate control signals for controlling vehicle lighting such as brake lights, turn signals, headlights, etc. In some embodiments, the vehicle control module 811 can be used to control vehicle audio related systems such as the vehicle's sound system, the vehicle's audio warnings, the vehicle's microphone system, the vehicle's horn system, etc.

[0111] In some embodiments, the vehicle control module 811 can be used to control notification systems, including a warning system for notifying passengers and / or drivers of driving events such as the approach of an intended destination or a potential collision. In some embodiments, the vehicle control module 811 can be used to adjust sensors such as the vehicle sensors 803. For example, the vehicle control module 811 can modify the orientation of the sensor 803, change the output resolution and / or format type of the sensor 803, increase or decrease the capture rate, adjust the dynamic range, and adjust the focus of the camera. Also, the vehicle control module 811 can turn sensors on / off individually or in groups.

[0112] In some embodiments, the vehicle control module 811 can be used to vary the parameters of the image pre-processor 805 in ways such as modifying the frequency range of a filter, adjusting edge detection parameters for feature and / or object detection, or adjusting channels and bit depth. In various embodiments, the vehicle control module 811 can be used to control the autonomous driving of the vehicle and / or the driver-assistance functions of the vehicle.

[0113] In some embodiments, the network interface 813 can be responsible for the internal interface between the block configuration of the autonomous driving control system 800 and the communication unit 815. Specifically, the network interface 813 may be a communication interface for receiving and / or transmitting data including voice data. In various embodiments, the network interface 813 can be connected to an external server to connect a voice call by the communication unit 815, receive and / or transmit text messages, transmit sensor data, update the software of the vehicle in the autonomous driving system, or update the software of the autonomous driving system of the vehicle.

[0114] In various embodiments, the communication unit 815 can include various wireless interfaces of cellular or WiFi types. For example, the network interface 813 can be used to receive updates for the operating parameters and / or instruction words for the sensor 803, the image pre-processor 805, the deep learning network 807, the AI processor 809, and the vehicle control module 811 from an external server accessed via the communication unit 815. For example, the machine learning model of the deep learning network 807 can be updated using the communication unit 815. According to still other examples, the communication unit 815 can be used to update the operating parameters of the image pre-processor 805 such as image processing parameters and / or the firmware of the sensor 803.

[0115] In other embodiments, the communication unit 815 can be used to activate communication for emergency services and emergency contact in the event of an accident or a near-accident event. For example, in a collision event, the communication unit 815 can be used to call for emergency services for help and can be used to notify external emergency services of collision details and the position of the vehicle. In various embodiments, the communication unit 815 can update or obtain an expected arrival time and / or the location of the destination.

[0116] According to one embodiment, the autonomous driving system 800 illustrated in FIG. 10 may be composed of vehicle electronic devices. According to one embodiment, when an autonomous driving cancellation event occurs from a user during autonomous driving of the vehicle, the AI processor 809 of the autonomous driving system 800 can be controlled to input autonomous driving cancellation event-related information as training set data of a deep learning network, thereby controlling the autonomous driving software of the vehicle to learn.

[0117] FIGS. 11 and 12 are diagrams illustrating an example of a block diagram showing an autonomous driving mobile body according to one embodiment. Referring to FIG. 18, the autonomous driving mobile body 900 according to this embodiment can include a control device 1000, sensing modules 904a, 904b, 904c, 904d, an engine 906, and a user interface 908.

[0118] The autonomous driving mobile body 900 can be provided with an autonomous driving mode or a manual mode. As an example, it can be switched from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode according to user input received via the user interface 908.

[0119] When the mobile body 900 operates in the autonomous driving mode, the autonomous driving mobile body 900 can operate under the control of the control device 1000.

[0120] In this embodiment, the control device 1000 can include a controller 1020 including a memory 1022 and a processor 1024, a sensor 1010, a communication device 1030, and an object detection device 1040.

[0121] Here, the object detection device 1040 can perform all or part of the functions of a distance measurement device (for example, the electronic device 101).

[0122] That is, in this embodiment, the object detection device 1040 is a device for detecting an object located outside the moving body 900, and the object detection device 1040 can detect an object located outside the moving body 900 and generate object information based on the detection result.

[0123] The object information can include information regarding the presence or absence of an object, the position information of the object, the distance information between the moving body and the object, and the relative speed information between the moving body and the object.

[0124] The object can include various objects located outside the moving body 900 such as a lane, other vehicles, pedestrians, traffic signals, light, roads, structures, speed bumps, terrain objects, animals, etc. Here, the traffic signal may be a concept including a traffic light, a traffic sign board, a pattern or text illustrated on the road surface. Also, the light may be light generated from a lamp provided on another vehicle, light generated from a street lamp, or sunlight.

[0125] Also, the structure may be an object located around the road and fixed to the ground. For example, the structure can include a street lamp, a street tree, a building, a utility pole, a traffic signal, a bridge. The terrain object can include a mountain, a hill, etc.

[0126] Such an object detection device 1040 can include a camera module. The controller 1020 can extract object information from an external image captured by the camera module and cause the controller 1020 to process information related thereto.

[0127] Further, the object detection device 1040 can further include an imaging device for recognizing the external environment. In addition to LIDAR, RADAR, a GPS device, an odometry device, and other computer vision devices, ultrasonic sensors and infrared sensors can be used, and these devices can operate selectively or simultaneously as needed to enable more precise sensing.

[0128] On the other hand, a distance measurement device according to an embodiment of the present invention can calculate the distance between the autonomous mobile body 900 and an object, and based on the calculated distance in cooperation with the control device 1000 of the autonomous mobile body 900, control the operation of the mobile body.

[0129] As an example, when there is a possibility of a collision according to the distance between the autonomous mobile body 900 and an object, the autonomous mobile body 900 can control the brakes to reduce the speed or stop. As another example, when the object is a moving object, the autonomous mobile body 900 can control the traveling speed of the autonomous mobile body 900 to maintain a predetermined distance or more from the object.

[0130] Such a distance measurement device according to an embodiment of the present invention can be constituted by a module within the control device 1000 of the autonomous mobile body 900. That is, the memory 1022 and the processor 1024 of the control device 1000 can implement the collision prevention method according to the present invention in software.

[0131] In addition, the sensor 1010 can be connected to the in-vehicle / external environment sensing modules 904a, 904b, 904c, and 904d to obtain various sensing information. Here, the sensor 1010 can include an attitude sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight sensing sensor, a heading sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on handle rotation, an in-vehicle temperature sensor, an in-vehicle humidity sensor, an ultrasonic sensor, an illuminance sensor, an acceleration pedal position sensor, a brake pedal position sensor, etc.

[0132] Thereby, the sensor 1010 can obtain sensing signals regarding the attitude information of the vehicle, the collision information of the vehicle, the direction information of the vehicle, the position information (GPS information) of the vehicle, the angle information of the vehicle, the speed information of the vehicle, the acceleration information of the vehicle, the gradient information of the vehicle, the forward / backward information of the vehicle, the battery information, the fuel information, the tire information, the lamp information of the vehicle, the in-vehicle temperature information, the in-vehicle humidity information, the rotation angle of the steering wheel, the external illuminance of the vehicle, the pressure applied to the acceleration pedal, the pressure applied to the brake pedal, etc.

[0133] In addition, the sensor 1010 can further include an acceleration pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0134] In this way, the sensor 1010 can generate the state information of the vehicle based on the sensing data.

[0135] The wireless communication device 1030 is configured to realize wireless communication between the autonomous driving mobile body 900. For example, it enables the autonomous driving mobile body 900 to communicate with the user's mobile phone, or other wireless communication devices 1030, other mobile bodies, a central device (traffic control device), a server, etc. The wireless communication device 1030 can transmit and receive wireless signals by means of an access wireless protocol. The wireless communication protocol may be Wi-Fi, Bluetooth, Long-Term Evolution (LTE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Global Systems for Mobile Communications (GSM), and the communication protocol is not limited to this.

[0136] Also, in this embodiment, the autonomous driving mobile body 900 can also realize communication between mobile bodies by means of the wireless communication device 1030. That is, the wireless communication device 1030 can communicate with other mobile bodies on the road and other mobile bodies by vehicle-to-vehicle (V2V) communication. The autonomous driving mobile body 900 can transmit and receive information such as driving warnings and traffic information by means of vehicle-to-vehicle communication, and can also request information from other mobile bodies or receive requests. For example, the wireless communication device 1030 can perform V2V communication with a dedicated short-range communication (DSRC) device or a Cellular-V2V (C-V2V) device. In addition to vehicle-to-vehicle communication, communication between a vehicle and other things (for example, an electronic device carried by a pedestrian, etc.) (V2X, Vehicle to Everything communication) can also be realized by means of the wireless communication device 1030.

[0137] In this embodiment, the controller 1020 is a unit that controls the overall operation of each unit within the moving body 900, and can be configured during manufacturing or additionally configured after manufacturing by the manufacturer of the moving body for performing the function of autonomous driving. Alternatively, through the upgrade of the controller 1020 configured during manufacturing, a configuration for continuously performing additional functions can be included. Such a controller 1020 may also be referred to as an ECU (Electronic Control Unit).

[0138] The controller 1020 can collect various data from the connected sensor 1010, object detection device 1040, communication device 1030, etc., and based on the collected data, transmit a control signal to the sensor 1010, engine 906, user interface 908, communication device 1030, object detection device 1040 included as other components within the moving body. Also, although not shown in the figure, a control signal can also be transmitted to an acceleration device, braking system, steering device, or navigation device related to the driving of the moving body.

[0139] In this embodiment, the controller 1020 can control the engine 906. For example, when the autonomous driving moving body 900 senses the speed limit of the road during driving, it can control the engine 906 so that the driving speed does not exceed the speed limit, or control the engine 906 to accelerate the driving speed of the autonomous driving moving body 900 within a range not exceeding the speed limit.

[0140] Also, while the autonomous mobile body 900 is in motion, if the controller 1020 determines that the autonomous mobile body 900 is approaching or departing from a lane, it can control the engine 906 to control the movement of the mobile body according to the determination result, determining whether such lane approach and departure are due to normal driving conditions or other driving conditions. Specifically, the autonomous mobile body 900 can detect lanes formed on both sides of the road on which the mobile body is traveling. In this case, the controller 1020 determines whether the autonomous mobile body 900 is approaching or departing from a lane. For example, if it is determined that the autonomous mobile body 900 is approaching or departing from a lane, it can be determined whether such driving is due to accurate driving conditions or other driving conditions. Here, an example of normal driving conditions may be a situation where a lane change of the mobile body is necessary. Also, as an example of other driving conditions, it may be a situation where a lane change of the mobile body is not necessary. For example, if the controller 1020 determines that the autonomous mobile body 900 is approaching or departing from a lane in a situation where a lane change of the mobile body is not necessary, it can control the movement of the autonomous mobile body 900 so that the autonomous mobile body 900 does not depart from the lane and travels normally with the mobile body.

[0141] When there is another mobile body or an obstacle in front of the mobile body, the engine 906 or the braking system can be controlled to decelerate the moving mobile body, and in addition to the speed, the trajectory, the travel route, and the steering angle can also be controlled. Alternatively, the controller 1020 can generate necessary control signals according to the recognition information of other external environments such as the travel lane and the travel signal of the mobile body, and control the movement of the mobile body.

[0142] In addition to generating its own control signals, the controller 1020 can also communicate with surrounding mobile bodies or a central server and transmit commands for controlling peripheral devices according to the received information, thereby controlling the movement of the mobile body.

[0143] In addition, when the position of the camera module 1050 is changed or the angle of view is changed, the controller 1020 may generate a control signal to control the calibration of the camera module 1050 in order to prevent the accurate recognition of the moving object or the lane according to this embodiment from being difficult. Therefore, in this embodiment, the controller 1020 can cause the camera module 1050 to generate a calibration control signal, so that even if the mounting position of the camera module 1050 is changed due to vibrations or impacts generated according to the movement of the autonomous driving vehicle 900, the normal mounting position, direction, angle of view, etc. of the camera module 1050 can be continuously maintained. The controller 1020 can generate a control signal to perform calibration of the camera module 1020 when the pre-stored initial mounting position, direction, and angle of view information of the camera module 1020 and the initial mounting position, direction, and angle of view information of the camera module 1020 measured during the running of the autonomous driving vehicle 900 change by more than a critical value.

[0144] In this embodiment, the controller 1020 may include a memory 1022 and a processor 1024. The processor 1024 can execute the software stored in the memory 1022 according to the control signal of the controller 1020. Specifically, the data and instructions for the controller 1020 to execute the lane detection method according to the present invention are stored in the memory 1022, and the instructions can be executed by the processor 1024 to implement one or more methods disclosed herein.

[0145] Here, the memory 1022 can be stored in a non-volatile recording medium executable by the processor 1024. The memory 1022 can store software and data by an appropriate internal or external device. The memory 1022 can be composed of a RAM (random access memory), a ROM (read only memory), a hard disk, and a memory 1022 device connected to a dongle.

[0146] Memory 1022 can store at least an operating system (OS), user applications, and executable instructions. Memory 1022 can also store application data and array data structures.

[0147] Processor 1024 is a microprocessor or a suitable electronic processor, and may be a controller, a microcontroller, or a state machine.

[0148] Processor 1024 can be implemented by a combination of computing devices, and the computing devices can be digital signal processors, microprocessors, or a suitable combination thereof.

[0149] On the other hand, the autonomous mobile 900 can further include a user interface 908 for user input to the above-described control device 1000. The user interface 908 can enable the user to input information through appropriate interaction. For example, it can be realized by a touch screen, a keypad, operation buttons, etc. The user interface 908 transmits an input or an instruction to the controller 1020, and the controller 1020 can perform a control operation of the mobile body in response to the input or the instruction.

[0150] Also, the user interface 908 is a device external to the autonomous mobile 900 and can communicate with the autonomous mobile 900 via the wireless communication device 1030. For example, the user interface 908 can be interlocked with a mobile phone, a tablet, or other computer devices.

[0151] Furthermore, in this embodiment, the autonomous driving vehicle 900 is described as including an engine 906, but it is also possible to include other types of propulsion systems. For example, the vehicle can operate on electrical energy, hydrogen energy, or a hybrid system combining these. Therefore, the controller 1020 includes a propulsion mechanism by the propulsion system of the autonomous driving vehicle 900, and can provide a control signal according to this to the configuration of each propulsion mechanism.

[0152] Hereinafter, with reference to FIG. 12, the detailed configuration of the control device 1000 according to this embodiment will be described in more detail.

[0153] The control device 1000 includes a processor 1024. The processor 1024 may be a general-purpose single or multi-chip microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, or the like. The processor may also be referred to as a central processing unit (CPU). Also, in this embodiment, the processor 1024 can be used as a combination of a plurality of processors.

[0154] The control device 1000 also includes a memory 1022. The memory 1022 may be any electronic component capable of storing electronic information. In addition to a single memory, the memory 1022 can also include a combination of memories 1022.

[0155] Data and instruction words 1022a for performing the distance measurement method of the distance measurement device according to the present invention may be stored in the memory 1022. When the processor 1024 executes the instruction words 1022a, all or part of the instruction words 1022a and the data 1022b necessary for the execution of the instructions may be loaded onto the processor 1024 as instruction words 1024a and data 1024b.

[0156] The control device 1000 may also include a transmitter 1030a, a receiver 1030b, or a transceiver 1030c for allowing transmission and reception of signals. One or more antennas 1032a, 1032b may be electrically connected to the transmitter 1030a, the receiver 1030b, or each transceiver 1030c, and may further include an antenna.

[0157] The control device 1000 may include a digital signal processor (DSP) 1070. The DSP 1070 can enable the mobile body to quickly process digital signals.

[0158] The control device 1000 may also include a communication interface 1080. The communication interface 1080 may include one or more ports and / or communication modules for connecting other devices to the control device 1000. The communication interface 1080 can enable the user and the control device 1000 to interact with each other.

[0159] The various components of the control device 1000 may all be connected by one or more buses 1090, and the buses 1090 may include a power bus, a control signal bus, a status signal bus, a data bus, etc. Under the control of the processor 1024, the components can transmit mutual information via the buses 1090 to perform the intended functions.

[0160] On the one hand, in various embodiments, the control device 1000 can be associated with a gateway for communication with a security cloud. For example, referring to FIG. 13, the control device 1000 can be associated with a gateway 1105 for providing information obtained from at least one of the components 1101 - 1104 of the vehicle 1100 to the security cloud 1106. For example, the gateway 1105 can be included within the control device 1000. As another example, the gateway 1105 may be configured as another device within the vehicle 1100 that is distinct from the control device 1000. The gateway 1105 communicatively connects the networks within the vehicle 1100 that are secured by the software management cloud 1109, the security cloud 1106, and the security software 1110 within the vehicle, which have different networks from each other.

[0161] For example, the component 1101 may be a sensor. For example, the sensor can be used to obtain information regarding at least one of the state of the vehicle 1100 or the state around the vehicle 1100. For example, the component 1101 can include a sensor 1410.

[0162] For example, the component 1102 may be an ECU (electronic control unit). For example, the ECU can be used for engine control, transmission control, airbag control, and tire air pressure management.

[0163] For example, component 1103 may be an instrument cluster. For example, the instrument cluster may mean a panel located in front of the driver's seat on the dashboard. For example, the instrument cluster can be configured to show the driver (or passenger) the information necessary for driving. For example, the instrument cluster can be used to display at least one of a visual element for indicating the number of revolutions per minute (RPM) of the engine, a visual element for indicating the speed of the vehicle 1100, a visual element for indicating the remaining fuel level, a visual element for indicating the gear state, or a visual element for indicating the information acquired by component 1101.

[0164] For example, component 1104 may be a telematics device. For example, the telematics device may mean a device that combines wireless communication technology and GPS (Global Positioning System) technology to provide various mobile communication services such as location information and safe driving within the vehicle 1100. For example, the telematics device can be used to connect the driver, the cloud (e.g., security cloud 1106), and / or the surrounding environment to the vehicle 1100. For example, the telematics device can be configured to support high bandwidth and low latency for 5G NR standard technology (e.g., 5G NR's V2X technology). For example, the telematics device can be configured to support the autonomous driving of the vehicle 1100.

[0165] For example, the gateway 1105 can be used to connect the network within the vehicle 1100 to the software management cloud 1109 and the security cloud 1106, which are networks outside the vehicle. For example, the software management cloud 1109 can be used to update or manage at least one software necessary for the driving and management of the vehicle 1100. For example, the software management cloud 1109 can interact with the in-car security software 1110 installed within the vehicle. For example, the in-car security software 1110 can be used to provide security functions within the vehicle 1100. For example, the in-car security software 1110 can encrypt data transmitted and received via the in-vehicle network using an encryption key obtained from an external authorized server for the encryption of the in-vehicle network. In various embodiments, the encryption key used by the in-car security software 1110 can be generated corresponding to the identification information of the vehicle (vehicle number plate, vehicle VIN (vehicle identification number)) or information uniquely assigned to each user (e.g., user identification information).

[0166] In various embodiments, the gateway 1105 can transmit data encrypted by the security software 1110 in the vehicle to the software management cloud 1109 and / or the security cloud 1106 based on the encryption key. The software management cloud 1109 and / or the security cloud 1106 can identify from which vehicle or user the data is received by decrypting the data encrypted by the encryption key of the security software 1110 in the vehicle using a decryption key. For example, since the decryption key is a unique key corresponding to the encryption key, the software management cloud 1109 and / or the security cloud 1106 can identify the sender of the data (e.g., the vehicle or the user) based on the data decrypted by the decryption key.

[0167] For example, the gateway 1105 can be configured to assist the security software 1110 in the vehicle and can be associated with the control device 1000. For example, the gateway 1105 can be associated with the control device 1000 to assist in connecting the control device 1000 with the client device 1107 connected to the security cloud 1106. As another example, the gateway 1105 can be associated with the control device 1000 to assist in connecting the control device 1000 with the third-party cloud 1108 connected to the security cloud 1106. However, it is not limited thereto.

[0168] In various embodiments, the gateway 1105 can be used to connect the vehicle 1100 with a software management cloud 1109 for managing the operating software of the vehicle 1100. For example, the software management cloud 1109 can monitor whether an update of the operating software of the vehicle 1100 is required, and based on monitoring that an update of the operating software of the vehicle 1100 is required, provide data for updating the operating software of the vehicle 1100 by the gateway 1105. As another example, the software management cloud 1109 can receive a request from a user who requests an update of the operating software of the vehicle 1100 from the vehicle 1100 via the gateway 1105, and based on the received request, provide data for updating the operating software of the vehicle 1100. However, it is not limited thereto.

[0169] On the other hand, the methods according to various embodiments of the present invention described above can be implemented by a program and provided to a server or a device. Thereby, each device can access a server or a device in which the program is stored and download the program.

[0170] Also, the methods according to various embodiments of the present invention described above can be implemented by a program and stored and provided in various non-transitory computer readable media. A non-transitory computer readable medium means a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short moment, such as a register, a cache, or a memory. Specifically, the various applications or programs described above can be stored and provided in non-transitory computer readable media such as CDs, DVDs, hard disks, Blu-ray disks, USBs, memory cards, ROMs, etc.

[0171] However, the concept of the present invention is not necessarily limited thereto, and the apparatus / method / system according to the exemplary embodiments of the present invention is applicable to various product / technical fields other than the above-described product / technical fields.

[0172] As described above in detail are various embodiments of the present invention. However, those having ordinary knowledge in the technical field to which the present invention pertains will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the rights of the present invention should not be defined by being limited to the above-described embodiments, but should be defined by not only the claims described below but also those equivalent to the scope of the present patent claims.

Explanation of Reference Numerals

[0173] 1 Image processing device around the vehicle 10 Camera module 20 Electronic device 30 Alarm device 40 Display 110 First camera 120 Second camera 130 Third camera 140 Fourth camera 210 Communication circuit 220 Video input unit 230 Control unit 240 Video conversion unit 250 Memory 260 Sensor unit

Claims

1. A video processing device for the vehicle surroundings, comprising a camera module installed on the vehicle and including a plurality of cameras equipped with wide-angle lenses, and an electronic device for processing the video acquired via the camera module, wherein the electronic device comprises a communication circuit, a video input unit for receiving the transmission of the video of the vehicle surroundings photographed via the communication circuit, a video conversion unit for correcting the video of the vehicle surroundings received by the video input unit, and a control unit for controlling the communication circuit, the video input unit, and the video conversion unit, wherein the video conversion unit is controlled by the control unit so that an overview screen in a circularly distorted form photographed by the camera module is corrected to a planar image, and the video processing device for the vehicle surroundings is characterized thereby.

2. The camera module comprises a first camera installed on the left side mirror of the vehicle, a second camera installed on the right side mirror of the vehicle, a third camera for photographing the front area of the vehicle, and a fourth camera for photographing the rear area of the vehicle, and the video processing device for the vehicle surroundings according to Claim 1 is characterized thereby.

3. The video conversion unit is controlled by the control unit to divide the received video of the vehicle surroundings according to the photographing area of the vehicle and to control the division into a plurality of parts, and the video processing device for the vehicle surroundings according to Claim 2 is characterized thereby.

4. The video of the vehicle surroundings is divided into a front area, a rear area, a left side area, and a right side area of the vehicle based on the mounting point of the wide-angle lens installed on the side mirror of the vehicle, and the left side area and the right side area are further divided into upper and lower end areas respectively, and the video processing device for the vehicle surroundings according to Claim 3 is characterized thereby.

5. The video conversion unit is controlled to correct a part of the divided plurality of videos of the vehicle surroundings into a planar image, and the video processing device for the vehicle surroundings according to Claim 4 is characterized thereby.

6. The video conversion unit is controlled to selectively merge a part of the divided plurality of videos of the vehicle surroundings and to correct them into one planar image, and the video processing device for the vehicle surroundings according to Claim 2 is characterized thereby.

7. further comprises a sensor unit for sensing the internal environment and the external environment of the vehicle, and the sensor unit is controlled by the control unit to sense the driving mode of the vehicle, and the video processing device for the vehicle surroundings according to Claim 6 is characterized thereby.

8. The vehicle surrounding video processing device according to claim 7, wherein the driving mode includes a forward mode, a reverse mode, a left turn mode, and a right turn mode.

9. The video conversion unit is controlled to selectively merge a part of the videos of the vehicle surroundings divided into the plurality according to the driving mode of the vehicle and correct the merged videos into one planar image, the vehicle surrounding video processing device according to claim 8.

10. When the driving mode is the forward mode, the video conversion unit is controlled by the control unit to merge the videos of the vehicle surroundings respectively captured by the first camera, the second camera, and the fourth camera and correct the merged videos into one planar image, the vehicle surrounding video processing device according to claim 9.

11. When the driving mode is the reverse mode, the video conversion unit is controlled to merge the videos of the vehicle surroundings respectively captured by the first camera, the second camera, and the fourth camera, correct the merged videos into one planar image, and extract and merge only the lower end regions of the left side region and the right side region respectively, the vehicle surrounding video processing device according to claim 9.

12. When the driving mode is the left turn mode, the video conversion unit is controlled to correct the video of the vehicle surroundings captured by the first camera into one planar image, the vehicle surrounding video processing device according to claim 9.

13. When the driving mode is the right turn mode, the video conversion unit is controlled to correct the video of the vehicle surroundings captured by the second camera into one planar image, the vehicle surrounding video processing device according to claim 9.

14. A method for processing videos of the vehicle surroundings, comprising: a step of sensing the driving mode of the vehicle; a step of capturing the surroundings of the vehicle using a camera module equipped with a wide-angle lens; a step of receiving the captured videos of the vehicle surroundings; a step of correcting the received videos of the vehicle surroundings; and a step of outputting the corrected videos of the vehicle surroundings, wherein the corrected videos of the vehicle surroundings are videos or images in which an overview screen in a form of being circularly distorted is corrected into a planar image, the method for processing videos of the vehicle surroundings.

15. The camera module includes a first camera and a second camera, a third camera and a fourth camera respectively installed on the left and right side mirrors of the vehicle. The method for processing an image around a vehicle according to claim 14, wherein the captured image around the vehicle includes the images captured by the first camera and the second camera, and the third camera and the fourth camera respectively.

16. The step of correcting is controlled to selectively merge a part of the images around the vehicle received after dividing the images around the vehicle received according to the driving mode of the vehicle into a plurality, and correct them into one planar image. The method for processing an image around a vehicle according to claim 15.

17. When the driving mode is the forward mode, the step of correcting is controlled to merge the images around the vehicle captured by the first camera, the second camera and the fourth camera respectively, and correct them into one planar image. The method for processing an image around a vehicle according to claim 15.

18. When the driving mode is the reverse mode, the step of correcting is controlled to merge the images around the vehicle captured by the first camera, the second camera and the fourth camera respectively, correct them into one planar image, and only the lower end regions of the images captured by the first and second cameras are selectively extracted and merged. The method for processing an image around a vehicle according to claim 15.

19. A computer-readable recording medium on which a program for executing the method for processing an image around a vehicle according to any one of claims 14 to 18 is recorded.

20. A computer program stored in a computer-readable recording medium and including program codes for executing the method for processing an image around a vehicle according to any one of claims 14 to 18.

Images