Augmented reality information providing apparatus based on digital mirror and control method thereof

Abstract

The present invention relates to an apparatus for displaying augmented reality information on a display on the basis of a digital mirror, comprising: an interface unit for receiving an image including a road environment around a vehicle and an object around the vehicle from a plurality of cameras including a first camera and a second camera, receiving detection information for detecting the driving state of the vehicle; the AR module is used for rendering at least one augmented reality object according to the detection information on the basis of an image shot by at least one camera in the plurality of cameras; and a processor controlling the AR module to render a first augmented reality object related to the first object among objects included in the image of the first camera, and controlling a display portion of the vehicle through the interface portion, and displaying a second augmented reality object related to the first augmented reality object in the image of the second camera in a direction opposite to the first object.

Description

Technical Field

[0001] The present invention relates to a display device provided in a vehicle, and more particularly, to a device for displaying augmented reality information on a display based on a digital mirror. Background Art

[0002] Currently, automotive-related technologies are constantly evolving and new technologies are gradually emerging. One of the new technologies emerging under this automotive-related technology is the digital mirror, which replaces the existing side mirror or rearview mirror with a camera.

[0003] The digital mirror may refer to a mirror that replaces a side mirror or an in-vehicle mirror for reflecting the rear of the vehicle, is equipped with a camera pointing to the rear of the vehicle, and displays an image obtained by the mounted camera through a display or a CID (Center Information Display) that replaces the in-vehicle mirror.

[0004] Compared with a general side mirror (with a rear view angle of 18 degrees based on the driver's seat), this digital mirror has the advantages of providing a wider view angle (with a rear view angle of 25 degrees or more based on the driver's seat) and having various additional functions such as a blind-spot collision warning (BCW). In addition, if a side mirror is used, the user needs to visually confirm the image of the rear of the vehicle reflected in the reflective surface of the mirror, and a specified level of reflective surface area needs to be ensured. However, if a camera is used, not only is a wide reflective surface area not required, but a compact configuration can be achieved through the configuration of camera components. Therefore, not only can the rear vision blind spot be minimized, but the front side vision can also be improved. In addition, compared with the existing side mirror, the air resistance can be minimized, so the fuel efficiency can be improved. As described above, due to having more excellent advantages than the side mirror, the trend of replacing the general side mirror with a digital mirror is increasing.

[0005] On the other hand, recently, augmented reality (AR) technology, which superimposes graphic objects on the image captured by a camera to additionally output virtual objects in the real world, has gradually emerged. Existing vehicles use such augmented reality technology to provide the driver with the environment around the vehicle and additional information related to the state and driving path of the vehicle. Therefore, the driver can intuitively recognize the vehicle and the driving environment of the vehicle. Therefore, the driving efficiency and convenience can be further improved. The advantage of applying this augmented reality technology is that various information required for vehicle driving can be provided based on the real world.

[0006] However, the digital mirror still displays the images captured by the camera through the display. Therefore, the user demand for applying the augmented reality technology to the digital mirror is also increasing. Summary of the Invention

[0007] Problems to be Solved by the Invention

[0008] An object of the present invention is to solve the above problems and other problems, and to provide an augmented reality information providing apparatus based on a digital mirror that identifies an object displayed in the digital mirror and provides information related to the identified object to a user as an augmented reality object, and a control method of the apparatus.

[0009] In addition, an object of the present invention is to solve the above problems and other problems, and to provide an augmented reality information providing apparatus based on a digital mirror that enables a first camera and a second camera that capture images including a first object among a plurality of cameras to be interlocked, so that an augmented reality object related to the first object can be displayed, and a control method of the apparatus.

[0010] Means for Solving the Problems

[0011] To achieve the above object or other objects, according to one aspect of the present invention, an augmented reality information providing apparatus according to an embodiment of the present invention includes: an interface unit that receives images including a road environment around a vehicle and an object around the vehicle from a plurality of cameras including a first camera and a second camera, and receives detection information for detecting a driving state of the vehicle; an AR module that renders at least one augmented reality object based on an image captured by at least one of the plurality of cameras according to the detection information; and a processor that controls the AR module to render a first augmented reality object related to a first object among the objects included in the image of the first camera, and controls a display unit of the vehicle through the interface unit to display a second augmented reality object related to the first augmented reality object in a direction opposite to the first object in the image of the second camera.

[0012] In one embodiment, the processor calculates a three-dimensional position and a relative speed of the first object detected from the image of the first camera, and estimates the three-dimensional position of the first object according to the elapsed time since the first object was detected; if the first object is not detected from the image of the first camera, the processor controls the display unit such that within a specified time, a first augmented reality object indicating the estimated three-dimensional position of the first object is displayed in the image of the first camera.

[0013] In one embodiment, it is characterized in that if there is a user selection of the augmented reality object, the processor displays information related to the first object.

[0014] In one embodiment, it is characterized in that the processor determines, based on the inferred three-dimensional position of the first object, any one of the plurality of cameras that captures images around the inferred three-dimensional position as the second camera.

[0015] In one embodiment, it is characterized in that if the second camera is determined, the processor controls the interface unit so that the image of the second camera including the second augmented reality object is displayed on the display unit, and the second augmented reality object predicts the appearance of the first object based on the inferred position of the first object.

[0016] In one embodiment, it is characterized in that if the first object is not detected from the image of the first camera, the processor predicts the moment when the first object will be included in the image of the second camera based on the inferred position of the first object; and the processor changes at least one of the first augmented reality object and the second augmented reality object according to the time elapsed from the time point when the first object is not detected in the image of the first camera.

[0017] In one embodiment, it is characterized in that the processor changes the transparency of the first augmented reality object and the second augmented reality object oppositely, or changes the saturation of the first augmented reality object and the second augmented reality object oppositely, according to the time elapsed from the time point when the first object is not detected.

[0018] In one embodiment, it is characterized in that if the first object is not detected from the determined image of the second camera, the processor controls the interface part so that the first object is detected based on the image taken by at least one other camera except the second camera, and an augmented reality object related to the first object is displayed based on the image taken by any other camera that detects the first object.

[0019] In one embodiment, it is characterized in that if the first object is not detected in the image captured by the at least one other camera, the processor controls the interface part so that an augmented reality object related to the first object is displayed at a position corresponding to the inferred three-dimensional position in the image captured by the second camera.

[0020] In one embodiment, the processor calculates the distance from the vehicle center to the first object, calculates the time when the first object exits the viewing angle of the first camera, and calculates the time taken based on the calculated time; the processor speculates the candidate positions of the plurality of first objects as the three-dimensional positions according to the different times taken for each preset unit time, determines the second camera based on at least one of the candidate positions among the plurality of candidate positions, and identifies the first object included in the image captured by the second camera based on any one of the candidate positions among the plurality of candidate positions.

[0021] In one embodiment, the processor projects the shapes corresponding to the first object onto the plurality of candidate positions within the image area including the area displayed by the display unit, and determines at least one candidate position including at least a part of the shape corresponding to the first object as the final candidate position in a part of the image area displayed by the display unit; the processor speculates, as the three-dimensional position of the first object, any one of the final candidate positions corresponding to an object whose shape overlaps with the shape of the projected first object by a preset ratio or more among the objects included in the image captured by the second camera.

[0022] In one embodiment, the processor detects the occurrence of a preset safe driving-related event based on at least one of the distance between the first object included in the image of the second camera and the vehicle, the relative speed of the first object, the traveling direction of the first object, and the traveling direction of the vehicle; if the safe driving-related event occurs, the processor crops a part of the image including the area where the first object is included in the image of the second camera, and controls the interface unit to scale and display the cropped image.

[0023] In one embodiment, the processor crops a part of the image of the second camera according to the aspect ratio of the display unit so as to include at least one of a specific surface of the first object and the approaching path of the first object.

[0024] In one embodiment, the processor detects an operation of a preset user on the vehicle, and determines a third camera from the plurality of cameras based on the detected operation; the processor crops a part of the area including at least one object in the image captured by the third camera, and controls the interface unit to scale and display the image of the cropped area on the display unit.

[0025] In one embodiment, the processor determines the degree of danger for at least one object included in the cropped area based on at least one of distance, speed, and driving direction, and controls the AR module to display different augmented reality objects according to the determined degree of danger.

[0026] In one embodiment, the operations of the user include: an operation of turning on a turn signal to enter a specific lane or gazing at a mirror corresponding to a specific direction for a time longer than a constant time; the processor determines the third camera as the camera that captures the lane corresponding to the turned-on turn signal or the specific direction corresponding to the mirror at which the user gazes for a time longer than a constant time.

[0027] To achieve the above object or other objects, according to one aspect of the present invention, a control method of an augmented reality information providing device according to an embodiment of the present invention is characterized by including: a step of identifying a first object included in a first camera among a plurality of cameras provided in a vehicle; a step of displaying a first augmented reality image including a first augmented reality object related to the identified first object; a step of predicting the position of the first object after at least a specified time; a step of determining a second camera that will capture an image of the first object after the specified time from among the plurality of cameras based on the predicted position of the first object; a step of determining a display position in a direction opposite to the first object from the image obtained from the second camera; and a step of controlling a display of the vehicle that displays the image obtained from the second camera so as to display a second augmented reality object different from the first augmented reality object and related to the first object at the determined display position.

[0028] Advantages of the Invention

[0029] According to at least one embodiment of the embodiments of the present invention, the present invention has the effect of being able to obtain information related to an object displayed in a digital mirror through augmented reality.

[0030] In addition, according to at least one embodiment of the embodiments of the present invention, the present invention has the effect of displaying augmented reality information related to an object that disappears from the field of view of a user gazing ahead through the digital mirror, so that even when the object enters a blind spot, the user can confirm information related to the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a block diagram for explaining the configuration of an augmented reality information providing device based on a digital mirror according to an embodiment of the present invention connected to a vehicle.

[0032] Figure 2It is a flowchart showing the operation process in which an augmented reality information providing device according to an embodiment of the present invention displays an augmented reality object related to an object recognized by a first camera in a second camera image.

[0033] Figure 3 It is a schematic diagram showing the relative movement of surrounding objects as a vehicle equipped with an augmented reality information providing device according to an embodiment of the present invention moves.

[0034] Figure 4 It is shown as the Figure 3 schematic diagram of the augmented reality object displayed on the CID and the in-vehicle mirror display as the object shown in

[0035] Figure 5 It is a flowchart showing the operation process in which an augmented reality information providing device according to an embodiment of the present invention determines a second camera based on the estimated position of a first object after a predetermined time.

[0036] Figure 6 It is a schematic diagram showing the augmented reality information providing device according to an embodiment of the present invention estimating the estimated position of a first object after a predetermined time.

[0037] Figure 7 It is a flowchart showing the operation process in which an augmented reality information providing device according to an embodiment of the present invention displays an augmented reality object related to a first object recognized from a first camera image in an image of a second camera.

[0038] Figure 8 It is a schematic diagram showing the augmented reality information providing device according to an embodiment of the present invention displaying the position of a first object blocked by an obstacle through an augmented reality object.

[0039] Figure 9 It is a flowchart showing the operation process in which an augmented reality information providing device according to an embodiment of the present invention detects an object corresponding to the first object from a second camera image.

[0040] Figure 10 It is a schematic diagram showing the augmented reality information providing device according to an embodiment of the present invention projecting the estimated position of a first object after a predetermined time and determining a candidate position corresponding thereto in a second camera image.

[0041] Figure 11 It is a schematic diagram showing the augmented reality information providing device according to an embodiment of the present invention detecting an object corresponding to the first object based on the generated candidate object.

[0042] Figure 12 and Figure 132 is a schematic diagram showing that the augmented reality information providing device according to an embodiment of the present invention displays different augmented reality information for objects around the vehicle according to the distance from the vehicle.

[0043] Figure 14 It is a schematic diagram showing that the augmented reality information providing apparatus according to an embodiment of the present invention displays different augmented reality objects that predict the appearance of the first object according to the estimated position of the first object. DETAILED DESCRIPTION

[0044] The technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, unless otherwise clearly specified in the context, the singular expressions used in this specification include plural expressions. The suffixes "module" and "unit" of the constituent elements used in the following description are given or mixed for the convenience of writing the specification, and they themselves do not have mutually distinguishable meanings or functions.

[0045] In the present specification, terms such as “compose” or “comprise” should not be interpreted as including all the various constituent elements or steps described in the specification, may not include some constituent elements or steps, or should be interpreted as further including other constituent elements or steps.

[0046] Furthermore, when describing the technology disclosed in this specification, if it is determined that the detailed description of the related known technology may make the gist of the technology disclosed in this specification unclear, the detailed description will be omitted.

[0047] In addition, the drawings are provided only to facilitate understanding of the embodiments disclosed in this specification. The technical ideas disclosed in this specification are not limited to the drawings, and it should be understood that the present invention includes all changes, equivalents and even substitutes made within the technical ideas and technical scope of the present invention. In addition, in addition to each embodiment described below, the combination of the embodiments as a change, equivalent or substitute within the idea and technical scope of the present invention can of course correspond to the idea and technical scope of the present invention.

[0048] Figure 1 This is a block diagram for explaining the structure of an augmented reality information providing device based on a digital mirror according to an embodiment of the present invention connected to a vehicle.

[0049] Just refer to Figure 1 In particular, the augmented reality information providing device based on the digital mirror of the embodiment of the present invention (hereinafter referred to as the augmented reality information providing device) includes a processor 100 and an interface unit 110 connected to the processor 100, an AR (Augmented Reality) module 120, a memory 130, and a communication unit 140. Figure 1The components shown are not essential for implementing the augmented reality information providing device of the embodiments of the present invention. The augmented reality information providing device described in this specification may include more or fewer components than those exemplified above.

[0050] First, look at the interface unit 110. The interface unit 110 can be connected to the interface unit of the vehicle 200 (not shown, hereinafter referred to as the vehicle interface unit), and can receive various information provided by the vehicle 200 through the vehicle interface unit. Among them, the vehicle interface unit can serve as a passage for various types of external devices connected to the vehicle 200 or each component of the vehicle 200. For example, the vehicle interface unit can be provided with a variety of ports connected to the interface unit 110, and use the ports to connect to the interface unit 110. And data can be exchanged with the interface unit 110.

[0051] The interface unit 110 can be connected to each component of the vehicle 200 through the vehicle interface unit. As an example, the interface unit 110 can be connected to the ADAS (Advanced Driver Assistance Systems) 210 provided in the vehicle 200. The ADAS 210 is a system for autonomously identifying some of the many situations that may occur during driving, judging the situations, and controlling the vehicle 100. The ADAS 210 may also include: an "Autonomous Emergency Braking (AEB)" system that can autonomously reduce speed or stop without the driver stepping on the brake when there is a collision risk; a "Lane Keep Assist System (LKAS)" that adjusts the driving direction to keep in the lane when deviating from the lane; an "Advanced Smart Cruise Control (ASCC)" that travels at a preset speed and can maintain a distance from the vehicle in front; an "Active Blind Spot Detection (ABSD)" that senses the risk of blind spot collision and helps switch to a safe lane; an "Around View Monitor (AVM)" that presents the situation around the vehicle in real time, etc.

[0052] Therefore, the ADAS 210 can be provided with a plurality of cameras (the first camera 211, the second camera 212... the nth camera 210n). Among them, the plurality of cameras are cameras disposed on the vehicle 200, and the viewing angles of the plurality of cameras are set to obtain images in different directions centered on the vehicle 200.

[0053] As an example, the first camera 211 can be a camera that obtains an image of the front of the vehicle 200. In addition, the second camera 212 can be a camera that obtains an image of the rear of the vehicle 200. And, the third camera (not shown) and the fourth camera (not shown) can be cameras that obtain images of the left side and the right side of the vehicle, respectively. In addition, the fifth camera (not shown) can be a camera for obtaining an image of the rear of the vehicle 200 from the left side of the vehicle 200 (e.g., a camera corresponding to the left side mirror), and the sixth camera (not shown) can be a camera for obtaining an image of the rear of the vehicle 200 from the right side of the vehicle 200 (e.g., a camera corresponding to the right side mirror).

[0054] Alternatively, a plurality of front cameras configured to obtain images of the front of the vehicle 200 can be provided. Similarly, a plurality of rear cameras configured to obtain images of the rear of the vehicle 200 can be provided; and a plurality of left side cameras and a plurality of right side cameras configured to obtain images of the left side and the right side of the vehicle 200 can be provided. In this case, the ADAS 210 can separately distinguish the cameras that obtain images in specific directions, and combine the images obtained from each camera to generate the images in the specific directions. As described above, by combining a plurality of cameras to obtain images in specific directions, it is possible to provide a user with an image with a wider viewing angle in the specific directions.

[0055] On the other hand, the ADAS 210 can use the plurality of cameras to identify various objects around the vehicle, and the ADAS 210 can calculate the relative positions of the identified objects based on the distance and angle between the vehicle 200 and the identified objects with the vehicle 200 as a reference. And, the information detected and calculated by the ADAS 210, that is, the information related to the identified objects and the relative positions of the identified objects can be transmitted to the processor 100 through the interface unit 110.

[0056] The interface unit 110 can be connected to the navigation system 220 of the vehicle 200 through the vehicle interface unit. And, the navigation information provided by the connected navigation system 220 can be transmitted to the processor 100. Then, the processor 100 can detect the position of the vehicle 200 on the current path based on the navigation information and the vehicle position detected from the sensing unit of the vehicle 200 and the like.

[0057] In addition, the interface unit 110 may be connected to a sensing unit 230 including a plurality of sensors provided in the vehicle 200. The plurality of sensors provided in the sensing unit 230 may include a sensor for identifying the position of the vehicle 200 and sensors for sensing the state of the vehicle and the state of the driver.

[0058] As an example, the sensing unit 230 may include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor); a collision sensor; a wheel sensor; a speed sensor; an inclination sensor; a weight sensing sensor; a heading sensor; a gyro sensor; a position module; a vehicle forward / backward sensor; a battery sensor; a fuel sensor; a tire sensor; a steering sensor based on the rotation of the steering wheel; a vehicle interior temperature sensor; a vehicle interior humidity sensor; a ultrasonic sensor; an illuminance sensor; an accelerator pedal position sensor; and a brake pedal position sensor, etc. In addition, it may further include an ultrasonic sensor, a LiDAR (Light Detection and Ranging) sensor, a radar sensor, or an infrared sensor for sensing the state of the occupant; or an interior camera for detecting the movement of the user. Based on the sensing results of such a sensing unit 230, the processor 100 may sense the gesture of the user, or sense that the user is looking at a specific direction or the movement of the mirror corresponding to the specific direction.

[0059] The interface unit 110 may be connected to a display unit 250 that outputs visual information or to a sound output unit (not shown) of the vehicle 200 that outputs sound information through a vehicle interface unit. As an example, the sound output unit may include at least one speaker provided in the vehicle 200. And, the event information generated under the control of the processor 100 may be provided to the sound output unit, so that the sound output unit can output the event information.

[0060] Moreover, the display unit 250 may include a plurality of displays. For example, the display unit 250 may include a CID (Center Information Display) 251, an in-vehicle mirror display, and a HUD (Head Up Display) 253. Among them, the in-vehicle mirror display 252 is a display for displaying an image obtained by a rear camera of the vehicle 200 and may be a display (digital mirror) that replaces the function of the existing in-vehicle mirror.

[0061] Each display included in the display unit 250, for example, the CID 251, the in-vehicle mirror display 252, and the HUD 253, may output images acquired by different cameras of the vehicle 200. Alternatively, under the control of the processor 100, any one of the displays may also display images acquired by two or more cameras. In this case, the any one of the displays may be divided into a plurality of regions, and each divided region may display images acquired from different cameras.

[0062] For example, the CID 251 may output, at the same time, an image acquired by a front camera of the vehicle 200 and an image acquired by at least one other camera. In this case, the CID 251 may be divided into three regions. That is, the left region may display an image acquired by a camera disposed on the left side of the vehicle 200 and capturing an image of the rear of the left side of the vehicle 200; the right region may display an image acquired by a camera disposed on the right side of the vehicle 200 and capturing an image of the rear of the right side of the vehicle 200. And, the central region may display an image acquired by the front camera of the vehicle 200. In this case, the left region of the CID 251 may perform the function of the left digital side view mirror, and the right region of the CID 251 may perform the function of the right digital side view mirror.

[0063] On the other hand, although only three displays, i.e., the CID 251, the in-vehicle mirror display 252, and the HUD 253, are shown Figure 1 herein, the display unit 250 may obviously include more displays. As an example, the display unit 250 may further include a display corresponding to the left digital side view mirror and a display corresponding to the right digital side view mirror.

[0064] The interface unit 110 may be connected to an input unit (not shown) of the vehicle 200 that receives user input through the vehicle interface unit. As an example, the input unit may include: a touch input unit that receives a touch input from a passenger of the vehicle 200, i.e., a user, or a microphone that receives a voice input from the user. And, when receiving a user input through the input unit, the interface unit 110 may input the received user input to the processor 100.

[0065] In addition, at least one display provided in the vehicle 200 may form a touch screen that supports both visual information output and touch input. In this case, the interface unit 110 outputs images acquired by at least one camera through the touch screen and may receive a touch input from the user sensed through the touch screen.

[0066] On the other hand, the augmented reality information providing apparatus according to an embodiment of the present invention may be provided with an AR module 120 for rendering and outputting an AR image including an augmented reality object. The AR module 120 may include: a scene forming unit (Scene manager) for forming an image including an augmented reality object, that is, an AR image; and an AR renderer for rendering an image including an augmented reality object on a display.

[0067] First, the AR module 120 may receive, through the interface unit 110, images detected by a plurality of cameras connected to the ADAS 210, gesture information or touch event information of a user sensed by an input unit of the vehicle 200, and information sensed by the ADAS 210, the navigation system 220, the sensing unit 230, and the like.

[0068] Among them, images detected by the plurality of cameras are received through the interface unit 110, and object recognition is implemented under the control of the processor 100. And the object recognition result may be input to the processor 100. In addition, the AR module 120 may determine an augmented reality object including information corresponding to the recognized object.

[0069] Among them, the augmented reality object may be stored in the memory 130. Or, the augmented reality object may be provided by a preset server, for example, provided by a cloud server. In this case, the AR module 120 transmits the recognized object and information related to the recognized object, such as the position of the current vehicle 200 and navigation information, etc., to the cloud server together with the object recognition result, and may receive, as a feedback of the information related to the transmitted object recognition result, an augmented reality object corresponding to the recognized object.

[0070] And, the scene forming unit may determine regions of each augmented reality object to be configured in the image, and configure the determined augmented reality objects in the image. And the AR renderer may receive, from the scene forming unit, scene forming information of objects in each region configured in the image. And, according to the received scene forming information, render an image on at least one display.

[0071] On the other hand, the augmented reality object may include various information related to the recognized object. For example, the augmented reality object corresponding to a specific place or building, i.e., the POI object, as the information related to the corresponding place or building, may include the information of commercial institutions located in the place or building. In this case, the information of the commercial institution may include the trade name or merchant type of the commercial institution, contact information, and information related to the services provided by the commercial institution, etc. And the information related to such an augmented reality object, i.e., augmented reality information, may be displayed on the display based on the input made by the user to the augmented reality object.

[0072] On the other hand, as described above, the augmented reality information providing device according to an embodiment of the present invention may include a plurality of displays. In this case, the AR module 120 may include a plurality of AR renderers for rendering an image including an augmented reality object on each display. In this case, the AR module 120, under the control of the processor 100, fuses the information detected from a plurality of cameras connected to the ADAS 210 or a plurality of sensors provided in the sensing unit 230, and based on the fused information, displays the augmented reality object on the plurality of displays so that the images acquired by the plurality of cameras are organically linked to each other.

[0073] As an example, the AR module 120 may have a fusion unit (not shown), which receives vehicle CAN (Controller Area Network), GPS (Global Positioning System), IMU (Inertial Measurement Unit), and navigation path information, etc. from the ADAS 210, the sensing unit 230, and the navigation system 220, and calculates a series of matrix information for projecting the three-dimensional position of surrounding objects onto the target display in a vehicle-based coordinate system. And each AR renderer may share the sensed values fused in the fusion unit. And each AR renderer may generate AR graphics to be displayed on different displays.

[0074] Or, as described above, instead of providing a plurality of AR renderers, the AR module may be provided in plurality according to the plurality of displays. In this case, different image forming units and AR renderers included in each AR module may render AR images to be displayed on different displays. As an example, Figure 1As shown, if the display unit 250 includes the CID 251, the in-vehicle mirror display 252, and the HUD 253, AR modules corresponding to the CID 251, the in-vehicle mirror display 252, and the HUD 253 can be provided. In this case, each AR module can be provided in each display. That is, AR modules can be provided in the CID 251, the in-vehicle mirror display 252, and the HUD 253, respectively.

[0075] At this time, the processor 100 fuses the information detected by a plurality of sensors provided in a plurality of cameras or the sensing unit 230 connected to the ADAS 210, and based on the fused information, can control the plurality of AR modules so that the images acquired by the plurality of cameras are organically linked to each other.

[0076] In this case, the plurality of AR modules can receive the sensed values of the ADAS 210 and the sensed values of the sensing unit 230 through the interface unit 110, respectively. Among them, if the sensed values received by each AR module 120 are different from each other, the relevance of the augmented reality objects displayed on each display may be reduced (for example, due to different results of GPS sensor fusion, the first AR module displays an augmented reality object, but the second AR module does not display an augmented reality object, etc.). Therefore, the processor 100 can control to make the plurality of AR modules share the intermediate operation data of each AR module (for example, the filtered position of the host vehicle's GPS, the filtered positions of surrounding objects, the types of the currently displayed augmented reality content, and the position matrix of the camera on the road surface for compensating for vehicle shaking, etc.). For example, for the matrix value related to the posture of the vehicle, the processor 100 can use the average value of the matrix value calculated in the second AR module and the matrix value calculated in the first AR module, or can make any one of the first AR module and the second AR module receive the matrix value calculated by the other module and directly use it.

[0077] On the other hand, the memory 130 can store data supporting various functions of the augmented reality information providing device. The memory 130 can store a plurality of application programs (application program) or applications (application) executed by the processor 100, data and instructions for the operation of the AR module 120 and the interface unit 110. In addition, data for supporting the ADAS 210 and a plurality of cameras connected to the ADAS 210 (for example, calibration data), data for the operation of the navigation system 220 and the sensing unit 230 can be stored.

[0078] Among them, as data for the operation of the navigation system 220, the memory 130 may include map information around the vehicle 200 and various pieces of information related to multiple POIs (Position Of Interest) included in the map. Additionally, augmented reality content corresponding to each POI included in the map, that is, augmented reality objects, may be included. Further, the augmented reality content may include various pieces of information related to the POI such as the business name of the POI, the type of merchant, the services provided, contact information, and selectable menus, that is, augmented reality information may be included. Therefore, based on the POI information stored in the memory 130, the AR module 120 may display an image including an augmented reality object on at least one display included in the display unit 250, that is, an augmented reality image may be displayed. And, based on an input from the user to the displayed augmented reality object, augmented reality information related to the selected POI may be displayed on at least one display.

[0079] On the other hand, the POI information and the augmented reality content may be provided by a preset server, such as a cloud server. In this case, the processor 100 may be wirelessly connected to the cloud server through the communication unit 140 and provide the position information of the vehicle 200 obtained from the sensing unit 230 to the cloud server. Then, the cloud server may transmit map information including POI information, or transmit information related to at least one POI around the current position of the vehicle 200 and the augmented reality content for each POI, and the POI information and the augmented reality content transmitted from the cloud server may be stored in the memory 130.

[0080] On the other hand, the processor 100 controls the connected components and generally controls the overall operation of the augmented reality information providing device. First, the processor 100 may receive information provided by the ADAS 210, the navigation system 220, and the sensing unit 230 of the vehicle 200 through the interface unit 110.

[0081] Here, the processor 100 may receive images obtained by a plurality of cameras through the ADAS 210. Additionally, the recognition result of an object included in the images obtained from each camera may be received through the ADAS 210. And the processor 100 may control the AR module 120 according to the recognition result to render an augmented reality image in which an augmented reality object for the recognized object is superimposed on the image obtained by the camera. And the processor 100 may control the display unit 250 through the interface unit 110 to display the augmented reality image rendered by the AR module 120 on at least one display.

[0082] In addition, the processor 100 can receive information related to the distance between each of the identified objects and the vehicle 200 (hereinafter referred to as the relative distance) and the angle of the position of each of the identified objects with respect to the vehicle 200 (hereinafter referred to as the relative angle) from the ADAS 210. In addition, information related to the traveling direction and moving speed of each of the identified objects corresponding to the current traveling direction and the current speed of the vehicle 200 (hereinafter referred to as the relative speed) can be received.

[0083] Next, the processor 100 can estimate the position of a specific object after a specified time based on the received object information, for example, information related to the relative speed, relative angle, and relative distance. And the camera that includes the specific object can be determined based on the estimated position.

[0084] As an example, if the specific object is a vehicle traveling in the opposite lane in the direction opposite to the vehicle 200, the front camera of the vehicle 200 can acquire an image including the specific object. And the processor 100 can control any one of the displays to display an augmented reality image in which an augmented reality object corresponding to the specific object is displayed around the specific object according to the recognition result of the specific object. Among them, the camera that first acquires the image including the specific object can be determined as the first camera.

[0085] Next, the ADAS 210 can calculate the relative speed, relative distance, and relative angle of the specific object included in the image of the first camera and provide them to the processor 100. Next, the processor 100 can estimate the position of the specific object after a specified time based on the relative speed, relative distance, and relative angle of the specific object received from the ADAS 210. For example, the processor 100 can estimate the position of the specific object after the specified time based on the time point when the front camera does not detect the specific object.

[0086] And based on the estimated position of the specific object, other cameras (hereinafter referred to as the second cameras) with a high possibility of the appearance of the specific object can be determined. In this case, the processor 100 can determine the camera having a perspective that includes the estimated position of the specific object after the specified time as the second camera. That is, as described above, if the specific object is a vehicle traveling in the opposite lane in the direction opposite to the vehicle 200, the processor 100 can determine the rear camera of the vehicle 200 as the camera with a high possibility of the appearance of the specific object, that is, the second camera.

[0087] Next, the processor 100 can detect the object corresponding to the identification information of the specific object identified in the image of the first camera, that is, the characteristic information such as shape, color, and size, from the recognition result of the object included in the image acquired by the second camera by the ADAS 210. And, if the corresponding object is detected, the AR module 120 can be controlled and any one of the displays can be controlled through the interface unit 110 to display the augmented reality image of the augmented reality object corresponding to the specific object displayed around the specific object.

[0088] Therefore, if the object around the vehicle 200 changes its relative position (with the vehicle 200 as the reference position) according to the moving speed of the vehicle 200 or the moving speed of the object, the processor 100 can link the plurality of cameras including the object to track the object according to the changed position of the object. In addition, the augmented reality object corresponding to the tracked object can be continuously displayed through the images obtained from the linked cameras. Therefore, the user can more easily and intuitively recognize the movement of the object detected in a specific direction of the vehicle 200.

[0089] In addition, if the objects included in the image acquired from the second camera do not include the specific object, the processor 100 can control the AR module and the interface unit 110 to display an augmented reality image in which an augmented reality object predicting the appearance of the specific object is superimposed on the image acquired from the second camera.

[0090] For example, if the specific object is in an area outside the respective viewing angles of the plurality of cameras, i.e., a blind spot, the specific object may not be included in the image acquired from the second camera. In this case, the processor 100 may control the AR module 120 to render the image of the second camera, which includes other augmented reality objects (e.g., the second augmented reality object) related to the augmented reality object that announces the appearance of the specific object (the augmented reality object corresponding to the specific object (e.g., the first augmented reality object)). In addition, the display unit 250 may be controlled through the interface unit 110 to display the rendered augmented reality image of the second augmented reality object on any one of the displays.

[0091] The processor 100 may control the AR module 120 so that, in the image captured by the second camera, the second augmented reality object is superimposed in a direction relative to the estimated position of the specific object after a specified time, that is, in a direction facing the estimated position of the specific object, and in a position corresponding to the direction pointing to the estimated position of the specific object. Therefore, even if the specific object is not yet included in the image of the second camera, the user can pre-identify the position where the specific object will appear through the second augmented reality object.

[0092] In addition, if the processor 100 determines that a part of the feature information of the first object recognized at the first time point recognized by the first camera has changed at the second time point recognized by the second camera, it can also control the AR module 120 to display the augmented reality information including the changed feature information in the image of the second camera.

[0093] Moreover, the processor 100 can control the AR module 120 and the interface unit 110 such that the second augmented reality object is displayed differently according to the distance between the vehicle 200 and the speculated position of the specific object after the specified time. As an example, the upcoming appearance of the specific object can also be indicated by increasing the size of the second augmented reality object or changing the saturation or brightness.

[0094] On the other hand, the first camera and the second camera can be determined differently according to the relative position, relative speed, or relative angle of the object. For example, as described above, if the specific object is a vehicle in another lane traveling in the direction opposite to the vehicle 200 in front of the vehicle 200, the front camera of the vehicle 200 may be the first to recognize the specific object. And the processor 100 speculates the position of the specific object after a specified time and can determine the rear camera as the camera that will subsequently detect the specific object according to the speculated position. Therefore, the front camera that first recognizes the specific object, i.e., the front camera, can be the first camera, and the rear camera determined according to the speculated position of the specific object can be the second camera.

[0095] If the specific object is a vehicle in another lane traveling in the same direction as the vehicle 200 and overtaking the vehicle 200 behind the vehicle 200, the rear camera of the vehicle 200 may be the first to recognize the specific object. And the processor 100 speculates the position of the specific object after a specified time and can determine the front camera as the camera that will subsequently detect the specific object according to the speculated position. In this case, the rear camera of the vehicle 200 can be the first camera, and the front camera of the vehicle 200 can be the second camera.

[0096] In addition, the processor 100 can further display an additional augmented reality object for the recognized specific object according to at least one of the relative speed, relative angle, and relative distance of the recognized specific object. For example, if the distance between the recognized specific object and the vehicle 200 is below a constant distance, the processor 100 can control the AR module 120 to render an augmented reality image in which an augmented reality object indicating that the specific object is approaching is superimposed on the image of the camera including the specific object.

[0097] On the other hand, such additional augmented reality objects can be further displayed according to the driving state of the vehicle 200 or the operating state of the vehicle 200. For example, if the indicator light for changing lanes of the vehicle 200 is turned on, the processor 100 can control the AR module 120 and the interface unit 110 based on the objects detected in the lane corresponding to the turned-on indicator light to display an augmented reality object indicating whether the lane to be changed is safe.

[0098] For example, the processor 100 determines whether it is safe to change lanes based on the distance between the other vehicle, which is the object detected in the lane corresponding to the indicator light, and the vehicle 200 and the speed of the other vehicle.

[0099] As an example, if the speed of the other vehicle is below a constant speed and the distance between the vehicle 200 and the other vehicle exceeds a preset safe distance, the AR module 120 can be controlled to display an augmented reality object indicating that it is safe to change lanes. Conversely, if the speed of the other vehicle on the lane corresponding to the indicator light or the relative speed of the other vehicle is above the constant speed, or the distance between the other vehicle and the vehicle 200 is below the preset distance, the AR module 120 can be controlled to display an augmented reality object indicating that it is dangerous to change lanes. In this case, the augmented reality object indicating that it is safe to change lanes and the augmented reality object indicating that it is dangerous to change lanes can be different.

[0100] Moreover, the augmented reality information providing device according to an embodiment of the present invention may further include a communication unit 140. Under the control of the processor 100, the communication unit 140 can be connected to a preset server, such as a cloud server, in a wireless communication manner and can receive information requested by the processor 100, such as POI information and augmented reality content, from the connected cloud server.

[0101] On the other hand, Figure 2 is a flowchart showing an operation process in which the augmented reality information providing device according to an embodiment of the present invention displays an augmented reality object related to an object recognized by a first camera in an image of a second camera.

[0102] Referring to Figure 2 it can be seen that the processor 100 of the augmented reality information providing device according to an embodiment of the present invention can first identify the objects included in the image acquired through the first camera. Therefore, a first object can be identified among the objects included in the image of the first camera (S200). The ADAS 210 can obtain information related to which camera is the camera that acquired the image including the first object. And the information of the camera obtained by the ADAS 210 can be the information of the first camera.

[0103] Here, the first object may be an object automatically selected according to preset conditions or an object selected by a user. As an example, the preset conditions may include at least one of a distance from the vehicle 200, a relative speed with the vehicle 200, and a relative angle. That is, if it is a vehicle whose moving speed exceeds the speed of the vehicle 200 by a specified level or more, it may be automatically selected as the first object. Or, if it is a vehicle whose distance from the vehicle 200 is less than a specified level, it may be automatically selected as the first object. Or, if there is a business point in a building that can provide services corresponding to a merchant type preset by a user, it may be automatically selected as the first object.

[0104] In the step S200, the processor 100 may collect various information obtained from the first object as feature information of the first object. For example, if the first object is a vehicle, the processor 100 may collect feature points such as the form of the vehicle (i.e., size, position of the license plate, position of the exhaust port, shape of the bumper, etc.) and the color of the vehicle as the feature information of the first object. Or, if the first object is a building, the processor 100 may collect the form of the building and its surroundings (e.g., window pattern), the position of the building, the color of the building, the text or image included in the signboard, the form or size of the signboard, etc. as the feature information of the first object.

[0105] Moreover, the processor 100 may control the AR module 120 to render an augmented reality image (first augmented reality image) in which an augmented reality object corresponding to the first object from which the feature information is collected, that is, the recognized first object, is included in the image acquired by the first camera. And, it may control the interface unit 110 to display the rendered augmented reality image on any one of the displays included in the display unit 250 (S202).

[0106] For example, if the first camera is a front camera, an augmented reality image including the augmented reality object (first augmented reality object) corresponding to the first object may be displayed on the CID 251. Or, if the first camera is a rear camera, an augmented reality image including the first augmented reality object may be displayed on the in-vehicle mirror display 252.

[0107] On the other hand, if an augmented reality image of an augmented reality object including the first object is displayed on at least one display, the processor 100 may estimate the position of the first object after a specified time has elapsed from a reference time point. And, based on the estimated position of the first object, at least one camera in which the possibility of acquiring an image including the first object in the camera of the vehicle 200 is high may be determined as the second camera (S204).

[0108] Here, the reference time point can be determined according to the first object. For example, the reference time point can be the time point when the first object is recognized. Or, the reference time point can be the time point when the first object disappears from the image of the first camera. In this case, the processor 100 can continuously monitor whether the first object is included in the image obtained from the first camera.

[0109] Therefore, in step S204, the processor 100 can determine, based on the movement of the vehicle 200 or the movement of the first object, the time point when at least a part of the first object in the image obtained by the first camera is no longer displayed as the reference time point. And the processor 100 can calculate at least one estimated position of the first object after a predetermined time has elapsed from the reference time point.

[0110] Here, the estimated position of the first object can be calculated based on converting the position of the first object detected in the image of the first camera (i.e., the position on the first camera image coordinate system) to the estimated position on the reference coordinate system. In this case, the processor 100 can control the ADAS 210 to convert the coordinates of the first object on the first camera image coordinate system to the coordinates on the reference coordinate system.

[0111] Among them, the reference coordinate system can be a coordinate system with the foot of the perpendicular drawn to the ground centered on the rear axle of the vehicle 200 as the origin, the driving direction of the vehicle 200, that is, the front as the X-axis component with a positive (+) value, the rear of the vehicle as the X-axis component with a negative (-) value, and the left and right sides of the vehicle 200 as the Y-axis. In addition, both the first camera image coordinate system and the reference coordinate system can be three-dimensional coordinate systems. Therefore, the coordinates of the first object on both the first camera image coordinate system and the reference coordinate system can be three-dimensional coordinates representing the three-dimensional position of the first object.

[0112] And the processor 100 can calculate the estimated position of the first object after the predetermined time based on the coordinates of the first object on the reference coordinate system, the speed and driving direction of the vehicle 200, and the speed and driving direction of the first object. And at least one camera different from the first camera can be determined as the second camera according to the calculated estimated position of the first object.

[0113] Among them, the second camera can be a camera whose viewing angle includes the estimated position of the first object after the predetermined time. That is, at least one camera that obtains an image pointing in the direction of the estimated position of the first object after the predetermined time can be determined as the second camera.

[0114] Hereinafter, with reference to Figure 5 , the step S204 of the Figure 2 can be observed in more detail, that is, the step of determining at least one second camera based on the estimated position of the first object after the specified time has elapsed.

[0115] On the other hand, if at least one second camera is determined in the step S204, the processor 100 may respectively determine the direction of the position of the first object or the estimated position of the estimated first object relative to the at least one second camera (S206). For example, if the first object has not disappeared from the image of the first camera, the direction relative to the first object may be the direction facing the first object according to the reference coordinate system, that is, the direction pointing to the position of the first object. However, if the first object disappears from the image of the first camera, the direction relative to the first object may be the direction looking at the estimated position of the first object, that is, the direction pointing to the estimated position of the first object.

[0116] The processor 100 may determine a position corresponding to the relative direction of the determined first object from the image obtained by the second camera. Among them, the processor 100 may determine a position corresponding to the relative direction of the determined first object within the display area of the image obtained by the second camera and displayed on the display.

[0117] Moreover, the processor 100 may control the AR module 120 to render an augmented reality image of an augmented reality object related to the first object at a position corresponding to the relative direction of the determined first object according to the image of the second camera (S208).

[0118] Among them, the augmented reality object related to the first object may be the first augmented reality object. Or, it may be a second augmented reality object different from the first augmented reality object.

[0119] For example, if the image obtained from the second camera includes the first object, the area where the first object is displayed may be the position corresponding to the relative direction of the first object. Then, the processor 100 may control the AR module 120 to render an augmented reality image in which the first augmented reality object is superimposed around the area where the first object is displayed in the image obtained by the second camera. And, it may control the interface unit 110 to display the augmented reality image (second augmented reality image) including the first augmented reality object on at least one display selected by the user.

[0120] Among them, the display for displaying the first augmented reality image, that is, the display for displaying the augmented reality image based on the image of the first camera, and the display for displaying the second augmented reality image, that is, the display for displaying the augmented reality image based on the image of the second camera, can be different displays from each other. For example, if the first augmented reality image is displayed on the CID251, the second augmented reality image can be displayed on the in-vehicle mirror display 252. Conversely, if the first augmented reality image is displayed on the in-vehicle mirror display 252, the second augmented reality image can be displayed on the CID251.

[0121] Alternatively, the first augmented reality image and the second augmented reality image can also be displayed on any one display. In this case, the processor 100 can distinguish the display areas of the any one display and display the first augmented reality image and the second augmented reality image in different areas.

[0122] On the other hand, if the image obtained from the second camera does not contain the first object, that is, the first object is located in the blind area between the first camera and the second camera, then in the image of the second camera, the position corresponding to the relative direction of the first object can be a certain position on the virtual extension line connecting the estimated position of the first object from the FOV (Field Of View) of the second camera. In this case, the position corresponding to the relative direction of the first object can be a certain position within the display area that is adjacent to the intersection of the edge of the display area displayed on the display in the image of the second camera and the virtual extension line.

[0123] Next, the processor 100 can display an augmented reality object related to the first object at the position corresponding to the relative direction of the determined first object in the image of the second camera. In this case, the augmented reality object related to the first object can be an augmented reality object for notifying the appearance of the first object. In this case, the augmented reality object for notifying the appearance of the first object can be an augmented reality object different from the first augmented reality object (hereinafter referred to as the second augmented reality object).

[0124] Next, the processor 100 can control the AR module 120 to render an augmented reality image in which the second augmented reality object is superimposed at the position corresponding to the relative direction of the determined first object in the image obtained from the second camera. And it can control the interface unit 110 to display the second augmented reality image including the second augmented reality object on at least one display selected by the user.

[0125] Hereinafter, refer to the following Figure 7, observe in more detail the operation process of the processor 100 that controls the AR module 120 and the interface unit 110 in step S208 to display a second augmented reality image including an augmented reality object related to the first object on the display.

[0126] Figure 3 and Figure 4 As shown, a second augmented reality object that pre-announces the appearance of the first object is shown as the augmented reality object related to the first object.

[0127] Figure 3 is a schematic diagram showing the relative movement of surrounding objects (first objects) as the vehicle equipped with the augmented reality information providing device according to an embodiment of the present invention moves. And, Figure 4 It is assumed that the augmented reality image corresponding to the image acquired by the front camera of the vehicle 200 is displayed on the CID 251, and the augmented reality image corresponding to the image acquired by the rear camera of the vehicle 200 is displayed on the in-vehicle mirror display 252.

[0128] First, referring to Figure 3 to see, if the vehicle 200 moves forward, the first object 301 detected from the right side of the vehicle 200 can move according to the speed of the vehicle 200 or the speed of the first object 301. In this case, if the first object 301 is a vehicle moving in the opposite direction to the vehicle 200, it can move in the direction opposite to the moving direction of the vehicle 200 at a speed equal to the sum of the moving speed of the first object 301 and the moving speed of the vehicle 200. Or, if the first object 301 is a building that cannot move autonomously, it can move in the direction opposite to the moving direction of the vehicle 200 according to the moving speed of the vehicle 200. Therefore, as Figure 3 shown, the first object 301 can move from the position at the first time point to the position at the third time point.

[0129] On the other hand, if the first object 301 is located at the position corresponding to the first time point, since the first object exists in front of the vehicle 200, the image acquired by the front camera of the vehicle 200 may include the first object. In this case, the processor 100 can recognize the first object based on the image acquired by the front camera. Therefore, as Figure 4 shown, if the image acquired by the front camera of the vehicle is displayed on the CID 251, the processor 100 can control the AR module 120 according to the result of recognizing the first object 301, so that a first augmented reality image including the first augmented reality object 411 related to the first object is displayed on the CID 251, that is, as shown in (a) of Figure 4 .

[0130] In such a state, as the vehicle 200 or the first object 301 moves, or both the vehicle 200 and the first object 301 move, the first object 301 can move to a position corresponding to the second time point. In this case, since the first object 301 is located in the blind spot between the front camera and the rear camera, both the front camera and the rear camera may not be able to detect the first object.

[0131] Thus, if the first object 301 disappears from the image of the front camera, which is the first camera that first detects the first object 301, the processor 100 can estimate the position of the first object 301 after a specified time. To this end, the processor 100 can estimate the position of the first object 301 based on the coordinates of the first object 301 corresponding to the reference coordinate system calculated by the ADAS 210 at the reference time point, and considering the relative speed, relative distance, and relative angle of the first object 301 with respect to the vehicle 200.

[0132] The estimation result is as described in Figure 3 As shown, as at least one of the vehicle 200 and the first object 301 moves, the first object 301 moves from the position at the first time point to the position at the third time point. Therefore, the processor 100 can determine the rear camera that captures the image of the rear of the vehicle as the second camera.

[0133] As described in Figure 2 Step S208, the processor 100 can determine the position corresponding to the relative direction of the first object, and control the AR module 120 to render a second augmented reality image in which a second augmented reality object 412 for predicting the appearance of the first object 301 is included in the image captured by the second camera. And the interface unit 110 can be controlled to display the second augmented reality image on the in-vehicle mirror display 252 that displays the image captured by the second camera.

[0134] Therefore, as shown in Figure 4 (b), as the first object 301 is located in the blind spot, neither the front camera, which is the first camera, nor the rear camera, which is the second camera, may display the first object. However, the processor 100 can determine the position corresponding to the relative direction of the first object in the display area displayed on the display (in-vehicle mirror display) 252 in the image of the second camera (rear camera) based on the estimated position of the first object 301 after a specified time. And at the determined position, the second augmented reality object 412 for predicting the appearance of the first object 301 can be displayed on the display that displays the image of the second camera. Therefore, asFigure 4 As shown in (b) of FIG.

[0135] On the other hand, as the vehicle 200 or the first object 301 moves, or as both the vehicle 200 and the first object 301 move, the first object 301 may move to a position corresponding to the third time point. In this case, since the first object 301 is located behind the vehicle 200, the first object 301 is included in the image obtained from the rear camera.

[0136] Therefore, the processor 100 may display an augmented reality object related to the first object 301 around the first object 301 displayed in the image of the rear camera, that is, display the first augmented reality object 411. That is, as the first object 301 is included in the rear camera image, the second augmented reality object 412 disappears, and the first augmented reality object 411 may be displayed.

[0137] In addition, the first augmented reality object 411 of the first object 301 displayed in the image of the first camera may be seamlessly displayed in the images of other cameras as the vehicle 200 or the first object 301 moves.

[0138] Therefore, even if the vehicle 200 passes a specific POI, the information of the passed POI can be confirmed and selected through the augmented reality object displayed by the other camera (second camera). In addition, as can be seen, different plural cameras displaying the first object 301 can be linked to provide POI information about the first object 301, thereby providing further improved navigation information.

[0139] Moreover, if the recognized object enters the blind spot, the augmented reality information providing device according to an embodiment of the present invention may provide information related to the position of the recognized object through the augmented reality object (second augmented reality object) displayed in the images of at least one camera. Therefore, even if the object is not displayed in the camera image because it is located in the blind spot, the user can know its existence and location through the augmented reality object.

[0140] The processor 100 may calculate the estimated position of the first object, and determine the second camera according to the calculated estimated position.

[0141] Figure 5It is a flowchart showing the operation process of the second camera determined by the augmented reality information providing apparatus according to an embodiment of the present invention based on the estimated position of the first object after a predetermined time. And, Figure 6 It is a schematic diagram showing the estimated position of at least one of the estimated first objects calculated.

[0142] Referring to Figure 5 As shown, if the processor 100 of the augmented reality information providing apparatus according to an embodiment of the present invention displays an augmented reality image including an augmented reality object (first augmented reality object) of the first object recognized from the image of the first camera on the display, the coordinates of the first object may be converted into a reference coordinate system (S500).

[0143] First, if the first object is recognized, the ADAS 210 may calculate the initial coordinates of the first object. In this case, the initial coordinates of the first object may be coordinates calculated based on the image coordinate system of the first camera including the first object. Then, the processor 100 may control the ADAS 210 to convert the initial coordinates of the first object based on the image coordinate system of the first camera into the reference coordinate system.

[0144] To convert to such a reference coordinate system, the processor 100 may use first camera calibration data. The first camera calibration data is data including information such as the position (X, Y, Z), pointing direction (Roll: roll, Pitch: pitch, Yaw: yaw), viewing angle of the first camera, and FPS (Frame Per Second: frame rate) of the first camera based on the origin of the reference coordinate system, and may be data for converting the coordinates of the first camera image coordinate system into the reference coordinate system. The first camera calibration data may be data pre-stored in the memory 130.

[0145] And, under the control of the processor 100, the ADAS 210 may obtain vehicle position information including the moving speed, position, and driving azimuth angle (heading) of the vehicle 200. And, based on the obtained vehicle position information and the three-dimensional position information of the first object, the relative distance, relative angle, and relative speed between the first object and the vehicle 200 may be calculated.

[0146] For example, when the coordinates of the first object 301 on the converted reference coordinate system are (x1, y1), the processor 100 may calculate the distance D (relative distance) from the origin (0, 0) of the reference coordinate system to the first object based on the moving distance S = speed (V) × time (T) according to the following mathematical formula 1.

[0147]

Formula 1

[0148]

[0149] Moreover, the angle θ (yaw, relative angle) from the origin (0, 0) of the reference coordinate system to the first object can be calculated according to the following Mathematical Formula 2.

[0150]

Formula 2

[0151]

[0152] Next, the processor 100 can estimate the position of the first object at at least one time point after a predetermined time from the reference time point (S502) based on the relative distance, relative angle, and relative speed of the first object calculated by the ADAS210.

[0153] For example, as Figure 6 shown, when the time point at which the first object 301 disappears from the first camera is set as the reference time point t1 (601), the reference time point t1 (601) can be the time point at which the relative distance D approaches below a preset distance and the relative angle θ exceeds a preset range.

[0154] In this case, when the longitudinal relative speed and the lateral relative speed detected from the first object 301 are set as Rx and Ry respectively, the estimated position (x2, y2) of the first object 301 at the time point t2 after a predetermined time from the reference time point t1 (601) can be calculated according to the following Mathematical Formula 3.

[0155]

Formula 3

[0156]

[0157] Among them, (x2, y2) are the estimated coordinates of the first object at the time point t2 after a predetermined time from the reference time point t1, (x1, y1) are the coordinates of the first object at the reference time point t1, Rx is the longitudinal relative speed of the first object, and Ry is the lateral relative speed of the first object.

[0158] Among them, the time point t2 (hereinafter referred to as the second time point) after the specified time can be a time point that monotonically increases from the reference time point t1 (601). That is, t2 = t1 + n, where n = {1, 2, 3, 4...}. Therefore, by sequentially increasing the preset unit time to make n increase uniformly, a plurality of different time points t2 after the specified time can be calculated. In this case, as the second time point t2 increases, the distance (x2) on the x-axis coordinate and the distance (y2) on the y-axis coordinate from the origin (0, 0) of the reference coordinate system also increase. Therefore, as Figure 6 shown, a plurality of different first object speculation positions 602 can be speculated.

[0159] The processor 100 can determine a second camera (S604) from a plurality of cameras provided in the vehicle 200 based on at least one of the speculated first object speculation positions 602. For example, the processor 100 can set the viewing angle to any camera that can capture an image of the area where the first object speculation positions corresponding to the plurality of time points t2 after the specified time are distributed, and determine it as the second camera. That is, as Figure 6 shown, if a plurality of first object speculation positions 602 are distributed within the viewing angle of the rear camera, the processor 100 can determine the rear camera as the second camera.

[0160] On the other hand, in the above description, an example is given in which the first object speculation position after the specified time is calculated based on the relative speed, relative distance, and relative angle speculated by the ADAS210. However, it can be different. The position of the first object can be speculated based on the position information of the vehicle 200 and the first object.

[0161] For example, the processor 100 can be based on the driving speed, GPS position, and traveling azimuth angle of the vehicle 200 to speculate the GPS position of the vehicle 200 at the second time point after the specified time from the reference time point (for example, t1 (601)). And based on this, the relative moving position of the first object can also be speculated on the vehicle reference coordinate system.

[0162] On the other hand, Figure 7 is a flowchart showing the operation process of the augmented reality information providing device according to an embodiment of the present invention to display an augmented reality object related to a first object recognized in a first camera image in an image of a second camera.

[0163] The processor 100 of the augmented reality information providing device according to an embodiment of the present invention, if it executes Figure 2In step S208 that displays the augmented reality object, first, an object included in the image obtained from the second camera can be recognized (S700). And, based on the recognition result, it can be detected whether the image obtained from the second camera includes a first object (S700).

[0164] The Figure 2 As described in step S200 above, when ADAS 210 recognizes the first object in the image of the first camera, it can recognize the first object by features related to the form such as the size, shape, and color detected from the first object. And, the features used to recognize the first object can be stored as the feature information of the first object. For example, if the first object is a vehicle, the processor 100 can collect feature points such as the form of the vehicle (i.e., the size or the position of the license plate, the position of the exhaust port, the shape of the bumper, etc.) and the color of the vehicle as the feature information of the first object. Or, if the first object is a building, the form around the building and the building, the position of the building, the color of the building, the text or image included in the signboard, the form or size of the signboard, etc. can be collected as the feature information of the first object.

[0165] As described above, the collected and stored feature information of the first object can be used to detect an object corresponding to the first object from the image obtained from the second camera. For example, if the first object is a building, the processor 100 can detect, from the objects detected by the second camera, an object corresponding to the collected and stored feature information from among the objects corresponding to "building". In addition, if the first object is "vehicle", the processor 100 can determine an object having features most similar to the form feature points of the recognized first object including the size and color included in the feature information of the first object as the first object. Or, the processor 100 can further use plural first object speculation positions to further improve the detection accuracy and detection speed of the object corresponding to the first object. For example, the processor 100 can generate virtual objects corresponding to the feature points of the first object at the plural first object speculation positions that are speculated, and based on the generated virtual objects, detect an object corresponding to the first object from the image of the second camera. Regarding the operation process of such a processor 100, a more detailed description will be given below with reference to Figure 9 for a more detailed description.

[0166] In the detection result of step S702, if the image acquired by the second camera contains at least a part of the object corresponding to the first object, the processor 100 can determine that the image acquired by the second camera contains the object corresponding to the first object. Therefore, the AR module 120 and the interface unit 110 can be controlled to display an augmented reality image (a second augmented reality image) around the area where the object corresponding to the first object is displayed in the image of the second camera, and the augmented reality image (the second augmented reality image) displays the first augmented reality object related to the first object (S704). On the other hand, if the check result of step S702 is that the first object is not detected from the image of the second camera, the processor 100 can check whether the time point at which the estimated position of the first object is included in the field of view of the second camera has been reached (S706). And, if the check result of step S706 is that the time point at which the estimated position of the first object is included in the field of view of the second camera has not been reached, the processor 100 can determine that the first object is located in the blind spot between the field of view of the second camera and the first camera.

[0167] Therefore, the processor 100 can determine the direction relative to the inferred position of the first object in the image acquired by the second camera. And the AR module 120 can be controlled to render an augmented reality image showing other augmented reality objects related to the first object, i.e., a second augmented reality object, according to the determined direction (S708). The second augmented reality object, as an augmented reality object that announces the appearance of the first object, can be an object different from the first augmented reality object.

[0168] Here, the position corresponding to the relative direction of the position of the first object may be a position determined based on a virtual line connecting the reference point of the image of the second camera to the inferred position of the first object. For example, the processor 100 may generate a virtual extension line connecting the center of the FOV of the second camera to the inferred position of the first object, and in the image of the second camera, determine the position where the edge of the display area displayed on the display intersects with the virtual extension line as the position corresponding to the relative direction of the position of the first object, and display the second augmented reality object around the determined position. In this case, the second augmented reality object will of course be displayed in the display area in the image of the second camera.

[0169] On the other hand, if, at the time point when the inspection result of step S706 indicates that the speculated position of the first object has fallen within the viewing angle of the second camera, the processor 100 confirms whether the object corresponding to the first object is in an occluded state based on the speculated position of the first object, that is, the speculated position of the first object that is speculated to be included within the viewing angle of the second camera (S710).

[0170] For example, the processor 100 may detect whether another object is displayed in the area corresponding to the speculated position of the first object in the image acquired by the second camera. And if another object is displayed in an area of the second camera corresponding to the speculated position of the first object, it can be determined that the first object is occluded by the displayed object (hereinafter referred to as the occluding object). Then, the processor 100 may control the AR module 120 to display an augmented reality image of the augmented reality object corresponding to the first object (e.g., the first augmented reality object) in the area where the occluding object is displayed, thereby indicating that the first object is located in the area occluded by the occluding object (S712).

[0171] The following will refer to Figure 8 to see an example where an augmented reality image of the first augmented reality object corresponding to the first object is displayed in the area occluded by the occluding object.

[0172] However, if, as a result of the confirmation in step S710, no other object is displayed in the area of the image acquired by the second camera corresponding to the speculated position of the first object, it can be determined that there is no occluding object. Then, the processor 100 may determine that the first object has disappeared based on the fact that the first object is no longer detected in the image of the second camera.

[0173] On the other hand, there may of course be a plurality of the second cameras. In this case, the Figure 7 process described above can be performed separately for each second camera. For example, in the above Figure 6 description, the processor 100 may determine one or more cameras as the second cameras according to the distribution state of the plurality of speculated positions 602 of the first object.

[0174] That is, as Figure 6As shown, if a plurality of first object estimated positions 602 are distributed at the rear right of the vehicle 200, the processor 100 can determine the rear camera as the second camera as described above. In addition, the processor 100 can determine the camera corresponding to the right digital mirror that detects the image of the rear area of the vehicle 200 on the right side of the vehicle 200 as the second camera. That is, both the rear camera and the camera corresponding to the right digital mirror can be determined as the second camera.

[0175] Alternatively, when the first camera is the front camera, the second camera can be the rear camera and the left digital side view mirror camera. In this case, even if the first object, for example, a vehicle in a different lane from the vehicle 200 is in the blind spot between the front camera and the rear camera, the left digital side view mirror camera can detect the first object. In this case, according to the Figure 7 operation process, an augmented reality object (second augmented reality object) predicting the appearance of the first object is displayed on the display showing the image of the rear camera, and in the image of the left digital side view mirror camera, as the first object is detected, of course, the first augmented reality object including the augmented reality information of the first object can also be displayed.

[0176] On the other hand, if more than one second camera is determined as described above, the processor 100 can execute the Figure 7 operation process for each second camera respectively. Therefore, for each second camera, a position pointing to the estimated position of the first object can be determined, and different augmented reality images showing the augmented reality object at the determined position can be rendered. And according to the user's selection, the rendered different augmented reality images can be displayed on at least one display.

[0177] On the other hand, if the first object can move, for example, it is a vehicle, the first object can move irregularly in a direction or at a speed that the processor 100 cannot predict. In this case, if the first object moves irregularly, after a specified time, the position of the first object can exceed the viewing angle of the currently determined second camera.

[0178] In order to detect the first object when such an irregular movement of the first object occurs, if the first object is not detected in the step S702, the processor 100 can check whether the first object is included in the images obtained by other cameras other than the currently determined second camera.

[0179] Therefore, if at least a part of the object corresponding to the first object is not included in the image acquired by the second camera, the processor 100 may check each image acquired by all the cameras provided in the vehicle 200 to see if an attempt has been made to detect the object corresponding to the first object.

[0180] And, if the object corresponding to the first object is not detected in any of the images acquired by all the cameras, the processor 100 may change any one of the other cameras other than the currently determined second camera to the second camera. Then, enter the step S700, identify the objects included in the image acquired by the changed second camera, and enter the step S702 to detect whether the identified objects include the object corresponding to the first object.

[0181] And, if the object corresponding to the first object is included in the image acquired by the changed second camera, the processor 100 enters the step S704 and may control the AR module 120 and the interface unit 110 to display an augmented reality image (second augmented reality image) around the area of the object corresponding to the first object in the image of the changed second camera, and the augmented reality image (second augmented reality image) displays the first augmented reality object related to the first object.

[0182] On the contrary, if the object corresponding to the first object is not detected in the images of all the cameras, the processor 100 may determine that the first object is in a blind area. Therefore, the processor 100 will restore the second camera initially determined for the first object to the second camera again and enter the Figure 7 step S710 to check whether there is an occluding object in the image of the restored second camera that occludes the first object.

[0183] On the other hand, as Figure 8 shown, it is a schematic diagram showing that the augmented reality information providing device according to an embodiment of the present invention displays the position of the first object occluded by an obstacle through an augmented reality object in the step S712. Figure 7

[0184] First, Figure 8 ​FIG. (a) shows an image obtained by the front camera of the vehicle 200 being displayed on the CID 251 and an image obtained by the rear camera of the vehicle 200 being displayed on the in-vehicle mirror display 252. In this case, the AR module 120 identifies an object (first object 301) recognized by the front camera and may display an augmented reality object (first augmented reality object) 411 related to the recognized first object 301 on the CID 251. In this case, the front camera may be the first camera.

[0185] In this case, the processor 100 may sense a user's selection of the first object 301 displayed on the CID 251. For example, it may sense a touch input on the first augmented reality object 411 displayed on the CID 251, or sense the movement of the user's eyeball gazing at the first augmented reality object 411, or sense a gesture of the user for selecting the first augmented reality object 411 (e.g., the pointing of the user's finger), etc. Or it may sense a voice command of the user for selecting the first object 301.

[0186] Next, the processor 100 may provide information related to the first object 301 in the form of augmented reality content according to the sensed user's selection. For example, service information (e.g., menu information) that the first object 301 can provide, or contact information about the first object 301, etc., may be displayed around the recognized first object 301 in the form of the augmented reality content (hereinafter referred to as augmented reality information). And if the user selects any one of the augmented reality information, services corresponding to the selected augmented reality information may be provided, such as selecting a menu or making a call connection to the first object 301, etc.

[0187] On the other hand, if the first object 301 is recognized from the first camera, the processor 100 may estimate the position after a specified time has elapsed from the reference time point according to whether the first object 301 has reached the reference time point (e.g., t1 (601)). And according to the estimated position, the camera with a high possibility of the first object 301 appearing, that is, the camera having a viewing angle including the estimated position of the first object 301, is determined as the second camera. In this case, the second camera may be the rear camera.

[0188] In addition, if the recognized augmented reality object is displayed in the image obtained by the first camera, as the vehicle 200 moves, the first object 301 may go out of the viewing angle of the first camera. For example, if as the vehicle 200 moves, the first object 301 goes out of Figure 6 the t1 (601) time point, then the first object 301 may be like in theFigure 3 At the second time point, the position is in the blind area between the front camera and the rear camera. In this case, the first object 301 may not be displayed on the CID 251 showing the image captured by the first camera (front camera), nor may it be displayed on the in-vehicle mirror display 252 showing the image captured by the rear camera.

[0189] On the other hand, as described above, if the first camera is the front camera and the second camera is the rear camera, then as Figure 7 in step S706 of, the processor 100 may check whether the time point when the estimated position of the first object 301 is included in the viewing angle position of the second camera has been reached. And, if the check result is that the time point when the estimated position of the first object 301 is included in the viewing angle position of the second camera has not been reached, then as shown in step S708 of Figure 7 it can control the AR module 120 to display the image of the second camera including the second augmented reality object, which is an augmented reality object for notifying the appearance of the first object.

[0190] To this end, the processor 100 may determine the direction opposite to the position of the estimated first object on the in-vehicle mirror display 252 showing the image of the second camera. And it can control the AR module 120 to superimpose and display the second augmented reality object pointing to the determined direction on the display showing the image of the second camera, that is, on the in-vehicle mirror display 252. Therefore, as shown in (b) of Figure 8 the second augmented reality object 412 may be displayed at a position corresponding to the direction in which the first object 301 will appear on the in-vehicle mirror display 252.

[0191] On the other hand, if the check result of step S706 of Figure 7 has reached the time point when the estimated position of the first object 301 is included in the viewing angle position of the second camera, then as shown in step S710 of Figure 7 the processor 100 may determine whether there is another object that blocks the first object 301 in the image of the second camera. Therefore, as shown in (c) of Figure 8 if the estimated position of the first object 301 overlaps with the shape of another vehicle 800 following the vehicle 200 from the rear of the vehicle 200, it can be determined that the other vehicle 800 is an occlusion object that blocks the first object 301. Then, as Figure 8As shown in (c), the processor 100 may superimpose and display the first augmented reality object 810, which corresponds to the first object 301, on other objects 800 included in the image of the occluding object, i.e., the second camera. Therefore, even if other objects included in the image of the second camera occlude the first object 301, the present invention can display the augmented reality object corresponding to the first object 301 so that the user can recognize it.

[0192] On the other hand, as described above, if the first augmented reality object is being displayed, the processor 100 may provide augmented reality information related to the first object 301 based on an input from the user with respect to the first augmented reality object. Therefore, even if the user fails to confirm the first object 301 in the image of the first camera, the first object 301 can be confirmed through the image of the second camera, and augmented reality information related to the first object 301 can be obtained. Moreover, the present invention can display the augmented reality object related to the first object 301 even when the first object 301 is in a state of being occluded by other objects, so that augmented reality information related to the first object 301 can be provided to the user regardless of whether it is occluded.

[0193] On the other hand, the processor 100 may display the image acquired by the camera according to an input made by the user to the display on which the image acquired by the camera is displayed, i.e., an input made by the user to the display on which the image received by the camera that performs the digital mirror function is displayed. For example, the processor 100 may magnify (zoom in) or reduce (zoom out) the image of a specific camera displayed on the display based on a multi-touch input from the user. Alternatively, the camera image may also be zoomed in or out based on the touch position of a single point applied to the display on which the camera image is displayed. That is, the reference point for zooming in or out may be different based on the touch position.

[0194] On the other hand, even for the same object, the shape of the object recognized by the first camera may be different from the shape of the object recognized by the second camera. For example, if the front camera and the rear camera are the first camera and the second camera, respectively, the first camera may include the front shape of the first object, while the second camera may include the rear image of the first object. Therefore, if the augmented reality information providing device according to an embodiment of the present invention recognizes the first object, it detects the feature points of the recognized first object, and based on the detected feature points, detects the object having the most similar features from the images of other cameras, and thus uses it as the object corresponding to the first object.

[0195] For example, if the first object is a vehicle, the processor 100 may collect feature points such as the form of the vehicle (i.e., the size, the position of the license plate, the position of the exhaust port, the shape of the bumper, etc.) and the color of the vehicle as the feature information of the first object. Or, if the first object is a building, the form around the building and the building (e.g., window patterns), the position of the building, the color of the building, the text or image included in the signboard, and the shape or size of the signboard, etc. may be collected as the feature information of the first object.

[0196] On the other hand, the processor 100 may, among the objects included in the image of the second camera, determine any object having a high similarity to the collected feature information of the first object as the object corresponding to the first object, that is, determine it as the object observing the first object from the perspective of the second camera.

[0197] To this end, the processor 100 projects the outer shape area obtained from the recognition result of the first object onto the positions of a plurality of speculated first objects, and then may detect the object in the image of the second camera corresponding to the first object according to whether the outer shape area is consistent with the projected outer shape area.

[0198] Figure 9 is a flowchart showing the operation process of the augmented reality information providing device according to an embodiment of the present invention for detecting the object corresponding to the first object in the image of the second camera in this way. And, Figures 10 to 11 is shown according to the Figure 9 operation process, a schematic diagram of detecting an object whose outer shape area is consistent with the outer shape area at the speculated position of the recognized first object as the object corresponding to the first object. Among them, the Figure 9 operation process may be performed to determine whether a first object is detected in the Figure 7 S702 step.

[0199] Referring to Figure 9 to see, the processor 100 of the augmented reality information providing device according to an embodiment of the present invention first projects the coordinate information of the positions of a plurality of speculated first objects, that is, candidate positions, speculated on the vehicle reference coordinate system onto the image area of the second camera using the currently determined calibration data of the second camera (S900).

[0200] Figure 10 is a schematic diagram showing that the augmented reality information providing device according to an embodiment of the present invention projects the speculated position of the first object after a specified time and determines the corresponding candidate position in the image of the second camera.

[0201] For example, as Figure 10As shown, the first object 301 can be recognized from the first camera (front camera) 211. In this case, the position of the first object 301 can be calculated based on the three-dimensional coordinates (primary coordinates) of the image coordinate system of the first camera 211.

[0202] Next, the processor 100 can control the ADAS 210 to convert the primary coordinates of the first object 301 into the vehicle reference coordinate system using the first camera calibration data 1010 corresponding to the first camera 211. Then, the first coordinates can be converted into coordinate information based on the vehicle reference coordinate system. And if the first coordinates are converted into coordinate information based on the vehicle reference coordinate system, the processor 100 can calculate the predicted position of the first object 301 after a specified time based on the relative speed, relative distance, and relative angle between the first object 301 and the vehicle 200. And at least one second camera can be determined based on the calculated predicted position.

[0203] On the other hand, if the second camera is the rear camera 212, the processor 100 can use the calibration data 1020 of the second camera 212 to convert the coordinates of each predicted position converted into the coordinate information based on the vehicle reference coordinate system into the coordinates (secondary coordinates 1000) of the image coordinate system of the second camera 212. That is, the candidate positions projected on the image area of the second camera can be determined.

[0204] In step S900, if the first coordinates of the first object are projected on the image area of the second camera based on the calibration data of the second camera, the processor 100 can generate an outline area corresponding to the feature information recognized from the first object at each position in the projected image of the second camera. Therefore, at multiple candidate positions in the image of the second camera, multiple candidate objects with the same outline area as the outline area based on the feature information recognized from the first object can be generated respectively (S902).

[0205] Next, the processor 100 can detect at least some of the candidate objects included in the display area displayed on the display in the image of the second camera (S904). And the processor 100 can check whether there is a candidate object among the objects recognized in the image of the second camera whose feature information including the outline area is consistent with the feature information obtained from the first object by more than a specified ratio (S906).

[0206] For example, when the processor 100 identifies an object from the image of the second camera, it can determine the size of the contour area associated with each object included in the image of the second camera. Moreover, it can detect an object whose size of the contour area is more than a specified ratio consistent with that of the first object. Furthermore, among the objects whose size of the contour area is more than a specified ratio consistent, it can detect an object whose display position is more than a specified ratio consistent with the contour area of the display candidate object.

[0207] In addition, the processor 100 can not only use the size and position of the contour area, but also use at least one other feature information identified from the first object to detect an object corresponding to the first object among the objects included in the image of the second camera. For example, if there are two or more objects whose display positions are more than a specified ratio consistent with the contour area of the display candidate object, the processor 100 can use other feature information of the first object, such as color information, to detect an object corresponding to a certain first object.

[0208] If, as a result of the detection in step S906, among the objects identified in the second camera, there is a candidate object whose contour area and features are more than a specified ratio consistent, the processor 100 can infer the object in the image of the second camera corresponding to the candidate object detected in step S906 as the object corresponding to the first object. Therefore, in step S702 Figure 7 if the second camera includes an object corresponding to the first object, it can enter step S704 Figure 7 for displaying the first augmented reality object.

[0209] Conversely, if, as a result of the detection in step S906, among the objects identified in the second camera, there is no candidate object whose contour area and features are more than a specified ratio consistent, the processor 100 can determine that the image of the second camera does not include an object corresponding to the first object (S910). Therefore, as in step S702 Figure 7 if the image of the second camera does not include an object corresponding to the first object, it can enter Figure 7 step S706 to check whether it has reached the time point when the inferred position of the first object is within the viewing angle of the second camera.

[0210] Figure 11 It is a schematic diagram showing that the augmented reality information providing device according to an embodiment of the present invention detects an object corresponding to the first object based on the generated candidate object.

[0211] Referring to Figure 11 and looking, first, as Figure 11As shown in (a) of [the figure], if the front camera (the first camera) recognizes the first object, the processor 100 can, based on the second camera calibration data, project the coordinate information on the reference coordinate system calculated based on the coordinate information (the first coordinate) of the first object 301 onto the coordinate on the second camera image coordinate system. Therefore, as shown in Figure 11 the (a) of [the figure], the first candidate position 1110 and the second candidate position 1120 can be determined.

[0212] Next, the processor 100 can respectively generate virtual objects (i.e., candidate objects 1111, 1121) with an outer shape area corresponding to the feature information of the first object 301, such as the size of the outer shape area, obtained when recognizing the first object 301 at the first candidate position 1110 and the second candidate position 1120. Among them, the first candidate object 1111 and the second candidate object 1121 can be three-dimensional objects with a three-dimensional outer shape area. Therefore, as shown in the Figure 11 the (b) of [the figure] showing the in-vehicle mirror display 252 displaying the image of the second camera, i.e., the rear camera, the first candidate object 1111 and the second candidate object 1121 with a three-dimensional shape can be generated.

[0213] On the other hand, as shown in Figure 11 the (a) and (b) of [the figure], if candidate objects 1111, 1121 with a three-dimensional shape are generated, the processor 100 can detect an object whose outer shape area is the same as that of the first object 301 by a specified ratio or more from the objects recognized in the image of the second camera. And among the detected objects, an object whose outer shape area is consistent with any one of the candidate objects 1111, 1121 by a specified ratio or more can be detected.

[0214] That is, as shown in Figure 11 the (a) and (b) of [the figure], among the objects recognized in the image of the second camera, if there is an object 1150 whose outer shape area overlaps with that of the first candidate object 1111 by a specified ratio or more and whose size of the outer shape area is similar to that of the first object 301, the processor 100 can determine that the image of the second camera contains an object corresponding to the first object 301 ( Figure 9 step S908 of [the figure]).

[0215] On the contrary, among the objects recognized in the image of the second camera, if there is no object whose size of the outer shape area is similar to that of the first object 301, or if there is no object whose size of the outer shape area is similar but whose outer shape area overlaps with that of the first candidate object 1111 or the second candidate object 1121 by a specified ratio or more, the processor 100 can determine that the image of the second camera does not contain an object corresponding to the first object 301 ( Figure 9step S910).

[0216] On the other hand, the above description illustrates that among the objects of the second camera, an object whose outer shape region overlaps with that of any one of the candidate objects by more than a specified ratio is presumed to be the object corresponding to the first object. However, of course, if there is a candidate position, that is, a position where the distance between the position obtained by converting the presumed position on the reference coordinate system into the image coordinate system of the second camera and the center position of the object is below a specified distance, then it can also be presumed that the image of the second camera contains the object corresponding to the first object. For example, the center position can be the position of at least one object within the range where the size of the outer shape region is the same as or similar to that of the first object among the objects included in the second camera on the two-dimensional plane (XY plane).

[0217] More specifically, as Figure 11 shown, the center position can be the center point of the upper edge in the outer shape region on the XY plane of the object recognized by the second camera. In this case, the plurality of candidate objects 1111, 1121 can be objects generated in such a way that the center of the upper edge of the bottom surface of the virtual object with an outer shape region having a size corresponding to the first object is located at each candidate position 1110, 1120.

[0218] On the other hand, if the augmented reality information providing device according to an embodiment of the present invention detects a safety driving-related event that needs to be provided to the driver within the image area of the second camera, it can display the specified image area of the second camera in various ways to highlight the specified image area of the second camera.

[0219] As an example, if the processor 100 detects that another object is approaching within a preset distance around the vehicle 200, in order to highlight the detected object, a region within the image of the second camera containing the detected object can be highlighted. Or, if the user sets a preset driving of the vehicle 200, the processor 100 will detect the surrounding situation of the vehicle 200 and use augmented reality objects to display to the user whether the detected surrounding situation of the vehicle 200 is safe according to the set driving. Figures 12 to 13 This kind of example is shown.

[0220] First, referring to Figure 12 as shown in Figure 12 (a) of, if another vehicle 1200 approaching from behind the vehicle 200 is at a distance greater than a preset distance from the vehicle 200, the processor 100 can determine that the other vehicle 1200 is at a distance above the safe distance. In this case, as Figure 12As shown in (a), the image of the rear camera that does not include augmented reality objects or the like can be displayed on the in-vehicle mirror display 252.

[0221] However, if the distance between the other object 1200 and the vehicle 200 approaches below a preset distance, the processor 100 can guide the rear collision warning with augmented reality information. To this end, the processor 100 can calculate an image range 1260 that simultaneously satisfies the front of the other vehicle 1200 approaching from behind the vehicle 200, the road surface of the approaching path, and the aspect ratio of the in-vehicle mirror display 252. Among them, the processor 100 calculates the image area in the rear camera image that simultaneously includes the lane (target lane) where the other vehicle 1200 is located and the adjacent vehicles within the dangerous distance on the target lane.

[0222] Next, the processor 100 can set a region corresponding to the image range 1260 from the image obtained by the rear camera with the other vehicle 1200 as the center according to the calculated image range 1260. And the image of the set region can be enlarged to match the resolution of the in-vehicle mirror display 252 on which the image of the rear camera is displayed.

[0223] Therefore, as Figure 12 shown in (b), the image of the other vehicle 1200 that is closer than the specified distance is enlarged and more prominently displayed on the in-vehicle mirror display 252. Moreover, as Figure 12 shown in (b), the processor 100 can display an augmented reality object 1252 for approach warning near the image of the other vehicle 1200 that is closer than the specified distance to more prominently display the approaching vehicle 1200.

[0224] On the other hand, the above description assumes the case where the distance between the other vehicle 1200 and the vehicle 200 approaches below a preset distance. However, of course, it can also occur according to the user's selection or the vehicle operation preset by the user.

[0225] For example, if the driver gazes at the in-vehicle mirror display 252 for a constant time for braking, or observes the rear situation through the in-vehicle mirror display 252 after braking, then as shown in (b) above, the processor 100 can detect other vehicles around the approaching vehicle 200, and calculate the time to collision (TTC, Time to Collision) based on the positions of the detected other vehicles and the vehicle 200. And if there is an object with a time to collision below a specified value, it can be prominently displayed using an augmented reality object as shown in (b) above for emphasis. Figure 12 shown in (b) above, the processor 100 can detect other vehicles around the approaching vehicle 200, and calculate the time to collision (TTC, Time to Collision) based on the positions of the detected other vehicles and the vehicle 200. And if there is an object with a time to collision below a specified value, it can be prominently displayed using an augmented reality object as shown in (b) above for emphasis. Figure 12 shown in (b), it can be prominently displayed using an augmented reality object for emphasis.

[0226] On the other hand, if the distance between the other approaching vehicle 1200 and the vehicle 200 exceeds the preset distance again, or if the approaching other vehicle 1200 goes beyond the set image area as the distance from the vehicle 200 increases, the processor 100 can restore the initial set area of the rear camera again. Conversely, if the distance between the approaching vehicle 1200 and the vehicle 200 becomes closer, the processor 100 will recalculate the image area and adjust the image of the in-vehicle mirror display 252 that displays the image of the rear camera according to the recalculated image area. In this case, the image on the adjusted in-vehicle mirror display 252 may include an augmented reality object for highlighting the other vehicle 1200 more prominently.

[0227] On the other hand, if the distance between the other vehicle 1200 and the vehicle 200 remains unchanged for a specified time or more, the processor 100 can keep the position of the other vehicle 1200 within the set image area for a specified time or more. Then, the processor 100 can determine that the rear collision warning has been sufficiently provided to the driver and can restore the image setting of the rear camera image to the initial set area.

[0228] The above description is given by taking the rear camera as an example, but it can also be applied to a side camera that obtains a side image of the vehicle 200 or a front camera that obtains a front image. In this case, the display that displays the acquired image may not be the in-vehicle mirror display but another display.

[0229] On the other hand, if the user operates the indicator light for driving the vehicle 200, the processor 100 of the augmented reality information providing device according to an embodiment of the present invention can also detect the surrounding situation of the vehicle 200 and use an augmented reality object to display to the user whether it is safe to drive according to the indicator light in the detected surrounding situation of the vehicle 200. Figure 13 Such an example is shown. For ease of explanation, it is assumed that the camera for detecting the object around the vehicle 200 is a rear camera, and further, it is assumed that the display for displaying the image of the rear camera is an in-vehicle mirror display.

[0230] Referring Figure 13 Looking at it, the processor 100 of the augmented reality information providing device according to an embodiment of the present invention can sense the user's selection to change lanes. For example, if the user turns on the turn signal in a specific direction, or the user gazes at the direction pointing to a specific lane or the mirror corresponding to a specific lane for a constant time or more, or the user selects to change lanes through a voice command, the processor 100 can detect whether the vehicle 200 is currently in a state where it can safely change lanes.

[0231] That is, as described above, if a lane to be changed is specified according to a turn signal or a user's selection, etc., the processor 100 can detect other vehicles 1310 adjacent to the vehicle 200 and located in the specific lane through a rear sensing sensor. And if the distance between the detected other vehicle and the vehicle 1310 is above a preset safety distance as shown in (a) of Figure 13 the following, the processor 100 can display, on the in-vehicle mirror display 252, an image of the rear camera including an augmented reality object 1320 indicating that it is safe to change lanes to the specific lane, that is, an augmented reality image. Among them, the augmented reality object 1320 indicating that the lane change is safe can be displayed in green.

[0232] On the contrary, according to a turn signal or a user's selection, if other vehicles 1350 or obstacles located in a specific other lane are within a safe distance from the vehicle 200, the processor 100 can determine that it is unsafe to change lanes to the specific other lane. Then, the processor 100 can further magnify and display an image of the rear camera including the other vehicle 1350 and the specific lane, making the adjacent other vehicle 1350 more prominent.

[0233] Moreover, as shown in (b) of Figure 13 the following, the processor 100 can display, on the in-vehicle mirror display 252 that displays the image of the rear camera, an augmented reality image including an augmented reality object having a virtual wall shape generated along the shape of the specific other lane in the image of the rear camera. Or, on the in-vehicle mirror display 252 that displays the image of the rear camera, the processor 100 can not only display the augmented reality object of the wall image, but also display an augmented reality object that marks the specific lane in red to show the user that it is unsafe to change lanes to the specific other lane.

[0234] On the other hand, the processor 100 can determine a camera for acquiring an image of the specific lane according to the operation of the user. For example, if the user selects to change lanes to the right lane, the processor 100 can generate an augmented reality image based on the image acquired by the right digital side view mirror camera. Or, if the user selects to change lanes to the left lane, the processor 100 can generate an augmented reality image based on the image acquired by the left digital side view mirror camera. That is, the processor 100 can detect the operation of the user on the vehicle 200 and determine, according to the detected operation, a camera for acquiring an image to which an augmented reality object is to be superimposed.

[0235] In addition, the processor 100 may crop a display area to be displayed on the display from the acquired image, so that the display area includes an area of other vehicles adjacent to the lane and the vehicle 200. In this case, the processor 100 may crop a part of the image acquired by the determined camera, and then scale the cropped part and display it on the display, so as to highlight the area including other vehicles adjacent to the lane and the vehicle 200.

[0236] On the other hand, according to the above description, if the first object enters the blind area as the vehicle 200 moves, the augmented reality information providing device according to the embodiment of the present invention may display a second augmented reality object for notifying the appearance of the first object.

[0237] In this case, if the first object reaches a reference time point, which is the time point when the first object exceeds the viewing angle of the first camera, the processor 100 may calculate a predicted position corresponding to a time point after a specified time has elapsed from the reference time point. And, based on the calculated predicted position, the second camera for acquiring an image including the first object after the specified time is determined.

[0238] On the other hand, the processor 100 may calculate the time required for the first object to enter the viewing angle of the second camera based on the calculated predicted position. And, if the first object passes the reference time point, as time goes by, that is, the closer the first object gets to the time point of entering the viewing angle of the second camera, the second augmented reality object for notifying the appearance of the first object may be displayed differently.

[0239] As described above, Figure 14 is a schematic diagram showing that the augmented reality information providing device according to the embodiment of the present invention displays the second augmented reality object differently according to the predicted position of the first object. In the following description, for the sake of convenience of explanation, it is assumed that the first camera is a front camera, and the second camera is a rear camera. In addition, an example is given in which the image of the rear camera, that is, the image of the second camera, is displayed through the in-vehicle mirror display 252. However, this is only an example for the convenience of explanation, and the present invention is not limited thereto.

[0240] First, referring to Figure 14 of (a), Figure 14 of (a) assumes that the first object has not passed the reference time point (for example, Figure 6 t1 (601) in Figure 14As shown in (a), as the first object is displayed on the CID 251 , the augmented reality image related to the first object may not be displayed on the interior mirror display 252 .

[0241] In this state, as the vehicle 200 moves, if the first object reaches the reference time point, the processor 100 can calculate a plurality of estimated positions of the first object at time points after a predetermined time from the reference time point. Furthermore, the second camera can be determined based on the calculated estimated positions. After the second camera is determined, the time from the reference time point to the time point when the first object enters the field of view of the second camera, i.e., the appearance time, can be calculated based on the estimated position of the first object.

[0242] Furthermore, if the first object enters the area between the viewing angles of the first camera and the second camera, that is, enters the blind spot, the processor 100 cannot detect the first object from the image of the first camera, and the determined image of the second camera does not include the first object. Then, an augmented reality image of an augmented reality object (second augmented reality object) 1400 that predicts the appearance of the first object can be displayed on the interior mirror display 252 in an area of ​​the second camera image corresponding to the estimated position of the first object.

[0243] In this case, as the time point at which the first object appears approaches, at least one of the transparency and color (eg, saturation) of the second augmented reality object 1400 may be different. That is, if the calculated appearance time is greater than the first time, then Figure 14 As shown in (b), the processor 100 may display the second augmented reality object 1400 on the interior mirror display 252 with high transparency or low saturation. However, as time passes, if the remaining time in the calculated appearance time is less than the second time (wherein the first time is greater than the second time), then Figure 14 As shown in (d), the processor 100 may display the second augmented reality object 1400 on the interior mirror display 252 with low transparency or high saturation.

[0244] That is, as time passes, the transparency or saturation of the second augmented reality object may change inversely according to the elapsed time. Figure 14 (b) to Figure 14 As shown in (d), if the first object disappears from the image of the first camera and the first object is getting closer and closer to the time point when it will be included in the image of the second camera, the transparency of the second augmented reality object can be gradually reduced or the saturation can be gradually increased.

[0245] Also, if the object recognized from the image of the second camera includes the object corresponding to the first object, i.e., the first object, as shown in (e) of Figure 14 the processor 100 may display an augmented reality image including the first augmented reality object replacing the second augmented reality object on the in-vehicle mirror display 252 that displays the image of the second camera, i.e., the rear camera.

[0246] On the other hand, although the above description is given by taking the change of displaying the second augmented reality object as an example, of course, the first augmented reality object may also be displayed differently over time. For example, even when the first object is included in the image of the second camera and the first augmented reality object is superimposed and displayed on the image of the second camera, the processor 100 may make the transparency or saturation of the first augmented reality object displayed in the image of the second camera different according to the time elapsed from the time point when the first object disappears from the image of the first camera, e.g., a reference time point. That is, the farther the first object is from the vehicle 200, the first augmented reality object displayed in the image of the second camera may be adjusted to have an increased transparency or a decreased saturation.

[0247] In summary, the present invention can be implemented as computer-readable code on a program recording medium. Computer-readable media include all kinds of recording devices storing data readable by a computer system. Examples of computer-readable media are hard disk drives (HDDs), solid state drives (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and can also be implemented in the form of a carrier wave (e.g., Internet-based transmission). In addition, the computer may also include a control unit. Therefore, the above detailed description should not be construed as restrictive in all aspects, but should be construed as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention should fall within the scope of the present invention.

Claims

1. An augmented reality information providing device, It is characterized in that include: An interface unit receives images including a road environment around a vehicle and objects around the vehicle from a plurality of cameras including a first camera and a second camera, and receives detection information for detecting a driving state of the vehicle; an AR module, which renders at least one augmented reality object according to the detection information based on an image captured by at least one of the plurality of cameras; as well as A processor controls the AR module to render a first augmented reality object related to the first object among the objects contained in the image of the first camera, and controls the display unit of the vehicle through the interface unit to display a second augmented reality object related to the first augmented reality object in a direction opposite to the first object in the image of the second camera.

2. The augmented reality information providing device according to claim 1, It is characterized in that The processor calculates the three-dimensional position and relative speed of the first object detected from the image of the first camera, and estimates the three-dimensional position of the first object according to the time elapsed from the moment when the first object is detected; If the first object is not detected from the image of the first camera, the processor controls the display unit to display a first augmented reality object pointing to the inferred three-dimensional position of the first object in the image of the first camera within a specified time.

3. The augmented reality information providing device according to claim 2, It is characterized in that If there is a user's selection of the augmented reality object, the processor displays information related to the first object.

4. The augmented reality information providing device according to claim 2, It is characterized in that Based on the inferred three-dimensional position of the first object, the processor determines any one of the plurality of cameras that captures an image around the inferred three-dimensional position as the second camera.

5. The augmented reality information providing device according to claim 4, It is characterized in that If the second camera is determined, the processor controls the interface unit so that an image of the second camera including a second augmented reality object is displayed on the display unit, and the second augmented reality object predicts the appearance of the first object based on the inferred position of the first object.

6. The augmented reality information providing device according to claim 5, It is characterized in that If the first object is not detected from the image of the first camera, the processor predicts a time when the first object will be included in the image of the second camera based on the estimated position of the first object; The processor changes at least one of the first augmented reality object and the second augmented reality object according to a time that has passed since a time point when the first object is not detected in an image of the first camera.

7. The augmented reality information providing device according to claim 5, It is characterized in that The processor inversely changes the transparency of the first augmented reality object and the second augmented reality object, or inversely changes the saturation of the first augmented reality object and the second augmented reality object according to the time elapsed since the time point when the first object has never been detected.

8. The augmented reality information providing apparatus according to claim 4, wherein, if the first object is not detected in the image of the determined second camera, the processor controls the interface unit to detect the first object based on an image captured by at least one other camera other than the second camera, and display an augmented reality object related to the first object based on the image captured by any one of the other cameras that has detected the first object.

9. The augmented reality information providing apparatus according to claim 8, wherein, if the first object is not detected in the image captured by the at least one other camera, the processor controls the interface unit to display an augmented reality object related to the first object at a position corresponding to the speculated three-dimensional position in the image captured by the second camera.

10. The augmented reality information providing apparatus according to claim 1, wherein, the processor calculates the distance from the vehicle center to the first object, calculates the moment when the first object exits the viewing angle of the first camera, and calculates the elapsed time based on the calculated moment; the processor speculates the candidate positions of the plurality of first objects as three-dimensional positions according to the different elapsed times for each preset unit time, determines the second camera based on at least one of the candidate positions among the plurality of candidate positions, and identifies the first object included in the image captured by the second camera based on any one of the candidate positions among the plurality of candidate positions.

11. The augmented reality information providing apparatus according to claim 10, wherein, the processor projects the shapes corresponding to the first object onto the plurality of candidate positions within an image area including the area displayed by the display unit, and determines at least one candidate position including at least a part of the shape corresponding to the first object as a final candidate position within a part of the image area displayed by the display unit; the processor speculates, as the three-dimensional position of the first object, any one of the final candidate positions corresponding to an object whose shape overlaps with the shape of the projected first object by a preset ratio or more among the objects included in the image captured by the second camera.

12. The augmented reality information providing apparatus according to claim 1, wherein, the processor detects the occurrence of a preset safety driving related event based on at least one of the distance between the first object included in the image of the second camera and the vehicle, the relative speed of the first object, the traveling direction of the first object, and the traveling direction of the vehicle. If the above-described event related to safe driving occurs, the processor crops a part of the image in the image of the second camera that includes a region containing the first object, and controls the interface unit to scale and display the cropped image.

13. The augmented reality information providing apparatus according to claim 12, wherein, the processor crops a part of the image of the second camera according to the aspect ratio of the display unit, such that at least one of a specific surface of the first object and a proximity path of the first object is included.

14. The augmented reality information providing apparatus according to claim 12, wherein, the processor detects an operation of a preset user on the vehicle, and determines a third camera from among the plurality of cameras based on the detected operation; the processor crops a part of the region in the image captured by the third camera that includes at least one object, and controls the interface unit to scale and display the image of the cropped region on the display unit.

15. The augmented reality information providing apparatus according to claim 12 or 14, wherein, the processor determines the degree of danger for at least one object included in the cropped region based on at least one of distance, speed, and driving direction, and controls the AR module to display different augmented reality objects according to the determined degree of danger.

16. The augmented reality information providing apparatus according to claim 14, wherein, the operation of the user includes: an operation of turning on a turn signal to enter a specific lane or gazing at a mirror corresponding to a specific direction for a constant time or more; the processor determines the camera that captures the lane corresponding to the turned-on turn signal or the specific direction corresponding to the mirror at which the user has gazed for a constant time or more as the third camera.

17. A control method for an augmented reality information providing apparatus, wherein, it includes: a step of identifying a first object included in a first camera among a plurality of cameras provided in a vehicle; a step of displaying a first augmented reality image including a first augmented reality object related to the identified first object; a step of predicting the position of the first object after a predetermined time; a step of determining a second camera that will capture an image of the first object after the predetermined time from among the plurality of cameras based on the predicted position of the first object; a step of determining a display position in a direction opposite to the first object in the image obtained from the second camera; and a step of controlling a display of the vehicle that displays the image obtained from the second camera, such that a second augmented reality object different from the first augmented reality object and related to the first object is displayed at the determined display position.

Images