Display device for a vehicle based on augmented reality and control method for the device

Abstract

The present application is characterized by including: an interface section that receives a sensed image that senses a surrounding environment of a vehicle from at least one sensor provided to the vehicle; a classification information database that includes classification information for classifying objects included in the sensed image into different categories from each other; an object recognition section that classifies each object included in the sensed image into any one of the categories within the classification information database according to the classification information and recognizes the object; a renderer that renders an image to be displayed on a display section provided to the vehicle; and a processor that detects an object corresponding to a category selected by a user from among the objects whose categories are recognized, controls the renderer to render an output image in which the detected object is replaced with a graphic object corresponding to the detected object, and controls the interface section to output the rendered output image on the display section through the interface.

Description

TECHNICAL FIELD

[0001] The present application relates to a display device that displays an image of a surrounding environment of a vehicle. BACKGROUND

[0002] In recent years, there has been an augmented reality (AR) technology that outputs a virtual object in addition to a real object by outputting a graphic object through a windshield and a HUD (Head Up Display) of a vehicle, or by superimposing a graphic object on an image captured by a camera. The AR technology is a technology that adds information related to a travel path of a vehicle to a real world, and enables a driver to intuitively recognize the vehicle and a travel environment of the vehicle. Therefore, it is possible to further improve efficiency and convenience of travel.

[0003] With such an AR technology, researches are actively conducted to provide user-customized information based on an actual real world. In addition, as a part of the researches, an AR information filtering technology has been developed to set a category of a POI (Point Of Interest) from a user in advance, and to display AR information based on POI information received from a cloud server only when a POI corresponding to the pre-set category exists in a surrounding of a vehicle, thereby enabling display of only AR information that the user is interested in.

[0004] However, the AR information filtering technology simply limits display of AR information related to each object of a real world in a surrounding of a user, and the objects of the real world are displayed as they are. That is, it is possible to provide user-customized information only to the displayed AR information, and there is a problem that objects of the real world that are not interesting or not desired to be seen by the user are displayed as they are.

[0005] Furthermore, there is a demand to apply the AR technology to the objects of the real world, for example, to continuously provide information of a specific POI that is desired by the user regardless of the objects of the real world in a surrounding of the user. SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] The present application has been made to solve the above problems and other problems, and provides a display device and a control method of the device using an AR technology, which can not display or simply display objects that are not interesting or not desired to be seen by a user, not only for AR information, but also for objects of a real world.

[0008] Furthermore, an object of the present application is to provide a display device and a control method thereof that can display information on the position and direction of a specific POI desired or intended by a user, independently of objects in the real world, or can display the position and direction of the specific POI that relatively change as a vehicle position moves, independently of objects in the real world, using augmented reality technology.

[0009] Technical solution to the problem

[0010] To achieve the above or other objects, according to an aspect of the present application, a display device according to an embodiment of the present application includes an interface section that receives a sensing image that senses a surrounding environment of a vehicle from at least one sensor provided in the vehicle; a memory that includes a classification information database including classification information for classifying each object included in the sensing image into a plurality of categories that are different from each other in advance; an object recognition section that classifies each object included in the sensing image into any one of the categories in the classification information database according to the classification information; a renderer that renders an image to be displayed on a display section provided in the vehicle; and a processor that detects an object corresponding to at least one category selected by a user from among the objects of which the categories are recognized, controls the renderer to render an output image in which the detected object is replaced with a graphic object corresponding to the detected object, and controls the interface section to output the rendered output image on the display section through the interface.

[0011] In an embodiment, the display device further includes a communication section that communicates with a cloud server that provides POI information including an augmented reality object corresponding to a POI (Position Of Interest) in a surrounding of the vehicle, the processor requests the cloud server for POI information for the POI in the surrounding of the vehicle including information on a current position of the vehicle and the at least one category selected by the user, and if at least one POI information corresponding to the at least one category in the POI in the surrounding of the vehicle is received from the cloud server in response to the request, the processor extracts an augmented reality object from the received POI information, and controls the renderer to render an output image in which the detected object is replaced with the extracted augmented reality object.

[0012] In an embodiment, the cloud server includes POI information classified into the plurality of categories in advance, and the classification information database includes classification information for classifying each object included in the sensing image into the plurality of categories in which the cloud server classifies POI information.

[0013] In an embodiment, the graphic object is a polygon object, which is a graphic object representing a shape of each of the detected objects.

[0014] In an embodiment, the processor deletes the detected object from the sensed image according to the selected at least one category, and controls the renderer to render an output image in which the deleted detected object is not displayed in the sensed image.

[0015] In an embodiment, the processor detects a region in which the detected object is displayed from the sensed image, and initializes color information of the detected region to delete the detected object from the sensed image, the initialized color of the region being different according to a selection of a user.

[0016] In an embodiment, if the detected object is deleted from the sensed image, the processor detects at least one of a color of a peripheral region of the deleted object and a color pattern of the peripheral region, generates a fill image corresponding to a region in which the detected object is deleted in the sensed image according to at least one of the color and the color pattern, and controls the renderer to render an output image in which the fill image is displayed in a region corresponding to the deleted object in the sensed image.

[0017] In an embodiment, the processor further detects an object corresponding to a specific category matching a satisfied condition from the sensed image, if the condition is satisfied.

[0018] In an embodiment, the condition is a condition for at least one of an illumination, a weather, and a time of the vehicle.

[0019] In an embodiment, the condition is a condition for biological information detected by at least one of occupants of the vehicle detected from a sensor portion of the vehicle.

[0020] In an embodiment, the processor detects at least one object located in a direction from the vehicle to a specific POI in the sensed image according to position information of the specific POI and position information of the vehicle, deletes the detected at least one object from the sensed image, and controls the renderer to render an output image in which a graphic object corresponding to the specific POI is displayed in a region corresponding to the deleted at least one object in the sensed image.

[0021] In an embodiment, the graphic object is an augmented reality object corresponding to the specific POI, and the processor receives the position information of the specific POI and the augmented reality object corresponding to the specific POI from the cloud server.

[0022] In an embodiment, the processor calculates a distance between the specific POI and the vehicle according to the position information of the specific POI and the position information of the vehicle, and determines the size of the graphic object according to the calculated distance.

[0023] In an embodiment, the processor detects the objects corresponding to the objects of the restricted deletion category in the objects in the direction pointing to the specific POI in the sensing image, and deletes only the objects other than the objects corresponding to the restricted deletion category.

[0024] In order to achieve the above or other objects, according to an aspect of the present application, an embodiment of the present application is a control method of a display device connected with a camera, at least one sensor, and a display part provided in a vehicle through an interface part, characterized by comprising: a step of receiving a sensing image sensed by the camera; a step of classifying and identifying a category corresponding to each object contained in the sensing image according to classification information for classifying in a preset category different from each other; a step of detecting an object corresponding to at least one specific category selected by a user from the objects of which the category is identified; a step of generating an output image by replacing the detected object with a graphic object corresponding to the detected object respectively; and a step of rendering the output image to output the generated output image on the display part.

[0025] In an embodiment, the step of generating the output image is a step of generating an output image in which the detected object is deleted from the sensing image by initializing a region corresponding to the detected object in the sensing image respectively.

[0026] In an embodiment, the step of generating the output image includes: a step of initializing a region corresponding to the detected object in the sensing image respectively; a step of detecting at least one of a color of a peripheral region of the deleted object and a color pattern of the peripheral region; a step of generating a fill image corresponding to a region in which the detected object is deleted in the sensing image according to at least one of the color and the color pattern; and a step of generating an output image in which the fill image is displayed in a region corresponding to the deleted object in the sensing image.

[0027] In an embodiment, the generating of the output image includes transmitting the specific category information and the location information of the vehicle to a cloud server, requesting POI information of an augmented reality object corresponding to a category of a POI (Position Of Interest) in a periphery of the location of the vehicle; receiving the POI information from the cloud server in response to the request; removing, from the sensed image, an object corresponding to the specific category among objects included in the sensed image; and generating an output image in which the augmented reality object corresponding to the removed object is displayed on an area corresponding to the removed object in the sensed image.

[0028] In an embodiment, the detecting of the object corresponding to the specific category includes detecting a location of a specific POI selected by a user with reference to the location of the vehicle, and detecting at least one object displayed in a direction from the location of the vehicle to the location of the specific POI from the sensed image; the detected object is an augmented reality object corresponding to the specific POI.

[0029] Effects of the Invention

[0030] The effects of the display device of the present invention and the control method of the device are described as follows.

[0031] According to at least one of the embodiments of the present invention, the present invention can classify and recognize each object included in an image sensed by a camera of a vehicle by a predetermined category, simplify the image by removing or replacing an object corresponding to at least one category set according to a predetermined condition or a user setting with a simplified object, and display the simplified image on a display unit. Thus, the present invention has an effect of providing an image of a periphery of the vehicle not including a specific category of object that a user does not want to see or pay attention to.

[0032] Further, according to at least one of the embodiments of the present invention, the present invention has an effect of providing information of a specific POI without an object between the specific POI and the vehicle in the real world by removing the object from an image sensed by a camera, and displaying the information of the specific POI on an area in which the object is removed in the image. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 FIG. 1 is an example view illustrating a configuration of a display device according to an embodiment of the present invention.

[0034] Figure 2 FIG. 2 is a conceptual view illustrating a structure in which a display device according to an embodiment of the present invention provides augmented reality information in cooperation with a cloud server.

[0035] Figure 3 is a flowchart illustrating an action process of a display device according to an embodiment of the present application to output an image including at least one simplified object according to a condition or a category selected by a user.

[0036] Figure 4 is an example diagram illustrating an example in which a display device according to an embodiment of the present application receives a category selection of a user.

[0037] Figure 5 is a flowchart illustrating a simplification action process of a display device according to an embodiment of the present application to delete an object corresponding to a specific category. Figure 3

[0038] Figure 6 is an example diagram illustrating a process of generating an image of a real object corresponding to a specific category to be deleted according to the action process of the display device. Figure 5

[0039] Figure 7 is a flowchart illustrating a simplification action process of a display device according to an embodiment of the present application to replace an object corresponding to a specific category with a simplified object. Figure 3

[0040] Figure 8 is an example diagram illustrating a generation process of an image of a real object corresponding to a specific category to be replaced with a simplified object according to the action process of the display device. Figure 7

[0041] Figure 9 is an example diagram illustrating an example in which a display device according to an embodiment of the present application simplifies an object corresponding to a specific category according to a preset illuminance.

[0042] Figure 10a and Figure 10b is a flowchart illustrating S304 and S306 of a display device according to an embodiment of the present application to replace a real object corresponding to a specific category with AR information corresponding to the specific category and to display the same. Figure 3

[0043] Figure 11 is an example diagram illustrating an example in which a display device according to the action processes of the display device and the display device replaces a real object corresponding to a specific category with AR information according to the action processes of the display device and the display device. Figure 10a Figure 10b

[0044] Figure 12 is a flowchart illustrating an action process of a display device according to an embodiment of the present application to detect an object of POI information corresponding to a specific category according to user biometric information.

[0045] Figure 13a and Figure 13b ​​​​​​​is a flowchart showing an action process of a display device according to an embodiment of the present application detecting at least one object from a sensed image and displaying an AR object of a specific POI instead of the detected object.

[0046] Figure 14 is an example diagram showing an example of an image in which a part of a real object included in a sensed image is replaced with an AR object of a specific POI according to an action process of the Figure 13a and Figure 13b

[0047] Figure 15 is an example diagram showing an example in which an AR object corresponding to a specific object selected by a user is displayed on an image displaying a real object in a display device according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] It should be noted that technical terms used in the present specification are used only to describe specific embodiments and are not intended to limit the present application. In addition, unless otherwise specified, a singular expression includes a plural expression. The suffix "module" and "part" used in the following description for a constituent element is given or mixed only for convenience of writing, and does not have a meaning or function distinguished from each other by itself.

[0049] In the present specification, the terms "comprise" or "include" and the like should not be interpreted as necessarily including all the stated elements or steps, but can include some elements or steps, or can further include additional elements or steps.

[0050] In addition, in explaining the disclosed technology, if it is judged that a detailed description of the related art can confuse the gist of the present technology, a detailed description thereof is omitted.

[0051] In addition, it should be understood that the accompanying drawings are for convenience of understanding the embodiments described in the present specification, and the technical idea of the present application is not limited to the drawings, including all modifications, equivalents, and alternatives included in the technical idea and technical scope of the present application. In addition, in addition to each of the following embodiments, combinations of the embodiments are of course included as modifications, equivalents, or alternatives in the technical idea and technical scope of the present application.

[0052] Figure 1 is an example diagram showing a configuration of a display device 10 according to an embodiment of the present application.

[0053] Referring to Figure 1 , the display device 10 according to an embodiment of the present application can include a processor 100 and an interface 110, a communication unit 120, an object recognition unit 130, a renderer 140, a display unit 150, and a storage 160 connected to the processor 100. The display device 10 can further include a sensor 170.​Figure 1 The plurality of components shown are not necessarily essential to achieve the display device 10, and the display device 10 described in this specification can have more or fewer components than those listed above.

[0054] First, the interface 110, which can be connected to an interface portion (not shown, hereinafter, vehicle interface portion) of the vehicle 20, can receive various information provided from the vehicle 20 through the vehicle interface portion. Among them, the vehicle interface portion can function as a passage to various external devices connected to the vehicle 20 or to each component of the vehicle 20. For example, the vehicle interface portion can have various ports connected to the interface 110, and can be connected to the interface 110 through the ports. In addition, data can be exchanged with the interface 110.

[0055] The interface 110 can be connected to each component of the vehicle 20 through the vehicle interface portion. As an example, the interface 110 can be connected to the camera 21 of the vehicle 20 and receive an image sensed at the camera 21. Hereinafter, the image sensed at the camera 21 will be referred to as a "sensing image".

[0056] In addition, the interface 110 is connected to the user input portion 22 provided in the vehicle 20 and receives a user input applied through the user input portion 22. Among them, the user input portion 22 can include a voice input portion capable of receiving a user's voice input, a gesture sensing portion capable of sensing a user's gesture, a touch input portion formed to include a touch sensor, and a mechanical input portion, etc.

[0057] In addition, the interface 110 can receive a sensing value sensed in at least one sensor provided in the vehicle 20 through the vehicle interface portion. Among them, the at least one sensor can be a sensor for detecting a position of the vehicle 20 (for example, a position sensor such as a GPS) and at least one state sensor for detecting a state of the vehicle 20 (for example, a speed sensor, an inclination sensor, etc.). In addition, a path guide device (for example, a navigation) for detecting a travel path of the vehicle 20 and a biological information detection sensor for detecting biological information of an occupant such as a driver and a passenger who ride in a cabin of the vehicle 20 can be included.

[0058] In addition, the communication portion 120 can perform wireless communication between the display device 10 and a predetermined server. To this end, the communication portion 120 can include at least one of a transmission antenna, a reception antenna, and an RF (Radio Frequency) circuit and an RF element capable of implementing various communication protocols.

[0059] The communication unit 120 can have at least one of a communication module for near distance communication and a communication module for V2X communication. As an example, the communication module for near distance communication (near distance communication module) can be formed to support near distance communication using at least one of Bluetooth (Bluetooth TM ), RFID (Radio Frequency Identification), IrDA (Infrared Data Association), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus) technology, and can perform communication connection with at least one external device connected to the vehicle 20 or at least one constituent part provided in the vehicle 20 through the near distance communication module.

[0060] On the other hand, the communication module for V2X communication (V2X communication module) can perform wireless communication with a server (V2I: Vehicle to Infra), another vehicle (V2V: Vehicle to Vehicle), or a pedestrian (V2P: Vehicle to Pedestrian). To this end, the V2X communication module can include at least one RF circuit that can implement a V2I (Vehicle to Infra) communication, a V2V (Vehicle to Vehicle) communication, and a V2P (Vehicle to Pedestrian) communication protocol.

[0061] Among them, the communication unit 120 can not only be a communication unit provided in the display device 10, but also be a communication unit provided in the vehicle 20. In this case, the display device 10 can be connected to the communication unit 120 of the vehicle 20 through the interface 110, and the processor 100 can also control the communication unit 120 through the interface 110.

[0062] In addition, the object recognition unit 130 recognizes each object in an image sensed by the camera 21 (hereinafter, a sensed image) and classifies it according to a predetermined category according to the control of the processor 100. The object recognition unit 130 can recognize each object in the sensed image based on information for recognizing and classifying each category (hereinafter, classification information). Among them, a plurality of classification information can form a database according to a related category and be stored in the memory 160 of the display device 10 of the present embodiment.

[0063] On the other hand, under the control of the processor 100, the renderer can render a sensed image configured to include at least one simplified object on the display unit 150. Additionally, the display unit 150 can output the image rendered by the renderer 140. Under the control of the processor 100, the renderer 140 can render an image on the display unit 150 that removes at least one of the objects included in the sensed image, i.e., real-world objects (hereinafter, real objects), or render an image that includes a simplified graphic object or augmented reality object replacing at least one of the real objects. Therefore, the display unit 150 can output an image that removes a portion of the real objects included in the sensed image, or an image that includes simplified objects or augmented reality objects replacing the portion of the real objects with polygonal shapes or similar shapes.

[0064] The display unit 150 may be a display such as a touch screen installed on the vehicle 20. In this case, the display unit 150 can be connected via interface 110, and the processor 100 can control the display unit 150 via the interface 110 to output the image rendered by the renderer 140.

[0065] On the other hand, the memory 160 can store data that supports various functions of the display device 10. The memory 160 can store a plurality of application programs that can be executed using the processor 100, data and instructions for the operation of the display device 10.

[0066] As an example, the memory 160 may be configured as a database including the classification information (hereinafter, classification information database 161). The classification information database 161 may include classification information for classifying objects included in the sensed image according to their respective categories. In this case, the classification information may be feature information for classifying image objects into specific categories.

[0067] Furthermore, the classification information can be downloaded from a preset external server. In this case, the processor 100 can update the classification information database 161 based on the classification information corresponding to at least a portion of the categories downloaded from the external server.

[0068] The processor 100 controls the respective constituent elements connected thereto, and generally controls the overall operation of the display device 10. First, the processor 100 can receive an image sensed by the camera 21 of the vehicle 20 through the interface 110. Also, the received sensed image can be input to the object recognition part 130, and the object recognition part 130 can be controlled to recognize and classify respective objects included in the sensed image by categories defined in the category information database 161.

[0069] Also, if respective objects in the sensed image are recognized and classified by categories defined in the category information database 161 using the object recognition part 130, the processor 100 can determine whether a predetermined condition is satisfied or whether a specific category selected by a user exists. Also, if the determination result is that the condition is satisfied or the specific category selected by the user exists, it can be detected whether the categories corresponding to respective objects in the sensed image include the satisfied condition or the specific category selected by the user. Also, if the categories corresponding to respective objects in the sensed image include the satisfied condition or the specific category selected by the user, the object in the sensed image corresponding to the specific category or objects other than the object corresponding to the specific category can be simplified.

[0070] Here, the simplification can mean that at least one object is deleted from the sensed image. Or, the simplification can mean that at least one object included in the sensed image is replaced with a simplified object such as a polygon object or with an augmented reality object. Hereinafter, a more detailed operation procedure of the processor 100 which simplifies the object in the sensed image will be described in detail. Figure 5 to Figure 10b

[0071] If at least one object included in the sensed image is simplified, the processor 100 configures the simplified object and a real object which is not simplified to constitute a final output image. Also, the renderer 140 and the display part 150 can be controlled to render the constituted final output image on the display part 150. Accordingly, an image of the vehicle surroundings including a real object and an object which is deleted or replaced with a simplified object or an augmented reality object can be output through the display part 150.

[0072] On the other hand, in order to replace a real object in a sensed image with an augmented reality object as described above, the display device 10 of the embodiment of the present application can be linked with a cloud server 30 which provides an augmented reality service.

[0073] Figure 2 is a conceptual diagram showing a structure in which the display device 10 of the embodiment of the present application is linked with a cloud server to provide augmented reality information as described above.

[0074] Referring to​Figure 2 The display device (Onboard) 10 of this embodiment may have an AR engine for rendering and outputting AR images. Furthermore, the AR engine may include an adapter that receives images sensed by the camera 21, user gesture information or touch event information sensed by the user input unit 22 of the vehicle 20, and sensing information sensed by ADAS (Advanced Driver Assistance Systems), navigation, environmental sensors, biometric sensors, etc. The adapter may be a device corresponding to the aforementioned... Figure 1 The configuration of interface 110 in the constituent parts.

[0075] Among them, such as Figure 1 As described above, the image sensed by the camera 21 is received via interface 110, and under the control of the processor 100, object recognition can be performed in the object recognition unit 130. Additionally, the object recognition result of the object recognition unit 130 can be input to the processor 100. Alternatively, as... Figure 2 As shown, the image sensed by the camera 21 can be directly input to the object recognition unit 130 to achieve object recognition. The object recognition result of the object recognition unit 130 can be sent to the AR engine through the interface 110, i.e., the adapter. The object recognition result can be the result of classifying each object included in the sensed image according to any category defined in the classification information database 161, based on the classification information database 161.

[0076] Furthermore, the classification manager 210 can receive user input detected at the user input unit 22 of the vehicle 20 via an adapter. Additionally, it can select at least one category from the categories in the classification information database 161 to be simplified based on the received user input. Furthermore, based on the object recognition results received via the adapter, it can detect objects in the categories corresponding to the user-selected category, which are respectively associated with objects included in the sensed image. Furthermore, the detected objects can be simplified by deleting or reducing them to graphical objects such as polygons.

[0077] Alternatively, the classification manager 210 can perform the simplification using an augmented reality graphical object (hereinafter, augmented reality object) corresponding to the detected object. For this purpose, the AR engine can have an AR service client that provides a wireless connection interface with the cloud server 30 that provides augmented reality services. The AR service client can be configured to correspond to a communication unit 120 that supports a wireless communication connection with the cloud server 30.

[0078] The cloud server 30 can include an AR service part 31 that provides POI information including an augmented reality object in response to a request from the AR service client, and an external data provider 32 that provides POI information and geographic information to be provided by the AR service part 31.

[0079] In this case, the AR service part 31 can receive a POI information request for the surroundings of the vehicle 20 including the location information of the vehicle 20 from the AR service client. Thereby, the AR service part 31 can request the external data provider 32 for POI information in an area corresponding to the received location information. In addition, if POI information including an augmented reality object is received from the external data provider 32 in response to the request, the AR service part 31 can provide the received POI information to the AR service client. Thereby, an augmented reality object corresponding to an object in a sensing image selected by the user can be transmitted to the category manager 210 through the AR service client.

[0080] On the other hand, if it is determined by the category manager 210 to delete an object in the sensing image corresponding to the category selected by the user or to replace it with a simplified object such as a polygon or an augmented reality object (simplification), the image composition part 220 can form an image after deleting the object in the sensing image corresponding to the category selected by the user or replacing it with the simplified object or the augmented reality object. That is, the image composition part 220 determines a region in the sensing image where each object is to be disposed, and can initialize (delete) a region in the sensing image corresponding to the category selected by the user among the detected objects, that is, a region occupied by an object in the sensing image, or dispose a simplified object or an augmented reality object received from the cloud server 30 in the region occupied by the detected object. The category manager 210 and the image composition part 220 can be configurations corresponding to the processor 100 of the Figure 1

[0081] In addition, the AR renderer can receive image composition information of objects disposed in each region in the sensing image from the image composition part 220. In addition, an image according to the received image composition information can be rendered on the display part 150. Therefore, an image in which at least one object in the sensing image corresponding to the category selected by the user is deleted or replaced with a simplified object or an augmented reality object can be output through the display part 150. The AR renderer can be a configuration corresponding to the renderer 140 of the Figure 1

[0082] ​​On the other hand, the POI information provided from the cloud server 30 can be information classified by a specified category. In addition, the classification category of the POI information in the cloud server 30 can be the same category as the category of the classification information database 161 of the display device 10 of the embodiment of the present application. That is, the classification information database 161 can include classification information classified according to a category corresponding to the classification category of the POI information in the cloud server 30. Therefore, the display device 10 can request only a specific category in the POI information of the current location vicinity of the vehicle 20 from the cloud server 30, for example, POI information satisfying a predetermined condition or corresponding to a corresponding category selected by the user, and can receive only POI information corresponding to the specific category in response to the request.

[0083] On the other hand, in the above description, the configuration including the AR engine connected to the display device 10 and the cloud server 30 of the embodiment of the present application is described in detail. Hereinafter, the action process of the display device 10 described above to output an image of at least one object simplified according to a category selected by the user on the display unit 150 is described. In addition, for convenience of description, the following is described based on the configuration of the display device 10 described in the above Figure 1

[0084] On the other hand, Figure 3 is a flowchart showing an action process of the display device 10 of the embodiment of the present application to output an image including at least one simplified object according to a condition or a category selected by the user. In addition, Figure 4 is an example diagram showing an example of the display device of the embodiment of the present application to receive a category setting from the user.

[0085] First, referring to Figure 3 , the display device 10 of the embodiment of the present application can receive an image (hereinafter, a sensed image) sensed by a camera 21 of a vehicle 20 (S300). Herein, the camera 21 can be a front camera sensing a front image of the vehicle 20. Alternatively, the camera 21 can be a rear or side camera sensing a rear or side of the vehicle 20.

[0086] Thereby, the processor 100 of the display device 10 can control the object recognition unit 130 to recognize a category corresponding to each object in the sensed image (S302). Herein, the object recognition unit 130 can utilize the classification information database 161 including classification information corresponding to each of the predefined categories in order to recognize a category corresponding to each of the objects.

[0087] ​As an example, in the S302 step, the object recognition unit 130 can distinguish each object included in the sensed image. In addition, under the control of the processor 100, a feature of each object distinguished in the sensed image can be detected. In addition, according to the classification information database 161, classification information corresponding to the feature detected from each object can be detected. In addition, according to the category corresponding to the detected classification information, a category corresponding to each object can be recognized. Thus, by the S302 step, each object included in the sensed image can be distinguished and a category on the classification information database 161 corresponding to each object distinguished can be recognized.

[0088] Thus, the object recognition unit 130 can provide the processor 100 with information of the recognized category corresponding to each object in the sensed image as an object recognition result. In addition, the processor 100 can detect an object in the sensed image corresponding to a category based on whether a preset condition is satisfied or a category based on a user selection, according to the received object recognition result (S304).

[0089] As an example, the processor 100 can determine whether a preset condition corresponding to at least one category is satisfied in the S304 step. For example, the preset condition can be an illuminance or a time or a weather, etc. In addition, in a case where the illuminance or the time or the weather (e.g., a weather state such as rain, snow, etc.) detected by the sensor unit 23 of the vehicle 20 satisfies the preset condition, an object in the sensed image corresponding to at least one category corresponding to the condition can be detected.

[0090] Alternatively, the processor 100 can determine whether there is a category pre-selected by a user for simplification in the S304 step for simplification. To this end, the processor 100 can control the display unit 150 to display a menu screen from which a category can be selected, and determine at least one category based on a user input applied through the menu screen as a category selected by a user for simplification. Here, the category can be a category pre-defined in the classification information database 161 for classifying objects.

[0091] Figure 4 FIG. 10 is an example view illustrating an example in which the display apparatus 10 according to an embodiment of the present application displays at least one category selected by a user for simplification through the menu screen.

[0092] Referring to Figure 4 (a), the processor 100 of the display apparatus 10 can display a menu screen 410 from which at least one category option can be selected according to a user's selection on the display unit 150. Here, the category option can be an option corresponding to each category pre-defined in the classification information database 161.

[0093] In this state, as shown inFigure 4 As shown in (a), the user can input information to select option 411 corresponding to the "Building" category. Thus, the processor 100 can distinguish and display the specific category option selected based on the user's input from other unselected category options. Additionally, as described... Figure 4 As shown in (a), if the category selection ends after the user selects the “building” category, the processor 100 can detect all objects in the sensed image whose classification category is “building” in step S304.

[0094] On the other hand, the processor 100 of the display device 10 in this embodiment of the invention can also allow the user to select more specific categories. For example, such as... Figure 4 As shown in (a), when the user selects the "Building" category, the processor 100 can, based on the user's selection, such as Figure 4 As shown in (b), a menu screen 420 is further displayed, including the options corresponding to the subcategory of the currently selected "Building".

[0095] In this case, the subcategories can be categories distinguished based on the building's primary use, for example, based on the services that can be provided. Thus, as... Figure 4 As shown in (b), the subcategories can be categories that allow selection of specific services such as hospitals, restaurants, cafes, and banks.

[0096] Additionally, as described Figure 4 As shown in (b), if the user selects the "Café" category from the menu screen 420 including subcategory options and then ends the category selection, the processor 100 can detect all objects in the sensed image that are classified as "Buildings" and further classified as "Cafés" in step S304. That is, if the user further selects a subcategory corresponding to a specific category, the processor 100 can detect only the objects in the sensed image that correspond to the subcategory.

[0097] On the other hand, the display device 10 of this embodiment can also allow users to select more specific categories. For example, the processor 100 of the display device 10 can, as described above... Figure 4In the state where a specific category (“building”) and its subcategory (“coffee shop”) (hereinafter, the primary subcategory) have been selected as shown in (b), a menu screen 430 is further provided that allows the user to select a secondary subcategory based on a different classification criterion (hereinafter, the secondary classification criterion) than the primary subcategory. In this case, the secondary subcategory selection menu screen 430 may be a menu screen in which the category corresponding to the primary subcategory (e.g., coffee shop) can be selected from the categories in the classification information database 161 according to the secondary classification criterion.

[0098] As an example, the secondary subcategory can be a merchant of a specific shop or operator. In this case, the secondary subcategory selection menu 430 can be as follows: Figure 4 As shown in (c), it includes the menu screen of the merchant corresponding to the first-level subcategory (coffee shop).

[0099] In this case, as described Figure 4 As shown in (c), the user can select any category (“starfall”). Thus, in step S304, the processor 100 can detect only objects from the objects included in the sensed image that not only belong to the “Building” category, but also, among the objects classified as “Building,” belong to the “Café” category and have the business name “starfall”. That is, if the user further selects at least one subcategory corresponding to a specific category, the processor 100 can detect only objects in the sensed image that correspond entirely to at least one subcategory.

[0100] On the other hand, although Figure 4 The description uses first-level and second-level subcategories as examples of user-selectable subcategories, but this is merely an example of an embodiment of the invention, and the invention is not limited thereto. That is, the user can select subcategories of any number of levels. Furthermore, the more subcategories selected, the fewer objects in the sensed image detected by the processor 100 as objects corresponding to the user-selected categories.

[0101] Furthermore, in the above description, although the example given is that the user selects a category option defined by the classification information database 161 through the menu screen and detects at least one object in the sensed image through the selected menu option, in contrast, a specific object or a specific category can also be selected based on user input applied to the display unit 150.

[0102] For example, the user can touch an area of the touch screen on which a specific object is displayed in a state in which the sensed image is displayed, through the touch screen. Thereby, the processor 100 can detect any object displayed in the touched area as an object selected by the user. Alternatively, the processor 100 can detect a category on the category information database 161 to which any object displayed in the touched area corresponds, and determine the detected category as a category selected by the user for simplification. In this case, the processor 100 further detects at least one subcategory to which any object according to the touch input corresponds, and determines at least one category including the detected subcategory as a category selected by the user for simplification.

[0103] On the other hand, in the S304 step, if an object corresponding to at least one category based on a preset condition or at least one category selected by the user is detected from the objects included in the sensed image, the processor 100 can perform simplification for the detected object (S306).

[0104] The simplification can include deletion of at least one object detected in the S304 step. In this case, the processor 100 can detect an area in which an object detected in the S304 step is displayed from the sensed image, and initialize pixel information of the detected area. In addition, color information and pattern information are detected from an area around the area in which the pixel information is initialized, and a fill image using the detected color and pattern is generated. Referring to Figure 5 to Figure 6 The operation process of the S306 step of deleting an object detected in the S304 step is described in detail.

[0105] On the other hand, the simplification can be a process of replacing at least one object detected in the S304 step with a simplified object such as a polygonal object. To this end, the processor 100 can generate a simplified object for at least one object detected, and can detect an area in which an object detected in the S304 step is displayed in the sensed image. In addition, the generated simplified object can be disposed in the detected area in the sensed image. Referring to Figure 7 to Figure 8 The operation process of the S306 step of replacing an object detected in the S304 step with a simplified object is described in detail.

[0106] Further, the simplification can be a process of replacing at least one object detected in the S304 step with an augmented reality object provided from the cloud server 30. Hereinafter, referring to Figure 10a , Figure 10b , and Figure 11The action process of the S304 step of detecting objects in the sensing image corresponding to the category selected by the user according to the POI information received from the cloud server 30 and the S306 step of replacing at least one of the objects detected in the S206 step with an augmented reality object included in the POI information will be described in detail.

[0107] If at least one of the objects within the captured image is deleted or replaced with a simplified object or an augmented reality object through the S306 step, the processor 100 can determine the positions of the objects in the image that are deleted or replaced with the simplified object or the augmented reality object and constitute a final output image (S308). In addition, if the constitution of the final output image including the objects that are deleted or replaced with the simplified object or the augmented reality object is completed, the renderer 140 can be controlled to render the final output image whose constitution is completed on the display part 150 (S310). Thus, an image in which at least one of the objects of the sensing image output through the display part 150 is deleted or replaced with the simplified object or the augmented reality object can be output.

[0108] Figure 5 FIG. 1 is a diagram illustrating a display device 10 according to an embodiment of the present application that deletes objects corresponding to a specific category. Figure 3 FIG. 6 is a diagram illustrating a simplified action process of the S306 step of FIG. 1. In addition, Figure 6 FIG. 7 is an example diagram illustrating a process of generating an image in which real objects corresponding to a specific category are deleted according to the action process of FIG. 6. Figure 5

[0109] First, referring to FIG. 5, Figure 5 If objects corresponding to a specific category selected by the user or satisfying a preset condition are detected in the S304 step of FIG. 1, the processor 100 can detect at least one region in the sensing image corresponding to each of the detected objects (S500). In addition, pixel information, for example, color information, of the region detected in the S500 step can be initialized. Thus, an image of the region detected in the S500 step can be deleted. Figure 3 Referring to FIG. 6,

[0110] (a) of FIG. 7 illustrates an example of a sensing image 600,(b) of FIG. 7 illustrates an example in which the sensing image 600 is divided into a plurality of regions according to each object included in the sensing image 600. Figure 6 Figure 6 As an example, the processor 100 can divide each object included in the sensing image 600 into different categories from each other according to the classification information included in the classification information database 161. In this case, as illustrated in (a) of FIG. 7, Figure 6

[0111] As an example, the processor 100 can divide each object included in the sensing image 600 into different categories from each other according to the classification information included in the classification information database 161. In this case, as illustrated in (a) of FIG. 7, Figure 6 ​​In the case where the sensing image 600 includes a plurality of buildings 601 to 604, a vehicle 608, a cloud 605, a road 606, and a lane 607 as shown in (a), the processor 100 can distinguish each of the plurality of buildings 601 to 604, the vehicle 608, the cloud 605, the road 606, and the lane 607 as different objects from each other. In addition, each region within the sensing image 600 in which each of the distinguished objects is displayed can be detected.

[0112] In this state, the specific building 601 can be an object corresponding to a category satisfying a preset condition or a user selection. Thus, the processor 100 can initialize pixel information of a region in the sensing image 600 in which the object corresponding to the category satisfying the preset condition or the user selection, i.e., the specific building 601, is displayed. Accordingly, as shown in (b), the region in the sensing image 600 in which the specific building 601 is displayed can be deleted. Figure 6 As shown in (c), a region in the sensing image 600 corresponding to the specific building 601 can be deleted.

[0113] The pixel information can indicate color information set for each pixel. In addition, the initialization of the pixel information can be a change of the color information set for each pixel to color information of a preset initial value. Thus, if the color of the initial value is white, a region in the sensing image 600 corresponding to the specific building 601 can be displayed in white in the case where the region in the sensing image 600 corresponding to the specific building 601 is deleted. The color information of the initial value can be a color selected by a user.

[0114] If the specific object corresponding to the category satisfying the preset condition or the user selection is deleted, the processor 100 can detect a peripheral region of a region in the sensing image 600 in which the pixel information is initialized by the deletion. In addition, information (pattern information) of a color pattern formed by color information and a color of the peripheral region can be detected from the detected peripheral region (S504). In addition, a fill image corresponding to the region in the sensing image 600 in which the pixel information is initialized can be generated based on the detected color information and the pattern information (S506). In addition, the generated fill image can be disposed in the region in the sensing image 600 in which the pixel information is initialized (S508).

[0115] Thus, as shown in (d), a fill image 602 corresponding to the region in which the pixel information is initialized can be disposed in the sensing image 600 in which the region in which the pixel information is initialized is deleted. Figure 6As shown in (c), if the area corresponding to the specific building 601 in the sensing image 600 is deleted, the processor 100 can detect color information and color pattern information from the area surrounding the deleted specific building 601. In this case, as the area surrounding the specific building 601, the sky surrounding the specific building 601, including the adjacent surrounding buildings 602 and clouds 605, can be detected. Therefore, the processor 100 can generate a filled image with a color and pattern similar to the detected surrounding area, i.e., the sky surrounding the specific building 601, including the adjacent surrounding buildings 602 and clouds 605. Additionally, as... Figure 6 As shown in (d), the filling image can be configured in the area corresponding to the specific building 601 that was deleted.

[0116] Therefore, as described Figure 6 As shown in (d), a fill image including a cloud image 610 and a pattern 611 similar in appearance to adjacent surrounding buildings can be configured in a region in the sensing image 600 corresponding to the specific building 601 to be deleted. Furthermore, the sensing image 600 with the fill image configured can be rendered on the display unit 150 by the renderer 140. Therefore, the display unit 150 can output an image showing that at least one object in the sensing image 600, based on a category satisfying preset conditions or user selection, has been naturally eliminated.

[0117] on the other hand, Figure 7 This describes in detail how the display device 10 of an embodiment of the present invention performs the replacement of objects corresponding to a specific category with simplified objects. Figure 3 A simplified diagram of the action process for step S306. Additionally, Figure 8 It is shown according to the above Figure 7 This is an example diagram illustrating the process of generating images that replace real-world objects of a specific category with simplified images.

[0118] First, refer to Figure 7 If in the Figure 3 If step S304 detects objects in the sensed image that correspond to the user's selection or a specific category that meets preset conditions, then the processor 100 of the display device 10 can detect each object included in the sensed image. Additionally, simplified objects corresponding to the detected objects can be generated (S700).

[0119] The simplified object can be a polygonal graphic model (hereinafter, a polygonal model) capable of representing the shape of an object. That is, the simplified object can be a virtual object with a polygonal shape that does not reflect texture.

[0120] The S700 step of generating the polygon model can be a step of transmitting the position information of the vehicle 20 to a preset server to request POI information of a location surrounding the vehicle 20, and receiving a polygon model corresponding to each object in the sensing image from the preset server in response to the request.

[0121] The polygon model corresponding to each object in the sensing image can include POI information of a building or each region surrounding the vehicle 20. To this end, the preset server providing the POI information can be a server including a POI database including polygon models of a building or a surrounding region corresponding to the vehicle 20 surrounding the vehicle 20 corresponding to the position information of the vehicle 20. In this case, the preset server can be a cloud server 30, and can be a server including a DTaaS (Digital Twin as a Service) server. Alternatively, the cloud server 30 can be a server in communication connection with the DTaaS server.

[0122] On the other hand, if the polygon model is generated in the S700 step or the polygon model is provided from the cloud server 30, the processor 100 can detect a region in the sensing image occupied by an object corresponding to a user's selection or a specific category satisfying a preset condition among the objects in the sensing image in the S304 step (S702). In addition, pixel information, for example, color information, of the detected region can be initialized (S704). Thereby, a region in the sensing image corresponding to the object detected in the S304 step can be deleted. Figure 3

[0123] In addition, the processor 100 can display the simplified object generated or received in the S700 step on each region in the sensing image in which the object detected in the S304 step is deleted (S706). Thereby, the sensing image in which a part of the objects is replaced with the simplified object in the S308 step can be rendered on the display 150 by the renderer 140. Figure 3 Figure 3

[0124] Figure 8 is an example view illustrating an example of an image in which a real object corresponding to a specific category is replaced with a simplified object according to the action process of the S308 step. Figure 7

[0125] ​​​​For example, the processor 100 of the display device 10 can detect only objects corresponding to a specific category based on a user's selection or satisfying a preset condition among objects included in the sensed image as objects to be simplified, or, conversely, detect objects of other categories than the specific category based on a user's selection or satisfying a preset condition as objects to be simplified. Figure 8 An example in which objects of other categories than objects corresponding to a specific category based on a user's selection or satisfying a preset condition among objects within the sensed image 800 are all detected as objects to be simplified is shown.

[0126] First, referring to Figure 8 (a) of FIG. 8, Figure 8 (a) of FIG. 8 shows an example of a case in which the category based on a user's selection or satisfying a preset condition among objects within the sensed image 800 is vehicles 810 and roads 811. In this case, all objects corresponding to other categories than the vehicles 810 and the roads 811, such as a building 820, can be selected as objects to be simplified.

[0127] Thereby, the processor 100 can generate simplified objects respectively corresponding to the objects to be simplified. In this case, the simplified objects can be polygon models corresponding to the shapes of the objects to be simplified, respectively. To this end, the processor 100 can detect the shapes of the objects to be simplified from the sensed image 800 by the contour lines or the like of the objects to be simplified, and then generate polygon models corresponding to the detected shapes.

[0128] Alternatively, the processor 100 can transmit a POI information request including location information for a current location of the vehicle 20 to the cloud server 30. Thereby, the cloud server 30 can provide POI information for POIs in the vicinity of the location of the vehicle 20 in response to the POI information request. Herein, the processor 100 can request the cloud server 30 for POI information corresponding to the detected category of the objects detected as objects to be simplified from among the respective objects included in the sensed image 800. Thereby, the cloud server 30 detects only POI information of POIs corresponding to the requested category from among the POI information for POIs in the vicinity of the vehicle 20, and transmits the same as a response to the request.

[0129] On the other hand, the POI information provided from the cloud server 30 can include information of simplified objects, such as polygon models, of objects. Accordingly, the processor 100 can acquire simplified objects respectively corresponding to the objects detected as objects to be simplified.

[0130] Thus, the processor 100 can detect the region occupied by the detected object as the object to be simplified from the sensed image 800. In addition, the detected region can be deleted. In addition, a simplified object corresponding to the deleted object can be arranged in each region of the deleted object on the sensed image 800. In addition, the renderer 140 can be controlled to render the sensed image 800 in which a part of the object is replaced by the simplified object on the display 150.

[0131] Thus, as shown in (b) of FIG. 8, the object 810 corresponding to the category "vehicle" and the object 811 corresponding to the category "road" are displayed as they are in the sensed state, and, in contrast thereto, the object other than the above, for example, the building object 820, can be replaced by the simplified object. Figure 8

[0132] On the other hand, according to the above description, the display apparatus 10 of the embodiment of the present application can detect the selection of the user and the object corresponding to the category satisfying the preset condition from the respective objects of the sensed image. In addition, the preset condition can be the illuminance or the time or the weather.

[0133] In this case, the preset condition can be set so that the specific category is selected in the case where the preset condition is satisfied. That is, in the case where the specific condition is satisfied, the specific category matching the specific condition can be selected as the category for detecting the object to be simplified.

[0134] Figure 9 For the purpose of explaining the display apparatus 10 of the embodiment of the present application set to automatically select the specific category in accordance with the satisfaction of the preset condition, it is assumed that the condition is the preset illuminance and the category matching the condition is the "building".

[0135] First, referring to (a) of FIG. 9, the processor 100 can receive the image (sensed image 900) sensed by the camera 21 in the travel of the vehicle 20. In addition, it can be determined whether the preset condition matching the specific category is satisfied in accordance with the information detected by the sensor portion 23 of the vehicle 20. Figure 9 In this case, the preset condition can be the illuminance. Thus, the processor 100 can determine whether the illuminance of the surroundings of the vehicle 20 reaches the preset illuminance in accordance with the illuminance detected by the illuminance sensor of the vehicle 20. In addition, if the determination result is that the illuminance of the surroundings of the vehicle 20 reaches the preset illuminance, the processor 100 can determine that the preset condition matching the specific category is satisfied.

[0136]

[0137] ​​On the other hand, if the preset conditions are met, the processor 100 can select a specific category matching the conditions as the category for simplifying the objects within the sensed image 900. In this case, if the category matching the preset illumination conditions is "buildings," the processor 100 can... Figure 9 As shown in (b), the object 910 corresponding to the category "building" that matches the conditions is detected from the sensed image 900.

[0138] Therefore, the processor 100 can perform simplification for the detected object 910. For example, the processor 100 can delete the detected object 910 from the sensed image 900. Thus, as Figure 9 As shown in (c), the image of the object 910 corresponding to the building in the object of the sensing image 900 can be output by the display unit 150.

[0139] Alternatively, the processor 100 can generate or receive simplified objects corresponding to the detected objects 910 from the cloud server 30 via POI information, and replace the objects 910 corresponding to buildings in the sensed image 900 with the generated or received simplified objects. Therefore, as... Figure 9 As shown in (d), the image output by the display unit 150 of the sensed image 900 can be obtained after the object 910 corresponding to the building in the object is replaced with a simplified object such as a polygon model.

[0140] On the other hand, according to the above description, the display device 10 of the present invention can also use the augmented reality object received through POI information as a simplified object.

[0141] Figure 10a and Figure 10b This describes in detail how a display device according to an embodiment of the present invention replaces real-world objects corresponding to a specific category with AR information corresponding to that specific category and then displays the corresponding information. Figure 3 The flowcharts for steps S304 and S306 are provided. Additionally, Figure 11 It is shown according to the above Figure 10a and Figure 10b The example diagram illustrates the process of replacing real-world objects corresponding to specific categories with AR information.

[0142] First, in the above Figure 3 In step S302, the processor 100 of the display device 10 can determine the respective categories of objects included in the image sensed by the camera 21 based on the categories specified in the classification information database 161. Additionally, if the detection is performed to identify objects corresponding to categories selected by the user or meeting preset conditions... Figure 3 The S304 step is as follows: Figure 10aAs illustrated, information on a category selected by the user or satisfying a preset condition (hereinafter, selected category) can be transmitted to the cloud server 30 together with the position information of the vehicle 20 (S1000). That is, the processor 100 can request the cloud server 30 for information on POIs corresponding to the selected category in the vicinity of the current position of the vehicle 20 (POI information).

[0143] In addition, if the POI information in the vicinity of the vehicle 20 corresponding to the selected category is received in response to the request (S1002), the processor 100 can detect an object corresponding to the category selected by the user or satisfying a preset condition, based on the category determined for each of the objects included in the sensed image (S1004). On the other hand, since the received POI information is information received from the cloud server 30 according to the category selected by the user or satisfying a preset condition, the object detection of the S1004 step can be a process of detecting an object corresponding to the category of the received POI information from among the objects included in the sensed image.

[0144] On the other hand, if the selected category is detected through the Figure 10a Figure 3 If at least one of the objects included in the sensed image is detected in the S304 step of the Figure 3 , the processor 100 can move to the S306 step of the Figure 10b , and perform simplification with respect to the object detected in the S304 step. Hereinafter Figure 3 is a diagram illustrating in detail a simplification process performed in the S306 step of the

[0145] Referring to Figure 10b , the processor 100 of the display apparatus 10 of the embodiment can first extract an augmented reality object corresponding to the category selected by the user or satisfying a preset condition from the POI information received from the cloud server 30 (S1050). In addition, an area occupied by an object corresponding to a specific category selected by the user or satisfying a preset condition from among the objects detected from the sensed image through the action process of the Figure 10a may be detected (S1052). In addition, pixel information, for example, color information, of the detected area can be initialized (S1054). Thereby, the area in the sensed image detected in the S1052 step can be deleted.

[0146] In addition, the processor 100 can display the augmented reality object extracted in the S1050 step in each area in the sensed image deleted in the S1054 step (S1506). Thereby, the sensed image in which a part of the object is replaced with the augmented reality object in the S308 step of the Figure 3 may be rendered on the display 150 through the renderer 140.​

[0147] Figure 11 is an example illustrating a process according to the Figure 10a and Figure 10b described in (a) of FIG. 10, the process of replacing at least one object included in a sensing image with an example of an augmented reality object is performed by the S304 step and the S306 step of the Figure 3

[0148] Referring to Figure 11 , Figure 11 (a) of FIG. 11 shows an example of a sensing image 1100 including buildings 1110, 1111, 1112, a vehicle, and a road object. In this case, the processor 100 can identify and classify each object of the sensing image 1100 according to the classification information database, as an object 1110 corresponding to a building category, an object corresponding to a vehicle category, and an object corresponding to a road category.

[0149] On the other hand, the processor 100 can request the cloud server 30 for information of POIs in the vicinity of the current vehicle 20 including information for a category selected by the user or satisfying a preset condition and information for a location of the current vehicle 20. Thereby, the cloud server 30 detects information of a POI corresponding to a category included in the received request from POIs in the vicinity of the location of the vehicle 20, and as a response to the received request, can provide information including an augmented reality object corresponding to the detected POI, i.e., POI information. Accordingly, the processor 100 can acquire POI information including an augmented reality object of a POI in the vicinity of the current vehicle 20 corresponding to a category selected by the user or satisfying a preset condition.

[0150] In this state, a first building among buildings in the vicinity of the vehicle 20 can correspond to a category selected by the user or satisfying a preset condition. Thereby, the processor 100 can detect an object 1110 corresponding to the first building from the sensing image 1100 shown in (a) of FIG. 11. In addition, by initializing a region in the sensing image 1100 corresponding to the detected object 1110 of the first building, the object 1110 corresponding to the first building can be deleted from the sensing image 1100. Figure 11 Thereby, the processor 100 can extract an augmented reality object 1120 corresponding to the first building from the POI information received from the cloud server 30. In addition, the extracted augmented reality object can be displayed on a region corresponding to the deleted object 1110 in the sensing image 1100. Accordingly, as shown in (b) of FIG. 11, the augmented reality object 1120 corresponding to the first building can be displayed on a region corresponding to the deleted object 1110 in the sensing image 1100.

[0151] Figure 11 ​​As shown in (b), the display unit 150 can output an image 1101 in which the object 1110 corresponding to the specific category is replaced with the augmented reality object 1120.

[0152] On the other hand, the processor 100 can detect color or pattern information of a surrounding area of the region corresponding to the deleted object 1110 in the sensed image 1100, and display a fill image generated based on the detected color or pattern information. In addition, the augmented reality object can be displayed in superposition in the region in which the fill image is displayed. Thus, the region in which the object corresponding to the first building is deleted can be displayed more naturally.

[0153] On the other hand, in the above description, as the preset condition, only the external environment of the vehicle such as illumination or weather is described, but unlike this, the physical state of the user can also be set as the preset condition. For example, in the case where the physical state of the user is an emergency situation such as illness, the processor 100 can automatically select a category such as a hospital based on the detected physical state of the user. Or, in the case where the result of detecting the physical state of the user is that the user is severely sunstroke (the temperature inside the vehicle is above a predetermined temperature), or the result of the detection of the DMS (Driver Monitoring System) is that the user is in a dehydrated state, the category of a place where water can be supplied, i.e., a restaurant (cafe, etc.) where beverages, etc. can be taken, can be automatically set.

[0154] Figure 12 is a flowchart showing the action process of the display device 10 according to an embodiment of the present application, in which the display device 10 automatically selects a specific category based on the detected physical state of the user, and detects an object corresponding to the selected category.

[0155] In Figure 3 In step S302, the processor 100 of the display device 10 can determine the respective categories of the objects included in the image sensed by the camera 21 based on the categories specified by the category information database 161. In addition, as shown in Figure 12 As shown, the processor 100 of the display device 10 can receive sensed biological information of the user from at least one device capable of sensing the biological information of the user (S1200).

[0156] The at least one device can be a wearable device such as a smart watch worn by the user. Or, it can be a sensor of the DMS of the vehicle 20. As an example, the sensor of the DMS can be an internal camera that acquires an image for detecting the state of a driver or a passenger seated in the cabin of the vehicle 20. The "user" can be any one of the occupants of the vehicle 20.

[0157] Additionally, the processor 100 can determine whether the received user's biometric information meets preset conditions for detection (S1202). That is, the processor 100 can determine the user's biometric state based on the received user's biometric information and determine whether the determined user's biometric state meets preset conditions corresponding to a specific category. These preset conditions can be the biometric states of different users, and different biometric state conditions can be matched with different categories (e.g., dehydration or thirst – matched with a coffee shop or restaurant; illness – matched with a hospital).

[0158] If the determination result in step S1202 is that the received user's biometric status meets preset biometric status conditions, the processor 100 can determine a specific category matching the met biometric status conditions. Additionally, by transmitting the determined specific category information and the current vehicle 20's location information to the cloud server 30, a request can be made for POI information surrounding the vehicle 20 corresponding to the specific category (S1204). Furthermore, if POI information surrounding the vehicle corresponding to the specific category is received from the cloud server 30, it can be... Figure 3 In step S302, at least one object in the sensed image is identified based on the received POI information and detected as an object corresponding to a category that matches the biological state conditions (S1206).

[0159] On the other hand, such as Figure 12 As shown, if an object corresponding to the POI information received from the cloud server 30 is detected from the sensed image, the processor 100 can replace the detected object with an augmented reality object extracted from the received POI information. Furthermore, an image in which a portion of the objects have been replaced with augmented reality objects can be output to the display unit 150. That is, it can be done by... Figure 10b A similar process is described in the text, where an image of the augmented reality object extracted from the received POI information is output on the display unit 150.

[0160] On the other hand, the above description illustrates an example where at least one of the objects in the sensed image is deleted or replaced with a simplified object or an augmented reality object, depending on a specific category selected by the user or meeting preset conditions.

[0161] However, unlike this, users can also select a specific POI to replace a specific category. In this case, the processor 100 of the display device 10 can delete or replace at least a portion of the sensed image with an augmented reality object based on the selected POI.

[0162] Figure 13a and Figure 13bThis is a flowchart illustrating the process by which a display device according to an embodiment of the present invention detects at least one object from a sensed image based on the location of a vehicle and a specific point of interest (POI), and then displays an augmented reality object of the specific POI to replace the detected object.

[0163] First, in the above Figure 3 In step S302, the processor 100 of the display device 10 can determine the respective categories of objects included in the image sensed by the camera 21 based on the categories specified in the classification information database 161. Additionally, if the following steps are performed... Figure 3 In step S304, the processor 100 can then proceed as follows: Figure 13a As shown, the cloud server 30 is requested to include POI information including the location of a specific POI selected by the user, and the cloud server 30 is received in response to the request (S1300).

[0164] Therefore, the processor 100 can calculate the direction and distance corresponding to the position of the specific POI from the sensing image based on the current position of the vehicle 20 and the position information of the specific POI (S1302). In addition, at least one object located in the direction corresponding to the position of the specific POI can be detected from the sensing image.

[0165] As an example, if the FOV (Field of View) of the sensed image includes the direction in which the vehicle 20 points towards the specific POI, then at least one object displayed in the sensed image may be located between the specific POI and the vehicle. Therefore, the processor 100 can detect at least one object in the sensed image that is located between the specific POI and the vehicle and thus occludes the specific POI as an object selected by the user. Therefore, the processor 100 can move to... Figure 3 Step S306, which is for the process described in step S304, i.e. Figure 13a The execution of objects detected during the action process is simplified. Below, Figure 13b The details described in such cases are shown in detail. Figure 3 The simplified process performed in step S306.

[0166] Reference Figure 13b First, the processor 100 of the display device 10 in this embodiment of the invention can extract the augmented reality object corresponding to the specific POI from the POI information received from the cloud server 30 (S1350). Additionally, it can detect objects passing through the sensed image. Figure 13a The action process detects at least one object, namely the region corresponding to at least one object in the sensing image that is located between the specific POI and the vehicle and occludes the specific POI (S1352).

[0167] In addition, the processor 100 can delete at least one object detected through the action process of the POI information from the sensed image by initializing pixel information of a region of the sensed image corresponding to the detected region (S1354). In addition, a specific POI corresponding to the augmented reality object extracted from the POI information can be displayed in a region of the sensed image in which the at least one object is deleted (S1356). Thus, the specific POI can be displayed in the region in which the at least one object is naturally deleted. Figure 13a In addition, the processor 100 can delete at least one object detected through the action process of the POI information from the sensed image by initializing pixel information of a region of the sensed image corresponding to the detected region (S1354). In addition, a specific POI corresponding to the augmented reality object extracted from the POI information can be displayed in a region of the sensed image in which the at least one object is deleted (S1356). Thus, the specific POI can be displayed in the region in which the at least one object is naturally deleted. Figure 3 In the S308 step, the image in which at least one object detected through the action process of the POI information is replaced with an image of an augmented reality object corresponding to a specific POI selected by the user can be rendered on the display 150 by the renderer 140. Figure 13a In the S308 step, the image in which at least one object detected through the action process of the POI information is replaced with an image of an augmented reality object corresponding to a specific POI selected by the user can be rendered on the display 150 by the renderer 140.

[0168] In this case, the size of the augmented reality object can be determined according to the distance between the current vehicle 20 and the location of the specific POI. That is, if the distance between the current vehicle 20 and the specific POI is closer, a larger augmented reality object can be displayed, and if the distance is farther, a smaller augmented reality object can be displayed.

[0169] On the other hand, the processor 100 can detect color or pattern information of a surrounding region in a region of the sensed image corresponding to the deleted at least one object and display a fill image generated according to the detected color or pattern information. In addition, the augmented reality object can be displayed in a region in which the fill image is displayed. Thus, the augmented reality object corresponding to the specific POI can be displayed in a region in which the at least one object is naturally deleted.

[0170] In addition, the number of objects to be deleted in the sensed image can be increased or decreased according to the size of the augmented reality object. As an example, if the size of the augmented reality object corresponding to the specific POI is increased as the distance between the vehicle 20 and the specific POI is shortened, the number of objects in the sensed image that occlude the specific POI can be increased. Thus, more objects in the sensed image can be deleted.

[0171] In contrast, if the size of the augmented reality object corresponding to the specific POI is decreased as the distance between the vehicle 20 and the specific POI is increased, the number of objects in the sensed image that occlude the specific POI can be decreased. Thus, fewer objects in the sensed image can be deleted.

[0172] Figure 14 is an example view illustrating an example of generating an image in which a portion of an object included in a sensed image is replaced with an augmented reality object of a specific POI according to the action processes of the POI information of Figure 13a and Figure 13b is an example view illustrating an example of generating an image in which a portion of an object included in a sensed image is replaced with an augmented reality object of a specific POI according to the action processes of the POI information of

[0173] Referring to Figure 14 (a), the processor 100 can receive an image (sensed image 1400) sensed by the camera 21 in the traveling of the vehicle 20. Also, in this state, the processor 100 can detect, from the sensed image, an object located in a direction toward a specific POI selected by the user and a current position of the vehicle, i.e., an object located between the specific POI and the vehicle and blocking the specific POI in the sensed image.

[0174] On the other hand, if the specific POI is the Eiffel Tower (in this case, it can be a state in which an augmented reality object corresponding to the Eiffel Tower has been received from the cloud server 30) and the object blocking the specific POI in the sensed image is an object (first building object 1410) corresponding to a first building and a vehicle object 1411, the processor 100 can detect a region corresponding to the first building object 1410 and the vehicle object 1411 in the sensed image as shown in (b) of FIG. 14. Figure 14

[0175] Also, the first building object 1410 and the vehicle object 1411 in the sensed image can be deleted by initializing pixel information of the detected region. Also, as shown in (b) of FIG. 14, an augmented reality object 1420 corresponding to the currently selected specific POI, i.e., the Eiffel Tower, can be displayed in a region corresponding to the deleted first building object 1410 and the vehicle object 1411 in the sensed image 1400. In this case, the size of the augmented reality object 1420 can be determined according to the distance between the vehicle 20 and the Eiffel Tower. Figure 14

[0176] On the other hand, as shown in (b) of FIG. 14, the user can also pre-set a specific category for which deletion is limited. In this case, even if an object blocking a specific POI in a sensed image is an object of a category set by the user to be limited for deletion, the object can not be deleted. This is to prevent a danger such as an accident that can occur when objects actually existing are not displayed on the display portion 150 (e.g., a nearby vehicle object, etc.) in a case in which objects are arbitrarily deleted in a sensed image according to a POI selected by the user. Figure 14

[0177] As an example, if an object corresponding to a vehicle category is pre-set by the user to be limited for deletion, even if the first building object 1410 and the vehicle object 1411 all block the specific POI, the processor 100 can not delete the vehicle object 1411. In this case, as shown in (b) of FIG. 14, the augmented reality object 1420 corresponding to the Eiffel Tower can be displayed in a region corresponding to the first building object 1410 in the sensed image 1400. Figure 11 ​​​In this case, the processor 100 can delete only the first building object 1410, and display the augmented reality object 1320 corresponding to the specific POI in a region of the sensed image corresponding to the deleted first building object 1410.

[0178] On the other hand, unlike the above, a specific category of objects that can be deleted among objects that occlude a specific POI in a sensed image can be designated in advance. In this case, only the object corresponding to the specific category that is allowed to be deleted among objects that occlude the specific POI in the sensed image can be deleted. In addition, the augmented reality object corresponding to the specific POI can be displayed in the deleted region.

[0179] On the other hand, the processor 100 of the display apparatus 10 according to an embodiment of the present application can also track a change in relative position between a specific object that changes according to movement of the vehicle 20 and the vehicle 20 using the augmented reality object displayed in the sensed image.

[0180] Figure 15 FIG. 1 is an example diagram illustrating an example in which the processor 100 of the display apparatus 10 according to an embodiment of the present application displays a position of a specific object that moves in a sensed image using an augmented reality object corresponding to the specific object selected by a user.

[0181] First, referring to Figure 15 (a), the processor 100 of the display apparatus 10 can accept selection of at least one object among objects displayed in a sensed image. In this case, the selected object can be an object directly selected by a user among the displayed objects or an object detected according to a category or a POI selected by a user. Hereinafter, for convenience of explanation, it is assumed that an object 1510 corresponding to the sun in a sensed image is a selected object.

[0182] In this case, the processor 100 can detect whether the selected object in the sensed image is occluded by other objects in the sensed image during movement of the vehicle or passage of time. In addition, in a case in which at least a portion of the selected object is occluded by the other objects, the at least a portion that is occluded can be displayed as an augmented reality object corresponding to the selected object. In addition, the selected object that moves during movement of the vehicle or passage of time can be displayed as movement of the augmented reality object.

[0183] That is, as illustrated in Figure 15 (a), in a case in which the selected object 1510 moves while being occluded by at least a portion of a building object in the sensed image as the vehicle 20 moves or time passes, the processor 100 can display the selected object 1510 as an augmented reality object 1520 as illustrated in Figure 15(b) to (d) as shown by the selected object 1510. That is, the movement of the selected object can be displayed by superimposing an augmented reality object 1511 having a hatching circle shape (b) or an augmented reality object 1512 having a three-dimensional shape with a central depression or protrusion (c) or an augmented reality object 1513 having a circular shape (d) on other objects in the sensed image. Figure 15 Figure 15 Figure 15

[0184] In this case, the augmented reality object can be displayed on top of other objects in the sensed image. Thus, the state in which the movement of the selected object is displayed can be tracked in the sensed image.

[0185] On the other hand, although the above description assumes that a specific object among the objects in the sensed image is selected by the user and is described, a specific POI or a specific category corresponding to an object selected by the user can be tracked and displayed in the same manner. In this case, even if the specific object corresponding to the specific POI or the specific category is occluded by other objects in the sensed image, the state in which the specific object moves in the sensed image can be displayed on the display unit 150 by superimposing an augmented reality object corresponding to the specific object on top of the other objects occluding the specific object at a position corresponding to the specific object.

[0186] The aforementioned application can be implemented in a medium storing a program as a code readable by a computer. The computer-readable medium includes all kinds of storage devices storing data readable by a computer system. Examples of the computer-readable medium include an HDD (Hard Disk Drive), an SSD (Solid State Disk), an SDD (Silicon Disk Drive), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and further includes a medium implemented in a carrier wave form (e.g., Internet transmission). In addition, the computer can include the processor 100 of the display device 10. Therefore, the above detailed description should not be interpreted as limiting in all aspects but should be considered as exemplary. The scope of the present application should be determined by reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present application are included in the scope of the present application.​​​

Claims

1. A display device, characterized in that, include: The interface receives sensing images of the vehicle's surrounding environment from at least one sensor installed on the vehicle. The memory includes a classification information database, which includes classification information for classifying the various objects contained in the sensed image into preset, distinct categories. The object recognition unit classifies and identifies each object contained in the sensed image into any category in the classification information database based on the classification information. A renderer renders the image to be displayed on the display unit of the vehicle. as well as The processor detects objects corresponding to at least one category selected by the user from the objects identified in the category, controls the renderer to render an output image that replaces the detected objects with the corresponding graphic objects, and controls the interface unit to output the rendered output image on the display unit through the interface. The processor removes objects detected according to the selected at least one category from the sensed image and controls the renderer to render an output image from which the detected objects have been removed from the sensed image.

2. The display device according to claim 1, characterized in that, It also includes a communication unit that communicates with a cloud server, which provides POI information, including augmented reality objects corresponding to POIs around the vehicle. The processor requests POI information from the cloud server, including the vehicle's current location and information on at least one category selected by the user, for the POIs surrounding the vehicle. If, in response to the request, at least one POI information corresponding to at least one category of POIs around the vehicle is received from the cloud server, the processor extracts augmented reality objects from the received POI information and controls the renderer to render an output image that replaces the detected objects with the extracted augmented reality objects.

3. The display device according to claim 2, characterized in that, The cloud server includes POI information categorized according to the preset plurality of categories. The classification information database includes a plurality of preset categories for classifying POI information according to the cloud server, and classification information for classifying each object contained in the sensed image.

4. The display device according to claim 1, characterized in that, The graphic object is a polygonal object, which represents the shape of each of the detected objects.

5. The display device according to claim 1, characterized in that, The processor detects regions in the sensed image where the detected object is displayed and initializes the color information of the detected regions to remove the detected object from the sensed image. The color of the initialized area varies depending on the user's selection.

6. The display device according to claim 1, characterized in that, If the detected object is deleted from the sensed image, the processor detects at least one of the color of the surrounding area of ​​the deleted object and the color pattern of the surrounding area. The processor generates a filled image corresponding to a region in the sensed image where the detected object has been deleted, based on at least one of the color and color pattern. The processor controls the renderer to render an output image that displays the filled image in a region corresponding to the deleted object in the sensed image.

7. The display device according to claim 1, characterized in that, If preset conditions are met, the processor further detects objects from the sensed image that correspond to a specific category that matches the met conditions.

8. The display device according to claim 2, characterized in that, The processor detects at least one object in the sensed image located in the direction from the vehicle toward the specific POI, based on preset location information of a specific POI and the vehicle's location information. The processor removes the detected at least one object from the sensed image and controls the renderer to render an output image in the sensed image that displays the graphic object corresponding to the specific POI in a region corresponding to the removed at least one object.

9. The display device according to claim 8, characterized in that, The graphical object is the augmented reality object corresponding to the specific POI. The processor receives the location information of the specific POI and the augmented reality object corresponding to the specific POI from the cloud server.

10. The display device according to claim 8, characterized in that, The processor calculates the distance between the specific POI and the vehicle based on the location information of the specific POI and the vehicle, and determines the size of the graphic object based on the calculated distance.

11. The display device according to claim 8, characterized in that, The processor detects objects in the sensed image that are located in the direction pointing to the specific POI and that correspond to objects of a deletion-restricted category, and deletes only objects other than those corresponding to the deletion-restricted category.

12. A method for controlling a display device, the display device being connected via an interface to a camera, at least one sensor, and a display unit disposed in a vehicle, characterized in that, The control method includes: The step of receiving the sensed image sensed by the camera; The step of classifying and identifying the categories corresponding to each object contained in the sensed image based on classification information used for classification according to preset categories that are different from each other; The step of detecting objects corresponding to at least one specific category selected by the user from the objects identified in the categories; The step of replacing the detected objects with corresponding graphic objects to generate an output image; and The step of rendering the output image to output the generated output image on the display unit; The step of generating the output image is to generate an output image from the sensing image by initializing a region in the sensing image that corresponds to the detected object.

13. The control method for the display device according to claim 12, characterized in that, The steps for generating the output image include: The step of initializing a region in the sensed image that corresponds to the detected object; The step of detecting at least one of the color of the surrounding area of ​​the deleted object and the color pattern of the surrounding area; The steps of generating a fill image corresponding to a region in the sensed image where the object was detected to have been deleted, based on at least one of the colors and color patterns; and The step of generating an output image in the sensed image that displays the filled image within a region corresponding to the deleted object.

14. The control method for the display device according to claim 12, characterized in that, The steps for generating the output image include: The steps include transmitting information about the specific category and the location information of the vehicle to the cloud server, and requesting POI information of augmented reality objects corresponding to the POIs of the specific category around the vehicle's location; The step of receiving POI information from the cloud server in response to the request; The step of deleting objects corresponding to the specific category from the objects contained in the sensed image; and The step of generating an output image that displays augmented reality objects corresponding to the objects deleted from the sensing image on a region corresponding to the deleted objects in the sensing image.

15. The control method for the display device according to claim 12, characterized in that, The steps for detecting objects corresponding to the specific category include: The steps of detecting the location of a specific POI selected by the user, based on the location of the vehicle; and The step of detecting at least one object shown in the direction from the location of the vehicle to the location of the specific POI from the sensed image; The detected object corresponds to a graphical object that is an augmented reality object corresponding to the specific POI.

Images