Augmented reality-based vehicle display device and control method of the device

Abstract

The present invention is characterized by comprising: an interface unit that receives a sensing image of the surrounding environment of the vehicle from at least one sensor provided in the vehicle; a classification information database including classification information for classifying objects included in the sensing image into different categories; an object identification unit that classifies and identifies each object included in the sensing image into one of the categories in the classification information database based on the classification information; a renderer that renders an image to be displayed on a display unit provided in the vehicle; and a processor that controls the renderer to detect objects corresponding to a category selected by a user among the objects whose categories have been identified, and to render an output image in which the detected objects are replaced with graphic objects corresponding to the detected objects, and controls the interface unit so that the rendered output image is output on the display unit through the interface.

Description

Technology Field

[0001] The present invention relates to a display device that displays an image of the surrounding environment of a vehicle. Background Technology

[0002] Recently, Augmented Reality (AR) technology has emerged, which adds virtual objects to the real world by displaying graphic objects through the vehicle's windshield or Head-Up Display (HUD), or by overlaying graphic objects onto video captured by cameras. Current vehicles utilize this AR technology to provide drivers with additional information regarding the surrounding environment, vehicle status, and driving path, allowing them to intuitively perceive the vehicle and its driving environment. Consequently, driving efficiency and convenience can be further enhanced.

[0003] Active research is underway to provide user-customized information based on the real world by utilizing such augmented reality technology. As part of this research, AR information filtering technology has been developed. This technology receives a category of Point of Interest (POI) from the user in advance and displays only augmented reality information corresponding to POI data provided by a cloud server when a POI corresponding to the pre-set category is present around the vehicle, thereby ensuring that only the augmented reality information of interest to the user is displayed.

[0004] However, this AR information filtering technology merely restricts the display of augmented reality information related to each object in the real world surrounding the user, while the real-world objects themselves are displayed as they are. In other words, while user-customized information can be provided only for the displayed augmented reality data, there is a problem in that real-world objects the user is not interested in or does not want to see are displayed as they are.

[0005] Furthermore, there is a need to apply the aforementioned augmented reality technology to real-world objects, such as continuously providing information on specific POIs desired by the user regardless of real-world objects surrounding the user. The problem to be solved

[0006] The present invention aims to solve the aforementioned problems and other problems, and aims to provide a display device and a control method for the device that utilize augmented reality technology to prevent or simplify the display of objects that the user does not have interest in or does not want to see, as well as augmented reality information and objects in the real world.

[0007] Furthermore, the present invention aims to provide a display device and a control method for the device, which utilize augmented reality technology to display information regarding the location and direction of a specific POI that is necessary or desired by the user, regardless of real-world objects, or to display the location and direction of the specific POI that changes relatively according to the position of a moving vehicle, regardless of real-world objects. means of solving the problem

[0008] According to one aspect of the present invention for achieving the above or other purposes, a display device according to an embodiment of the present invention comprises: an interface unit that receives a sensing image of the surrounding environment of the vehicle from at least one sensor provided in the vehicle; a memory that includes a classification information database containing classification information for classifying each object included in the sensing image into different categories set in advance; an object identification unit that classifies and identifies each object included in the sensing image into one of the categories in the classification information database based on the classification information; a renderer that renders an image to be displayed on a display unit provided in the vehicle; and a processor that controls the renderer to detect objects corresponding to at least one category selected by a user among the objects whose categories are identified, and to render an output image in which the detected objects are replaced with graphic objects corresponding to the detected objects, and controls the interface unit so that the rendered output image is output on the display unit through the interface.

[0009] In one embodiment, the system further includes a communication unit that communicates with a cloud server providing POI information including an augmented reality object corresponding to a POI (Position Of Interest) around the vehicle, and the processor requests POI information about a POI around the vehicle, which includes the current location of the vehicle and information about at least one category selected by the user, from the cloud server, and when at least one POI information corresponding to at least one category among the POIs around the vehicle is received from the cloud server in response to the request, the processor controls the renderer to extract an augmented reality object from the received POI information and render an output image in which the detected objects are replaced with the extracted augmented reality object.

[0010] In one embodiment, the cloud server includes POI information classified according to the plurality of preset categories, and the classification information database includes classification information for classifying each object included in the sensing image according to the plurality of preset categories in which the cloud server classified the POI information.

[0011] In one embodiment, the graphic object is characterized as being a polygon object representing the shape of each of the detected objects.

[0012] In one embodiment, the processor deletes objects detected according to at least one selected category from the sensing image, and controls the renderer to render an output image in which the detected objects are deleted from the sensing image.

[0013] In one embodiment, the processor detects an area where the detected objects are displayed from the sensing image and initializes the color information of the detected area to delete the detected objects from the sensing image, wherein the color of the initialized area varies according to the user's selection.

[0014] In one embodiment, the processor detects at least one of the color of a surrounding area of ​​the deleted objects and the color pattern of the surrounding area when the detected objects are deleted from the sensing image, generates a fill image corresponding to a part of the sensing image in which the detected objects are deleted based on at least one of the color and the color pattern, and controls the renderer to render an output image in which the fill image is displayed within a part of the sensing image in which the deleted objects are deleted.

[0015] In one embodiment, the processor is characterized by further detecting objects corresponding to a specific category that matches the satisfied condition from the sensing image when a preset condition is satisfied.

[0016] In one embodiment, the preset condition is characterized as being a condition regarding at least one of the illuminance, weather, or time of the vehicle.

[0017] In one embodiment, the preset condition is characterized as being a condition for biometric information detected from at least one of the occupants of the vehicle detected by the sensor unit of the vehicle.

[0018] In one embodiment, the processor detects at least one object within the sensing image located in a direction toward the specific POI from the vehicle based on location information of a pre-set specific POI and location information of a vehicle, deletes the detected at least one object from the sensing image, and controls the renderer to render an output image in which a graphic object corresponding to the specific POI is displayed in an area within the sensing image corresponding to the deleted at least one object.

[0019] In one embodiment, the graphic object is an augmented reality object corresponding to the specific POI, and the processor receives location information of the specific POI and the augmented reality object corresponding to the specific POI from the cloud server.

[0020] In one embodiment, the processor calculates the distance between the specific POI and the vehicle based on the location information of the specific POI and the vehicle, and determines the size of the graphic object according to the calculated distance.

[0021] In one embodiment, the processor detects objects within the sensing image located in a direction toward the specific POI that correspond to a category where deletion is restricted, and deletes only the remaining objects excluding the objects corresponding to the category where deletion is restricted.

[0022] According to one aspect of the present invention for achieving the above or other purposes, a control method for a display device connected to a display unit and a camera and at least one sensor provided in a vehicle through an interface unit according to an embodiment of the present invention comprises: receiving a sensing image sensed through the camera; classifying and identifying categories corresponding to each of the objects included in the sensing image based on classification information for classifying into different categories set in advance; detecting objects corresponding to at least one specific category selected by a user among the objects whose categories have been identified; generating an output image by replacing the detected objects with graphic objects corresponding to each of the detected objects; and rendering the output image so that the generated output image is displayed on the display unit.

[0023] In one embodiment, the step of generating the output image is characterized by initializing a region within the sensing image corresponding to each of the detected objects to generate an output image in which the detected objects are deleted from the sensing image.

[0024] In one embodiment, the step of generating the output image comprises: a step of initializing a region within the sensing image corresponding to each of the detected objects; a step of detecting at least one of the color of the surrounding region of the deleted objects and the color pattern of the surrounding region; a step of generating a fill image corresponding to a region within the sensing image where the detected objects have been deleted, based on at least one of the color and the color pattern; and a step of generating an output image in which the fill image is displayed within a region of the sensing image corresponding to the deleted objects.

[0025] In one embodiment, the step of generating the output image comprises: transmitting information of the specific category and location information of the vehicle to a cloud server to request POI information including an augmented reality object corresponding to a POI (Position Of Interest) of the category corresponding to the specific category around the location of the vehicle; receiving the POI information from the cloud server in response to the request; deleting an object corresponding to the specific category from the sensing image among the objects included in the sensing image; and generating an output image in which an augmented reality object corresponding to each object deleted from the sensing image is displayed in a part of the sensing image corresponding to the deleted object.

[0026] In one embodiment, the step of detecting objects corresponding to the specific category comprises: detecting the location of a specific POI selected by a user based on the location of the vehicle; and detecting at least one object displayed in a direction oriented toward the location of the specific POI from the location of the vehicle from the sensing image, wherein the graphic object corresponding to the detected object is an augmented reality object corresponding to the specific POI. Effects of the invention

[0027] The effects of the display device and the control method of the device according to the present invention are described as follows.

[0028] According to at least one embodiment of the present invention, the present invention classifies and identifies each object included in an image sensed by a camera of a vehicle according to a preset category, and simplifies and displays on a display unit objects corresponding to at least one category according to preset conditions or user settings by deleting or replacing them with simplified objects. Accordingly, there is an effect of providing the user with an image of the vehicle's surroundings that does not include specific types of objects that the user is not interested in or does not wish to view.

[0029] In addition, according to at least one embodiment of the present invention, the present invention has the effect of providing information about a specific POI without obscuring it by real-world objects by deleting an object between a vehicle and the user's current state or a specific POI set by the user from an image sensed by a camera, and displaying information about the specific POI in an area on the image where the object has been deleted. Brief explanation of the drawing

[0030] FIG. 1 is an exemplary diagram illustrating the configuration of a display device according to an embodiment of the present invention. FIG. 2 is a conceptual diagram illustrating a structure in which a display device according to an embodiment of the present invention provides augmented reality information in conjunction with a cloud server. FIG. 3 is a flowchart illustrating the operation process of a display device according to an embodiment of the present invention outputting an image including at least one simplified object based on a condition or a category selected by a user. FIG. 4 is an exemplary diagram illustrating an example in which a display device according to an embodiment of the present invention receives a category selection from a user. FIG. 5 illustrates the flow of the simplification operation process of FIG. 3, in which a display device according to an embodiment of the present invention deletes objects corresponding to a specific category. FIG. 6 is an example diagram illustrating the process of generating an image in which a real object corresponding to a specific category is deleted according to the operation process of FIG. 5. FIG. 7 illustrates the flow of the simplification operation process of FIG. 3, in which a display device according to an embodiment of the present invention replaces objects corresponding to a specific category with simplified objects. FIG. 8 is an example diagram illustrating the process of generating an image in which a real object corresponding to a specific category is replaced with a simplified object according to the operation process of FIG. 7. FIG. 9 is an exemplary diagram illustrating an example in which a display device according to an embodiment of the present invention simplifies objects corresponding to a specific category based on a preset illuminance. FIGS. 10a and FIGS. 10b are flowcharts illustrating in more detail steps S304 and S306 of FIG. 3, in which a display device according to an embodiment of the present invention replaces a real object corresponding to a specific category with AR information corresponding to the specific category and displays it. FIG. 11 is an example diagram illustrating an example in which a real object corresponding to a specific category is replaced with AR information according to the operation process of FIG. 10a and FIG. 10b. FIG. 12 is a flowchart illustrating the operation process of a display device according to an embodiment of the present invention detecting an object corresponding to POI information corresponding to a specific category based on the user's biometric information. FIGS. 13a and FIGS. 13b are flowcharts illustrating the operation process of a display device according to an embodiment of the present invention, which detects at least one object from an image sensed according to the location of a vehicle and a specific POI, and displays an AR object of the specific POI instead of the detected object. FIG. 14 is an example diagram illustrating an example in which some of the real objects included in the sensed image are replaced with AR objects of a specific POI according to the operation process of FIG. 13a and FIG. 13b. FIG. 15 is an exemplary diagram illustrating an example in which an AR object corresponding to a specific object selected by a user is displayed on an image in which a real object is displayed, in a display device according to an embodiment of the present invention. Specific details for implementing the invention

[0031] It should be noted that technical terms used in this specification are used merely to describe specific embodiments and are not intended to limit the invention. Additionally, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. The suffixes "module" and "part" for components used in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles.

[0032] In this specification, terms such as "composed of" or "comprising" should not be interpreted as necessarily including all of the various components or steps described in the specification, and should be interpreted as potentially excluding some of the components or steps, or including additional components or steps.

[0033] In addition, when describing the technology disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the technology disclosed in this specification, such detailed description is omitted.

[0034] In addition, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification, and the technical concept disclosed in this specification is not limited by the attached drawings; it should be understood that they include all modifications, equivalents, and substitutions that fall within the concept and technical scope of the present invention. Furthermore, not only each of the embodiments described below, but also combinations of embodiments may fall within the concept and technical scope of the present invention as modifications, equivalents, and substitutions that fall within the concept and technical scope of the present invention.

[0035] FIG. 1 is an exemplary diagram illustrating the configuration of a display device (10) according to an embodiment of the present invention.

[0036] Referring to FIG. 1, a display device (10) according to an embodiment of the present invention may be configured to include a processor (100), an interface (110) connected to the processor (100), a communication unit (120), an object identification unit (130), a renderer (140), a display unit (150), and a memory (160). Since the components illustrated in FIG. 1 are not essential for implementing the display device (10), the display device (10) described in this specification may have more or fewer components than those listed above.

[0037] First, looking at the interface (110), the interface (110) can be connected to an interface unit (not shown, hereinafter referred to as the vehicle interface unit) of the vehicle (20), and can receive various information provided by the vehicle (20) through the vehicle interface unit. Here, the vehicle interface unit can serve as a passage to various types of external devices connected to the vehicle (20) or to each component of the vehicle (20). For example, the vehicle interface unit may be equipped with various ports connected to the interface (110), and can be connected to the interface (110) through the ports. It can also exchange data with the interface (110).

[0038] The above interface (110) can be connected to each component of the vehicle (20) through the vehicle interface section. For example, the above interface (110) can be connected to the camera (21) of the vehicle (20) to receive an image sensed by the camera (21). Hereinafter, the image sensed by the camera (21) will be referred to as a 'sensed image'.

[0039] In addition, the interface (110) is connected to a user input unit (22) provided in the vehicle (20) and can receive user input authorized through the user input unit (22). Here, the user input unit (22) may include a voice input unit capable of receiving voice input from a user, a gesture detection unit capable of detecting a user's gesture, a touch input unit formed by including a touch sensor, and a mechanical input unit.

[0040] Additionally, the interface (110) can receive a detection value detected by at least one sensor equipped in the vehicle (20) through the vehicle interface unit. Here, the at least one sensor may be a sensor for detecting the position of the vehicle (20) (e.g., a position sensor such as GPS), and at least one state sensor for detecting the state of the vehicle (20) (e.g., a speed sensor, a tilt sensor, etc.). It may also include a route guidance device (e.g., navigation) for detecting the driving path of the vehicle (20), and a biometric information detection sensor for detecting biometric information of a passenger, such as a driver and a passenger, who is in the cabin of the vehicle (20).

[0041] And the communication unit (120) can perform wireless communication between the display device (10) and a pre-configured server. To this end, the communication unit (120) may include at least one of a transmitting antenna, a receiving antenna, an RF (Radio Frequency) circuit capable of implementing various communication protocols, and an RF element.

[0042] The communication unit (120) may be equipped with at least one communication module for short-range communication and at least one communication module for V2X communication. For example, the communication module for short-range communication (short-range communication module) may be formed to support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies, and may perform a communication connection with at least one external device connected to the vehicle (20) or at least one component provided in the vehicle (20) through the short-range communication module.

[0043] Meanwhile, the communication module for the above V2X communication (V2X communication module) can perform wireless communication with a server (V2I: Vehicle to Infra), other vehicles (V2V: Vehicle to Vehicle), or pedestrians (V2P: Vehicle to Pedestrian). To this end, the above V2X communication module may include at least one RF circuit capable of implementing protocols for communication with infrastructure (V2I), communication between vehicles (V2V), and communication with pedestrians (V2P).

[0044] Here, the communication unit (120) may be a communication unit provided in a vehicle (20) rather than a display device (10). In this case, the display device (10) may be connected to the communication unit (120) of the vehicle (20) through an interface (110), and the processor (100) may control the communication unit (120) via the interface (110).

[0045] And the object identification unit (130) can identify each object within the image (hereinafter referred to as the sensing image) sensed through the camera (21) under the control of the processor (100) and classify it according to a pre-set category. The object identification unit (130) can identify each object within the sensing image based on information (hereinafter referred to as classification information) for identifying and classifying each object according to a pre-set category. Here, the classification information may be databased by related category and included in the memory (160) of the display device (10) according to an embodiment of the present invention.

[0046] Meanwhile, the renderer can render a sensing image configured to include at least one simplified object on the display unit (150) under the control of the processor (100). The display unit (150) can output the image rendered by the renderer (140). Under the control of the processor (100), the renderer (140) can render an image on the display unit (150) in which at least one of the objects included in the sensing image, i.e., real-world objects displayed as actual images (hereinafter referred to as real objects), is deleted, or render an image including a simplified graphic object or an augmented reality object that replaces at least one of the real objects. Accordingly, the display unit (150) may output an image in which some of the real objects included in the sensing image are deleted, or an image including a simplified object or an augmented reality object, such as a polygon shape, instead of some of the real objects.

[0047] Here, the display unit (150) may be a display such as a touch screen provided in the vehicle (20). In this case, the display unit (150) may be connected through an interface (110), and the processor (100) may control the display unit (150) through the interface unit (110) to output an image rendered by the renderer (140).

[0048] Meanwhile, the memory (160) can store data that supports various functions of the display device (10). The memory (160) can store a number of application programs (or applications) that can be executed by the processor (100), data for the operation of the display device (10), and instructions.

[0049] For example, the memory (160) may be formed to include a database of classification information (hereinafter, a classification information database (161)). The classification information database (161) may include classification information for classifying each object included in the sensing image into a respective category. In this case, the classification information may be feature information for classifying an image object into a specific category.

[0050] Additionally, the classification information may be downloaded from an external server configured to provide the classification information. In this case, the processor (100) may update the classification information database (161) according to the classification information corresponding to at least some of the categories downloaded from the external server.

[0051] The processor (100) controls each connected component and typically controls the overall operation of the display device (10). First, the processor (100) can receive an image sensed from a camera (21) of a vehicle (20) through an interface (110). Then, the received sensed image can be input into an object identification unit (130), and the object identification unit (130) can be controlled so that the object identification unit (130) can identify and classify each object included in the sensed image according to a category defined in a classification information database (161).

[0052] And when each object in the sensing image is identified and classified according to a category defined in the classification information database (161) by the object identification unit (130), the processor (100) can determine whether a pre-set condition is satisfied or whether there is a specific category selected by the user. And if the determination result indicates that the condition is satisfied or there is a specific category selected by the user, the processor can detect whether the categories corresponding to each object in the sensing image include the satisfied condition or the specific category selected by the user. And if the categories corresponding to each object in the sensing image include the satisfied condition or the specific category selected by the user, the objects in the sensing image corresponding to the specific category or the remaining objects excluding the objects corresponding to the specific category can be simplified.

[0053] Here, the above simplification may mean deleting at least one object from the sensing image. Alternatively, the above simplification may mean replacing at least one object included in the sensing image with a simplified object, such as a polygon object, or with an augmented reality object. Below, a more detailed operation process of the processor (100) that simplifies objects within the sensing image will be examined with reference to FIGS. 5 to 10b.

[0054] When at least one object included in the above-mentioned sensing image is simplified, the processor (100) can arrange the simplified object and the real objects that have not been simplified to form a final output image. Then, by controlling the renderer (140) and the display unit (150), the configured final output image can be rendered on the display unit (150). Thus, an image of the vehicle surroundings, including real objects and objects that have been deleted or simplified or replaced with augmented reality objects, can be output through the display unit (150).

[0055] Meanwhile, in order to replace a real object within a sensing image with an augmented reality object, the display device (10) according to an embodiment of the present invention can be linked with a cloud server (30) that provides an augmented reality service.

[0056] FIG. 2 is a conceptual diagram illustrating a structure in which a display device (10) according to an embodiment of the present invention provides augmented reality information in conjunction with a cloud server.

[0057] Referring to FIG. 2, a display device (Onboard) (10) according to an embodiment of the present invention may be equipped with an AR engine that renders and outputs an AR image. The AR engine may include an adapter that receives an image sensed by a camera (21), user gesture information or touch event information detected through a user input unit (22) of a vehicle (20), and sensing information detected by ADAS (Advanced Driver Assistance Systems), navigation, environmental sensors, biometric information detection sensors, etc. Here, the adapter may be a component corresponding to the interface (110) among the components of FIG. 1.

[0058] Here, the image sensed by the camera (21) is received through the interface (110) as described in FIG. 1, and object identification can be performed in the object identification unit (130) under the control of the processor (100). The object identification result of the object identification unit (130) can be input to the processor (100). Alternatively, as shown in FIG. 2, the image sensed by the camera (21) is directly input to the object identification unit (130) to perform object identification, and the object identification result of the object identification unit (130) can be received by the AR engine through the interface (110), that is, the adapter. Here, the object identification result may be the result of classifying each object included in the sensed image into one of the categories defined through the classification information database (161) based on the classification information of the classification information database (161).

[0059] The classification manager (210) can receive user input detected from the user input section (22) of the vehicle (20) through an adapter. Based on the received user input, at least one category to be simplified can be selected from the categories of the classification information database (161). Based on the object identification result received through the adapter, an object corresponding to the category selected by the user can be detected among the categories corresponding to each object included in the sensing image. The detected object can be deleted or simplified into a graphic object such as a polygon.

[0060] Alternatively, the classification manager (210) may perform the simplification using an augmented reality graphic object (hereinafter referred to as an augmented reality object) corresponding to the detected object. To this end, the AR engine may be equipped with an AR service client that provides a wireless connection interface with a cloud server (30) that provides an augmented reality service. Here, the AR service client may be configured to correspond to a communication unit (120) that supports a wireless communication connection with the cloud server (30).

[0061] Here, the cloud server (30) may include an AR service unit (31) that provides POI information including augmented reality objects in response to a request from the AR service client, and an External Data Provider (32) that includes POI information and geographic information to be provided through the AR service unit (31).

[0062] In this case, the AR service unit (31) may receive a request for POI information around the vehicle (20) from the AR service client, which includes location information of the vehicle (20). Then, the AR service unit (31) may request POI information within a specific area corresponding to the received location information from the data provider (32). And when POI information including an augmented reality object is received from the data provider (32) in response to the request, the AR service unit (31) may provide the received POI information to the AR service client. Then, through the AR service client, an augmented reality object corresponding to an object on a sensing image corresponding to a category selected by the user may be transmitted to the classification manager (210).

[0063] Meanwhile, when the classification manager (210) decides to delete an object on the sensing image corresponding to the category selected by the user, or to replace it with a simplified object such as a polygon, or an augmented reality object (simplification), the image composition unit (Scene manager) (220) can form an image in which the object on the sensing image corresponding to the category selected by the user is deleted, or replaced with the simplified object or the augmented reality object. That is, the image composition unit (220) determines the area on the sensing image where each object is to be placed, and can initialize (delete) the area occupied by the detected object, that is, the object on the sensing image corresponding to the category selected by the user among the determined areas, or place a simplified object or an augmented reality object received from the cloud server (30) in the area occupied by the detected object. Here, the classification manager (210) and the image composition unit (220) may be configurations corresponding to the processor (100) of FIG. 1.

[0064] And the AR renderer can receive image composition information of objects placed in each area on the sensing image from the image composition unit (220). And it can render an image according to the received image composition information on the display unit (150). Accordingly, an image in which at least one object on the sensing image corresponding to the category selected by the user is deleted, or replaced with a simplified object or an augmented reality object, can be output through the display unit (150). Here, the AR renderer may be a configuration corresponding to the renderer (140) of FIG. 1.

[0065] Meanwhile, the POI information provided by the cloud server (30) may be information classified according to a designated category. The category in which the POI information is classified in the cloud server (30) may be the same category as the category of the classification information database (161) of the display device (10) according to an embodiment of the present invention. That is, the classification information database (161) may include classification information classified according to a category corresponding to the category in which the POI information is classified in the cloud server (30). Accordingly, the display device (10) may request only the POI information corresponding to a specific category, for example, a category corresponding to the satisfaction of a preset condition or a user's selection, from the cloud server (30) among the POI information around the current location of the vehicle (20), and may receive only the POI information corresponding to the specific category in response to the request.

[0066] Meanwhile, the above description has described in detail a configuration including an AR engine connected to a display device (10) and a cloud server (30) according to an embodiment of the present invention. In the following description, the operation process of the display device (10) described above outputting a simplified image of at least one object corresponding to a category selected by the user onto a display unit (150) will be described. In addition, for convenience of explanation, the following description will be based on the configuration of the display device (10) described in FIG. 1 above.

[0067] Meanwhile, FIG. 3 is a flowchart illustrating the operation process of a display device (10) according to an embodiment of the present invention outputting an image including at least one simplified object based on a condition or a category selected by a user. FIG. 4 is an example diagram illustrating an example of a display device according to an embodiment of the present invention receiving a category from a user.

[0068] First, referring to FIG. 3, a display device (10) according to an embodiment of the present invention can receive an image (hereinafter referred to as a sensed image) sensed by a camera (21) of a vehicle (20) (S300). Here, the camera (21) may be a front camera that senses a front image of the vehicle (20). Alternatively, the camera (21) may be a rear or side camera that senses the rear or side of the vehicle (20).

[0069] Then, the processor (100) of the display device (10) can control the object identification unit (130) to identify categories corresponding to each object in the sensing image (S302). Here, the object identification unit (130) may use a classification information database (161) containing classification information corresponding to each of the previously defined categories to identify the categories corresponding to each object.

[0070] For example, in step S302 above, the object identification unit (130) can distinguish each object included in the sensing image. And, under the control of the processor (100), it can detect features for each distinguished object in the sensing image. And, based on the classification information database (161), it can detect classification information corresponding to the features detected from each object. And, based on the categories corresponding to the detected classification information, it can identify categories corresponding to each object. Therefore, through step S302 above, each object included in the sensing image can be distinguished, and categories on the classification information database (161) corresponding to each distinguished object can be identified.

[0071] Then, the object identification unit (130) can provide information of identified categories corresponding to each object in the sensing image as an object identification result to the processor (100). And based on the received object identification result, the processor (100) can detect an object corresponding to a category based on whether a preset condition is satisfied or a category based on the user's selection among the objects in the sensing image (S304).

[0072] For example, the processor (100) can determine whether a preset condition corresponding to at least one category is satisfied in step S304. For example, the preset condition may be illuminance, time, or weather. And if the illuminance, time, or weather (e.g., weather conditions such as rain, snow, etc.) detected by the sensor unit (23) of the vehicle (20) satisfies the preset condition, an object corresponding to at least one category corresponding to the condition can be detected among the objects in the sensing image.

[0073] Alternatively, the processor (100) may determine whether there is a category pre-selected by the user for simplification in step S304. To do this, the processor (100) may control the display unit (150) to display a menu screen for selecting a category, and may determine at least one category based on user input through the menu screen as a category selected by the user for simplification. Here, the category may be a category pre-defined for classifying objects in the classification information database (161).

[0074] FIG. 4 is an exemplary diagram illustrating an example in which a display device (10) according to an embodiment of the present invention receives at least one category for simplification from a user through the menu screen.

[0075] Referring to FIG. 4(a), the processor (100) of the display device (10) can display a menu screen (410) on the display unit (150) that allows the selection of at least one category item according to the user's selection. Here, the category items may be items corresponding to each category predefined in the classification information database (161).

[0076] In this state, as shown in FIG. 4(a), the user may provide input to select an item (411) corresponding to the 'building' category. Then, the processor (100) may display the specific category item selected according to the user's input, distinguishing it from other unselected category items. And as shown in FIG. 4(a), if the user selects the 'building' category and then terminates the category selection, the processor (100) may detect all objects among the objects included in the sensing image whose classified category is 'building' in step S304.

[0077] Meanwhile, the processor (100) of the display device (10) according to an embodiment of the present invention may, of course, allow the user to select a more subdivided category. For example, as shown in FIG. 4 (a), if the user selects the 'building' category, the processor (100) may further display a menu screen (420) containing items corresponding to the subcategories of the currently selected 'building' according to the user's selection, as shown in FIG. 4 (b).

[0078] In this case, the above subcategories may be categories distinguished according to the main use of the building, for example, the services that can be provided. Accordingly, the above subcategories may be categories where specific services can be selected, such as hospitals, restaurants, coffee shops, and banks, as shown in FIG. 4 (b).

[0079] And as shown in (b) of FIG. 4 above, if the user selects the 'Coffee Shop' category from the menu screen (420) containing sub-category items and then terminates the category selection, the processor (100) can detect, in step S304, not only the objects included in the sensing image whose classified category is the 'Building' category, but also all objects classified as the 'Coffee Shop' category among the objects classified as the 'Building' category. That is, if the user selects a sub-category corresponding to a specific category, the processor (100) can detect only the objects corresponding to the sub-category among the objects included in the sensing image.

[0080] Meanwhile, the display device (10) according to an embodiment of the present invention may, of course, allow the user to select a more detailed category. For example, the processor (100) of the display device (10) may further provide a menu screen (430) that allows the user to select a secondary subcategory based on a classification criterion different from the classification criterion of the subcategory (hereinafter secondary classification criterion), as shown in (b) of FIG. 4, when a subcategory ('coffee shop') (hereinafter primary subcategory) of a specific category ('building') is selected, according to the user's selection. In this case, the secondary subcategory selection menu screen (430) may be a menu screen that allows the user to select a category corresponding to the primary subcategory (e.g., coffee shop) among the categories on the classification information database (161) based on the secondary classification criterion.

[0081] For example, the above secondary subcategory may be the trade name of a specific store or business operator. In this case, the secondary subcategory selection menu (430) may be a menu screen containing trade names corresponding to the above primary subcategory (coffee shop), as shown in FIG. 4 (c).

[0082] In this case, as shown in (c) of FIG. 4 above, the user can select one of the categories ('Starfall'). Then, in step S304, the processor (100) can detect only the objects included in the sensing image that are classified into the 'Building' category, and among the objects classified into the 'Building' category, the 'Coffee Shop' category and the business name 'Starfall'. That is, if the user selects at least one additional subcategory corresponding to a specific category, the processor (100) can detect only the objects included in the sensing image that correspond to at least one subcategory.

[0083] Meanwhile, in FIG. 4, first and second subcategories are described as examples of subcategories selectable by the user, but this is merely an example according to an embodiment of the present invention, and it goes without saying that the present invention is not limited thereto. That is, any number of more subcategories can be selected by the user. And as more subcategories are selected, the number of objects on the sensing image that are detected by the processor (100) on the sensing image as objects corresponding to the categories selected by the user may decrease.

[0084] In addition, the above description describes, for example, that a user selects category items defined through the classification information database (161) via a menu screen and detects at least one object on a sensing image via the selected menu item, but it is also known that a specific object or a specific category may be selected based on user input authorized to the display unit (150).

[0085] For example, while a sensing image is displayed through the touch screen, the user may touch a specific area on the touch screen where a specific object is displayed. Then, the processor (100) may detect any one object displayed in the touched area as an object selected by the user. Alternatively, the processor (100) may detect a category in the classification information database (161) corresponding to any one object displayed in the touched area and determine the detected category as the category selected by the user for simplification. In this case, the processor (100) may further detect at least one subcategory corresponding to any one object based on the touch input and determine at least one category including the detected subcategory as the category selected by the user for simplification.

[0086] Meanwhile, in step S304 above, if an object corresponding to at least one category based on the satisfaction of a preset condition or at least one category based on the user's selection is detected among the objects included in the sensing image, the processor (100) can perform simplification on the detected objects (S306).

[0087] Here, the above simplification may include the deletion of at least one object detected in step S304. In this case, the processor (100) may detect an area where the objects detected in step S304 are displayed from the sensing image and initialize the pixel information of the detected area. Then, it may detect color information and pattern information from an area surrounding the area where the pixel information was initialized and generate a fill image according to the detected color and pattern. The operation process of step S306, in which the object detected in step S304 is deleted in this manner, will be examined in more detail with reference to FIGS. 5 and 6 below.

[0088] Meanwhile, the above simplification may be a process of replacing at least one object detected in step S304 with a simplified object such as a polygon. To this end, the processor (100) may generate a simplified object for the at least one object detected and may detect a region on a sensing image where the objects detected in step S304 are displayed. Then, the generated simplified object may be placed on the region on the detected sensing image. The operation process of step S306, in which the object detected in step S304 is replaced with a simplified object, will be examined in more detail with reference to FIGS. 7 and 8 below.

[0089] In addition, the above simplification may be a process of replacing at least one object detected in step S304 with an augmented reality object provided from the cloud server (30). Below, the operation process of step S304, which detects an object within a sensing image corresponding to a category selected by the user based on POI information received from the cloud server (30), and step S306, in which at least one object detected in step S206 is replaced with an augmented reality object included in the POI information, will be examined in more detail with reference to FIGS. 10a, 10b, and 11.

[0090] When at least one of the objects in the captured image is deleted or replaced with a simplified object or an augmented reality object through the above step S306, the processor (100) can determine the location of the deleted or simplified object or the object replaced with an augmented reality object within the image to construct the final output image (S308). And when the construction of the final output image including the deleted or simplified object or the object replaced with an augmented reality object is completed, the renderer (140) can be controlled to render the final output image with completed construction onto the display unit (150) (S310). Thus, an image in which at least one of the objects in the sensing image is deleted or replaced with a simplified object or an augmented reality object can be output through the display unit (150).

[0091] FIG. 5 illustrates in more detail the simplified operation process of step S306 of FIG. 3, in which a display device (10) according to an embodiment of the present invention deletes objects corresponding to a specific category. FIG. 6 is an example diagram illustrating the process of generating an image in which real objects corresponding to a specific category are deleted according to the operation process of FIG. 5.

[0092] First, referring to FIG. 5, if an object corresponding to a specific category is detected among the objects in the sensing image in step S304 of FIG. 3 based on the user's selection or the satisfaction of a preset condition, the processor (100) can detect at least one area on the sensing image corresponding to each of the detected objects (S500). Then, pixel information, such as color information, of the area detected in step S500 can be initialized. Then, the image of the area detected in step S500 can be deleted.

[0093] Referring to FIG. 6, FIG. 6 (a) shows an example of a sensing image (600), and FIG. 6 (b) shows an example in which the sensing image (600) is divided into multiple regions according to each object included in the sensing image (600).

[0094] For example, the processor (100) can classify each object included in the sensed image (600) into different categories based on classification information included in the classification information database (161). In this case, as shown in FIG. 6 (a), if the sensed image (600) includes a plurality of buildings (601 to 604), a vehicle (608), a cloud (605), a road (606), and a lane (607), the processor (100) can classify each of the plurality of buildings (601 to 604), the vehicle (608), the cloud (605), the road (606), and the lane (607) into different objects. And each region within the sensed image (600) where each classified object is displayed can be detected.

[0095] In this state, the specific building (601) may be an object corresponding to the satisfaction of a preset condition or a category selected by the user. Then, the processor (100) can initialize pixel information for a region on the sensing image (600) where the object corresponding to the satisfaction of a preset condition or a category selected by the user, i.e., the specific building (601), is displayed. Thus, as shown in FIG. 6 (c), the region corresponding to the specific building (601) within the sensing image (600) can be deleted.

[0096] Here, the pixel information may refer to color information set for each pixel. The initialization of the pixel information may involve changing the color information set for each pixel to color information based on a preset initial value. Therefore, if the color based on the initial value is white, when the area corresponding to the specific building (601) within the sensing image (600) is deleted, the area corresponding to the specific building (601) within the sensing image (600) may be displayed as white. Here, the color information based on the initial value may be a color selected by the user.

[0097] When a specific object corresponding to a category selected by a user is deleted or when a pre-set condition is satisfied, the processor (100) can detect a surrounding area of ​​a part of the sensing image (600) in which pixel information has been initialized due to the deletion. Then, color information and information about a color pattern formed by the colors of the surrounding area (pattern information) can be detected from the detected surrounding area (S504). Then, based on the detected color information and pattern information, a fill image corresponding to a part of the sensor image (600) in which pixel information has been initialized can be generated (S506). Then, the generated fill image can be placed on a part of the sensor image (600) in which pixel information has been initialized (S508).

[0098] Accordingly, as shown in FIG. 6(c), when the area corresponding to the specific building (601) within the sensing image (600) is deleted, the processor (100) can detect color information and color pattern information from the area surrounding the deleted specific building (601). In this case, the sky surrounding the specific building (601), including the adjacent surrounding building (602) and clouds (605), can be detected as the area surrounding the specific building (601). Then, the processor (100) can generate a fill image having a color and pattern similar to the detected surrounding area, that is, the sky surrounding the specific building (601), including the adjacent surrounding building (602) and clouds (605). And as shown in FIG. 6(d), the fill image can be placed in the area corresponding to the deleted specific building (601).

[0099] Accordingly, as shown in (d) of FIG. 6 above, a fill image including a cloud image (610) and a pattern (611) similar to the shape of an adjacent surrounding building can be placed in a part of the sensing image (600) corresponding to the deleted specific building (601). Then, the sensing image (600) with the fill image placed thereon can be rendered on the display unit (150) through the renderer (140). Thus, an image in which at least one object corresponding to a category selected by the user or satisfying a preset condition among the objects in the sensing image (600) has naturally disappeared can be output through the display unit (150).

[0100] Meanwhile, FIG. 7 illustrates in more detail the process of the simplification operation of step S306 of FIG. 3, in which a display device (10) according to an embodiment of the present invention replaces objects corresponding to a specific category with simplified objects. FIG. 8 is an example diagram illustrating the process of generating an image in which a real object corresponding to a specific category is replaced with a simplified object according to the operation process of FIG. 7.

[0101] First, referring to FIG. 7, if an object corresponding to a specific category is detected among the objects in the sensing image in step S304 of FIG. 3 based on the user's selection or the satisfaction of a preset condition, the processor (100) of the display device (10) can detect each object included in the sensing image. And, a simplified object corresponding to each of the detected objects can be generated (S700).

[0102] Here, the simplified object may be a polygonal graphic model (hereinafter referred to as a polygonal model) capable of representing the shape of an object. That is, the simplified object may be a polygonal virtual object in which textures are not reflected.

[0103] The step of generating the above polygon model S700 may be a step of transmitting location information of the vehicle (20) to a pre-configured server to request POI information around the location of the vehicle (20), and receiving polygon models corresponding to each object in the sensing image from the pre-configured server in response to the request.

[0104] Here, polygon models corresponding to each object within the sensing image may be included in POI information for buildings or areas around the location of the vehicle (20). To this end, a pre-configured server providing the POI information may be a server that includes a POI database containing polygon models for buildings or surrounding areas around the vehicle (20) corresponding to the location information of the vehicle (20). In this case, the pre-configured server may be a cloud server (30) and may be a server that includes a DTaaS (Digital Twin as a Service) server. Alternatively, the cloud server (30) may be a server that communicates with the DTaaS server.

[0105] Meanwhile, if a polygon model is generated in step S700 or if a polygon model is provided from a cloud server (30), the processor (100) can detect areas on the sensing image occupied by objects corresponding to a specific category based on user selection or the satisfaction of a preset condition among the objects in the sensing image in step S304 of FIG. 3 (S702). Then, pixel information, for example, color information of the detected areas can be initialized (S704). Then, areas on the sensing image corresponding to the objects detected in step S304 can be deleted.

[0106] And the processor (100) can display simplified objects created or received in step S700 in each area of ​​the sensing image where the objects detected in step S304 of FIG. 3 have been deleted (S706). Then, the sensing image in which some objects have been replaced with simplified objects in step S308 of FIG. 3 can be rendered on the display unit (150) through the renderer (140).

[0107] FIG. 8 is an example diagram illustrating an example of an image in which a real object corresponding to a specific category is replaced with a simplified object according to the operation process of FIG. 7.

[0108] For example, the processor (100) of the display device (10) may detect only objects corresponding to a specific category based on user selection or the fulfillment of preset conditions among the objects included in the sensing image as objects to be simplified, or conversely, may detect objects of other categories excluding the specific category based on user selection or the fulfillment of preset conditions as objects to be simplified. FIG. 8 illustrates an example in which, among the objects in the sensing image (800), all objects of other categories excluding objects corresponding to a specific category based on user selection or the fulfillment of preset conditions are detected as objects to be simplified.

[0109] First, referring to FIG. 8(a), FIG. 8(a) illustrates an example in which the category of objects within a sensing image (800) selected by the user or satisfied by a pre-set condition is a vehicle (810) and a road (811). In this case, all objects corresponding to other categories excluding the vehicle (810) and the road (811), such as a building (820), can be selected as objects to be simplified.

[0110] Then, the processor (100) can generate a simplified object corresponding to each of the objects to be simplified. In this case, the simplified object may be a polygon model corresponding to the shape of each of the objects to be simplified. To this end, the processor (100) can detect the shape of the object to be simplified through the outline of the object to be simplified from the sensing image (800) and generate a polygon model corresponding to the detected shape.

[0111] Alternatively, the processor (100) may transmit a request for POI information containing location information regarding the current location of the vehicle (20) to the cloud server (30). Then, the cloud server (30) may provide POI information regarding POIs around the location of the vehicle (20) in response to the request for POI information. Here, the processor (100) may detect the categories of objects detected as targets for simplification among each object included in the sensing image (800), and request POI information corresponding to the detected categories from the cloud server (30). Then, the cloud server (30) may detect only the POI information of the POIs corresponding to the requested categories among the POI information regarding POIs around the vehicle (20) and transmit it as a response to the request.

[0112] Meanwhile, the POI information provided from the cloud server (30) may include information about simplified objects, for example, polygon models. Accordingly, the processor (100) can obtain simplified objects corresponding to each of the objects detected as targets for simplification.

[0113] Then, the processor (100) can detect the area occupied by the objects detected as targets for simplification from the sensing image (800). And the detected area can be deleted. And simplified objects corresponding to each of the deleted objects can be placed in each area on the sensing image (800) from which the objects have been deleted. And the renderer (140) can be controlled so that the sensing image (800) in which some of the objects have been replaced with simplified objects is rendered on the display unit (150).

[0114] Accordingly, as shown in FIG. 8(b), objects (810) corresponding to category 'vehicle' and objects (811) corresponding to category 'road' are displayed as they are sensed, while other objects, such as building objects (820), can be replaced with simplified objects.

[0115] Meanwhile, according to the above description, it has been mentioned that the display device (10) according to an embodiment of the present invention can detect objects corresponding to a category based on the satisfaction of a preset condition as well as the user's selection among each object of the sensed image. It has also been mentioned that the preset condition may be illuminance, time, or weather.

[0116] In such cases, the aforementioned pre-set condition may be configured so that a specific category is selected when the pre-set condition is satisfied. That is, when a specific condition is satisfied, a specific category matched to the specific condition may be selected as a category for detecting objects to be simplified.

[0117] FIG. 9 is for explaining a display device (10) according to an embodiment of the present invention in which a specific category is automatically selected based on the satisfaction of a preset condition, assuming that the condition is a preset illuminance and the category matching the condition is 'building'.

[0118] First, referring to FIG. 9 (a), the processor (100) can receive an image (sensing image (900)) sensed by the camera (21) while the vehicle (20) is in motion. Then, based on the information detected by the sensor unit (23) of the vehicle (20), it can determine whether a pre-set condition matched to a specific category is satisfied.

[0119] In this case, the above-mentioned preset condition may be illuminance. Then, the processor (100) can determine whether the illuminance around the vehicle (20) has reached the preset illuminance based on the illuminance detected by the illuminance sensor of the vehicle (20). And if the determination result shows that the illuminance around the vehicle (20) has reached the preset illuminance, the processor (100) can determine that the preset condition matched to the specific category is satisfied.

[0120] Meanwhile, when the above-mentioned pre-set condition is satisfied, the processor (100) may select a specific category that matches the above condition as a category for simplifying objects within the sensing image (900). In this case, if the category that matches the above-mentioned illuminance condition is 'building', the processor (100) may detect objects (910) corresponding to the category 'building' that matches the satisfied condition from the sensing image (900), as shown in FIG. 9 (b).

[0121] Then, the processor (100) can perform simplification on the detected objects (910). For example, the processor (100) can delete the detected objects (910) from the sensing image (900). Thus, as shown in (c) of FIG. 9, an image in which the objects (910) corresponding to buildings among the objects of the sensing image (900) have been deleted can be output through the display unit (150).

[0122] Alternatively, the processor (100) may generate a simplified object corresponding to each of the detected objects (910) or receive it from the cloud server (30) via POI information, and replace the objects (910) corresponding to buildings among the objects of the sensing image (900) with the generated or received simplified objects. Accordingly, as shown in (d) of FIG. 9, an image in which the objects (910) corresponding to buildings among the objects of the sensing image (900) are replaced with simplified objects such as polygon models can be output through the display unit (150).

[0123] Meanwhile, according to the above description, it has been mentioned that the display device (10) according to the embodiment of the present invention may use an augmented reality object received through POI information as a simplified object.

[0124] FIGS. 10a and 10b are flowcharts illustrating in more detail steps S304 and S306 of FIG. 3, in which a display device according to an embodiment of the present invention replaces a real-world object corresponding to a specific category with AR information corresponding to the specific category. FIG. 11 is an example diagram illustrating an example in which a real-world object corresponding to a specific category is replaced with AR information according to the operation process of FIGS. 10a and 10b.

[0125] First, in step S302 of FIG. 3, the processor (100) of the display device (10) can determine the category of each object included in the image sensed by the camera (21) based on the category specified in the classification information database (161). Then, when step S304 of FIG. 3 proceeds to detect objects corresponding to the category selected by the user or satisfied by a preset condition, as shown in FIG. 10a, information about the category selected by the user or satisfied by a preset condition (hereinafter the selected category) can be transmitted to the cloud server (30) along with the location information of the vehicle (20) (S1000). That is, the processor (100) can request information (POI information) of POIs corresponding to the selected category from around the current location of the vehicle (20) from the cloud server (30).

[0126] And in response to the above request, when POI information around the vehicle (20) corresponding to the selected category is received (S1002), the processor (100) can detect objects corresponding to a category selected by the user or satisfied by a pre-set condition based on the categories determined for each object included in the sensing image (S1004). Meanwhile, since the received POI information is received from the cloud server (30) according to the category selected by the user or satisfied by a pre-set condition, the object detection in step S1004 may be a process of detecting objects among the objects included in the sensing image that correspond to the category of the received POI information.

[0127] Meanwhile, if at least one object among the objects included in the sensing image is detected through step S304 of Fig. 3 as illustrated in Fig. 10a, the processor (100) may proceed to step S306 of Fig. 3 to perform simplification on the objects detected in step S304. Fig. 10b below illustrates in more detail the simplification process performed in step S306 of Fig. 3 in such a case.

[0128] Referring to FIG. 10b, the processor (100) of the display device (10) according to an embodiment of the present invention can first extract an augmented reality object corresponding to a category selected by a user or satisfied by a preset condition from POI information received from a cloud server (30) (S1050). Then, through the operation process of FIG. 10a, the areas occupied by objects corresponding to a specific category selected by a user or satisfied by a preset condition among the objects detected from the sensing image can be detected (S1052). Then, pixel information, for example, color information of the detected areas can be initialized (S1054). Then, the areas on the sensing image detected in step S1052 can be deleted.

[0129] And the processor (100) can display the augmented reality objects extracted in step S1050 on each area of ​​the sensing image deleted in step S1054 (S1506). Then, the sensing image in which some objects are replaced with augmented reality objects in step S308 of FIG. 3 can be rendered on the display unit (150) through the renderer (140).

[0130] FIG. 11 is a drawing showing an example in which at least one object included in a sensing image is replaced with an augmented reality object through steps S304 and S306 of FIG. 3 according to the process described in FIG. 10a and FIG. 10b.

[0131] Referring to FIG. 11, FIG. 11 (a) shows an example of a sensing image (1100) including buildings (1110, 1111, 1112) and vehicle and road objects. In this case, the processor (100) can identify and classify each object of the sensing image (1100) into objects (1110, 1111, 1112) corresponding to the building category, objects corresponding to the vehicle category, and objects corresponding to the road category, based on a classification information database.

[0132] Meanwhile, the processor (100) may request information about POIs around the current vehicle (20), including information about a category based on the user's selection or the fulfillment of pre-set conditions, and information about the location of the current vehicle (20), from the cloud server (30). Then, the cloud server (30) may detect information about a POI corresponding to the category included in the received request among the POIs around the location of the vehicle (20), and provide information including an augmented reality object corresponding to the detected POI, i.e., POI information, as a response to the received request. Accordingly, the processor (100) can obtain POI information including an augmented reality object for a POI around the current vehicle (20), corresponding to a category based on the user's selection or the fulfillment of pre-set conditions.

[0133] In this state, among the buildings around the vehicle (20), the first building may correspond to a category based on the user's selection or the fulfillment of a pre-set condition. Then, the processor (100) can detect an object (1110) corresponding to the first building from the sensing image (1100) shown in (a) of FIG. 11. And by initializing a region on the sensing image (1100) corresponding to the detected object (1110) of the first building, the object (1110) corresponding to the first building can be deleted from the sensing image (1100).

[0134] Then, the processor (100) can extract an augmented reality object (1120) corresponding to the first building from the POI information received from the cloud server (30). And the extracted augmented reality object can be displayed in a part of the sensing image (1100) corresponding to the deleted object (1110). Thus, as shown in FIG. 11 (b), instead of the object (1110) corresponding to a specific category, an image (1101) containing the augmented reality object (1120) can be output on the display unit (150).

[0135] Meanwhile, the processor (100) can detect color or pattern information of a surrounding area for a part of the sensing image (1100) corresponding to the deleted object (1110) and display a fill image generated according to the detected color or pattern information. Additionally, the augmented reality object may be superimposed and displayed on the area where the fill image is displayed. Thus, the area where the object corresponding to the first building was deleted can be displayed more naturally.

[0136] Meanwhile, although the above description only used external vehicle environments such as lighting or weather as examples of pre-set conditions, it is obvious that the user's physical condition may also be set as the pre-set condition. For example, if the user's physical condition is an emergency situation such as a seizure, the processor (100) may automatically select a category such as a hospital based on the detected physical condition of the user. Alternatively, if the user's physical condition is detected and it is determined that the user is in a state of severe heat (vehicle interior temperature is above a certain temperature) or that the user is dehydrated based on the detection results of the DMS (Driver Monitoring System), a category such as a restaurant (coffee shop, etc.) where fluids can be supplied, i.e., where beverages can be consumed, may be automatically set.

[0137] FIG. 12 is a flowchart illustrating the operation process of a display device (10) according to an embodiment of the present invention, which automatically selects a specific category based on the detected physical condition of a user and detects an object corresponding to the selected category.

[0138] In step S302 of FIG. 3, the processor (100) of the display device (10) can determine the category of each object included in the image sensed by the camera (21) based on the category specified in the classification information database (161). And as shown in FIG. 12, the processor (100) of the display device (10) can receive the user's biometric information sensed from at least one device capable of sensing the user's biometric information (S1200).

[0139] Here, the at least one device may be a wearable device, such as a smart watch worn by a user, or a sensor of the DMS of the vehicle (20). For example, the sensor of the DMS may be an internal camera that acquires images to detect the condition of a driver or passenger in the cabin of the vehicle (20). Here, the 'user' may be any one of the passengers in the vehicle (20).

[0140] And the processor (100) can determine whether the received user's biometric information satisfies a pre-set condition for object detection (S1202). That is, the processor (100) determines the user's biometric state based on the received user's biometric information and determines whether the determined user's biometric state satisfies a pre-set condition corresponding to a specific category. Here, the pre-set condition may be the biometric state of different users, and different biometric state conditions may be matched to different categories (e.g., dehydration or thirst state - matched to a coffee shop or restaurant, seizure state - matched to a hospital).

[0141] If, as a result of the determination in step S1202 above, the received user's bio-state satisfies a preset bio-state condition, the processor (100) can determine a specific category that matches the satisfied bio-state condition. Then, the information of the determined specific category and the location information of the current vehicle (20) can be transmitted to a cloud server (30) to request POI information around the vehicle (20) corresponding to the specific category (S1204). Then, when POI information around the vehicle corresponding to the specific category is received from the cloud server (30), at least one object according to the received POI information among each object in the sensing image identified in step S302 of FIG. 3 can be detected as an object corresponding to the category that matches the satisfied bio-state condition (S1206).

[0142] Meanwhile, as described in FIG. 12, when an object corresponding to POI information received from a cloud server (30) is detected from a sensing image, the processor (100) can replace the detected object with an augmented reality object extracted from the received POI information. Then, an image in which some of the objects are replaced with augmented reality objects can be output on the display unit (150). That is, through a process similar to that described in FIG. 10b, an image containing an augmented reality object extracted from the received POI information can be output on the display unit (150).

[0143] Meanwhile, the above description explains an example in which at least one of the objects in a sensed image is deleted or replaced with a simplified object or an augmented reality object based on a specific category selected according to the user's choice or the fulfillment of pre-set conditions.

[0144] However, alternatively, the user may select a specific POI instead of a specific category. In this case, the processor (100) of the display device (10) may delete at least a portion of the sensing image or replace it with an augmented reality object based on the selected POI.

[0145] FIGS. 13a and 13b are flowcharts illustrating the operation process of a display device according to an embodiment of the present invention detecting at least one object from a sensing image according to the location of a vehicle and a specific POI, and displaying an augmented reality object of the specific POI instead of the detected object.

[0146] First, in step S302 of FIG. 3, the processor (100) of the display device (10) can determine the category of each object included in the image sensed by the camera (21) based on the category specified in the classification information database (161). Then, when step S304 of FIG. 3 is performed, the processor (100) can request POI information including the location of a specific POI selected by the user from the cloud server (30), as shown in FIG. 13a, and receive POI information including the location of the specific POI in response to the request (S1300).

[0147] Then, the processor (100) can calculate the direction and distance corresponding to the location of the specific POI from the sensing image based on the current location of the vehicle (20) and the location information of the specific POI (S1302). And from the sensing image, at least one object located in the direction corresponding to the location of the specific POI can be detected.

[0148] For example, if the direction in which the vehicle (20) faces the specific POI is included within the Field of View (FOV) of the sensing image, at least one object displayed within the sensing image may be located between the specific POI and the vehicle. Then, the processor (100) can detect at least one object within the sensing image that is located between the specific POI and the vehicle and obscures the specific POI as an object selected by the user. Then, the processor (100) can proceed to step S306 of FIG. 3 to perform simplification on the objects detected through step S304, i.e., the operation process of FIG. 13a. FIG. 13b below illustrates in more detail the simplification process performed in step S306 of FIG. 3 in such a case.

[0149] Referring to FIG. 13b, the processor (100) of the display device (10) according to an embodiment of the present invention can first extract an augmented reality object corresponding to the specific POI from POI information received from the cloud server (30) (S1350). Then, from the sensing image, it can detect areas corresponding to at least one object detected through the operation process of FIG. 13a, that is, at least one object within the sensing image located between the specific POI and the vehicle and obscuring the specific POI (S1352).

[0150] Then, the processor (100) can initialize pixel information of one area within the sensing image corresponding to the detected areas to delete at least one object detected through the operation process of FIG. 13a from the sensing image (S1354). Then, an augmented reality object corresponding to a specific POI extracted from the POI information can be displayed in the area within the sensing image where the at least one object has been deleted (S1356). Then, in step S308 of FIG. 3, an image in which the at least one object detected through the operation process of FIG. 13a is replaced with an augmented reality object corresponding to a specific POI selected by the user can be rendered on the display unit (150) through the renderer (140).

[0151] In this case, the size of the augmented reality object can be determined according to the distance between the current vehicle (20) and the location of the specific POI. That is, the closer the distance between the current vehicle (20) and the specific POI, the larger the size of the augmented reality object can be displayed, and the closer the distance, the smaller the size of the augmented reality object can be displayed.

[0152] Meanwhile, the processor (100) may detect color or pattern information of a surrounding area for a region on a sensing image corresponding to at least one deleted object and display a fill image generated according to the detected color or pattern information. Additionally, the augmented reality object may be superimposed and displayed on the region where the fill image is displayed. Thus, an augmented reality object corresponding to the specific POI may be displayed more naturally in the region where the at least one object was deleted.

[0153] Additionally, the number of objects to be deleted within the sensing image may increase or decrease depending on the size of the augmented reality object. For example, as the distance between the vehicle (20) and the specific POI decreases, if the size of the augmented reality object corresponding to the specific POI increases, there may be more objects within the sensing image that obscure the specific POI. Therefore, a larger number of objects within the sensing image may be deleted.

[0154] On the other hand, as the distance between the vehicle (20) and the specific POI increases, the size of the augmented reality object corresponding to the specific POI decreases, so that fewer objects in the sensing image may obscure the specific POI. Therefore, fewer objects in the sensing image may be deleted.

[0155] FIG. 14 is an example diagram illustrating an example in which some of the objects included in the sensed image are replaced with augmented reality objects of a specific POI according to the operation process of FIG. 13a and FIG. 13b.

[0156] Referring to FIG. 14 (a), the processor (100) can receive an image (sensing image (1400)) sensed by a camera (21) while the vehicle (20) is in motion. In this state, the processor (100) can detect an object located in a direction toward the specific POI within the sensing image, that is, an object within the sensing image located between the specific POI and the vehicle that obscures the specific POI, based on the location of the specific POI selected by the user and the current location of the vehicle.

[0157] Meanwhile, if the specific POI is the Eiffel Tower (in this case, an augmented reality object corresponding to the Eiffel Tower may be received from the cloud server (30)) and the object in the sensing image covering the specific POI is an object corresponding to the first building (first building object (1410)) and a vehicle object (1411), the processor (100) can detect areas corresponding to the first building object (1410) and the vehicle object (1411) within the sensing image, as shown in (b) of FIG. 14.

[0158] Then, by initializing the pixel information of the detected areas, the first building object (1410) and the vehicle object (1411) can be deleted from the sensing image. And as shown in FIG. 14 (b), an augmented reality object (1420) corresponding to a currently selected specific POI, namely the Eiffel Tower, can be displayed in an area within the sensing image (1400) corresponding to the deleted first building object (1410) and vehicle object (1411). In this case, the size of the augmented reality object (1420) can be determined according to the distance between the vehicle (20) and the Eiffel Tower.

[0159] Meanwhile, unlike what is shown in (b) of Fig. 14 above, the user may, of course, set specific categories for which deletion is restricted in advance. In this case, even if an object is obscured by a specific POI within the sensing image, an object in a category for which deletion is restricted by the user may not be deleted. This is to prevent the risk of accidents occurring, such as when objects are indiscriminately deleted within the sensing image according to the POI selected by the user, and even though the objects actually exist, they are not displayed on the display unit (150) (e.g., nearby vehicle objects).

[0160] For example, if the user has pre-set the deletion of an object corresponding to a vehicle category to be restricted, the processor (100) may not delete the vehicle object (1411) even though both the first building object (1410) and the vehicle object (1411) obscure the specific POI. In this case, as shown in (c) of FIG. 11, only the first building object (1410) may be deleted, and an augmented reality object (1320) corresponding to the specific POI may be displayed in a part of the sensing image corresponding to the deleted first building object (1410).

[0161] On the other hand, unlike the above, it is also possible to pre-designate a specific category of objects that can be deleted among the objects obscuring a specific POI within the sensing image. In this case, among the objects obscuring a specific POI within the sensing image, only those objects corresponding to the specific category for which deletion is permitted can be deleted. Additionally, an augmented reality object corresponding to the specific POI may be displayed in the deleted area.

[0162] Meanwhile, the processor (100) of the display device (10) according to an embodiment of the present invention may track and display the relative position change between a specific object and the vehicle (20) that changes according to the movement of the vehicle (20) through an augmented reality object displayed on the sensing image.

[0163] FIG. 15 is an example diagram illustrating an example in which the position of a specific object moving on a sensing image is displayed using an augmented reality object corresponding to a specific object selected by a user in a display device (10) according to an embodiment of the present invention.

[0164] First, referring to FIG. 15 (a), the processor (100) of the display device (10) may receive a selection of at least one object among the objects displayed in the sensing image. In this case, the selected object may be an object directly selected by the user among the displayed objects, or an object detected according to a category or POI selected by the user. For convenience of explanation, in the following description, it will be assumed that the object (1510) corresponding to the sun in the sensing image is selected by the user.

[0165] In such cases, the processor (100) can detect whether the selected object within the sensing image is obscured by another object within the sensing image as the vehicle moves or as time flows. And if at least a portion of the selected object is obscured by the other object, the obscured portion can be displayed as an augmented reality object corresponding to the selected object. And the selected object moving as the vehicle moves or as time flows can be displayed as the movement of the augmented reality object.

[0166] That is, as shown in FIG. 15 (a), when a selected object (1510) moves according to the movement of the vehicle (20) or the flow of time, and at least a part of it is obscured by a building object in the sensing image, the processor (100) can display the selected object (1510) through various augmented reality objects corresponding to the selected object (1510), as shown in FIG. 15 (b) to (d). That is, the movement of the selected object can be displayed by superimposing an augmented reality object (1511) having a hatched circular shape (Fig. 15 (b)), an augmented reality object (1512) having a three-dimensional shape with a concave or convex center (Fig. 15 (c)), or an augmented reality object (1513) having a cylindrical shape (Fig. 15 (d)) on another object in the sensing image.

[0167] In this case, the augmented reality object can be displayed on top of other objects within the sensing image. Therefore, the state of movement of the selected object within the sensing image can be tracked and displayed.

[0168] Meanwhile, although the above description assumes that a specific object among the objects in the sensing image is selected by the user, it is obvious that an object corresponding to a specific POI or specific category selected by the user may be tracked and displayed in the same manner. In such a case, even if the specific object corresponding to the specific POI or specific category is obscured by another object in the sensing image, the augmented reality object corresponding to the specific object is superimposed on the other object obscuring the specific object at a position corresponding to the specific object, thereby allowing the state of movement of the specific object within the sensing image to be displayed on the display unit (150).

[0169] The above-described invention may be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also include implementation in the form of a carrier wave (e.g., transmission over the Internet). Furthermore, the computer may include a processor (100) of the display device (10). Accordingly, the above detailed description should not be interpreted restrictively in all respects and should be considered exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

Claim 1 An interface unit for receiving a sensing image of the surrounding environment of the vehicle from at least one sensor provided in the vehicle; a memory including a classification information database including classification information for classifying each object included in the sensing image into different pre-set categories; an object identification unit for classifying and identifying each object included in the sensing image into one of the categories in the classification information database based on the classification information; and a renderer for rendering an image to be displayed on a display unit provided in the vehicle. A display device comprising: a processor that detects objects corresponding to at least one category selected by a user among the objects identified by the category, controls the renderer to render an output image in which the detected objects are replaced with graphic objects corresponding to the detected objects, and controls the interface unit to output the rendered output image on the display unit through the interface; wherein the processor deletes objects detected according to the at least one selected category from the sensing image, controls the renderer to render the output image in which the detected objects are deleted from the sensing image, and further detects objects corresponding to a specific category that matches the satisfied condition from the sensing image when a preset condition is satisfied. Claim 2 A display device according to claim 1, further comprising a communication unit that communicates with a cloud server providing POI information including an augmented reality object corresponding to a POI (Position Of Interest) around the vehicle, wherein the processor requests POI information about a POI around the vehicle, including the current location of the vehicle and information about at least one category selected by the user, from the cloud server, and when at least one POI information corresponding to at least one category among the POIs around the vehicle is received from the cloud server in response to the request, the processor extracts an augmented reality object from the received POI information and controls the renderer to render an output image in which the detected objects are replaced with the extracted augmented reality object. Claim 3 A display device according to paragraph 2, wherein the cloud server includes POI information classified according to the plurality of pre-set categories, and the classification information database includes classification information for classifying each object included in the sensing image according to the plurality of pre-set categories in which the cloud server classified the POI information. Claim 4 A display device according to claim 1, characterized in that the graphic object is a polygon object representing the shape of each of the detected objects. Claim 5 delete Claim 6 A display device according to claim 1, wherein the processor detects an area where the detected objects are displayed from the sensing image and initializes the color information of the detected area to delete the detected objects from the sensing image, and the color of the area where the color information is initialized varies according to the user's selection. Claim 7 A display device according to claim 1, wherein the processor detects at least one of the color of a surrounding area of ​​the deleted objects and the color pattern of the surrounding area when the detected objects are deleted from the sensing image, generates a fill image corresponding to a part of the sensing image in which the detected objects are deleted based on at least one of the color and the color pattern, and controls the renderer to render an output image in which the fill image is displayed within a part of the sensing image corresponding to the deleted objects. Claim 8 delete Claim 9 A display device according to claim 1, wherein the above-mentioned pre-set condition is a condition regarding at least one of the illuminance, weather, or time of the vehicle. Claim 10 A display device according to claim 1, characterized in that the above-mentioned pre-set condition is a condition for biometric information detected from at least one of the occupants of the vehicle detected by the sensor unit of the vehicle. Claim 11 A display device according to claim 2, wherein the processor detects at least one object within the sensing image located in a direction toward the specific POI from the vehicle based on location information of a pre-set specific POI and location information of a vehicle, deletes the detected at least one object from the sensing image, and controls the renderer to render an output image in which a graphic object corresponding to the specific POI is displayed in a part of the sensing image corresponding to the deleted at least one object. Claim 12 A display device according to claim 11, wherein the graphic object is an augmented reality object corresponding to the specific POI, and the processor receives location information of the specific POI and the augmented reality object corresponding to the specific POI from the cloud server. Claim 13 A display device according to claim 11, wherein the processor calculates the distance between the specific POI and the vehicle based on the location information of the specific POI and the location information of the vehicle, and determines the size of the graphic object according to the calculated distance. Claim 14 A display device according to claim 11, wherein the processor detects objects within the sensing image located in a direction toward the specific POI that correspond to a category where deletion is restricted, and deletes only the remaining objects excluding the objects corresponding to the category where deletion is restricted. Claim 15 A method for controlling a display device connected to a camera and at least one sensor and a display unit, which are provided in a vehicle through an interface unit, comprising: receiving a sensing image sensed through the camera; classifying and identifying categories corresponding to each of the objects included in the sensing image based on classification information for classifying into different categories set by a user; detecting objects corresponding to at least one specific category selected by a user among the objects whose categories are identified; generating an output image by replacing the detected objects with graphic objects corresponding to each of the detected objects; and rendering the output image so that the generated output image is displayed on the display unit, wherein the step of generating the output image further comprises initializing a region within the sensing image corresponding to each of the detected objects to generate an output image in which the detected objects are deleted from the sensing image, and the step of detecting objects corresponding to at least one specific category selected by a user among the objects whose categories are identified further comprises detecting objects corresponding to the specific category that matches the satisfied condition from the sensing image when a set condition is satisfied. Claim 16 delete Claim 17 A control method for a display device according to claim 15, wherein the step of generating the output image comprises: initializing a region within the sensing image corresponding to each of the detected objects; detecting at least one of the color of the surrounding region of the deleted objects and the color pattern of the surrounding region; generating a fill image corresponding to a region within the sensing image where the detected objects have been deleted based on at least one of the color and the color pattern; and generating an output image in which the fill image is displayed within a region of the sensing image corresponding to the deleted objects. Claim 18 A control method for a display device according to claim 15, wherein the step of generating the output image comprises: transmitting information of the specific category and location information of the vehicle to a cloud server to request POI information including an augmented reality object corresponding to a POI (Position Of Interest) of the category corresponding to the specific category around the location of the vehicle; receiving POI information from the cloud server in response to the request; deleting an object corresponding to the specific category from the sensing image among the objects included in the sensing image; and generating an output image in which an augmented reality object corresponding to each object deleted from the sensing image is displayed in a part of the sensing image corresponding to the deleted object. Claim 19 A control method for a display device according to claim 15, wherein the step of detecting objects corresponding to the specific category comprises: a step of detecting the location of a specific POI selected by a user based on the location of the vehicle; and a step of detecting at least one object displayed in a direction oriented toward the location of the specific POI from the location of the vehicle from the sensing image, wherein the graphic object corresponding to the detected object is an augmented reality object corresponding to the specific POI.

Images