Display apparatus for augmented reality-based vehicle and method for controlling same

Abstract

The present invention is characterized by comprising: an interface unit for receiving, from at least one sensor provided in a vehicle, a sensing image for sensing the surroundings of the vehicle; a classification information database including classification information for classifying objects included in the sensing image by classes different from each other; an object recognition unit that classifies and recognizes, on the basis of the classification information, each object included in the sensing image as any one of the categories in the classification information database; a renderer that renders an image to be displayed on a display unit provided in the vehicle; and a processor that detects an object corresponding to the category selected by the user from among the objects whose categories are recognized, controls the renderer to render an output image in which the detected object is replaced with a graphic object corresponding to the detected object, and controls the interface section to output the rendered output image on the display section through the interface.

Description

Technical Field

[0001] The present invention relates to a display device for displaying an image of the surrounding environment of a vehicle. Background Art

[0002] In recent years, augmented reality (AR) has emerged, which outputs graphic objects through the vehicle's windshield and HUD (Head Up Display), or superimposes graphic objects on images captured by a camera to add virtual objects to the real world. This technology provides additional information about the vehicle's driving path, allowing the driver to intuitively identify the vehicle and its driving environment. This can further improve driving efficiency and convenience.

[0003] Research is actively underway to use this augmented reality technology to provide user-customized information based on the real world. As part of this research, AR information filtering technology has been developed. This technology pre-configures POI (Point of Interest) categories from the user and displays augmented reality information based on the POI information received from a cloud server only when there are POIs matching these pre-configured categories near the vehicle. This allows users to view only the augmented reality information they are interested in.

[0004] However, this AR information filtering technology restricts the display of AR information to only those related to the real-world objects surrounding the user, leaving the real-world objects displayed as is. In other words, it only provides user-customized information for the displayed AR information, leaving real-world objects that the user is not interested in or does not want to see untouched.

[0005] Furthermore, there is a need to apply the augmented reality technology to objects in the real world, for example, to continuously provide information of a specific POI required by the user regardless of the objects in the real world surrounding the user. Summary of the Invention

[0006] Problems to be solved by the invention

[0007] The purpose of the present invention is to solve the above-mentioned problems and other problems, and to provide a display device and a control method for the device that use augmented reality technology to not only augment reality information but also objects in the real world that the user is not interested in or does not want to see, or to simplify the display of objects.

[0008] In addition, an object of the present invention is to provide a display device and a control method for the device that uses augmented reality technology to display information such as the position and direction of a specific POI required or desired by a user independently of real-world objects, or can display the position and direction of the specific POI that changes relatively with the position of a moving vehicle independently of real-world objects.

[0009] Technical solutions to the problem

[0010] In order to achieve the above or other purposes, according to one aspect of the present invention, a display device of an embodiment of the present invention includes: an interface unit, which receives a sensing image of the surrounding environment of the vehicle from at least one sensor provided in the vehicle; a memory, which includes a classification information database, and the classification information database includes classification information for classifying each object contained in the sensing image according to preset different categories; an object recognition unit, which classifies and identifies each object contained in the sensing image as any one of the categories in the classification information database according to the classification information; a renderer, which renders an image to be displayed on a display unit provided in the vehicle; and a processor, which detects an object corresponding to at least one category selected by a user from the objects of the identified category, controls the renderer to render an output image that replaces the detected object with a graphic object corresponding to the detected object, and controls the interface unit to output the rendered output image on the display unit through the interface.

[0011] In one embodiment, the system further includes a communication unit that communicates with a cloud server, the cloud server providing POI information including augmented reality objects corresponding to POIs around the vehicle, the processor requesting POI information for POIs around the vehicle including the current position of the vehicle and information of at least one category selected by the user from the cloud server, and if at least one POI information corresponding to the at least one category among the POIs around the vehicle is received from the cloud server in response to the request, the processor extracts the augmented reality object from the received POI information and controls the renderer to render an output image that replaces the detected object with the extracted augmented reality object.

[0012] In one embodiment, the cloud server includes POI information classified according to the preset multiple categories, and the classification information database includes classification information for classifying each object contained in the sensed image according to the preset multiple categories used by the cloud server to classify the POI information.

[0013] In one embodiment, the graphic object is a polygon object, which is a graphic object representing the shape of each of the detected objects.

[0014] In one embodiment, the processor deletes the object detected according to the selected at least one category from the sensed image, and controls the renderer to render the output image with the detected object deleted from the sensed image.

[0015] In one embodiment, the processor detects an area showing the detected object from the sensed image and initializes color information of the detected area to delete the detected object from the sensed image, and the color of the initialized area varies according to a user's selection.

[0016] In one embodiment, if the detected object is deleted from the sensed image, the processor detects at least one of the color of the peripheral area of ​​the deleted object and the color pattern of the peripheral area, generates a fill image corresponding to an area in the sensed image where the detected object is deleted based on at least one of the color and the color pattern, and controls the renderer to render an output image displaying the fill image in an area corresponding to the deleted object in the sensed image.

[0017] In one embodiment, when a preset condition is met, the processor further detects an object corresponding to a specific category that matches the met condition from the sensed image.

[0018] In one embodiment, the preset condition is a condition for at least one of illumination, weather and time of the vehicle.

[0019] In one embodiment, the preset condition is a condition of biological information detected by a sensor unit of the vehicle from at least one of the passengers of the vehicle.

[0020] In one embodiment, the processor detects at least one object in the sensed image located in a direction from the vehicle to the specific POI based on preset location information of the specific POI and location information of the vehicle, deletes the detected at least one object from the sensed image, and controls the renderer to render an output image in which a graphic object corresponding to the specific POI is displayed in an area corresponding to the deleted at least one object in the sensed image.

[0021] In one embodiment, the graphic object is an augmented reality object corresponding to the specific POI, and the processor receives the location information of the specific POI and the augmented reality object corresponding to the specific POI from the cloud server.

[0022] In one embodiment, the processor calculates the distance between the specific POI and the vehicle according to the location information of the specific POI and the location information of the vehicle, and determines the size of the graphic object according to the calculated distance.

[0023] In one embodiment, the processor detects objects corresponding to a category with deletion restrictions among objects located in a direction pointing to the specific POI in the sensed image, and deletes only objects other than the objects corresponding to the category with deletion restrictions.

[0024] In order to achieve the above or other purposes, according to one aspect of the present invention, an embodiment of the present invention is a control method for a display device connected to a camera, at least one sensor and a display unit provided in a vehicle through an interface unit, and is characterized in that it includes: a step of receiving a sensing image sensed by the camera; a step of classifying and identifying categories corresponding to each object contained in the sensing image according to classification information for classifying according to preset different categories; a step of detecting objects corresponding to at least one specific category selected by a user from objects of the identified categories; a step of replacing the detected objects with graphic objects respectively corresponding to the detected objects to generate an output image; and a step of rendering the output image to output the generated output image on the display unit.

[0025] In one embodiment, the step of generating the output image is a step of initializing a region in the sensed image corresponding to each of the detected objects, thereby generating the output image in which the detected objects are deleted from the sensed image.

[0026] In one embodiment, the step of generating the output image includes: a step of initializing an area in the sensed image corresponding to the detected objects; a step of detecting at least one of the color of the peripheral area of ​​the deleted object and the color pattern of the peripheral area; a step of generating a fill image corresponding to an area in the sensed image where the detected object is deleted based on at least one of the color and the color pattern; and a step of generating an output image in which the fill image is displayed in an area in the sensed image corresponding to the deleted object.

[0027] In one embodiment, the step of generating the output image includes: transmitting the information of the specific category and the location information of the vehicle to a cloud server, and requesting POI information of augmented reality objects corresponding to POIs of the category corresponding to the specific category around the location of the vehicle; receiving POI information from the cloud server in response to the request; deleting objects corresponding to the specific category from the objects contained in the sensing image; and generating an output image in which augmented reality objects corresponding to the objects deleted from the sensing image are displayed on an area corresponding to the deleted objects in the sensing image.

[0028] In one embodiment, the step of detecting an object corresponding to the specific category includes: a step of detecting the position of a specific POI selected by a user based on the position of the vehicle; and a step of detecting at least one object displayed in a direction pointing from the position of the vehicle to the position of the specific POI from the sensed image; the graphic object corresponding to the detected object is an augmented reality object corresponding to the specific POI.

[0029] Effects of the Invention

[0030] The effects of the display device and the method for controlling the device according to the present invention are described below.

[0031] According to at least one embodiment of the present invention, the present invention can classify and identify objects included in an image sensed by a vehicle camera according to preset categories, and then simplify the image by deleting or replacing objects corresponding to at least one category set by preset conditions or the user with simplified objects, thereby displaying the simplified image on a display unit. This provides the user with an image of the vehicle's surroundings that does not include objects of a specific type that the user is not interested in or does not want to see.

[0032] In addition, according to at least one of the embodiments of the present invention, the present invention has the ability to delete the user's current state or an object between a specific POI set by the user and the vehicle from an image sensed by a camera, and display information of the specific POI in an area of ​​the image from which the object is deleted, thereby providing information of the specific POI without the effect of occlusion by objects in the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 1 is a diagram showing an example of the configuration of a display device according to an embodiment of the present invention.

[0034] Figure 2 This is a conceptual diagram showing a structure in which a display device according to an embodiment of the present invention cooperates with a cloud server to provide augmented reality information.

[0035] Figure 3 FIG. 1 is a flowchart illustrating an operation process of a display device according to an embodiment of the present invention outputting an image including at least one simplified object according to a condition or a category selected by a user.

[0036] Figure 4 1 is an exemplary diagram showing an example in which a display device according to an embodiment of the present invention receives a category selection by a user.

[0037] Figure 5 The display device of the embodiment of the present invention performs deletion of objects corresponding to a specific category. Figure 3 A simplified flowchart of the action process.

[0038] Figure 6 According to the Figure 5 An example diagram of a process of generating an image of a real object corresponding to a specific category by performing the action process.

[0039] Figure 7 The display device of the embodiment of the present invention replaces the object corresponding to a specific category with a simplified object. Figure 3 A simplified flowchart of the action process.

[0040] Figure 8 It is shown that according to the Figure 7 An example diagram of the process of generating an image by replacing the real object corresponding to a specific category with a simplified object.

[0041] Figure 9 1 is an exemplary diagram illustrating an example in which a display device according to an embodiment of the present invention simplifies objects corresponding to a specific category according to a preset illumination.

[0042] Figure 10a and Figure 10b The display device of the embodiment of the present invention replaces the real object corresponding to the specific category with the AR information corresponding to the specific category and displays it in detail. Figure 3 Flowchart of steps S304 and S306.

[0043] Figure 11 It is shown that according to the Figure 10a and Figure 10b The example diagram of the action process of replacing the real object corresponding to a specific category with an AR information example.

[0044] Figure 12 This is a flowchart illustrating an operation process of a display device according to an embodiment of the present invention detecting an object of POI information corresponding to a specific category based on user biometric information.

[0045] Figure 13a and Figure 13bThe flowchart shows an operation process of a display device according to an embodiment of the present invention detecting at least one object from a sensing image based on the position of a vehicle and a specific POI and displaying an AR object of the specific POI to replace the detected object.

[0046] Figure 14 It is shown that according to the Figure 13a and Figure 13b An example diagram of an image in which a portion of a real object included in a sensed image is replaced with an AR object of a specific POI is generated through the action process.

[0047] Figure 15 1 is an exemplary diagram showing an example in which an AR object corresponding to a specific object selected by a user is displayed on an image showing a real object in a display device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] It should be noted that the technical terms used in this specification are intended only to describe specific embodiments and are not intended to limit the present invention. Furthermore, unless otherwise specified, expressions in the singular encompass expressions in the plural. The suffixes "module" and "unit" used in the following description for components are assigned or used interchangeably for ease of writing and do not inherently distinguish between them in meaning or function.

[0049] In this specification, the terms "compose" or "comprise" should not be interpreted as necessarily including all the constituent elements or steps contained in the specification, but should be interpreted as some of the elements or steps may be excluded, or additional elements or steps may be included.

[0050] Furthermore, when describing the technology disclosed in the present invention, if it is determined that a detailed description of a related known technology may obscure the technical gist of the present invention, the detailed description will be omitted.

[0051] In addition, it should be understood that the accompanying drawings are only used to facilitate understanding of the embodiments of the present invention, and the technical concept of the present invention is not limited to the drawings, and includes all changes, equivalents, and substitutes included in the concept and technical scope of the present invention. In addition, in addition to the following embodiments, combinations of the embodiments as changes, equivalents, or substitutes included in the concept and technical scope of the present invention naturally also fall within the concept and technical scope of the present invention.

[0052] Figure 1 1 is a diagram showing an example of the configuration of the display device 10 according to the embodiment of the present invention.

[0053] Reference Figure 1 The display device 10 of the embodiment of the present invention may include a processor 100 and an interface 110 connected to the processor 100, a communication unit 120, an object recognition unit 130, a renderer 140, a display unit 150, and a memory 160. Figure 1 The multiple components shown are not essential for realizing the display device 10 . The display device 10 described in this specification may have more or fewer components than the components listed above.

[0054] First, the interface 110 will be described. The interface 110 can be connected to an interface unit (not shown, hereinafter referred to as the vehicle interface unit) of the vehicle 20 and can receive various information provided by the vehicle 20 through the vehicle interface unit. The vehicle interface unit can serve as a channel for various external devices connected to the vehicle 20 or various components of the vehicle 20. For example, the vehicle interface unit can have various ports connected to the interface 110, allowing connection to the interface 110 through these ports. Furthermore, data can be exchanged with the interface 110.

[0055] The interface 110 can be connected to various components of the vehicle 20 through the vehicle interface unit. As an example, the interface 110 can be connected to the camera 21 of the vehicle 20 and receive an image sensed by the camera 21. Hereinafter, the image sensed by the camera 21 is referred to as a "sensing image."

[0056] In addition, the interface 110 is connected to a user input unit 22 provided in the vehicle 20 and receives user input applied through the user input unit 22. The user input unit 22 may include: a voice input unit capable of receiving a user's voice input; a gesture sensing unit capable of sensing a user's gesture; a touch input unit formed to include a touch sensor; and a mechanical input unit.

[0057] Furthermore, the interface 110 can receive, through the vehicle interface unit, a sensed value from at least one sensor provided in the vehicle 20. The at least one sensor can include a sensor for detecting the position of the vehicle 20 (e.g., a position sensor such as GPS) and at least one state sensor for detecting the state of the vehicle 20 (e.g., a speed sensor, an inclination sensor, etc.). Furthermore, the interface 110 can include a route guidance device (e.g., navigation) for detecting the driving path of the vehicle 20, and a biometric information detection sensor for detecting biometric information of occupants such as the driver and passengers in the cabin of the vehicle 20.

[0058] In addition, the communication unit 120 can perform wireless communication between the display device 10 and a preset server. To this end, the communication unit 120 can include a transmitting antenna, a receiving antenna, and at least one of an RF (Radio Frequency) circuit and an RF element capable of implementing various communication protocols.

[0059] The communication unit 120 may include at least one of a communication module for short-range communication and a communication module for V2X communication. As an example, the communication module for short-range communication (short-range communication module) may be formed using Bluetooth. TM ), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies to support short-range communication, and a communication connection can be established with at least one external device connected to the vehicle 20 or at least one component provided in the vehicle 20 through the short-range communication module.

[0060] On the other hand, the communication module for V2X communication (V2X communication module) can perform wireless communication with a server (V2I: Vehicle to Infrastructure), another vehicle (V2V: Vehicle to Vehicle), or a pedestrian (V2P: Vehicle to Pedestrian). To this end, the V2X communication module may include at least one RF circuit that can implement communication protocols with infrastructure (V2I), vehicle-to-vehicle communication (V2V), and communication with pedestrians (V2P).

[0061] The communication unit 120 may be provided not only on the display device 10 but also on the vehicle 20. In this case, the display device 10 may be connected to the communication unit 120 of the vehicle 20 via the interface 110, and the processor 100 may control the communication unit 120 via the interface 110.

[0062] Furthermore, the object recognition unit 130, under the control of the processor 100, identifies each object within an image sensed by the camera 21 (hereinafter, the sensed image) and classifies the object according to a preset category. The object recognition unit 130 may identify each object in the sensed image based on information for identifying and classifying the objects according to each preset category (hereinafter, the classification information). The plurality of pieces of classification information may be formed into a database by relevant category and stored in the memory 160 of the display device 10 according to an embodiment of the present invention.

[0063] On the other hand, the renderer can render a sensed image including at least one simplified object on the display unit 150 under the control of the processor 100. In addition, the display unit 150 can output the image rendered by the renderer 140. The renderer 140 can render an image on the display unit 150 under the control of the processor 100, in which at least one of the objects included in the sensed image, that is, the objects in the real world (hereinafter, the real objects) displayed in reality, is deleted, or an image including a simplified graphic object or an augmented reality object replacing at least one of the real objects. Therefore, the display unit 150 can output an image in which a portion of the real object included in the sensed image is deleted, or an image including a simplified object or an augmented reality object such as a polygonal shape replacing the portion of the real object.

[0064] The display unit 150 may be a display such as a touch screen provided in the vehicle 20. In this case, the display unit 150 may be connected via the interface 110, and the processor 100 may control the display unit 150 via the interface 110 to output the image rendered by the renderer 140.

[0065] On the other hand, the memory 160 may store data supporting various functions of the display device 10. The memory 160 may store a plurality of application programs (applications) executable by the processor 100, data for the operation of the display device 10, and instructions.

[0066] As an example, the memory 160 may be configured as a database including the classification information (hereinafter referred to as the classification information database 161). The classification information database 161 may include classification information for classifying objects included in the sensed image into respective categories. In this case, the classification information may be feature information for classifying the image objects into specific categories.

[0067] In addition, the classification information may be downloaded from a preset external server to provide the classification information. In this case, the processor 100 may update the classification information database 161 according to the classification information corresponding to at least a portion of the categories downloaded from the external server.

[0068] The processor 100 controls the connected components and, in general, controls the overall operation of the display device 10. First, the processor 100 may receive an image sensed by the camera 21 of the vehicle 20 via the interface 110. Furthermore, the processor 100 may input the received sensed image to the object recognition unit 130 and control the object recognition unit 130 so that the object recognition unit 130 can recognize and classify each object included in the sensed image according to the category defined in the classification information database 161.

[0069] Furthermore, if the object recognition unit 130 is used to identify and classify each object in the sensed image according to a category defined in the classification information database 161, the processor 100 may determine whether a preset condition is satisfied or whether a specific category selected by the user exists. Furthermore, if the determination result is that the condition is satisfied or the specific category selected by the user exists, the processor 100 may detect whether the category corresponding to each object in the sensed image includes the satisfied condition or the specific category selected by the user. Furthermore, if the category corresponding to each object in the sensed image includes the satisfied condition or the specific category selected by the user, the objects in the sensed image corresponding to the specific category or objects other than those corresponding to the specific category may be simplified.

[0070] The simplification may mean deleting at least one object from the sensed image. Alternatively, the simplification may mean replacing at least one object included in the sensed image with a simplified object such as a polygonal object or with an augmented reality object. Figures 5 to 10b , a more detailed action process of the processor 100 for simplifying the object in the sensing image is described in detail.

[0071] If at least one object included in the sensed image is simplified, the processor 100 arranges the simplified object and the unsimplified real object to form a final output image. Furthermore, the processor 100 controls the renderer 140 and the display unit 150 to render the constructed final output image on the display unit 150. Thus, the display unit 150 can output an image of the vehicle's surroundings, including the real object and objects that have been deleted or replaced with simplified objects or augmented reality objects.

[0072] On the other hand, in order to replace the real object in the sensed image with the augmented reality object as described above, the display device 10 according to the embodiment of the present invention can be linked with the cloud server 30 that provides augmented reality services.

[0073] Figure 2 1 is a conceptual diagram showing a structure in which the display device 10 according to the embodiment of the present invention described above cooperates with a cloud server to provide augmented reality information.

[0074] Reference Figure 2 , the display device (Onboard) 10 of the embodiment of the present invention may have an AR engine for rendering and outputting AR images. In addition, the AR engine may include an adapter that receives images sensed by the camera 21, user gesture information or touch event information sensed by the user input unit 22 of the vehicle 20, and sensing information sensed in ADAS (Advanced Driver Assistance Systems), navigation, environmental sensors, biometric information sensors, etc. The adapter may be a corresponding Figure 1 The structure of the interface 110 in the component.

[0075] Among them, such as Figure 1 As described above, the image sensed by the camera 21 is received through the interface 110, and object recognition can be implemented in the object recognition unit 130 under the control of the processor 100. In addition, the object recognition result of the object recognition unit 130 can be input to the processor 100. Or, as Figure 2 As shown, the image sensed by the camera 21 can be directly input into the object recognition unit 130 to realize object recognition. The object recognition result of the object recognition unit 130 can be sent to the AR engine through the interface 110, i.e., the adapter. The object recognition result can be the result of classifying each object included in the sensed image according to any category defined in the classification information database 161 based on the classification information in the classification information database 161.

[0076] Furthermore, the classification manager 210 can receive user input detected by the user input unit 22 of the vehicle 20 via an adapter. Furthermore, based on the received user input, at least one category in the classification information database 161 can be selected for simplification. Furthermore, based on the object recognition results received via the adapter, objects corresponding to the user-selected category can be detected from the categories corresponding to the objects included in the sensed image. Furthermore, the detected objects can be simplified by deleting them or simplifying them into graphic objects such as polygons.

[0077] Alternatively, the classification manager 210 may perform the simplification using an augmented reality graphic object (hereinafter, an augmented reality object) corresponding to the detected object. To this end, the AR engine may include an AR service client (Service Client) that provides a wireless connection interface with the cloud server 30 providing the augmented reality service. The AR service client may be a component corresponding to the communication unit 120 that supports wireless communication with the cloud server 30.

[0078] The cloud server 30 may include: an AR service unit 31 that provides POI information including augmented reality objects in response to a request from the AR service client; and a data provider (External Data Providers) 32 that includes POI information and geographic information to be provided by the AR service unit 31.

[0079] In this case, the AR service unit 31 may receive a request for POI information around the vehicle 20, including the location information of the vehicle 20, from the AR service client. Consequently, the AR service unit 31 may request POI information within an area corresponding to the received location information from the data provider 32. Furthermore, if POI information including augmented reality objects is received from the data provider 32 in response to the request, the AR service unit 31 may provide the received POI information to the AR service client. Consequently, the augmented reality objects corresponding to the objects in the sensed image corresponding to the category selected by the user may be transmitted to the category manager 210 via the AR service client.

[0080] On the other hand, if the classification manager 210 is used to determine whether to delete the object in the sensing image corresponding to the category selected by the user or replace it with a simplified object such as a polygon or an augmented reality object (simplification), the image composition unit (Scene manager) 220 can form an image after deleting the object in the sensing image corresponding to the category selected by the user or replacing it with the simplified object or augmented reality object. That is, the image composition unit 220 determines the area in the sensing image where each object is to be configured, and can initialize (delete) the area occupied by the detected object, that is, the object in the sensing image corresponding to the category selected by the user in the determined area, or configure a simplified object or an augmented reality object received from the cloud server 30 in the area occupied by the detected object. The classification manager 210 and the image composition unit 220 can be the same as the Figure 1 The configuration corresponding to the processor 100.

[0081] In addition, the AR renderer can receive image composition information of objects arranged in each area of ​​the sensed image from the image composition unit 220. In addition, an image based on the received image composition information can be rendered on the display unit 150. Therefore, an image in which at least one object in the sensed image corresponding to the category selected by the user is deleted or replaced with a simplified object or an augmented reality object can be output through the display unit 150. Figure 1 The renderer 140 corresponds to the composition.

[0082] On the other hand, the POI information provided from the cloud server 30 may be information categorized by a specified category. Furthermore, the POI information in the cloud server 30 may be categorized into the same categories as those in the categorized information database 161 of the display device 10 according to this embodiment of the present invention. That is, the categorized information database 161 may include categorized information categorized according to the categories corresponding to the categorized POI information in the cloud server 30. Therefore, the display device 10 can request the cloud server 30 only for POI information in a specific category around the current location of the vehicle 20. For example, POI information that meets preset conditions or corresponds to a category selected by the user can be received in response to the request.

[0083] On the other hand, the above description has described in detail the configuration of the AR engine connected to the display device 10 and the cloud server 30 of the embodiment of the present invention. The following describes the operation process of the display device 10 outputting a simplified image of at least one object corresponding to the category selected by the user on the display unit 150. In addition, for the sake of convenience, the following is based on the above Figure 1 The configuration of the display device 10 described in will be described.

[0084] on the other hand, Figure 3 1 is a flowchart showing an operation process of the display device 10 according to an embodiment of the present invention outputting an image including at least one simplified object according to a condition or a category selected by the user. Figure 4 This is an exemplary diagram showing an example in which the display device according to the embodiment of the present invention receives category settings from the user.

[0085] First, refer to Figure 3 The display device 10 of the embodiment of the present invention may receive an image sensed by the camera 21 of the vehicle 20 (hereinafter referred to as a sensed image) (S300). The camera 21 may be a front camera that senses an image in front of the vehicle 20. Alternatively, the camera 21 may be a rear or side camera that senses the rear or side of the vehicle 20.

[0086] Thus, the processor 100 of the display device 10 may control the object recognition unit 130 to identify the category corresponding to each object in the sensed image (S302). In order to identify the category corresponding to each object, the object recognition unit 130 may use the classification information database 161 including classification information corresponding to each predefined category.

[0087] As an example, in step S302, the object recognition unit 130 may distinguish each object included in the sensed image. Furthermore, under the control of the processor 100, features of each distinguished object in the sensed image may be detected. Furthermore, classification information corresponding to the features detected from each object may be detected from the classification information database 161. Furthermore, based on the category corresponding to the detected classification information, the category corresponding to each object may be identified. Thus, through step S302, each object included in the sensed image may be distinguished, and the category corresponding to each distinguished object may be identified from the classification information database 161.

[0088] Thus, the object recognition unit 130 can provide information on the recognized categories corresponding to the respective objects in the sensed image as object recognition results to the processor 100. Furthermore, the processor 100 can detect, based on the received object recognition results, objects in the sensed image that correspond to categories based on whether a preset condition is satisfied or based on a category selected by the user (S304).

[0089] As an example, the processor 100 may determine whether a preset condition corresponding to at least one category is satisfied in step S304. For example, the preset condition may be illumination, time, or weather. Furthermore, if the illumination, time, or weather (e.g., rain, snow, or other weather conditions) detected by the sensor unit 23 of the vehicle 20 satisfies the preset condition, the processor 100 may detect objects in the sensed image that correspond to at least one category corresponding to the condition.

[0090] Alternatively, the processor 100 may determine whether a category pre-selected by the user exists for simplification in step S304. To this end, the processor 100 may control the display unit 150 to display a menu screen for selecting a category, and determine at least one category based on user input applied through the menu screen as the category selected by the user for simplification. The category may be a category pre-defined in the classification information database 161 for classifying objects.

[0091] Figure 4 1 is an exemplary diagram showing an example of at least one category selected for simplification by a user through the menu screen of the display device 10 according to the embodiment of the present invention.

[0092] Reference Figure 4 In (a), the processor 100 of the display device 10 may display a menu screen 410 on the display unit 150 from which at least one category option may be selected according to a user's selection. The category option may be an option corresponding to each category predefined in the classification information database 161 .

[0093] In such a state, if Figure 4 As shown in (a) of FIG. 1 , the user may input an option 411 corresponding to the category “building”. Thus, the processor 100 may distinguish and display the specific category option selected according to the user's input from other category options that are not selected. Figure 4 As shown in (a), if the category selection is ended after the user selects the "building" category, the processor 100 may detect all objects classified into the "building" category among the objects included in the sensed image in step S304.

[0094] On the other hand, the processor 100 of the display device 10 according to the embodiment of the present invention can also allow the user to select a more detailed category. Figure 4 As shown in (a), when the user selects the "building" category, the processor 100 can select the building according to the user's selection, such as Figure 4 As shown in (b) of FIG. 4 , a menu screen 420 including options corresponding to the currently selected subcategory of “Building” is further displayed.

[0095] In this case, the subcategories may be categories differentiated according to the main purpose of the building, for example, according to the services that can be provided. Figure 4 As shown in (b), the subcategory may be a category where specific services such as hospitals, restaurants, cafes, banks, etc. can be selected.

[0096] In addition, as stated Figure 4 As shown in (b) of FIG. 4 , if the user selects the "cafe" category from menu screen 420 including subcategory options and then ends category selection, the processor 100 may detect, in step S304, all objects included in the sensed image that are classified as "building" and classified as "cafe" among the objects included in the sensed image. In other words, if the user further selects a subcategory corresponding to a specific category, the processor 100 may detect only objects corresponding to the subcategory among the objects included in the sensed image.

[0097] On the other hand, the display device 10 of the embodiment of the present invention can also allow the user to select a more detailed category. For example, the processor 100 of the display device 10 can be as described above. Figure 4As shown in (b) of FIG. 1 , when a subcategory ("cafe") (hereinafter, a first-level subcategory) of a specific category ("building") is selected, a menu screen 430 is further provided, which allows the user to select a second-level subcategory based on a classification standard (hereinafter, a second-level classification standard) different from the classification standard of the subcategory. In this case, the second-level subcategory selection menu screen 430 can be a menu screen that allows the user to select a category corresponding to the first-level subcategory (e.g., cafe) from the categories in the classification information database 161 based on the second-level classification standard.

[0098] As an example, the secondary subcategory may be a specific store or business operator. In this case, the secondary subcategory selection menu 430 may be as follows: Figure 4 As shown in (c), the menu screen includes the merchants corresponding to the first-level subcategory (cafe).

[0099] In this case, as described Figure 4 As shown in (c), the user can select any category ("starfall"). Thus, in step S304, the processor 100 can detect only the following objects from the objects included in the sensed image: objects whose classification category is not only "building," but also objects classified as "building" that belong to the "cafe" category and whose business name is classified as "starfall." That is, if the user further selects at least one subcategory corresponding to a specific category, the processor 100 can detect only objects in the sensed image that correspond to all of the at least one subcategory.

[0100] On the other hand, although Figure 4 While the user-selectable subcategories are described using first-level and second-level subcategories as examples, this is merely an example of an embodiment of the present invention and is not limited thereto. Specifically, the user can select any number of subcategories. Furthermore, the greater the number of subcategories selected, the fewer objects in the sensed image that the processor 100 detects as corresponding to the user-selected category.

[0101] In addition, in the above description, although the user selects a category option defined by the classification information database 161 through a menu screen and detects at least one object in the sensed image through the selected menu option is used as an example, differently from this, a specific object or a specific category can also be selected based on the user input applied to the display unit 150.

[0102] For example, a user may touch an area on the touch screen displaying a specific object while a sensed image is displayed. Thus, the processor 100 may detect any object displayed in the touched area as an object selected by the user. Alternatively, the processor 100 may detect a category in the classification information database 161 corresponding to any object displayed in the touched area, and determine the detected category as the category selected by the user for simplification. In this case, the processor 100 further detects at least one subcategory corresponding to any object input according to the touch input, and determines at least one category including the detected subcategory as the category selected by the user for simplification.

[0103] On the other hand, in step S304, if an object corresponding to at least one category based on satisfying a preset condition or at least one category selected by the user is detected from the objects included in the sensed image, the processor 100 may perform simplification for the detected object (S306).

[0104] The simplification may include deleting at least one object detected in the step S304. In this case, the processor 100 may detect an area showing the object detected in the step S304 from the sensed image and initialize pixel information of the detected area. In addition, color information and pattern information are detected from an area surrounding the area where the pixel information is initialized, and a filling image using the detected color and pattern is generated. Figures 5 and 6 , the action process of step S306 for deleting the object detected in step S304 is described in detail.

[0105] On the other hand, the simplification may be a process of replacing at least one object detected in step S304 with a simplified object such as a polygon. To this end, the processor 100 may generate a simplified object for the at least one object detected, and may detect an area in the sensed image where the object detected in step S304 is displayed. In addition, the generated simplified object may be configured in an area in the detected sensed image. Figures 7 and 8 , the action process of step S306 in which the object detected in step S304 is replaced with a simplified object is described in detail.

[0106] In addition, the simplification may be a process of replacing at least one object detected in the step S304 with an augmented reality object provided from the cloud server 30. Figure 10a 、 Figure 10b as well as Figure 11, the action process of step S304 of detecting an object in a sensed image corresponding to a category selected by a user based on POI information received from the cloud server 30 and step S306 of replacing at least one object detected in step S206 with an augmented reality object included in the POI information is described in detail.

[0107] If at least one of the objects in the captured image is deleted or replaced with a simplified object or an augmented reality object in step S306, the processor 100 may determine the position of the object in the image from which the simplified object or augmented reality object was deleted or replaced, and construct a final output image (S308). Furthermore, if the final output image including the object deleted or replaced with the simplified object or augmented reality object is completed, the processor 100 may control the renderer 140 to render the constructed final output image on the display unit 150 (S310). Thus, an image in which at least one of the objects in the sensed image is deleted or replaced with a simplified object or augmented reality object can be output via the display unit 150.

[0108] Figure 5 The display device 10 of the embodiment of the present invention deletes the object corresponding to the specific category in detail. Figure 3 FIG S306 is a simplified diagram of the operation process. In addition, Figure 6 It is shown that according to the Figure 5 An example diagram of a process of generating an image from which real objects corresponding to a specific category are deleted.

[0109] First, refer to Figure 5 , if in the Figure 3 If, in step S304, an object corresponding to a specific category selected by the user or satisfying a preset condition is detected in the sensed image, the processor 100 may detect at least one region in the sensed image corresponding to each of the detected objects (S500). Furthermore, pixel information, such as color information, of the region detected in step S500 may be initialized. Thus, the image of the region detected in step S500 may be deleted.

[0110] Reference Figure 6 , Figure 6 (a) shows an example of a sensed image 600, Figure 6 (b) shows an example of dividing the sensed image 600 into a plurality of regions according to respective objects included in the sensed image 600 .

[0111] As an example, the processor 100 may distinguish the objects included in the sensed image 600 into different categories based on the classification information included in the classification information database 161. In this case, Figure 6As shown in (a) of FIG. 5 , when the sensed image 600 includes a plurality of buildings 601 to 604, a vehicle 608, a cloud 605, a road 606, and a lane 607, the processor 100 may distinguish each of the plurality of buildings 601 to 604, the vehicle 608, the cloud 605, the road 606, and the lane 607 as different objects. Furthermore, each region within the sensed image 600 in which the respective distinguished objects are displayed may be detected.

[0112] In this state, the specific building 601 may be an object that satisfies a preset condition or corresponds to a category selected by the user. Thus, the processor 100 may initialize pixel information of an area in the sensed image 600 that shows an object that satisfies a preset condition or corresponds to a category selected by the user, i.e., the specific building 601. Figure 6 As shown in (c), the area corresponding to the specific building 601 in the sensed image 600 can be deleted.

[0113] The pixel information may represent color information set for each pixel. Furthermore, initializing the pixel information may involve changing the color information set for each pixel to a preset initial value. Therefore, if the initial value is white, then if the area corresponding to the specific building 601 in the sensed image 600 is deleted, the area corresponding to the specific building 601 in the sensed image 600 may be displayed in white. The initial value color information may be a color selected by the user.

[0114] If a specific object corresponding to a category that meets a preset condition or is selected by the user is deleted, the processor 100 may detect a surrounding area of ​​the area in the sensed image 600 whose pixel information was initialized due to the deletion. Furthermore, color information and information (pattern information) of a color pattern formed by the color of the surrounding area may be detected from the detected surrounding area (S504). Furthermore, a fill image corresponding to the area in the sensed image 600 whose pixel information was initialized may be generated based on the detected color information and pattern information (S506). Furthermore, the generated fill image may be placed in the area in the sensed image 600 whose pixel information was initialized (S508).

[0115] Therefore, if Figure 6As shown in (c), if the area corresponding to the specific building 601 in the sensed image 600 is deleted, the processor 100 can detect color information and color pattern information from the area around the deleted specific building 601. In this case, as the area around the specific building 601, the sky around the specific building 601 including the surrounding buildings 602 and clouds 605 adjacent to the specific building 601 can be detected. Thus, the processor 100 can generate a filled image with a color and pattern similar to the detected surrounding area, that is, the sky around the specific building 601 including the adjacent surrounding buildings 602 and clouds 605. In addition, as Figure 6 As shown in (d), the filling image can be configured in the area corresponding to the deleted specific building 601.

[0116] Therefore, as stated Figure 6 As shown in (d), a fill image including a cloud image 610 and a pattern 611 similar in appearance to adjacent surrounding buildings can be configured in an area corresponding to the deleted specific building 601 in the sensed image 600. Furthermore, the sensed image 600 configured with the fill image can be rendered on the display unit 150 by the renderer 140. Thus, the display unit 150 can display an image in which at least one object corresponding to a category that satisfies a preset condition or a user selection is naturally eliminated from the sensed image 600.

[0117] on the other hand, Figure 7 The display device 10 of the embodiment of the present invention is used to replace the object corresponding to the specific category with a simplified object. Figure 3 FIG S306 is a simplified diagram of the operation process. In addition, Figure 8 It is shown that according to the Figure 7 An example diagram of the process of generating an image in which a real object corresponding to a specific category is replaced with an image of a simplified object.

[0118] First, refer to Figure 7 , if in the Figure 3 In step S304, if the objects in the sensed image are detected as corresponding to a specific category selected by the user or satisfying a preset condition, the processor 100 of the display device 10 may detect each object included in the sensed image and generate simplified objects corresponding to each of the detected objects (S700).

[0119] The simplified object may be a polygonal graphic model (hereinafter referred to as a polygonal model) capable of representing the shape of an object. In other words, the simplified object may be a polygonal virtual object without a texture.

[0120] The step S700 of generating the polygonal model may be a step of transmitting the location information of the vehicle 20 to a preset server to request POI information around the location of the vehicle 20, and receiving polygonal models corresponding to various objects in the sensed image from the preset server in response to the request.

[0121] The polygonal models corresponding to the various objects in the sensed image may include POI information for buildings or areas surrounding the location of vehicle 20. To this end, the pre-set server providing the POI information may be a server that includes a POI database containing polygonal models of buildings or areas surrounding vehicle 20 corresponding to the location information of vehicle 20. In this case, the pre-set server may be a cloud server 30, or a server including a DTaaS (Digital Twin as a Service) server. Alternatively, the cloud server 30 may be a server in communication with the DTaaS server.

[0122] On the other hand, if the polygonal model is generated in step S700 or provided from the cloud server 30, the processor 100 may detect the polygonal model in step S700. Figure 3 The area in the sensed image occupied by an object selected by the user or corresponding to a specific category that meets preset conditions among the objects in the sensed image in step S304 (S702) can be initialized (S704). Furthermore, pixel information, such as color information, of the detected area can be initialized. This allows the area in the sensed image corresponding to the object detected in step S304 to be deleted.

[0123] In addition, the processor 100 may delete the Figure 3 The simplified object generated or received in the step S700 is displayed on each area of ​​the object detected in the step S304 (S706). Figure 3 In step S308 , a portion of the object is replaced with a sensed image of a simplified object.

[0124] Figure 8 It is shown that according to the Figure 7 An example diagram of an example of an action process in which a real object corresponding to a specific category is replaced with an image of a simplified object.

[0125] For example, the processor 100 of the display device 10 may detect only objects corresponding to specific categories based on user selection or satisfying preset conditions among the objects included in the sensed image as objects to be simplified, or, conversely, detect objects of other categories other than specific categories based on user selection or satisfying preset conditions as objects to be simplified. Figure 8 This illustrates an example in which objects in the sensed image 800 , except for objects corresponding to a specific category selected by the user or meeting a preset condition, are all detected as objects to be simplified.

[0126] First, refer to Figure 8 (a), Figure 8 (a) shows an example of a case where the categories of objects in the sensed image 800 selected based on a user's selection or satisfying a preset condition are vehicles 810 and roads 811. In this case, all objects corresponding to categories other than vehicles 810 and roads 811, such as buildings 820, can be selected as objects to be simplified.

[0127] Thus, the processor 100 can generate simplified objects corresponding to the objects to be simplified. In this case, the simplified objects can be polygonal models corresponding to the shapes of the objects to be simplified. To this end, the processor 100 can detect the shapes of the objects to be simplified from the sensed image 800 using the outline of the objects to be simplified, and then generate polygonal models corresponding to the detected shapes.

[0128] Alternatively, the processor 100 may transmit a POI information request including location information for the current location of the vehicle 20 to the cloud server 30. In response to the POI information request, the cloud server 30 may provide POI information for POIs near the location of the vehicle 20. The processor 100 may detect the category of objects detected as objects to be simplified from the various objects included in the sensed image 800 and request the cloud server 30 for POI information corresponding to the detected category. The cloud server 30 then detects only POI information corresponding to the requested category from the POI information for POIs near the vehicle 20 and transmits the information in response to the request.

[0129] On the other hand, the POI information provided from the cloud server 30 may include simplified objects of the objects, such as information of polygonal models. Therefore, the processor 100 may obtain simplified objects corresponding to the objects detected as objects to be simplified.

[0130] Thus, the processor 100 can detect, from the sensed image 800, the areas occupied by the objects detected as objects to be simplified. Alternatively, the detected areas can be deleted. Furthermore, simplified objects corresponding to the deleted objects can be allocated to the areas of the deleted objects on the sensed image 800. Furthermore, the renderer 140 can be controlled to render the sensed image 800 on the display unit 150, with a portion of the objects replaced by simplified objects.

[0131] Therefore, if Figure 8 As shown in (b), the object 810 corresponding to the category "vehicle" and the object 811 corresponding to the category "road" are displayed as they are sensed, whereas other objects, such as the building object 820, can be replaced with simplified objects.

[0132] On the other hand, according to the above description, the display device 10 of the embodiment of the present invention can detect the user's selection and the object corresponding to the category that meets the preset conditions from the various objects in the sensed image. In addition, the preset conditions can be illumination, time, or weather.

[0133] In this case, the preset condition may be preset to select a specific category when the preset condition is met. That is, when the specific condition is met, the specific category matching the specific condition may be selected as the category for detecting the object to be simplified.

[0134] Figure 9 For illustrating the display device 10 according to an embodiment of the present invention configured to automatically select a specific category based on a preset condition, it is assumed that the condition is a preset illumination and the category matching the condition is “building”.

[0135] First, refer to Figure 9 In (a), the processor 100 may receive an image (sensed image 900) sensed by the camera 21 while the vehicle 20 is traveling. Furthermore, based on the information detected by the sensor unit 23 of the vehicle 20, it may be determined whether a preset condition matching a specific category is satisfied.

[0136] In this case, the preset condition may be illumination. Thus, processor 100 can determine whether the illumination around vehicle 20 reaches the preset illumination level based on the illumination detected by the illumination sensor of vehicle 20. Furthermore, if the illumination around vehicle 20 reaches the preset illumination level, processor 100 can determine that the preset condition matching the specific category is satisfied.

[0137] On the other hand, if the preset condition is satisfied, the processor 100 may select a specific category matching the condition as a category for simplifying the sensing of objects within the image 900. In this case, if the category matching the preset illumination condition is "building," the processor 100 may select the specific category matching the condition as a category for simplifying the sensing of objects within the image 900. Figure 9 As shown in (b) of FIG. 8 , an object 910 corresponding to the class “building” that matches the satisfied condition is detected from the sensed image 900 .

[0138] Thus, the processor 100 may perform simplification on the detected object 910. For example, the processor 100 may delete the detected object 910 from the sensed image 900. Figure 9 As shown in (c), an image in which an object 910 corresponding to a building among the objects in the sensed image 900 is deleted may be outputted through the display unit 150 .

[0139] Alternatively, the processor 100 may generate or receive simplified objects corresponding to the detected objects 910 from the cloud server 30 through POI information, and replace the objects 910 corresponding to the buildings in the sensed image 900 with the generated or received simplified objects. Figure 9 As shown in (d), an image in which the object 910 corresponding to the building among the objects in the sensed image 900 is replaced with a simplified object such as a polygonal model may be outputted through the display unit 150 .

[0140] On the other hand, according to the above description, the display device 10 according to the embodiment of the present invention can also use the augmented reality object received through the POI information as a simplified object.

[0141] Figure 10a and Figure 10b The present invention is a detailed description of the display device of the embodiment of the present invention, which replaces the real object corresponding to the specific category with the AR information corresponding to the specific category. Figure 3 Flowchart of step S304 and step S306. Figure 11 It is shown that according to the Figure 10a and Figure 10b An example diagram of an action process in which the real object corresponding to a specific category is replaced by AR information.

[0142] First, in the Figure 3 In step S302, the processor 100 of the display device 10 may determine the categories of the objects included in the image sensed by the camera 21 according to the categories specified in the classification information database 161. In addition, if the detection of the objects corresponding to the category selected by the user or meeting the preset conditions is performed Figure 3 In step S304, Figure 10aAs shown, information about the category selected by the user or satisfying a preset condition (hereinafter, the selected category) may be transmitted to the cloud server 30 along with the location information of the vehicle 20 (S1000). That is, the processor 100 may request the cloud server 30 for information about POIs (POI information) corresponding to the selected category in the vicinity of the current location of the vehicle 20.

[0143] In addition, if POI information surrounding the vehicle 20 corresponding to the selected category is received in response to the request (S1002), the processor 100 may detect objects corresponding to the category selected by the user or satisfying a preset condition based on each category determined for the objects included in the sensed image (S1004). On the other hand, since the received POI information is received from the cloud server 30 based on the category selected by the user or satisfying the preset condition, the object detection in step S1004 may be a process of detecting objects corresponding to the category of the received POI information from the objects included in the sensed image.

[0144] On the other hand, if the Figure 10a As shown Figure 3 If at least one object among the objects included in the sensed image is detected in step S304, the processor 100 may move to Figure 3 In step S306, simplification of the object detected in step S304 is performed. Figure 10b is shown in detail in such a case Figure 3 FIG. 1 is a diagram of the simplified process performed in step S306 of FIG.

[0145] Reference Figure 10b , the processor 100 of the display device 10 of the embodiment of the present invention can first extract the augmented reality object corresponding to the category selected by the user or meeting the preset conditions from the POI information received from the cloud server 30 (S1050). Figure 10a The process of detecting an area occupied by an object corresponding to a specific category selected by a user or meeting a preset condition from among the objects detected in the sensed image (S1052) can be performed. Furthermore, pixel information, such as color information, of the detected area can be initialized (S1054). Thus, the area in the sensed image detected in step S1052 can be deleted.

[0146] In addition, the processor 100 may display the augmented reality object extracted in step S1050 in each area of ​​the sensed image deleted in step S1054 (S1506). Thus, the renderer 140 may render the augmented reality object on the display unit 150. Figure 3 In step S308 , a portion of the object is replaced with a sensed image of the augmented reality object.

[0147] Figure 11 It is shown that according to the Figure 10a and Figure 10b The process described in Figure 3 Steps S304 and S306 replace at least one object included in the sensed image with an augmented reality object.

[0148] Reference Figure 11 , Figure 11 (a) shows an example of a sensed image 1100 including buildings 1110, 1111, 1112, vehicles, and roads. In this case, the processor 100 can identify and classify the objects in the sensed image 1100 into objects 1110, 1111, 1112 corresponding to the building category, objects corresponding to the vehicle category, and objects corresponding to the road category based on the classification information database.

[0149] Alternatively, the processor 100 may request information about POIs surrounding the current vehicle 20 from the cloud server 30, including information about a category selected by the user or that meets preset conditions and information about the current location of the vehicle 20. The cloud server 30 then detects information about POIs corresponding to the category included in the received request from the POIs surrounding the vehicle 20 and, in response to the received request, provides information about augmented reality objects corresponding to the detected POIs, i.e., POI information. Thus, the processor 100 can obtain POI information, including augmented reality objects, for POIs surrounding the current vehicle 20 corresponding to the category selected by the user or that meets preset conditions.

[0150] In this state, the first building among the buildings around the vehicle 20 may correspond to the category selected by the user or meeting the preset conditions. Figure 11 The object 1110 corresponding to the first building is detected from the sensed image 1100 shown in (a). In addition, by initializing an area in the sensed image 1100 corresponding to the detected object 1110 of the first building, the object 1110 corresponding to the first building can be deleted from the sensed image 1100.

[0151] Thus, the processor 100 can extract the augmented reality object 1120 corresponding to the first building from the POI information received from the cloud server 30. In addition, the extracted augmented reality object can be displayed on an area corresponding to the deleted object 1110 in the sensed image 1100. Figure 11As shown in (b) of FIG. 1 , an image 1101 including an augmented reality object 1120 instead of an object 1110 corresponding to a specific category may be output on the display unit 150 .

[0152] Alternatively, the processor 100 may detect color or pattern information of the surrounding area of ​​the area corresponding to the deleted object 1110 in the sensed image 1100, and display a fill image generated based on the detected color or pattern information. Furthermore, the augmented reality object may be displayed overlapping the area where the fill image is displayed. Thus, the area where the object corresponding to the first building has been deleted can be displayed more naturally.

[0153] On the other hand, in the above description, only the external environment of the vehicle, such as illumination or weather, is used as an example of a preset condition. However, differently from this, the user's physical condition can also be set as the preset condition. For example, when the user's physical condition is an emergency such as an illness, the processor 100 can automatically select a category such as a hospital based on the detected physical condition of the user. Alternatively, when the result of detecting the user's physical condition is that the user has severe heatstroke (the temperature inside the vehicle is above the specified temperature), or the detection result of the DMS (Driver Monitoring System) determines that the user is dehydrated, the category can be automatically set to a place where water can be supplied, that is, a restaurant (cafe, etc.) where drinks can be taken.

[0154] Figure 12 1 is a flowchart illustrating an operation process in which the display device 10 according to an embodiment of the present invention automatically selects a specific category according to the detected physical state of the user and detects an object corresponding to the selected category.

[0155] exist Figure 3 In step S302, the processor 100 of the display device 10 may determine the categories of the objects included in the image sensed by the camera 21 according to the categories specified by the classification information database 161. Figure 12 As shown, the processor 100 of the display apparatus 10 may receive sensed biometric information of a user from at least one device capable of sensing biometric information of a user ( S1200 ).

[0156] At least one of the devices may be a wearable device such as a smartwatch worn by a user. Alternatively, it may be a sensor of a DMS in vehicle 20. As an example, the DMS sensor may be an internal camera that captures images for detecting the status of a driver or passenger in the cabin of vehicle 20. The "user" may be any person in vehicle 20.

[0157] In addition, the processor 100 may determine whether the received user's biometric information satisfies preset conditions for detecting the object (S1202). Specifically, the processor 100 may determine the user's biological state based on the received user's biometric information, and determine whether the determined user's biological state satisfies preset conditions corresponding to a specific category. The preset conditions may be the biological states of different users, and different biological state conditions may be matched to different categories (e.g., dehydration or thirst is matched to a cafe or restaurant, and illness is matched to a hospital).

[0158] If the result of the determination in step S1202 is that the received biological state of the user satisfies the preset biological state condition, the processor 100 may determine a specific category that matches the biological state condition. In addition, the information of the determined specific category and the current position information of the vehicle 20 may be transmitted to the cloud server 30 to request POI information around the vehicle 20 corresponding to the specific category (S1204). In addition, if the POI information around the vehicle corresponding to the specific category is received from the cloud server 30, the POI information may be transmitted to the cloud server 30. Figure 3 At least one object among the objects within the sensed image identified in step S302 based on the received POI information is detected as an object corresponding to a category matching the satisfied biological state condition (S1206).

[0159] On the other hand, Figure 12 As shown, if an object corresponding to the POI information received from the cloud server 30 is detected from the sensing image, the processor 100 can replace the detected object with an augmented reality object extracted from the received POI information. In addition, an image in which a part of the object is replaced with the augmented reality object can be output on the display unit 150. That is, the image can be displayed by the processor 100. Figure 10b The image including the augmented reality object extracted from the received POI information is output on the display unit 150 through a process similar to that described in FIG.

[0160] On the other hand, in the above description, an example is described in which at least one of the objects in the sensed image is deleted or replaced with a simplified object or an augmented reality object according to a specific category selected by a user or satisfying a preset condition.

[0161] However, differently from this, the user may also select a specific POI to replace a specific category. In this case, the processor 100 of the display device 10 may delete or replace at least a portion of the sensed image with an augmented reality object according to the selected POI.

[0162] Figure 13a and Figure 13bThe flowchart shows the operation process of the display device according to the embodiment of the present invention detecting at least one object from the sensing image based on the positions of the vehicle and the specific POI, and displaying the augmented reality object of the specific POI to replace the detected object.

[0163] First, in the Figure 3 In step S302, the processor 100 of the display device 10 may determine the categories of the objects included in the image sensed by the camera 21 according to the categories specified in the classification information database 161. Figure 3 In step S304, the processor 100 may Figure 13a As shown, POI information including the location of a specific POI selected by the user is requested from the cloud server 30 , and POI information including the location of the specific POI is received as a response to the request ( S1300 ).

[0164] Thus, the processor 100 can calculate the direction and distance corresponding to the location of the specific POI from the sensed image based on the current location of the vehicle 20 and the location information of the specific POI (S1302). In addition, at least one object located in the direction corresponding to the location of the specific POI can be detected from the sensed image.

[0165] For example, if the FOV (Field Of View) of the sensing image includes the direction in which the vehicle 20 is pointing to the specific POI, then at least one object displayed in the sensing image may be located between the specific POI and the vehicle. Thus, the processor 100 may detect at least one object in the sensing image that is located between the specific POI and the vehicle and blocks the specific POI as an object selected by the user. Thus, the processor 100 may move to Figure 3 The S306 step is for the step S304, i.e. Figure 13a The action process of the detected object is simplified. Figure 13b Detailed description of the case Figure 3 The simplified process is performed in step S306.

[0166] Reference Figure 13b First, the processor 100 of the display device 10 of the embodiment of the present invention can extract the augmented reality object corresponding to the specific POI from the POI information received from the cloud server 30 (S1350). In addition, the processor 100 can detect the augmented reality object corresponding to the specific POI from the sensing image. Figure 13a At least one object detected during the action process of the specific POI is located between the specific POI and the vehicle and blocks the area corresponding to the at least one object in the sensing image of the specific POI (S1352).

[0167] In addition, the processor 100 can delete the pixel information of an area in the sensing image corresponding to the detected area from the sensing image. Figure 13a At least one object is detected during the action process (S1354). In addition, an augmented reality object corresponding to a specific POI extracted from the POI information can be displayed in an area in the sensed image from which the at least one object is deleted (S1356). Figure 3 In step S308, the renderer 140 may render the image on the display unit 150. Figure 13a At least one object detected during the action process is replaced with an image of an augmented reality object corresponding to a specific POI selected by the user.

[0168] In this case, the size of the augmented reality object may be determined based on the distance between the current vehicle 20 and the specific POI. That is, if the distance between the current vehicle 20 and the specific POI is closer, a larger augmented reality object may be displayed, and if the distance is farther, a smaller augmented reality object may be displayed.

[0169] On the other hand, the processor 100 can detect color or pattern information of the surrounding area of ​​an area in the sensed image corresponding to the at least one deleted object and display a fill image generated based on the detected color or pattern information. In addition, the augmented reality object can be displayed overlappingly in the area where the fill image is displayed. Therefore, the augmented reality object corresponding to the specific POI can be displayed in the area where the at least one object was naturally deleted.

[0170] Furthermore, the number of objects to be deleted from the sensed image can be increased or decreased based on the size of the augmented reality object. For example, if the distance between the vehicle 20 and the specific POI decreases, the size of the augmented reality object corresponding to the specific POI increases, and more objects in the sensed image may obstruct the specific POI. Therefore, more objects in the sensed image can be deleted.

[0171] On the contrary, if the size of the augmented reality object corresponding to the specific POI decreases as the distance between the vehicle 20 and the specific POI increases, fewer objects in the sensed image may block the specific POI. Therefore, fewer objects in the sensed image may be deleted.

[0172] Figure 14 It is shown that according to the Figure 13a and Figure 13b An example diagram of an image in which a portion of an object included in a sensed image is replaced with an augmented reality object of a specific POI is generated during the action process.

[0173] Reference Figure 14 In (a), the processor 100 may receive an image (sensed image 1400) sensed by the camera 21 while the vehicle 20 is traveling. Furthermore, in this state, the processor 100 may detect, based on the location of a specific POI selected by the user and the current location of the vehicle, an object in the sensed image that is located in a direction toward the specific POI, i.e., an object in the sensed image that is located between the specific POI and the vehicle and obstructs the specific POI.

[0174] On the other hand, if the specific POI is the Eiffel Tower (in this case, the augmented reality object corresponding to the Eiffel Tower may be in a state where it has been received from the cloud server 30), and the objects blocking the specific POI in the sensed image are the object corresponding to the first building (the first building object 1410) and the vehicle object 1411, the processor 100 may perform the following operations: Figure 14 As shown in (b), the areas corresponding to the first building object 1410 and the vehicle object 1411 in the sensing image are detected.

[0175] In addition, the first building object 1410 and the vehicle object 1411 in the sensed image can be deleted by initializing the pixel information of the detected area. Figure 14 As shown in (b), an augmented reality object 1420 corresponding to the currently selected specific POI, namely, the Eiffel Tower, can be displayed in an area corresponding to the deleted first building object 1410 and vehicle object 1411 within the sensed image 1400. In this case, the size of the augmented reality object 1420 can be determined based on the distance between the vehicle 20 and the Eiffel Tower.

[0176] On the other hand, as mentioned Figure 14 As shown in (b), the user can also pre-set specific categories for deletion restrictions. In this case, even if an object in the sensed image blocks a specific POI, objects in the category that the user has set as deletion restricted will not be deleted. This is to prevent the situation where objects in the sensed image are arbitrarily deleted based on the user-selected POI, resulting in actual objects not being displayed on the display unit 150 (e.g., nearby vehicles), which could lead to accidents and other dangers.

[0177] For example, if the object corresponding to the vehicle category is preset by the user to be restricted from being deleted, then even if the first building object 1410 and the vehicle object 1411 completely block the specific POI, the processor 100 will not delete the vehicle object 1411. Figure 11As shown in (c), only the first building object 1410 may be deleted, and the augmented reality object 1320 corresponding to the specific POI may be displayed in an area of ​​the sensed image corresponding to the deleted first building object 1410.

[0178] Alternatively, and in a different manner from the above, it is also possible to pre-specify specific categories of objects that can be deleted from the sensed image and obstruct a specific POI. In this case, only the objects in the sensed image that obstruct the specific POI and correspond to the specific categories that can be deleted can be deleted. Furthermore, the augmented reality object corresponding to the specific POI can be displayed in the deleted area.

[0179] On the other hand, the processor 100 of the display device 10 according to the embodiment of the present invention may also track the relative position change between the specific object and the vehicle 20 that changes according to the movement of the vehicle 20 through the augmented reality object displayed in the sensing image.

[0180] Figure 15 1 is an exemplary diagram illustrating an example in which the position of a specific object moving in a sensed image is displayed using an augmented reality object corresponding to a specific object selected by a user in the display device 10 according to an embodiment of the present invention.

[0181] First, refer to Figure 15 In step (a), the processor 100 of the display device 10 may accept a selection of at least one object from among the objects displayed in the sensed image. In this case, the selected object may be an object directly selected by the user from among the displayed objects, or an object detected based on a category or POI selected by the user. For ease of explanation, it is assumed that object 1510 corresponding to the sun in the sensed image is the object selected by the user.

[0182] In this case, the processor 100 may detect whether the selected object in the sensed image is obscured by other objects in the sensed image during the movement of the vehicle or the passage of time. Furthermore, if the selected object is at least partially obscured by the other objects, the obscured portion may be displayed as an augmented reality object corresponding to the selected object. Furthermore, the movement of the selected object during the movement of the vehicle or the passage of time may be displayed as movement of the augmented reality object.

[0183] That is, Figure 15 As shown in (a), when the selected object 1510 moves with the movement of the vehicle 20 or the passage of time and is at least partially blocked by the building object in the sensed image, the processor 100 may be as follows Figure 15As shown in (b) to (d) of FIG. 15 , various augmented reality objects corresponding to the selected object 1510 are displayed. That is, a hatching circle ( Figure 15 (b)) of the augmented reality object 1511 or a three-dimensional shape having a central depression or a protrusion ( Figure 15 (c)) of the augmented reality object 1512 or having a circular shape ( Figure 15 (d)) of the augmented reality object 1513 to display the movement of the selected object.

[0184] In this case, the augmented reality object can be displayed on top of other objects in the sensed image. Therefore, the state of movement of the selected object can be tracked and displayed in the sensed image.

[0185] On the other hand, although the above description assumes that a specific object among the objects in the sensed image is selected by the user, the same method can also be used to track and display objects corresponding to a specific POI or specific category selected by the user. In this case, even if the specific object corresponding to the specific POI or specific category is blocked by other objects in the sensed image, the movement of the specific object in the sensed image can be displayed on the display unit 150 by overlaying the augmented reality object corresponding to the specific object at the position corresponding to the specific object on the other objects blocking the specific object.

[0186] The present invention described above can be implemented as a computer-readable code in a medium storing a program. Computer-readable media include all kinds of storage devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also include media implemented in the form of a carrier wave (such as Internet transmission). In addition, the computer may include a processor 100 of the display device 10. Therefore, the above detailed description should not be interpreted as restrictive in all aspects, but should be considered as exemplary. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims

1. A display device, characterized in that: include: an interface unit that receives a sensing image of a surrounding environment of the vehicle from at least one sensor provided on the vehicle; a memory including a classification information database, wherein the classification information database includes classification information for classifying each object included in the sensed image into preset categories different from each other; an object recognition unit that classifies and recognizes each object contained in the sensed image as any one of the categories in the classification information database based on the classification information; a renderer that renders an image to be displayed on a display unit provided on the vehicle; as well as The processor detects an object corresponding to at least one category selected by a user from among the objects of the identified category, controls the renderer to render an output image that replaces the detected object with a graphic object corresponding to the detected object, and controls the interface unit to output the rendered output image on the display unit through the interface.

2. The display device according to claim 1, wherein The vehicle further includes a communication unit configured to communicate with a cloud server, wherein the cloud server provides POI information including augmented reality objects corresponding to POIs around the vehicle. The processor requests the cloud server for POI information of POIs around the vehicle, including the current location of the vehicle and information of at least one category selected by the user. If at least one POI information corresponding to the at least one category among the POIs around the vehicle is received from the cloud server in response to the request, the processor extracts an augmented reality object from the received POI information and controls the renderer to render an output image that replaces the detected object with the extracted augmented reality object.

3. The display device according to claim 1, wherein The cloud server includes POI information classified according to the preset multiple categories, The classification information database includes classification information for classifying each object included in the sensed image according to the preset plurality of categories into which the cloud server classifies the POI information.

4. The display device according to claim 1, wherein The graphic objects are polygonal objects, which are graphic objects representing the shapes of the detected objects.

5. The display device according to claim 1, wherein The processor deletes the object detected according to the selected at least one category from the sensed image, and controls the renderer to render an output image in which the detected object is deleted from the sensed image.

6. The display device according to claim 5, wherein: The processor detects an area showing the detected object from the sensed image and initializes color information of the detected area to delete the detected object from the sensed image. The color of the initialized area varies according to the user's selection.

7. The display device according to claim 5, wherein: If the detected object is deleted from the sensed image, the processor detects at least one of a color of a peripheral area of ​​the deleted object and a color pattern of the peripheral area, The processor generates a fill image corresponding to a region of the sensed image where the object is detected to be deleted, based on at least one of the color and the color pattern. The processor controls the renderer to render an output image in which the filling image is displayed in a region corresponding to the deleted object in the sensed image.

8. The display device according to claim 1 or 5, characterized in that When a preset condition is satisfied, the processor further detects an object corresponding to a specific category matching the satisfied condition from the sensed image.

9. The display device according to claim 1 or 5, characterized in that: The preset condition is a condition for at least one of illumination, weather, and time of the vehicle.

10. The display device according to claim 1 or 5, characterized in that The preset condition is a condition of biological information detected by a sensor unit of the vehicle from at least one of the passengers of the vehicle.

11. The display device according to claim 2, wherein: The processor detects at least one object in the sensed image that is located in a direction from the vehicle to the specific POI based on preset location information of the specific POI and location information of the vehicle, The processor deletes the detected at least one object from the sensed image, and controls the renderer to render an output image in which a region corresponding to the deleted at least one object in the sensed image displays a graphic object corresponding to the specific POI.

12. The display device according to claim 11, wherein The graphic object is an augmented reality object corresponding to the specific POI, The processor receives the location information of the specific POI and the augmented reality object corresponding to the specific POI from the cloud server.

13. The display device according to claim 11, wherein The processor calculates a distance between the specific POI and the vehicle based on the location information of the specific POI and the location information of the vehicle, and determines a size of the graphic object based on the calculated distance.

14. The display device according to claim 11, wherein The processor detects objects corresponding to a category with deletion restrictions among objects located in a direction pointing to the specific POI in the sensed image, and deletes only objects other than the objects corresponding to the category with deletion restrictions.

15. A method for controlling a display device, wherein the display device is connected to a camera, at least one sensor, and a display unit provided in a vehicle via an interface unit, wherein: The control method includes: receiving a sensing image sensed by the camera; a step of classifying and identifying categories corresponding to respective objects included in the sensed image based on classification information for classifying into preset categories different from each other; a step of detecting objects corresponding to at least one specific category selected by a user from among the objects of the identified categories; a step of replacing the detected objects with graphic objects respectively corresponding to the detected objects to generate an output image; and The step of rendering the output image to output the generated output image on the display unit.

16. The control method of the display device according to claim 15, characterized in that: The step of generating the output image is a step of initializing a region in the sensed image corresponding to each of the detected objects, thereby generating an output image in which the detected objects are deleted from the sensed image.

17. The control method of the display device according to claim 16, characterized in that: The steps of generating the output image include: Initializing an area in the sensed image corresponding to each of the detected objects; detecting at least one of a color of a peripheral area of ​​the deleted object and a color pattern of the peripheral area; generating a filling image corresponding to a region of the sensed image where the object is detected to be deleted, based on at least one of the color and the color pattern; and and generating an output image in which the filling image is displayed in an area corresponding to the deleted object in the sensed image.

18. The method for controlling a display device according to claim 15, wherein: The steps of generating the output image include: The step of transmitting the specific category information and the location information of the vehicle to a cloud server, and requesting POI information including augmented reality objects corresponding to POIs of the category corresponding to the specific category around the location of the vehicle; In response to the request, receiving POI information from the cloud server; a step of deleting, from the sensed image, objects corresponding to the specific category among objects contained in the sensed image; and The step of generating an output image in which augmented reality objects respectively corresponding to the objects deleted from the sensed image are displayed in an area corresponding to the deleted object in the sensed image.

19. The method for controlling a display device according to claim 15, wherein: The step of detecting an object corresponding to the specific category includes: a step of detecting the location of a specific POI selected by a user based on the location of the vehicle; and a step of detecting, from the sensed image, at least one object displayed in a direction directed from the position of the vehicle to the position of the specific POI; The graphic object corresponding to the detected object is the augmented reality object corresponding to the specific POI.

Images