A method for providing navigation and device for providing navigation
The method processes navigation images using reference lines and AI to remove vehicle interior components, improving lane detection and providing accurate route guidance by optimizing the navigation screen for enhanced visibility.
Patent Information
- Application Number
- KR1020200035935
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-08-29
- Filing Date
- 2020-03-24
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2040-03-24
AI Technical Summary
Navigation devices equipped with cameras capture images of vehicle interiors, which obscure lane visibility and hinder accurate AR road marking effects due to vehicle components like the bonnet or dashboard, and incorrect camera installation exacerbates this issue, leading to inaccurate route guidance.
A method and device that processes images using reference lines to remove vehicle interior components, separate the image into upper and lower sections, and utilize AI algorithms to enhance lane detection, providing accurate navigation images by removing unnecessary surroundings.
The device enhances navigation accuracy by removing obstructions and optimizing the screen ratio for better lane detection, ensuring stable route guidance by enlarging only the necessary navigation screen for easier driver viewing.
Smart Images

Figure R1020200035935_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a device for providing navigation images and a method for providing the same, and more specifically, it is applicable to the fields of 5G communication technology, robot technology, autonomous driving technology, and AI (Artificial Intelligence) technology. Background Technology
[0002] VR (Virtual Reality) technology provides real-world objects or backgrounds solely as CG (Computer Graphic) images, AR (Augmented Reality) technology provides virtual CG images on top of real object images, and MR (Mixed) technology provides virtual objects mixed and combined with the real world. The aforementioned VR, AR, and MR are all sometimes simply referred to as XR (extended reality) technology. The problem to be solved
[0003] Devices equipped with navigation functions can provide vehicle driving route information through a navigation screen that adds road display effects using XR content.
[0004] A device equipped with a navigation function may be a mobile device or a mobile phone (hereinafter, mobile).
[0005] The device captures an image of the vehicle's current road ahead using a camera and regenerates (or renders) the preview video onto the navigation screen.
[0006] When a device (e.g., a mobile) captures a preview video, the camera must be installed and set so that part of the driving route screen is not obscured by the vehicle's components.
[0007] However, vehicle components such as the bonnet or dashboard are easily included within the field of view of the mobile camera.
[0008] Such vehicle components are often included in preview videos as images of obstacles that obscure the lane.
[0009] In addition, if the mobile unit is installed incorrectly inside the vehicle due to vehicle shaking caused by road irregularities, the captured preview video may include a larger portion of the sky or surrounding footage skewed to one side of the road than the road footage ahead.
[0010] If lane images are obscured by obstacle images, the device cannot accurately recognize the lane images, making it difficult to add accurate AR (Augmented Reality) road marking effects to the road.
[0011] In addition, if the device's camera captures the road at an incorrect angle, a larger portion of the image is unrelated to the road, resulting in a problem where the device cannot provide accurate driving route information.
[0012] However, the scope of the present invention is not limited to the aforementioned purposes only, and may be extended to other purposes that can be inferred by a person skilled in the art based on the entire contents of this specification. means of solving the problem
[0014] A control method for a mobile device equipped with a navigation function according to an embodiment of the present invention may include: a step of executing a navigation function; a step of capturing a first image of the surroundings of the mobile device by means of a camera; a step of displaying the captured first image, wherein the first image includes both an object inside the vehicle and an object outside the vehicle; a step of generating a second image based on a plurality of reference lines extracted from the first image, wherein the second image does not include images of the object inside the vehicle and the object outside the vehicle; and a step of displaying the generated second image.
[0015] In addition, in a control method for a mobile device equipped with a navigation function according to an embodiment of the present invention, the step of generating a second image based on a plurality of reference lines extracted from a first image may include: a step of generating a third image in which vehicle interior objects and vehicle exterior objects are removed from the first image; a step of separating the third image into upper and lower images centered on the first reference line—wherein the vertical height of the upper image corresponds to a first value and the vertical height of the lower image corresponds to a second value—; a step of obtaining a height value of the second image based on a preset first ratio and the smaller value between the first value and the second value, wherein the first ratio is the ratio of the second value to the first value—; a step of obtaining a width value of the second image based on a preset second ratio and the height value of the second image, wherein the second ratio is the ratio of the width value to the height value of the second image—.
[0016] A mobile device equipped with a navigation function according to one embodiment of the present invention may include: a controller for executing a navigation function; a camera for capturing a first image of the surroundings of the mobile device; a display for displaying the captured first image, wherein the first image includes both an object inside the vehicle and an object outside the vehicle; a video rendering processor for generating a second image based on a plurality of reference lines extracted from the first image, wherein the second image does not include images of the object inside the vehicle and the object outside the vehicle; and a display for displaying the generated second image.
[0017] However, the scope of the present invention is not limited to the aforementioned purposes only, and may be extended to other purposes that can be inferred by a person skilled in the art based on the entire contents of this specification. Effects of the invention
[0018] A device according to embodiments of the present invention processes a driving road image (hereinafter referred to as the third image) in which an image of an obstacle obscuring a lane is removed based on a plurality of reference lines from an initial driving road image (hereinafter referred to as the first image) and provides it as a navigation image.
[0019] The device according to the embodiments processes the navigation image to be output using a program such as a DNN of an AI (Artificial Intelligence) algorithm.
[0020] The device can provide a final navigation image by deleting surrounding areas (e.g., left area) separated from the lane image in the third image to facilitate the detection of lanes on the driving road.
[0021] The device according to the embodiments extracts (crops) the final navigation screen at the best screen ratio suitable for user convenience based on a plurality of reference lines including a vanishing point.
[0022] Therefore, the device removes unnecessary images around the driving road and provides a navigation screen that includes accurate route guidance information using AR.
[0023] The device provides navigation services by enlarging only the navigation screen that the driver needs to make it easier to view, allowing the driver to drive with a more stable route guidance service.
[0024] However, the scope of the present invention is not limited to the technical effects described above, and may be extended to other technical effects that can be inferred by a person skilled in the art based on the entire content of this specification. Brief explanation of the drawing
[0025] Figure 1 illustrates a resource grid mapped to physical signals / channels in a 3GPP-based system. Figure 2 is a diagram showing an example of a 3GPP signal transmission / reception method. Figure 3 illustrates an SSB structure. Figure 4 illustrates an example of an arbitrary connection process. Figure 5 shows an example of UL transmission according to an uplink grant. Figure 6 shows an example of a conceptual diagram of physical channel processing. Figure 7 is a diagram showing an example of a block diagram of a transmitting end and a receiving end for hybrid beamforming. FIG. 8 (a) is a figure showing an example of narrowband operation, and FIG. 8 (b) is a figure showing an example of MTC channel repetition with RF retuning. FIG. 9 illustrates a block diagram of a wireless communication system to which the methods proposed in this specification can be applied. FIG. 10 shows an AI device (100) according to one embodiment of the present invention. FIG. 11 shows an AI server (200) according to one embodiment of the present invention. FIG. 12 shows an AI system (1) according to one embodiment of the present invention. FIG. 13 is a block diagram illustrating an XR device according to embodiments of the present invention. FIG. 14 is a block diagram illustrating the memory shown in FIG. 13 in more detail. Figure 15 shows a point cloud data processing system. Figure 16 shows a device including a learning processor. FIG. 17 illustrates the process of the XR device (1600) of the present invention shown in FIG. 16 providing XR services. Figure 18 illustrates the appearance of an XR device and a robot. FIG. 19 is a flowchart illustrating the process of controlling a robot using a device equipped with XR technology. Figure 20 shows a vehicle providing autonomous driving services. Figure 21 illustrates the process of providing AR / VR services during autonomous driving services. FIG. 22 shows an HDM type XR device according to embodiments of the present invention. FIG. 23 shows an AR glass type XR device according to embodiments of the present invention. FIG. 24 is a flowchart illustrating a control method of a device according to embodiments. FIG. 25 is a block diagram illustrating the components of a device according to embodiments. FIG. 26 is an example drawing of a device according to embodiments rendering an image based on a plurality of reference lines. FIG. 27 illustrates a rendering in which a device according to embodiments deletes an obstruction image based on a second baseline. FIGS. 28(a) to 28(b) are a flowchart (2800) and a drawing illustrating a method in which a device according to embodiments manually sets a first reference line to render an image. FIGS. 29(a) to 29(c) are a flowchart (2900) and a drawing illustrating a method in which a device according to embodiments automatically sets a first baseline to render an image. FIGS. 30 to 31 are a flowchart (3000) and a drawing illustrating a method in which a device according to embodiments automatically calculates and renders a second baseline using an algorithm such as a DNN (Neural Network). FIG. 32 is an example of a device according to embodiments extracting a display image by applying a lane recognition algorithm through a DNN. FIG. 33 is an example of rendering by the device when the first baseline is positioned at half the vertical height of the third image. FIG. 34 is an example of rendering by the device when the first reference line is positioned above the halfway point of the vertical height of the third image. FIG. 35 is an example of rendering by the device when the first reference line is positioned below the halfway point of the vertical height of the third image. FIGS. 36 and 37 are flowcharts for the case where the device switches from a mode displaying navigation images in map mode to a mode displaying navigation images in AR mode. FIG. 38 illustrates a method in which a device re-renders and displays a navigation video as the aspect ratio of the screen displaying the navigation video changes. FIG. 39 is a flowchart illustrating a method for controlling a device according to embodiments. FIGS. 40 to 41 are a flowchart (4000) and a drawing illustrating a method in which a device according to embodiments automatically calculates a second reference line using an algorithm such as a DNN (Neural Network) to render a navigation image. FIG. 42 is a block diagram of a mobile terminal according to an embodiment of the present invention. FIG. 43 is a detailed block diagram of a processor according to an embodiment of the present invention. FIG. 44 is a drawing referenced to explain a navigation screen according to an embodiment of the present invention. FIG. 45 is a drawing referenced to explain the operation of generating a navigation screen according to an embodiment of the present invention. FIG. 46 is a flowchart according to an embodiment of the present invention. FIGS. 47a to 47b are drawings referenced to explain the vanishing line, bonnet line, and center line according to an embodiment of the present invention. FIGS. 48a to 48d are drawings referenced to explain a cropping operation according to an embodiment of the present invention. FIGS. 49a to 49b are drawings referenced to explain a cropping operation according to an embodiment of the present invention. FIGS. 50a to 52d are drawings referenced to explain the operation of using a wide-angle image according to an embodiment of the present invention. FIGS. 53a to 53b are drawings referenced to explain the vanishing line adjustment operation according to an embodiment of the present invention. FIGS. 54a to 54c are drawings referenced to explain the operation of compensating the tilt value of a mobile terminal installation according to an embodiment of the present invention. FIGS. 55a to 55c are drawings referenced to explain the operation of a mobile terminal in a specific situation according to an embodiment of the present invention. FIG. 56 is a drawing referenced to explain the operation of a mobile terminal in walking mode according to an embodiment of the present invention. Specific details for implementing the invention
[0026] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols are assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the invention.
[0027] The following embodiments of the present invention are intended merely to embody the invention and do not limit or restrict the scope of the rights thereof. Anything that can be easily inferred by a person skilled in the art from the detailed description and embodiments of the present invention is interpreted as falling within the scope of the rights thereof.
[0028] The foregoing detailed description should not be interpreted restrictively in all respects and should be considered exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.
[0029] Introduction
[0030] In the following, the downlink (DL) refers to communication from a base station (BS) to user equipment (UE), and the uplink (UL) refers to communication from the UE to the BS. In the downlink, the transmitter may be part of the BS, and the receiver may be part of the UE. In the uplink, the transmitter may be part of the UE, and the receiver may be part of the BS. In this specification, the UE may be referred to as the first communication device, and the BS as the second communication device. The BS may be replaced by terms such as fixed station, Node B, eNB (evolved-NodeB), gNB (Next Generation NodeB), BTS (base transceiver system), access point (AP), network or 5G (5th generation) network node, AI (Artificial Intelligence) system, RSU (road side unit), robot, AR / VR (Augmented Reality / Virtual Reality) system, etc. In addition, UE can be replaced with terms such as terminal, MS (Mobile Station), UT (User Terminal), MSS (Mobile Subscriber Station), SS (Subscriber Station), AMS (Advanced Mobile Station), WT (Wireless terminal), MTC (Machine-Type Communication) device, M2M (Machine-to-Machine) device, D2D (Device-to-Device) device, vehicle, robot, AI device (or module), AR / VR device (or module).
[0031] The following technology can be used in various wireless access systems such as CDMA (Code Division Multiple Access), FDMA (Frequency Division Multiple Access), TDMA (Time Division Multiple Access), OFDMA (Orthogonal Frequency Division Multiple Access), and SC-FDMA (Single Carrier FDMA).
[0032] For convenience of explanation, this specification is based on 3GPP communication systems (e.g., LTE-A, NR), but the present invention is not limited thereto. For reference, 3GPP (3rd Generation Partnership Project) LTE (Long Term Evolution) is part of E-UMTS (Evolved UMTS) using E-UTRA, and LTE-A (Advanced) / LTE-A pro is an evolved version of 3GPP LTE. 3GPP NR (New Radio or New Radio Access Technology) is an evolved version of 3GPP LTE / LTE-A / LTE-A pro.
[0033] In this disclosure, the term "node" refers to a fixed point capable of transmitting or receiving wireless signals by communicating with a UE. Various types of BSs may be used as nodes regardless of their designation. For example, BSs, NBs, eNBs, pico-cell eNBs (PeNBs), home eNBs (HeNBs), relays, repeaters, etc., may serve as nodes. At least one antenna is installed on a node. The antenna may refer to a physical antenna, an antenna port, a virtual antenna, or an antenna group. A node is also referred to as a point.
[0034] In this specification, the term "cell" may refer to a specific geographical area or wireless resource where one or more nodes provide communication services. A "cell" of a geographical area may be understood as the coverage over which a node can provide services using a carrier wave, and a "cell" of a wireless resource is associated with the bandwidth (BW), which is the frequency size configured by said carrier wave. Since downlink coverage, which is the range over which a node can transmit a valid signal, and uplink coverage, which is the range over which a valid signal can be received from a UE, depend on the carrier wave carrying the signal, the coverage of a node is also associated with the coverage of the "cell" of the wireless resource used by said node. Therefore, the term "cell" may be used to refer sometimes to the coverage of a service provided by a node, sometimes to a wireless resource, and sometimes to the range over which a signal using said wireless resource can reach with effective strength.
[0035] In this specification, "communicating with a specific cell" may mean communicating with a BS or node that provides communication services to said specific cell. Additionally, the downlink / uplink signals of a specific cell refer to downlink / uplink signals from to or to the BS or node that provides communication services to said specific cell. A cell that provides uplink / downlink communication services to a UE is specifically referred to as a serving cell. Furthermore, the channel state / quality of a specific cell refers to the channel or communication link formed between the BS or node that provides communication services to said specific cell and the UE.
[0036] Meanwhile, a "cell" associated with wireless resources can be defined as a combination of downlink resources (DL resources) and uplink resources (UL resources), that is, a combination of a DL component carrier (CC) and a UL CC. A cell may be configured as a DL resource alone or as a combination of a DL resource and a UL resource. Where carrier aggregation is supported, the linkage between the carrier frequency of a DL resource (or DL CC) and the carrier frequency of a UL resource (or UL CC) may be indicated by system information transmitted through the cell. Here, the carrier frequency may be the same as or different from the center frequency of each cell or CC. Hereinafter, a cell operating on the primary frequency is referred to as a primary cell (Pcell) or PCC, and a cell operating on the secondary frequency is referred to as a secondary cell (Scell) or SCC. An Scell can be established after the UE performs the RRC (Radio Resource Control) connection establishment process with the BS and an RRC connection is established between the UE and the BS, that is, after the UE has reached the RRC_CONNECTED state. Here, an RRC connection may refer to a channel through which the UE's RRC and the BS's RRC can exchange RRC messages. An Scell may be established to provide additional radio resources to the UE. Depending on the capabilities of the UE, the Scell, together with the Pcell, may form a set of serving cells for the UE.For UEs in the RRC_CONNECTED state but where carrier aggregation is not configured or does not support carrier aggregation, there is only one serving cell configured as a Pcell.
[0037] Cells support their own radio access technology. For example, on an LTE cell, transmission and reception are performed according to LTE radio access technology (RAT), and on a 5G cell, transmission and reception are performed according to 5G RAT.
[0038] Carrier aggregation technology refers to a technique that aggregates multiple carriers having a system bandwidth smaller than the target bandwidth to support broadband. Carrier aggregation is distinguished from OFDMA technology, which performs downlink or uplink communication by carrying a fundamental frequency band divided into multiple orthogonal subcarriers onto a single carrier frequency, in that it performs downlink or uplink communication using multiple carrier frequencies, each of which forms a system bandwidth (also called channel bandwidth). For example, in the case of OFDMA or Orthogonal Frequency Division Multiplexing (OFDM), a single frequency band with a certain system bandwidth is divided into multiple subcarriers with a certain subcarrier spacing, information / data is mapped within these multiple subcarriers, and the frequency band in which the information / data is mapped undergoes frequency upconversion to be transmitted to the carrier frequency of the said frequency band. In the case of wireless carrier aggregation, frequency bands each having its own system bandwidth and carrier frequency can be used for communication simultaneously, and each frequency band used for carrier aggregation can be divided into multiple subcarriers with a certain subcarrier spacing.
[0039] 3GPP-based communication standards define downlink physical channels corresponding to resource elements carrying information originating from upper layers of the physical layer (e.g., medium access control (MAC) layer, radio link control (RLC) layer, protocol data convergence protocol (PDCP) layer, radio resource control (RRC) layer, service data adaptation protocol (SDAP), non-access stratum (NAS) layer), and downlink physical signals corresponding to resource elements used by the physical layer but not carrying information originating from upper layers. For example, the physical downlink shared channel (PDSCH), physical broadcast channel (PBCH), physical multicast channel (PMCH), physical control format indicator channel (PCFICH), and physical downlink control channel (PDCCH) are defined as downlink physical channels, and the reference signal and synchronization signal are defined as downlink physical signals.A reference signal (RS), also referred to as a pilot, refers to a signal of a specific waveform that is known to both the BS and the UE. For example, cell-specific RS, UE-specific RS (UE-RS), positioning RS (PRS), channel state information RS (CSI-RS), and demodulation reference signal (DMRS) are defined as downlink reference signals. Meanwhile, 3GPP-based communication standards define uplink physical channels corresponding to resource elements that carry information originating from upper layers, and uplink physical signals corresponding to resource elements used by the physical layer but that do not carry information originating from upper layers. For example, the physical uplink shared channel (PUSCH), physical uplink control channel (PUCCH), and physical random access channel (PRACH) are defined as uplink physical channels, and the demodulation reference signal (DMRS) for uplink control / data signals and the sounding reference signal (SRS) used for uplink channel measurement are defined.
[0040] In this specification, physical shared channels (e.g., PUSCH, PDSCH) are used to carry information originating from upper layers of the physical layer (e.g., medium access control (MAC) layer, radio link control (RLC) layer, protocol data convergence protocol (PDCP) layer, radio resource control (RRC) layer, service data adaptation protocol (SDAP), non-access stratum (NAS) layer).
[0041] In this specification, a reference signal (RS) refers to a signal of a specific waveform that is known to both the BS and the UE. In 3GPP-based communication systems, for example, cell-specific RS (cell specific RS) which is a cell common RS, UE-specific RS (UE-RS) for demodulating a physical channel for a specific UE, channel state information RS (CSI-RS) for measuring / estimating downlink channel state, and demodulation reference signal (DMRS) for demodulating a physical channel are defined as downlink RSs, and DMRS for demodulating uplink control / data signals and sounding reference signal (SRS) used for measuring / estimating uplink channel state are defined as uplink RSs.
[0042] In this specification, a transport block is a payload for the physical layer. For example, data given to the physical layer from a higher layer or the medium access control (MAC) layer is basically referred to as a transport block. A UE, which is a device (AR / VR device) including an AR / VR module, may transmit a transport block containing AR / VR data to a wireless communication network (e.g., a 5G network) via PUSCH. Alternatively, the UE may receive from the wireless communication network a transport block containing AR / VR data from the 5G network or a transport block containing a response related to the AR / VR data transmitted by the UE.
[0043] In this specification, HARQ (Hybrid Automatic Repeat and reQuest) is a type of error control method. HARQ-ACK (HARQ acknowledgment) transmitted over the downlink is used for error control of uplink data, and HARQ-ACK transmitted over the uplink is used for error control of downlink data. The transmitting end performing the HARQ operation waits for a positive acknowledgment (ACK; acknowledgment) after transmitting data (e.g., transport block, codeword). The receiving end performing the HARQ operation sends a positive acknowledgment (ACK) only if the data is received correctly, and sends a negative acknowledgment (NACK) if there is an error in the received data. If the transmitting end receives an ACK, it can transmit (new) data, and if it receives a NACK, it can retransmit the data.
[0044] In this specification, channel state information (CSI) is a general term for information that can indicate the quality of a radio channel (also called a link) formed between a UE and an antenna port. CSI may include at least one of a channel quality indicator (CQI), a precoding matrix indicator (PMI), a CSI-RS resource indicator (CRI), an SSB resource indicator (SSBRI), a layer indicator (LI), a rank indicator (RI), or a reference signal received power (RSRP).
[0045] In this specification, frequency division multiplexing (FDM) may mean transmitting / receiving signals / channels / users in different frequency resources, and time division multiplexing (TDM) may mean transmitting / receiving signals / channels / users in different time resources.
[0046] In this specification, frequency division duplex (FDD) refers to a communication method in which uplink communication is performed on an uplink carrier and downlink communication is performed on a downlink carrier linked to said uplink carrier, and time division duplex (TDD) refers to a communication method in which uplink communication and downlink communication are performed on the same carrier in separate times. Meanwhile, in this specification, semi-duplex refers to a communication device operating as either an uplink or only an uplink on one frequency at one time, and operating as either a downlink or an uplink on another frequency at another time. For example, when a communication device operates as semi-duplex, it communicates using an uplink frequency and a downlink frequency, but said communication device cannot use the uplink frequency and the downlink frequency simultaneously; instead, it divides time to perform uplink transmission via the uplink frequency for a certain period and performs downlink reception by retuning to the downlink frequency for another certain period.
[0047] Figure 1 illustrates a resource grid mapped to physical signals / channels in a 3GPP-based system.
[0048] Referring to Fig. 1, for each subcarrier interval setting and carrier, individual subcarriers and A resource grid of OFDM symbols is defined, where is indicated by RRC signaling from BS. μ represents the subcarrier spacing △f = 2μ*15 [kHz], and in a 5G system, μ∈{0, 1, 2, 3, 4}.
[0049] This can vary not only for the subcarrier spacing setting μ but also between the uplink and downlink. There is a single resource grid for the subcarrier spacing setting μ, antenna port p, and transmission direction (uplink or downlink). Each element of the resource grid for the subcarrier spacing setting μ and antenna port p is referred to as a resource element and is uniquely identified by an index pair (k,l), where k is an index in the frequency domain and l refers to the symbol position in the time domain relative to a reference point. A resource block (RB), which is a frequency unit used for mapping physical channels to resource elements, is in the frequency domain It is defined as consecutive subcarriers. In a 5G system, considering that the UE may not be able to support the wide bandwidth supported by the 5G system at once, the UE may be configured to operate in a portion of the cell's frequency bandwidth (hereinafter referred to as the bandwidth part, BWP).
[0050] Regarding the background technology, terms, abbreviations, etc. used in this specification, reference may be made to matters described in standard documents published prior to the present invention. For example, the following documents may be referenced.
[0051] 3GPP LTE
[0052] - 3GPP TS 36.211: Physical channels and modulation
[0053] - 3GPP TS 36.212: Multiplexing and channel coding
[0054] - 3GPP TS 36.213: Physical layer procedures
[0055] - 3GPP TS 36.214: Physical layer; Measurements
[0056] - 3GPP TS 36.300: Overall description
[0057] - 3GPP TS 36.304: User Equipment (UE) procedures in idle mode
[0058] - 3GPP TS 36.314: Layer 2 - Measurements
[0059] - 3GPP TS 36.321: Medium Access Control (MAC) protocol
[0060] - 3GPP TS 36.322: Radio Link Control (RLC) protocol
[0061] - 3GPP TS 36.323: Packet Data Convergence Protocol (PDCP)
[0062] - 3GPP TS 36.331: Radio Resource Control (RRC) protocol
[0063] - 3GPP TS 23.303: Proximity-based services (Prose); Stage 2
[0064] - 3GPP TS 23.285: Architecture enhancements for V2X services
[0065] - 3GPP TS 23.401: General Packet Radio Service (GPRS) enhancements for Evolved Universal Terrestrial Radio Access Network (E-UTRAN) access
[0066] - 3GPP TS 23.402: Architecture enhancements for non-3GPP accesses
[0067] - 3GPP TS 23.286: Application layer support for V2X services; Functional architecture and information flows
[0068] - 3GPP TS 24.301: Non-Access-Stratum (NAS) protocol for Evolved Packet System (EPS); Stage 3
[0069] - 3GPP TS 24.302: Access to the 3GPP Evolved Packet Core (EPC) via non-3GPP access networks; Stage 3
[0070] - 3GPP TS 24.334: Proximity-services (ProSe) User Equipment (UE) to ProSe function protocol aspects; Stage 3
[0071] - 3GPP TS 24.386: User Equipment (UE) to V2X control function; protocol aspects; Stage 3
[0072] 3GPP NR (e.g. 5G)
[0073] - 3GPP TS 38.211: Physical channels and modulation
[0074] - 3GPP TS 38.212: Multiplexing and channel coding
[0075] - 3GPP TS 38.213: Physical layer procedures for control
[0076] - 3GPP TS 38.214: Physical layer procedures for data
[0077] - 3GPP TS 38.215: Physical layer measurements
[0078] - 3GPP TS 38.300: NR and NG-RAN Overall Description
[0079] - 3GPP TS 38.304: User Equipment (UE) procedures in idle mode and in RRC inactive state
[0080] - 3GPP TS 38.321: Medium Access Control (MAC) protocol
[0081] - 3GPP TS 38.322: Radio Link Control (RLC) protocol
[0082] - 3GPP TS 38.323: Packet Data Convergence Protocol (PDCP)
[0083] - 3GPP TS 38.331: Radio Resource Control (RRC) protocol
[0084] - 3GPP TS 37.324: Service Data Adaptation Protocol (SDAP)
[0085] - 3GPP TS 37.340: Multi-connectivity; Overall description
[0086] - 3GPP TS 23.287: Application layer support for V2X services; Functional architecture and information flows
[0087] - 3GPP TS 23.501: System Architecture for the 5G System
[0088] - 3GPP TS 23.502: Procedures for the 5G System
[0089] - 3GPP TS 23.503: Policy and Charging Control Framework for the 5G System; Stage 2
[0090] - 3GPP TS 24.501: Non-Access-Stratum (NAS) protocol for 5G System (5GS); Stage 3
[0091] - 3GPP TS 24.502: Access to the 3GPP 5G Core Network (5GCN) via non-3GPP access networks
[0092] - 3GPP TS 24.526: User Equipment (UE) policies for 5G System (5GS); Stage 3
[0093] Figure 2 is a figure showing an example of a 3GPP signal transmission / reception method.
[0094] Referring to FIG. 2, when the UE is powered on or enters a new cell, it performs an initial cell search operation, such as synchronizing with the BS (S201). To this end, the UE receives a primary synchronization channel (P-SCH) and a secondary synchronization channel (S-SCH) from the BS to synchronize with the BS and obtain information such as a cell ID. In LTE systems and NR systems, the P-SCH and S-SCH are referred to as the primary synchronization signal (PSS) and the secondary synchronization signal (SSS), respectively. The above initial cell search procedure is described in more detail below.
[0095] After the initial cell search, the UE can obtain broadcast information within the cell by receiving a physical broadcast channel (PBCH) from the BS. Meanwhile, the UE can check the downlink channel status by receiving a downlink reference signal (DL RS) during the initial cell search phase.
[0096] After completing the initial cell search, the UE can obtain more specific system information by receiving the PDCCH and the PDSCH according to the information contained in the PDCCH (S202).
[0097] Meanwhile, when first connecting to the BS or when there are no wireless resources available for signal transmission, the UE may perform a random access procedure for the BS (steps S203 to S206). To this end, the UE transmits a specific sequence as a preamble via PRACH (S203 and S205) and may receive a random access response (RAR) message for the preamble via PDCCH and the corresponding PDSCH (S204 and S206). In the case of a contention-based random access procedure, a contention resolution procedure may additionally be performed. The random access procedure is described in more detail below.
[0098] The UE that has performed the process described above can subsequently perform PDCCH / PDSCH reception (S207) and PUSCH / PUCCH transmission (S208) as a general uplink / downlink signal transmission process. In particular, the UE receives DCI through PDCCH.
[0099] The UE monitors a set of PDCCH candidates at monitoring occasions configured in one or more control element sets (CORESETs) on the serving cell according to the corresponding search space configurations. The set of PDCCH candidates to be monitored by the UE is defined in terms of search space sets, and the search space sets may be common search space sets or UE-specific search space sets. A CORESET consists of a set of (physical) resource blocks having durations of 1 to 3 OFDM symbols. The network may be configured so that the UE has multiple CORESETs. The UE monitors PDCCH candidates within one or more search space sets. Here, monitoring means attempting to decode the PDCCH candidate(s) within the search space. If the UE succeeds in decoding one of the PDCCH candidates within the search space, the UE determines that it has detected a PDCCH from that candidate and performs PDSCH reception or PUSCH transmission based on the DCI within the detected PDCCH.
[0100] PDCCH can be used to schedule DL transmissions on PDSCH and UL transmissions on PUSCH. Here, the DCI on PDCCH includes a downlink assignment (i.e., a DL grant) that includes at least modulation and coding format and resource allocation information associated with a downlink shared channel, or an uplink grant (UL grant) that includes modulation and coding format and resource allocation information associated with an uplink shared channel.
[0101] Initial Access (IA) Process
[0102] SSB (Synchronization Signal Block) transmission and related operations
[0103] Figure 3 illustrates an SSB structure. Based on the SSB, the UE can perform cell search, acquire system information, beam alignment for initial connection, DL measurement, etc. The term SSB is used interchangeably with the term SS / PBCH (Synchronization Signal / Physical Broadcast channel) block.
[0104] Referring to FIG. 3, the SSB consists of a PSS, an SSS, and a PBCH. The SSB is composed of four consecutive OFDM symbols, and for each OFDM symbol, a PSS, a PBCH, an SSS / PBCH, or a PBCH is transmitted. The PBCH is encoded / decoded based on a polar code and modulated / demodulated according to Quadrature Phase Shift Keying (QPSK). The PBCH within an OFDM symbol consists of data resource elements (REs) to which the complex modulation value of the PBCH is mapped, and DMRS REs to which the demodulation reference signal (DMRS) for the PBCH is mapped. There are three DMRS REs for each resource block of the OFDM symbol, and three data REs exist between the DMRS REs.
[0105] Cell search
[0106] Cell search refers to the process in which the UE acquires the time / frequency synchronization of a cell and detects the cell ID (Identifier) of said cell (e.g., Physical layer Cell ID, PCI). PSS is used to detect a cell ID within a cell ID group, and SSS is used to detect a cell ID group. PBCH is used for detecting the SSB (time) index and half-frame detection.
[0107] In the 5G system, there are 336 cell ID groups, and each cell ID group contains 3 cell IDs. A total of 1,008 cell IDs exist. Information regarding the cell ID group to which a cell's cell ID belongs is provided / acquired through the cell's SSS, and information regarding the cell ID among the 336 cells within the cell ID is provided / acquired through the PSS.
[0108] SSBs are transmitted periodically in accordance with the SSB periododicity. During initial cell search, the default SSB period assumed by the UE is defined as 20ms. After cell connection, the SSB period can be set by the network (e.g., BS) to one of {5ms, 10ms, 20ms, 40ms, 80ms, 160ms}. An SSB burst set is configured at the beginning of the SSB period. The SSB burst set consists of a 5ms time window (i.e., a half-frame), and SSBs can be transmitted up to L times within the SSB burst set. The maximum number of SSB transmissions L can be given as follows, depending on the carrier frequency band.
[0109] - For frequency range up to 3 GHz, L = 4
[0110] - For frequency range from 3GHz to 6 GHz, L = 8
[0111] - For frequency range from 6 GHz to 52.6 GHz, L = 64
[0112] The possible time positions of SSBs within a half-frame are determined by the subcarrier interval, and the periododicity of the half-frames in which the SSBs are transmitted is set by the network. The time positions of SSB candidates are indexed from 0 to L-1 in chronological order within the set of SSB bursts (i.e., half-frames) (SSB index). During a half-frame, different SSBs may be transmitted in different spatial directions (using different beams that span the cell's coverage area). Therefore, in a 5G system, the SSBI can be associated with the BS Tx beam direction.
[0113] The UE can obtain DL synchronization by detecting an SSB. The UE can identify the structure of an SSB burst set based on the detected SSB (time) index (SSB index, SSBI) and, accordingly, detect symbol / slot / half-frame boundaries. The frame / half-frame number to which the detected SSB belongs can be identified using system frame number (SFN) information and half-frame indication information.
[0114] Specifically, the UE can obtain a 10-bit SFN from the PBCH for the frame to which the PBCH belongs. Next, the UE can obtain a 1-bit half-frame indication information. For example, if the UE detects a PBCH with the half-frame indication bit set to 0, it can determine that the SSB to which the PBCH belongs belongs to the first half-frame within the frame, and if the UE detects a PBCH with the half-frame indication bit set to 1, it can determine that the SSB to which the PBCH belongs belongs to the second half-frame within the frame. Finally, the UE can obtain the SSBI of the SSB to which the PBCH belongs based on the DMRS sequence and the PBCH payload carried by the PBCH.
[0115] Acquisition of system information (SI)
[0116] SI is divided into a master information block (MIB) and multiple system information blocks (SIB). SI other than the MIB may be referred to as RMSI (Remaining Minimum System Information). For details, please refer to the following.
[0117] - The MIB contains information / parameters for monitoring the PDCCH that schedules the PDSCH carrying SIB1 (SystemInformationBlock1) and is transmitted by the BS via the SSB's PBCH. For example, the UE can determine based on the MIB whether a CORESET (Control Resource Set) exists for the Type0-PDCCH common search space. The Type0-PDCCH common search space is a type of PDCCH search space and is used to transmit the PDCCH that schedules SI messages. If the Type0-PDCCH common search space exists, the UE can determine (i) multiple contiguous resource blocks and one or more consecutive symbols that constitute the CORESET, and (ii) a PDCCH occasion (e.g., a time domain location for receiving the PDCCH) based on the information within the MIB (e.g., pdcch-ConfigSIB1).
[0118] - SIB1 contains information related to the availability and scheduling (e.g., transmission period, SI-window size) of the remaining SIBs (hereinafter SIBx, where x is an integer greater than or equal to 2). For example, SIB1 may indicate whether SIBx is broadcast periodically or provided upon a UE's request via an on-demand method. If SIBx is provided via an on-demand method, SIB1 may contain information necessary for the UE to perform an SI request. The PDCCH scheduling SIB1 is transmitted through the Type0-PDCCH common seek space, and SIB1 is transmitted through the PDSCH directed by said PDCCH.
[0119] - SIBx is included in SI messages and transmitted via PDSCH. Each SI message is transmitted within a periodically occurring time window (i.e., SI window).
[0120] Random Access Process
[0121] Random access processes are used for various purposes. For example, random access processes can be used for network initial access, handover, and UE-triggered UL data transmission. Through random access processes, UEs can acquire UL synchronization and UL transmission resources. Random access processes are classified into contention-based random access processes and contention-free random access processes.
[0122] FIG. 4 illustrates an example of a random access process. In particular, FIG. 4 illustrates a contention-based random access process.
[0123] First, the UE may transmit the random access preamble via PRACH as Msg1 of the random access process in the UL. In this specification, the random access process and the random access preamble are also referred to respectively as the RACH process and the RACH preamble.
[0124] Multiple preamble formats are defined by one or more RACH OFDM symbols and different cyclic prefixes (CPs) (and / or guard times). A RACH configuration for a cell is provided to the UE in the cell's system information. The RACH configuration includes information regarding the subcarrier spacing of the PRACH, available preambles, preamble formats, etc. The RACH configuration includes association information between SSBs and RACH (time-frequency) resources, that is, association information between SSB indices (SSBIs) and RACH (time-frequency) resources. SSBIs are respectively associated with the Tx beams of the BS. The UE transmits a RACH preamble in the RACH time-frequency resource associated with the detected or selected SSB. The BS can determine the BS Tx beam preferred by the UE based on the time-frequency resource where the RACH preamble was detected.
[0125] A threshold for an SSB for RACH resource association can be set by the network, and the transmission (i.e., PRACH transmission) or retransmission of a RACH preamble is performed based on an SSB whose RSRP measured based on the SSB satisfies the threshold. For example, a UE can select one of the SSB(s) that satisfy the threshold and transmit or retransmit a RACH preamble based on the RACH resource associated with the selected SSB.
[0126] When the BS receives a RACH preamble from the UE, the BS transmits a RAR message (Msg2) to the UE. The PDCCH scheduling the PDSCH carrying the RAR is transmitted with CRC masking using a random access (RA) radio network temporary identifier (RNTI) (RA-RNTI). A UE that detects the PDCCH masked by RA-RNTI can receive the RAR from the PDSCH scheduling the DCI carrying the PDCCH. The UE checks whether RAR information for the preamble it transmitted, i.e., Msg1, is present in the RAR. Whether random access information for the Msg1 it transmitted exists can be determined by whether a RACH preamble ID for the preamble transmitted by the UE exists. If there is no response to Msg1, the UE may retransmit the RACH preamble within a specified number of times while performing power ramping. The UE calculates the PRACH transmission power for the retransmission of the preamble based on the most recent path loss and power ramping counter.
[0127] When a UE receives RAR information about itself on PDSCH, the UE can obtain timing advance information for UL synchronization, an initial UL grant, and a UE temporary cell RNTI (C-RNTI). The timing advance information is used to control uplink signal transmission timing. To ensure that PUSCH / PUCCH transmissions by the UE are better aligned with subframe timing at the network end, the network (e.g., BS) may measure the time difference between PUSCH / PUCCH / SRS receptions and subframes and send timing advance information based on this. Based on the RAR information, the UE may transmit a UL transmission on PUSCH as Msg3 in the RACH process. Msg3 may include an RRC connection request and a UE identifier. In response to Msg3, the network may transmit Msg4, which can be treated as a contention resolution message on DL. By receiving Msg4, the UE can enter an RRC connected state.
[0128] Meanwhile, the contention-free RACH process may be performed when a UE handes over to another cell or BS, or when requested by a command from the BS. The basic process of the contention-free RACH process is similar to the contention-based RACH process. However, unlike the contention-based RACH process, where the UE arbitrarily selects a preamble to use from among multiple RACH preambles, in the case of the contention-free RACH process, the preamble to be used by the UE (hereinafter referred to as the dedicated RACH preamble) is assigned to the UE by the BS. Information regarding the dedicated RACH preamble may be included in an RRC message (e.g., a handover command) or provided to the UE through a PDCCH order. When the RACH process is initiated, the UE transmits the dedicated RACH preamble to the BS. When the UE receives the RACH process from the BS, the RACH process is completed.
[0129] DL and UL transmission / reception operations
[0130] DL transmission / reception operation
[0131] Downlink grants (also called assignments) can be divided into (1) dynamic grants and (2) configured grants. Dynamic grants are intended to maximize resource utilization and refer to a data transmission / reception method based on dynamic scheduling by BS.
[0132] The BS schedules downlink transmissions via the DCI. The UE receives a DCI (hereinafter referred to as the DL Grant DCI) for downlink scheduling (i.e., containing scheduling information for the PDSCH) from the BS on the PDCCH. The DCI for downlink scheduling may include, for example, the following information: a bandwidth part indicator, frequency domain resource assignment, time domain resource assignment, and a modulation and coding scheme (MCS).
[0133] The UE can determine the modulation order, target code rate, and transport block size (TBS) for the PDSCH based on the MCS field within the DCI. The UE can receive the PDSCH from a time-frequency resource according to frequency domain resource allocation information and time domain resource allocation information.
[0134] A DL-configured grant is also referred to as semi-persistent scheduling (SPS). The UE may receive an RRC message from the BS containing resource configuration for the transmission of DL data. In the case of a DL SPS, the actual DL-configured grant is provided by the PDCCH and is enabled or deactivated by said PDCCH. When a DL SPS is configured, at least the following parameters are provided to the UE via RRC signaling from the BS: a configured scheduling RNTI (CS-RNTI) for activation, deactivation, and retransmission; and a period. The actual DL grant of the DL SPS (e.g., frequency resource allocation) is provided to the UE by the DCI within the PDCCH addressed in the CS-RNTI. The UE enables the SPS associated with said CS-RNTI if specific fields of the DCI within the PDCCH addressed in the CS-RNTI are set to specific values for scheduling activation. The DCI within the PDCCH addressed in the above CS-RNTI includes actual frequency resource allocation information, MCS index values, etc. The UE can receive downlink data via the PDSCH based on the SPS.
[0135] UL transmission / reception operation
[0136] Uplink grants can be divided into (1) dynamic grants, which dynamically schedule PUSCH by UL grant DCI, and (2) configured grants, which semi-statically schedule PUSCH by RRC signaling.
[0137] FIG. 5 illustrates an example of UL transmission based on an uplink grant. In particular, FIG. 5 (a) illustrates a UL transmission process based on a dynamic grant, and FIG. 5 (b) illustrates a UL transmission process based on a set grant.
[0138] In the case of a UL dynamic grant, the BS transmits a DCI containing uplink scheduling information to the UE. The UE receives a DCI for uplink scheduling (i.e., containing scheduling information for the PUSCH) (hereinafter referred to as the UL Grant DCI) from the BS on the PDCCH. The DCI for uplink scheduling may include, for example, the following information: a Bandwidth part indicator, a frequency domain resource assignment, a time domain resource assignment, and an MCS. For the efficient allocation of uplink radio resources by the BS, the UE transmits information regarding the uplink data it intends to transmit to the BS, and the BS may allocate uplink resources to the UE based on this information. In this case, the information regarding the uplink data transmitted by the UE to the BS is called a buffer status report (BSR), and the BSR is related to the amount of uplink data stored in the UE's own buffer.
[0139] Referring to Fig. 5 (a), the UL transmission process is illustrated when the UE does not have uplink radio resources available for transmitting BSR. Since the UE, which does not have UL grants available for UL data transmission, cannot transmit BSR via PUSCH, it must request resources for uplink data starting with the transmission of a scheduling request via PUCCH, and in this case, a five-step uplink resource allocation process is used.
[0140] Referring to FIG. 5 (a), if there are no PUSCH resources available to transmit the BSR, the UE first transmits a scheduling request (SR) to the BS to be allocated a PUSCH resource. The SR is used to request a PUSCH resource from the BS for uplink transmission when a buffer status reporting event occurs but there are no PUSCH resources available to the UE. If there is a valid PUCCH resource for the SR, the UE transmits the SR via the PUCCH; if there is no valid PUCCH resource, the aforementioned (contention-based) RACH process is initiated. When the UE receives a UL Grant from the BS via the UL Grant DCI, it transmits the BSR to the BS via the PUSCH resource allocated by the UL Grant. Based on the BSR, the BS determines the amount of data the UE will transmit over the uplink and transmits the UL Grant to the UE via the UL Grant DCI. A UE that detects a PDCCH containing the above UL Grant DCI transmits actual uplink data to a BS via a PUSCH based on the UL Grant within the above UL Grant DCI.
[0141] In the case of a configured grant, referring to FIG. 5(b), the UE receives an RRC message from the BS containing resource configuration for the transmission of UL data. In an NR system, there are two types of UL configured grants: Type 1 and Type 2. In the case of UL configured grant Type 1, the actual UL grant (e.g., time resource, frequency resource) is provided by RRC signaling, and in the case of UL configured grant Type 2, the actual UL grant is provided by PDCCH and is activated or deactivated by said PDCCH. When configured grant Type 1 is configured, at least the following parameters are provided to the UE via RRC signaling from the BS: CS-RNTI for retransmission; periododicity of configured grant Type 1; information regarding the start symbol index S and symbol count L for a PUSCH within the slot; and a time domain offset indicating the offset of the resource for SFN=0 in the time domain. An MCS index indicating the modulation order, target code rate, and transport block size. When a configured grant type 2 is configured, at least the following parameters are provided to the UE via RRC signaling from the BS: a CS-RNTI for activation, deactivation, and retransmission; and the period of the configured grant type 2. The actual UL grant of the configured grant type 2 is provided to the UE by the DCI within the PDCCH addressed in the CS-RNTI. The UE activates the configured grant type 2 associated with the CS-RNTI if specific fields of the DCI within the PDCCH addressed in the CS-RNTI are set to specific values for scheduling activation. The DCI within the PDCCH set to specific values for scheduling activation includes actual resource allocation information, MCS index values, etc. The UE can perform uplink transmission via PUSCH based on the configured grant according to type 1 or type 2.
[0142] Figure 6 shows an example of a conceptual diagram of physical channel processing.
[0143] Each of the blocks illustrated in FIG. 6 can be performed in each module within the physical layer block of the transmission device. More specifically, signal processing in FIG. 6 can be performed for UL transmission in the processor of the UE described herein. Signal processing that excludes transform precoding in FIG. 6 and includes CP-OFDM signal generation instead of SC-FDMA signal generation can be performed for DL transmission in the processor of the BS described herein. Referring to FIG. H5, uplink physical channel processing can be performed through the processes of scrambling, modulation mapping, layer mapping, transform precoding, precoding, resource element mapping, and SC-FDMA signal generation. Each of the above processes can be performed separately or together in each module of the transmission device. The above transform precoding spreads UL data in a special way that reduces the peak-to-average power ratio (PAPR) of the waveform, and is a type of Discrete Fourier Transform (DFT). OFDM using CP along with transform precoding that performs DFT spreading is called DFT-s-OFDM, and OFDM using CP without DFT spreading is called CP-OFDM. An SC-FDMA signal is generated by DFT-s-OFDM. In an NR system, transform precoding can be applied optionally when enabled for UL. That is, the NR system supports two options for UL waveforms, one of which is CP-OFDM and the other is DFT-s-OFDM.Whether the UE should use CP-OFDM or DFT-s-OFDM as the UL transmission waveform is provided to the UE from the BS via RRC parameters. FIG. H5 is a conceptual diagram of uplink physical channel processing for DFT-s-OFDM, and in the case of CP-OFDM, transform precoding among the processes in FIG. H5 is omitted. For DL transmission, CP-OFDM is used for DL waveform transmission.
[0144] Looking more specifically at each of the above processes, for a single codeword, the transmission device can transmit the coded bits within the codeword by scrambling them using a scrambling module and then transmit them through a physical channel. Here, the codeword is obtained by encoding a transport block. The scrambled bits are modulated into complex value modulation symbols by a modulation mapping module. The modulation mapping module can modulate the scrambled bits according to a predetermined modulation scheme to arrange them into complex value modulation symbols representing positions on a signal constellation. Pi / 2-BPSK (pi / 2-Binary Phase Shift Keying), m-PSK (m-Phase Shift Keying), or m-QAM (m-Quadrature Amplitude Modulation), etc., may be used for modulating the coded data. The complex value modulation symbols may be mapped to one or more transmission layers by a layer mapping module. The complex value modulation symbols on each layer may be precoded by a precoding module for transmission over an antenna port. When transform precoding is enabled for UL transmission, the precoding module can perform precoding after performing transform precoding for complex value modulation symbols as illustrated in FIG. H5. The precoding module can process the complex value modulation symbols using a MIMO method according to multiple transmission antennas to output antenna-specific symbols and distribute the antenna-specific symbols to the corresponding resource element mapping module. The output z of the precoding module can be obtained by multiplying the output y of the layer mapping module by an N×M precoding matrix W, where N is the number of antenna ports and M is the number of layers. The resource element mapping module maps the demodulation value modulation symbols for each antenna port to appropriate resource elements within the resource block allocated for transmission.The resource element mapping module maps complex-valued modulated symbols to appropriate subcarriers and can multiplex them according to the user. The SC-FDMA signal generation module (or the CP-OFDM signal generation module in the case of DL transmission or when transform precoding is disabled for UL transmission) can generate a complex-valued time domain OFDM symbol signal by modulating the complex-valued modulated symbols using a specific modulation method, such as OFDM. The signal generation module can perform an Inverse Fast Fourier Transform (IFFT) on antenna-specific symbols, and a CP can be inserted into the time domain symbols after the IFFT is performed. The OFDM symbols are transmitted to a receiving device through each transmitting antenna after undergoing digital-to-analog conversion, frequency upconversion, etc. The signal generation module may include an IFFT module, a CP inserter, a Digital-to-Analog Converter (DAC), a frequency uplink converter, etc.
[0145] The signal processing process of the receiving device can be configured as the inverse of the signal processing process of the transmitting device. For specific details, refer to the above and Figure 6.
[0146] Next, we will look at PUCCH.
[0147] PUCCH is used for the transmission of UCI. UCI includes a scheduling request (SR) requesting uplink transmission resources, channel state information (CSI) indicating the downlink channel state measured by the UE based on DL RS, and / or HARQ-ACK indicating whether downlink data has been successfully received by the UE.
[0148] PUCCH supports multiple formats, and PUCCH formats can be classified by symbol duration, payload size, and whether multiplexing is enabled. Table 1 below provides examples of PUCCH formats.
[0149]
[0150] PUCCH resources are configured for the UE by the BS via RRC signaling. As an example of allocating PUCCH resources, the BS configures multiple sets of PUCCH resources for the UE, and the UE can select a specific set of PUCCH resources corresponding to a specific range based on the range of UCI (payload) sizes (e.g., number of UCI bits). For example, the UE [selects] the number of UCI bits Depending on this, you can select one of the following PUCCH resource sets: - PUCCH resource set #0, if UCI bit count ≤ 2
[0151] - PUCCH resource set #1, if 2 < number of UCI bits ≤
[0152] - PUCCH resource set #(K-1), if NK-2 < UCI bit count ≤
[0153] Here, K represents the number of PUCCH resource sets (K>1), and Ni is the maximum number of UCI bits supported by PUCCH resource set #i. For example, PUCCH resource set #1 may consist of resources of PUCCH format 0 to 1, and other PUCCH resource sets may consist of resources of PUCCH format 2 to 4.
[0154] Subsequently, the BS transmits the DCI to the UE via the PDCCH, and through the ARI (ACK / NACK Resource Indicator) within the DCI, it can indicate the PUCCH resource to be used for UCI transmission among the PUCCH resources within a specific set of PUCCH resources. The ARI is used to indicate the PUCCH resource for HARQ-ACK transmission and may also be referred to as the PRI (PUCCH Resource Indicator).
[0155] eMBB (enhanced Mobile Broadband communication)
[0156] In the case of NR systems, massive multiple input multiple output (MIMO) environments, where the number of transmitting and receiving antennas increases significantly, are being considered. Meanwhile, for NR systems using bands above 6 GHz, beamforming techniques are being considered to collect and transmit energy in a specific direction rather than in all directions to compensate for rapid signal attenuation characteristics. Accordingly, to reduce the complexity of hardware implementation, increase performance using multiple antennas, provide flexibility in resource allocation, and facilitate frequency-specific beam control, a hybrid beamforming technique combining analog beamforming and digital beamforming techniques is required depending on the location where the beamforming weight vector / precoding vector is applied.
[0157] Hybrid Beamforming
[0158] Figure 7 is a diagram showing an example of a block diagram of a transmitting end and a receiving end for hybrid beamforming.
[0159] According to hybrid beamforming, a narrow beam can be formed by transmitting the same signal to a large number of antennas in the BS or UE using an appropriate phase difference, thereby increasing the energy only in a specific direction.
[0160] Beam Management (BM)
[0161] The BM process is a process for acquiring and maintaining a set of BS (or transmission and reception point (TRP)) and / or UE beams that can be used for downlink (DL) and uplink (UL) transmission / reception, and may include the following processes and terms.
[0162] - Beam measurement: An operation in which a BS or UE measures the characteristics of a received beamforming signal.
[0163] - Beam determination: The operation in which a BS or UE selects its transmit beam (Tx beam) / receive beam (Rx beam).
[0164] - Beam sweeping: An operation that covers a spatial domain using a transmitted and / or received beam for a set time interval in a predetermined manner.
[0165] - Beam report: An operation in which the UE reports information about the beamformed signal based on beam measurements.
[0166] The BM process can be divided into (1) a DL BM process using SSB or CSI-RS and (2) a UL BM process using SRS (sounding reference signal). Additionally, each BM process may include Tx beam sweeping to determine the Tx beam and Rx beam sweeping to determine the Rx beam. The following mainly describes the DL BM process using SSB.
[0167] The DL BM process using SSB may include (1) beam-formed SSB transmission by BS and (2) beam reporting by UE. SSB can be used for both Tx beam sweeping and Rx beam sweeping. Rx beam sweeping using SSB can be performed by the UE attempting to receive the SSB while changing the Rx beam.
[0168] Configuration for beam reporting using SSB is performed during channel state information (CSI) / beam configuration in RRC_CONNECTED.
[0169] - The UE receives information from the BS regarding the set of SSB resources used for the BM. The set of SSB resources can be configured with one or more SSB indices (SSBIs). The SSBIs for each set of SSB resources can be defined from 0 to a maximum of 63.
[0170] - The UE receives signals on the SSB resources from the BS based on information about the SSB resource set.
[0171] - If the UE is set by the BS to perform reporting on the SSB resource indicator (SSBRI) and RSRP, the UE reports the best SSBRI and the corresponding RSRP to the BS.
[0172] BS can determine the BS Tx beam to be used for DL transmission to the UE based on the beam report from the UE.
[0173] Beam failure recovery (BFR) process
[0174] In beamformed systems, Radio Link Failure (RLF) can frequently occur due to rotation, movement, or beamforming blockage of the UE. Therefore, BFR is supported in NR to prevent frequent RLF.
[0175] To detect beam failure, the BS sets beam failure detection reference signals to the UE, and the UE declares a beam failure when the number of beam failure indications from the physical layer of the UE reaches a threshold set by the RRC signaling within a period set by the BS's RRC signaling.
[0176] After a beam failure is detected, the UE triggers beam failure recovery by initiating a RACH process on the PCell; and performs beam failure recovery by selecting a suitable beam (if the BS has provided dedicated RACH resources for certain beams, the UE uses these preferentially to perform the RACH process for BFR). Upon completion of the RACH process, beam failure recovery is considered complete.
[0177] URLLC (Ultra-Reliable and Low Latency Communication)
[0178] URLLC transmission defined in NR may mean (1) relatively low traffic size, (2) relatively low arrival rate, (3) extremely low latency requirement (e.g., 0.5, 1ms), (4) relatively short transmission duration (e.g., 2 OFDM symbols), (5) transmission for urgent services / messages, etc.
[0179] Pre-emption indication
[0180] eMBB and URLLC services may be scheduled on non-overlapping time / frequency resources, but URLLC transmissions may occur on resources where ongoing eMBB traffic is scheduled. A preemption indication may be used to allow a UE receiving a PDSCH to know that the PDSCH has been partially punctured by a URLLC transmission by another UE. The preemption indication may also be referred to as an interrupted transmission indication.
[0181] In relation to preemption instructions, the UE receives downlink preemption RRC information (e.g., DownlinkPreemption IE) through RRC signaling from the BS.
[0182] The above UE receives DCI format 2_1 from the BS based on downlink preemption RRC information. For example, the UE attempts to detect a PDCCH that conveys DCI format 2_1, which is a DCI related to preemption instructions, using an int-RNTI set by the downlink preemption RRC information.
[0183] When a UE detects DCI format 2_1 for a serving cell(s) set by downlink preemption RRC information, the UE may assume that there is no transmission to the UE within the RBs and symbols indicated by DCI format 2_1 among the sets of resource blocks (RBs) and symbols of the last monitoring period immediately preceding the monitoring period to which the DCI format 2_1 belongs. For example, referring to FIG. J2, the UE determines that the signal within the time-frequency resource indicated by the preemption is not a DL transmission scheduled to it and decodes the data based on the signals received in the remaining resource area.
[0184] mMTC (massive MTC)
[0185] Massive Machine Type Communication (mmTC) is one of the 5G scenarios designed to support hyper-connected services that communicate simultaneously with a large number of UEs. In this environment, UEs communicate intermittently with extremely low transmission speeds and limited mobility. Therefore, the primary goal of mMTC is to enable UEs to operate for extended periods at a low cost. In this regard, we will examine MTC and NB-IoT as addressed by 3GPP.
[0186] The following description is based on the example where the transmission time interval of a physical channel is a subframe. For example, the description is based on the example where the minimum time interval from the start of transmission in one physical channel to the start of transmission in the next physical channel is one subframe. However, in the following description, subframes may be replaced by slots, mini-slots, or multiple slots.
[0187] MTC (Machine Type Communication)
[0188] MTC (Machine Type Communication) refers to a communication technology adopted by the 3GPP (3rd Generation Partnership Project) to meet the requirements of IoT services, as it is an application that does not require a large amount of throughput and can be applied to M2M (Machine-to-Machine) or IoT (Internet-of-Things).
[0189] The following description mainly concerns features related to eMTC, but unless otherwise noted, the same applies to MTC, eMTC, and MTC to be applied to 5G (or NR). The term MTC described below may be referred to by other terms such as eMTC (enhanced MTC), LTE-M1 / M2, BL (Bandwidth reduced low complexity) / CE (coverage enhanced), non-BL UE (in enhanced coverage), NR MTC, enhanced BL / CE, etc.
[0190] General characteristics of MTC
[0191] (1) MTC operates only within a specific system bandwidth (or channel bandwidth).
[0192] MTC may use a specific number of resource blocks (RBs) within the system band of a legacy LTE system or an NR system. The frequency band in which MTC operates can be defined by considering the frequency range and subcarrier spacing of the NR. Hereinafter, the specific system or frequency band in which MTC operates is referred to as the MTC narrowband (NB) or MTC subband. In NR, MTC may operate in at least one bandwidth part (BOP) or in a specific band of the BWP.
[0193] MTC can be supported by cells with a bandwidth much larger than 1.08 MHz (e.g., 10 MHz), but the physical channels and signals transmitted and received by the MTC are always limited to 1.08 MHz or 6 (LTE) RBs. For example, in an LTE system, narrowband is defined as 6 non-overlapping consecutive blocks of physical resources in the frequency domain.
[0194] In MTC, some channels of the downlink and uplink may be allocated within a narrowband, and a single channel does not occupy multiple narrowbands in one time unit. FIG. 8 (a) is a figure showing an example of narrowband operation, and FIG. 8 (b) is a figure showing an example of MTC channel repetition with RF retuning.
[0195] The narrowband of the MTC can be set to the UE through system information transmitted by the BS or DCI (downlink control information).
[0196] (2) MTC does not use channels (defined in existing LTE or NR) that must be distributed across the entire system bandwidth of existing LTE or NR. For example, since the existing LTE PDCCH is distributed across the entire system bandwidth and transmitted, the existing PDCCH is not used in MTC. Instead, a new control channel called MPDCCH (MTC PDCCH) is used in MTC. MPDCCH is transmitted / received within up to 6 RBs in the frequency domain. In the time domain, MPDCCH can be transmitted using one or more OFDM symbols, starting from the OFDM symbol with the starting OFDM symbol index indicated by the RRC parameter from the BS among the OFDM symbols within the subframe.
[0197] (3) In the case of MTC, PBCH, PRACH, MPDCCH, PDSCH, PUCCH, and PUSCH can be transmitted repeatedly. Such repeated transmission of MTC allows the MTC channel to be decoded even in environments with very poor signal quality or power, such as basements, which can lead to increased cell radius and signal penetration effects.
[0198] MTC Operation Mode and Level
[0199] MTC uses two operation modes (CE Mode A, CE Mode B) and four different CE levels for coverage enhancement (CE), as shown in Table 2 below.
[0200]
[0201] The MTC operation mode is determined by the BS, and the CE level is determined by the MTC UE. MTC protection period (guard period)
[0202] The location of the narrowband used in the MTC may vary for each specific time unit (e.g., subframe or slot). The MTC UE can be tuned to different frequencies depending on the time units. Frequency retuning requires a certain amount of time, and this time can be used as the guard period of the MTC. During the guard period, no transmission or reception occurs.
[0203] MTC signal transmission / reception method
[0204] The signal transmission / reception process in the MTC is similar to the process in Fig. 2, except for matters specific to the MTC. The process described in S201 of Fig. 2 can also be performed in the MTC. The PSS / SSS used for the initial cell search operation of the MTC may be the PSS / SSS of existing LTE.
[0205] After synchronizing with the BS using PSS / SSS, the MTC UE can receive a PBCH signal from the BS to obtain in-cell broadcast information. The broadcast information transmitted via the PBCH is an MIB. In the case of the MTC, reserved bits among the bits in the existing LTE MIB are used to transmit scheduling information for a new SIB1-BR (system information block for bandwidth reduced device). The scheduling information for the SIB1-BR may include information regarding the number of iterations for the PDSCH carrying the SIB1-BR and information regarding the transport block size (TBS). The frequency resource allocation for the PDSCH carrying the SIB-BR may be a set of six consecutive RBs within the narrowband. The SIB-BR is transmitted directly over the PDSCH without a control channel (e.g., PDCCH, MPDDCH) associated with the SIB-BR.
[0206] After completing the initial cell search, the MTC UE receives MPDCCH and PDSCH based on the MPDCCH information to obtain more specific system information (S202).
[0207] Subsequently, the MTC UE may perform a RACH process to complete the connection to the BS (S203 ~ S206). Basic configurations related to the MTC UE's RACH process may be transmitted via SIB2. Additionally, SIB2 includes parameters related to paging. In the 3GPP system, a Paging Occasion (PO) refers to a time unit in which a UE can attempt to receive paging. Paging means that the network notifies the UE that it has data to transmit. The MTC UE attempts to receive an MPDCCH based on the P-RNTI within the time unit corresponding to its PO on the narrowband (PNB) configured for paging. A UE that successfully decodes the MPDCCH based on the P-RNTI receives the PDSCH scheduled by the MPDCCH and can confirm the paging message for itself. If there is a paging message for itself, it performs a RACH process to establish a connection to the network.
[0208] Signals and / or messages (Msg1, Msg2, Msg3, Msg4) transmitted during the RACH process in MTC may be transmitted repeatedly, and this repetition pattern is set differently depending on the CE level.
[0209] PRACH resources for different CE levels are signaled by the BS for random access. Different PRACH resources for up to four CE levels may be signaled to the MTC UE, respectively. The MTC UE estimates the RSRP using a downlink RS (e.g., CRS, CSI-RS, TRS, etc.) and determines one of the CE levels signaled by the BS based on the measurement result. Based on the determined CE level, the UE selects one of the different PRACH resources for random access (e.g., frequency, time, and preamble resources for PRACH) and performs PRACH transmission. The BS can determine the CE level of the UE based on the PRACH resource used by the UE for PRACH transmission. The BS can determine the CE mode for the UE based on the CE level announced by the UE through PRACH transmission. The BS can transmit DCI to the UE according to the CE mode for the UE.
[0210] The search spaces for RAR and contention resolution messages for PRACH are also signaled by BS through system information.
[0211] The MTC UE that has performed the process described above may subsequently perform the reception of MPDCCH signals and / or PDSCH signals (S207) and the transmission of physical uplink shared channel (PUSCH) signals and / or physical uplink control channel (PUCCH) signals (S208) as a general uplink / downlink signal transmission process. The MTC UE may transmit UCI to the BS via PUCCH or PUSCH.
[0212] When an RRC connection to the MTC UE is established, the MTC UE attempts to receive the MDCCH by monitoring the MPDCCH in the search space configured to obtain uplink and downlink data allocations.
[0213] In legacy LTE, PDSCH is scheduled using PDCCH. Specifically, PDCCH can be transmitted in the first N OFDM symbols of a subframe (SF) (N=1 to 3), and PDSCH scheduled by said PDCCH is transmitted in the same subframe.
[0214] Unlike in conventional LTE, in the case of MTC, the MPDCCH and the PDSCH scheduled by the MPDCCH are transmitted / received in different subframes. For example, the MPDCCH having its last iteration in subframe #n schedules the PDSCH starting in subframe #n+2. The MPDCCH may be transmitted only once or repeatedly. The maximum number of iterations of the MPDCCH is set to the UE by RRC signaling from the BS. The DCI transmitted by the MPDCCH provides information on how many times the MPDCCH is repeated so that the MTC UE can know when the PDSCH transmission begins. For example, if the DCI within the MPDCCH that started transmission from subframe #n contains information that the MPDCCH is repeated 10 times, the last subframe in which the MPDCCH is transmitted is subframe #n+9, and the transmission of the PDSCH may start in subframe #n+11. The DCI transmitted by the MPDCCH may include information regarding the number of repetitions of the physical data channel (e.g., PUSCH, PDSCH) scheduled by the DCI. The UE may repeatedly transmit / receive the physical data channel in the time domain according to the information regarding the number of repetitions for the physical data channel scheduled by the DCI. The PDSCH may be scheduled in the same or a different narrowband as the narrowband where the MPDCCH scheduling the PDSCH is located. If the MPDCCH and the corresponding PDSCH are located in different narrowbands, the MTC UE needs to retun the frequency to the narrowband where the PDSCH is located before decoding the PDSCH. For uplink scheduling, the same timing as legacy LTE may be followed. For example, an MPDCCH with the last transmission in subframe #n may schedule a PUSCH transmission starting in subframe #n+4.When repetitive transmission is applied to a physical channel, frequency hopping between different MTC subbands is supported by RF retuning. For example, when PDSCH is repetitively transmitted in 32 subframes, PDSCH may be transmitted in the first MTC subband in the first 16 subframes and in the second MTC subband in the remaining 16 subframes. The MTC may operate in semi-duplex mode.
[0215] NB-IoT (Narrowband-Internet of Things)
[0216] NB-IoT can refer to a system designed to support low complexity, low power consumption, and efficient use of frequency resources through a system bandwidth (system BW) corresponding to a single resource block (RB) of a wireless communication system (e.g., LTE system, NR system, etc.). NB-IoT can operate in semi-duplex mode. NB-IoT may also be used as a communication method to implement IoT (i.e., the Internet of Things) by supporting devices (or UEs), such as machine-type communication (MTC), in a cellular system.
[0217] In the case of NB-IoT, since each UE recognizes one resource block (RB) as one carrier, the RB and carrier mentioned in this specification in relation to NB-IoT may be interpreted as having the same meaning.
[0218] Hereinafter, the frame structure, physical channel, multi-carrier operation, general signal transmission / reception, etc. related to NB-IoT in this specification are described with consideration of the case of existing LTE systems, but it goes without saying that they can also be applied to next-generation systems (e.g., NR systems, etc.). In addition, the content related to NB-IoT in this specification may also be applied to MTCs that aim for similar technical objectives (e.g., low power, low cost, improved coverage, etc.).
[0219] NB-IoT Frame Structure and Physical Resources
[0220] The NB-IoT frame structure can be configured differently depending on the subcarrier spacing. For example, the NB-IoT frame structure for a 15 kHz subcarrier spacing can be configured to be the same as the frame structure of a legacy system (e.g., LTE system).
[0221] For example, a 10ms NB-IoT frame may contain 10 1ms NB-IoT subframes, and a 1ms NB-IoT subframe may contain 2 0.5ms NB-IoT slots. Additionally, each 0.5ms NB-IoT may contain 7 OFDM symbols.
[0222] As another example, for a BWP or cell / carrier with a 3.75 kHz subcarrier spacing, a 10 ms NB-IoT frame contains five 2 ms NB-IoT subframes, and a 2 ms NB-IoT subframe may contain seven OFDM symbols and one guard period (GP).
[0223] In addition, the above 2ms NB-IoT subframe may also be represented by an NB-IoT slot or an NB-IoT resource unit (RU), etc. The NB-IoT frame structure is not limited to 15kHz and 3.75kHz, and it goes without saying that NB-IoT for other subcarrier intervals (e.g., 30kHz, etc.) can also be considered with different time / frequency units.
[0224] The physical resources of the NB-IoT downlink can be configured by referring to the physical resources of other wireless communication systems (e.g., LTE systems, NR systems, etc.), except that the system bandwidth is limited to a specific number of RBs (e.g., 1 RB, i.e., 180 kHz).
[0225] For example, as described above, if the NB-IoT downlink supports only a 15 kHz subcarrier spacing, the physical resources of the NB-IoT downlink can be set to a resource region in which the resource grid exemplified in Fig. 1 described above is limited to one RB in the frequency domain.
[0226] In the case of physical resources for the NB-IoT uplink, the system bandwidth can also be configured to be limited to one RB, just as in the case of the downlink. In NB-IoT, the number of subcarriers in the uplink band and slot period It can be given as shown in Table 3 below.
[0227] In the case of NB-IoT in LTE systems, the slot period of one slot It is defined by 7 SC-FDMA symbols in the time domain.
[0228]
[0229] In NB-IoT, resource units (RUs) are used for mapping to resource elements of PUSCH for NB-IoT (hereinafter, NPUSCH). In the time domain, RUs... It consists of SC-FDMA symbols and in the frequency domain It can be composed of consecutive subcarriers. For example, and For cell / carrier with a frame structure for FDD, it is given by Table 4 below, and for cell / carrier with a frame structure for TDD, it can be given by Table 5.
[0230]
[0231]
[0232] NB-IoT physical channels
[0233] The OFDMA method can be applied to the NB-IoT downlink based on a subcarrier spacing of 15 kHz. This provides orthogonality between subcarriers, thereby efficiently supporting coexistence with other systems (e.g., LTE systems, NR systems). To distinguish the downlink physical channels / signals of the NB-IoT system from existing systems, they may be expressed with 'N (Narrowband)' added. For example, downlink physical channels may be referred to as NPBCH, NPDCCH, NPDSCH, etc., and downlink physical signals may be referred to as NPSS, NSSS, NRS (Narrowband Reference Signal), NPRS (Narrowband Positioning Reference Signal), NWUS (Narrowband Wake Up Signal), etc. In the case of downlink channels of the NB-IoT system, such as NPBCH, NPDCCH, and NPDSCH, repetition transmission may be performed to improve coverage. In addition, NB-IoT uses a newly defined DCI format, and for example, the DCI format for NB-IoT can be defined as DCI format N0, DCI format N1, DCI format N2, etc.
[0234] SC-FDMA can be applied to the NB-IoT uplink based on a subcarrier spacing of 15 kHz or 3.75 kHz. As mentioned in the downlink section, the physical channel of the NB-IoT system may be expressed with 'N (Narrowband)' added to distinguish it from existing systems. For example, the uplink physical channel may be expressed as NPRACH and NPUSCH, and the uplink physical signal may be expressed as NDMRS. NPUSCH can be divided into NPUSCH Format 1 and NPUSCH Format 2. For instance, NPUSCH Format 1 is used for uplink shared channel (UL-SCH) transmission (or transport), while NPUSCH Format 2 may be used for UCI transmission, such as HARQ ACK signaling. In the case of NPRACH, which is the uplink channel of the NB-IoT system, repetitive transmission may be performed to improve coverage. In this case, iterative transmission may be performed by applying frequency hopping.
[0235] Multi-carrier operation of NB-IoT
[0236] NB-IoT can operate in a multi-carrier mode. Multi-carrier operation may mean that multiple carriers configured for different purposes (i.e., different types) are used when the BS and / or UE transmit / receive channels and / or signals to / from each other in NB-IoT.
[0237] In the multi-carrier mode of NB-IoT, carriers can be classified into anchor type carriers (i.e., anchor carrier, anchor PRB) and non-anchor type carriers (i.e., non-anchor carrier, non-anchor PRB).
[0238] An anchor carrier may refer to a carrier that transmits NPSS, NSSS, NPBCH, and NPDSCH for the System Information Block (N-SIB) for initial access from the perspective of BS. That is, in NB-IoT, the carrier for initial access is referred to as the anchor carrier, and the others may be referred to as non-anchor carriers.
[0239] NB-IoT signal transmission / reception process
[0240] The signal transmission / reception process in NB-IoT is similar to the process of FIG. 2, except for matters specific to NB-IoT. Referring to FIG. 2, an NB-IoT UE that is turned on again after being turned off or has newly entered a cell can perform an initial cell search (S201). To do this, the NB-IoT UE receives NPSS and NSSS from the BS to synchronize with the BS and obtain information such as cell ID. Additionally, the NB-IoT UE can receive NPBCH from the BS to obtain broadcast information within the cell.
[0241] After completing the initial cell search, the NB-IoT UE can obtain more specific system information by receiving the NPDCCH and the corresponding NPDSCH (S202). In other words, the BS can transmit the NPDCCH and the corresponding NPDSCH to the NB-IoT UE after completing the initial cell search to provide more specific system information.
[0242] Subsequently, the NB-IoT UE can perform a RACH process to complete the connection to the BS (S203 to S206). Specifically, the NB-IoT UE can transmit a preamble to the BS via NPRACH (S203), and as described above, NPRACH can be configured to be transmitted repeatedly based on frequency hopping, etc., to improve coverage. In other words, the BS can receive the preamble (repeatedly) from the NB-IoT UE via NPRACH. Subsequently, the NB-IoT UE can receive a RAR for the preamble from the BS via NPDCCH and the corresponding NPDSCH (S204). In other words, the BS can transmit a RAR for the preamble to the NB-IoT UE via NPDCCH and the corresponding NPDSCH. Afterwards, the NB-IoT UE can transmit NPUSCH to BS using scheduling information in RAR (S205), and receive NPDCCH and the corresponding NPDSCH to perform a collision resolution procedure (S206).
[0243] The NB-IoT UE that has performed the process described above can subsequently perform NPDCCH / NPDSCH reception (S207) and NPUSCH transmission (S208) as a general uplink / downlink signal transmission process. In other words, after performing the processes described above, the BS can perform NPDCCH / NPDSCH transmission and NPUSCH reception as a general signal transmission / reception process to the NB-IoT UE.
[0244] In the case of NB-IoT, as mentioned earlier, NPBCH, NPDCCH, NPDSCH, etc., may be transmitted repeatedly to improve coverage. Additionally, in the case of NB-IoT, UL-SCH (i.e., general uplink data) and UCI may be transmitted via NPUSCH. At this time, UL-SCH and UCI may be configured to be transmitted through different NPUSCH formats (e.g., NPUSCH format 1, NPUSCH format 2, etc.).
[0245] In NB-IoT, UCI can generally be transmitted via NPUSCH. Additionally, depending on the request / instruction of a network (e.g., BS), the UE can transmit UCI via NPUSCH periodically, aperiodisically, or semi-persistently.
[0246] wireless communication device
[0247] FIG. 9 illustrates a block diagram of a wireless communication system to which the methods proposed in this specification can be applied.
[0248] Referring to FIG. 9, the wireless communication system includes a first communication device (910) and / or a second communication device (920). 'A and / or B' can be interpreted as having the same meaning as 'including at least one of A or B'. The first communication device may represent BS and the second communication device may represent UE (or the first communication device may represent UE and the second communication device may represent BS).
[0249] The first communication device and the second communication device include a processor (911, 921), memory (914, 924), one or more Tx / Rx RF modules (radio frequency modules, 915, 925), a Tx processor (912, 922), an Rx processor (913, 923), and an antenna (916, 926). The Tx / Rx module is also called a transceiver. The processor implements the functions, processes, and / or methods described above. More specifically, in DL (communication from the first communication device to the second communication device), upper layer packets from the core network are provided to the processor (911). The processor implements the functions of the Layer 2 (i.e., L2) layer. In DL, the processor provides the second communication device (920) with multiplexing between the logical channel and the transmission channel, radio resource allocation, and is responsible for signaling to the second communication device. The transmission (TX) processor (912) implements various signal processing functions for the L1 layer (i.e., the physical layer). These signal processing functions facilitate forward error correction (FEC) in the second communication device and include coding and interleaving. The signal that has undergone encoding and interleaving undergoes scrambling and modulation to be modulated into complex valued modulation symbols. Depending on the channel, BPSK, QPSK, 16QAM, 64QAM, 246QAM, etc. may be used for modulation. The complex valued modulation symbols (hereinafter referred to as modulation symbols) are divided into parallel streams, each stream is mapped to an OFDM subcarrier, multiplexed with a reference signal in the time and / or frequency domain, and combined together using IFFT to create a physical channel carrying a time-domain OFDM symbol stream. The OFDM symbol stream is spatially precoded to generate a multi-spatial stream.Each spatial stream may be provided to different antennas (916) via individual Tx / Rx modules (or transceivers, 915). Each Tx / Rx module may upconvert each spatial stream to an RF carrier for transmission. In the second communication device, each Tx / Rx module (or transceiver, 925) receives a signal of the RF carrier via each antenna (926) of each Tx / Rx module. Each Tx / Rx module restores the signal of the RF carrier to a baseband signal and provides it to a receive (RX) processor (923). The RX processor implements various signal processing functions of L1 (i.e., the physical layer). The RX processor may perform spatial processing on information to recover any spatial stream destined for the second communication device. If multiple spatial streams are destined for the second communication device, they may be combined into a single OFDMA symbol stream by multiple RX processors. The RX processor uses a Fast Fourier Transform (FFT) to convert the OFDM symbol stream, which is a time-domain signal, into a frequency-domain signal. The frequency-domain signal contains individual OFDM symbol streams for each subcarrier of the OFDM signal. The modulation symbols and reference signals on each subcarrier are restored and demodulated by determining the most likely signal constellation points transmitted by the first communication device. These soft decisions may be based on channel estimate values. The soft decisions are decoded and deinterleaved to restore the data and control signals originally transmitted by the first communication device on the physical channel. The corresponding data and control signals are provided to the processor (921).
[0250] UL (communication from the second communication device to the first communication device) is processed in the first communication device (910) in a manner similar to that described in relation to the receiver function in the second communication device (920). Each Tx / Rx module (925) receives a signal through each antenna (926). Each Tx / Rx module provides RF carriers and information to the RX processor (923). The processor (921) may be associated with a memory (924) that stores program code and data. The memory may be referred to as a computer-readable medium.
[0251] Artificial Intelligence (AI)
[0252] Artificial intelligence refers to the field of researching artificial intelligence or the methodologies to create it, while machine learning refers to the field of researching methodologies to define and solve various problems addressed within the field of artificial intelligence. Machine learning is also defined as an algorithm that improves performance on a task through continuous experience.
[0253] An Artificial Neural Network (ANN) is a model used in machine learning that can refer to any model capable of problem-solving, composed of artificial neurons (nodes) that form a network through the connection of synapses. An artificial neural network can be defined by connection patterns between neurons in different layers, a learning process that updates model parameters, and an activation function that generates output values.
[0254] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron may output a function value of an activation function for input signals, weights, and biases input through the synapses.
[0255] Model parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters, on the other hand, refer to parameters that must be set prior to training in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.
[0256] The objective of training an artificial neural network can be viewed as determining model parameters that minimize the loss function. The loss function can be used as an indicator to determine optimal model parameters during the training process of an artificial neural network.
[0257] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.
[0258] Supervised learning refers to a method of training an artificial neural network with labels provided for the training data; a label can refer to the correct answer (or result) that the neural network must infer when the training data is input. Unsupervised learning refers to a method of training an artificial neural network without labels provided for the training data. Reinforcement learning refers to a learning method in which an agent defined within an environment is trained to select an action or sequence of actions that maximizes the cumulative reward in each state.
[0259] Machine learning implemented using a Deep Neural Network (DNN) that includes multiple hidden layers among artificial neural networks is also called Deep Learning, and Deep Learning is a part of Machine Learning. Hereinafter, Machine Learning is used in a sense that includes Deep Learning.
[0260] Robot
[0261] A robot can refer to a machine that automatically processes or operates a given task based on its own capabilities. In particular, a robot that has the ability to perceive its environment, make decisions on its own, and perform actions can be called an intelligent robot.
[0262] Robots can be classified into industrial, medical, household, military, etc., depending on their purpose of use or field.
[0263] A robot can perform various physical movements, such as moving robot joints, by being equipped with a drive unit including actuators or motors. Additionally, a mobile robot may include wheels, brakes, propellers, etc., in the drive unit, enabling it to drive on the ground or fly in the air.
[0264] Self-Driving
[0265] Autonomous driving refers to technology that drives itself, and an autonomous vehicle refers to a vehicle that drives without user intervention or with minimal user intervention.
[0266] For example, autonomous driving can include technologies such as maintaining the driving lane, automatically adjusting speed like adaptive cruise control, automatically driving along a set route, and automatically setting a route and driving once a destination is set.
[0267] Vehicles encompass vehicles equipped solely with internal combustion engines, hybrid vehicles equipped with both internal combustion engines and electric motors, and electric vehicles equipped solely with electric motors, and may include not only automobiles but also trains, motorcycles, etc.
[0268] In this case, an autonomous vehicle can be viewed as a robot with autonomous driving capabilities.
[0269] Extended Reality (XR)
[0270] Extended Reality is a collective term for Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). VR technology provides real-world objects or backgrounds solely as CG images, AR technology provides virtual CG images superimposed on real-world images, and MR technology is a computer graphics technology that mixes and combines virtual objects with the real world.
[0271] MR technology is similar to AR technology in that it displays real-world objects and virtual objects together. However, there is a difference in that while virtual objects in AR technology are used to complement real-world objects, virtual objects and real-world objects are used as equals in MR technology.
[0272] XR technology can be applied to HMDs (Head-Mount Displays), HUDs (Head-Up Displays), mobile phones, tablet PCs, laptops, desktops, TVs, digital signage, etc., and devices to which XR technology is applied can be called XR devices.
[0273] FIG. 10 shows an AI device (1000) according to one embodiment of the present invention.
[0274] The AI device (1000) illustrated in FIG. 10 can be implemented as a stationary device or a mobile device, such as a TV, projector, mobile phone, smartphone, desktop computer, laptop, digital broadcasting terminal, PDA (personal digital assistants), PMP (portable multimedia player), navigation, tablet PC, wearable device, set-top box (STB), DMB receiver, radio, washing machine, refrigerator, desktop computer, digital signage, robot, vehicle, etc.
[0275] Referring to FIG. 10, the AI device (1000) may include a communication unit (1010), an input unit (1020), a learning processor (1030), a sensing unit (1040), an output unit (1050), a memory (1070), and a processor (1080), etc.
[0276] The communication unit (1010) can transmit and receive data with external devices, such as other AI devices or AI servers, using wired or wireless communication technology. For example, the communication unit (1010) can transmit and receive sensor information, user input, learning models, control signals, etc., with external devices.
[0277] At this time, the communication technologies used by the communication unit (1010) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth™), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc. In particular, the 5G technology described above in the previous FIGS. 1 to 9 may also be applied.
[0278] The input unit (1020) can acquire various types of data. At this time, the input unit (1020) may include a camera for inputting video signals, a microphone for receiving audio signals, a user input unit for receiving information from a user, etc. Here, the camera or microphone may be treated as a sensor, and the signal acquired from the camera or microphone may be referred to as sensing data or sensor information.
[0279] The input unit (1020) can obtain training data for model training and input data to be used when obtaining an output using a training model. The input unit (1020) may also obtain unprocessed input data, in which case the processor (1080) or the learning processor (1030) can extract input feature points as a preprocessing step for the input data.
[0280] The learning processor (1030) can train a model composed of an artificial neural network using training data. Here, the trained artificial neural network may be referred to as a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for judgment to perform an action.
[0281] At this time, the learning processor (1030) can perform AI processing together with the learning processor of the AI server.
[0282] At this time, the learning processor (1030) may include memory integrated into or implemented in the AI device (1000). Alternatively, the learning processor (1030) may be implemented using memory (1070), external memory directly coupled to the AI device (1000), or memory maintained in an external device.
[0283] The sensing unit (1040) can acquire at least one of internal information of the AI device (1000), surrounding environment information of the AI device (1000), and user information using various sensors.
[0284] At this time, the sensors included in the sensing unit (1040) include a proximity sensor, an illuminance sensor, an accelerometer, a magnetic sensor, a gyroscope, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar, a radar, etc.
[0285] The output unit (1050) can generate output related to sight, hearing, or touch.
[0286] At this time, the output unit (1050) may include a display unit that outputs visual information, a speaker that outputs auditory information, a haptic module that outputs tactile information, etc.
[0287] The memory (1070) can store data that supports various functions of the AI device (1000). For example, the memory (1070) can store input data, training data, training models, training history, etc. obtained from the input unit (1020).
[0288] The processor (1080) can determine at least one executable action of the AI device (1000) based on information determined or generated using a data analysis algorithm or a machine learning algorithm. The processor (1080) can perform the determined action by controlling the components of the AI device (1000).
[0289] To this end, the processor (1080) can request, search, receive, or utilize data from the learning processor (1030) or memory (1070), and can control the components of the AI device (1000) to execute a predicted operation or a preferred operation among the at least one executable operation.
[0290] At this time, if the processor (1080) requires the connection of an external device to perform a determined operation, it can generate a control signal to control the external device and transmit the generated control signal to the external device.
[0291] The processor (1080) can obtain intent information regarding user input and determine the user's requirements based on the obtained intent information.
[0292] At this time, the processor (1080) can obtain intent information corresponding to the user input by using at least one of a Speech To Text (STT) engine for converting voice input into a string or a Natural Language Processing (NLP) engine for obtaining intent information of natural language.
[0293] At this time, at least one of the STT engine or NLP engine may be composed of an artificial neural network in which at least a portion is learned according to a machine learning algorithm. Also, at least one of the STT engine or NLP engine may be learned by a learning processor (1030), learned by a learning processor of an AI server, or learned by distributed processing thereof. For reference, specific components of the AI server are illustrated in detail in FIG. 11 below.
[0294] The processor (1080) can collect history information, including the operation details of the AI device (1000) or user feedback regarding the operation, and store it in memory (1070) or a learning processor (1030), or transmit it to an external device such as an AI server. The collected history information can be used to update a learning model.
[0295] The processor (1080) can control at least some of the components of the AI device (1000) to run an application stored in memory (1070). Furthermore, the processor (1080) can operate two or more of the components included in the AI device (1000) in combination with each other to run the application.
[0296] FIG. 11 shows an AI server (1120) according to one embodiment of the present invention.
[0297] Referring to FIG. 11, the AI server (1120) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network. Here, the AI server (1120) may be composed of multiple servers to perform distributed processing, or it may be defined as a 5G network. At this time, the AI server (1120) may be included as part of the configuration of the AI device (1100) to perform at least part of the AI processing together.
[0298] The AI server (1120) may include a communication unit (1121), memory (1123), a learning processor (1124), and a processor (1126), etc.
[0299] The communication unit (1121) can transmit and receive data with external devices such as AI devices (1100).
[0300] The memory (1123) may include a model storage unit (1124). The model storage unit (1124) may store a model (or artificial neural network, 1125) that is being learned or has been learned through the learning processor (1124).
[0301] The learning processor (1124) can train an artificial neural network (1125) using training data. The training model may be used while mounted on the AI server (1120) of the artificial neural network, or it may be used while mounted on an external device such as an AI device (1100).
[0302] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (1123).
[0303] The processor (1126) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.
[0304] FIG. 12 shows an AI system according to one embodiment of the present invention.
[0305] Referring to FIG. 12, the AI system is connected to a cloud network (1210) at least one of an AI server (1260), a robot (1210), an autonomous vehicle (1220), an XR device (1230), a smartphone (1240), or a home appliance (1250). Here, a robot (1210), an autonomous vehicle (1220), an XR device (1230), a smartphone (1240), or a home appliance (1250) to which AI technology is applied may be referred to as an AI device.
[0306] A cloud network (1210) may refer to a network that constitutes part of a cloud computing infrastructure or exists within a cloud computing infrastructure. Here, the cloud network (1210) may be configured using a 3G network, a 4G or LTE (Long Term Evolution) network, or a 5G network, etc.
[0307] That is, each device (1210 to 1260) constituting the AI system can be connected to each other through a cloud network (1210). In particular, each device (1210 to 1260) may communicate with each other through a base station, but may also communicate directly with each other without going through a base station.
[0308] The AI server (1260) may include a server that performs AI processing and a server that performs operations on big data.
[0309] The AI server (1260) is connected via a cloud network (1210) to at least one of the AI devices constituting the AI system, such as a robot (1210), an autonomous vehicle (1220), an XR device (1230), a smartphone (1240), or a home appliance (1250), and can assist in at least some of the AI processing of the connected AI devices (1210 to 1250).
[0310] At this time, the AI server (1260) can train an artificial neural network according to a machine learning algorithm on behalf of the AI device (1210 to 1250), and can directly save the training model or transmit it to the AI device (1210 to 1250).
[0311] At this time, the AI server (1260) receives input data from the AI device (1210 to 1250), infers a result value for the received input data using a learning model, and generates a response or control command based on the inferred result value and transmits it to the AI device (1210 to 1250).
[0312] Alternatively, the AI device (1210 to 1250) may use a direct learning model to infer a result value for input data and generate a response or control command based on the inferred result value.
[0313] Hereinafter, various embodiments of AI devices (1210 to 1250) to which the above-described technology is applied will be described. Here, the AI devices (1210 to 1250) illustrated in FIG. 12 can be seen as specific embodiments of the AI device (1000) illustrated in FIG. 10.
[0314] <AI+XR>
[0315] The XR device (1230) may be implemented with AI technology applied, such as a Head-Mount Display (HMD), a Head-Up Display (HUD) equipped in a vehicle, a television, a mobile phone, a smartphone, a computer, a wearable device, a home appliance, digital signage, a vehicle, a stationary robot, or a mobile robot.
[0316] The XR device (1230) can obtain information about surrounding space or real objects by analyzing 3D point cloud data or image data obtained through various sensors or from an external device to generate location data and attribute data for 3D points, and can render and output an XR object to be output. For example, the XR device (1230) can output an XR object containing additional information about a recognized object by corresponding it to the recognized object.
[0317] The XR device (1230) can perform the above-mentioned operations using a learning model composed of at least one artificial neural network. For example, the XR device (1230) can recognize real-world objects in 3D point cloud data or image data using the learning model and can provide information corresponding to the recognized real-world objects. Here, the learning model may be learned directly by the XR device (1230) or learned from an external device such as an AI server (1260).
[0318] At this time, the XR device (1230) may perform an operation by generating a result using a direct learning model, but it may also perform an operation by transmitting sensor information to an external device such as an AI server (1260) and receiving the result generated accordingly.
[0319] <AI+로봇+XR>
[0320] The robot (1210) can be implemented as a guide robot, transport robot, cleaning robot, wearable robot, entertainment robot, pet robot, unmanned flying robot, drone, etc. by applying AI technology and XR technology.
[0321] A robot (1210) to which XR technology is applied may refer to a robot that is the subject of control / interaction within an XR image. In this case, the robot (1210) is distinguished from the XR device (1230) and can be interconnected with it.
[0322] When a robot (1210) that is the subject of control / interaction within an XR image acquires sensor information from sensors including a camera, the robot (1210) or the XR device (1230) can generate an XR image based on the sensor information, and the XR device (1230) can output the generated XR image. Furthermore, the robot (1210) can operate based on a control signal input through the XR device (1230) or user interaction.
[0323] For example, the user can view an XR image corresponding to the viewpoint of the remotely linked robot (1210) through an external device such as an XR device (1230), and through interaction, can adjust the autonomous driving path of the robot (1210), control its movement or driving, or check information about surrounding objects.
[0324] <AI+자율주행+XR>
[0325] The autonomous vehicle (1220) can be implemented as a mobile robot, vehicle, unmanned aerial vehicle, etc. by applying AI technology and XR technology.
[0326] An autonomous vehicle (1220) equipped with XR technology may refer to an autonomous vehicle equipped with means for providing XR images, or an autonomous vehicle that is the subject of control / interaction within the XR images. In particular, the autonomous vehicle (1220) that is the subject of control / interaction within the XR images is distinguished from the XR device (1230) and can be interconnected with it.
[0327] An autonomous vehicle (1220) equipped with means for providing XR images can acquire sensor information from sensors including cameras and output an XR image generated based on the acquired sensor information. For example, the autonomous vehicle (1220) can provide an XR object corresponding to a real object or an object in the screen to the occupant by providing an XR image by outputting an XR image with a HUD.
[0328] At this time, when the XR object is displayed on the HUD, at least a portion of the XR object may be displayed so as to overlap with the actual object to which the passenger's gaze is directed. On the other hand, when the XR object is displayed on a display provided inside the autonomous vehicle (100b), at least a portion of the XR object may be displayed so as to overlap with an object on the screen. For example, the autonomous vehicle (1220) may display XR objects corresponding to objects such as lanes, other vehicles, traffic lights, traffic signs, motorcycles, pedestrians, buildings, etc.
[0329] When an autonomous vehicle (1220) that is the subject of control / interaction within an XR image acquires sensor information from sensors including a camera, the autonomous vehicle (1220) or the XR device (1230) can generate an XR image based on the sensor information, and the XR device (1230) can output the generated XR image. Furthermore, the autonomous vehicle (1220) can operate based on control signals input through an external device such as the XR device (1230) or user interaction.
[0330] The VR (Virtual Reality) technology, AR (Augmented Reality) technology, and MR (Mixed Reality) technology according to the present invention can be applied to various devices, and more specifically, for example, to HMD (Head-Mount Display), HUD (Head-Up Display) attached to a vehicle, mobile phones, tablet PCs, laptops, desktops, TVs, signage, etc. In addition, it can be applied to devices equipped with flexible or rollable displays.
[0331] Furthermore, the aforementioned VR, AR, and MR technologies are implemented based on computer graphics and can be distinguished according to the proportion of CG (Computer Graphic) images in the images displayed in the user's field of vision.
[0332] In other words, VR technology is a display technology that provides real-world objects or backgrounds solely as CG images. On the other hand, AR technology refers to a technology that displays virtual CG images alongside images of real objects.
[0333] Furthermore, MR technology is similar to the aforementioned AR technology in that it mixes and combines virtual objects with the real world. However, while AR technology clearly distinguishes between real-world objects and virtual objects created from CG images and uses virtual objects to complement real-world objects, MR technology is distinguished from AR technology in that virtual objects are considered to be of the same nature as real-world objects. To give a more specific example, the aforementioned MR technology is applied in hologram services.
[0334] However, recently, rather than clearly distinguishing between VR, AR, and MR technologies, they are also referred to as XR (extended Reality) technology. Therefore, the embodiments of the present invention are applicable to all VR, AR, MR, and XR technologies.
[0335] Meanwhile, hardware (HW) related elemental technologies applied to VR, AR, MR, and XR technologies include, for example, wired / wireless communication technology, input interface technology, output interface technology, and computing device technology. In addition, software (SW) related elemental technologies include, for example, tracking and matching technology, speech recognition technology, interaction and user interface technology, location-based service technology, search technology, and AI (Artificial Intelligence) technology.
[0336] In particular, embodiments of the present invention aim to solve at least one of the following problems by utilizing the aforementioned hardware / software related element technologies: communication problems with other devices, inefficient memory usage problems, problems with low data processing speed due to inconvenient UX / UI, video problems, sound problems, motion sickness, or other problems.
[0337] FIG. 13 is a block diagram illustrating an XR device according to embodiments of the present invention. The XR device includes a camera (1310), a display (1320), a sensor (1330), a processor (1340), a memory (1350), and a communication module (1360), etc. Of course, deleting, changing, or adding some modules as needed by those skilled in the art is also within the scope of the present invention.
[0338] The communication module (1360) communicates with an external device or server via wired or wireless means. For short-range wireless communication, examples such as Wi-Fi or Bluetooth may be used, and for long-range wireless communication, examples such as 3GPP communication standards may be used. LTE refers to technology from 3GPP TS 36.xxx Release 8 onwards. Specifically, LTE technology from 3GPP TS 36.xxx Release 10 onwards is referred to as LTE-A, and LTE technology from 3GPP TS 36.xxx Release 13 onwards is referred to as LTE-A pro. 3GPP 5G (5th generation) technology refers to technology from TS 36.xxx Release 15 onwards and technology from TS 38.XXX Release 15 onwards. Among these, technology from TS 38.xxx Release 15 onwards is referred to as 3GPP NR, and technology from TS 36.xxx Release 15 onwards may be referred to as enhanced LTE. "xxx" indicates a specific standard document number. LTE / NR can be collectively referred to as a 3GPP system.
[0339] The camera (1310) can capture the surrounding environment of the XR device (1300) and convert it into an electrical signal. An image captured by the camera (1310) and converted into an electrical signal can be stored in memory (1350) and then displayed on a display (1320) via a processor (1340). Alternatively, the image can be displayed directly on the display (1320) using the processor (1340) without being stored in memory (1350). Additionally, the camera (110) may have a field of view. In this case, the field of view refers to an area where, for example, a real object located around the camera (1310) can be detected. The camera (1310) can detect only real objects located within the field of view. When a real object is located within the field of view of the camera (1310), the XR device (1300) can display an augmented reality object corresponding to the real object. Additionally, the camera (1310) can detect the angle between the camera (1310) and the real object.
[0340] The sensor (1330) may include at least one sensor, for example, a gravity sensor, a geomagnetic sensor, a motion sensor, a gyroscope sensor, an accelerometer sensor, an inclination sensor, a brightness sensor, an altitude sensor, an olfactory sensor, a temperature sensor, a depth sensor, a pressure sensor, a bending sensor, an audio sensor, a video sensor, a GPS (Global Positioning System) sensor, a touch sensor, etc. Furthermore, the display (1320) may be fixed, but can be implemented as an LCD (Liquid Crystal Display), OLED (Organic Light Emitting Diode), ELD (Electro Luminescent Display), or M-LED (Micro LED) to have high flexibility. In this case, the sensor (1330) is designed to detect the degree of bending of the display (1320) implemented as the aforementioned LCD, OLED, ELD, M-LED (Micro LED), etc.
[0341] In addition, the memory (1350) has the function of storing images captured by the camera (1310), as well as the function of storing all or part of the results of wired / wireless communication with an external device or server. In particular, considering the trend of increasing communication data traffic (e.g., in a 5G communication environment), efficient memory management is required. In this regard, it will be described in detail below in FIG. 14.
[0342] FIG. 14 is a block diagram illustrating the memory (1350) illustrated in FIG. 13 in more detail. Hereinafter, with reference to FIG. 14, a swap-out process between RAM and flash memory will be described according to an embodiment of the present invention.
[0343] When the control unit (1430) swaps out AR / VR related page data in the RAM (1410) to the flash memory (1420), among the AR / VR related page data to be swapped out, if there are two or more AR / VR related page data with identical contents, only one can be swapped out to the flash memory (1420).
[0344] That is, the control unit (1430) calculates distinguishing values (e.g., hash functions) that distinguish the contents of the AR / VR related page data to be swapped out, and can determine that the contents of two or more AR / VR page data having the same distinguishing value among the calculated distinguishing values are identical. Accordingly, the problem of unnecessary AR / VR related page data being stored in the flash memory (1420) and the lifespan of the flash memory (1420) as well as the AR / VR device containing it being shortened can be resolved.
[0345] The operation of the above control unit (1430) may be implemented in software form or in hardware form, and this is also within the scope of the present invention. Furthermore, more specifically, the memory shown in FIG. 14 is included in an HMD (Head-Mount Display), vehicle, mobile phone, tablet PC, laptop, desktop, TV, signage, etc., and performs a swap function.
[0346] Meanwhile, the device according to the embodiments of the present invention can process three-dimensional point cloud data and provide various services to the user, such as VR, AR, MR, XR, and autonomous driving services.
[0347] Sensors that collect three-dimensional point cloud data may be, for example, LiDAR (light detection and ranging), RGB-D (Red Green Blue Depth), 3D laser scanners, etc., and said sensors can be mounted inside or outside of HMD (Head-Mount Display), vehicle, mobile phone, tablet PC, laptop, desktop, TV, signage, etc.
[0348] Figure 15 shows a point cloud data processing system.
[0349] The point cloud processing system (1500) illustrated in FIG. 15 includes a transmission device that acquires point cloud data, performs encoding processing, and transmits it, and a receiving device that receives video data, performs decoding processing, and acquires point cloud data. As illustrated in FIG. 15, point cloud data according to embodiments of the present invention may be acquired through a process of capturing, synthesizing, or generating point cloud data (S1510). During the acquisition process, 3D position (x, y, z) / attribute (color, reflectance, transparency, etc.) data for points (e.g., a PLY (Polygon File format or the Stanford Triangle format) file, etc.) may be generated. In the case of a video having multiple frames, one or more files may be acquired. During the capture process, metadata related to point cloud data (e.g., metadata related to the capture, etc.) may be generated. A transmission device or encoder according to embodiments of the present invention may output one or more video streams by encoding point cloud data using a Video-based Point Cloud Compression (V-PCC) or Geometry-based Point Cloud Compression (G-PCC) method (S1520). V-PCC is a method for compressing point cloud data based on 2D video codecs such as HEVC and VVC, and G-PCC is a method for encoding point cloud data by dividing it into two streams: geometry and attribute. The geometry stream may be generated by reconstructing and encoding location information of points, and the attribute stream may be generated by reconstructing and encoding attribute information (e.g., color) associated with each point.In the case of V-PCC, it is compatible with 2D video, but because more data (e.g., geometry video, attribute video, occupancy map video, and auxiliary information) is required to recover V-PCC processed data compared to G-PCC, longer latency may occur when providing services. One or more output bit streams may be encapsulated in the form of a file (e.g., file format such as ISOBMFF) along with related metadata and transmitted via a network or digital storage medium (S1530).
[0350] A device or processor according to embodiments of the present invention can decapsulate received video data to obtain one or more bit streams and related metadata, and decode the obtained bit streams using a V-PCC or G-PCC method to restore three-dimensional point cloud data (S1540). A renderer can render the decoded point cloud data and provide content suitable for VR / AR / MR / service to a user through a display unit (S1550).
[0351] As illustrated in FIG. 15, a device or processor according to embodiments of the present invention may perform a feedback process that transmits various feedback information obtained during the rendering / display process to a transmitting device or to a decoding process (S1560). The feedback information according to embodiments of the present invention may include head orientation information, viewport information indicating the area currently being viewed by the user, etc. Since interaction between the user and the service (or content) provider is achieved through the feedback process, the device according to embodiments of the present invention can not only provide various services with greater user convenience, but also has the technical effect of providing faster data processing speeds or enabling clear video composition by utilizing the aforementioned V-PCC or G-PCC method.
[0352] FIG. 16 shows an XR device (1600) including a learning processor. Since only the learning processor (1670) has been added compared to FIG. 13, other components can be interpreted by referring to FIG. 13, so redundant descriptions are omitted.
[0353] The XR device (160) illustrated in FIG. 16 may be equipped with a learning model. The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (1650).
[0354] A learning processor (1670) according to embodiments of the present invention may be connected to communicate with a processor (1640) and may repeatedly train a model composed of an artificial neural network using training data. An artificial neural network is an information processing system in which a plurality of neurons, referred to as nodes or processing elements, are connected in the form of a layer structure, modeling the operating principles of biological neurons and the relationships between neurons. An artificial neural network is a model used in machine learning and is a statistical learning algorithm in machine learning and cognitive science inspired by biological neural networks (particularly the brain within the central nervous system of animals). The term "machine learning" may be used interchangeably with "machine learning." Machine learning is a field of Artificial Intelligence (AI) that is a technology capable of giving computers the ability to learn without explicit programming. Machine learning is a technology that studies and builds systems and algorithms for learning, making predictions, and improving their own performance based on empirical data. Accordingly, the learning processor (1670) according to the embodiments of the present invention can infer result values for new input data by determining optimized model parameters of the artificial neural network through iterative training of the artificial neural network. Accordingly, the learning processor (1670) can analyze the user's device usage patterns based on the user's device usage history information. In addition, the learning processor (1670) may be configured to receive, classify, store, and output information to be used for data mining, data analysis, intelligent decision making, and machine learning algorithms and technologies.
[0355] A processor (1640) according to embodiments of the present invention can determine or predict at least one executable action of a device based on data analyzed or generated by a learning processor (1670). Additionally, the processor (1640) can request, search, receive, or utilize data from the learning processor (1670) and control the XR device (1600) to execute a predicted action or an action deemed desirable among at least one executable action. The processor (1640) according to embodiments of the present invention can perform various functions that implement intelligent emulation (i.e., knowledge-based systems, inference systems, and knowledge acquisition systems). This can be applied to various types of systems (e.g., fuzzy logic systems), including adaptive systems, machine learning systems, artificial neural networks, etc. That is, the processor (1640) can predict future user device usage patterns based on data analyzed from the learning processor (1670) regarding the user's device usage patterns, and control the XR device (1600) so that it can provide a more suitable XR service to the user. Here, XR services include at least one of AR services, VR services, and MR services.
[0356] FIG. 17 illustrates the process of the XR device (1600) of the present invention shown in FIG. 16 providing XR services.
[0357] A processor (1670) according to embodiments of the present invention may store user's device usage history information in memory (1650) (S1710). The device usage history information may include information such as the name, category, and content of content provided to the user, information on the time the device was used, information on the location where the user used the device, time information, and information on the use of applications installed on the device.
[0358] A learning processor (1670) according to embodiments of the present invention can obtain user's device usage pattern information by analyzing device usage history information (S1720). For example, when an XR device (1600) provides specific content A to a user, the learning processor (1670) can learn the pattern information of the user using content A on the device by comprehensively combining specific information about content A (e.g., age information regarding users who mainly use content A, content information of content A, content information similar to content A, etc.) and information such as the time, place, and number of times the user using the terminal consumed specific content A.
[0359] A processor (1640) according to embodiments of the present invention may acquire user device pattern information generated based on information learned from a learning processor (16470) and generate device usage pattern prediction information (S1730). Additionally, the processor (1640) may, for example, instruct the device (1600) to operate when the user is not using the device (1640) and it is determined that the user is in a place where the user frequently used the device (1640) or when it is close to the time when the user mainly uses the device (1640). In this case, the device according to embodiments of the present invention may provide AR content based on the user pattern prediction information (S1740).
[0360] Additionally, when a user uses the device (1600), the processor (1640) identifies information about the content currently being provided to the user and can generate information predicting the user's device usage pattern related to the content (e.g., when the user requests other related content or requests additional data related to the current content). Additionally, the processor (1640) can direct the operation of the device (1600) to provide AR content based on the user pattern prediction information (S1740). AR content according to embodiments of the present invention may include advertisements, navigation information, risk information, etc.
[0361] Figure 18 illustrates the appearance of an XR device and a robot.
[0362] As the configuration modules of the device (18000) equipped with XR technology according to one embodiment of the present invention have been described in detail in the previous drawings, redundant descriptions are omitted.
[0363] The appearance of the robot (1810) shown in FIG. 18 is merely an example, and the robot of the present invention can be implemented with various appearances. For example, the robot (1810) shown in FIG. 18 can be a drone, a vacuum cleaner, a cooking robot, a wearable robot, etc., and in particular, each component can be placed in different positions, such as up, down, left, right, front, and back, depending on the shape of the robot.
[0364] The robot may have various sensors placed on the exterior of the robot (1810) to identify external objects. Additionally, the robot has an interface unit (1811) placed on the top or rear (1812) of the robot (1810) to provide specific information to the user.
[0365] A robot control module (1850) is installed inside the robot (1810) to control the robot by detecting the robot's movement and surrounding objects. The robot control module (1850) can be implemented as a software module or a chip that implements it in hardware. The robot control module (1850) may further include, in detail, a deep learning unit (1851), a sensing information processing unit (1852), a movement path generation unit (1853), a communication module (1854), etc.
[0366] The sensing information processing unit (1852) collects and processes information sensed by various types of sensors (lidar sensor, infrared sensor, ultrasonic sensor, depth sensor, image sensor, microphone, etc.) placed on the robot (1810).
[0367] The deep learning unit (1851) can input information processed by the sensing information processing unit (1852) or information accumulated and stored by the robot (1810) during the movement process, and output results necessary for the robot (1810) to determine external conditions, process information, or generate a movement path.
[0368] The movement path generation unit (1853) can calculate the movement path of the robot using data produced by the deep learning unit (1851) or data processed by the sensing information processing unit (1852).
[0369] However, since both the device (1800) equipped with XR technology and the robot (1810) have communication modules, it is possible to transmit and receive data through short-range wireless communication such as Wi-Fi or Bluetooth, or long-range wireless communication such as 5G. The technology for controlling the robot (1810) using the device (1800) equipped with XR technology will be described below in FIG. 19.
[0370] FIG. 19 is a flowchart illustrating the process of controlling a robot using a device equipped with XR technology.
[0371] First, devices and robots equipped with XR technology are connected to a 5G network (S1901). Of course, transmitting and receiving data to and from each other through other short-range and long-range communication technologies is also within the scope of the present invention.
[0372] The robot captures an image or video of the robot's surroundings using at least one camera installed inside or outside (S1902) and transmits the captured image / video to an XR device (S1903). The XR device displays the captured image / video (S1904) and transmits a command to the robot to control the robot (S1905). The command may be manually entered by a user of the XR device, or it may be automatically generated through AI (Artificial Intelligent) technology, which falls within the scope of the present invention.
[0373] The robot executes the corresponding function according to the command received in step S405 (S1906) and transmits the result value to the XR device (S1907). The result value may be an indicator regarding the success or failure of normal data processing, currently captured images / videos, or specific data considering the XR device. The specific data is designed to change, for example, depending on the state of the XR device. If the display of the XR device is off, a command to turn on the display of the XR device is included in step S1907. Therefore, in the event of an emergency situation near the robot, there is a technical effect that a notification message can be delivered even if the display of the remote XR device is turned off.
[0374] And, according to the result value received in step S1907 above, AR / VR related content is displayed (S1908).
[0375] Additionally, according to another embodiment of the present invention, it is also possible to display the robot's location information on an XR device using a GPS module attached to the robot.
[0376] The XR device (1300) described in FIG. 13 can be connected to a vehicle providing autonomous driving services to enable wired / wireless communication, or can be installed in a vehicle providing autonomous driving services. Therefore, various services including AR / VR can be provided in a vehicle providing autonomous driving services.
[0377] Figure 20 shows a vehicle providing autonomous driving services.
[0378] A vehicle (2010) according to embodiments of the present invention may include automobiles, trains, and motorcycles as means of transport traveling on roads or tracks. A vehicle (2010) according to embodiments of the present invention may include all of the following: an internal combustion engine vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, and an electric vehicle equipped with an electric motor as a power source.
[0379] A vehicle (2010) according to embodiments of the present invention may include the following components to control the operation of the vehicle: a user interface device, an object detection device, a communication device, a driving control device, a main ECU, a driving control device, an autonomous driving device (260), a sensing unit and a position data generation device;
[0380] The object detection device, communication device, driving control device, main ECU, drive control device, autonomous driving device, sensing unit, and position data generation device can each be implemented as electronic devices that generate electrical signals and exchange electrical signals with one another.
[0381] A user interface device receives user input and can provide information generated from the vehicle (2010) to the user in the form of a User Interface (UI) or User Experience (UX). The user interface device may include an input / output device and a user monitoring device. An object detection device can detect the presence or absence of an object outside the vehicle (2010) and generate information about the object. The object detection device may include, for example, at least one of a camera, a lidar, an infrared sensor, and an ultrasonic sensor. The camera may generate object information outside the vehicle (2010) based on images. The camera may include one or more lenses, one or more image sensors, and one or more processors for generating object information. The camera may acquire location information of an object, distance information from the object, or relative speed information from the object by using various image processing algorithms. Additionally, the camera may be mounted in a location where a field of view (FOV) can be secured on the vehicle to photograph the exterior of the vehicle, and may be used to provide AR / VR-based services. Lidar can generate information about an object outside the vehicle (K600) using laser light. Lidar may include an optical transmitter, an optical receiver, and at least one processor electrically connected to the optical transmitter and optical receiver to process the received signal and generate data about the object based on the processed signal.
[0382] The communication device can exchange signals with a device located outside the vehicle (2010) (e.g., infrastructure (e.g., server, broadcasting station), other vehicle, terminal, etc.). The driving control device is a device that receives user input for driving. In manual mode, the vehicle (2010) can be operated based on signals provided by the driving control device. The driving control device may include a steering input device (e.g., steering wheel), an acceleration input device (e.g., accelerator pedal), and a brake input device (e.g., brake pedal).
[0383] The sensing unit can sense the state of the vehicle (2010) and generate state information. The position data generating device can generate position data of the vehicle (2010). The position data generating device may include at least one of a Global Positioning System (GPS) and a Differential Global Positioning System (DGPS). The position data generating device can generate position data of the vehicle (K600) based on signals generated from at least one of the GPS and DGPS. The main ECU can control the overall operation of at least one electronic device provided in the vehicle (2010), and the drive control device can electrically control the vehicle drive device in the vehicle (2010).
[0384] The autonomous driving device can generate a path for an autonomous driving service based on data obtained from an object detection device, a sensing unit, a position data generation device, etc. The autonomous driving device can generate a driving plan for driving along the generated path and generate a signal to control the movement of the vehicle according to the driving plan. Since the signal generated by the autonomous driving device is transmitted to a drive control device, the drive control device can control the vehicle drive unit within the vehicle (2010).
[0385] As illustrated in FIG. 20, a vehicle (2010) providing autonomous driving services is connected to an XR device (2000) to enable wired / wireless communication. The XR device (2000) illustrated in FIG. 20 may include a processor (2001) and a memory (2002). Additionally, although not illustrated in the drawing, the XR device (2000) of FIG. 20 may further include components of the XR device (1300) described in FIG. 13.
[0386] When the XR device (2000) of FIG. 20 is connected to a vehicle (2010) so as to enable wired / wireless communication, the XR device (2000) of FIG. 20 can receive and process AR / VR service-related content data that can be provided along with an autonomous driving service and transmit it to the vehicle (2010). Additionally, when the XR device (2000) of FIG. 20 is mounted on the vehicle (2010), the XR device (2000) of FIG. 20 can receive and process AR / VR service-related content data according to a user input signal input through a user interface device and provide it to the user. In this case, the processor (2001) can receive and process AR / VR service-related content data based on data obtained from an object detection device, a sensing unit, a location data generation device, an autonomous driving device, etc. Content data related to AR / VR services according to embodiments of the present invention may include not only information related to autonomous driving services, such as driving information, route information for autonomous driving services, driving operation information, vehicle status information, and object information, but also entertainment content unrelated to autonomous driving services, weather information, etc.
[0387] Figure 21 illustrates the process of providing AR / VR services during autonomous driving services.
[0388] A vehicle or user interface device according to embodiments of the present invention may receive a user input signal (S2110). The user input signal according to embodiments of the present invention may include a signal indicating an autonomous driving service. The autonomous driving service according to embodiments of the present invention may include a fully autonomous driving service and a general autonomous driving service. A fully autonomous driving service refers to a service in which the vehicle is driven entirely autonomously without manual driving by the user to the destination, and a general autonomous driving service refers to a service in which the vehicle is driven by a combination of manual driving by the user and autonomous driving to the destination.
[0389] It can be determined whether a user input signal according to embodiments of the present invention corresponds to a fully autonomous driving service (S2120). If, as a result of the determination, the user input signal corresponds to a fully autonomous driving service, the vehicle according to embodiments of the present invention can provide a fully autonomous driving service (S2130). Since user operation is not required for a fully autonomous driving service, the vehicle according to embodiments of the present invention can provide content related to a VR service to the user through the vehicle's windows, side mirrors, HMD, smartphone, etc. (S2130). The content related to a VR service according to embodiments of the present invention may be content related to fully autonomous driving (e.g., navigation information, driving information, external object information, etc.), or, depending on the user's selection, may be content unrelated to fully autonomous driving (e.g., weather information, street images, nature images, video call images, etc.).
[0390] If, as a result of the judgment, the user input signal does not correspond to a fully autonomous driving service, the vehicle according to the embodiments of the present invention may provide a general autonomous driving service (S2140). In the case of a general autonomous driving service, since the user's field of vision must be secured for manual driving, the vehicle according to the embodiments of the present invention may provide content related to an AR service to the user through the vehicle's windows, side mirrors, HMD, smartphone, etc. (S2140).
[0391] Content related to the AR service according to the embodiments of the present invention may be content related to fully autonomous driving (e.g., navigation information, driving information, external object information, etc.), or, depending on the user's choice, may be content unrelated to fully autonomous driving (e.g., weather information, street images, nature images, video call images, etc.).
[0392] FIG. 22 illustrates a case in which an XR device according to an embodiment of the present invention is implemented as an HMD type. Various embodiments described above may also be implemented as the HMD type shown in FIG. 22.
[0393] The HMD-type XR device (100a) illustrated in FIG. 22 includes a communication unit (110), a control unit (120), a memory unit (130), an I / O unit (140a), a sensor unit (140b), and a power supply unit (140c), etc. In particular, the communication unit (110) within the XR device (10a) communicates with a mobile terminal (100b) via wired or wireless communication.
[0394] Also, FIG. 23 illustrates a case in which an XR device according to an embodiment of the present invention is implemented as an AR glass type. The various embodiments described above may also be implemented as an AR glass type as shown in FIG. 44.
[0395] As illustrated in FIG. 23, AR glasses may include a frame, a control unit (200), and an optical display unit (300).
[0396] As shown in FIG. 23, the frame may have the form of glasses worn on the face of the user (10), but is not necessarily limited thereto and may have the form of goggles worn in close contact with the face of the user (10).
[0397] Such a frame may include a front frame (110) and first and second side frames.
[0398] The front frame (110) has at least one opening and can be extended in a first horizontal direction (x), and the first and second side frames can be extended in a second horizontal direction (y) that intersects the front frame (110) and can be extended parallel to each other.
[0399] The control unit (200) can generate an image to be shown to the user (10) or a continuous video of images. Such a control unit (200) may include an image source that generates an image and a plurality of lenses that diffuse and converge light generated from the image source. The image generated by the control unit (200) in this manner can be emitted to the optical display unit (300) through a guide lens (P200) located between the control unit (200) and the optical display unit (300).
[0400] The control unit (200) can be fixed to either of the first or second side frames. For example, the control unit (200) may be fixed to the inside or outside of either side frame, or may be formed integrally by being embedded inside either side frame.
[0401] The optical display unit (300) can perform the role of showing an image generated by the control unit (200) to the user (10), and can be formed of a translucent material so that the external environment can be seen through the opening while the image is shown to the user (10).
[0402] The optical display unit (300) can be fixed by being inserted into an opening included in the front frame (110), or it can be provided by being fixed to the front frame (110) by being located on the back side of the opening [i.e., between the opening and the user (10)]. In the present invention, as an example, the case in which the optical display unit (300) is fixed to the front frame (110) by being located on the back side of the opening is illustrated.
[0403] As shown in FIG. 23, such an XR device can be configured so that when an image of an image is incident from a control unit (200) into an incident area (S1) of an optical display unit (300), the image light is emitted through the optical display unit (300) to an output area (S2) of the optical display unit (300), thereby allowing the image generated by the control unit (200) to be shown to the user (10).
[0404] Accordingly, the user (10) can view the external environment through the opening of the frame (100) and simultaneously view the image generated by the control unit (200).
[0405] As described above, the present invention is applicable to the fields of 5G communication technology, robotics technology, autonomous driving technology, and AI technology; however, the following drawings will focus on describing the present invention applicable to multimedia devices such as XR devices, digital signage, and TVs. However, with reference to the previous FIGS. 1 to 23, it is also within the scope of the present invention for a person skilled in the art to implement other embodiments by combining the drawings described below.
[0406] In particular, since the multimedia device described in the drawings to be described later is sufficient as a device with a display function, it is not limited to an XR device, and it is also possible to additionally perform communication according to 5G by corresponding to the UE (User Equipment) described in the previous Figures 1 to 9.
[0407] A device equipped with a navigation function according to the embodiments reprocesses an image (hereinafter referred to as the third image) from which images of internal and external objects of a vehicle that obscure part of a driving lane have been removed, and provides a navigation screen containing accurate navigation information to a user.
[0408] A device equipped with a navigation function may be a mobile device or a mobile phone (hereinafter, mobile).
[0409] The device captures the forward view of the road using a camera built into the mobile device or a separate camera installed in the vehicle.
[0410] The device regenerates the preview video captured by the camera onto the navigation screen.
[0411] The device adds information necessary for driving to the navigation video using XR (Extended Reality) effects.
[0412] The device adds road display images and other elements to the navigation video using AR (Augment Reality) effects.
[0413] The device removes images of surrounding devices inside and outside the vehicle that obscure the lane in the direction the vehicle is currently traveling from the forward view image.
[0414] For example, even if part of the lane of the driving road image is obscured by an image of a vehicle component such as a dashboard or bonnet, the device according to the embodiments renders a navigation image containing accurate route guidance information through image rendering.
[0415] The device according to the embodiments provides the user with a navigation image including a road display image with an Augmented Reality (AR) effect on a forward view where the lane is clearly visible and on the road in the direction of travel.
[0416] A device according to the embodiments renders a driving road image or / and adds an AR road sign image based on a plurality of baselines.
[0417] The device can automatically recognize multiple baselines using AI (artificial intelligence) algorithms.
[0418] The device can manually recognize multiple reference lines based on user input signals.
[0419] When rendering an image, the device according to the embodiments obtains height and width values corresponding to the best aspect ratio of the display screen based on at least one reference line including a vanishing point among a plurality of set reference lines.
[0420] The device according to the embodiments displays a navigation video including accurate driving paths and navigation information through additional rendering, such as adding an AR road marker image to a processed screen.
[0421] FIG. 24 is a flow chart (2400) illustrating a control method of a device according to embodiments.
[0422] The device according to the embodiments performs a navigation function (2411).
[0423] The device according to the embodiments performs a determination of whether AR is supported (2412).
[0424] If the device determines that AR support is not supported (2420), it starts route guidance in map mode (2415).
[0425] When the device determines that AR support is possible (2421), it searches for past usage history (e.g., AR mode route guidance or map mode route guidance) by referring to memory (2413).
[0426] The device refers to the memory of past usage history, and when a history of using map mode route guidance is found (2430), it starts route guidance in map mode (2415).
[0427] For example, map mode route guidance indicates a map mode navigation function that guides the user by displaying the path to follow as an arrow image on a 2D or 3D map image.
[0428] The device refers to the memory of past usage history, and if a history of starting AR support mode or performing AR mode route guidance is found (2440), it starts AR support mode (2414).
[0429] The device searches for past usage history by referring to memory, and even if it is found that there is no usage history (2441), it performs the start of AR support mode (2414).
[0430] The device's AR support mode includes a function that executes the AR support feature to render and display driving footage captured as navigation video, which guides the route in AR mode.
[0431] The device's AR support feature includes a rendering function that adds AR (Augmented Reality) effect road display images to driving video images captured by the device's camera.
[0432] The device's AR support function includes a feature that technically supports deleting unnecessary obstacle images from driving footage for user convenience and rendering the remaining images by deleting or enlarging parts of them according to a pre-set aspect ratio.
[0433] The device performs an AR support mode start (2414) to capture an image of the vehicle's forward driving path with a camera and provides it to the user.
[0434] The camera may be a camera installed on a device, for example, a mobile device (hereinafter referred to as mobile), or a camera installed separately on a vehicle.
[0435] For example, the device can display a message on the screen saying, “AR mode will automatically activate when you drive after aligning the camera screen with the line below,” and receive a user input signal to start the AR support mode.
[0436] For example, when the user adjusts the position of the camera so that the center of the driving video screen aligns with the dotted line (2415) displayed on the screen, or aligns the center dotted line with the center of the driving road, the device starts rendering the driving video screen (2450).
[0437] If the device includes a touch screen, a user input signal may be input by the user's touch or drag, and a voice-recognizing device may receive the user's voice as an input signal and start an AR support mode.
[0438] The input signals according to the embodiments are not limited to the embodiments described above and may include various input signals for using the device.
[0439] The device's video rendering processor receives video data of the captured driving path and renders the forward view images in AR mode.
[0440] When rendering is successful (Yes, 2460), the device starts route guidance in AR mode (2416).
[0441] If rendering fails (No, 2470), for example, if it fails after 3 attempts, the device displays a message on the screen such as “….Would you like to receive route guidance in map mode?” to the user (2417).
[0442] Although not implemented, messages provided by the device can be delivered to the user in various ways, such as voice.
[0443] When the device receives a user selection signal for “retry,” it performs the AR support mode start again (2414) (2490).
[0444] When the device receives a user selection signal for “guidance in map mode,” it performs route guidance in map mode (2415) (2480).
[0445] Although not illustrated, user selection signals on the device can be designed to be input in the form of screen touch and drag selection, user voice selection, or user eye tracking or position tracking through the device's HMD.
[0446] Meanwhile, the flowchart (2400) of FIG. 24 may represent the process of an XR device, for example described in FIG. 1 to FIG. 23, providing a navigation service.
[0447] FIG. 25 is a block diagram illustrating the components of a device to which the flowchart illustrated in FIG. 24 is applied.
[0448] Of course, a person skilled in the art may implement the device shown in FIG. 25 in other devices shown in FIG. 1 to FIG. 23, and this also falls within the scope of the present invention.
[0449] The device (2500) performs the functions / operations described in FIGS. 1 to 24. Additionally, the device (2500) illustrated in FIG. 25 further includes modules, etc., for performing the functions / operations described in FIGS. 1 to 24, although not illustrated in the drawings.
[0450] A device equipped with a navigation function according to the embodiments receives forward view image data of a driving road, renders images of the image data based on a plurality of set reference lines, and generates and provides a navigation image for AR mode route guidance.
[0451] The device (2500) includes a sensor (2510), a video rendering processor (2560), a display (2550), a controller (2530), a memory (2520), and a camera (2540).
[0452] The device (2500) may be, for example, a mobile device (hereinafter, mobile).
[0453] The controller (2530) according to the embodiments controls the overall function of the device and the operation of each component module.
[0454] However, the configuration modules illustrated in FIG. 25 are merely examples, and deleting, adding, or changing some modules to implement the technical concept of the present invention is also within the scope of the present invention.
[0455] The device (2500) according to the embodiments performs a navigation function.
[0456] The controller (2530) of the device (2500) according to the embodiments determines whether AR is supported (e.g., 2412 in FIG. 24).
[0457] If the controller (2530) according to the embodiments determines that AR support is not possible (2420 in FIG. 24), it continues to execute the navigation function of the map mode.
[0458] When the controller (2530) determines that AR support is possible (2421 in FIG. 24), it searches for the usage history of the previous navigation mode by referring to the memory (2520).
[0459] The controller (2530) according to the embodiments performs a navigation function that guides the route in map mode when it detects that the map mode function was used in the usage history of the previous navigation mode (2430 in FIG. 24).
[0460] The controller (2530) according to the embodiments searches for the usage history of the previous navigation mode by referring to the memory (2520), and if the history of using the AR support mode is found (2440 in FIG. 24), it performs the start of the AR support mode.
[0461] The controller (2530) of the device (2500) according to the embodiments performs an AR support mode start (2414 in FIG. 24).
[0462] The camera (2540) of the device (2500) according to the embodiments photographs the road ahead of the vehicle upon the start of the AR support mode and transmits the image data to the sensor (2510).
[0463] The camera (2540) can acquire image data by photographing the road ahead according to the set pixel values.
[0464] The camera (2540) of the device (2500) may be a camera built into the device, or a separate camera installed in the vehicle.
[0465] Depending on where the mobile is installed in the vehicle, vehicle components (hereinafter referred to as objects), such as the vehicle's bonnet or dashboard, are often included within the field of view of the mobile's camera.
[0466] The sensor (2510) according to the embodiments receives driving path image data captured by the camera (2540) as an input signal and transmits it to a video rendering processor (2560).
[0467] A video rendering processor (2560) according to the embodiments starts an AR support mode and renders a driving path video captured as a navigation video for route guidance in AR mode.
[0468] The video rendering processor (2560) starts the AR support mode automatically or manually according to the user input signal.
[0469] A video rendering processor (2560) extracts one or more forward view image frames (hereinafter, first images) from a driving path image.
[0470] A video rendering processor (2560) generates a navigation image (hereinafter, a second image) for route guidance in AR mode based on multiple reference lines set in a first image.
[0471] The video rendering processor (2560) extracts a third image that does not include an image of an internal or external vehicle component (hereinafter, an image of an internal or external vehicle object), e.g., a bonnet or a dashboard, that obscures the vehicle's forward path from the first image.
[0472] A plurality of reference lines according to the embodiments include a first reference line including a vanishing point or a second reference line associated with an image of an obstruction obscuring the lane.
[0473] The vanishing point according to the embodiments includes the point where the left and right lanes of the vehicle's driving road intersect the horizon.
[0474] The first reference line according to the embodiments represents a horizontal horizontal line including vanishing points in the first image.
[0475] According to the embodiments, the second baseline may be a horizontal boundary line formed in the first image by the highest horizontal line of the object image when the object image of the interior or exterior of the vehicle is located at the bottom of the first image.
[0476] The video rendering processor (2560) generates a third image by removing the first image within the range from the second baseline to one end of the first image.
[0477] The video rendering processor (2560) separates the third image into upper and lower images based on the first baseline.
[0478] The vertical height of the upper image corresponds to the first value, and the vertical height of the lower image corresponds to the second value.
[0479] The video rendering processor (2560) obtains the height value of the final navigation image (hereinafter, the second image) based on the smaller value between the first value and the second value and the first ratio set.
[0480] The first ratio is the ratio of the second value to the first value.
[0481] The video rendering processor (2560) obtains the width value of the second image based on the pre-set second ratio and the height value of the second image.
[0482] The second ratio is the ratio of the width value to the height value of the second image above.
[0483] A video rendering processor (2560) according to the embodiments can generate a second image by excluding an image corresponding to a polygonal area having a width value corresponding to (horizontal pixel value of the first image - width value of the second image) / 2.
[0484] The controller (2530) of the device (2500) according to the embodiments can control the video rendering processor (2560) to start path guidance in AR mode when it succeeds in rendering (2460 in FIG. 24) (2416 in FIG. 24).
[0485] The display (2550) according to the embodiments displays a navigation video that provides a route guidance service in AR mode.
[0486] The display (2550) according to the embodiments is designed to display a navigation image with a height:width ratio of 3:4 or 4:3.
[0487] A video rendering processor (2560) according to the embodiments renders a third image such that the height:width ratio of the second image becomes 3:4 when the second image is displayed on a display (2550) such that the height of the second image is shorter than its width.
[0488] A video rendering processor (2560) according to the embodiments renders a third image such that the height:width ratio of the second image becomes 4:3 when the second image is displayed on a display (2550) such that the width of the second image is shorter than its height.
[0489] The device (2500) according to the embodiments may be configured to include a memory (2520) that stores a program including a file rendering a captured image or other instructions. The input signals according to the embodiments of the present invention are not limited to the embodiments described above and include various input signals for using the device.
[0490] The controller (2530) controls the overall operation of the device. The controller (2530) can determine other user input signals, including the received driving video of the current vehicle, and controls the video rendering processor (2560) to render the input forward driving path video data and display it as a navigation screen on the device's display (2550). The controller (2530) can be connected to communicate with the sensor (2510), camera (2540), video rendering processor (2560), memory (2520), and display (2550). The controller (2540) may be composed of one or more processors configured to communicate with the device's memory (2520), at least one of the sensor (2510), camera (2540), video rendering processor (2560), and display (2550) according to embodiments of the present invention.
[0491] The memory (2520) stores a program, etc., that includes instructions for controlling the operation of the device in relation to the driving video or rendered video captured by the input current vehicle.
[0492] Additionally, the device (2500) illustrated in FIG. 25 performs the functions / operations described in FIG. 1 to FIG. 24. Additionally, the device (2500) further includes modules, etc., for performing the functions / operations described in FIG. 1 to FIG. 24, although not illustrated in the drawings.
[0493] FIG. 26 is an example drawing of a device according to embodiments rendering an image based on a plurality of reference lines.
[0494] The device renders a captured driving video image (hereinafter, the first image) based on multiple reference lines to generate a navigation screen (hereinafter, the second image) for route guidance in AR mode.
[0495] The device sets multiple baselines automatically or manually.
[0496] FIG. 26(a) shows that the device receives a user input signal and manually sets the first reference line (2612) and the second reference line (2621) in the first image (2610).
[0497] The device can receive input signals from the user, for example, screen touch and drag in the case of a touch screen, and can also receive voice input signals from the user to perform settings.
[0498] The device according to the embodiments can be designed to receive and input user input signals through the HMD, such as user eye tracking or position tracking.
[0499] The first reference line (2612) according to the embodiments represents a horizontal line including the vanishing point (2611) in the first image (2610).
[0500] The vanishing point (2611) according to the embodiments represents the point where the left and right lane images (2613, 2614, 2615, 2616) of the vehicle's driving lanes intersect with the horizon.
[0501] The device sets the boundary line (2617) formed by the image of the vehicle interior object or exterior object in the first image (2610, 2620) as the second reference line (2621).
[0502] The second reference line (2621) according to the embodiment includes a boundary line formed within the first image of the image of the bonnet or dashboard.
[0503] The device deletes images of vehicle interior and exterior objects based on the second reference line (2621) in the first image, and regenerates the second image (2630) based on the first reference line (2611).
[0504] The device regenerates a second image (2630) by deleting some images and enlarging the remaining images according to the pre-set aspect ratio.
[0505] The device provides a navigation video including a route guidance (2631) of an AR (Augmented Reality) road display image and a navigation information (2632) image of the driving direction that the vehicle must follow.
[0506] The first image according to the embodiments may include images of obstacles inside or outside the vehicle that obstruct the vehicle's forward driving road, such as a front pillar (A pillar) between the front windshield and the side window, a dashboard, a bonnet, a room mirror, a sun visor, a steering wheel, etc., which obstruct the driver's forward view.
[0507] FIG. 26(b) shows that the device receives a user input signal and automatically sets the first reference line (2641) and the second reference line (2642) in the first image (2640).
[0508] The device deletes an image of an interior or exterior object of a vehicle based on a second baseline (2642) in a first image (2640).
[0509] The device deletes some images based on the first baseline (2641) and extracts and enlarges the remaining images according to a pre-set screen or height ratio.
[0510] The device provides a navigation video including a route guidance (2661) of an AR (Augmented Reality) road display image and a navigation information (2662) image of the driving direction that the vehicle must follow.
[0511] FIG. 27 illustrates a rendering in which a device according to embodiments deletes an obstruction image based on a second baseline.
[0512] The rendering method of FIG. 27 illustrates an example of a device described in FIG. 1 to 26 (e.g., the video rendering processor, 2560, of FIG. 25) performing a rendering operation.
[0513] FIG. 27(a) illustrates a device searching the area around a vehicle and selecting a destination based on a user input signal.
[0514] In FIG. 27(b), the device shows an image of the vehicle's current driving road in the forward view.
[0515] The device can set the second baseline manually or automatically by receiving a user-selected input signal.
[0516] The device displays a pre-reference line (2725) on the initial screen that can set a second reference line on the captured image as in drawing (2720).
[0517] The device displays or provides a request message (2724) asking to manually set a second baseline, and performs the process according to the user's input signal.
[0518] For example, display an image that says, “To provide accurate AR navigation, remove the parts that block the view ahead. Please move the red dots on both sides to set the parts that block the view ahead” (2724).
[0519] The device receives a user input signal in which the user touches or drags to move the positions of the two endpoints (2721, 2722) of the baseline (2725).
[0520] When the two endpoints (2721, 2722) of a reference line are moved to and overlap the two endpoints (2731, 2732) of the image of the bonnet or dashboard at the bottom of the first image according to a user input signal, the device displays a message (2734, e.g., “Setting complete”) indicating that the second reference line (2735) has been set.
[0521] The device may also automatically perform the setting of a second baseline using AI (artificial intelligence) algorithms, DNN programs, etc.
[0522] The device can automatically set the reference line (2725) in the first image (2720) to be formed or moved to the top (2735) of the object image (2723) inside or outside the vehicle.
[0523] The device deletes the object image included in the lower region (2733) of the second baseline (2735) in the first image (2730) and regenerates the second image (2741).
[0524] The device renders the second image (2740) to include an AR road display (2742) image that shows the driving direction the vehicle must follow as an AR image, and navigation information (2743) images.
[0525] The device deletes the images of the vehicle's interior and exterior objects and enlarges the remaining image (2730) to match the ratio of the pre-set display screen, or deletes part of the image to create and display a second image (2740) that is the same size as the first image (2720).
[0526] The rendering operation of the device according to the embodiments is identical to one or more rendering operations described in FIGS. 24 to 26 (for example, the operation of the video rendering processor of FIG. 24 rendering forward view images of the driving path into AR mode navigation images). A detailed description of one or more methods / operations described in FIGS. 24 to 26 is omitted.
[0527] FIGS. 28(a) to 28(b) are a flowchart (2800) and a drawing illustrating a method in which a device according to embodiments manually sets a first reference line to render an image.
[0528] The rendering method of FIG. 28 illustrates an example of a device described in FIG. 1 to 27 (e.g., the video rendering processor, 2560, of FIG. 25) performing a rendering operation.
[0529] The device according to the embodiments establishes a first reference line (2810).
[0530] The device can predesignate a first baseline.
[0531] For example, the device can search for a horizon in the first image using a method such as a DNN of an AI (artificial intelligence) algorithm and set it as the first baseline.
[0532] The device receives the location of the vanishing point manually via a user input signal (2811).
[0533] FIG. 28(b) is an example in which the device receives the position of the vanishing point as a user input signal (e.g., touch or drag) and manually sets the position of the first reference line.
[0534] The device stores a threshold value range (preference scope) (2812).
[0535] When the device finds that the location of the set vanishing point image is included within the threshold range, it inputs that point as the location of the vanishing point image and saves the threshold range (preference scope) (2812).
[0536] For example, store a threshold range that measures the distance from the horizon to the vanishing point image.
[0537] The device calculates where the vanishing point is located in the upper and lower images at the 1 / 2 point in the third image (2830) to generate a second image based on the location of the vanishing point (Get Vanish point in Crop Class, 2813).
[0538] The third image (2830) according to the embodiments is an image in which the device removes the internal and external object images of the vehicle from the first image based on the second reference line (2822 in FIG. 28(b)).
[0539] The device performs calculations and rendering for the extraction of a second image (2814).
[0540] According to embodiments, the device generates a second image (2840) based on the position of a first baseline and the set height or width ratios of the second image.
[0541] For example, the device generates a second image (2840) based on where the first baseline is included in the upper and lower images, or the best ratio of height to width of the second image that is set.
[0542] When the device succeeds in rendering the second image (2815), it starts AR mode (2816).
[0543] The device displays route guidance navigation video in AR mode.
[0544] If the device fails to render the second image, it displays the navigation image in default mode (e.g., map mode) (2817).
[0545] The rendering operation of the device according to the embodiments is identical to one or more rendering operations described in FIGS. 24 to 27 (for example, the operation of the video rendering processor of FIG. 24 rendering forward view images of the driving path into AR mode navigation images). A detailed description of one or more methods / operations described in FIGS. 24 to 27 is omitted.
[0546] FIGS. 29(a) to 29(c) are a flowchart (2900) and a drawing illustrating a method in which a device according to embodiments automatically sets a first baseline to render an image.
[0547] The rendering method of FIG. 29 illustrates an example of a device described in FIG. 1 to 28 (e.g., the video rendering processor, 2560, of FIG. 25) performing a rendering operation.
[0548] In FIG. 29(a), the device according to the embodiments starts operation (2909) and starts AR mode (2910).
[0549] The device detects vehicle shaking and vibration (2911).
[0550] If shaking occurs while the vehicle is driving due to vibrations caused by shaking or unevenness of the road surface, the camera capturing the road ahead may also change position, causing the view of the road to deviate from a stable position.
[0551] The device checks if there are any problems with rendering (Rendering OK?, 2912).
[0552] When the device determines that there is no problem with rendering, it continues to display navigation video in AR mode (2913).
[0553] If the device determines that there is a problem with rendering, it detects multiple straight lines in the captured image (2914).
[0554] The device first searches for and extracts all multiple straight lines to extract the first reference line from the captured image of the road ahead.
[0555] For example, if the device detects that images of multiple straight lines forming a lane are broken in the middle, it connects the broken lines using an AI algorithm or the like.
[0556] The device calculates the projection position created by the endpoints of the connected lines intersecting the horizon as the vanishing point (2915).
[0557] In the image of the driving video captured by the camera (hereinafter, the first image), the vanishing point includes a point where the endpoints of the multiple connected straight lines intersect the horizon, and the left and right lane images of the vehicle's driving road lane are each connected by a single line.
[0558] The device calculates the vanishing point location (2916).
[0559] FIG. 29(b) shows that a point (2920) where multiple straight lines intersect at the horizon includes the location of the vanishing point in the first image.
[0560] FIG. 29(c) illustrates a vanishing point (2930) where multiple straight lines (2940) extracted from the left and right sides forming multiple lanes meet the horizon at one point.
[0561] The device calculates the location of the point created by the straight lines (2940) connecting the lanes intersecting the horizon as the location of the vanishing point (2930).
[0562] The device performs rendering of an image based on the position of the vanishing point (Do Rendering, 2917).
[0563] The device according to the embodiments generates a navigation image based on the position of the vanishing point (or the position of the first baseline), for example, whether it is located above or below the halfway point of the image, and based on the best ratio of height to width of the image to be extracted, etc.
[0564] The device continues to provide a navigation video that guides the route in AR mode (AR Mode Continue, 2913).
[0565] The rendering operation of the device according to the embodiments is identical to one or more rendering operations described in FIGS. 24 to 28 (for example, the operation of the video rendering processor of FIG. 24 rendering forward view images of the driving path into AR mode navigation images). A detailed description of one or more methods / operations described in FIGS. 24 to 28 is omitted.
[0566] FIGS. 30 to 31 are a flowchart (3000) and a drawing illustrating a method in which a device according to embodiments automatically calculates and renders a second baseline using an algorithm such as a DNN (Neural Network).
[0567] The rendering method of FIG. 30 illustrates an example of a device described in FIG. 1 to 29 (e.g., the video rendering processor, 2560, of FIG. 25) performing a rendering operation.
[0568] The device according to the embodiments starts the execution of the navigation function (3010).
[0569] The device starts a process for rendering an AR mode navigation image (AR Mode Srart, 3011).
[0570] The device uses a DNN method of an AI (artificial intelligence) algorithm to detect obstacles in the forward field of view (3012).
[0571] The device uses deep learning of a DNN (deep neural network) of an AI (artificial intelligence) algorithm to detect obstacles in the forward field of view.
[0572] The device searches for images of the bonnet in driving video images (Bonnet Detection OK?, 3013)
[0573] When the device determines that it has succeeded in detecting the bonnet image (3020), it sets the position of the boundary line (Crop Position) to delete the bonnet image from the driving video image and saves the set value (Get Crop Position & Save preference, 3014).
[0574] For example, the device saves the position to be deleted (or the location of the second baseline) of the bonnet image searched by the AI algorithm as a preference if it matches the boundary line formed by the actual bonnet image in the first image, or if the distance from the boundary line is calculated and falls within a threshold value range.
[0575] If the device determines that it failed to detect the bonnet image (3030), it sets the position of the boundary line to delete the bonnet image to the default (Set default Crop Position, 3022)
[0576] The device obtains vanishing point location information to generate a final navigation image (hereinafter, second image) based on a first reference line (Get Vanish point in Crop Class, 3015, 3018).
[0577] The device calculates the image region to be deleted, excluding the region of the second image to be extracted (Calculate Crop region, 3016, 3019).
[0578] The device deletes the image (Do Crop, 3017, 3020) and extracts the second image.
[0579] The device terminates the process of rendering the AR mode navigation video (3021).
[0580] The device generates a second image based on the location of the vanishing point and the aspect ratio of the final navigation video screen, etc.
[0581] The device deletes images outside the set area based on the best height ratio of the upper and lower image areas relative to the vanishing point, and generates a second image by enlarging the remaining image according to the best ratio of the screen to be displayed.
[0582] FIGS. 31(a) to 31(c) are drawings illustrating the operation of rendering an image in AR mode of the device of FIG. 30.
[0583] The drawing shows that, according to the rendering operation (3012, 3013) of the device of FIG. 30, the device detects the boundary lines (3111, 3121, 3131) formed by the bonnet image within the first image (3110, 3120, 3130).
[0584] The rendering operation of the device according to the embodiments of FIGS. 30 to 31 is identical to one or more rendering operations described in FIGS. 24 to 29 (for example, the operation of the video rendering processor of FIG. 24 rendering forward view images of the driving path into AR mode navigation images).
[0585] A detailed description of one or more methods / operations described in FIGS. 24 to 29 is omitted.
[0586] FIG. 32 is an example in which a device according to embodiments applies a lane recognition algorithm through a DNN to render an AR mode navigation image and calculate accurate lane information.
[0587] The rendering method of FIG. 32 illustrates an example of a device described in FIG. 1 to 31 (e.g., the video rendering processor of FIG. 25, 2560) performing a rendering operation.
[0588] The device extracts one or more first images from a driving road preview video captured by a camera and converts them into JPEG images.
[0589] The device (e.g., the video rendering processor (2560) of FIG. 25) receives the converted JPEG image as input data for the DNN modeling process and performs a process of accurately detecting the lanes of the road.
[0590] At this time, the device extracts the area of the image (3220, hereinafter referred to as the second image) to accurately detect the lane image using a DNN algorithm.
[0591] The device, for example, sets pixel values to capture a preview video of the driving road with the camera.
[0592] Based on the set pixels, the device extracts the second image (3220) area by setting the left (3210) value so that the images (3230) of the current vehicle's driving lane are centered in the second image (3220) area.
[0593] The device captures a video by setting the pixels of the driving road preview video to, for example, 1280*720.
[0594] The device obtains a left value for extracting an area to be excluded from the second image region based on the width value of the second image and the set horizontal pixel value of the first image.
[0595] The device obtains the left value of the drawing according to the formula, for example, (set pixel width value - width value of the acquired second image) / 2.
[0596] First, the device obtains the width value of the second image based on the best height ratio of the upper and lower regions of the second image relative to the first baseline and the best height:width ratio of the second image screen.
[0597] The device obtains the value of an area to be excluded from the second image (e.g., Left value, 3210) based on the width value of the acquired second image and the horizontal pixel value of the set first image (e.g., 1280 in the case of 1280*720).
[0598] For example, the device can set the left value (3210) in the drawing to 115.5, which is half the value obtained by subtracting the width value (e.g., 1049) of the second image from the horizontal pixel value (e.g., 1280) of the first image (e.g., 1049) (1280-1049=231).
[0599] The device can accurately detect lanes by extracting the area of the second image (3220) based on the Left value and using a DNN algorithm.
[0600] The rendering operation of the device according to the embodiments is identical to one or more rendering operations described in FIGS. 24 to 31 (for example, the operation of the video rendering processor of FIG. 24 rendering forward view images of the driving path into AR mode navigation images).
[0601] A detailed description of one or more methods / operations described in FIGS. 24 to 31 is omitted.
[0602] FIGS. 33 to 35 are examples illustrating a method of rendering a driving image captured by a device according to embodiments into a second image based on a first reference line position and a second reference line.
[0603] The rendering method of FIGS. 33 to 35 illustrates an example of a device described in FIGS. 1 to 32 (e.g., the video rendering processor, 2560, of FIG. 25) performing a rendering operation.
[0604] In FIG. 33, the device sets the image extracted (cropped) by deleting the image of the obstacle at the bottom of the first image (3309) based on the second reference line (3308) as the third image (3310).
[0605] FIG. 33 is an example in which a device renders the third image (3310) as the second image when the first reference line (3320) is located at the halfway point (3330) of the vertical height of the third image (3310).
[0606] The image of the obstacle at the bottom may include, for example, an image of a bonnet.
[0607] To remove the obstruction image at the bottom, the device sets any polygon that includes the first image at the bottom within a range based on the second baseline (3308) in the first image (3309).
[0608] The device sets any polygon to include the second baseline (3308) as one side.
[0609] The device removes a portion of the first image corresponding to an arbitrary polygon from the first image (3309).
[0610] The device establishes a first baseline including a vanishing point (3340).
[0611] When the first baseline is located at the halfway point (3330) of the vertical height of the third image (3310), the device sets the range of the second image (3391) based on the total vertical height value (3350 height + 3360 height) of the third image (3310).
[0612] When the device extracts a third image from a first image, it can extract (crop) the image by setting the height of the upper image area (3350) and the height of the lower image area (3360) to be the same with respect to the first reference line.
[0613] The second image (3391) can be the final navigation image that the device displays to the user.
[0614] The device sets the vertical height value (3350+3360 height) of the third image (3310) to the vertical height value of the second image (hereinafter, the fourth value, 3391).
[0615] The device can calculate the width value of the second image using the horizontal:vertical ratio of the second image (hereinafter, the second ratio) that is set in advance.
[0616] The second ratio is the ratio of the width value to the height value of the second image. For example, the pre-set second ratio can be 4:3.
[0617] The device can calculate the width value of the second image using the pre-set second ratio (4:3) and the vertical height value of the second image (fourth value, 3391).
[0618] For example, if the fourth value (3391) is 10 cm, then 4:3=x:10, and since x=4*10 / 3=13.3, the width value (3380) of the second image can be set to approximately 13.3 cm.
[0619] The device obtains the width value (3380) of the second image (3391) based on the previously set second ratio and fourth value.
[0620] For example, the device obtains the width value of the second image by using the height value of the image after deleting the bonnet image (the third image) according to the best width-to-height ratio of the final navigation image. Next, the device extracts the second image (3391) by using the difference between the total width value of the third image and the width value of the second image as the left offset value (3370) of the third image.
[0621] When a user receives a navigation video with the mobile device laid flat, that is, when the device provides a navigation video including a second image in a form where the horizontal direction of the display is longer than the vertical direction, the device sets the Best ratio value of the height:width of the second image to 3:4.
[0622] When a user receives a navigation video while holding the mobile device upright, that is, when the device provides a navigation video including a second image in a form where the vertical direction of the display is longer than the horizontal direction, the Best ratio value of the height:width of the second ratio is set to 4:3.
[0623] The device may set the Best value of the second ratio's height:width to 4:4 or 3:4 for the user's convenience, but the second ratio is not limited to this and the ratio may be changed depending on the display's setting value.
[0624] The device obtains the width value (3380) of the second image (3391), then obtains the left value (3370) from the first image (e.g., obtained through a rendering operation in FIG. 32), and extracts the area of the second image.
[0625] FIG. 34 is an example of a device rendering an image when the first reference line is positioned above the halfway point of the vertical height of the third image.
[0626] This is a rendering example in which the device removes the obstruction image based on the second baseline in the first image, and the first baseline (3420) is included above the point (3430) of the vertical height of the extracted (cropped) third image (3410).
[0627] The device analyzes the third image (3410) by dividing it into upper and lower images based on the first reference line (3420) including the vanishing point (3440).
[0628] The device sets the vertical height of the upper image to the 6th value (3450) and the vertical height of the lower image to the 7th value (3460), and obtains the 6th value (3450) by measuring it.
[0629] If the height of the upper image is smaller than the height of the lower image, the device excludes the area of the bottom edge of the lower image from the area of the second image based on the height of the upper image.
[0630] The height value of the area to be excluded from the area to be extracted from the upper image to the second image can be set to “0” to obtain the sixth value (Top=0).
[0631] The device calculates and obtains a seventh value based on a third ratio (e.g., 2:3) and a sixth value (3450).
[0632] The third ratio is set as the ratio of the seventh value to the sixth value, centered on the first baseline.
[0633] The device obtains the value obtained by adding the 6th and 7th values as the 8th value (3490), which is the vertical height value of the 2nd image.
[0634] The device calculates and obtains the width value (3480) of the second image based on a preset second ratio (e.g., 3:4) and an eighth value (3490).
[0635] The second ratio is set as the ratio of the width value to the height value of the second image.
[0636] The device obtains the width value (3480) of the second image (3491), then obtains the left value (3470) from the first image (e.g., obtained through a rendering operation in FIG. 32), and obtains the area of the second image.
[0637] FIG. 35 is an example of rendering by the device when the first reference line is positioned below the halfway point of the vertical height of the third image.
[0638] This indicates the case where the device removes the obstruction image from the first image based on the second reference line, and the first reference line (3520) is located below the point (3530) at half the vertical height of the third image (3510) that was cropped.
[0639] The device analyzes the third image (3510) by dividing it into upper and lower images based on the first reference line (3520) including the vanishing point (3540).
[0640] The device sets the vertical height of the upper image to a first value (3550) and the vertical height of the lower image to a second value (3560), and obtains the second value (3560) by measuring it.
[0641] If the height of the lower image is smaller than the height of the upper image, the device excludes the top part of the upper image from the area of the second image based on the height of the lower image.
[0642] The device calculates and obtains a first value based on a second value (3560) according to a first ratio (e.g., 3:2) that is pre-set.
[0643] The first ratio is set as the ratio of the first value to the second value (upper height: lower height) centered on the first baseline.
[0644] The device obtains the value obtained by adding the first value and the second value as the third value (3590), which is the vertical height value of the second image.
[0645] The device calculates and obtains the width value (3580) of the second image based on a pre-set second ratio (e.g., 3:4) and a third value (3590).
[0646] The second ratio is set as the ratio of the width value to the height value of the second image (3591).
[0647] The device obtains the width value (3580) of the second image (3591), then obtains the left value (3570) from the first image (e.g., obtained through a rendering operation in FIG. 32), and extracts the second image.
[0648] The rendering operation of the device according to the embodiments illustrated in FIGS. 33 to 35 is identical to one or more rendering operations described in FIGS. 24 to 32 (for example, the operation of the video rendering processor of FIG. 24 rendering forward view images of the driving path into AR mode navigation images).
[0649] A detailed description of one or more methods / operations described in FIGS. 24 to 32 is omitted.
[0650] FIGS. 36 and 37 are flowcharts for the case where the device switches from a mode displaying navigation images in map mode to a mode displaying navigation images in AR mode.
[0651] Figure 36 shows a case where a device provides a navigation video service through a display in which the vertical length is set larger than the horizontal length (e.g., a mobile device displayed in a standing position).
[0652] The device starts an AR support mode by a user input signal while displaying a navigation image (3610) in map mode.
[0653] For example, the device receives a user input signal through the AR mode button (3611) and performs a mode for AR mode setting (3620) (3612).
[0654] The AR mode setting (3620) of the device is a mode that receives a user input signal to start rendering an AR mode route guidance navigation video.
[0655] The device performs AR mode navigation video rendering and determines whether the rendering is successful (3630).
[0656] When the device determines that rendering has been successful (3631), it displays an AR mode route guidance navigation video (3640).
[0657] If the device determines that rendering has failed (3632), it displays the map mode route guidance navigation video (3610) again (3632).
[0658] The device provides a map mode route guidance navigation video (3610) again in response to a user input signal requesting a switch to map mode while displaying an AR mode route guidance navigation video (3640) (3641).
[0659] The device can receive a user input signal requesting a map mode route guidance navigation video through the map mode button (3641).
[0660] FIG. 37 is a case where the device provides a navigation video mode switching service through a display in which the vertical length is set smaller than the horizontal length (e.g., a mobile device laid flat).
[0661] As in FIG. 36, the device displays buttons (3711, 3741) on the navigation image that can receive a user input signal requesting a mode switch, for example.
[0662] FIG. 38 illustrates a method for readjusting and rendering a navigation image and displaying it when the aspect ratio of the screen displaying the navigation image of a device (e.g., mobile) changes.
[0663] The rendering method of FIG. 38 illustrates an example of a device described in FIG. 1 to 37 (e.g., the video rendering processor of FIG. 25, 2560) performing a rendering operation.
[0664] Depending on how the mobile is installed in the vehicle, the height-to-width ratio of the display screen may change.
[0665] The device recalculates the height and width of the second image based on the height-to-width ratio of the changed display screen.
[0666] The device recalculates the width of the second image based on the aspect ratio of the reset display screen and the height of the recalculated second image.
[0667] The device performs a process of re-cropping the second image by obtaining a left value based on the pixels of the initially captured image and the width of the second image obtained by recalculation.
[0668] The condition under which the device starts the process of re-extracting the second image is, for example, when the mobile is tilted at an angle that falls within a specific range set as in FIG. 38(b).
[0669] For example, in FIG. 38(a), the reference line (3810) set with the mobile y-axis is rotated or tilted clockwise (3820) or counterclockwise (3830) and is included in the pre-set angle range (a, b) as in FIG. 38(b).
[0670] The device may be configured to start a process of re-extracting a second image when the angle formed by the baseline (3810) with the x-axis is included within the set angle range (a, b).
[0671] Similarly, an angle range can be set for the x-axis and z-axis as well, so that when the mobile baseline falls within the angle range, the device can be configured to start the process of re-extracting the second image.
[0672] Rotation or tilting of the mobile can be detected by the device's sensors.
[0673] The setting angle range (a, b) at which the device initiates the process of re-extracting the second image may be changed depending on the device or the user's selection.
[0674] The rendering operation of the device according to the embodiments illustrated in FIG. 38 is identical to one or more rendering operations described in FIG. 24 to FIG. 37 (for example, the operation of the video rendering processor of FIG. 24 rendering forward view images of the driving path in AR mode).
[0675] A detailed description of one or more methods / operations described in FIGS. 24 to 37 is omitted.
[0676] FIG. 39 is a flowchart (3900) illustrating a method of controlling a device according to embodiments.
[0677] FIG. 39 shows a flow diagram (3900) illustrating a method in which a device according to the embodiments described in FIG. 24 to 38 renders and provides a navigation image.
[0678] A device equipped with a navigation function according to the embodiments (or a controller of the device (2530)) starts operation (3911) and executes the navigation function (3910).
[0679] A device (e.g., the device (2500) of FIG. 25) includes a mobile or cell phone (hereinafter, a mobile device).
[0680] A device (or camera of the device (2540)) according to the embodiments captures a first image of the mobile device surrounding the device (3920).
[0681] A device (or a video rendering processor (2560) of the device) according to the embodiments performs a rendering step (3930) to generate a second image based on a plurality of reference lines extracted from a captured first image, and the second image does not include images of the vehicle interior and exterior objects.
[0682] A device (or a video rendering processor (2560) of the device) according to the embodiments obtains a third image (3940) by deleting object images inside and outside the vehicle based on a second baseline.
[0683] Among the multiple reference lines, the second reference line includes the boundary line of an object image inside and outside the vehicle that obscures the lane.
[0684] A device according to the embodiments (or a video rendering processor (2560) of the device) detects whether the first baseline is located at the ½ point of the third image, or is located above or below the ½ point, and calculates the height value of the AR mode image (second image) (3950).
[0685] Among the multiple reference lines, the first reference line includes a vanishing point.
[0686] The device (or controller of the device (2530)) according to the embodiments calculates the width of the second image based on the height value of the second image and the best ratio of height to width of the screen to be displayed (3960).
[0687] A device (or controller of the device (2530)) according to the embodiments obtains a left value based on the width value of the second image and the set pixel value when capturing the first image (3970).
[0688] The device (or the video rendering processor (2560) of the device) according to the embodiments extracts (crops) the second image and displays it (3980).
[0689] The XR content providing device (or control unit (2530)) according to the embodiments performs at least one of the one or more methods / operations described in FIGS. 24 to 37.
[0690] A detailed description of one or more methods / operations described in FIGS. 24 to 37 is omitted.
[0691] FIGS. 40 to 41 are a flowchart (4000) and a drawing illustrating a method in which a device according to embodiments automatically calculates a second reference line using an algorithm such as a DNN (Neural Network) to render a navigation image.
[0692] The rendering method of FIG. 40 illustrates an example of a device described in FIG. 1 to 39 (e.g., the video rendering processor of FIG. 25, 2560) performing a rendering operation.
[0693] The device according to the embodiments starts the execution of a navigation function.
[0694] The device starts a process for rendering a navigation image, for example (4010).
[0695] The device can detect obstacles in the forward field of view from preview images (preview driving video images) captured by a camera using the DNN method of an AI (artificial intelligence) algorithm.
[0696] Obstacles to the forward view may include images of the bonnet or dashboard.
[0697] The device uses deep learning of a DNN (deep neural network) of an AI (artificial intelligence) algorithm to detect obstacles in the forward field of view.
[0698] The device extracts the position of the bonnet or dashboard from the preview image (e.g., 4111 in FIG. 41).
[0699] The device sets the height of the area to be cropped (Crop Height, e.g., 4113 in FIG. 41) using the Bonnet Position of the bonnet image in the driving video image (Crop Height = Bonnet Position) (4011).
[0700] The device calculates the ratio (diff) between the height of the preview image and the position of the bonnet image (diff = Preview Height / Bonnet Position) (4012)
[0701] To obtain the width of the crop area, the device calculates the height of the preview image, the diff of the position of the bonnet image, and the width of the preview image (Crop Width = Preview Width * diff) (4013).
[0702] The device calculates the width offset of the crop area using the width of the preview image and the crop width (Width offset = (Preview Width - Crop Width) / 2) (4014).
[0703] The offset of the width of the crop area includes the width length offset to the left when extracting the crop area from the preview image (e.g., 4124 in FIG. 41).
[0704] The device extracts an image by applying the acquired values as the left and top values of the area to be cropped (Crop Left = Width offset, Crop Top = 0)(4015).
[0705] The device can set the top value of the area to be cropped to “0”. For example, in FIG. 41(b), the top of the preview image is set as the top of the area to be cropped.
[0706] The device validates the validity of the area to be cropped (4016).
[0707] When the device determines that the validation result is successful, it applies cropping by scaling up the area to be cropped to match the ratio of the preview image size (4017).
[0708] The device enlarges the image of the cropped area to the size of the preview image and sets it as the navigation video image (for example, enlarges 4120 in FIG. 41(b) to 4130 in FIG. 41(c).
[0709] If the device determines that the validation result is not successful, it applies a crop by applying a default value to the area to be cropped (4019).
[0710] The device finishes rendering the navigation image (4018, 4020)
[0711] FIGS. 41(a) to 41(c) are drawings illustrating the operation of rendering a navigation image of the device of FIG. 40.
[0712] FIG. 41(a) shows that, according to the rendering operation (4011) of the device of FIG. 40, the device extracts the position (4111, 4121) of the bonnet or dashboard and sets the height (4113) of the area to be cropped.
[0713] FIG. 41(b) shows obtaining the width (4112, 4122) of a crop area according to the rendering operation (4012, 4013) of the device of FIG. 40.
[0714] The device calculates the ratio (diff) of the height (height, 4114) of the preview image and the position (Bonnet Position, 4113) of the bonnet image (diff = Preview Height / Bonnet Position).
[0715] To obtain the width (4122) of the crop area, the device calculates the ratio (diff) of the height (4114) of the preview image and the position (4113, 4123) of the bonnet image and the width (4115) of the preview image (Crop Width = Preview Width * diff).
[0716] The device calculates the width offset (Width offset, 4124) of the crop area using the width of the preview image (4115) and the crop width (4122) (Width offset = (Preview Width - Crop Width) / 2).
[0717] The device extracts an image by applying the acquired values as the left and top values of the area to be cropped (4120) (Crop Left = Width offset, Crop Top = 0).
[0718] The device can set the top value of the area to be cropped to “0”.
[0719] In Fig. 41(b), the top part of the preview image (4110) is set as the top part of the cropping area (4120).
[0720] When the device verifies the validity of the area to be cropped and determines it to be successful, it scales up the area to be cropped (4120) in the direction of the arrow (4125) to match the ratio of the preview image size, and applies the crop by scaling it up to the same size as the preview image (4130).
[0721] The rendering operation of the device according to the embodiments of FIGS. 40 to 41 is identical to one or more rendering operations described in FIGS. 24 to 39 (for example, the operation of the video rendering processor of FIG. 24 rendering forward view images of the driving path into AR mode navigation images).
[0722] A detailed description of one or more methods / operations described in FIGS. 24 to 39 is omitted.
[0723] As described in FIGS. 24 to 41, the components of the device may be composed of one or more processors combined with memory, and one or more controllers execute instructions stored in memory to perform the operation of the device described in FIGS. 24 to 41. Since the specific details are the same as those described in FIGS. 24 to 41, a detailed description is omitted.
[0724] Executable instructions for performing operations of a device according to embodiments of the present invention may be stored in non-transient CRMs or other computer program products configured to be executed by one or more processors, or may be stored in transient CRMs or other computer program products configured to be executed by one or more processors. In addition, memory according to embodiments of the present invention may be used as a concept that includes not only volatile memory (e.g., RAM, etc.) but also non-volatile memory, flash memory, PROM, etc.
[0725] The components of the device according to the embodiments of the present invention described in FIGS. 1 to 41 may each be composed of separate hardware (e.g., a chip, a hardware circuit, a communication-capable device, etc.) or may be composed of a single piece of hardware. In addition, at least one of the components of the device according to the embodiments of the present invention may be composed of one or more processors capable of executing a program.
[0727] FIG. 42 is a block diagram of a mobile terminal according to an embodiment of the present invention.
[0728] Referring to FIG. 42, the mobile terminal (100) may include a mobile phone, a smartphone, a laptop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, a head-mounted display (HMD)), etc.
[0729] The mobile terminal (100) can provide an augmented reality navigation screen while mounted in a vehicle. The mobile terminal (100) can be mounted inside a vehicle such that the display (151) faces the cabin of the vehicle and at least one camera (121) faces the front of the vehicle.
[0730] A mobile terminal (100) may include a wireless communication unit (110), at least one camera (121), a gyroscope sensor (143), an accelerometer sensor (144), a display (151), a memory (170), at least one processor (180), and a power supply unit (190).
[0731] The mobile terminal (100) may further include a wireless communication unit comprising a broadcast reception module, a mobile communication module, a wireless internet module, a short-range communication module, and a location information module. The mobile terminal (100) may further include an input unit comprising a microphone and a user input unit. According to an embodiment, a camera (121) may be classified as a sub-component of the input unit. The mobile terminal (100) may further include a sensing unit comprising a proximity sensor and an illuminance sensor. The mobile terminal (100) may include an output unit comprising an audio output unit, a haptic module, and an optical output unit. According to an embodiment, a display (151) may be classified as a sub-component of the output unit. The mobile terminal (100) may further include an interface unit for exchanging power or data with other devices.
[0732] The wireless communication unit (110) may include one or more modules that enable wireless communication between the mobile terminal (100) and a wireless communication system, between the mobile terminal (100) and another mobile terminal (100), or between the mobile terminal (100) and an external server. The wireless communication unit (110) may include one or more modules that connect the mobile terminal (100) to one or more networks.
[0733] The wireless communication unit (110) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network built according to technical standards or communication methods for mobile communication (e.g., GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced, 5G), etc.).
[0734] The camera (121) can process image frames, such as still images or video, obtained by an image sensor. The processed image frames can be displayed on a display unit (151) or stored in a memory (170). The camera (121) includes at least one of a camera sensor (e.g., CCD, CMOS, etc.), a photo sensor (or image sensor), and a laser sensor.
[0735] The camera (121) can acquire a front image of the vehicle. The acquired front image of the vehicle can be stored in memory (170) or transmitted to a processor (180). The camera (121) can be activated or deactivated based on a control signal generated by the processor (180).
[0736] The camera (121) may include a first camera and a second camera.
[0737] The first camera can acquire a front image of the vehicle. The first camera can acquire a narrow-angle image compared to the second camera. The first camera can be positioned around the second camera. The first camera can be activated or deactivated based on a control signal generated by the processor (180).
[0738] The second camera can acquire a front image of the vehicle. The second camera can be activated to acquire a front image in a deactivated state according to a request signal from the processor (180). The second camera can acquire a wide-angle image compared to the first camera. The second camera can be activated or deactivated based on a control signal generated by the processor (180).
[0739] The gyroscope sensor (143) allows the mobile terminal (100) to measure the angular velocity of the mobile terminal (100). The gyroscope sensor (143) can generate first sensing data based on the measured angular velocity.
[0740] An acceleration sensor (144) can measure the acceleration of a mobile terminal (100). The acceleration sensor (144) can generate second sensing data based on the measured acceleration.
[0741] The display (151) displays (outputs) information processed by the mobile terminal (100). For example, the display unit (151) can display information on the execution screen of an application running on the mobile terminal (100), or UI (User Interface) and GUI (Graphic User Interface) information based on such execution screen information.
[0742] The display unit (151) may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.
[0743] Additionally, there may be two or more display units depending on the implementation form of the mobile terminal (100). In this case, a plurality of display units may be spaced apart or integrally arranged on one surface of the mobile terminal (100), or they may be arranged on different surfaces respectively.
[0744] The display unit (151) may include a touch sensor that detects a touch on the display unit (151) so that it can receive control commands via a touch method. By using this, when a touch occurs on the display unit (151), the touch sensor detects the touch, and the processor (180) may generate a control command corresponding to the touch based on this. The content input via the touch method may be characters or numbers, instructions in various modes, or selectable menu items. In this way, the display unit (151) may form a touch screen together with the touch sensor, and in this case, the touch screen may function as a user input unit.
[0745] The memory (170) stores data that supports various functions of the mobile terminal (100). The memory (170) can store a number of application programs (or applications) running on the mobile terminal (100), data for the operation of the mobile terminal (100), and instructions. At least some of these application programs may be downloaded from an external server via wireless communication. In addition, at least some of these application programs may exist on the mobile terminal (100) from the time of shipment for the basic functions of the mobile terminal (100) (e.g., phone incoming and outgoing functions, message reception and outgoing functions). Meanwhile, the application programs may be stored in the memory (170), installed on the mobile terminal (100), and driven by the processor (180) to perform the operation (or function) of the mobile terminal.
[0746] The memory (170) can store a program for the operation of the processor (180) and can also temporarily store input / output data (e.g., phonebook, message, still image, video, etc.). The memory (170) can store data regarding various patterns of vibration and sound output when a touch input is made on the touch screen.
[0747] The memory (170) may include at least one type of storage medium among flash memory type, hard disk type, SSD type (Solid State Disk type), SSD type (Silicon Disk Drive type), multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. The mobile terminal (100) may be operated in conjunction with web storage that performs the storage function of the memory (170) on the internet.
[0748] The processor (180) can be electrically connected to and exchange signals with the camera (121), gyroscope sensor (143), accelerometer sensor (144), display (151), memory (170), and power supply (190). The processor (180) can be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, and other electrical units for performing functions. The processor (180) can be driven by power provided from the power supply (190).
[0749] The processor (180) controls the overall operation of the mobile terminal (100). The processor (180) can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output from above, or by running an application stored in memory (170).
[0750] The processor (180) can receive a front view of the vehicle.
[0751] The processor (180) can calibrate the front image.
[0752] The processor (180) can perform calibration by cropping the front image. The front image may contain areas that are unnecessary for the navigation screen configuration. The processor (180) can remove these unnecessary areas. The processor (180) can crop a first area in the front image where at least a part of the vehicle is located. Here, at least a part of the vehicle may include at least one of a hood, a roof, an A-pillar, a dashboard, and a mount. The processor (180) can crop the first area based on image processing using a histogram.
[0753] The processor (180) can output navigation-related information to the cropped area. For example, the processor (180) can display at least one of time information, location information, information on the road being driven, origin information, departure time information, destination information, distance remaining to the destination, time remaining to the destination, and estimated time of arrival information in the cropped area.
[0754] The processor (180) can perform calibration by using a wide-angle image. The processor (180) can determine the balance of the front image in the up, down, left, and right directions based on at least one of a horizontal line passing through a vanishing point detectable in the front image acquired by the first camera and a vertical line passing through said vanishing point. The processor (180) can determine whether to use the wide-angle image acquired by the second camera based on the balance of the front image in the up, down, left, and right directions. The processor (180) can overlay a first AR graphic object at a point in the front image corresponding to a first object detected in the front image acquired by the first camera. The processor (180) can determine whether to use the wide-angle image based on the position of the first AR graphic object in the front image.
[0755] The processor (180) can perform calibration by adjusting the position of a vanishing line detectable in the front image. The processor (180) can position a vanishing line detectable in the left-right direction in the front image at the center of the top-bottom direction of the navigation screen.
[0756] The processor (180) can perform calibration by compensating for the mounting tilt value of the mobile terminal (100). The processor (180) can determine the mounting tilt value based on first sensing data received from the gyroscope sensor (143) and second sensing data received from the accelerometer sensor (144). The processor (180) can compensate for the mounting tilt value on the navigation screen. The processor (180) can compensate for the first sensing data and the second sensing data using a Kalman filter. According to an embodiment, the processor (180) can determine the mounting tilt value based further on data received from the geomagnetic sensor and the temperature sensor. According to an embodiment, the processor (180) can determine and compensate for the mounting tilt value based on third sensing data received from the gravity sensor.
[0757] The processor (180) can run an augmented reality navigation application so that an augmented reality navigation screen is displayed on the display (151). The augmented reality navigation screen may include at least one AR (Augmented Reality) graphic object and a calibrated forward image.
[0758] The processor (180) can receive at least one of a home button input signal and another application execution input signal through a user input unit. In this case, the processor (180) can disable the camera (121) so that the acquisition of the front image is stopped. The processor (180) can run an augmented reality navigation application in the background. Through such control, the waste of battery and processing power caused by unnecessarily acquiring and processing the front image can be prevented.
[0759] The processor (180) can obtain information on the vehicle's stationary state by processing the front image or receiving a signal from the vehicle. When the processor (180) obtains information on the vehicle's stationary state, it can disable the camera (121). In this case, the processor (180) can run an augmented reality navigation application so that a navigation screen is displayed with the front image and AR graphic objects excluded. The processor (180) can obtain information on the vehicle's movement state after the vehicle's stationary state by processing the front image or receiving a signal from the vehicle. When the processor (180) obtains information on the vehicle's movement state after the vehicle's stationary state, it can enable the camera (121). The processor (180) can calibrate the front image obtained from the enabled camera based on calibration data prior to the deactivation of the camera (121).
[0760] The processor (180) can receive at least one of a signal, information, and data from a server through a wireless communication unit (110). The processor (180) can receive augmented reality navigation application data from the server.
[0761] The processor (180) can receive navigation information from a server and run a navigation application based on the navigation information. The navigation information may include map data and AR graphic objects.
[0762] The processor (180) can receive map data from a server. The map data may include at least one of SD (standard definition) map data and HD (high definition) map data. The processor (180) can download the entire map data at a specific time and store it in memory (170) for use. The processor (180) can download only a portion of the entire map data at a preset price, store it in memory (170), and use it temporarily. The map data that has been used may be deleted. The processor (180) can receive AR graphic object (e.g., AR indicator) data from a server.
[0763] The processor (180) can download only a portion of the data from the entire map data based on the set path, store it in memory (170), and use it temporarily. The processor (180) can receive map data corresponding to the path generated by the destination entered by the user and store the received map data in memory (170). The processor (170) can receive AR graphic objects corresponding to the path along with the map data and store the received AR graphic objects in memory (170). The processor (170) can delete the map data and AR graphic objects stored in memory (170) after using them.
[0764] The processor (180) can run a navigation application by receiving updated navigation information when navigation information is updated during navigation operation based on navigation information received from a server. The processor (180) can change a pre-set route according to the updated navigation information.
[0765] The processor (180) can receive navigation information corresponding to the predicted driving route in real time. The processor (180) can adjust the amount of navigation information received corresponding to the section including the predicted driving route according to the communication strength. For example, when the communication strength is greater than or equal to a first reference value, the processor (180) can receive navigation information corresponding to a relatively long section.
[0766] According to the embodiment, a plurality of surfaces (navigation map surface, AR camera surface, AR GUI surface, and navigation GUI surface) described below may be generated on a server. A processor (180) may receive a plurality of surfaces from a server and configure a navigation screen. The processor (180) may adjust the number of surfaces received according to the communication strength. For example, when the communication strength is weak, the processor (180) may receive only navigation GUI surfaces, and then receive additional AR GUI surfaces as the communication strength gradually increases.
[0767] The processor (180) receives AR graphic information processed by the server and can output an AR indicator based on the received AR graphic information. If a Point of Interest (POI) pre-set by the augmented reality navigation application exists, the processor (180) can additionally display an AR indicator during AR navigation. The processor (180) can receive POI information from the server in real time. When receiving POI information, the processor (180) can also receive an AR indicator corresponding to the POI.
[0768] The power supply unit (190) receives external power and internal power under the control of the processor (180) and supplies power necessary for the operation of each component. The power supply unit (190) includes a battery, and the battery may be an internal battery that is rechargeable and may be detachably coupled to the terminal body for charging, etc. The power supply unit (190) may be provided with a connection port, and the connection port may be configured as an example of an interface (160) to which an external charger that supplies power for charging the battery is electrically connected.
[0769] As another example, the power supply unit (190) may be configured to charge the battery wirelessly without using the connection port. In this case, the power supply unit (190) may receive power from an external wireless power transmission device using one or more of an inductive coupling method based on magnetic induction or a magnetic resonance coupling method based on electromagnetic resonance.
[0770] Meanwhile, the mobile terminal (100) may include a computer-readable medium that executes a plurality of steps when driven by a processor (180). The plurality of steps may include receiving a forward image, calibrating the forward image, and running an augmented reality navigation application such that an augmented reality navigation screen including at least one AR graphic object and the calibrated forward image is displayed on a display. The plurality of steps are described in more detail with reference to FIG. 46.
[0771] FIG. 43 is a detailed block diagram of a processor according to an embodiment of the present invention.
[0772] FIG. 44 is a drawing referenced to explain a navigation screen according to an embodiment of the present invention.
[0773] FIG. 45 is a drawing referenced to explain the operation of generating a navigation screen according to an embodiment of the present invention.
[0774] Referring to the drawing, the processor (180) may include a navigation engine (210), an augmented reality engine (220), and a navigation application (230).
[0775] The navigation engine (210) can receive map data and GPS data. The navigation engine (210) can perform map matching based on the map data and GPS data. The navigation engine (210) can perform route planning. The navigation engine (210) can display a map and perform route guidance. The navigation engine (210) can provide route guidance information to the navigation application (230). Meanwhile, the navigation engine (210) may include a navigation controller (211). The navigation controller (211) can receive map matching data, map display data, and route guidance data. The navigation controller (211) can provide route data, POI (point of interest) data, etc. to the AR engine (220). The navigation controller (211) can provide route guidance data and a map display frame to the navigation application (230).
[0776] The AR engine (220) may include an adapter (221) and a renderer (222). The adapter (221) may receive forward image data from a camera (121), first sensing data from a gyroscope sensor (143), and second sensing data from an accelerometer sensor (144). According to an embodiment, the adapter may receive sensing data from at least one of a geomagnetic sensor, a temperature sensor, and a gravity sensor.
[0777] Calibration of the front image can be performed based on data provided in the calibration factor database. The AR engine (220) can perform object detection based on the front image data and root data. The AR engine (220) can perform prediction and interpolation.
[0778] The renderer (222) can perform rendering based on root data, POI data, and prediction and interpolation result data. The renderer (222) can provide AR GUI (graphical user interface) frames and AR camera frames to the navigation application (230).
[0779] A navigation application (230) can generate an augmented reality navigation screen. As illustrated in FIG. 44, the augmented reality navigation screen may include a navigation map surface (310), an AR camera surface (320), an AR GUI surface (330), and a navigation GUI surface (340). The navigation application (230) can generate a navigation map surface (310) based on a map display frame provided by a navigation controller (211). The navigation application (230) can generate an AR camera surface (320) based on an AR camera frame provided by a renderer (222). The navigation application (230) can generate an AR GUI surface (330) based on an AR GUI frame provided by a renderer (222). The navigation application (230) can generate a navigation GUI surface (340) based on the route guidance data provided by the navigation controller (211).
[0780] Referring to FIG. 45, when the navigation application (230) starts (S410), the navigation application (230) can create a navigation map surface (310), an AR camera surface (320), an AR GUI surface (330), and a navigation GUI surface (340) (S420). The navigation application (230) can provide parameters of the AR camera surface (320) and parameters of the AR GUI surface (330) to the AR engine (220) (S430). The AR engine (220) can register a callback function to receive forward image data from the camera server (171) (S440). The camera server (171) can be understood as a concept included in memory (170). The AR engine (220) can receive and crop the forward image data (S450). The navigation application (230) can display a cropped front image on the AR camera surface (320) (S460). The AR engine (230) can perform AR (S470). The navigation application (230) can display an AR GUI on the AR GUI surface (330) based on the cropped front image (S480).
[0781] FIG. 46 is a flowchart according to an embodiment of the present invention.
[0782] Referring to FIGS. 45 and 46, a method of operation (S500) of a mobile terminal is described. The mobile terminal (100) can provide an augmented reality navigation screen while mounted in a vehicle.
[0783] The processor (180) can receive a vehicle front image (S510). The camera (121) can capture a vehicle front image. The processor (180) can receive a vehicle front image captured by the camera (121). Step S510 may include steps S511, S512, and S513.
[0784] The processor (180) can generate an AR GUI surface (330) and an AR camera surface (320) (S511). The processor (180) can acquire control of the camera (121) and receive a vehicle front image (S512). The processor (180) can display the vehicle front image from the camera (121) on the AR GUI surface (330) and the AR camera surface (320) (S513).
[0785] The processor (180) can calibrate the vehicle front image (S520). Step S520 may include steps S521, S522, S523, S524, S525, and S526.
[0786] When calibration is initiated (S521), the processor (180) can detect and extract a vanishing line, a hood line, and a center line from the vehicle front image (S522, S523). Data regarding the vanishing line, hood line, and center line extracted in step S523 can be stored in a calibration factor DB (172). The calibration factor DB (172) can be included in memory (170). The processor (180) can calculate the height and tilt of the camera (121) (S524). For example, the processor (180) can calculate the height and tilt of the camera (121) based on sensing data (e.g., first sensing data and second sensing data) (S524). The processor (180) can perform calibration on the vehicle front image based on the data obtained in steps S523 and S524 (S525). The processor (180) can terminate the calibration (S526).
[0787] The step of performing calibration (S525) may include the step of the processor (180) cropping a first region in which at least a part of the vehicle is located in the front image. At least a part of the vehicle may include at least one of a hood, a roof, an A-pillar, a dashboard, and a mount.
[0788] The step of performing calibration (S525) may include the step of the processor (180) determining the balance of the upper, lower, left, and right sides of the front image based on at least one of a horizontal line passing through a vanishing point detectable in the front image and a vertical line passing through the vanishing point, and the step of the processor determining whether to use the wide-angle image based on the balance.
[0789] Meanwhile, the method of operation of the mobile terminal (S500) may include the step of overlaying a first AR graphic object at a point in the front image corresponding to the first object detected in the vehicle front image. The step of overlaying may be a sub-component of step S544.
[0790] The step of performing calibration (S525) may include a step in which the processor (180) determines whether to use the wide-angle image based on the position of the first AR graphic object in the front image. The step of performing calibration (S525) may include a step in which the processor (180) positions a vanishing line detectable in the front image in the left-right direction in the center of the navigation screen in the upper-lower direction.
[0791] The method of operation (S500) of the mobile terminal may further include the step of the gyroscope sensor (143) generating first sensing data and the step of the accelerometer sensor (144) generating second sensing data. The step of performing calibration (S525) may include the step of the processor determining the mounting tilt based on the first sensing data and the second sensing data, and the step of the processor compensating the mounting tilt on the navigation screen.
[0792] The processor (180) can drive an augmented reality navigation application so that an augmented reality navigation screen (300) including at least one AR graphic object and a calibrated forward image is displayed on the display (151) (S530). Step S530 may include steps S531 through S549.
[0793] Example of a case where calibration was successful
[0794] The processor (180) can display the calibration result (S531). If it is determined that the calibration is successful (S532), the processor (180) can start the AR mode and then start the navigation mode (S533, S534). The processor (180) can search for a path (S535). The processor (180) can receive and parse path information (S536).
[0795] Meanwhile, the processor (180) can start sensor control (S540). The processor (180) can register various sensor data reception callbacks (S541). The processor (180) can receive and parse IMU sensor data. The IMU sensor includes a gyroscope sensor (143) and an accelerometer sensor (144), and the processor (180) can receive first sensing data and second sensing data as IMU sensor data. The processor (180) can predict and interpolate the driving of the vehicle based on the parsing result of step S546 and the parsing result of step S542 (S543). The processor (180) can display a maneuver on the AR GUI surface (330) (S544). The processor (180) can display an AR graphic object on the AR GUI surface (330). The processor (180) can overlay a first AR graphic object at a point in the front image corresponding to the first object detected in the front image through step S546. Afterward, the processor (180) can guide the path (S537). The processor (180) can guide the path based on the path discovered in step S535 and the maneuver in step S544. The processor (180) can determine whether AR guidance is appropriate and determine whether it is an AR guidance point (S538, S539). If it is determined that it is an AR guidance point, the processor (180) can perform step S544. If it is determined that it is not an AR guidance point, the processor (180) can perform step S537.
[0796] The processor (180) can receive and parse sensor data from the camera (121) (S546). The processor (180) can detect objects (e.g., lanes, other vehicles, motorcycles, pedestrians, etc.) based on images acquired from the camera (121). The processor (180) can recognize lanes based on information about the detected objects (S547). The processor (180) can crop the camera image (S548). The processor (180) can crop the front image based on data stored in the calibration factor DB (172). The processor (180) can display a 3D carpet on the AR camera surface (320) based on the lane recognition data of S547 and the image data of S548 (S549).
[0797] Example of a case where calibration failed
[0798] The processor (180) can display the calibration result (S531). If it is determined that the calibration failed (S532), the processor (180) can start the navigation mode without starting the AR mode (S534). The processor (180) can search for a route (S535). The processor (180) can provide route guidance according to the searched route (S537).
[0799] FIGS. 47a to 47b are drawings referenced to explain the vanishing line, bonnet line, and center line according to an embodiment of the present invention.
[0800] Referring to the drawing, the processor (180) can detect a vanishing line (610), a hood line (620), and a center line (630) in a vehicle front image captured by a camera (121). It can be defined as a horizontal line passing through a vanishing point (601) detected in the vehicle front image. The hood line (620) can be defined as a horizontal line passing through the top of the hood detected in the vehicle front image. The center line (630) can be defined as a vertical line passing through a vanishing point (601).
[0801] According to an embodiment, the processor (180) can perform calibration based on the vanishing line (610), the bonnet line (620), and the center line (630) in response to user input. As illustrated in FIG. 47a, when the vanishing line (610), the bonnet line (620), and the center line (630) are set according to user input, the processor (180) can calibrate the front image to match the position set on each of the vanishing line (610), the bonnet line (620), and the center line (630).
[0802] FIGS. 48a to 48d are drawings referenced to explain a cropping operation according to an embodiment of the present invention.
[0803] As illustrated in FIG. 48a, the processor (180) can detect a vanishing line (710), a hood line (720), and a center line (730) in a vehicle front image captured by a camera (121). As illustrated in FIG. 48b, the processor (180) can delete the area below the hood line (710) in the vehicle front image. The processor (180) can output navigation-related information (740) to the deleted area below the hood line (710). For example, the processor (180) can display at least one of time information, location information, information on the road being driven on, origin information, departure time information, destination information, distance remaining to destination information, time remaining to destination information, and estimated time of arrival information in the area below the hood line (710). The processor (180) can display a navigation map (750) and a navigation GUI (755) in one area of the display (151).
[0804] As illustrated in FIG. 48c, the processor (180) can delete unnecessary areas from the vehicle front image. The processor (180) can delete object areas that do not affect the vehicle's driving. For example, the processor (180) can delete areas corresponding to fixed objects located on the sidewalk in the vehicle front image. In this way, by deleting object areas that do not affect the vehicle's driving, distraction during driving can be prevented. As illustrated in FIG. 48d, the processor (180) can display an AR navigation screen (300).
[0805] FIGS. 49a to 49b are drawings referenced to explain a cropping operation according to an embodiment of the present invention.
[0806] As illustrated in FIG. 49a, the processor (180) can detect a stand line (810). The stand line (810) can be defined as a line formed in a vertical or horizontal direction at the part of the stand of the mobile terminal that protrudes most from the center of the screen. As illustrated in FIG. 49b, the processor (180) can delete the area where the stand is displayed centered on the stand line (810) in order to delete the stand (820).
[0807] FIGS. 50a to 52d are drawings referenced to explain the operation of using a wide-angle image according to an embodiment of the present invention.
[0808] Referring to the drawing, the camera (121) may include a first camera and a second camera. The first camera can acquire a front image of the vehicle. The second camera can acquire a wide-angle image compared to the first camera. The second camera may be activated when the processor (180) determines the use of the wide-angle image while the second camera is in a deactivated state.
[0809] The processor (180) can determine the up-down-left-right balance of the front image based on the vanishing line (910) and the center line (920) in the front image. The processor (180) can determine whether to use the wide-angle image based on the up-down-left-right balance. For example, if the processor (180) determines that the center line (920) is skewed to the left of the front image, it can activate the second camera. The processor (180) can take the image outside the left side of the image acquired by the first camera from the image acquired by the activated second camera and supplement the front image of the first camera.
[0810] Meanwhile, when the vehicle (10) approaches the intersection, the processor (180) can gradually enlarge and display the front image while gradually changing the field of view of the first camera.
[0811] Meanwhile, the processor (180) can determine whether to use a wide-angle image based on road shape data. For example, the processor (180) can determine whether to use a wide-angle image based on road curve data formed to the left or right.
[0812] Reference numeral 1010 of FIG. 51a illustrates a forward image acquired by a first camera. Reference numeral 1020 illustrates a forward image acquired by a second camera. A processor (180) can overlay a first AR graphic object at a point in the forward image corresponding to a first object detected in the forward image. The processor (180) can determine whether to utilize a wide-angle image based on the position of the first AR graphic object in the forward image. As illustrated in FIG. 51b, when a path is formed to the right and an AR graphic object (1030) is located on the right side of the forward image, the processor (180) can take a right area of the wide-angle image of the second camera that deviates from the forward image of the first camera and use it to complement the forward image of the first camera. In this case, as illustrated in FIG. 51c, when using a wide-angle image, the processor (180) may display a combination of the front image of the first camera and a portion of the wide-angle image of the second camera by making the scale smaller than before using the wide-angle image. Alternatively, the processor (180) may process and display only the wide-angle image of the second camera. In this case, the processor (180) may maintain the size of the front area display area. As illustrated in FIG. 51d, when using a wide-angle image, the processor (180) may display a combination of the front image of the first camera and a portion of the wide-angle image of the second camera by making the scale the same as before using the wide-angle image. Alternatively, the processor (180) may process and display only the wide-angle image of the second camera. In this case, the processor (180) may increase the size of the front area display area. The processor (180) may reduce the navigation map display area.
[0813] Reference numeral 1110 of FIG. 52a illustrates a forward image acquired by a first camera. Reference numeral 1120 illustrates a forward image acquired by a second camera. A processor (180) can overlay a first AR graphic object at a point in the forward image corresponding to a first object detected in the forward image. A processor (180) can determine whether to utilize a wide-angle image based on the position of the first AR graphic object in the forward image. As illustrated in FIG. 52b, when a path is formed to the left and an AR graphic object (1130) is located on the left side of the forward image, the processor (180) can take a left area of the wide-angle image of the second camera that deviates from the forward image of the first camera and use it to complement the forward image of the first camera. In this case, as illustrated in FIG. 52c, when using a wide-angle image, the processor (180) may display a combination of the front image of the first camera and a portion of the wide-angle image of the second camera by making the scale smaller than before using the wide-angle image. Alternatively, the processor (180) may process and display only the wide-angle image of the second camera. In this case, the processor (180) may maintain the size of the front area display area. As illustrated in FIG. 52d, when using a wide-angle image, the processor (180) may display a combination of the front image of the first camera and a portion of the wide-angle image of the second camera by making the scale the same as before using the wide-angle image. Alternatively, the processor (180) may process and display only the wide-angle image of the second camera. In this case, the processor (180) may increase the size of the front area display area. The processor (180) may reduce the navigation map display area.
[0814] FIGS. 53a to 53b are drawings referenced to explain the vanishing line adjustment operation according to an embodiment of the present invention.
[0815] Referring to the drawings, the processor (180) can detect a vanishing line (1210) in the forward image. As illustrated in FIG. 53a, the processor (180) can determine whether the vanishing line (1210) is located below the center (1220) in the vertical direction of the navigation screen. In this case, the processor (180) can adjust the area displayed on the navigation screen in the forward image so that the vanishing line (1210) is located at the center (1220). As illustrated in FIG. 53b, the processor (180) can position the vanishing line (1210) at the center (1220) in the vertical direction of the navigation screen.
[0816] The processor (180) can determine whether the vanishing line is positioned above the center of the vertical direction of the navigation screen. In this case, the processor (180) can position the vanishing line at the center of the vertical direction of the navigation screen.
[0817] FIGS. 54a to 54c are drawings referenced to explain the slope value correction operation when a vehicle according to an embodiment of the present invention travels on a slope.
[0818] As illustrated in FIGS. 54a and 54b, the processor (180) can calculate the tilt value of the mobile terminal and calculate the vanishing line when the vehicle is traveling on flat ground to construct a lookup table (1310). The lookup table (1310) can be stored in memory (170). For example, when the tilt of the mobile terminal (100) is 90 degrees, 85 degrees, 80 degrees, and 75 degrees, the respective vanishing line can be calculated to construct the lookup table (1310).
[0819] As illustrated in FIGS. 54c and 54d, when a vehicle is traveling on an incline, the processor (180) can correct the installation incline value of the mobile terminal (100) based on the lookup table (1310). For example, the processor (180) can determine that the installation incline value of the mobile terminal (100) is 75 degrees based on at least one of the first sensing data and the second sensing data. The processor (180) can confirm that the value of the vanishing line is 540. The processor (180) can confirm that the incline value of the mobile terminal corresponding to the vanishing line value is 90 degrees based on the lookup table (1310). The processor (180) can correct the installation incline of the mobile terminal (100) to 90 degrees when the vehicle is traveling on an incline.
[0820] FIGS. 55a to 55c are drawings referenced to explain the operation of a mobile terminal in a specific situation according to an embodiment of the present invention.
[0821] FIG. 55a illustrates a case where an AR navigation application is run in the foreground. The foreground operation of the AR navigation application is as described with reference to FIG. 42 to FIG. 54c.
[0822] FIG. 55b illustrates a case where an AR navigation application is run in the foreground while the camera is disabled. The method of operation of the mobile terminal (100) may further include the steps of the processor (180) obtaining information on the stationary state of the vehicle, the processor (180) disabling the camera (121), and the processor (180) running the augmented reality navigation application so that a navigation screen excluding the front image and AR graphic objects is displayed.
[0823] When the processor (180) obtains information on the vehicle's stationary state, it can disable the camera and run the augmented reality navigation application so that a navigation screen is displayed with the front image and AR graphic objects excluded. The processor (180) can configure the AR navigation screen by excluding the AR GUI surface and AR camera surface from the AR navigation screen and including only the navigation GUI surface and navigation map surface.
[0824] Meanwhile, the method of operation of the mobile terminal (100) may further include the steps of the processor (180) acquiring information on the moving state of the vehicle after the vehicle is in a stopped state, the processor (180) activating the camera, and the processor (180) calibrating the front image acquired from the activated camera (121) based on calibration data prior to the deactivation of the camera (121).
[0825] When the processor (180) obtains information on the vehicle's moving state after the vehicle's stationary state, it can activate the camera (121) and calibrate the front image obtained from the activated camera (121) based on calibration data prior to the deactivation of the camera (121). The processor (180) can calibrate the front image obtained from the activated camera (121) by calling calibration data from the calibration factor DB (172).
[0826] FIG. 55c illustrates a case where an AR navigation application is run in the background. The operation method of the mobile terminal (100) may include the steps of the processor (180) receiving at least one of a home button input signal and another application execution input signal, the processor (180) deactivating the camera (121), and the processor (180) running the augmented reality navigation application in the background.
[0827] When the processor (180) receives at least one of a home button input signal and another application execution input signal, it can disable the camera (121) and run the augmented reality navigation application in the background.
[0828] FIG. 56 is a drawing referenced to explain the operation of a mobile terminal in walking mode according to an embodiment of the present invention.
[0829] Referring to FIG. 56, the mobile terminal (100) can be used in vehicle mode and walking mode. When the mobile terminal (100) is used in vehicle mode, the mobile terminal (100) operates as described with reference to FIG. 42 to FIG. 55.
[0830] When the mobile terminal (100) is used in walking mode, the processor (180) can provide a walking AR navigation screen. The processor (180) can perform calibration so that areas of objects that do not interfere with walking are excluded from the pedestrian front image. The processor (180) can enlarge and display an area for pedestrian guidance in the pedestrian front image and delete the remaining area.
[0831] The foregoing invention may be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also include implementations in the form of a carrier wave (e.g., transmission over the Internet). Furthermore, the computer may include a processor or a processor. Accordingly, the above detailed description should not be interpreted restrictively in all respects and should be considered exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.
Claims
Claim 1 In a mobile device that provides augmented reality images, the mobile device comprises one or more processors; A mobile device comprising: a communication interface for receiving information from a camera; wherein one or more processors determine a vanishing line and an unnecessary area in a first image, which is a front image received from the camera; determine a crop area including an image of an obstacle unnecessary for navigation based on the determination result; when a signal to change the display aspect ratio is received, re-determine the vanishing line and the unnecessary area; re-determine the crop area based on the re-determine result; generate a second image by rendering an image with the crop area removed; transmit the second image to a display; perform calibration on the front image; if the calibration is successful, output a route guidance screen based on the second image in which at least one AR graphic object related to route guidance is overlaid on a point in the front image corresponding to an object detected in the front image; and if the calibration fails, output a route guidance screen based on general map data. Claim 2 A mobile device according to claim 1, wherein, as at least part of the operation of determining the crop area, the one or more processors determine the crop area based on a plurality of reference lines extracted from the first image, wherein the first reference line, which is at least one of the plurality of reference lines, includes a straight line of a horizontal line including one or more vanishing point images in the first image, and the vanishing point image includes a point where the left lane and the right lane of the vehicle's driving road intersect the horizon. Claim 3 In paragraph 2, the one or more processors, as at least part of the operation of determining the crop area: generate a third image in which an image of an obstruction unnecessary for navigation is removed from a first image, and the third image is a mobile device in which a first image is removed within a range from a second baseline to one end of the first image. Claim 4 A mobile device according to claim 1, wherein the size of the first image and the second image are the same. Claim 5 In claim 1, the mobile device, wherein the image of an obstacle unnecessary for the navigation overlaps with the image of the left lane or the right lane in the first image. Claim 6 A mobile device according to claim 1, wherein the image of an obstacle unnecessary for the navigation is an image that obscures a part of the left lane or right lane in the first image so as not to be displayed. Claim 7 In paragraph 3, a mobile device in which one end of the image of an obstacle unnecessary to the navigation is included in the second baseline. Claim 8 In paragraph 2, the one or more processors, as part of an operation to determine a crop area based on a plurality of reference lines extracted from the first image, generate a third image from which an image of an obstacle unnecessary for navigation has been removed from the first image, separate the third image into upper and lower images based on the first reference line—the height of the upper image corresponds to a first value and the height of the lower image corresponds to a second value—obtain a height value of the second image based on a preset first ratio and the smaller value between the first value and the second value—the first ratio is the ratio of the second value to the first value—obtain a width value of the second image based on a preset second ratio and the height value of the second image—the second ratio is the ratio of the width value to the height value of the second image—mobile device. Claim 9 A mobile device according to claim 8, wherein the one or more processors further remove the third image corresponding to a polygonal range including a height from the bottom of the third image to the top of the third image, where the first value is greater than the second value. Claim 10 In paragraph 2, the one or more processors, as part of an operation to determine a crop area based on a plurality of reference lines extracted from the first image, generate a third image from which an image of an obstacle unnecessary for navigation has been removed from the first image, separate the third image into upper and lower images centered on the first reference line - the height of the upper image corresponds to a first value, and the height of the lower image corresponds to a second value -, set the height value of the third image as the height value of the second image, and obtain the width value of the second image based on a pre-set second ratio and the height value of the second image - the second ratio is the ratio of the width value to the height value of the second image -, mobile device. Claim 11 A mobile device according to claim 8, wherein the one or more processors further remove the third image corresponding to a polygonal range including a height from the top of the third image to the bottom of the third image, where the first value is smaller than the second value. Claim 12 A mobile device according to claim 1, wherein the one or more processors acquire a first image, which is a front image received from a camera, according to a set pixel value, and further exclude a third image corresponding to a polygonal area having a width value corresponding to (horizontal pixel value of the first image - width value of the second image) / 2. Claim 13 In paragraph 2, the plurality of reference lines are manually set according to a user input signal, in a mobile device. Claim 14 delete Claim 15 A mobile device according to claim 1, wherein one or more processors acquire calibration factor data from the received first image, store the acquired calibration factor data, and calibrate the first image based on the calibration factor data. Claim 16 A method for providing an augmented reality image of a mobile device comprises: a step of determining a vanishing line and an unnecessary area in a first image, which is a forward image received from a camera; a step of determining a crop area including an image of an obstacle unnecessary for navigation based on the result of the determination; a step of re-determining the vanishing line and the unnecessary area when a signal to change the display aspect ratio is received, and re-determining the crop area based on the result of the re-determining; a step of generating a second image by rendering an image from which the crop area has been removed, and transmitting the second image to a display; a step of performing calibration on the forward image; a step of outputting a route guidance screen based on the second image in which at least one AR graphic object related to route guidance is overlaid on a point in the forward image corresponding to an object detected in the forward image when the calibration is successful; and a step of outputting a route guidance screen based on general map data when the calibration fails. Claim 17 In claim 16, the step of determining the crop area comprises determining the crop area based on a plurality of reference lines extracted from the first image, wherein the first reference line, which is at least one of the plurality of reference lines, includes a straight line of a horizontal line including one or more vanishing point images in the first image, and the vanishing point image includes a point where the left lane and the right lane of the vehicle's driving road intersect the horizon. A method for providing an augmented reality image. Claim 18 In claim 16, the step of determining the crop area further comprises the step of generating a third image in which an image of an obstruction unnecessary for navigation is removed from a first image, and the third image is a method of providing an augmented reality image in which a first image within a range from a second baseline to one end of the first image is removed. Claim 19 In claim 17, the step of determining a crop area based on a plurality of reference lines extracted from the first image comprises: a step of generating a third image in which an image of an obstacle unnecessary for navigation is removed from the first image; a step of separating the third image into upper and lower images based on the first reference line, wherein the height of the upper image corresponds to a first value and the height of the lower image corresponds to a second value; a step of obtaining a height value of the second image based on a pre-set first ratio and the smaller value between the first value and the second value, wherein the first ratio is the ratio of the second value to the first value; and a step of obtaining a width value of the second image based on a pre-set second ratio and the height value of the second image, wherein the second ratio is the ratio of the width value to the height value of the second image. Claim 20 In claim 17, the step of determining a crop area based on a plurality of reference lines extracted from the first image comprises: a step of generating a third image in which an image of an obstacle unnecessary for navigation is removed from the first image; a step of separating the third image into upper and lower images based on the first reference line—where the height of the upper image corresponds to a first value and the height of the lower image corresponds to a second value—; a step of setting the height value of the third image to the height value of the second image; and a step of obtaining the width value of the second image based on a previously set second ratio and the height value of the second image—where the second ratio is the ratio of the width value to the height value of the second image—a method for providing an augmented reality image.
Citation Information
Patent Citations
System for providing image around vehicle with portable terminal and method thereof
KR1020120066194A
Methods and systems for generating route data
KR1020170016473A
Mobile terminal
KR1020180005067A
Method and system for video-based positioning and mapping
KR1020190094405A
Driving path determination for autonomous vehicles
US20170371336A1
