Methods and electronic devices with independent point-of-day control

Abstract

To provide an electronic device having independent time management and a method thereof.SOLUTION: An electronic device according to an embodiment of the present invention determines, based on a clock speed of a time keeping module, detection point of reference detection data generated through detection of a reference sensor of a plurality of sensors, detection points of other detection data of other sensors, time difference between detection points, and delay in work from the detection point of the reference detection data to work completion point, and can correct, based on the work delay, a work result processed by measurement of the reference detection point to output it.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The following disclosure relates to an electronic device and method having independent time management.

Background Art

[0002] In the case of a camera, it is used as a simple video shooting device and precise time control is not required. In a camera, calculations are performed through sequential processing according to the performance of the calculation device via a simple schedule ring. In a camera, calculations are performed in a manner of outputting as soon as a result is obtained. On the other hand, in a system that requires strict time control, such as an aircraft or a robot, calculations are performed using an RTOS (real time OS) to ensure accurate processing time.

[0003] Recently, inexpensive sensor systems have become widespread in general consumer electronic devices such as mobile phones. With the development of new control systems including autonomous driving, sensor fusion operations are being used in various devices. Due to cost and convenience issues in such application fields, it is difficult to use systems such as the aforementioned RTOS. For example, in the case of the SLAM (simultaneous localization and mapping) algorithm in a robot or a vehicle, strict time control is required, but it has been developed as a combination of sensors that is not synchronized with the time of a general-purpose OS, different from such requirements.

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to provide an electronic device and method having independent time management.

Means for Solving the Problems

[0005] An electronic device according to one embodiment includes a timekeeping processor that determines, based on the clock speed of a timekeeping module, the detection time of reference detection data generated via detection by a reference sensor among a plurality of sensors, and the detection time of other detection data from other sensors among the plurality of sensors, determines the time difference between the detection time of the reference detection data and the detection time of the other detection data, and determines the work delay from the detection time of the reference detection data to the completion of the work based on the clock speed of the timekeeping module, and one or more other processors that correct the work result processed by positioning of the electronic device determined based on the determined time difference, the reference detection data, and the other detection data with respect to the detection time of the reference detection data, based on the work delay.

[0006] The electronic device may further include a plurality of sensors that generate the reference detection data and the other detection data, and an output unit that outputs the work result corrected based on the work delay.

[0007] The timekeeping processor determines the rendering delay from the time of detection of the reference detection data to the time when rendering of the image for augmented reality (AR) is completed in order to determine the work delay, and one or more other processors can generate an output image by correcting the rendered image based on the rendering delay, which is determined by the positioning of the electronic device relative to the time of detection of the reference detection data for the purpose of correction.

[0008] The reference sensor may include a vision sensor comprising one or more combinations of a camera sensor for capturing camera images, a radar sensor for detecting radar data, and a lidar sensor for detecting lidar images.

[0009] The reference sensor includes a camera sensor that captures a camera image for generating the detection data, and the timekeeping processor can calibrate the detection time of the camera sensor based on one or both of the exposure time and readout time of the camera image for determining the detection time of the reference detection data.

[0010] The positioning of the electronic device may include one or two of the attitude and position of the electronic device.

[0011] The one or more other processors may perform warping based on one or more combinations of rotational transformation, translation, and scaling changes corresponding to the positioning change of the electronic device during the work delay from the time of detection of the reference detection data to the time of completion of the work for the purpose of correction.

[0012] The other sensors may include one or more combinations of an inertial measurement unit (IMU) for detecting the inertia of the electronic device, a passive sensor that performs detection when a trigger input is received, and a passive sensor that performs detection at predetermined intervals.

[0013] The main processor, among the one or more other processors, determines the positioning of the electronic device based on the calculated time difference, the reference detection data, and the other detection data with respect to the detection time of the reference detection data for the purpose of correction, and the sub-processor, among the one or more other processors, can generate a rendered image as a result of the work on the positioning.

[0014] The electronic device further includes a communication module that transmits the detection time of the detection data from the plurality of sensors and the time difference to an external device and receives the work result from the external device in order to determine the time difference, and the timekeeping processor can determine the work delay based on the time the work result is received in order to determine the work delay.

[0015] The aforementioned work delay may include the transmission delay and reception delay caused by the communication module, and the time it takes for the external device to perform the work.

[0016] A method implemented by a processor according to one embodiment includes the steps of: determining the detection time of reference detection data generated via detection by a reference sensor among a plurality of sensors and the detection time of other detection data from other sensors among the plurality of sensors, based on the clock speed of a timekeeping processor; determining the time difference between the detection time of the reference detection data and the detection time of the other detection data; determining the work delay from the detection time of the reference detection data to the completion of the work, based on the clock speed of the timekeeping processor; and correcting the work result processed by positioning of the electronic device determined based on the determined time difference, the reference detection data and the other detection data, with respect to the detection time of the reference detection data, using one or more other processors based on the work delay.

[0017] An electronic device according to one embodiment includes a processor that, depending on the clock speed of the processor, determines the detection time of first detection data of a first sensor and the detection time of second detection data of a second sensor, determines the time difference between the detection time of the first detection data and the detection time of the second detection data, and determines the work delay of the executed work from the detection time of the first detection data to the completion time of the executed work based on the clock speed of the processor, wherein the work is performed by one or more other processors relative to the detection time of the first detection data based on the determined time, and the one or more other processors can be configured to control the electronic device based on the determined work delay.

[0018] The processor may be configured to determine the detection time of the second detection data in response to receiving a detection trigger for the second detection data from the second sensor, and to determine the detection time of the second detection data at the next subsequent timing, which is updated according to the clock speed of the processor after the detection trigger has been received.

[0019] The processor may be configured to determine the detection time of the first detection data by, in response to receiving a detection trigger for the first detection data from the first sensor, determining the next subsequent timing which is updated according to the clock speed of the processor after the detection trigger is received, determining the calibration time of the first sensor based on one or more processing times of the first sensor, and determining the detection time of the first detection data based on the difference between the calibration time and the next subsequent timing.

[0020] The one or more processing times may include the readout time of the first sensor and the exposure time of the first sensor.

[0021] The clock speed of the processor may differ from the clock speeds of the first sensor and the second sensor, respectively.

[0022] The one or more other processors may be configured to compensate for the work results of the work performed based on the work delay for the control. [Effects of the Invention]

[0023] According to the present invention, it is possible to provide an electronic device and method having independent time control. [Brief explanation of the drawing]

[0024] [Figure 1] An electronic device including a timekeeping module according to one embodiment is shown. [Figure 2]The structure and flowchart of an electronic device according to an embodiment performing operations for work inside the device are shown. [Figure 3] The structure and flowchart of an electronic device according to an embodiment performing operations for work inside the device are shown. [Figure 4] The time stamp determination by a timekeeping module according to an embodiment is described. [Figure 5] The time stamp determination by a timekeeping module according to an embodiment is described. [Figure 6] The timing of performing positioning determination, rendering, and displaying an output image in an electronic device according to an embodiment is shown. [Figure 7] The structure and flowchart of an electronic device according to an embodiment requesting operations for work from an external server are shown. [Figure 8] The structure and flowchart of an electronic device according to an embodiment requesting operations for work from an external server are shown. [Figure 9] An example of image correction considering rendering delay in an electronic device providing augmented reality according to an embodiment is described. [Figure 10] It is a flowchart explaining a method of independently managing and using detection points for positioning according to an embodiment.

Best Mode for Carrying Out the Invention

[0025] The specific structural or functional description of the embodiment is disclosed for the purpose of mere exemplification and can be changed into various forms. Therefore, the embodiment is not limited to a specific disclosed form, and the scope of this specification includes modifications, equivalents, or alternatives included in the technical idea.

[0026] Terms such as "first" or "second" may be used to describe multiple components, but such terms should be interpreted solely for the purpose of distinguishing one component from others. For example, the first component may be named the second component, and similarly, the second component may also be named the first component.

[0027] When it is mentioned that one component is “linked” or “connected” to another component, it should be understood that it is directly linked to or connected to the other component, but that other components may be present in between.

[0028] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood not to presuppose the existence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0029] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this embodiment belongs. Commonly used, predefined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless expressly defined herein.

[0030] The embodiments will be described in detail below with reference to the attached drawings. When describing with reference to the drawings, the same components will be given the same reference numerals regardless of the reference numerals used in the drawings, and redundant explanations for them will be omitted.

[0031] Figure 1 shows an electronic device including a timekeeping module according to one embodiment.

[0032] An electronic device 100 according to one embodiment includes a sensor 110 (e.g., one or more sensors), a timekeeping module 120, a processor 130 (e.g., one or more processors), an output module 140 (e.g., a user interface including a display), and a memory 150 (e.g., one or more memories). For example, the electronic device 100 can determine its position based on detection data generated via multiple sensors. The positioning of the electronic device 100 includes one or both of the pose and location of the electronic device 100. The electronic device 100 may perform a sensor fusion operation on the detection data generated via multiple sensors. The electronic device 100 may perform software synchronization of the multiple sensors in order to perform the sensor fusion operation. For example, software synchronization indicates that in the sensor fusion operation, the timing of detection between angle sensors and the difference between the timings of detection are accurately monitored. The electronic device 100 can determine its position by performing sensor fusion operation, taking into account the difference in the time of detection, even without a forced agreement of the measurement times for detection between angle sensors.

[0033] Sensor 110 may include multiple sensors. The multiple sensors generate detection data for positioning through detection. For example, one or more of the multiple sensors may be heterogeneous sensors that generate a different type of detection data than the other sensors. The reference sensor among the multiple sensors is a sensor that generates reference detection data that serves as the basis for positioning. The other sensors are sensors that generate other detection data used for positioning in addition to the reference detection data. As will be described later with reference to Figure 6, positioning may be determined based on the reference detection data and other detection data, with reference to the detection time of the reference detection data (e.g., the reference detection time).

[0034] For example, the reference sensor may include a vision sensor. The vision sensor may include, for example, a camera sensor, a radar sensor, a LiDAR sensor, and an ultrasonic sensor, as a sensor that performs vision-based detection. The camera sensor captures a camera image. The radar sensor detects radar data. The LiDAR sensor detects a LiDAR image. However, the vision sensor may also include, without limitation, a depth camera sensor, a stereo camera sensor, a multi-camera sensor, and a single-camera sensor.

[0035] Other sensors include one or more combinations of an inertial measurement unit (IMU) for detecting the inertia of the electronic device 100, a passive sensor that performs detection when it receives a trigger input, and a passive sensor that performs detection at predetermined intervals. The IMU can detect 3-axis acceleration and / or 3-axis angular velocity as a sensor that measures the inertia applied to the IMU. The passive sensor performs detection when it receives a trigger input, and can generate detection data irregularly based on the received trigger input. The passive sensor can generate detection data periodically because it performs detection at intervals determined by its own clock. For reference, the timekeeping module 120 can control the timing of the trigger input to the passive sensor and apply the trigger input to the passive sensor.

[0036] For illustrative purposes, this specification primarily describes the reference sensor as a vision sensor (e.g., a camera sensor) and the other sensors as an IMU, but the reference sensor and other sensors are not limited to these.

[0037] The timekeeping module 120, operating at a separate clock speed independent of the processor 130 (described later), can generate, record, calibrate, store, and manage time information related to the target task within the electronic device 100. The time information related to the target task includes the detection time determined for each detected data point, the time difference between detected data points (described later), and the task delay (described later). The timekeeping module 120 can store accurate time information for each detected data point. When calculating the target task, the timekeeping module 120 can provide the stored time information of the detected data to the processor (e.g., an internal application processor 130 or a processor in an external device).

[0038] The target task is an operation performed relative to a determined position and includes one or more actions and / or calculations performed relative to the electronic device 100 at the determined position. For example, if the electronic device 100 is a device that provides augmented reality (AR) and / or virtual reality (VR), the target task includes an action to render an image for augmented reality and / or virtual reality (e.g., an image containing virtual content) that must be provided to the user at the determined position relative to the electronic device 100. In a different example, if the electronic device 100 is embodied as a robot, the target task may include a series of actions (e.g., surrounding recognition and output of recognition results) to perform an action (e.g., an item service, voice guidance, and interaction with surrounding objects such as opening a door) that must be performed relative to the robot at the determined position. As a further example, if the electronic device 100 is mounted on a vehicle (e.g., a car, aircraft, or watercraft), the target operation may include actions for driving and / or flying with positioning determined for the electronic device 100 (e.g., surrounding area recognition and steering and speed adjustment based on the recognition results). However, the aforementioned target operations are merely examples and are not limited thereto.

[0039] The timekeeping module 120 can record the detection time (e.g., timestamp) for each angle sensor. For example, the timekeeping module 120 can determine the detection time (e.g., reference detection time) of reference detection data generated via the detection of a reference sensor among multiple sensors, and the detection time (e.g., other detection time) of other detection data from other sensors, based on the clock speed of the timekeeping module 120. The timekeeping module 120 can record and manage the timestamp corresponding to the reference detection time and the timestamp corresponding to the other detection time, respectively. For reference, the timekeeping module 120 may include a clock generation circuit that operates at a clock speed (e.g., clock frequency) independent of the clock (e.g., clock speed) of the processor 130. The clock speed of the timekeeping module 120 is faster than the detection frequencies of the multiple sensors. The timekeeping module 120 can record and manage each detection time more accurately to determine it independently of the processor 130, using a clock speed faster than the sensor detection frequencies. For example, the clock speed of the timekeeping module 120 may be more than 100 times faster than the fastest detection frequency of the multiple sensors. However, this is merely an example of a clock speed and is not limited to it.

[0040] The timekeeping module 120 may include a memory for storing time information related to the target work and timing calibration information for timing calibration of the time information. The timekeeping module 120 can also calculate (e.g., determine) a more precise detection point by calibrating the detection point using the timing calibration information. The timing calibration information will be explained with reference to Figure 5 below.

[0041] The timekeeping module 120 can calculate the time difference between the detection time of the reference detection data and the detection time of other detection data. For example, the timekeeping module 120 calculates the time difference between other detection times based on the reference detection time. As will be described later, the electronic device 100 acquires (for example, determines) positioning based on the reference detection time.

[0042] The timekeeping module 120 can determine the task delay from the time of detection of reference detection data to the task completion time point, based on the clock speed of the timekeeping module 120. The task delay represents the latency time required from the reference detection time to the task completion time. The timekeeping module 120 can record the task completion time (for example, the time when rendering is completed) and calculate the time from the last reference time immediately before the start of the task to the task completion time as the task delay. For example, if the target task is rendering for augmented reality, the task completion time may be the time when rendering is completed. The task delay includes the rendering delay. In one embodiment, the timekeeping module 120 can determine the rendering delay from the time of detection of reference detection data to the time when rendering of the image for augmented reality is completed.

[0043] The processor 130 can acquire the positioning of the electronic device 100, which is determined based on the time difference calculated with respect to the detection time of the reference detection data, the reference detection data, and other detection data. The processor 130 can acquire the work results processed for the positioning. If the target work is rendering for augmented reality, the work results include an image rendered for the user to be provided with the positioning at the reference detection time. For example, a field of view corresponding to the viewing direction that the user looks at may be determined based on the position and orientation of the electronic device 100 included in the positioning. The work results may include an augmented reality image having virtual content that must be provided as a field of view corresponding to the viewing direction.

[0044] The processor 130 can correct the work result processed by the positioning of the electronic device 100 determined based on the time difference calculated relative to the detection time of the reference detection data, and the work delay. The processor 130 can compensate for changes in positioning (e.g., movement and change of direction) that occurred during the work after the reference detection time in the work result. For example, if the target work is rendering for augmented reality, the processor 130 can generate an output image by correcting the rendered image based on the rendering delay, according to the positioning of the electronic device determined relative to the detection time of the reference detection data.

[0045] Exemplary, the processor 130 can be embodied as one or more processors, such as an application processor including a main processor and subprocessors. As will be described later with reference to Figure 2, the operation of determining the position of the electronic device 100 and the operation of processing tasks corresponding to the determined position (e.g., rendering) may be performed by the application processor. However, without limiting this, as a different example, as will be described later with reference to Figure 7, the operation of determining the position of the electronic device 100 and the operation of processing tasks corresponding to the determined position may be performed by an external device, and the processor 130 of the electronic device 100 may obtain the determined position and the results of the work performed by the position from the external device.

[0046] Hardware including the output module 140 can output work results corrected based on work delays. Depending on the form of the work result, the output module 140 can output one or more combinations of visual, auditory, and tactile information. For example, if the target work is rendering for augmented reality, the output module 140 includes a display that shows the output image generated for providing augmented reality. The output module 140 can, for example, generate a user interface and render an image using the generated user interface.

[0047] Memory 150 can store instruction words that, when performed by the processor 130, configure the processor 130 to perform one, a combination of, or all of its operations. Memory 150 may include, for example, RAM (random-access memory), DRAM (dynamic RAM), SRAM (static RAM), or other types of non-volatile memory known in the relevant art.

[0048] In one embodiment, the electronic device 100 can perform more precise motion estimation by acquiring positioning (e.g., attitude and position) relative to a reference detection point based on detection data from various sensors during the actual calculation of the target work. Furthermore, the electronic device 100 can ultimately provide the user with a result that compensates for positioning changes from the time the positioning determination at the reference detection point up to the time the work result is output (e.g., a corrected work result). This enables the user to be provided with accurate results.

[0049] As described above, the electronic device 100 may include a separate timekeeping module 120 independent of the application processor. The timekeeping module 120 includes a clock generation circuit that operates as a relatively accurate clock (e.g., a clock more accurate than the clocks of one or more sensors of sensor 110). The timekeeping module 120 can monitor the operating time of the sensor, the time required for the sensor to operate (e.g., detection), and the time required for the processor 130 and / or external devices to perform calculations for the target task. Thus, the timekeeping module 120 can calculate the end-to-end latency (e.g., work delay) from the time of sensor detection (e.g., a timestamp recorded for detection) based on the monitored time information. Therefore, the electronic device 100, when implemented in a real-time system for measuring or controlling motion (e.g., AR devices, SLAM devices, and various mechanical control devices), can minimize errors caused by end-to-end latency in end-to-end calculations.

[0050] For sensor fusion, precise time synchronization of sensors included in the hardware can be achieved. In one embodiment, the electronic device 100 can apply sensor fusion even to sensors with different clock speeds by recording and managing the detection time for each sensor via the timekeeping module 120, as described above. For example, since the output of the angle sensor is generated by the individual clock of each sensor, the output periods between different types of sensors may be asynchronous. Also, the commonly used sensors periodically output and transmit sensor data according to the clock speed they each maintain, and the clock speeds generated by the clock circuits installed in each sensor exhibit considerable errors.

[0051] In addition to errors or differences in clock speed between sensors, the sampling period for detected data from sensors may change due to electromagnetic or communication reasons. For example, in the case of low-cost IMUs commonly used in mobile phones, the clock speed can vary by approximately 10% depending on the circumstances. According to a comparative embodiment, the electronic device 100 can calculate velocity or attitude by integrating the angular velocity or acceleration detected via the IMU. When using the integration of angular velocity or acceleration, changes based on time can be linearly reflected in the measurement error in estimation using IMU data. In camera sensors, even when capturing images at the same fps (frames per second), the shutter speed and data processing speed can vary from camera sensor to camera sensor. Therefore, data with different delay times and different sampling periods may be generated when measuring each frame image. This can make it difficult to identify data measured at the same time as data generated via the first camera sensor, among data generated via the first camera sensor, each with different delay times. For reference, some sensors (e.g., camera sensors) are designed to receive trigger inputs, but controlling the trigger input timing can be difficult.

[0052] Despite the aforementioned errors, the electronic device 100 accurately records the detection time based on the clock speed of the timekeeping module 120, enabling positioning operations (e.g., SLAM) based on sensor fusion. In one embodiment of the electronic device 100, the detection time is recorded for each sensor regardless of whether a trigger is present or not, ensuring diversity and flexibility in sensor selection during the development phase. Furthermore, the electronic device 100 can prevent or assist in preventing delay problems caused by computation time that occur in augmented reality (AR). The electronic device 100 can minimize errors due to end-to-end latency by accurately managing the variable delay caused by the difference in computation amount in each situation between the work scheduling results of the general-purpose operating system (OS) and the processor 130.

[0053] Figures 2 and 3 show the structure and flowchart of an electronic device according to one embodiment, in which calculations for work are performed inside the device.

[0054] In Figure 2, the sensor 210 (for example, the sensor 110 shown in Figure 1) includes a first sensor and a second sensor, the timekeeping module 220 (for example, the timekeeping module 120 shown in Figure 2) includes a timekeeping processor 221 and memory 222, and the application processor 230 (for example, the processor 130 shown in Figure 1) includes a main processor 231 and subprocessor 232. The main processor may be a CPU (central processing unit) and the subprocessor may be a GPU (graphics processing unit), but are not limited to these. The output module 140 shown in Figure 1 can be implemented as a display 240. The following description will mainly focus on examples where the target task is to provide augmented reality images for augmented reality.

[0055] If the target task is to generate augmented reality images for augmented reality, the main operations of the electronic device 200 include determining position (e.g., SLAM (Simultaneous Localization and Mapping) vision), modeling the environment surrounding the electronic device 200 into a three-dimensional geometric structure (3D geometry) (e.g., interpreting and / or modeling surrounding physical objects, people, and background as three-dimensional coordinates), and generating and / or rendering augmented reality images that include virtual content (e.g., virtual objects). The operations to determine position and model the surrounding environment into a three-dimensional geometric structure may be performed by the main processor 231. Rendering of the augmented reality images is performed by the subprocessor 232. For example, one or more processors of the application processor 230, the main processor 231, can determine the positioning of the electronic device 200 based on the time difference calculated relative to the detection time of reference detection data (e.g., first detection data) (e.g., the difference between detection times from the first detection to the second detection), the reference detection data (e.g., first detection data), and other detection data (e.g., second detection data). One or more processors, the subprocessor 232, can generate a rendered image as a result of the positioning work. An example of the electronic device 200, including the main processor 231 and the subprocessor 232, performing calculations related to the work itself will be described with reference to the following.

[0056] For example, in step S311, the first sensor 211 generates first detection data by performing a first detection. The timekeeping module 220 can receive a detection trigger for the first detection in response to the occurrence of the first detection (for example, the first sensor 211 can generate and transmit a detection trigger for the first detection to the timekeeping module 220 in response to the execution of the first detection). The detection trigger is a signal that informs the timekeeping module 220 that the angle sensor has generated detection data, and exemplifies that the sensor has completed image capture and / or detected acceleration. In step S312, the timekeeping module 220 determines and records a timestamp for the first detection data at the clock timing when the detection trigger was received. The recording of clock timing and timestamp will be explained with reference to Figures 4 and 5 below. In step S321, the second sensor 212 generates second detection data by performing a second detection. The timekeeping module 220 can receive a detection trigger for the second detection in response to the occurrence of the second detection. In step S322, the timekeeping module 220 determines and records a timestamp for the second detection data based on the clock speed of the timekeeping module 220. For reference, the timekeeping module 220 can determine the aforementioned detection point based on the clock speed of the timekeeping processor 221 and store the determined detection point in the memory 222.

[0057] In steps S313 and S323, the first sensor 211 and the second sensor 212 each transmit detection data to the main processor 231. For reference, Figures 2 and 3 show only the first sensor 211 and the second sensor 212 as examples, but are not limited to these. In this specification, the sensor may include n sensors, where n is an integer of 2 or more.

[0058] For reference, the timekeeping module 220 may generate and provide a trigger input to a sensor (e.g., a passive sensor) among the first sensor 211 and the second sensor 212 that is capable of receiving (or configured to receive) a trigger input. The sensor configured to receive a trigger input performs a detection in response to receiving the trigger input. Therefore, the timing of the trigger input may also be determined based on the clock speed of the timekeeping module 220. In this specification, the first sensor 211 (e.g., a reference sensor) is mainly described using a camera sensor as an example, but other vision sensors such as radar sensors and lidar sensors can also be used as reference sensors in the same manner as camera sensors, without being limited thereto.

[0059] In step S331, the timekeeping module 220 calculates the time difference for each detected data. For example, the timekeeping module 220 can calculate the time difference between detected data based on the last timestamp of the first sensor 211. The timekeeping module 220 can calculate and record the time difference between the detection time of other detected data (e.g., the timestamp of the second detected data) and the detection time of other detected data (e.g., the timestamp of the second detected data) based on the last detection time of the first detected data collected up to the time the positioning is determined, in other words, the most recent detection time (e.g., the most recent reference time) (e.g., the timestamp of the first detected data).

[0060] In step S332, the timekeeping module 220 transmits time-related information to the main processor 231. For example, the timekeeping module 220 can transmit to the main processor 231 time information monitored for the detected data (e.g., timestamp for each detected data and the time difference calculated between detected data). For reference, the timekeeping module 220 may transmit all the time information at once, but is not limited to this. While the main processor 231 and subprocessor 232 perform calculations for their work, the timekeeping module 220 may selectively provide the main processor 231 and / or subprocessor 232 with some of the time information requested for each calculation.

[0061] In step S333, the main processor 231 of the electronic device 200 determines the position of the device at a reference detection time (e.g., the timestamp of the first detection data) of the first sensor 211 (e.g., the reference sensor) based on the aforementioned time information. The main processor 231 can determine the position of the electronic device 200 based on the reference detection time that is temporally adjacent to the positioning time (e.g., the latest reference detection time). The positioning time indicates the time when the electronic device 200 starts calculations for determining the position. In step S334, the main processor 231 transmits the determined position (e.g., AR device positioning) to the subprocessor 232.

[0062] In step S341, the subprocessor 232 renders an image for a reference detection time (e.g., the timestamp of the first detection data) based on the determined positioning. For example, the subprocessor 232 renders and generates an augmented reality image corresponding to the field of view in the positioning of the electronic device 200 at the reference detection time. In step S342, the subprocessor 232 transmits a rendering completion trigger to the timekeeping module 220. For example, the subprocessor 232 notifies the timekeeping module 220 of rendering completion in response to generating a rendered image. In step S343, the timekeeping module 220 determines the rendering completion time based on the clock speed of the timekeeping module 220.

[0063] In step S351, the timekeeping module 220 calculates the rendering delay from the latest reference detection time (e.g., the timestamp of the first detection data) to the rendering completion time. For example, the timekeeping module 220 can calculate the rendering delay by subtracting the latest reference detection time from the rendering completion time. In step S352, the timekeeping module 220 transmits the calculated rendering delay to the main processor 231. In step S353, the main processor 231 can generate an output image by compensating the rendered image based on the calculated rendering delay. For example, in step S353, the main processor 231 may perform a delay compensation calculation 239. The delay compensation calculation 239 includes warping based on one or more combinations of rotational transformation, translation, and scale change corresponding to the positioning change of the electronic device 200 during the work delay from the detection time of the reference detection data to the work completion time. For reference, although it has been explained that the delay compensation calculation 239 is performed by one of the processors of the application processor 230, it is not limited to this and may be performed by a separate processor in the electronic device (for example, a dedicated processor for delay compensation). In step S354, the main processor 231 transmits the corrected image to the display 240. In step S355, the display 240 displays the generated output image.

[0064] In one embodiment, the electronic device 200 can synchronize the coordinates between real and virtual objects with low latency in the AR and MR fields. The electronic device 200 also achieves low latency in mobile computing and / or edge computing, preventing coordinate mismatch between real and virtual objects due to delay. To prevent coordinate mismatch, the electronic device 200 has a timekeeping module 220 that operates on a separate clock from the main processor 231 and subprocessor 232, which separately manages time information related to positioning and can provide the main processor 231 with time information obtained through individual calculations.

[0065] Figures 4 and 5 illustrate timestamp determination by a timekeeping module according to one embodiment.

[0066] A timekeeping module according to one embodiment (for example, one or two of timekeeping modules 120 and 220) can determine the detection time of detection data generated based on detections from all sensors, based on the clock speed of a timekeeping processor (for example, timekeeping processor 221). The timekeeping module records a timestamp determined based on the clock speed of the timekeeping module for each generated detection data. For example, the kth detection data may be generated based on the kth detection of a certain sensor. The timekeeping module receives a detection trigger from the sensor indicating the generation of the kth detection data, where k may be an integer of 1 or more. The timekeeping module can determine the clock timing immediately after receiving the detection trigger (the next clock timing generated after the detection trigger has been received once) as the detection time for the detection data. The detection time is recorded as a timestamp. All sensors included in the electronic device can detect, generate, and transmit data at a period determined by their own clock circuit (for example, a period determined by the clock speed determined by the individual clock circuit). Since the output of each angle sensor is generated by an individual clock, the output period for each angle sensor can be asynchronous.

[0067] Figure 4 illustrates the operation of determining the detection time of a passive sensor (for example, a second sensor 212 in a non-limiting embodiment). An IMU is used as an example of a passive sensor. As an inertial sensor, the IMU can detect, for example, acceleration and / or angular velocity in three axes. The IMU can operate with its own clock 420, but it can operate at a faster speed than a camera sensor (for example, a first sensor 211 in a non-limiting embodiment). If the IMU is a low-cost sensor, it can operate at a predetermined clock speed without a trigger. The IMU can generate IMU data as detection data by detecting the acceleration and / or angular velocity in three axes applied to the electronic device at each period determined by its own clock speed. The clock error of the IMU can be up to about 10% if it is a low-cost sensor. In response to receiving an IMU data detection trigger 421 from the IMU, the timekeeping module may store a timestamp (e.g., a timestamp for the second detection data) indicating the most recent (e.g., next subsequent) timing 411, updated by the clock rate of the timekeeping processor after the detection trigger 421, as the detection time 429 for the IMU data.

[0068] For reference, if the clock 410 of the timekeeping module is sufficiently fast, the difference 490 between the actual time of generation of the IMU data and the detection time indicated by the timestamp is trivial. For example, if the clock 410 of the timekeeping module is faster than the clock 420 of the IMU, the error can be ignored. Exemplaryly, the clock speed of the timekeeping module may be more than 100 times faster than the fastest sampling rate of any of the sensors. Therefore, the error can be ignored. The timekeeping module can have a clock speed that is better than the error level of the IMU.

[0069] Figure 5 illustrates the operation that determines the detection timing of the camera sensor among the vision sensors.

[0070] According to one embodiment, the reference sensor of the electronic device (for example, a first sensor 211, as a non-limiting example) includes a camera sensor among vision sensors that captures camera images. The camera sensor can generate image data (for example, camera data) by capturing a scene. The camera sensor according to one embodiment can perform continuous shooting at a determined FPS. Figure 5 shows the clock 510 of the timekeeping module and the detection time axis 520 of the camera sensor.

[0071] The timekeeping module may also use the most recent clock time (e.g., a timestamp for the first detection data) for the camera sensor, similar to the IMU described above with reference to Figure 4. For example, the detection time of camera detection data from the camera sensor may be determined by a timestamp indicating the clock timing 511 of the timekeeping module after the detection trigger 521 indicating the acquisition of the k-th image. For example, in response to receiving a detection trigger 521 for camera detection data from the camera sensor, the timekeeping module stores a timestamp indicating the most recent time updated by its own clock speed after receiving the detection trigger 521 as the detection time of the camera detection data. Thus, the timekeeping module can record the detection time of the k-th image data based on the clock timing at the time when the reading of camera data is completed. The timekeeping module may be designed to have a clock speed that is sufficiently faster than the sampling rate (e.g., the data generation frequency) of the sensor, including the camera sensor.

[0072] However, it is not limited to this. Additionally, the detection time of camera data may be calibrated. For example, the timekeeping module can calibrate the detection time of the camera sensor based on one or two of the exposure time 522 and readout time 523 of the camera image. For example, the generation time 524 of the camera data may be treated as a point in time 524 corresponding to the middle of the exposure time 522. On the other hand, the detection trigger for camera detection data may occur after the exposure time 522 and readout time 523 have elapsed and readout is complete. In other words, there may be a difference between the generation time 524 of the camera detection data and the readout time of the camera detection data. The difference between the generation time 524 of the camera detection data and the readout time of the camera detection data can be calculated by the timekeeping module as a calibration time 525. As shown in the example in Figure 5, the difference between the generation time 524 of the camera detection data and the readout time can be expressed as shown in the following formula (1).

[0073]

number

[0074] For example, the camera sensor can provide the timekeeping module with an exposure value indicating an exposure time of 522. The timekeeping module can store the exposure time 522 in memory (e.g., memory 222). The timekeeping module can determine a detection point calibrated for the camera detection data by subtracting a time corresponding to half of the exposure time 522 from the clock timing 511 immediately after the occurrence of the detection trigger 521.

[0075] Furthermore, it may be assumed that the readout time 523 is constant for each shot taken by the camera sensor. The timekeeping module may pre-store the readout time 523 measured for the camera sensor in memory. The timekeeping module may determine a detection time 529 that is more precisely calibrated for the camera detection data by subtracting the calibration time 525 (for example, half of the exposure time 522 and the sum of the readout time 523) from the clock timing 511 immediately after the occurrence of the detection trigger 521.

[0076] Figure 6 shows the timing of position determination, rendering, and display of the output image in an electronic device according to one embodiment.

[0077] In one embodiment of the electronic device, the timekeeping module 620 can determine the detection time for the detection data of the sensor 610. For example, the timekeeping module 620 may determine the detection time for each of the camera sensor 611, IMU 612, passive sensor 613, and passive sensor 614. The electronic device may correspond to one, a combination of, or all of the electronic devices shown in Figures 1 to 5, as non-limiting examples.

[0078] In step S601, the timekeeping module determines a positioning timestamp when positioning is initiated by the main processor 631. The positioning timestamp is determined based on the reference detection data from the most recent reference detection time among the reference detection times (e.g., reference detection times) of the reference detection data collected up to the time point in which positioning is performed. The detection time of the vision sensor may be the reference detection time for positioning. As described above, the reference detection data may be camera detection data detected by the camera sensor 611 as a vision sensor.

[0079] In step S602, the main processor 631 (for example, the main processor 231 shown in Figure 2) determines the position based on the positioning timestamp. For example, the main processor 631 can determine the position of the electronic device (e.g., attitude and position) based on the positioning timestamp, using the detection data and detection time of the sensor 610 as a reference to the time corresponding to the positioning timestamp.

[0080] In step S603, if positioning is determined for a positioning timestamp, the main processor 631 instructs the start of work (e.g., rendering). For example, the main processor 631 may instruct the subprocessor 632 (e.g., subprocessor 232 shown in Figure 2) to start rendering.

[0081] In step S604, the timekeeping module 620 determines and records the point in time that signals the completion of rendering based on the clock speed of the timekeeping processor (e.g., the timekeeping processor of the timekeeping module 620). The timekeeping module 620 can calculate the rendering delay, which is the time required from the positioning timestamp to the completion of rendering. Therefore, the timekeeping module 620 can calculate the rendering delay, which includes the time from the point in time corresponding to the positioning timestamp 601 to the start of rendering, and the time from the start of rendering to the completion of rendering.

[0082] In step S606, the main processor 631 warps the rendered image based on the rendering delay. The main processor 631 may predict the positioning change of the electronic device from the positioning timestamp to the rendering completion time and perform warping corresponding to the predicted positioning change. The main processor 631 can generate an output image by warping the rendered image. Warping is an operation that transforms the coordinate system of pixels contained in the rendered image, and warping will be explained with reference to Figure 9 below.

[0083] In step S609, the display 640 (for example, the display 240 shown in Figure 2) displays an output image. For example, the display 640 can provide the user with an output image by forming an image on a transparent or translucent material. The output image may include a pair of first images (for example, a left image) and second images (for example, a right image), and the display 640 can provide the user with a stereoscopic image with depth by directing the first image and the second image toward the part corresponding to the user's left eye and the part corresponding to the user's right eye, respectively.

[0084] As described above, the electronic device can monitor and record the moment data calculation is completed (for example, the moment the rendered image is transmitted to the display) via the timekeeping module 620. Therefore, the electronic device can directly calculate the end-to-end delay from sensor detection to rendering completion. The electronic device according to one embodiment can reduce not only the error due to the difference in detection times between the sensors 610 described above with reference to Figures 1 to 5, but also the error due to rendering delay explained with reference to Figure 6. Therefore, the electronic device can significantly reduce recognition and rendering errors while the electronic device is in operation.

[0085] Figures 7 and 8 show the structure and flowchart of an electronic device according to one embodiment that requests calculations for a task from an external server.

[0086] In the electronic device 700 shown in Figure 7, the sensor 710 including the first sensor 711 and the second sensor 712, the timekeeping module 720 including the timekeeping processor 721 and memory 722, and the display 740 operate in the same manner as described above with reference to Figure 2, and therefore their description is omitted. The electronic device 700 can correspond to one, a combination of, or all of the electronic devices shown in Figures 1 to 6, as non-limiting examples. Unlike the electronic device 200 shown in Figure 2, the electronic device 700 shown in Figure 7 can request a series of calculations for the target task from a separate computing device 760 (e.g., an external device). The separate computing device 760 is embodied as an edge computing device or cloud server having a main processor 761 and subprocessors 762 for calculating the target task.

[0087] The electronic device 700 may further include a communication module 750. In one embodiment, the transmitter 751 of the communication module 750 can transmit the detection time and time difference between detection data from a plurality of sensors 710 to an external device (for example, another computing device 760). The receiver 752 of the communication module 750 can receive work results from the external device.

[0088] The timekeeping module 720 can calculate the work delay based on the time the work result is received. The work delay includes the transmission delay and reception delay due to communication by the communication module 750, and the time spent on work by external devices. Therefore, the electronic device 700 calculates only accurate time information, including the delay due to communication, and a separate computing device 760 performs calculations for the target work, thereby increasing the efficiency of all calculations.

[0089] The operation of the electronic device 700 and the external device will be explained in more detail with reference to Figure 8.

[0090] For example, in step S813, the first sensor 711 and the second sensor 712 transmit detection data to the communication module 750. The communication module 750 can then transmit the detection data to the main processor 761 of a separate computing device 760.

[0091] In step S831, the timekeeping module 720 calculates the time difference between detection data based on the last timestamp of the first sensor 711. In step S832, the communication module 750 transmits the calculated time difference and time information, including the detection time of each detection data, to a separate computing device 760. In step S833, the main processor 761 of the separate computing device 760 determines the positioning of the electronic device 700 at the last timestamp. In step S834, the main processor 761 transmits the positioning of the AR device (e.g., electronic device 700) to the subprocessor 762.

[0092] In step S841, the subprocessor 762 renders an image for the determined positioning. In step S842, the communication module 750 receives the rendered image from a separate computing device 760. In step S843, the timekeeping module 720 determines the time when the communication module 750 receives the rendered image as the rendering completion time.

[0093] In step S851, the timekeeping module 720 determines the rendering delay from the timestamp corresponding to the positioning time to the time the rendered image is received. In step S852, the timekeeping module 720 transmits the calculated rendering delay to the processor 730. In step S853, the processor 730 corrects the rendered image using the calculated rendering delay. For example, the processor 730 may perform a delay compensation calculation 731 (e.g., image warping). In step S854, the processor 730 transmits the corrected image (e.g., output image) to the display 740. In step S855, the display 740 outputs the corrected image.

[0094] Figure 9 illustrates an example of image correction considering rendering delay in an electronic device providing augmented reality according to one embodiment.

[0095] An electronic device 900 according to one embodiment can be implemented as an augmented reality glasses device, as shown in Figure 9. As described above with reference to Figures 1 to 8, the electronic device 900 can correct the rendered image based on rendering delay. One or more processors of the electronic device 900 may perform warping based on one or more combinations of rotational transformations, translations, and scale changes corresponding to changes in the positioning of the electronic device 900 during the work delay from the time of detection of reference detection data to the time of work completion. Rotational transformations, translations, and scale changes may be represented as coordinate transformation matrices between views of the electronic device 900. The views of the electronic device 900 change with the positioning of the electronic device 900 (e.g., attitude and position) and, exemplary, correspond to the direction and position viewed by the vision sensor (e.g., camera sensor) of the electronic device 900. In other words, the views of the electronic device 900 are interpreted as corresponding to camera views, and the coordinate transformation matrices between views can be represented similarly to the coordinate transformation matrices of external camera calibration. One or more processors of the electronic device 900 may perform one, a combination of, or all of the operations described above with reference to Figures 1 to 8. Furthermore, one or more memories of the electronic device 900 may store instruction words that configure one or more processors to perform one, a combination of, or all of the operations described above with reference to Figures 1 to 8, when performed by one or more processors. For example, the electronic device 900 corresponds to one, a combination of, or all of the electronic devices shown in Figures 1 to 8, as a non-limiting example.

[0096] Warping refers to the operation of generating an output image by applying a coordinate transformation matrix, which includes at least one of rotation transformation parameters, translation parameters, and scale change parameters, to the rendered image. The coordinate transformation matrix calculated based on the rendering delay may be a coordinate transformation matrix that transforms coordinates in a first coordinate system corresponding to a view located at a position determined as a positioning timestamp (e.g., a first view direction) to coordinates in a second coordinate system corresponding to a view located at a position predicted at the time rendering is complete (e.g., a second view direction). The electronic device 900 can rotate and translate each pixel coordinate of the augmented reality image containing virtual content generated in the first coordinate system, using the coordinates in the aforementioned second coordinate system.

[0097] For example, in Figure 9, the electronic device 900 may determine a positioning timestamp while facing a first view direction 911. Then, while rendering is taking place, the positioning of the electronic device 900 may change from the first view direction 911 to a second view direction 912. The view direction may be, exemplarily, the direction in which the reference vector of the electronic device 900 is facing, the direction in which a vision sensor (e.g., a camera sensor) is being viewed. The electronic device 900 may perform warping 950, which transforms the coordinates of the virtual content generated in a first coordinate system corresponding to the first view direction 911 to a second coordinate system corresponding to the second view direction 912. Thus, despite the positioning change 919 during the rendering delay, the electronic device 900 can output the virtual content 991 to world coordinates that precisely fit to an actual object 990 (e.g., a desk).

[0098] Figure 10 is a flowchart illustrating a method for independently managing and using detection time points for positioning according to one embodiment.

[0099] First, in step S1010, the electronic device generates detection data for positioning through detection by multiple sensors. For example, the electronic device can generate reference detection data by performing one of the following actions: capturing a camera image with a camera sensor, detecting radar data with a radar sensor, and detecting a lidar image with a lidar sensor. The electronic device calibrates the detection time of the camera sensor based on one or both of the exposure time and readout time of the camera image. The electronic device can also generate other detection data by performing one or more combinations of the following actions: detecting the inertia of the electronic device with an inertial measurement unit, performing detection when a trigger input is received by a passive sensor, and performing detection at predetermined intervals by a passive sensor.

[0100] Then, in step S1020, the electronic device determines the detection time of the reference detection data generated via the detection of the reference sensor and the detection time of other detection data from the other sensors based on the clock speed of the timekeeping module.

[0101] Next, in step S1030, the electronic device calculates the time difference between the detection time of the reference detection data and the detection time of the other detection data.

[0102] Then, in step S1040, the electronic device determines the work delay from the time of detection of the reference detection data to the time of completion of the work, based on the clock speed of the timekeeping module. For example, the electronic device can determine the rendering delay from the time of detection of the reference detection data to the time when the rendering of the image for augmented reality is completed.

[0103] Next, in step S1050, the electronic device corrects the work result processed by the positioning of the electronic device, which is determined based on the time difference calculated relative to the detection time of the reference detection data, the reference detection data, and other detection data, based on the work delay. For example, the electronic device can generate an output image by correcting the image rendered by the positioning of the electronic device, which is determined relative to the detection time of the reference detection data, based on the rendering delay. As explained with reference to Figure 9, the electronic device can perform warping based on one or more combinations of rotational transformation, translation, and scaling changes corresponding to changes in the positioning of the electronic device during the work delay from the detection time of the reference detection data to the completion of the work.

[0104] For example, an electronic device may determine its position based on a time difference calculated with respect to the detection point of a reference detection data, the reference detection data, and other detection data. As a result of the positioning work, the electronic device can generate a rendered image.

[0105] As a different example, the electronic device may request rendering from an external device (e.g., a separate computing device) and receive the rendered image from the external device. In this case, in step S1030 described above, the electronic device transmits the detection time and time difference of detection data from multiple sensors to the external device. Alternatively, in step S1040, the electronic device may receive the work result from the external device and calculate the work delay based on the time the work result was received.

[0106] Then, in step S1060, the electronic device outputs the work result corrected based on the work delay. For example, the electronic device can display the output image on a display.

[0107] The operation of the electronic device is not limited to what has been described above, and may be performed in parallel or chronologically along with the operations described above, as shown in Figures 1 to 9.

[0108] An electronic device according to one embodiment can be applied to all forms of devices that incorporate a sensor system requiring time synchronization. The timekeeping module can be implemented as chip-type hardware or as a combination of hardware and software. The electronic device can be implemented, for example, as an AR device, a vehicle sensor system, and a wireless communication device equipped with a camera and IMU. Furthermore, an electronic device according to one embodiment can be applied to robot control systems, and electronic systems for mechanical, aerospace, and marine control in general.

[0109] An electronic device according to one embodiment can measure and record the precise time of data measured by an angle sensor and the precise time of occurrence of calculation results using the measured data. The electronic device can correct motion estimation errors using the measured time and improve the delay of rendering results. The electronic device can store and calibrate a pre-calculated measurement time error (e.g., calibration information) according to the characteristics of the sensor in order to accurately calculate the estimated delay.

[0110] The electronic devices, sensors, timekeeping module, processor, output module, first sensor, second sensor, memory, main processor, subprocessor, display, timekeeping processor, communication module, transmitter, receiver, separate computing device, camera sensor, IMU, passive sensor, passive sensor, electronic device 100, sensor 110, timekeeping module 120, processor 130, output module 140, memory 150, electronic device 200, sensor 210, first sensor 211, second sensor 212, timekeeping module 220, processor 221, memory 222, processor 230, main processor 231, subprocessor 232, and D The display 240, sensor 610, camera sensor 611, IMU 612, passive sensor 613, passive sensor 614, timekeeping module 620, main processor 631, subprocessor 632, display 640, electronic device 700, sensor 710, first sensor 711, second sensor 712, timekeeping module 720, timekeeping processor 721, memory 722, processor 730, display 740, communication module 750, transmitter 751, receiver 752, separate computing device 760, main processor 761, subprocessor 762, electronic device 900, and other devices, units, modules, devices, and components can be implemented as hardware components.

[0111] The embodiments described above are embodied in hardware components, software components, or combinations of hardware and software components. For example, the devices and components described in these embodiments are embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or different devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the OS. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For convenience of understanding, the processing device may sometimes be described as being used as a single unit, but a person with ordinary skill in the art will understand that the processing device includes multiple processing elements and / or multiple types of processing elements. For example, the processing device includes multiple processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

[0112] Software includes computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or instruct the processing unit independently or in combination. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, for interpretation by a processing unit or for providing instructions or data to a processing unit. Software can be distributed across a network of computer systems and stored and executed in a distributed manner. Software and data can be stored on a recording medium readable by one or more computers.

[0113] The method according to this embodiment is embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executed by a computer using an interpreter or the like.

[0114] The hardware device described above may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0115] As described above, although embodiments have been illustrated with limited drawings, a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described technique may be performed in a different order than described, and / or the described system, structure, apparatus, circuit, and other components may be combined or assembled in a different manner than described, or replaced or substituted by other components or equivalents, and still achieve the desired results.

[0116] Therefore, other embodiments, other embodiments, and claims equivalent to those described below also fall within the scope of the claims. [Explanation of Symbols]

[0117] 110: Sensor 120: Timekeeping Module 130: Processor 140: Output module

Claims

1. A method implemented using electronic devices, The steps include determining, based on the clock speed of the timekeeping processor of the electronic device, the detection time of the reference detection data generated via the detection of a reference sensor among the multiple sensors of the electronic device, and the detection time of other detection data from other sensors among the multiple sensors, A step of determining the time difference between the detection time of the aforementioned reference detection data and the detection time of the other detection data, The steps include determining the work delay from the time of detection of the reference detection data to the time of completion of the work based on the clock speed of the timekeeping processor, A step of correcting the work result processed by positioning of the electronic device, determined based on the determined time difference, the reference detection data, and the other detection data, with respect to the detection time of the reference detection data, using one or more other processors of the electronic device based on the work delay, A method that includes this.

2. The steps include generating the reference detection data and the other detection data using the plurality of sensors, The steps include outputting the work result corrected based on the aforementioned work delay, The method according to claim 1, further comprising:

3. The step of determining the work delay includes determining the rendering delay from the time the reference detection data is detected until the rendering of the image for augmented reality is completed. The method according to claim 1, wherein the correction step includes a step of generating an output image by correcting the image rendered by the positioning of the electronic device determined with respect to the detection time of the reference detection data based on the rendering delay.

4. The method according to claim 1, wherein the step of generating the detection data includes generating the reference detection data by performing one or more combinations of the following operations: capturing a camera image with the camera sensor of the reference sensor; detecting radar data with the radar sensor of the reference sensor; and detecting a lidar image with the lidar sensor of the reference sensor.

5. The step of generating the detection data includes the step of capturing a camera image using the camera sensor, which is the reference sensor. The method according to claim 1, wherein the step of determining the detection time of the reference detection data includes the step of calibrating the detection time of the camera sensor based on one or two of the exposure time and readout time of the camera image.

6. The method according to claim 1, wherein the positioning of the electronic device includes one or both of the attitude and position of the electronic device.

7. The method according to claim 1, wherein the correction step includes a step of performing warping based on one or more combinations of rotational transformation, translation, and scale change corresponding to the positioning change of the electronic device during the work delay from the time of detection of the reference detection data to the time of completion of the work.

8. The method according to claim 1, wherein the step of generating the detection data includes generating the other detection data by performing one or more combinations of the following: an operation to detect the inertia of the electronic device by the inertia measuring unit of the sensor; an operation to perform detection when a trigger input is received by the passive sensor of the sensor; and an operation to perform detection at predetermined intervals by the passive sensor of the sensor.

9. The correction step includes determining the position of the electronic device based on the determined time difference with respect to the detection time of the reference detection data, the reference detection data, and the other detection data, The steps include generating a rendered image as a result of the positioning operation, The method according to claim 1, including the method described in claim 1.

10. The step of determining the time difference includes transmitting the detection time of the detection data from the plurality of sensors and the time difference to an external device. The method according to claim 1, wherein the step of determining the work delay includes receiving the work result from the external device and determining the work delay based on the time at which the work result was received.

11. The method according to claim 10, wherein the work delay includes a transmission delay in the step of transmitting to the external device, a reception delay when receiving the work result from the external device, and the time spent performing the work on the external device.

12. A computer-readable recording medium storing one or more computer programs including instruction words for performing the method according to any one of claims 1 to 11.

13. An electronic device comprising a timekeeping processor and one or more other processors, configured to perform the method according to any one of claims 1 to 11.

14. An electronic device, The system includes a processor that, depending on the processor's clock speed, determines the detection time of first detection data from a first sensor and the detection time of second detection data from a second sensor, determines the time difference between the detection time of the first detection data and the detection time of the second detection data, and determines the work delay of the executed work from the detection time of the first detection data to the completion time of the executed work based on the processor's clock speed. The above operation is performed by one or more other processors on the detection time of the first detection data based on the determined time, The one or more other processors are configured to control the electronic device based on the determined work delay.

15. The electronic device according to claim 14, wherein the processor is configured to determine the detection time of the second detection data in response to receiving a detection trigger for the second detection data from the second sensor, and to determine the detection time of the second detection data at the next subsequent timing which is updated according to the clock speed of the processor after the detection trigger has been received.

16. The electronic device according to claim 14, wherein the processor is configured to determine the detection time of the first detection data in response to receiving a detection trigger for the first detection data from the first sensor, to determine the next subsequent timing which is updated according to the clock speed of the processor after the detection trigger has been received, to determine a calibration time for the first sensor based on one or more processing times of the first sensor, and to determine the detection time of the first detection data based on the difference between the calibration time and the next subsequent timing.

17. The electronic device according to claim 16, wherein the one or more processing times include the readout time of the first sensor and the exposure time of the first sensor.

18. The electronic device according to claim 14, wherein the clock speed of the processor is different from the clock speeds of the first sensor and the second sensor, respectively.

19. The electronic device according to claim 14, wherein the one or more other processors are configured to compensate for the work results of the work performed based on the work delay for the control.

Images