Processing device and method for processing detected data

Abstract

To provide a detection data processing device that detects and manages a key point using sensors with overlapping viewing angles.SOLUTION: A detection data processing device according to one embodiment tracks, on the basis of detection data collected in a previous frame and a current frame via a reference sensor, a plurality of key points within a total observation range determined on the basis of a combination of viewing angles of sensors, manages a new key point together with the plurality of key points in response to detection of the new key point, and thereby can position the device on the basis of the managed key points.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The following disclosure relates to the processing of detection data.

Background Art

[0002] Estimating the two-dimensional or three-dimensional pose of an object is an important technology for many vision applications such as augmented reality, CCTV, navigation, control devices, and robot applications.

[0003] Over the years, the interest in augmented reality technology has been gradually increasing. One of the basic functions of augmented reality technology is three-dimensional interaction. That is, it is to display enhanced information by overlapping it on three-dimensional objects in the real world.

[0004] In order to obtain an actual visual effect with three-dimensional interaction, the augmented information (or enhanced information) and the three-dimensional pose of the actual object must match. Here, it is necessary to obtain the two-dimensional or three-dimensional pose information of the actual object. For obtaining the pose information, estimation using key points can be used.

Summary of the Invention

Problems to be Solved by the Invention

[0005] A detection data processing device according to an embodiment is to detect and manage key points using sensors with overlapping viewing angles.

Means for Solving the Problems

[0006] The detection data processing device according to the present invention includes a reference sensor that generates detection data by capturing a scene in the field of view (FOV), and a processor that tracks multiple keypoints within the overall observation range of the detection data processing device based on detection data collected in previous frames via the reference sensor and detection data collected in the current frame, and manages new keypoints together with the tracked multiple keypoints in response to the detection of new keypoints, wherein the overall observation range is determined based on a combination of the field of view of the reference sensor and the second sensor.

[0007] The processor can determine the moved positions of keypoints in at least some of the multiple keypoints for at least a portion of the overall observation range.

[0008] The processor can determine the moved position of keypoints among the keypoints tracked based on detection by the reference sensor for regions of the reference sensor's field of view that do not overlap with the field of view of the second sensor, and can also determine the moved position of keypoints among the keypoints tracked based on detection by the second sensor for regions of the second sensor's field of view that do not overlap with the field of view of the reference sensor.

[0009] The processor can determine the moved position of keypoints among those tracked based on detection by the reference sensor for areas within the reference sensor's field of view that overlap with the second sensor's field of view, and can skip tracking keypoints among those tracked based on detection by the second sensor for areas within the second sensor's field of view that overlap with the reference sensor's field of view.

[0010] The processor can update the region to which each keypoint belongs within the overall observation range, based on the movement of multiple tracked keypoints.

[0011] The processor can exclude keypoints from tracking if their moved location falls outside the overall observation range.

[0012] The processor can classify multiple tracked keypoints into one of two regions—non-overlapping regions or overlapping regions—based on a non-overlapping mask indicating non-overlapping regions and an overlapping mask indicating overlapping regions.

[0013] The processor can remove outliers from among the multiple tracked keypoints based on detection data collected in previous frames via a reference sensor and detection data collected in the current frame.

[0014] The processor can detect new keypoints from overlapping areas where the field of view of the second sensor overlaps with the detection data collected via the reference sensor.

[0015] The detection data processing device further includes an output device that outputs estimated positioning results to the detection data processing device, and the processor can estimate at least one of the position and orientation of the detection data processing device as a positioning result based on the refined results of the tracked keypoints.

[0016] A detection data processing method implemented by a processor according to one embodiment includes the steps of: tracking a plurality of key points within the overall observation range of a detection data processing device based on detection data collected in previous frames and detection data collected in the current frame by capturing a scene in the field of view with a reference sensor; refining the tracked plurality of key points; and, in response to the detection of a new key point, managing the new key point together with the tracked plurality of key points, wherein the overall observation range can be determined based on a combination of the field of view angles of the reference sensor and the second sensor.

[0017] A detection data processing device according to one embodiment includes a first sensor that captures a first image at a first field of view, a second sensor that captures a second image at a second field of view and differs from the first sensor, and one or more processors that determine non-overlapping and overlapping regions between the first and second images, detect keypoints from the non-overlapping and overlapping regions, track changes in the position of keypoints in the non-overlapping and overlapping regions for each consecutive frame of the first and second images, refine the tracked keypoints by removing outliers from the tracked keypoints, and manage the new keypoints together with the tracked keypoints in response to the detection of new keypoints in the current frame.

[0018] The new keypoint can be detected from the overlapping region that overlaps with the field of view of the second sensor, based on the detection data collected by the first sensor.

[0019] The processor can track keypoints in non-overlapping regions and keypoints in overlapping regions, and apply the tracked keypoints to positioning. [Effects of the Invention]

[0020] In one embodiment, the detection data processing device performs positioning by considering both the key points in the overlapping and non-overlapping regions of each sensor, and therefore can provide stable tracking performance even with rapid rotation (for example, the rapid head rotation of a user wearing augmented reality glasses). [Brief explanation of the drawing]

[0021] [Figure 1] The field of view of the sensor according to one embodiment is shown. [Figure 2] An example of region differentiation in detection data according to one embodiment is shown. [Figure 3] This is a flowchart showing a method for processing detected data according to one embodiment. [Figure 4] This is a flowchart showing the tracking of key points according to one embodiment. [Figure 5]Shows the region determination by the position movement of key points according to an embodiment. [Figure 6] Shows a mask for region determination of other key points according to an embodiment. [Figure 7] It is a flowchart showing key point refinement according to an embodiment. [Figure 8] It is a flowchart showing an example of utilization of key points according to an embodiment. [Figure 9] It is a block diagram showing a processing device for detection data according to an embodiment.

Modes for Carrying Out the Invention

[0022] The specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and can be changed into various forms. Therefore, the embodiments are not limited to the specific disclosed forms, and the scope of this specification includes modifications, equivalents or alternatives included in the technical idea.

[0023] Terms such as the first or the second may be used to describe a plurality of components, but such terms must be interpreted only for the purpose of distinguishing one component from another. For example, the first component can be named the second component, and similarly, the second component can also be named the first component.

[0024] When it is mentioned that any component is "connected" or "joined" to another component, it should be understood that it is directly connected or joined to the other component, but there may be other components in between.

[0025] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood not to presuppose the existence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0026] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this embodiment belongs. Commonly used, predefined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless expressly defined herein.

[0027] The embodiments will be described in detail below with reference to the attached drawings. When describing with reference to the drawings, the same components will be given the same reference numerals regardless of the reference numerals used in the drawings, and redundant explanations will be omitted.

[0028] Figure 1 shows the field of view of a sensor according to one embodiment.

[0029] According to one embodiment, the detection data processing device 100 can perform localization operations using detection data acquired through multiple sensors. For example, the detection data processing device 100 can estimate the position and / or orientation of the device based on the detection data. In this specification, camera sensors are mainly described as examples of sensors, but the invention is not limited thereto.

[0030] Figure 1 shows an example in which the detection data processing device 100 includes a pair of camera sensors (e.g., stereo camera sensors). The detection data processing device 100 can acquire image pairs (e.g., stereo images) in the same time frame via the stereo camera sensors. Based on the image pairs, the detection data processing device 100 can calculate the depth from itself to a point in the scene captured by the stereo camera sensors. In the case of parallel stereo camera sensors, where the principal axis of each camera sensor is configured to be parallel to the principal axis of the other camera sensor, the overlapping region 153 where the fields of view overlap may be wider than the non-overlapping regions 151,152. The detection data processing device 100 using parallel stereo camera sensors can calculate the depth to a point corresponding to a pixel for the wide overlapping region 153 as described above.

[0031] This specification primarily describes examples in which the detection data processing device 100 uses a divergent stereo camera sensor. A divergent stereo camera sensor is a camera sensor in which, of a pair of camera sensors, the principal axis of one camera sensor is arranged non-parallel to the principal axis of the other camera sensor. In other words, the camera sensors may be arranged so that the principal axes of the camera sensors diverge outward with respect to the line of sight axis. In other words, the distance between the principal axes of the camera sensors may increase as you move away from the camera sensors. In Figure 1, when the detection data processing device 100 is embodied in an HMD (Head Mounted Display) and worn by a user 190, the principal axes of the camera sensors form an angle θ with respect to the line of sight axis of the user 190. The divergent stereo camera sensor has a narrower overlapping region 153 than a parallel stereo camera sensor. On the other hand, the divergent stereo camera sensor has a wider non-overlapping region 151,152 than a parallel stereo camera, and can therefore provide a relatively wide field of view. Therefore, the detection data processing device 100 using a divergent stereo camera sensor can utilize more video information acquired based on a wide field of view for tracking the position and / or attitude of the device.

[0032] For reference, the full observable range 150 of the detection data processing device 100 may be determined based on a combination of field of view angles of the sensors (e.g., a reference sensor 110 and other sensors 120). The combination of field of view angles represents a combination of overlapping and non-overlapping field of view angles between the sensors. As the overlapping area between the sensor field of view angles decreases and the non-overlapping area increases, the field of view angle of the sensor combination increases, and therefore the full observable range 150 also increases. Conversely, as the overlapping area between the sensor field of view angles increases and the non-overlapping area decreases, the field of view angle of the sensor combination decreases, and therefore the full observable range 150 decreases. In Figure 1, the reference sensor 110 is shown as the left camera sensor relative to the user, and the other sensors 120 are shown as the right camera sensors, but this is not limited to the above. Depending on the design, the reference sensor 110 may be the right camera sensor, the camera sensors may be arranged along an axis other than one horizontal axis to the ground (e.g., an axis perpendicular to the ground), and the number of camera sensors may be n. Here, n may be an integer greater than or equal to 2.

[0033] The following describes a detection data processing device 100 that performs effective positioning not only for parallel stereo camera sensors but also for divergent stereo camera sensors.

[0034] Figure 2 shows an example of region distinction in detection data according to one embodiment.

[0035] In Figure 1, if the aforementioned pair of sensors are divergent stereo camera sensors, the detection data processing device can generate images individually through each camera sensor. For example, the detection data processing device can generate a first image 210 through the first camera sensor and a second image 220 through the second camera sensor.

[0036] In one embodiment, the detection data processing device 100 may divide the detection data collected from multiple sensors into overlapping regions that overlap with the field of view of other sensors and non-overlapping regions that do not overlap with the field of view of other sensors. For example, the detection data processing device can acquire a first image 210 (e.g., left image) via the first camera sensor (e.g., camera sensor 110) of the stereo camera sensors 110 and 120, and a second image 220 (e.g., right image) via the second camera sensor (e.g., camera sensor 120). The detection data processing device 100 can determine the region in the first image 210 where the field of view of the first camera sensor (e.g., camera sensor 110) and the field of view of the second camera sensor (e.g., camera sensor 120) overlap as the first overlapping region 212 (e.g., SL (stereo left)). The detection data processing device 100 can determine in the first video 210 the remaining area of ​​the field of view of the first camera sensor that does not overlap with the field of view of the second camera sensor as the first non-overlapping area 211 (e.g., LO (left only)). Similarly, in the second video 220, the detection data processing device 100 can determine in the second video 220 the area where the field of view of the second camera sensor and the field of view of the first camera sensor overlap as the second overlapping area 222 (e.g., SR (stereo right)). In the second video 220, the detection data processing device 100 can determine in the second video 220 the remaining area of ​​the field of view of the second camera sensor (e.g., camera sensor 120) that does not overlap with the field of view of the first camera sensor (e.g., camera sensor 110) as the second non-overlapping area 221 (e.g., RO (right only)). The first overlapping area 212 and the second overlapping area 222 may contain the same object and / or background in the scene.

[0037] As described above, the detection data processing device 100 can track and manage keypoints based on regions obtained by dividing the field of view of each sensor. The detection data processing device 100 maintains records of keypoints belonging to overlapping regions and keypoints belonging to non-overlapping regions for each sensor, and can estimate the position and / or orientation of the device using keypoints within the overall observation range. The operation of the detection data processing device will be described later.

[0038] Figure 3 is a flowchart illustrating a method for processing detection data according to one embodiment. The operations shown in Figure 3 are executed based on the time-series sequence shown, but are not limited thereto; some operations may be omitted or modified. The operations shown in Figure 3 may be executed in parallel or simultaneously. One or more blocks and combinations of blocks shown in Figure 3 can be implemented using a special-purpose hardware-based computer that performs specialized functions, or a combination of special-purpose hardware and computer instructions. In addition to the later explanation shown in Figure 3, the explanations shown in Figures 1 and 2 are also applicable to Figure 3.

[0039] For example, a method for estimating the position and / or orientation of a device (e.g., a detection data processing device 100) using a camera sensor may include a direct tracking method and an indirect tracking method. The direct tracking method refers to a method that estimates the position and / or orientation using all the pixel information of the video information. The indirect tracking method refers to a method that detects keypoints in the video and estimates the position and / or orientation of the device using the detected keypoints.

[0040] A detection data processing device 100 according to one embodiment can detect key points in video from detection data (e.g., video data) collected via a sensor. The detection data processing device 100 performs indirect positioning and / or attitude estimation using the detected key points. In the indirect tracking method, the position and / or attitude estimation performance of the equipment can be improved by managing key points. For reference, in this specification, key points are used as reference points for positioning in the detection data to estimate changes in the position and / or attitude of the detection data processing device 100 and the sensor, and may exemplary be corner points. Corner points are points in video data where two or more edge components intersect or vertices exist. For example, when a check pattern board is captured via a camera sensor, each vertex may be extracted as a key point. Key points may also be referred to as landmarks and / or features.

[0041] First, in step S310, the detection data processing device 100 tracks keypoints. For example, based on detection data collected in previous and current frames via a reference sensor, the detection data processing device 100 can track multiple keypoints within the overall observation range determined based on a combination of the field of view angles of the reference sensor and other sensors. As described above, the detection data processing device 100 can acquire detection data corresponding to frames (e.g., time frames) from each sensor. The detection data processing device 100 can detect multiple keypoints from the detection data of each frame. The detection data processing device 100 can track the positional movement and region changes of multiple keypoints relative to a frame. In this specification, examples of detecting and tracking multiple keypoints for each frame of consecutive frames are mainly described, but are not limited thereto. Keypoint tracking is described with reference to Figure 4 below.

[0042] Then, in step S320, the detection data processing device 100 refines the tracked keypoints. For example, the detection data processing device 100 may remove outliers from the tracked keypoints. The removal of outliers will be explained with reference to Figure 7 below.

[0043] Next, in step S330, the detection data processing device 100 manages new keypoints together with multiple keypoints (e.g., tracked or existing keypoints). The detection data processing device 100 can manage new keypoints together with multiple keypoints (e.g., tracked or existing keypoints) in response to detecting new keypoints. The detection data processing device 100 may further detect new keypoints in a new frame (e.g., the current frame). For example, the detection data processing device 100 may further detect new keypoints based on the ORB (Oriented FAST and Rotated BRIEF) algorithm and SIFT (Scale-Invariant Feature Transform). For convenience of explanation, the operation of adding new keypoints in step S330 has been described as being performed later, but is not limited to this. Depending on the design, the operation of adding new keypoints may be performed before step S310.

[0044] According to one embodiment, the processor of the detection data processing device 100 can detect new keypoints in the detection data collected via the reference sensor in overlapping regions that overlap with the field of view of other sensors. Since keypoints can be detected in the overlapping regions of the reference sensor and other sensors, the detection data processing device 100 can detect new keypoints and calculate the three-dimensional coordinates of the new keypoints. The three-dimensional coordinates of the keypoints may be in the world coordinate system and / or the camera coordinate system. The origin point of the world coordinate system may, for example, be set as a single point in the space where the initial positioning was started. As described above, the detection data processing device 100 can store and manage the three-dimensional coordinate values ​​corresponding to multiple keypoints separately from the two-dimensional coordinates of the keypoints (e.g., coordinates on the image plane). The detection data processing device 100 can generate and manage a three-dimensional map based on the three-dimensional coordinate values.

[0045] According to one embodiment, at the initial stage of operation of the detection data processing device 100, new keypoints are detected and managed only in the overlapping region, and the keypoints may be distributed in the remaining non-overlapping region by the movement and / or rotation of the device. After the keypoints are uniformly distributed in the overlapping and non-overlapping regions, the detection data processing device 100 can stably provide position and / or attitude estimation using the keypoints.

[0046] As described above, the detection data processing device 100 can have improved tracking performance for key points by applying key points in both overlapping and non-overlapping regions to positioning. For example, although a divergent stereo camera sensor has a relatively narrow overlapping region, the detection data processing device 100 according to one embodiment can provide stable positioning results by tracking key points in both overlapping and non-overlapping regions and applying them to positioning. Furthermore, even if the angular distance between the principal axes of the sensors changes dynamically, the detection data processing device 100 can perform stable positioning by using all key points in both overlapping and non-overlapping regions as described above. Camera calibration for matching key points acquired by each camera sensor may be performed in response to changes in the angular distance between the principal axes of the sensors, or the detection data processing device 100 may load camera calibration parameters corresponding to the changed angle from a pre-built database.

[0047] For reference, this specification primarily describes detection through two sensors (e.g., a pair of camera sensors), but is not limited thereto. The detection data processing device 100 collects detection data frame by frame through multiple sensors, and each of the multiple sensors may have a field of view that overlaps with at least the other sensors.

[0048] Figure 4 is a flowchart illustrating the tracking of keypoints according to one embodiment. The operations shown in Figure 4 are performed in a chronological manner, but are not limited thereto; some operations may be omitted or modified. The operations shown in Figure 4 may be performed in parallel or simultaneously. One or more blocks and combinations of blocks shown in Figure 4 can be implemented by a special-purpose hardware-based computer that performs specialized functions, or by a combination of special-purpose hardware and computer instructions. In addition to the explanation described later with reference to Figure 4, the explanations shown in Figures 1 to 3 are also applicable to Figure 4.

[0049] First, in step S411, the detection data processing device 100 determines the position to which the keypoint has moved. The detection data processing device 100 can track the movement of the keypoint in frames. According to one embodiment, the detection data processing device 100 can track the movement of each keypoint from its position in a previous frame to its position in the current frame. The detection data processing device 100 can determine the position of the keypoint in the current frame that was detected in a previous frame. In this specification, the previous frame is mainly described as the frame immediately preceding the current frame, but is not limited thereto. The detection data processing device 100 can track the keypoint until it moves out of the overall observation range described above.

[0050] The detection data processing device 100 can track the movement of keypoints from two frames using an optical flow tracking method based on the optical flow between images (for example, the KLT (Kanade-Lucas-Tomasi) tracking method). The detection data processing device 100 compares keypoints detected in the previous frame with keypoints detected in the current frame. The detection data processing device 100 may also search for corresponding pairs of keypoints detected in the previous frame and keypoints detected in the current frame. Here, the detection data processing device 100 can obtain an initial estimation of keypoint position changes using inertial information measured via an inertial measurement unit (IMU), and can compensate for the initial estimation using the aforementioned KLT.

[0051] According to one embodiment, the detection data processing device 100 can determine the position of keypoints based on the detection of the reference sensor for regions of the reference sensor's field of view that do not overlap with the field of view of other sensors (non-overlap). The detection data processing device 100 can estimate the position of keypoints within the non-overlap region of the reference sensor from a previous frame to the current frame using the detection data of the reference sensor from the previous frame and the detection data of the current frame. The detection data processing device 100 can determine the position of keypoints based on the detection of other sensors for regions of the other sensors' field of view that do not overlap with the field of view of the reference sensor. Similarly, the detection data processing device 100 can estimate the position of keypoints within the non-overlap region of other sensors from a previous frame to the current frame using the detection data of the other sensors from the previous frame and the detection data of the current frame. For overlapping regions between sensors, the detection data processing device 100 can estimate the movement of keypoints using detection data collected from the reference sensor and at least one of the other sensors.

[0052] The detection data processing device 100 can manage keypoint information for each individual keypoint. The keypoint information may include the keypoint identifier (ID), its position in the video (e.g., coordinates on a 2D video plane), and a confidence score (e.g., a score indicating the accuracy of the extracted keypoint).

[0053] In step S412, the detection data processing device 100 updates the region to which the key point belongs in accordance with the movement of the key point's position. The detection data processing device 100 can classify the region to which the key point belongs into one of the following regions: the overlapping region of the reference sensor, the non-overlapping region of the reference sensor, the overlapping region of other sensors, and the non-overlapping region of other sensors. The updating of the region to which the key point belongs will be explained with reference to Figure 5 below.

[0054] For reference, the detection data processing device 100 can track and detect keypoints across the entire observation range, but is not limited to this. The detection data processing device 100 may track keypoints for each region of individual video, or it may exclude keypoint tracking for some regions. For example, the detection data processing device 100 can determine the moved positions of keypoints in at least some regions of a plurality of keypoints for at least some regions of the overall observation range. In other words, the processor of the detection data processing device 100 can determine the moved positions of keypoints based on the detection of the reference sensor for regions of the reference sensor's field of view that overlap with the field of view of other sensors. The detection data processing device 100 may skip tracking keypoints based on the detection of other sensors for regions of other sensors' field of view that overlap with the field of view of the reference sensor (for example, the SR region 222 shown in Figure 2). Keypoints indicated in the overlapping regions of other sensors correspond to keypoints indicated in the overlapping regions of the reference sensor. Therefore, even if tracking of keypoints within the overlapping regions of other sensors is omitted, the changed positions of keypoints within the overlapping regions of other sensors can be estimated through the tracking results (e.g., positional movement) of keypoints within the overlapping regions of the reference sensor.

[0055] For example, the detection data processing device 100 can calculate the position of the other sensor in the image plane corresponding to the point based on the detection of the reference sensor, based on camera parameters (e.g., internal parameters) between the reference sensor and the other sensor. The modified position of keypoints in the overlapping region of the other sensor may be performed in a refinement operation described later with reference to Figure 7. In other words, for the overlapping region of the other sensor, the tracking of keypoints as described with reference to Figure 4 may be omitted, and the modified position of keypoints in the overlapping region of the other sensor may be estimated by tracking based on the detection of the reference sensor in a keypoint refinement operation described later with reference to Figure 7. However, this is merely an example, and the order in which the determination of the modified position of keypoints in the overlapping region of the other sensor is performed is not limited as described above, and may be performed in parallel with other operations and / or chronologically.

[0056] Figure 5 shows the determination of a region by moving the position of a keypoint according to one embodiment.

[0057] According to one embodiment, the processor of the detection data processing device 100 can update the region to which each key point belongs within the overall observation range based on the moved positions of the multiple key points. In Figure 5, the positions of the key points may be positions in the image coordinate system of the first image 510 and / or positions in the image coordinate system of the second image 520.

[0058] For example, the detection data processing device 100 may exclude from tracking keypoints whose moved position falls outside the overall observation range. The detection data processing device 100 may also remove keypoints from keypoint management that fall outside the observable range (e.g., the overall observation range) among the tracked keypoints. Exemplarily, Figure 5 shows the movement of keypoints by frame for each region of the video. In Figure 5, keypoint 503 in the overlapping region 512 of the first video 510 and keypoint 501 in the non-overlapping region 511 have moved outside the overall observation range due to frame movement. Keypoint 506 in the non-overlapping region 521 of the second video 520 has also moved outside the overall observation range due to frame movement. Therefore, the detection data processing device 100 can remove keypoints 501, 503, and 506 that exceed the observable range corresponding to the first video 510 and the second video 520. The modified position of keypoint 503 can be transformed from its coordinates in the first image 510 to its coordinates in the second image 520 using the corresponding three-dimensional coordinates 503a and / or camera parameters.

[0059] For reference, keypoint 507 that has moved from the non-overlapping region 521 to the overlapping region 522 of the second image 520 may be removed for management efficiency. Alternatively, the detection data processing device 100 can generate a keypoint corresponding to the removed keypoint 507 in the overlapping region 512 of the first image 510. As described above with reference to Figure 4, since keypoints in the overlapping regions 512,522 can be converted to positions corresponding to other sensors via camera parameters, the detection data processing device 100 can hold keypoints for only one of the two sensors.

[0060] According to one embodiment, if a keypoint moves away from the boundary of the region to which it previously belonged in a frame and enters another region within the overall observation range in the current frame, the processor of the detection data processing device 100 can update the region to which a keypoint belongs to the other region. For example, the detection data processing device 100 can classify a detected keypoint as belonging to one of the regions (e.g., the LO region, SL region, SR region, and RO region shown in Figure 2). Exemplarily, an LO keypoint indicates a keypoint located in the first non-overlapping region of the first image acquired via the first camera sensor (e.g., the left camera sensor). An SL keypoint indicates a keypoint located in the first overlapping region of the first image. An SR keypoint indicates a keypoint located in the second overlapping region of the second image acquired via the second camera sensor (e.g., the right camera sensor). An RO keypoint indicates a keypoint located in the second non-overlapping region of the second image. For reference, this specification mainly describes examples where the first image is a reference image and the second image is another image, but is not limited thereto.

[0061] For example, the detection data processing device 100 may determine that a keypoint 502 that has moved from a non-overlapping region 511 to an overlapping region 512 of the first video 510 belongs to the overlapping region 512. The detection data processing device 100 determines that the changed position of a keypoint 505 that belongs to the overlapping region 512 of the first video 510 belongs to a non-overlapping region 521 of the second video 520. The detection data processing device 100 can calculate the corresponding coordinates in the second video 520 from the world coordinates 505a corresponding to the keypoint 505, or calculate the corresponding coordinates in the second video 520 using camera parameters. The detection data processing device 100 can update the region of a keypoint 504 that has moved from an overlapping region 512 to a non-overlapping region 511 of the first video 510. The keypoint position classification described with reference to Figure 5 may be efficiently performed via a mask described later with reference to Figure 6.

[0062] Figure 6 shows a mask for determining the region of other keypoints in one embodiment.

[0063] In one embodiment of the detection data processing device, the processor can classify multiple keypoints into one of the non-overlapping and overlapping regions based on a non-overlapping mask indicating non-overlapping regions and an overlapping mask indicating overlapping regions. The non-overlapping mask may have an activation value (e.g., 1) for regions that do not overlap with other sensors within the field of view of the sensor (e.g., reference sensor) and an inactivation value (e.g., 0) for the remaining regions. The overlapping mask may have an activation value (e.g., 1) for regions that overlap with other sensors within the field of view of the sensor and an inactivation value (e.g., 0) for the remaining regions. In Figure 6, activation values ​​are shown brightly and inactivation values ​​are shown darkly. The mask 610 for the first sensor (e.g., reference sensor) may include an overlapping mask 612 and a non-overlapping mask 611. The mask 620 for the second sensor (e.g., other sensors) may also include an overlapping mask 622 and a non-overlapping mask 621. The detection data processing device can apply the mask set for each sensor to the keypoints that have been moved. Each mask may be predefined based on calibration information between sensors (e.g., camera parameters), field of view, distance, and distortion.

[0064] The detection data processing device can individually apply pixel-wise multiplication to the coordinates of one or more keypoints whose positional movement has been tracked for each sensor, based on the aforementioned masks 611, 612, 621, and 622. A keypoint multiplied by the activation value of a mask indicating a region may be determined to belong to that region. By utilizing the predefined masks described above to determine and transform the region to which a keypoint belongs, the detection data processing device can update the region changes of keypoints with a small amount of computation.

[0065] Figure 7 is a flowchart illustrating keypoint refinement according to one embodiment. The operations shown in Figure 7 are performed in the order shown chronologically, but are not limited to this, and some operations may be omitted or modified. The operations shown in Figure 7 may be performed in parallel or simultaneously. One or more blocks and combinations of blocks shown in Figure 7 can be realized by a special-purpose hardware-based computer that performs specialized functions, or by a combination of special-purpose hardware and computer instructions. In addition to the later explanation shown in Figure 7, the explanations in Figures 1 to 6 are also applicable to Figure 7.

[0066] A detection data processing device according to one embodiment can remove outliers from among a plurality of keypoints based on detection data collected in previous and current frames via a reference sensor. Outliers are keypoints among a plurality of keypoints that have positions that were incorrectly calculated due to sensor errors and / or calculation errors. Since outliers degrade performance in estimating the position and / or attitude of the equipment, they can be removed as described below.

[0067] For example, in step S721, the detection data processing device compares the overall detection data of each sensor frame by frame. For example, the detection data processing device can compare the detection data frame by frame using algorithms such as RANSAC (RANDOM SAmple Consensus) and stereo matching. The detection data processing device can integrate detection data collected from any sensor without distinguishing regions and compare them frame by frame. Exemplarily, the detection data processing device compares the keypoints shown in the detection data of the first sensor in a previous frame with the keypoints shown in the detection data of the current frame. The detection data processing device can compare the keypoints shown in the detection data of the second sensor in a previous frame with the keypoints shown in the detection data of the current frame. In other words, the detection data processing device integrates the overlapping and non-overlapping regions of the first sensor and the overlapping and non-overlapping regions of the second sensor frame by frame to perform keypoint comparison. For reference, in an example where tracking of keypoints within the overlapping region of the second sensor is skipped, the detection data processing device can convert the keypoints within the overlapping region of the first sensor to the coordinates of the second sensor and copy the result, which can then be used for comparison to remove outliers. Exemplarily, in the RANSAC algorithm, the outlier detection performance increases as the number of samples increases, but as mentioned above, sufficient keypoints for comparison can be secured by integrating the overlapping and non-overlapping regions.

[0068] However, as mentioned above, the example is not limited to integrating overlapping and non-overlapping regions to compare keypoints for each frame. For example, while the likelihood of outliers increases with increasing motion, it may also be interpreted that outliers are more likely to occur in the non-overlapping regions of each sensor. The detection data processing device can also perform calculations efficiently by comparing keypoints for each frame in the non-overlapping regions of each sensor.

[0069] Then, in step S722, the detection data processing device determines and removes outliers based on the detections of each sensor. The detection data processing device can remove keypoints determined by outliers via algorithms such as RANSAC and stereo matching, as described above.

[0070] Figure 8 is a flowchart illustrating an example of the use of key points according to one embodiment.

[0071] In step S840, the detection data processing device performs attitude estimation. In one embodiment, the detection data processing device can estimate at least one of the position and attitude of the detection data processing device as a positioning result based on the results of purifying the tracked keypoints. In other words, the detection data processing device can estimate the position and attitude of the device using the remaining keypoints from a plurality of keypoints, excluding the outliers. The detection data processing device may perform odometry based on the purified keypoints. The detection data processing device can output the estimated positioning result to the detection data processing device.

[0072] For example, the detection data processing device may estimate the position and / or orientation of the device based on the inertial data 841 in addition to the aforementioned keypoints.

[0073] As described above, the detection data processing device according to one embodiment performs positioning by considering both the key points in the overlapping and non-overlapping regions of each sensor, and therefore can provide stable tracking performance even with rapid rotation (for example, the rapid head rotation of a user wearing augmented reality glasses).

[0074] Figure 9 is a block diagram showing a detection data processing device according to one embodiment.

[0075] A detection data processing device 900 according to one embodiment includes a sensor 910, one or more processors 920, and one or more memories 930. The detection data processing device 900 may further include an output device 940.

[0076] Sensor 910 can generate detection data by capturing a scene corresponding to the field of view. For example, sensor 910 may include one or more combinations of camera sensors, infrared sensors, radar sensors, lidar sensors, and ultrasonic sensors. Detection data processing device 900 may include a pair of sensors. For example, if sensor 910 is a camera sensor, the camera sensor can generate video data by capturing a scene within the field of view. As a different example, if sensor 910 is an infrared sensor, the infrared sensor can generate infrared data by capturing a scene within the field of view.

[0077] One or more processors 920 can track multiple keypoints within a full observable range determined based on a combination of the field of view angles of the reference sensor and other sensors, based on detection data collected in previous and current frames via a reference sensor. In response to the detection of new keypoints, one or more processors 920 can manage new keypoints along with the existing keypoints. However, the operation of one or more processors 920 is not limited to this, and the operations described above may be performed in parallel and / or chronologically, as shown in Figures 1 to 8.

[0078] One or more memories 930 store information required for keypoint management and positioning using keypoints. For example, one or more memories 930 can store data indicating the location of a keypoint, the region to which the keypoint belongs, and the estimated position and / or attitude to the detection data processing device 900.

[0079] The output device 940 outputs the positioning results described above (e.g., the estimated position and / or orientation relative to the detection data processing device 900). The output device 940 can output the positioning results visually, tactilely, and / or audibly. For example, if the detection data processing device 900 is implemented as an augmented reality device 900, the output device 940 may be an augmented reality display, and the positioning results described above can be output visually via the display. In a different example, the detection data processing device 900 may output virtual content to a position determined based on the positioning results (e.g., a position overlaid on a targeted real object).

[0080] According to one embodiment, one or more processors 920 can control other operations of the detection data processing device 900 based on the positioning results described above. For example, if the detection data processing device 900 is implemented as a vehicle, robot, and / or drone, one or more processors 920 may change one or more combinations of steering, speed, and acceleration of the device 900 based on the positioning results.

[0081] According to one embodiment, the detection data processing device 900 may be implemented as a device 900 including a head-mounted display (HMD) in an augmented reality (AR) environment, and the position and / or orientation of the device 900 can be estimated. However, without limitation, the detection data processing device 900 may be implemented in a robot or a drone. The detection data processing device 900 may be applied to the aforementioned augmented reality device 900 (e.g., AR glasses), a robot, a drone, and a chip, software, or a web service implemented thereon.

[0082] The embodiments described above are embodied in hardware components, software components, or combinations of hardware and software components. The detection data processing device 100, the detection data processing device 900, the sensor 910, the processor 920, the memory 930, the output device 940, and the remaining devices, units, modules, equipment, and other components can be embodied in the hardware and software components described herein. For example, the devices and components described in this embodiment are embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or different devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the OS. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For the sake of understanding, a single processing unit may sometimes be described as being used; however, those with ordinary skill in the art will understand that a processing unit may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0083] Software includes computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or instruct the processing unit independently or in combination. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, for interpretation by the processing unit or for providing instructions or data to the processing unit. Software can be distributed across a network of computer systems and stored and executed in a distributed manner. Software and data can be stored on a recording medium readable by one or more computers.

[0084] The method according to this embodiment is embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executed by a computer using an interpreter or the like.

[0085] The hardware device described above may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0086] As described above, although embodiments have been illustrated with limited drawings, a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or assembled in a different manner than described, or replaced or substituted with other components or equivalents, and still achieve suitable results.

[0087] Therefore, other embodiments, other embodiments, and claims equivalent to those described below also fall within the scope of the claims.

Claims

1. A detection data processing device, A reference sensor that generates detection data by capturing the scene within the field of view, The detection data processing device includes a processor that tracks multiple keypoints within its overall observation range based on detection data collected in previous frames via the reference sensor and detection data collected in the current frame, and manages the new keypoint together with the tracked multiple keypoints in response to the detection of a new keypoint. The overall observation range is determined based on the combination of the field of view angles of the reference sensor and the second sensor. The aforementioned processor, For the region of the field of view of the reference sensor that overlaps with the field of view of the second sensor, the moved position of the key point among the tracked key points is determined based on the detection of the reference sensor. For the region of the second sensor's field of view that overlaps with the field of view of the reference sensor, the tracking of keypoints among the tracked keypoints based on the detection of the second sensor is skipped. Detection data processing device.

2. The detection data processing device according to claim 1, wherein the processor determines the moved position of the key points in at least a portion of the plurality of key points with respect to at least a portion of the overall observation range.

3. The aforementioned processor, For the region of the field of view of the reference sensor that does not overlap with the field of view of the second sensor, the moved position of the key point among the tracked key points is determined based on the detection of the reference sensor. For the region of the second sensor's field of view that does not overlap with the field of view of the reference sensor, the moved position of the keypoint among the tracked keypoints is determined based on the detection by the second sensor. The detection data processing device according to claim 1.

4. The detection data processing device according to claim 1, wherein the processor updates the region to which each key point belongs within the overall observation range based on the moved position of the plurality of tracked key points.

5. The detection data processing device according to claim 1, wherein the processor excludes from tracking any keypoints whose moved position among the tracked keypoints falls outside the overall observation range.

6. The detection data processing device according to claim 1, wherein the processor classifies the traced plurality of keypoints into one of the non-overlapping regions and the overlapping regions based on a non-overlapping mask indicating non-overlapping regions and an overlapping mask indicating overlapping regions.

7. The detection data processing device according to claim 1, wherein the processor removes outliers from among the tracked keypoints based on detection data collected in previous frames and detection data collected in the current frame via the reference sensor.

8. The detection data processing device according to claim 1, wherein the processor detects a new key point from an overlapping region that overlaps with the field of view of the second sensor using detection data collected via the reference sensor.

9. The system further includes an output device that outputs the estimated positioning result to the detection data processing device, Based on the refined results of the tracked keypoints, the processor estimates at least one of the position and orientation of the detection data processing device as the positioning result. The detection data processing device according to claim 1.

10. A detection data processing method implemented by a processor, The process involves capturing the scene within the field of view using a reference sensor, and then tracking multiple keypoints within the overall observation range of the detection data processing device based on detection data collected in previous frames and detection data collected in the current frame. The steps include: purifying the aforementioned traced key points, The step of managing the new keypoint together with the tracked keypoints in response to the detection of a new keypoint, The overall observation range is determined based on the combination of the field of view angles of the reference sensor and the second sensor. The aforementioned tracking step is, For the region of the field of view of the reference sensor that overlaps with the field of view of the second sensor, the steps include determining the moved position of the key point among the multiple tracked key points based on the detection of the reference sensor, The step includes skipping the tracking of keypoints among the plurality of keypoints tracked based on the detection of the second sensor for regions of the second sensor's field of view that overlap with the field of view of the reference sensor, Method for processing detected data.

11. The detection data processing method according to claim 10, wherein the tracking step includes determining the moved position of the key points in at least a portion of the tracked key points in at least a portion of the overall observation range.

12. The aforementioned tracking step is, For the region of the field of view of the reference sensor that does not overlap with the field of view of the second sensor, the step of determining the moved position of the key point among the multiple tracked key points based on the detection of the reference sensor, For regions of the second sensor's field of view that do not overlap with the reference sensor's field of view, the step of determining the moved position of a key point among the multiple tracked key points based on the detection by the second sensor is included. The detection data processing method according to claim 10.

13. The detection data processing method according to claim 10, wherein the tracking step includes updating the region to which each key point belongs within the overall observation range based on the moved position of the tracked plurality of key points.

14. The detection data processing method according to claim 10, wherein the tracking step includes excluding from tracking any keypoints whose moved position among the tracked keypoints falls outside the overall observation range.

15. The detection data processing method according to claim 10, wherein the tracking step includes classifying the tracked keypoints into one of the non-overlapping regions and the overlapping regions based on a non-overlapping mask indicating non-overlapping regions and an overlapping mask indicating overlapping regions.

16. The detection data processing method according to claim 10, wherein the purification step includes removing outliers from among the tracked keypoints based on detection data collected in previous frames and detection data collected in the current frame via the reference sensor.

17. The detection data processing method according to claim 10, wherein the step of managing the new keypoints is to detect new keypoints from an overlapping region that overlaps with the field of view of the second sensor using detection data collected via the reference sensor.

18. A computer-readable recording medium storing a computer program that includes a computer program for executing the detection data processing method described in any one of claims 10 to 17.

19. A detection data processing device, A first sensor that captures the first image at the first field of view, The second image is captured at the second field of view, and the second sensor differs from the first sensor. The system includes one or more processors that determine non-overlapping and overlapping regions between the first and second video, detect keypoints from the non-overlapping and overlapping regions, track changes in the position of the keypoints in the non-overlapping and overlapping regions for each consecutive frame of the first and second video, refine the tracked keypoints by removing outliers from the tracked keypoints, and manage the new keypoints together with the tracked keypoints in response to the detection of new keypoints in the current frame. The aforementioned processor, For the overlapping region of the first image that overlaps with the second image, the moved position of the key point among the tracked key points is determined based on the detection by the first sensor. For overlapping regions of the second video that overlap with the second video, the tracking of keypoints among the tracked keypoints based on detection by the second sensor is skipped. Detection data processing device.

20. The detection data processing device according to claim 19, wherein the new key point is detected from the overlapping region based on detection data collected by the first sensor.

21. The aforementioned processor, Track keypoints in the non-overlapping region and keypoints in the overlapping region. Apply the tracked keypoints to positioning. The detection data processing device according to claim 19.

Images