Enabling automatic measurement

Abstract

The present disclosure relates to enabling automatic measurements. According to some embodiments, a technique is described that enables an electronic device with a camera to automatically gather and generate the necessary data from a real-world environment to allow the electronic device to quickly and efficiently determine and provide accurate measurements of physical spaces and / or objects within the real-world environment.

Description

[0001] Related application citation

[0002] This application is a divisional application of Chinese national application number 202010125848.9, filed on February 27, 2020, entitled "Enabling Automatic Measurement". Technical Field

[0003] This disclosure relates in general to providing measurements, and more specifically to methods and techniques for enabling automated measurements using electronic devices. Background Technology

[0004] Electronic devices equipped with cameras can be used to provide measurements of real-world distances. However, conventional measurement techniques require users to follow specific steps so that the device can acquire and process the necessary data from the real-world environment to determine and provide accurate measurements. Therefore, conventional measurement techniques can be improved to increase efficiency, allowing devices to conserve more energy, which is especially important for battery-powered devices. Summary of the Invention

[0005] Therefore, there is a need for a more efficient and improved technique for providing measurements using electronic devices. The technique presented in this paper enables electronic devices with cameras, such as smartphones or wearable devices (e.g., smartwatches or smart glasses), to automatically collect and generate the necessary data from the real-world environment, allowing the device to quickly and efficiently determine and provide accurate measurements of physical spaces and / or objects within the real-world environment.

[0006] According to some embodiments, a method performed at an electronic device having a camera is described. The method includes: capturing a plurality of images along a movement path of the electronic device using the camera before detecting activation of an application for measuring distances between real-world points on the electronic device; determining keyframes based on one or more features of the plurality of images; storing the keyframes; and in response to detecting activation of the application for measuring the distances: determining the distances between the real-world points using one or more of the stored keyframes; and displaying a representation of the distances in the application's user interface.

[0007] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera, the one or more programs including instructions for performing the following operations: capturing a plurality of images along a movement path of the electronic device via the camera before detecting activation of an application for measuring distances between real-world points on the electronic device; determining keyframes based on one or more features of the plurality of images; storing the keyframes; and in response to detecting activation of the application for measuring the distances: determining the distances between the real-world points using one or more of the stored keyframes; and displaying a representation of the distances in the user interface of the application.

[0008] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera, the one or more programs including instructions for performing the following operations: capturing a plurality of images along a movement path of the electronic device via the camera before detecting activation of an application for measuring distances between real-world points on the electronic device; determining keyframes based on one or more features of the plurality of images; storing the keyframes; and in response to detecting activation of the application for measuring the distances: determining the distances between the real-world points using one or more of the stored keyframes; and displaying a representation of the distances in the user interface of the application.

[0009] According to some embodiments, an electronic device is described. The electronic device includes: a camera; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: capturing a plurality of images along a movement path of the electronic device via the camera before detecting activation of an application for measuring distances between real-world points on the electronic device; determining keyframes based on one or more features of the plurality of images; storing the keyframes; and in response to detecting activation of the application for measuring the distances: determining the distances between real-world points using one or more of the stored keyframes; and displaying a representation of the distances in the user interface of the application.

[0010] According to some embodiments, an electronic device is described. The electronic device includes: a camera; means for capturing a plurality of images via the camera along a movement path of the electronic device before detecting activation of an application for measuring distances between real-world points on the electronic device; means for determining keyframes based on one or more features of the plurality of images; means for storing the keyframes; and means for performing the following operations in response to detecting activation of the application for measuring distances: determining the distances between real-world points using one or more of the stored keyframes; and displaying a representation of the distances in the user interface of the application.

[0011] According to some embodiments, a method performed at an electronic device having a camera is described. The method includes: capturing a plurality of images by the camera along a movement path of the electronic device from a first position to a second position, wherein the first position corresponds to the position of the electronic device when the application is activated on the electronic device, prior to detecting a request to measure the distance between real-world points using an application on the electronic device; determining keyframes based on one or more features of the plurality of images; storing the keyframes; and in response to detecting a request to measure distance: determining the distance between real-world points using one or more of the stored keyframes; and displaying a representation of the distance in the user interface of the application.

[0012] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera, the one or more programs including instructions for performing the following operations: capturing a plurality of images by the camera along a movement path of the electronic device from a first position to a second position, wherein the first position corresponds to the position of the electronic device when the application is activated on the electronic device, prior to detecting a request to measure the distance between real-world points using an application on the electronic device; determining keyframes based on one or more features of the plurality of images; storing the keyframes; and in response to detecting a request to measure distance: determining the distance between real-world points using one or more of the stored keyframes; and displaying a representation of the distance in the user interface of the application.

[0013] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera, the one or more programs including instructions for performing the following operations: capturing a plurality of images by the camera along a movement path of the electronic device from a first position to a second position, wherein the first position corresponds to the position of the electronic device when the application is activated on the electronic device, prior to detecting a request to measure the distance between real-world points using an application on the electronic device; determining keyframes based on one or more features of the plurality of images; storing the keyframes; and in response to detecting a request to measure distance: determining the distance between real-world points using one or more of the stored keyframes; and displaying a representation of the distance in the user interface of the application.

[0014] According to some embodiments, an electronic device is described. The electronic device includes: a camera; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: capturing a plurality of images by the camera along a movement path of the electronic device from a first position to a second position, wherein the first position corresponds to the position of the electronic device when the application is activated on the electronic device, prior to detecting a request to measure the distance between real-world points using an application on the electronic device; determining keyframes based on one or more features of the plurality of images; storing the keyframes; and in response to detecting a request to measure distance: determining the distance between real-world points using one or more of the stored keyframes; and displaying a representation of the distance in the user interface of the application.

[0015] According to some embodiments, an electronic device is described. The electronic device includes: a camera; means for capturing a plurality of images via the camera along a movement path of the electronic device from a first position to a second position before detecting a request to measure the distance between real-world points using an application on the electronic device, wherein the first position corresponds to the position of the electronic device when the application is activated on the electronic device; means for determining keyframes based on one or more features of the plurality of images; means for storing the keyframes; and means for performing the following operations in response to detecting a request to measure distance: determining the distance between real-world points using one or more of the stored keyframes; and displaying a representation of the distance in the user interface of the application. Attached Figure Description

[0016] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, in which similar reference numerals indicate corresponding parts throughout the drawings.

[0017] Figures 1A to 1G and Figures 2A to 2I Exemplary techniques for enabling electronic devices to provide measurements, according to some implementation schemes, are shown.

[0018] Figures 3A to 3F Another exemplary technique for enabling electronic devices to provide measurements, according to some implementation schemes, is shown.

[0019] Figure 4 A flowchart illustrating a method for enabling an electronic device to provide measurements, according to some embodiments.

[0020] Figure 5 This is a flowchart illustrating another method for enabling electronic devices to provide measurements, according to some implementation schemes. Detailed Implementation

[0021] This article describes various techniques that enable electronic devices with cameras (such as smartphones or wearable devices) to automatically provide measurements of real-world environments (specifically, measurements of physical space and objects therein).

[0022] Figures 1A to 1G Exemplary techniques for enabling electronic device 100 to provide measurements, according to some embodiments, are illustrated. Electronic device 100 includes a display 102 (e.g., integrated with a touch-sensitive surface) and a camera 104 (e.g., located on the device such that the camera's field of view is pointed toward the user when the device is held in a natural position by the user). In some embodiments, electronic device 100 is a smartphone or tablet. In some embodiments, electronic device 100 is a wearable electronic device (e.g., a smartwatch, smart glasses).

[0023] Figure 1A A user 106 is shown holding electronic device 100 in a physical location 108 (e.g., a room). Physical location 108 includes multiple objects, including object 110 (e.g., a desk), object 112 (e.g., a chair), object 114 (e.g., a mug), object 116 (e.g., a mouse), and object 118 (e.g., a laptop computer). Although Figures 1A to 1G The physical location shown is an indoor room, but it can also be any location in the real-world environment where a user can use the device (e.g., indoor location, outdoor location). Figure 1AIn this case, user 106 is holding electronic device 100 (e.g., in a natural position, allowing the user to easily and comfortably view display 102 and navigate the content displayed on display 102).

[0024] Figure 1B As shown in Figure 1A The first example shown is an electronic device 100 that displays a user interface 120 on a display 102 when the user 106 is in physical position 108 and holds the device in its natural position. Figure 1B As shown, user interface 120 is the main user interface of the operating system of the electronic device, and includes multiple application icons 122A-122D corresponding to various applications installed on the device. In some embodiments, when user interface 120 is displayed, the operating system of the electronic device 100 is executing one or more open applications in the background of the device, such that the user interfaces corresponding to these applications are not currently displayed on the display. In other examples, the electronic device 100 may be displaying the user interface of a third-party application (instead of a first-party application) installed on the device.

[0025] As used herein, the term “open application” or “running application” refers to an application that retains state information and is optionally any of the following three types of applications: (1) “active applications” or “activated applications” that are currently displayed on the display of the device on which they are being used; (2) “background applications” (or background processes) that are not currently displayed but one or more processes of these applications are being processed by one or more processors of the device; and (3) “suspended applications” or “dormant applications” that are not currently running but have state information stored in memory (volatile or non-volatile) that can be used to resume the execution of the application.

[0026] As used herein, the term "closed application" refers to an application that does not retain state information (i.e., the state information of the application used to close it is not stored in the device's memory). Therefore, closing an application includes stopping and / or removing the application's application process and removing the application's state information from the device's memory. Generally, opening a second application while the first application is in the first application does not close the first application. When the second application is displayed and the first application stops displaying, the first application becomes a background application.

[0027] exist Figure 1BIn the process of displaying the user interface 120, the electronic device 100 detects (e.g., via a touch-sensitive surface of the device) a user selection to an application icon 122B corresponding to an application for measuring distances in a real-world environment. For simplicity, the application for measuring distances in a real-world environment is referred to hereinafter as a measurement application.

[0028] In response to detecting a user selection of the application icon 122B corresponding to the measurement application, the electronic device 100 activates the measurement application on the device (and thus the device displays the user interface of the measurement application on the display 102) to display the user interface 124 of the measurement application. Figure 1C For example, if the measurement application is closed, the electronic device 100 activates the measurement application by opening it and displaying its user interface 124. If the measurement application is in the background or suspended, the electronic device 100 activates it by displaying its user interface 124. Figure 1C As shown, electronic device 100 displays a real-time feed image 126 of the area of ​​physical location 108 in user interface 124, wherein the real-time feed image 126 is being captured by camera 104 and corresponds to the area of ​​physical location 108 within the field of view of camera 104. Additionally, in some embodiments, in response to detecting activation of a measurement application, electronic device 100 automatically begins capturing multiple images of the area of ​​physical location 108 within the field of view of camera 104 (e.g., without providing feedback to the user, thus making it invisible to the user whether the device is capturing) through camera 104.

[0029] exist Figures 1C to 1E In the middle, user 106 removes electronic device 100 from its Figure 1B The position in the measurement application (when the measurement application is activated) is moved to Figure 1E The position in the middle, in Figure 1E At the location indicated, object 118 (e.g., a laptop computer) is within the field of view of camera 104. Figures 1C to 1E During the movement of the electronic device 100, when the device is removed from its place by the user Figure 1B As the device moves from its position in 1E to its position in 1E, the electronic device 100 continues to automatically capture multiple images along the device's moving path via the camera 104 (e.g., without providing feedback to the user, thus making it invisible to the user whether the device is capturing).

[0030] In response to (e.g., and) along the movement path of the device (e.g., Figures 1C to 1EThe device moves along a path and captures multiple images. Electronic device 100 determines (e.g., selects) keyframes from the captured images based on one or more features of the captured images (e.g., filtering captured images based on one or more features to determine / identify keyframes). In response to determining keyframes, electronic device 100 stores (e.g., in a storage buffer) the keyframes (e.g., a predetermined number of keyframes). In some embodiments, if the device has already stored (e.g., in a storage buffer) a predefined maximum number of keyframes (e.g., 11 keyframes), electronic device 100 replaces the oldest stored keyframe with the newly determined keyframe.

[0031] In some implementations, one or more features used to determine (e.g., select, identify) keyframes from captured images include the amount of movement (e.g., distance, angular distance, rotation) of the device between the determined keyframes (e.g., determined / detected using the device's accelerometer). For example, if the amount of movement of the device between the capture of a first captured image and a second captured image is at least a predefined amount, then the electronic device 100 selects the first captured image from a plurality of captured images as a first keyframe and selects the second captured image from a plurality of captured images as a second keyframe.

[0032] In some implementations, one or more features used to determine (e.g., select, identify) keyframes from captured images include the image quality of the captured images (e.g., whether one or more visually salient attributes of the captured images or a weighted combination of visually salient attributes of the captured images meet a specific threshold). For example, if a first captured image meets the image quality criteria, the electronic device 100 selects the first captured image from a plurality of captured images, but if a second captured image does not meet the image quality criteria, the electronic device abandons the selection of the second captured image from the plurality of captured images.

[0033] In some implementations, one or more features used to determine (e.g., select, identify) keyframes from captured images include the image's brightness / illuminance level, where the brightness / illuminance level is determined based on data extracted from the captured images. For example, if the brightness / illuminance data extracted from a first captured image meets a threshold criterion, the electronic device 100 selects the first captured image from a plurality of captured images; however, if a second captured image does not meet the threshold criterion, the electronic device abandons the selection of the second captured image from the plurality of captured images.

[0034] Based on its existence Figure 1B The position in Figure 1E After moving along the path of the position in the middle, in Figure 1EIn this scenario, user 106 points the camera 104 of electronic device 100 toward the object 118. While pointing toward the object 118, electronic device 100 displays a live feed image 126 including the object 118 in user interface 124.

[0035] When a live feed image 126 including object 118 is displayed in user interface 124, electronic device 100 detects a request to use a measurement application to measure a distance (e.g., the distance between real-world points corresponding to points within objects such as object 118 shown in the live feed image). In some embodiments, the request to measure a distance includes user selection of one or more points within the user interface 124 of the measurement application that correspond to real-world points (e.g., the two endpoints to which the distance is to be measured), as referenced below. Figures 1E to 1F As described above. In some implementations, the request to measure distance includes user selection of various icons (e.g., application icons), display icons, and / or controls.

[0036] exist Figure 1E In the process of displaying a real-time feed image 126 including object 118, electronic device 100 detects (e.g., via the touch-sensitive surface of the device) a user selection of a first selection point 128 in the real-time feed image 126 corresponding to a real-world first measurement point 130 of object 118 (e.g., a corner, edge, section, or portion of the object).

[0037] In response to detecting a user selection of a first selection point 128 corresponding to a first measurement point 130 of object 118, electronic device 100 sets (e.g., defines, designates) the first measurement point 130 of object 118 as the first endpoint of measurement. In some embodiments, in response to detecting a user selection of the first selection point 128, electronic device 100 displays an indication 132 of the user selection of the first selection point 128 in user interface 124, such as... Figure 1F As shown.

[0038] exist Figure 1F While the live feed image 126 including object 118 is still being displayed, the electronic device 100 detects (e.g., via the touch-sensitive surface of the display) a user selection of a second selection point 134 in the live feed image 126 that corresponds to another real-world second measurement point 136 of object 118 (e.g., different corners, different edges, different sections, different parts of the object).

[0039] In response to detecting a user selection of a second selection point 134 corresponding to the second measurement point 136 of object 118, electronic device 100 sets (e.g., defines, designates) the second measurement point 136 of object 118 as the second endpoint of the measurement. In some embodiments, in response to detecting a user selection of the second selection point 134, electronic device 100 displays an indication 138 of the user selection of the second selection point 134 in user interface 124, such as... Figure 1G As shown.

[0040] exist Figure 1G In response to a detected request to measure distance (e.g., in response to a detected user selection of a first selection point 128 (corresponding to a first measurement point 130 of object 118) and a second selection point 134 (corresponding to a second measurement point 136 of object 118)), the electronic device 100 uses one or more keyframes from a stored set of keyframes to determine (e.g., generate, calculate) measurements corresponding to the first and second selection points (e.g., the real-world straight-line distance between the first and second measurement points 130 and 136 of object 118). For example, if the first measurement point 130 corresponds to a first corner of object 118 and the second measurement point 136 corresponds to a second corner of object 118, where the two corners are connected by a straight edge, the electronic device 100 provides a straight-line distance between the first and second measurement points 130 and 136 corresponding to the edge length. In some embodiments, the electronic device 100 stops capturing multiple images in response to a detected request to measure distance (e.g., in response to a detected user selection of the second selection point 134).

[0041] In some embodiments, electronic device 100 uses only keyframes of images captured during the device's movement path between when the device detects activation of a measurement application and when a request to measure distance is detected on the device. In some embodiments, providing the measurement includes a representation 140 of the measurement between selected portions of object 118 displayed in real-time feed image 126, such as... Figure 1G As shown, the measurement representation 140 includes the measured value 142.

[0042] In some embodiments, using one or more keyframes from the stored keyframes to determine measurements corresponding to selected portions of a real-world object (e.g., between first measurement point 130 and second measurement point 136 of object 118) involves combining machine learning techniques capable of detecting the real-world object (e.g., object 118) in the region of physical location 108 captured by the stored keyframes with techniques for extracting 3D feature points of the real-world object using the keyframes (e.g., using pixel information extracted from the keyframes). In some embodiments, the electronic device 100 then uses the extracted feature points of the object (e.g., object 118) to automatically determine (e.g., define) the boundary volume of the object used to determine the requested measurement. In some embodiments, tracking the placement of the measurement (e.g., the measurement between first measurement point 130 and second measurement point 136 of object 118) in its real-world physical location involves tracking the intersection of the view vector and the surface plane of the requested measurement.

[0043] In some implementations, after detecting a user selection for one or more points used for measurement (e.g., first selection point 128, second selection point 134), the electronic device 100 continues to automatically capture multiple images via the camera 104 and determines new keyframes based on one or more features of the captured images (e.g., to continuously update keyframes stored in a storage buffer). This continuous collection of real-world data enables the electronic device 100 to refine measurements in real time to improve measurement accuracy.

[0044] Figure 2A A user 106 is shown holding an electronic device 100 at a physical location 202 (e.g., an outdoor sports field). The physical location 202 includes multiple objects, including object 204 (e.g., a slide) and object 206 (e.g., a tree).

[0045] Figure 2B An electronic device 100 is shown when user 106 holds the device in a natural position. The electronic device 100 is displaying a user interface 226 on a display 102 for an application active on the device (e.g., a map application), wherein the application corresponding to user interface 226 is not a measurement application. In some embodiments, the application is a third-party application rather than a first-party application.

[0046] In some embodiments, while displaying user interface 226 (e.g., the user interface of a map application), electronic device 100 detects user input (e.g., user input on the device's home button) involving navigation to the device's home user interface (e.g., home user interface 120). In response to detecting user input involving navigation to the home user interface, electronic device 100 displays the home user interface on display 102 (and stops displaying user interface 226). In some embodiments, while displaying the home user interface, electronic device 100 detects user selection of an application icon corresponding to the measurement application (e.g., application icon 122B). In response to detecting user selection of the application icon corresponding to the measurement application on the home user interface, electronic device 100 activates the measurement application by displaying the measurement application's user interface 124 on display 102 (and stops displaying the home user interface).

[0047] Alternatively, in some implementation schemes, see the following references. Figures 2F to 2G More specifically, when displaying user interface 226 (e.g., the user interface of a map application), electronic device 100 detects user input involving the selection of a different application to be viewed on the display. In some embodiments, in response to detecting user input involving the selection of a different application to be viewed on the display, electronic device 100 displays multiple user interface items on display 102 corresponding to applications currently running on the device, including user interface items corresponding to user interface 226 (e.g., the user interface of a map application) and user interface items corresponding to user interface 124 of a measurement application (which is already running on the device). In some embodiments, while displaying multiple user interface items, electronic device 100 detects a user selection for a user interface item corresponding to user interface 124 of the measurement application. In response to detecting a user selection for a user interface item corresponding to user interface 124 of the measurement application, electronic device 100 activates the measurement application (and stops displaying multiple user interface items) by displaying user interface 124 of the measurement application on display 102.

[0048] In some implementations, the electronic device 100 captures multiple images via the camera 104 before detecting activation of the measurement application, as shown in the reference image. Figures 2B to 2E As stated above. Figures 2B to 2E In the event that, while electronic device 100 is displaying user interface 226 (e.g., the user interface of a map application) on display 102, user 106 moves electronic device 100 from its natural location (e.g., in...). Figure 2B (middle) moves to different positions within the field of view of camera 104, where object 204 (e.g., slide) is located (e.g., in the middle) Figure 2Emiddle).

[0049] When electronic device 100 is in Figure 2B Move the position in to its position in Figure 2E When positioned, the device automatically captures multiple images along its movement path using camera 104 (e.g., without providing feedback to the user, thus making it invisible to the user whether the device is capturing). In some embodiments, the electronic device 100 is positioned in its... Figure 2B Before being positioned in the middle, multiple images are automatically captured. In some implementations, the electronic device 100 is positioned in its middle Figure 2E After the location is determined, multiple images continue to be automatically captured. In some implementations, when the device is in its position... Figure 2B Move the position in to its position in Figure 2E When the device is in a certain position, the measurement application is opened / executed on the device (as a background application). In some implementations, when the device is in its position... Figure 2B Move the position in to its position in Figure 2E When the measurement application is closed on the device, multiple images are captured without the measurement application being opened / executed on the device.

[0050] As mentioned above, the first reference is... Figures 1A to 1G In response to (e.g., in) capturing multiple images, electronic device 100 determines (e.g., selects) keyframes from the captured images based on one or more features of the captured images (e.g., filtering the captured images based on one or more features of the images to determine / identify keyframes) and stores (e.g., in a storage buffer) the keyframes (e.g., a predetermined number of keyframes). Examples of one or more features captured for determining keyframes are referenced above. Figures 1A to 1G As mentioned above, and for the sake of simplicity, this article will not repeat it.

[0051] exist Figure 2E In the process of displaying user interface 226 (e.g., the user interface of a map application), user 106 points the camera 104 of electronic device 100 at object 204. While displaying user interface 226, electronic device 100 detects user input involving the selection of different applications to view on the display. Figure 2F As shown, in response to detecting user input involving the selection of different applications to view on the display, the electronic device 100 displays multiple user interface items on the display 102 corresponding to the applications currently running on the device, including those related to... Figure 2EThe electronic device displays user interface item 228 corresponding to the user interface 226 of the active application (e.g., a map application) and user interface item 230 corresponding to the user interface 124 of the measurement application that was previously and is still running on the device. When displaying multiple user interface items, the electronic device detects the user selection for user interface item 228 corresponding to the user interface 124 of the measurement application, such as... Figure 2F As shown.

[0052] exist Figure 2G In response to detecting a user selection on the user interface 228 corresponding to the measurement application, the electronic device 100 activates the measurement application by displaying the user interface 124 of the measurement application on the display 102. Figure 2G As shown, the user interface 124 includes the display of a real-time feed image 208 of the field of view of the camera 104, wherein the real-time feed image includes the object 204. In some embodiments, in response to the detection of activation of the measurement application, and while the user interface 124 of the measurement application is displayed, the electronic device 100 continues to automatically capture multiple images.

[0053] When a live feed image 208 including object 204 is displayed in user interface 124, electronic device 100 detects a request to measure distance using a measurement application (e.g., the distance between real-world points corresponding to objects shown in the live feed image, such as points within object 204). In some embodiments, the request to measure distance includes user selection of one or more points within user interface 124 corresponding to real-world points, as first referenced above. Figures 1E to 1F The above, and referred to again below. Figures 2G to 2H As mentioned above.

[0054] exist Figure 2G In the process of displaying the real-time feed image 208, the electronic device 100 detects (e.g., via the touch-sensitive surface of the device) the user selection of the first selection point 210 in the real-time feed image 208 that corresponds to the first real-world measurement point 212 of the object 204 (e.g., a corner, edge, section, or part of the object).

[0055] In response to detecting a user selection of a first selection point 210 corresponding to a first measurement point 212 of object 204, electronic device 100 sets (e.g., defines, designates) the first measurement point 212 of object 204 as the first endpoint of measurement. In some embodiments, in response to detecting a user selection of the first selection point 210, electronic device 100 displays an indication 214 of the user selection of the first selection point 210 in user interface 124, such as... Figure 2H As shown.

[0056] exist Figure 2HWhile the live feed image 208 including object 204 is still being displayed, the electronic device 100 detects (e.g., via the touch-sensitive surface of the display) a user selection of a second selection point 216 in the live feed image 208 that corresponds to a real-world second measurement point 218 of object 204 (e.g., different corners, different edges, different sections, different parts of the object).

[0057] In response to detecting a user selection of a second selection point 216 corresponding to a second measurement point 218 of object 204, electronic device 100 sets (e.g., defines, designates) the second measurement point 218 of object 204 as a second endpoint of measurement. In some embodiments, in response to detecting a user selection of the second selection point 216, electronic device 100 displays an indication 220 of the user selection of the second selection point 216 in user interface 124, such as... Figure 2I As shown.

[0058] exist Figure 2I In response to a detected request to measure distance (e.g., in response to a detected user selection of a first selection point 210 (corresponding to a first measurement point 212 of object 204) and a second selection point 216 (corresponding to a second measurement point 218 of object 204)), electronic device 100 uses one or more keyframes from a stored set of keyframes to determine (e.g., generate, calculate) the measurement values ​​corresponding to the first and second selection points (e.g., the real-world straight-line distance between the first and second measurement points 212 and 218 of object 204). In other words, in some embodiments, electronic device 100 uses only keyframes from images captured by the device prior to the detection of a request to measure distance on the device. Reference above Figures 1A to 1G The use of stored keyframes to determine measurements is described in more detail, and will not be repeated here for the sake of simplicity.

[0059] In response to determining measurements corresponding to a first selection point and a second selection point, electronic device 100 provides measurements. In some embodiments, providing measurements includes electronic device 100 displaying a representation 222 of the measurements between selected portions of object 204 in a real-time feed image 208, such as... Figure 2I As shown, the measurement representation 222 includes the measured value 224.

[0060] Figure 3A A user 106 is shown at a physical location 302 (e.g., an indoor area, such as a room or hallway). Physical location 302 includes a wall 304 and an object 306 (e.g., a portrait) placed on the wall 304. Figure 3AIn this context, user 106 holds electronic device 100 in physical position 302 (e.g., in a natural position, allowing the user to easily and comfortably view display 102 and navigate the content shown on display 102).

[0061] Figure 3B This illustrates an electronic device 100 displaying a user interface 120 on a display 102 when the user 106 is in physical position 302 and holds the device in its natural position. Figure 3B In the process of displaying the user interface 120, the electronic device 100 detects (e.g., via the touch-sensitive surface of the device) the user's selection of the application icon 122B corresponding to the measurement application.

[0062] exist Figure 3C In response to detecting a user selection of the application icon 122B corresponding to the measurement application, the electronic device 100 activates the measurement application by displaying the user interface 124 of the measurement application on the display 102.

[0063] like Figure 3C As shown, the user interface 124 includes a real-time feed image 308 of the area of ​​physical location 302, wherein the real-time feed image 308 is being captured by camera 104 and corresponds to the area of ​​physical location 302 within the field of view of camera 104. In some embodiments, in response to detecting activation of a measurement application, the electronic device 100 begins to automatically capture (e.g., without providing feedback to the user, thus making it invisible to the user whether the device is capturing) multiple images of the area of ​​physical location 302 within the field of view of camera 104.

[0064] exist Figures 3C to 3E In the middle, user 106 removes electronic device 100 from its Figure 3B The position in the measurement application (when the measurement application is activated) is moved to Figure 3E The wall 304 is positioned within the field of view of camera 104 (e.g., completely within the view of the camera's field of view, such that all four corners of the wall are visible within the real-time feed image 308). Figures 3C to 3E During the movement of the electronic device 100, when the device is taken from its location by the user 106 Figure 3B As the device moves from its position in 3E to its position in 3E, the electronic device 100 continues to automatically capture multiple images along the device's moving path via the camera 104 (e.g., without providing feedback to the user, thus making it invisible to the user whether the device is capturing).

[0065] As mentioned above, the first reference is... Figures 1A to 1G As described, along the movement path of the device (e.g., Figures 3C to 3EDuring or after the capture of multiple images (during the movement path of the device), the electronic device 100 determines (e.g., selects) keyframes from the captured images based on one or more features of the captured images (e.g., filters the captured images based on one or more features of the images to determine / identify keyframes) and stores (e.g., in a storage buffer) the keyframes (e.g., a predetermined number of keyframes). Examples of one or more features captured for determining keyframes are referenced above. Figures 1A to 1G As mentioned above, and for the sake of simplicity, this article will not repeat it.

[0066] In following its Figure 3B The position in Figure 3E After the user moves along the path of the location, the user 106 points the camera 104 of the electronic device 100 toward the direction of the wall 304. In response, the electronic device 100 displays a live feed image 308 in the user interface 124 that includes the wall 304 (e.g., including all four sides of the wall 304).

[0067] In some implementations, in response to or when a real-world object is detected (e.g., using machine learning techniques that utilize 3D feature points extracted from stored keyframes to detect a real-world object within a physical location) when the measurement application is displayed on the display 102, with the wall 304 within the field of view of the camera 104, the electronics 100 automatically determines one or more measurements corresponding to the wall 304 (e.g., the layout of the wall dimensions, such as the horizontal and vertical lengths of the wall; the dimensions of the object 306 suspended on the wall, such as the horizontal and vertical lengths of the object 306; the distance from the edge of the wall 304 to the edge of the object 306).

[0068] In some implementations, the electronic device 100 also automatically displays one or more measurements corresponding to the wall 304 (e.g., the measurement layout 310 of the wall) in the user interface 124, such as Figure 3F As shown in the reference above. Figures 1A to 1G In more detail, electronic device 100 is used in the device from its Figure 3B The initial position (when the measurement application is activated) to its location in Figure 3F The stored keyframes captured during the movement path of the device's current position (when the camera is pointed at the wall) determine one or more measurements corresponding to the wall 304. Therefore, in some embodiments, the measurement application provides one or more measurements of the object (e.g., upon detecting or determining the presence of the object) even if no user request for measurement or specific user input requesting measurement is detected, or before such detection.

[0069] Figure 4This is a flowchart illustrating a method for enabling an electronic device (e.g., 100) to provide measurements according to some embodiments. The electronic device has a camera (e.g., 104). In some embodiments, the electronic device also has a display (e.g., 102) and one or more input devices (e.g., touch-sensitive surfaces).

[0070] Before the activation of an application (e.g., a measurement application) for measuring distances between real-world points (e.g., 130, 136, 212, 218) on an electronic device (e.g., 100) is detected, the electronic device captures multiple images (e.g., at box 402) along its movement path using a camera (e.g., 104). In some embodiments, the movement path of the electronic device (e.g., 100) includes movement of the electronic device in the x and y directions in a three-dimensional plane.

[0071] An electronic device (e.g., 100) determines a keyframe (e.g., at box 404) based on one or more features of multiple images. In some embodiments, determining a keyframe based on one or more features of multiple images includes the electronic device selecting one or more images from the multiple images as keyframes based on one or more features.

[0072] In some embodiments, one or more features include the amount of movement of the electronic device (e.g., 100) between a position of the electronic device corresponding to a first image among a plurality of images and a position of the electronic device corresponding to a second image among a plurality of images. In some embodiments, one or more features include the image quality of the first image among a plurality of images. In some embodiments, one or more features include the brightness level of the first image among a plurality of images.

[0073] The electronic device (e.g., 100) stores (e.g., at box 406) keyframes (e.g., in the device's storage buffer).

[0074] In some implementations, before detecting activation of an application (e.g., a measurement application), the electronic device (e.g., 100) determines feature points based on keyframes, where the feature points identify objects captured in the keyframes (e.g., 118, 204, 304), and where real-world points (e.g., 130, 136, 212, 218) correspond to points on the objects. In some implementations, the electronic device determines feature points based on keyframes and simultaneously determines the distances between real-world points (e.g., in response to detecting activation of the application).

[0075] In some implementations, an electronic device (e.g., 100) detects (e.g., at box 408) activation of an application used for measuring distance (e.g., a measurement application).

[0076] In response to the detection of activation of an application for measuring distance (e.g., a measurement application), an electronic device (e.g., 100) uses one or more keyframes from the stored keyframes to determine (e.g., at box 410) the distance between real-world points (e.g., 130, 136, 212, 218).

[0077] In response to the detection of activation of an application for measuring distance (e.g., a measurement application), an electronic device (e.g., 100) displays (e.g., at box 412) a representation of the distance in the user interface (e.g., 124) of the application (e.g., 140, 142, 222, 224, 310).

[0078] In some implementations, in response to detecting activation of an application for measuring distance (e.g., a measurement application), an electronic device (e.g., 100) detects the selection of a first measurement point (e.g., 128, 210) and a second measurement point (e.g., 134, 216) in a user interface (e.g., 124), wherein the first measurement point corresponds to a first real-world point (e.g., 130, 212) among real-world points in a real-world environment and the second measurement point corresponds to a second real-world point (e.g., 136, 218) among real-world points in a real-world environment, and wherein determining the distance between real-world points includes determining the distance between the first real-world point and the second real-world point.

[0079] In some embodiments, after determining the distance between real-world points (e.g., 130, 136, 212, 218) using one or more keyframes from the stored keyframes, the electronic device (e.g., 100) captures a second plurality of images along a second movement path of the electronic device via a camera (e.g., 104). In some embodiments, the electronic device determines new keyframes based on one or more features of the second plurality of images. In some embodiments, the electronic device stores the new keyframes. In some embodiments, one or more keyframes from the stored new keyframes are used to determine updated distances between real-world points. In some embodiments, the electronic device displays a representation of the updated distances in a user interface (e.g., 124) of an application (e.g., a measurement application).

[0080] Figure 5 This is a flowchart illustrating another method for enabling an electronic device (e.g., 100) to provide measurements, according to some embodiments. The electronic device has a camera (e.g., 104). In some embodiments, the electronic device also has a display (e.g., 102) and one or more input devices (e.g., touch-sensitive surfaces).

[0081] Before detecting a request to measure the distance between real-world points (e.g., 130, 136, 212, 218) using an application (e.g., a measurement application) on an electronic device (e.g., 100), the electronic device captures (e.g., at box 502) multiple images along a movement path of the electronic device from a first position to a second position using a camera (e.g., 104), where the first position corresponds to the position of the electronic device when the application is activated on the electronic device (thus causing the device to display the user interface of the application (e.g., 124)).

[0082] In some embodiments, the second position corresponds to the position of the electronic device (e.g., 100) when a request to measure distance is detected on the electronic device. In some embodiments, the movement path of the electronic device (e.g., 100) from the first position to the second position includes movement of the electronic device in the x and y directions in a three-dimensional plane.

[0083] In some implementations, when the application is not activated on the electronic device, the electronic device (e.g., 100) displays a second user interface (e.g., 120) that does not correspond to the application (e.g., a measurement application). In some implementations, when the application is activated on the electronic device, the electronic device displays the application's user interface (e.g., 124).

[0084] An electronic device (e.g., 100) determines a keyframe (e.g., at box 504) based on one or more features of multiple images. In some embodiments, determining a keyframe based on one or more features of multiple images includes the electronic device selecting one or more images from the multiple images as keyframes based on one or more features.

[0085] In some embodiments, one or more features include the amount of movement of the electronic device (e.g., 100) between a position of the electronic device corresponding to a first image among a plurality of images and a position of the electronic device corresponding to a second image among a plurality of images. In some embodiments, one or more features include the image quality of the first image among a plurality of images. In some embodiments, one or more features include the brightness level of the first image among a plurality of images.

[0086] The electronic device (e.g., 100) stores (e.g., at box 506) keyframes (e.g., in the device's storage buffer).

[0087] In some implementations, prior to detecting a request to measure distance, an electronic device (e.g., 100) determines feature points based on keyframes, wherein the feature points identify real-world objects captured in the keyframes (e.g., 118, 204, 304), and wherein real-world points (e.g., 130, 136, 212, 218) correspond to points on the real-world objects.

[0088] In some embodiments, the electronic device (e.g., 100) detects (e.g., at box 508) a request to measure distance. In some embodiments, detecting the request to measure distance includes the electronic device detecting the selection of a measurement point (e.g., a first measurement point and a second measurement point) in the user interface of an application (e.g., a measurement application), wherein the measurement point in the user interface (e.g., 124) corresponds to a first real-world point among real-world points (e.g., 130, 136, 212, 218) in the real-world environment (e.g., 108, 202, 302).

[0089] In response to a request to detect a distance measurement, an electronic device (e.g., 100) uses one or more keyframes from the stored keyframes to determine (e.g., at box 510) the distance between real-world points (e.g., 130, 136, 212, 218).

[0090] In response to a request to detect a distance measurement, an electronic device (e.g., 100) displays (e.g., at box 512) a representation of the distance in the user interface (e.g., 124) of an application (e.g., a measurement application) of the application (e.g., at box 512).

[0091] In some embodiments, after determining the distance between real-world points (e.g., 130, 136, 212, 218) using one or more keyframes from the stored keyframes, the electronic device (e.g., 100) captures a second plurality of images along a second movement path of the electronic device using a camera (e.g., 104). In some embodiments, the electronic device determines new keyframes based on one or more features of the second plurality of images. In some embodiments, the electronic device stores the new keyframes. In some embodiments, the electronic device uses one or more of the stored new keyframes to determine updated distances between real-world points. In some embodiments, the electronic device displays a representation of the updated distances in the user interface (e.g., 124) of an application (e.g., a measurement application).

[0092] For purposes of explanation, the foregoing description has been given by reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. For example, a wearable device could be a smartwatch, headphones, a pair of glasses, etc. Many modifications and variations are possible based on the teachings above. These embodiments were chosen and described in order to best explain the principles of these techniques and their practical applications. Others skilled in the art will thus be able to best utilize these techniques and the various embodiments with various modifications suitable for the particular intended use.

[0093] While this disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of this disclosure and examples as defined by the claims.

[0094] As described above, one aspect of this technology lies in acquiring and using image data to provide measurements using an application on an electronic device. Entities implementing these features should be careful to distinguish between image processing occurring on the device and information transmitted outside the device. This disclosure envisions that measurement information can be provided using local processing, meaning that images of the user's surrounding environment captured by the device are not transmitted to a remote server.

[0095] When an entity implementing the above technology chooses to transmit images captured by the device as part of a measurement activity outside the device, such an entity should proactively notify its users of the possible data transmission and instruct the users to choose "opt in" or "opt out" so that the device can capture and / or process image data when the relevant application (e.g., a measurement application) is not active.

[0096] Therefore, while this disclosure broadly covers the use of camera images for measurement activities, it also envisions that the captured images are inherently transient.

Claims

1. An electronic device, comprising: camera; One or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: Before detecting a request to measure the distance between real-world points using an application on the electronic device, multiple images are captured via the camera along a movement path of the electronic device from a first position to a second position, wherein the first position corresponds to the position of the electronic device when the application is activated on the electronic device; Keyframes are determined based on one or more features of the plurality of images; Store the keyframe; Feature points are determined based on the keyframe, wherein the feature points identify real-world objects captured in the keyframe, and wherein the real-world points correspond to points on the real-world objects. and In response to the detection of a request to measure the distance: The distance between the real-world points is determined using one or more stored keyframes; as well as The distance is displayed in the user interface of the application.

2. The electronic device of claim 1, wherein the one or more programs further include instructions for: When the application is not activated on the electronic device, a second user interface not corresponding to the application is displayed; and When the application is activated on the electronic device, the application's user interface is displayed instead of the second user interface.

3. The electronic device of claim 1, wherein when the request to measure the distance is detected on the electronic device, the second position corresponds to the position of the electronic device.

4. The electronic device of claim 1, wherein detecting the request to measure the distance includes detecting the selection of a measurement point in the user interface of the application, wherein the measurement point in the user interface corresponds to a first real-world point among the real-world points in the real-world environment.

5. The electronic device of claim 1, wherein determining the keyframe based on one or more features of the plurality of images comprises: Based on the one or more features, one or more images from the plurality of images are selected as the keyframes.

6. The electronic device of claim 1, wherein one or more features include an amount of movement of the electronic device between a position of the electronic device corresponding to a first image in the plurality of images and a position of the electronic device corresponding to a second image in the plurality of images.

7. The electronic device of claim 1, wherein the one or more features include the image quality of the first image among the plurality of images.

8. The electronic device of claim 1, wherein the one or more features include the brightness level of a first image among the plurality of images.

9. The electronic device of claim 1, wherein the movement path of the electronic device from the first position to the second position includes movement of the electronic device in a three-dimensional plane along the x and y directions.

10. The electronic device of claim 1, wherein the one or more programs further include instructions for: After determining the distance between the real-world points using one or more keyframes from the stored keyframes, a second plurality of images are captured by the camera along a second movement path of the electronic device; New keyframes are determined based on one or more features of the second plurality of images; Store the new keyframe; Use one or more of the newly stored keyframes to determine the updated distance between the real-world points; as well as The updated distance is displayed in the user interface of the application.

11. A method of operating an electronic device, comprising: At electronic devices with cameras: Before detecting a request to measure the distance between real-world points using an application on the electronic device, multiple images are captured via the camera along a movement path of the electronic device from a first position to a second position, wherein the first position corresponds to the position of the electronic device when the application is activated on the electronic device; Keyframes are determined based on one or more features of the plurality of images; Store the keyframe; Feature points are determined based on the keyframe, wherein the feature points identify real-world objects captured in the keyframe, and wherein the real-world points correspond to points on the real-world objects. and In response to the detection of a request to measure the distance: The distance between the real-world points is determined using one or more stored keyframes; as well as The distance is displayed in the user interface of the application.

12. The method of claim 11, further comprising: When the application is not activated on the electronic device, a second user interface that does not correspond to the application is displayed. as well as When the application is activated on the electronic device, the application's user interface is displayed instead of the second user interface.

13. The method of claim 11, wherein when the request to measure the distance is detected on the electronic device, the second position corresponds to the position of the electronic device.

14. The method of claim 11, wherein detecting the request to measure the distance includes detecting the selection of a measurement point in the user interface of the application, wherein the measurement point in the user interface corresponds to a first real-world point among the real-world points in the real-world environment.

15. The method of claim 11, wherein determining the keyframe based on one or more features of the plurality of images comprises: Based on the one or more features, one or more images from the plurality of images are selected as the keyframes.

16. The method of claim 11, wherein one or more features include an amount of movement of the electronic device between a position of the electronic device corresponding to a first image in the plurality of images and a position of the electronic device corresponding to a second image in the plurality of images.

17. The method of claim 11, wherein the one or more features include the image quality of the first image among the plurality of images.

18. The method of claim 11, wherein the one or more features include the brightness level of the first image among the plurality of images.

19. The method of claim 11, wherein the movement path of the electronic device from the first position to the second position comprises movement of the electronic device in a three-dimensional plane along the x and y directions.

20. The method of claim 11, further comprising: After determining the distance between the real-world points using one or more keyframes from the stored keyframes, a second plurality of images are captured by the camera along a second movement path of the electronic device; New keyframes are determined based on one or more features of the second plurality of images; Store the new keyframe; Use one or more of the newly stored keyframes to determine the updated distance between the real-world points; as well as The updated distance is displayed in the user interface of the application.

21. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having a camera, the one or more programs including instructions for: Before detecting a request to measure the distance between real-world points using an application on the electronic device, multiple images are captured via the camera along a movement path of the electronic device from a first position to a second position, wherein the first position corresponds to the position of the electronic device when the application is activated on the electronic device; Keyframes are determined based on one or more features of the plurality of images; Store the keyframe; Feature points are determined based on the keyframe, wherein the feature points identify real-world objects captured in the keyframe, and wherein the real-world points correspond to points on the real-world objects. and In response to the detection of a request to measure the distance: The distances between real-world points are determined using one or more stored keyframes; as well as The distance is displayed in the user interface of the application.

22. The computer-readable storage medium of claim 21, wherein the one or more programs further include instructions for: When the application is not activated on the electronic device, a second user interface not corresponding to the application is displayed; and When the application is activated on the electronic device, the application's user interface is displayed instead of the second user interface.

23. The computer-readable storage medium of claim 21, wherein when the request to measure the distance is detected on the electronic device, the second location corresponds to the location of the electronic device.

24. The computer-readable storage medium of claim 21, wherein detecting the request to measure the distance includes detecting the selection of a measurement point in the user interface of the application, wherein the measurement point in the user interface corresponds to a first real-world point among the real-world points in the real-world environment.

25. The computer-readable storage medium of claim 21, wherein determining the keyframe based on one or more features of the plurality of images comprises: Based on the one or more features, one or more images from the plurality of images are selected as the keyframes.

26. The computer-readable storage medium of claim 21, wherein one or more features include an amount of movement of the electronic device between a position of the electronic device corresponding to a first image in the plurality of images and a position of the electronic device corresponding to a second image in the plurality of images.

27. The computer-readable storage medium of claim 21, wherein one or more features include the image quality of the first image among the plurality of images.

28. The computer-readable storage medium of claim 21, wherein one or more features include the brightness level of a first image among the plurality of images.

29. The computer-readable storage medium of claim 21, wherein the movement path of the electronic device from the first position to the second position includes movement of the electronic device in a three-dimensional plane along the x and y directions.

30. The computer-readable storage medium of claim 21, wherein the one or more programs further include instructions for: After determining the distance between the real-world points using one or more keyframes from the stored keyframes, a second plurality of images are captured by the camera along a second movement path of the electronic device; New keyframes are determined based on one or more features of the second plurality of images; Store the new keyframe; Use one or more of the newly stored keyframes to determine the updated distance between the real-world points; as well as The updated distance is displayed in the user interface of the application.

31. An electronic device, comprising: camera; Apparatus for performing the method according to any one of claims 11 to 20.

Images