Location-aware text search and visualization capabilities for physical environments

Abstract

A computing system (100) may include an image access engine (108) configured to access a panoramic point cloud image (210, 220) of a physical environment. The computing system (100) may also include an ambient location aware text engine (110) configured to convert the panoramic point cloud image (210, 220) to an alternative representation (230) to reduce distortion in the panoramic point cloud image (210, 220), and perform an optical character recognition (OCR) process on the alternative representation (230) to determine text in the panoramic point cloud image (210, 220). The environment location aware text engine (110) may also be configured to construct a text tag (310) to track the determined text in the panoramic point cloud image (210, 220) and to support a text search for the physical environment through the text tag.

Description

Background Art

[0001] Computer systems can be used to create, use, and manage data for products, projects, and other objects. Examples of computer systems include computer-aided design (CAD) systems (which may include computer-aided engineering (CAE) systems), visualization and manufacturing systems, product data management (PDM) systems, product lifecycle management (PLM) systems, etc. These systems may include components that facilitate the design, visualization, and simulation testing of product structures and product manufacturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Some examples will be described in detail below with reference to the accompanying drawings.

[0003] Figure 1 An example of a computing system supporting location-aware text capabilities for a physical environment is shown.

[0004] Figure 2 An example of converting a panoramic point cloud image into an alternative representation that supports location-aware text capabilities for the physical environment is shown.

[0005] Figure 3 An example of text label construction by an ambient location-aware text engine supporting location-aware text capabilities in a physical environment is shown.

[0006] Figure 4 An example of constructing a directional view of the physical environment for a text search term through text tags constructed by an ambient location-aware text engine is shown.

[0007] Figure 5 An example of the logic that a system may implement to support location-aware text capabilities for a physical environment is shown.

[0008] Figure 6 An example of a computing system supporting location-aware text capabilities for a physical environment is shown. DETAILED DESCRIPTION

[0009] As technology continues to advance, it becomes increasingly possible to achieve digital representations of physical environments. For example, three-dimensional (3D) scanning cameras can create point clouds by determining a large number of points on the surface of a physical environment. This point cloud technology can be used for increasingly complex analysis and design in textile factories, automotive production lines, microcircuit manufacturing centers, or any other industrial setting. Almost all physical environments include text, whether as safety signs, logos, machine instructions, or countless other forms of textual information.

[0010] Scans of a physical environment (e.g., a point cloud) can capture text information contained in the physical environment as pixel or point data, but typically provide little or no further context about the location of text-specific or other text information in the physical environment. Such scans also cannot identify the actual text itself in the physical environment. Determining and annotating such text information in a physical environment is typically performed as a manual process. Text information in a physical environment is not indexed in any searchable database, and simple text recognition processes cannot take into account the location of the text in the physical environment.

[0011] What is disclosed herein can provide systems, methods, devices, and logic for location-aware text capabilities for physical environments. As described in more detail, the environmental location-aware text technology of the present invention can allow text to be identified and tracked in a physical environment in a manner that tracks location data (e.g., 3D coordinates) of the identified text. The environmental location-aware text technology of the present invention can allow text tags to be constructed to track identified text, doing so in a searchable manner to allow text-specific searches that are tracked to specific locations in a physical environment. Through various visualization capabilities, the location of the physical environment in which the search text is present can be presented through a directional view, wherein a high-contrast text overlay object is used to present the text with improved clarity and accuracy in a given virtual view of the physical environment. Therefore, the environmental location-aware text technology of the present invention can improve virtual navigation of a physical environment, thereby improving system capabilities and new functionality.

[0012] These and other features and technical advantages of the context-aware text technology of the present disclosure are described in more detail herein.

[0013] Figure 1 An example of a computing system 100 that supports location-aware text capabilities for a physical environment is shown. The computing system 100 can take the form of a single or multiple computing devices, such as an application server, a computing node, a desktop or laptop computer, a smartphone or other mobile device, a tablet device, an embedded controller, etc. In some embodiments, the computing system 100 hosts, supports, executes, or implements any combination of applications that provide the ability to visualize a physical environment.

[0014] As an exemplary implementation supporting any combination of the contextual location-aware text features described herein, Figure 1The computing system 100 shown in includes an image access engine 108 and an ambient location-aware text engine 110. The computing system 100 can implement the engines 108 and 110 (including their components) in various ways, for example, as hardware and programming. The programming of the engines 108 and 110 can take the form of processor-executable instructions stored on a non-transitory machine-readable storage medium, and the hardware of the engines 108 and 110 can include a processor that executes these instructions. The processor can take the form of a single processor or a multi-processor system, and in some examples, the computing system 100 uses the same computing system features or hardware components (e.g., a common processor or a common storage medium) to implement multiple engines.

[0015] In operation, the image access engine 108 may access a panoramic point cloud image of a physical environment, which may include location data of points in the panoramic point cloud image. In operation, the environment location-aware text engine 110 may convert the panoramic point cloud image into an alternative representation to reduce distortion in the panoramic point cloud image, perform an optical character recognition (OCR) process on the alternative representation to determine text in the panoramic point cloud image, construct text tags to track text determined in the panoramic point cloud image, and support text searches for the physical environment through the text tags.

[0016] These and other context-aware text features and technical advantages are described in more detail below.

[0017] Figure 2 An example of converting a panoramic point cloud image into an alternative representation that supports location-aware text capabilities for the physical environment is shown. Figure 2 In the example of , the image access engine 108 accesses the panoramic point cloud image 210, which the image access engine 108 can access in any manner. The image access engine 108 can receive the panoramic point cloud image 210 through a network connection, for example, from an imaging device that captures the panoramic point cloud image 210 or from an image processing system that constructs the panoramic point cloud image 210 from point cloud data for a given physical environment. As another example, the image access engine 108 can retrieve the panoramic point cloud image 210 from a configured network drive or by any additional or alternative means.

[0018] Panoramic point cloud image 210 may take the form of any type of image generated for a physical environment through point cloud data. For example, panoramic point cloud image 210 may be captured by a 3D scanning device to capture a factory floor, and various point cloud images in panoramic point cloud image 210 may depict different parts of the factory floor. As another illustrative example, panoramic point cloud image 210 may be generated through laser scanning of a facility, which may be performed periodically to track or monitor a physical environment.

[0019] The panoramic point cloud image 210 accessed by the image access engine 108 may be a point cloud image, because the image may be composed of, related to, associated with, or constructed by a point cloud, such as a collection of data points in a space (e.g., a three-dimensional coordinate system). Although the environmental location awareness technology of the present disclosure may utilize point cloud data to improve text recognition, search, and visualization capabilities for a physical environment, the panoramic point cloud image 210 may be logically different from the point cloud from which the panoramic point cloud image 210 is generated. For example, the point cloud from which the panoramic point cloud image 210 is generated may track, represent, include, or otherwise contain location data about the physical environment, such as through 3D coordinates. A 3D scanning or photogrammetry program may measure such points on the surface of the physical environment, assign coordinates in a 3D space, and utilize the point cloud data to generate the panoramic point cloud image 210. Therefore, discrete location data may be determined for portions (e.g., pixels) in the panoramic point cloud image 210 by the point cloud from which the panoramic point cloud image 210 is generated.

[0020] In some embodiments, the panoramic point cloud image 210 may be generated by capturing an image in conjunction with (e.g., synchronously with) performing a point cloud scan. Construction of the panoramic point cloud image may then be performed by coloring the point cloud pixels based on the image color captured at a particular point cloud location. Thus, pixels in the panoramic point cloud image may be directly related to point cloud pixels, and such point cloud pixels may provide position data (e.g., 3D coordinates) for pixels of the panoramic point cloud image 210. Thus, the panoramic point cloud image 210 may refer to and include a panoramic image of a physical environment in which pixels of the panoramic point cloud image are associated with corresponding point cloud data. Thus, individual pixels in a given panoramic point cloud image may be mapped to corresponding locations in the physical environment, such as by mapping 3D coordinates of a 3D space of the physical environment.

[0021] The panoramic point cloud image 210 may be panoramic in that the image may represent a particular view or visible area in a physical environment, which may be due to a fixed position in the physical environment at which the omnidirectional 3D scanning device captures point data for a given panoramic point cloud image. In this regard, the panoramic point cloud image may represent an “unwrapped” 360° view of the physical environment in a 2D rectangular image format. In other words, the panoramic point cloud image may provide a 360° field of view in a horizontal direction from a given point in the physical environment. Therefore, the panoramic point cloud image 210 accessed by the image access engine 108 may be equirectangular because a spherical view (e.g., a point cloud captured by a 360° field of view of a laser scanning device) is projected onto a rectangular image.

[0022] Although many physical environments may include text (e.g., signs, equipment or site identifiers, safety warnings, etc.), a panoramic point cloud image generated by laser scanning or other imaging techniques may lack any type of data identifying such text beyond image pixel or point data at the location of such text. The context location-aware text engine 110 may support recognition and tracking of text included in the physical environment by processing the panoramic point cloud image 210. However, equirectangular images may provide a distorted view of portions of the physical environment because projection of a spherical view into a rectangular image format may result in stretching, line distortion, skewing, or other visual effects that affect the accuracy of representing text and objects in the physical environment.

[0023] To address this image distortion, the context-aware text engine 110 may convert the panoramic point cloud image 210 into an alternative representation to reduce the distortion. Figure 2 In the example of , the context location-aware text engine 110 converts the panoramic point cloud image 220 into the alternative representation 230, doing so to reduce (e.g., eliminate) distortion caused by the equirectangular projection in the panoramic point cloud image 220. As used herein, reducing distortion may refer to or include any conversion to a different image format to improve the effectiveness of text recognition by the OCR process. Thus, the image conversion performed by the context location-aware text engine 110 to reduce distortion may include image sharpening processing or conversion to a different image format to reduce skew or stretching, or otherwise eliminate visual artifacts that reduce the clarity of text in the panoramic point cloud image.

[0024] As an illustrative example, the context-aware text engine 110 may convert the panoramic point cloud image to a cubemap, cylindrical, Mercator, sinusoidal, or any other suitable image format. Figure 2In the specific example shown in , the context location-aware text engine 110 converts the panorama point cloud image 220 into an alternative representation 230 as a cube map. However, any other suitable image format that the alternative representation may take is contemplated herein. Thus, the context location-aware text engine 110 may implement any image conversion capabilities in order to process the panorama point cloud image to reduce distortion, improve text clarity, etc.

[0025] By converting the alternative representation from the panoramic point cloud image, the context location aware text engine 110 can recognize text in the physical environment and construct text tags to track the recognized text. Figure 3 Examples of these features are described in more detail.

[0026] Figure 3 An example of text tag construction is shown by an ambient location-aware text engine 110 that supports location-aware text capabilities for the physical environment. Figure 3 In the example of , the context location-aware text engine 110 processes the alternative representation 230 of the panoramic point cloud image 220 to identify text in the physical environment described by the panoramic point cloud image 220. To do so, the context location-aware text engine 110 may perform an OCR process on the alternative representation 230 to determine the text in the panoramic point cloud image 220, and the OCR process may be performed by any suitable OCR technique, process, or algorithm.

[0027] In some implementations, the OCR process applied by the context location-aware text engine 110 may output a bounding box around the text identified in the alternate representation 230. Each instance of text identified in the alternate representation 230 may be tracked by the context location-aware text engine 110 via a corresponding bounding box. However, because the context location-aware text engine 110 may perform the OCR process on the alternate representation 230 (rather than the panoramic point cloud image 220 itself), the pixels of the alternate representation 230 where the bounding box is identified may lack location data for the physical environment in which the text appears.

[0028] In order to associate the text and bounding box recognized in the alternative representation 230, the context location-aware text engine 110 may map the text recognized in the alternative representation 230 to the corresponding position in the panoramic point cloud image 220. To do so, the context location-aware text engine 110 may convert the alternative representation 230 back to the panoramic point cloud image 220, or otherwise access the panoramic point cloud image 220 from which the alternative representation 230 was generated. When converting the panoramic point cloud image to the alternative representation, the context location-aware text engine 110 may maintain a 1:1 pixel association between the images. Thus, the context location-aware text engine 110 may map a single pixel in the alternative representation 230 to a corresponding single pixel in the panoramic point cloud image 220, and vice versa.

[0029] By such pixel association, the context location-aware text engine 110 can determine, assign, or otherwise specify corresponding locations in the panoramic point cloud image 220 where the text recognized in the alternative representation 230 is located. In doing so, the context location-aware text engine 110 can assign 3D coordinates to the text recognized in the physical environment. In some embodiments, the context location-aware text engine 110 can associate the recognized text by selected points of a bounding box of the text recognized in the alternative representation 230. To provide an illustrative example, the context location-aware text engine 110 can determine a bounding box for a given instance of text recognized in the alternative representation 230. The context location-aware text engine 110 can identify selected points in the bounding box in the alternative representation 230, such as corner coordinates or center coordinates of the bounding box, and identify corresponding coordinates of these selected points in the panoramic point cloud image 220. Through these corresponding coordinates in the panoramic point cloud image 220, the context location-aware text engine 110 can construct a bounding box for a given instance of text in the panoramic point cloud image 220, even if the given instance of the text is recognized in the alternative representation 230.

[0030] In addition to plain text recognition, the environmental location-aware text technology of the present disclosure may provide the ability to combine location data with text recognized in a physical environment, which may support location-aware text capabilities in analyzing physical environments. To support these features, the environmental location-aware text engine 110 may utilize location data embedded or otherwise included as part of the point cloud data of the panoramic point cloud image. Since the environmental location-aware text engine 110 may determine (e.g., through selected bounding box points) specific pixels where text is recognized in the panoramic point cloud image 220, the environmental location-aware text engine 110 may extract location data for these specific pixels, such as 3D coordinate data of data points in the point cloud at these specific pixels in the panoramic point cloud image 220. Such location data (e.g., 3D coordinates) may be directly embedded in, associated with, or otherwise associated with these specific pixels through the point cloud construction of the panoramic point cloud image 220 (or by accessing the corresponding point cloud from which the panoramic point cloud image 220 is constructed). Thus, the context-location-aware text engine 110 may track the location of recognized text in the physical environment, for example, by assigning 3D coordinates to a given text tag for the text that is tracked by the given text tag.

[0031] The context location-aware text engine 110 may construct text tags to track text recognized in the physical environment and mapped to the panoramic point cloud image 220 via the alternative representation 230. A text tag may refer to any data element storing text information of text recognized in the physical environment, and the context location-aware text engine 110 may construct the text tag in any number of forms or formats. Figure 3 In the example of , the context location-aware text engine 110 constructs text tags 310 for text recognized in the panoramic point cloud image 220 through an OCR process performed on the alternative representation 230 .

[0032] The context location-aware text engine 110 may construct text tags 310 such that each instance of text recognized in the physical environment is tracked by specific text information. Example forms of text information that the context location-aware text engine 110 may track in text tags may include the ID of a specific panoramic point cloud image in which the text instance was recognized (e.g., a specific panoramic point cloud image in a collection of panoramic point cloud images 210 depicting the physical environment), the 3D coordinates (or any other form of location data) at which the text instance is located (which may be specified by a selected point of the text instance or a bounding box surrounding the text instance), the recognized text itself (e.g., "Site #1" is the text itself as determined by OCR), etc.

[0033] In some embodiments, the context location-aware text engine 110 may construct a text label 310 to include, for a given text portion, given text identified in the panoramic point cloud image 220, an identifier of the panoramic point cloud image 220, location data of a selected point in the text portion, and the given text included in the given text portion. The given text portion may include a bounding box of the given text in the panoramic point cloud image 220 or be otherwise tracked by a bounding box of the given text in the panoramic point cloud image 220, and the location data in the text label may be a selected point on or within the bounding box (e.g., a bounding box corner or a bounding box center).

[0034] As another example form of text information to be included in a text label, the context location-aware text engine 110 may store visualization data to depict text in a virtual view of a physical environment in a constructed text label. Such visualization data may include spherical θ (theta) angles and spherical φ (phi) angles for coordinates of a selected location in a portion of the text in the panoramic point cloud image 220 that includes the text, and these angles may represent the vertical angle and horizontal angle, respectively, from the center of the sphere in the 360° field of view to a specific point on the sphere where the text appears in the projection of the panoramic point cloud image 220 into the 360° field of view (e.g., when the panoramic point cloud image 220 is wrapped into a sphere for 360° viewing). Through such visualization data (e.g., spherical θ angles and spherical φ angles), the context location-aware text engine 110 may provide or support a directional view of the physical environment in which the text appears, as described in more detail below.

[0035] The environment location-aware text engine 110 can store the constructed text tags 310 in a searchable database through which queries can be performed to easily search for and identify specific text located in the physical environment. Accordingly, by constructing the text tags 310, the environment location-aware text engine 110 can support various location-aware text searches and visualization capabilities for the physical environment. The environment location-aware text engine 110 supports querying specific text in the physical environment, which can be specified by a text search term. For a given text search term, the environment location-aware text engine 110 can search for specific text in the text tags 310, and if found, the 3D coordinates of any occurrence of the text in the physical environment, a series of any (e.g., all) panoramic point cloud images (which contain text, spherical θ angles and spherical φ angles for selected points of the bounding box for such occurrences, or any additional or alternative text information related to the determined occurrences) can be returned (e.g., by the 3D coordinates of the center of the bounding box).

[0036] With such text tags 310 and a searchable data structure storing the text tags 310, the context location-aware text engine 110 can present any search result "hits" for a given text search term in any number of ways. For example, the context location-aware text engine 110 can provide a directional view of the occurrences of text matching the text search term in the physical environment, as shown below in conjunction with Figure 4 Its exemplary features are given.

[0037] Figure 4 An example of constructing a directional view of the physical environment for a text search term through text tags constructed by the context location-aware text engine 110 is shown. Figure 4In the example of , the context location-aware text engine 110 identifies (e.g., receives) a text search term 410 provided via a search query. The search query may be implemented via a user interface in which a user may specify a particular text search term to be found in a physical environment captured via a panoramic point cloud image. The context location-aware text engine 110 may use the text search term 410 to search in the constructed text tags 310 for the physical environment. Since the text tags 310 may store text information in the form of text recognized in the physical environment, the particular text search term may be searched for via any suitable search algorithm or technique, capability that the context location-aware text engine 110 may implement or otherwise access.

[0038] In response to determining that one or more of the text tags 310 matches the text search term 410, the context location-aware text engine 110 may return search results for the text search term 410. The return of search results may be provided by the context location-aware text engine 110 in any number of ways, such as by providing a search hit list for the text search term 410, which may include providing any combination of text information stored in the text tags 310, such as a panoramic point cloud image ID, 3D coordinate data, a thumbnail image of the text location in the physical environment, and the like.

[0039] In some examples, the context location-aware text engine 110 can visualize the search results for the text search term 410 by providing a directional view 420 of the physical environment that includes the text search term 410. The directional view can be oriented relative to the location (in terms of viewing angle) in the physical environment that includes the text search term 410. To provide the directional view 420, the context location-aware text engine 110 can utilize the spherical θ angle and φ angle stored for a specific text tag of the text that includes the text search term 410. Using the spherical θ angle and φ angle (or any other relevant visualization data), the context location-aware text engine 110 can project a view of the physical environment (e.g., which can be displayed in a rectangular format via a monitor, screen, or user interface) specifically at a viewpoint identified by the spherical θ angle and spherical φ angle in a 360° view that is projected for a specific panoramic point cloud image identified by a panoramic point cloud image ID in a specific text tag.

[0040] In this way, the directional view 420 can be projected directly to the location in the physical environment where the text specified in the text search term 420 appears. The directional view 420 can be angled and presented along a specific line of sight so as to target the text appearing in the physical environment. This can improve visual clarity and reduce skew or distortion of the text in the virtual navigation of the physical environment.

[0041] Although the directional view 420 may be generated specifically for the perspective at which a particular text appears, as described for the text search term 410, other text portions (text portions other than the text search term 410) may exist in the directional view 420. The ambient location-aware text engine 110 may de-skew these other text portions in the directional view 420 that do not include the text search term 410. In some implementations, the location-aware text engine 110 may determine all text present in a given view of the physical environment (e.g., the directional view 420 or any other rendered view). To do so, the ambient location-aware text engine may determine the coordinate boundaries of the given view and search the text tags 310 to determine any text contained within the coordinate boundaries of the given view.

[0042] The context-aware text engine 110 can then deskew the text portion that includes the text present in the given view of the physical environment and present the deskewed text portion together with the text present in the given physical environment view. The context-aware text engine 110 deskewing the text portion can include changing a bounding box of the text portion (and, in some implementations, the text content of the bounding box) so that the sides of the bounding box are parallel to the edges of the given view.

[0043] In order to de-skew or otherwise visualize a portion of text, the environment location aware text engine 110 can generate a text overlay. For text in a given view of a physical environment, the environment location aware text engine 110 can generate a new visualization object that covers the top of a specific view in a 360° field of view or a projected panoramic point cloud image, referred to herein as a text overlay object. The text overlay object may include a background color and text included at a specific location in the physical environment. The environment location aware text engine 110 can configure various text overlay objects, such as by constructing the text overlay object as a rectangle and determining the background and text colors based on a contrast standard (e.g., having different color values, e.g., different RGB values ​​exceeding a difference threshold). One such example is a text overlay object configured with a black background color and white text, and vice versa. By supporting the generation and visualization of text overlay objects in high contrast mode, the environment location aware text engine 110 can improve the readability of the virtual view of the physical environment.

[0044] As another example, the context-aware text engine 110 can construct text overlay objects such that the background color and / or text color is based on the physical environment itself, e.g., a text color that matches the color of text in the physical environment, and / or a background color that matches a selected portion of the physical environment. Thus, the text overlay objects can be constructed to simulate the physical environment and reduce variations in the virtual view that may distract a user viewing the physical environment, while providing accurate and readable text for the various text portions included in the virtual view of the physical environment.

[0045] Any suitable configuration of text overlay objects is contemplated herein, and such overlay object configuration may be customized based on location in the physical environment (e.g., different rooms for a particular factory route) or according to any other attribute, context, or criteria, whether specific to location in the physical environment or any additional or alternative constraints. In any of the ways described herein, the context-location-aware text engine 110 may visualize text in a given view of the physical environment via a text overlay object.

[0046] When visualizing text for a given (virtual) view of a physical environment, the environment location-aware text engine 110 can selectively determine which text instances in the physical environment are visualized by text overlay objects, and how the determined text overlay objects are visualized in a given view. In some embodiments, the environment location-aware text engine 110 constructs and overlays the text overlay object in a given virtual view normally to the viewpoint depicted for the physical environment. Based on the angle difference between the text instance in the physical environment and the viewpoint depicted in a given view (e.g., as specified by the spherical θ angle and spherical φ angle of the text instance and the current view), selectively determine which text instances overlap with the text overlay object. When the angle difference exceeds a configurable threshold (e.g., 20°), the environment location-aware text engine 110 can determine not to generate a text overlay object for the specific text instance or not to visualize the text overlay object for the specific text instance. When the angle difference does not exceed a configurable threshold (e.g., 20°), the environment location-aware text engine 110 can construct the text overlay object to overlay the specific text instance in a given view (e.g., normal to the current view to do so).

[0047] As yet another feature provided by the context location awareness technology of the present invention, the context location awareness engine 110 can construct a text overlay object to redact, hide, or obscure text in a given view of a physical environment. For confidential, sensitive, high priority, or text identified according to any configurable redaction criteria for a physical environment, the context location awareness engine 110 can generate an overlay text object that obscures the actual text in a given virtual view of the physical environment. Such a text overlay object can be constructed as a black box without any text to obscure and hide text instances in a physical environment that meet the redaction criteria.

[0048] Although many of the contextual location-aware text features herein are described through illustrative examples presented via various figures, the image access engine 108 or the contextual location-aware text engine 110 may implement any combination of the contextual location-aware text techniques described herein.

[0049] Figure 5 An example of logic 500 is shown, and the system can implement the logic 500 to support location-aware text capabilities for physical environments. For example, the computing system 100 can implement the logic 500 as hardware, executable instructions stored on a machine-readable medium, or a combination of the two. The computing system 100 can implement the logic 500 through the image access engine 108 and the environmental location-aware text engine 110, and the computing system 100 can execute or implement the logic 500 through the logic 500 as a method for providing any combination of environmental location-aware text features presented herein. The following describes the logic 500 using the image access engine 108 and the environmental location-aware text engine 110 as examples. However, various other implementation options of the computing system are possible.

[0050] In implementing the logic 500, the image access engine 108 may access a panoramic point cloud image of a physical environment (502), and the panoramic point cloud image may include location data of points in the panoramic point cloud image. In implementing the logic 500, the context location-aware text engine 110 may convert the panoramic point cloud image into an alternative representation to reduce distortion in the panoramic point cloud image (504), perform an OCR process on the alternative representation to determine text in the panoramic point cloud image (506), construct text tags to track the determined text in the panoramic point cloud image (508), and support text searching for the physical environment through the text tags (510), performed in any manner described herein.

[0051] Figure 5The logic 500 shown in provides an illustrative example by which the computing system 100 can support the contextual location-aware text capability according to the present invention. Additional or alternative steps in the logic 500 are contemplated herein, including any of the various features described herein for the image access engine 108, the contextual location-aware text engine 110, or any combination thereof.

[0052] Figure 6 An example of a computing system 600 that supports location-aware text capabilities for a physical environment is shown. The computing system 600 may include a processor 610, which may take the form of a single processor or multiple processors. The processor 610 may include a central processing unit (CPU), a microprocessor, or any hardware device suitable for executing instructions stored on a machine-readable medium. The computing system 600 may include a machine-readable medium 620. The machine-readable medium 620 may take the form of any non-transitory electronic, magnetic, optical, or other physical storage device that stores executable instructions, such as Figure 6 622 and context-sensitive text instructions 624. Thus, the machine-readable medium 620 may be, for example, a random access memory (RAM) such as a dynamic RAM (DRAM), a flash memory, a spin transfer torque memory, an electrically erasable programmable read-only memory (EEPROM), a storage drive, an optical disk, or the like.

[0053] The computing system 600 may execute instructions stored on the machine-readable medium 620 via the processor 610. Executing the instructions (e.g., the image access instructions 622 and / or the ambient location-aware text instructions 624) may cause the computing system 600 to perform any aspects of the ambient location-aware text techniques described herein, including any features according to the image access engine 108, the ambient location-aware text engine 110, or a combination of both.

[0054] For example, execution of the image access instructions 622 by the processor 610 may enable the computing system 600 to access a panoramic point cloud image of the physical environment, which may include location data of points in the panoramic point cloud image. Execution of the environment position-aware text instructions 624 by the processor 610 may enable the computing system 600 to convert the panoramic point cloud image into an alternative representation to reduce distortion in the panoramic point cloud image, perform an OCR process on the alternative representation to determine text in the panoramic point cloud image, construct text tags to track the determined text in the panoramic point cloud image, and support text search for the physical environment through the text tags.

[0055] Any additional or alternative ambient location-aware text features described herein may be implemented via image access instructions 622 , ambient location-aware text instructions 624 , or a combination of both.

[0056] The systems, methods, devices, and logic described above, including the image access engine 108 and the context location aware text engine 110, may be implemented in a variety of different ways in a variety of different combinations of hardware, logic, circuitry, and executable instructions stored on machine-readable media. For example, the image access engine 108, the context location aware text engine 110, or a combination thereof may include circuitry in a controller, microprocessor, or application specific integrated circuit (ASIC), or may be implemented by a combination of discrete logic or components or other types of analog or digital circuitry, which are combined on a single integrated circuit or distributed among multiple integrated circuits. A product, such as a computer program product, may include a storage medium and machine-readable instructions stored in the medium, which, when executed in an endpoint, computer system, or other device, causes the device to perform operations described in accordance with any of the above, including any features of the image access engine 108, the context location aware text engine 110, or a combination thereof.

[0057] The processing capabilities of the systems, devices, and engines described herein, including the image access engine 108 and the context location aware text engine 110, can be distributed among multiple system components, such as multiple processors and memories, optionally including multiple distributed processing systems or clouds / network elements. Parameters, databases, and other data structures can be stored and managed separately, can be incorporated into a single memory or database, can be logically and physically organized in a variety of different ways, and can be implemented in a variety of ways, including data structures such as linked lists, hash tables, or implicit storage mechanisms. Programs can be a single program, parts of separate programs (e.g., subroutines), distributed among multiple memories and processors, or implemented in many different ways, for example, in a library (e.g., a shared library).

[0058] While various examples are described above, many more implementations are possible.

Claims

1. A method comprising: By calculating system (100, 600): accessing (502) a panoramic point cloud image (210, 220) of a physical environment, wherein the panoramic point cloud image (210, 220) includes position data of points in the panoramic point cloud image (210, 220); converting (504) the panoramic point cloud image (210, 220) to an alternative representation (230) to reduce distortion in the panoramic point cloud image (210, 220); performing (506) an optical character recognition (OCR) process on the alternative representation (230) to determine text in the panoramic point cloud image (210, 220); constructing (508) a text tag (310) to track the text identified in the panoramic point cloud image (210, 220); and Supporting (510) textual search for the physical environment through the textual tags (310).

2. The method according to claim 1, comprising: For a given text portion of a given text identified in the panoramic point cloud image (210, 220), the text label (310) is constructed to include: an identifier of the panoramic point cloud image (210, 220); Position data of a selected point in the text portion; and The given text included in the given text portion.

3. The method according to claim 2, wherein: The given text portion includes a bounding box of the given text in the panoramic point cloud image (210, 220), and the position data is for a selected point on or within the bounding box.

4. The method according to any one of claims 1 to 3, wherein: The alternative representation (230) comprises a cube map, and wherein the text label (310) comprises spherical θ angles and spherical φ angles of coordinates of a selected location in a text portion of the panoramic point cloud image (210, 220) that includes the text.

5. The method according to any one of claims 1 to 4, wherein: Supporting text search for the physical environment through the text tag (310) includes: identifying textual search terms provided via a search query (410); and A directional view (420) of the physical environment including the text search term (410) is provided, wherein the directional view (420) includes a virtual view of the physical environment, and the virtual view of the physical environment is oriented relative to a location in the physical environment including the text search term (410).

6. The method according to claim 5, further comprising: Other text portions in the directional view (420) that do not include the text search term (410) are de-skewed.

7. The method according to any one of claims 1 to 6, wherein: The supporting of text search for the physical environment through the text tag (310) includes: determining all text present in a given view of the physical environment; de-skewing a text portion of the text present in the given view that includes the physical environment; and The de-skewed portion of the text present in the given view with the physical environment is presented.

8. A system (100), wherein: include: an image access engine (108) configured to access a panoramic point cloud image (210, 220) of a physical environment, wherein the panoramic point cloud image (210, 220) includes position data of points in the panoramic point cloud image (210, 220); and The context location aware text engine (110) is configured to: converting the panoramic point cloud image (210, 220) to an alternative representation (230) to reduce distortion in the panoramic point cloud image (210, 220); performing an optical character recognition (OCR) process on the alternative representation (230) to determine text in the panoramic point cloud image (210, 220); constructing a text tag (310) to track the text identified in the panoramic point cloud image (210, 220); and Supporting text search for the physical environment through text tags (310).

9. The system (100) according to claim 8, wherein: The context location-aware text engine (110) is configured to construct the text tag (310) for a given text portion of a given text identified in the panoramic point cloud image (210, 220) to include: an identifier of the panoramic point cloud image (210, 220); Position data of a selected point in the text portion; and The given text included in the given text portion.

10. The system (100) according to claim 8, wherein: The given text portion includes a bounding box of a given text in the panoramic point cloud image (210, 220), and the position data is for a selected point on or within the bounding box.

11. The system (100) according to any one of claims 8 to 10, wherein: The alternative representation (230) comprises a cube map, and wherein the text label (310) comprises spherical θ angles and spherical φ angles of coordinates of a selected location in a text portion of the panoramic point cloud image (210, 220) that includes the text.

12. The system (100) according to any one of claims 8 to 11, wherein: The context location aware text engine (110) is configured to support text search for the physical environment through the text tag (310) by: identifying textual search terms provided via a search query (410); and A directional view (420) of the physical environment including the text search term (410) is provided, wherein the directional view (420) includes a virtual view of the physical environment, and the virtual view of the physical environment is oriented relative to a location in the physical environment including the text search term (410).

13. The system (100) of claim 12, wherein: The ambient location-aware text engine (110) is also configured to de-skew other text portions in the directional view (420) that do not include the text search term (410).

14. The system (100) according to any one of claims 8 to 14, wherein: The context location aware text engine (110) is configured to support text search for the physical environment through the text tag (310) by: determining all text present in a given view of the physical environment; de-skews a text portion of the text present in the given view that includes the physical environment; and The de-skewed portion of the text present in the given view with the physical environment is presented.

15. A non-transitory machine-readable medium (620) comprising instructions (622, 624) which, when executed by a processor (610), cause a computing system (600) to perform the method of any one of claims 1 to 7.

Images