Building an environmental view from selectively determined environmental images

Abstract

A computing system can include a client device and a server. The client device can be configured to access a stream of image frames depicting an environment, determine, from the stream of image frames, an environment image that satisfies selection criteria, and transmit the environment image to the server. The server can be configured to receive, from the client device, the environment image, construct a spatial view of the environment based on the environment image and location data contained therein, and navigate the spatial view, including by receiving a movement direction and advancing from a current environment image depicted for the spatial view to a next environment image based on the movement direction.

Description

Background Technology

[0001] Computer systems can be used to create, use, and manage data for products and other items. For example, Computer-Aided Technology (CAx) systems can be used to assist in the design, analysis, simulation, or manufacturing of products. Examples of CAx systems include Computer-Aided Design (CAD) systems, Computer-Aided Engineering (CAE) systems, visualization and computer-aided manufacturing (CAA) systems, Manufacturing Operations Management (MOM) systems, Product Data Management (PDM) systems, and Product Lifecycle Management (PLM) systems. These CAx systems may include components (such as CAx applications) that facilitate the design, simulation, testing, and management of product structures and the manufacturing of products. Attached Figure Description

[0002] Some examples are described in the following detailed description and with reference to the accompanying drawings.

[0003] Figure 1 An example of a computing system that supports constructing an environmental view from selectively determined environmental images is shown.

[0004] Figure 2 An example is shown where an environmental image is determined based on a stream of image frames captured by a client device.

[0005] Figure 3 An example is shown of the logic that a computing system can implement to determine an environmental image from an image stream via selection criteria.

[0006] Figure 4 An example of constructing an environment view from an environment image transmitted by a client device is shown.

[0007] Figure 5 An example is shown of the logic that a computing system can implement to construct an environmental view from a transmitted environmental image.

[0008] Figure 6 An example of a computing system that supports constructing an environmental view from selectively determined environmental images is shown. Detailed Implementation

[0009] As technology advances, modern industrial environments are becoming increasingly complex in design, layout, and operation. For example, modern assembly lines and manufacturing plants can incorporate sophisticated robotic machinery and other factory technologies that support the semi-automated or fully automated construction of highly complex components, ranging from household appliances to semiconductor components, motor vehicles, and virtually countless other types of products. Maintaining complex industrial environments can be challenging, and a thorough understanding of every plant component that may be included in a modern industrial environment is practically impossible. Therefore, maintenance of plants and other manufacturing environments is often outsourced to external service providers specializing in the maintenance or repair of specific physical hardware. Service appointments for repairing or maintaining facility resources can take the form of on-site visits by mechanical experts, which can incur time and travel costs and limit the efficiency of such maintenance procedures.

[0010] Alternatively, service hotlines can provide a telephone mechanism for verbally describing maintenance issues, but verbal descriptions alone may be incomplete in explaining, diagnosing, or resolving faulty industrial systems. Teleconferencing can provide maintenance personnel or experts with a mechanism to digitally view the industrial environment and discuss maintenance requirements with field operators. However, general video conferencing technology can provide an inefficient mechanism through which remote experts or maintenance personnel can assess a given environment and provide necessary feedback. For example, indiscriminate video transmission from client devices physically located on the factory floor can consume significant amounts of network bandwidth and computing power in order to properly compress and transmit video streams. Continuous video streaming can also deplete the battery resources of client devices used by local operators to provide remote maintenance experts with a digital view of the industrial environment.

[0011] Furthermore, operator-driven video streaming relies on the local operator's guidance to move the local client device to capture the environmental portions necessary for a remote expert to fully assess the environment's condition. Rapid hand or head movements can result in blurry images and low-quality video streams, potentially reducing the efficiency of maintenance and repair work. As another drawback, teleconferencing technology provides a real-time, operator-controlled view, and remote experts cannot rewind or review relevant environmental portions unless the operator redirects the teleconferencing view to a specific location within the environment.

[0012] The disclosure herein provides systems, methods, apparatus, and logic for constructing environmental views from selectively determined environmental images. The environmental construction techniques described herein provide the ability to selectively determine environmental images from image streams captured by client devices physically located within an environment. Instead of indiscriminately transmitting individual image frames (e.g., video frames) captured by the client device, the environmental construction techniques described herein can apply any number of selection criteria to selectively determine specific environmental images to be transmitted for processing and visualization by a remote server (e.g., accessible and usable by a remote technician or expert). Selection criteria applied by the client device to determine the environmental images may include, for example, frequency criteria to reduce excessive consumption of network resources, ambiguity criteria to enforce threshold image quality requirements, and / or difference criteria to reduce redundancy in the transmitted images. These selection criteria can together improve the effectiveness and efficiency of capturing environmental images depicting a given physical environment. Therefore, the environmental construction techniques described herein can provide technical benefits by reducing network latency, reducing network and computing resource consumption, and improving the power efficiency of client devices used to transmit images of the environment for remote consideration.

[0013] Additionally or alternatively, the environment construction technique presented herein can provide various server-side environment view construction features, through which spatial and temporal views of the environment can be generated from transmitted environment images. Using timestamps, pose data, location data, and orientation data that can communicate with the environment images, the proposed environment construction technique can construct a spatial view of a given environment, which users can navigate to view different parts of the environment. This allows remote users to control navigation of the environment without expending time and network bandwidth to instruct local operators to move client device views to specific locations. Moreover, by classifying environment images into different temporal sets, the environment construction technique described herein allows remote experts to view specific environmental portions over time. By providing such time-based navigation, the environment construction technique presented herein can improve the ability of remote technicians to assess the condition of specific environmental components and repair environmental defects.

[0014] The following sections will describe in more detail these and other features and technical benefits of the environment construction technology proposed in this paper.

[0015] Figure 1 An example of a computing system 100 that supports constructing an environmental view from selectively determined environmental images is shown. Figure 1 In the example shown, computing system 100 includes client device 102. Client device 102 can take the form of any device that can be used to capture and transmit images of the environment. Figure 1In the specific example shown, client device 102 takes the form of augmented reality (AR) glasses or smart glasses. In other examples, client device 102 may be a mobile phone (e.g., a smartphone), a tablet, a laptop computer, or any other suitable computing device. Client device 102 may be environment-local and therefore supports the capture of images of the environment (e.g., as a video stream of image frames capturing the environment). As described in more detail herein, client device 102 may selectively determine environment frames from the captured image stream, and then the environment frames may be transmitted for subsequent processing, environment viewing, and navigation.

[0016] Figure 1 The illustrated computing system 100 also includes a server 104. Server 104 can be any computing entity that receives environmental images transmitted by client device 102. Thus, server 104 can take the form of any number of computing devices (e.g., application servers, compute nodes, desktop or laptop computers, smartphones or other mobile devices, tablets, embedded controllers, etc.). Server 104 can be located remotely from client device 102, for example, as a computing system accessed and used by remote experts or technicians to collaborate with local operators in the industrial environment using client device 102. As described herein, server 104 can construct various environmental views from the environmental images transmitted by client device 102. The constructed environmental views (e.g., spatial or temporal) can allow remote experts to view and evaluate the environment or provide feedback and maintenance instructions to local operators. The combination of client device 102 and server 104 can support, for example, remote-based collaboration between local operators (physically located within an industrial environment) and remote maintenance experts. Through the various environmental construction features described herein, client device 102 and server 104 can provide this remote-based collaboration with improved efficiency and effectiveness.

[0017] This serves as an example implementation of any combination of the environment building features described herein. Figure 1 The computing system 100 shown includes an environment capture engine 110 (implemented by client device 102) and an environment view building engine 112 (implemented by server 104). The computing system 100 can implement engines 110 and 112 (including their components) in various ways, such as as hardware and programming. Programming for engines 110 and 112 can take the form of processor-executable instructions stored on a non-transient machine-readable storage medium, and the hardware for engines 110 and 112 can include processors that execute these instructions. The processor can take the form of a single-processor or multi-processor system, and in some examples, the computing system 100 uses the same computing system features or hardware components (e.g., a common processor or a common storage medium) to implement multiple engines.

[0018] In operation, the environment capture engine 110 can access a stream of image frames (image frames depicting the environment) captured by the client device 102, determine environment images that meet selection criteria from the stream of image frames, and transmit the environment images to the server 104. In operation, the environment view construction engine 112 can receive environment images from the client device 102, construct a spatial view of the environment based on the included location data along with the environment images, and navigate the spatial view, including by receiving a movement direction and moving from the current environment image depicted for the spatial view to the next environment image based on the movement direction.

[0019] These and other environment-building features are described in more detail below. Many of the examples described herein are provided in the context of industrial environments (e.g., factory environments, robot equipment repair, etc.). However, the invention is not limited thereto, and the environment-building techniques proposed herein can be consistently applied to any type of physical (or digital) environment.

[0020] Figure 2 An example is shown where an environmental image is determined from a stream of image frames captured by a client device. Figure 2 In the example shown, client device 102 implements environment capture engine 110. Client device 102 can be used by local personnel within environment 200, which can be any physical (or digital) environment or location, such as a factory floor, assembly line, machine shop, manufacturing plant, integrated circuit foundry, or any other industrial environment. Local operators can use client device 102 to capture images of environment 200, for example, to support the maintenance, repair, or servicing of machinery, robotic tools, or other components located within environment 200. In this regard, client device 102 can include a digital camera or any suitable digital imaging hardware to capture images of the environment. Image capture by client device 102 can be in the form of discrete digital images or a video stream containing video frames.

[0021] The environment capture engine 110 can selectively determine environmental images from image frames depicting the environment 200 captured by the client device 102, rather than indiscriminately sending every captured image as a live video stream to a remote technician or expert. An environmental image can refer to any image frame selected by the environment capture engine 110 for transmission to a remote server, and the environment capture engine 110 can selectively determine the environmental image from a set of candidate image frames via selection criteria. The selection criteria applied by the environment capture engine 110 can specify various conditions, thresholds, or parameters that the environment capture engine 110 can use to evaluate the image frames captured by the client device 102 in order to selectively determine the environmental image. In this respect, the environmental image determined by the environment capture engine 110 may include a subset of candidate image frames captured by the client device 102 and determined to satisfy the selection criteria.

[0022] In order to pass Figure 2 As an example, the environment capture engine 110 may apply selection criteria to image frames 210 (captured by client device 102) depicting environment 200. Image frames 210 may be any collection of images captured by client device 102, such as video streams of various portions of environment 200 captured by client device 102 via (e.g., initiated and used by a local operator) a digital camera of client device 102. Image frames 210 may thus be captured by client device 102 as video data and at a rate specified by the video technology implemented or used by client device 102 (e.g., 30 frames / second at a pre-configured video resolution). Instead of indiscriminately sending the entire video feed captured by client device 102 (e.g., all image frames 210), environment capture engine 110 may selectively determine some (but not all) of image frames 210 as environment images for transmission to a remote server for processing and environment visualization. Figure 2 In the specific example shown, the environment capture engine 110 selectively... Figure 2 Three (3) of the fifteen (15) illustrated images in image frame 210 are identified as environmental images to be transmitted to a remote server for processing.

[0023] By transmitting selectively determined environmental images from image frame 210 (and determining not to transmit image frames that do not meet the selection criteria), the environment capture engine 110 can improve the efficiency and effectiveness of constructing an environmental view of environment 200 for evaluation, analysis, and review. The determination of environmental images based on selection criteria and the selective transmission of only determined environmental images can reduce network consumption (especially when video streaming may consume significant network bandwidth for transmission, particularly for high-definition video). Furthermore, based on the specific selection criteria applied to determine the environmental images, the evaluation of image frame 210 can be performed to identify and select specific image frames with reduced blur and content redundancy relative to other transmitted environmental frames. Various examples of selection criteria that the environment capture engine 110 can apply are described below.

[0024] As an example of selection criteria, the environment capture engine 110 may apply frequency criteria when determining an environment image from an image frame captured by a client device. Frequency criteria can set any conditions or thresholds that control the rate at which the environment capture engine 110 determines (and / or transmits) the environment image. For example, frequency criteria can restrict the determination and transmission of the environment image to a threshold frequency, and thus prevent this from happening if selecting an environment image from image frame 210 would violate a specified threshold frequency (e.g., doing so would result in transmitting the environment image at a higher / faster rate than the threshold frequency). Such frequency criteria can be implemented by the environment capture engine 110 in various ways.

[0025] For example, the environment capture engine 110 may use a timer-based implementation of a frequency criterion. The environment capture engine 110 may set a timer based on a threshold frequency, and in this case, the frequency criterion is satisfied when it is determined that a time value set for the timer has elapsed since the last environmental image was determined (or transmitted). To provide an illustrative example, the frequency criterion may be set to a threshold frequency of 5 frames per second, which sets a threshold or upper limit for the environment capture engine 110 to selectively determine and transmit environmental images. In this example, the environment capture engine 110 may set a timer with a timer value of 200 milliseconds (ms), which the environment capture engine 110 may calculate or set based on the 5 frames per second frequency threshold. When an environmental image is determined (i.e., the image frame is identified by the frequency criterion and any other selection criteria), the environment capture engine 110 may reset the timer. The environment capture engine 110 may determine that any candidate image frames subsequently accessed, considered, or acquired by the environment capture engine 110 do not satisfy the frequency criterion until the timer reaches the 200 ms timer value. This can happen when any subsequently considered candidate image frames that are considered before the 200ms timer has elapsed would violate the frequency threshold set by the frequency criteria. In response to determining that the current candidate image frame has been accessed or acquired after the 200ms timer value has elapsed, the environment capture engine 110 may determine that the frequency criteria are met and further apply other selection criteria when determining the environment image from image frame 210.

[0026] As another implementation example, the environment capture engine 110 may perform a frequency criterion based on the number of image frames acquired or considered since the last environment image was determined or transmitted. Therefore, instead of determining whether a frequency threshold is met based on a timer, the environment capture engine 110 may apply a counter-based approach, where the required counter value for the acquired image frames to be considered is based on a frequency threshold (e.g., a counter value of six (6) for a video stream of image frames 210 captured at 30 frames per second, and a threshold frequency set for the frequency criterion is 5 frames per second). In such an example, the environment capture engine 110 may increment the counter for each acquired candidate image frame and determine that the frequency criterion is met when the counter value reaches or exceeds a threshold counter value calculated based on the threshold frequency (e.g., >6). The environment capture engine 110 may reset the counter when an environment image that meets the selection criteria is determined.

[0027] While some implementation examples of frequency criteria are provided, the environment capture engine 110 can implement any suitable mechanism to control (e.g., limit) the selective determination of the frequency or rate of environment frames from image frames 210 captured by the client device 102. Note that in some implementations, the frequency criteria applied by the environment capture engine 110 may set an upper limit on the rate at which the environment image is determined. In such cases, the environment capture engine 110 may determine and transmit the environment image at a rate below a frequency threshold (e.g., below a frequency threshold set to 5 frames / second). This can be the case when the environment capture engine 110 can apply other selection criteria when determining the environment image (even if the frequency criteria are met). Therefore, instead of enforcing a rule-based or fixed transmission rate for the determined environment image, the environment capture engine 110 can flexibly determine the environment image from image frames 210 that fully satisfy the selection criteria, which can result in the construction of an environment view with improved efficiency and accuracy.

[0028] As another example of selection criteria, the environment capture engine 110 may apply a fuzzing criterion when determining an environment image from image frames captured by a client device. The fuzzing criterion can be any condition, threshold, or evaluation form satisfied based on the image quality of the image frame. Therefore, the environment capture engine 110 may perform any number of image processing algorithms on candidate image frames to measure the fuzzing effects present in the candidate image frames and evaluate the fuzzing criterion. As an illustrative example, the environment capture engine 110 may perform any number of Laplacian image processing techniques to measure, detect, or identify sharp edges in the candidate image. The fuzzing criterion may specify a threshold number of sharp edges, a percentage of sharp features in the candidate image frame, or any other suitable condition that the candidate image frame must meet to pass the fuzzing criterion. In some implementations, the environment capture engine 110 evaluates candidate image frames by performing image processing on a selected subset of the candidate image frames (e.g., processing selected pixels in the candidate image frame instead of all pixels in the candidate image frame). Doing so can reduce computational latency and power strain on the client device 102, and thus improve the computational efficiency of the client device 102.

[0029] In some implementations, the ambient capture engine 110 may evaluate blur criteria without performing image processing on image frame 210. Instead, the ambient capture engine 110 may characterize the image quality of image frame 210 while evaluating blur criteria. As an example, the ambient capture engine 110 may access gyroscope or accelerometer data applicable to client device 102 for image frame 210. The gyroscope or accelerometer may specify motion data measured by client device 102 (and underlying camera or other imaging device) during the capture of image frame 210. Blur criteria may evaluate such accelerometer data as an indirect measurement of image quality. For example, the blur criteria may specify that the acceleration of client device 102 is less than a threshold acceleration or threshold velocity during the capture of candidate image frames of image frame 210. Such a threshold acceleration may be set relative to the capture frequency of the video stream, which may reflect the image capture rate of the digital camera of client device 102.

[0030] As an illustrative example, the digital camera of client device 102 can capture image frames 210 of a video stream at 30 frames per second, which translates to capturing one frame every 33.33 ms. For accelerometer or gyroscope data indicating that client device 102 is moving or accelerating at a rate exceeding a threshold rate relative to the 33.33 ms image capture rate of client device 102, ambient capture engine 110 can indirectly determine that the captured image frame is blurry (e.g., due to latency of the camera sensor). Therefore, the blur criteria applied by ambient capture engine 110 can specify a blur threshold based on accelerometer, gyroscope, or any other motion-based data to indirectly determine and evaluate the image quality of image frame 210 based on the blur criteria.

[0031] As another example of selection criteria, the environment capture engine 110 may apply a difference criterion, which specifies that the content of a given environment image must differ from the content of a previously determined environment image by a threshold degree. The environment capture engine 110 may implement and evaluate the difference criterion in various ways. In some implementations, the environment capture engine 110 may evaluate the difference criterion via location data or orientation data from the client device 102. The environment capture engine 110 may extract location data and orientation data for candidate image frames and previously determined environment images. If the difference between at least one of the location data or orientation data exceeds a predetermined threshold, the environment capture engine 110 may determine that the difference criterion is met.

[0032] As an illustrative example, the difference criterion could require a difference in positioning data caused by a movement of at least 30 centimeters (cm) by the client device 102, or a difference in orientation data caused by a movement of at least 15° by the client device 102. Such a difference in positioning or orientation can provide an indirect determination that the image content of the candidate image frame and the previously determined environmental image depict sufficiently different portions of the environment 200 (but allow for at least some overlap via the difference criterion). The environment capture engine 110 can apply any other suitable motion data or comparison as a proxy for the content differences between the candidate image frame and the previously determined environmental image.

[0033] As another implementation example, the environment capture engine 110 can perform any number of image processing techniques to evaluate the difference criteria. In some cases, the environment capture engine 110 can use motion flow image computation to determine the overlap between candidate image frames of image frame 210 and previously determined environment images. When the overlap determined by such image processing exceeds an overlap threshold (e.g., >85% overlap or any other configurable threshold), the environment capture engine 110 can determine that the candidate image frame does not meet the difference criteria. When the determined overlap does not exceed the overlap threshold, the environment capture engine 110 can determine that the candidate image frames in image frame 210 meet the difference criteria.

[0034] While some examples of selection criteria are presented herein via frequency, ambiguity, and difference criteria, the environment capture engine 110 can apply any number of additional or alternative selection criteria when selectively determining and transmitting environmental images. Through such selective determination, the environment capture engine 110 can support rate-controlled selection of content-relevant images from the image stream captured by the client device 102. In doing so, the environment capture engine 110 can transmit a selected subset of image frames captured by the client device 102 in a rate-controlled manner that does not overwhelm network bandwidth, conserves computing power, and reduces battery strain on the client device 102. Therefore, the selective determination and transmission of environmental images by the environment capture engine 110 can improve the efficiency and effectiveness of constructing environmental views in remote collaboration.

[0035] In some implementations, the environment capture engine 110 can adaptively adjust any number of thresholds set in the selection criteria. Such adaptive adjustment can be based on the network characteristics or attributes of the client device 102. For example, the environment capture engine 110 can maintain different selection criteria (or underlying thresholds) based on the power level of the client device 102. In full power (90-100% battery capacity remaining), low power (less than 15% battery capacity remaining), or any other configurable state, the environment capture engine 110 can apply different thresholds in the selection criteria. In the low power state of the client device 102, the environment capture engine 110 can select and transmit environmental images at a lower frequency (e.g., 1 frame / second) compared to other power states. The environment capture engine 110 can make similar adjustments and adjustments based on network availability (e.g., based on network upload speed, congestion measurements, or any other network metric).

[0036] As another implementation example, the environment capture engine 110 or environment view construction engine 112 may initiate an override period during which the environment capture engine 110 may pause some or all of the selection criteria. For example, the override period may be a burst of time (e.g., 10 seconds) determined by the environment capture engine 110 to transmit the entire video stream of environment 200 captured by the client device 102. Alternatively, the environment capture engine 110 may pause only the frequency criteria during the override period, but continue to apply blur and content criteria when selectively determining the environment image from image frames 210. In any of the ways described herein, the environment construction techniques provided herein can support the selective determination of the environment image from an image stream captured by a client device.

[0037] Any example adjustments to the selection criteria may be additionally or alternatively triggered or initiated by a remote server. For example, the environment view building engine 112 of server 104 may specify any adjustments to the selection criteria as described herein, for example, in response to a request initiated by a remote technician. The environment view building engine 110 may implement adjustments to the selection criteria by transmitting a selection criterion adjustment message to client device 102, which may take the form of any communication format supported between client device 102 and server 104. The selection criterion adjustment message may cause the environment capture engine 110 to temporarily adjust the selection criteria used to determine the environment image or to completely suspend the selection criteria. In some implementations, the environment capture engine 110 may reject selection criterion adjustment requests from the environment capture engine 110, for example, when releasing or suspending the selection criteria would deplete battery reserves or consume network bandwidth exceeding a tolerable threshold.

[0038] Figure 3An example of logic 300 that a computing system can implement to determine an environmental image from an image stream via selection criteria is shown. For example, computing system 100 may implement logic 300 as hardware, executable instructions stored on a machine-readable medium, or a combination of both. Client device 102 of computing system 100 may implement logic 300 via an environment capture engine 110, through which computing system 100 may execute or implement logic 300 as a method supporting the determination and transmission of environmental images. The following description of logic 300 is provided using environment capture engine 110 as an example. However, various other implementation options for the system are possible.

[0039] The environment capture engine 110 can evaluate a set of image frames depicting the environment captured by the client device. The evaluation of image frames can occur frame-by-frame. Therefore, in implementing logic 300, the environment capture engine 110 can access the next image frame (302) from the image stream captured by the client device. As described herein, the current image frame accessed by the environment capture engine 110 for evaluation may be referred to as a candidate image frame. The rate at which the environment capture engine 110 acquires image frames for evaluation can be configurable. In some cases, the environment capture engine 110 can acquire and evaluate individual image frames in the image stream captured by the client device. In other cases, the environment capture engine 110 can acquire candidate image frames from the image stream at a rate lower than the capture rate of the image stream, discarding image frames not acquired in the image stream.

[0040] For candidate image frames, the environment capture engine 110 may apply selection criteria to evaluate the candidate image frames and determine the environment image. The environment capture engine 110 may apply a frequency criterion (304) that specifies that the environment image is determined or transmitted at a rate below a threshold frequency. As another selection criterion, the environment capture engine 110 may apply a fuzziness criterion (306) that specifies a threshold image quality that the determined environment image must meet. As yet another selection criterion, the environment capture engine 110 may apply a difference criterion (308) that specifies that the content of a given environment image must differ from the content of a previously determined environment image by a threshold degree. The environment capture engine 110 may apply the frequency criterion, fuzziness criterion, and difference criterion in any of the ways described herein.

[0041] In response to determining that a candidate image frame does not meet a frequency criterion, ambiguity criterion, difference criterion, or any other application selection criterion, the environment capture engine 110 may determine that the candidate image frame does not meet the selection criterion and obtain the next image frame from the image stream (302). In response to determining that a candidate image frame meets the selection criterion, the environment capture engine 110 may identify the candidate image frame as an environment frame. In response to such determination, the environment capture engine 110 may reset a timer, counter, or any other logic element used to measure or evaluate the frequency criterion. In some embodiments, the environment capture engine 110 may cache the determined environment image (310), and the cached environment image may be used to compare the image content with other candidate image frames for evaluating the difference criterion. The environment capture engine 110 may also transmit the environment image to a remote server (312) for processing and environment view construction. In some embodiments, the environment capture engine 110 may compress the environment image before transmission to reduce network strain.

[0042] Figure 3 The illustrated logic 300 provides an illustrative example of how client device 102 can support the determination and transmission of environmental images. Additional or alternative steps in logic 300 are contemplated herein, including any of the various features described herein for client device 102, environmental capture engine 110, server 104, environmental view construction engine 112, and any combination thereof.

[0043] In any of the ways described herein, a client device may selectively transmit some, rather than all, of the captured image stream depicting the environment. As described herein, the selectively determined transmitted images may be referred to as environmental images. A remote server may process the environmental images to construct a view of the environment, which, for example, a remote maintenance expert may use to assess the environment and provide any relevant feedback or instructions. The construction of an environmental view from the transmitted environmental images is described below.

[0044] Figure 4 An example of constructing an environment view from an environment image transmitted by a client device is shown. Figure 4 In this example, server 104 receives an environmental image 410 transmitted by a client device (e.g., client device 102). The environmental image 410 may depict an environment and may be selectively determined by an environment capture engine 110 in any of the manner described herein. Server 104 may implement or include an environment view building engine 112 configured to build an environment view from the environmental image 410. Environment view building can facilitate collaboration between remote experts or maintenance personnel and local operators using the client device transmitting the environmental image 410.

[0045] Environment view construction engine 112 can construct an environment view of the environment from environment image 410. As an example, environment view construction engine 112 can construct a spatial view of the environment from environment image 410. Location data and orientation data may accompany the environment image 410 received by environment view construction engine 112, which can utilize this data to construct the spatial view. Environment views (including spatial views) can take the form of any data structure or set of environment images that form a visual depiction of the environment (or a portion thereof). Using the location data and orientation data of the individual environment images in environment image 410, environment view construction engine 112 can position the individual environment images 410 in their corresponding locations to form a spatial view of the environment. In doing so, environment view construction engine 112 can construct a spatial view where the individual environment images can form specific tiles or portions of the environment view. Since some of these environment images 410 may overlap in location, environment view construction engine 112 can construct a spatial view of the environment where a given location or position within the environment view can be visualized by multiple different environment images.

[0046] An example of a spatial view that Environment View Building Engine 112 can build is shown in Figure 4 The environment is shown as spatial view 420. Spatial view 420 may include multiple different environmental images of environmental image 410, which together form a visualization of a specific environment. Since the client device can transmit environmental images (with different pose and orientation data) individually, environment view building engine 112 can construct spatial view 420 of the environment, allowing a user (e.g., a remote expert) to view the environment from different angles and positions. In some embodiments, environment view building engine 112 can visualize spatial view 420 of the environment as individual environmental images. In some embodiments, environment view building engine 112 does not need to stitch together the environmental images forming spatial view 420, because stitching images from different environmental images captured at different times, in different poses and orientations, may be impractical, inaccurate, or impossible. Instead, environment view building engine 112 can visualize the environment of the constructed spatial view 420 image by image. The current environmental image visualized for spatial view 420 may represent the current location, orientation, or position of spatial view 420. Navigation to other locations within the spatial view 420 can be supported by switching to different environmental images that form the spatial view 420.

[0047] The environment view construction engine 112 can support navigation of a spatial view 420 visualized for a user (e.g., a remote expert) in various ways. In some cases, the environment view construction engine 112 can support navigation of the spatial view 420 via user commands, which may take the form of a movement direction. The movement direction can instruct the user to specify any direction of movement within the spatial view 420 (e.g., via arrow keys or any other method supporting user input). The environment view construction engine 112 can then navigate the spatial view 420 by receiving the movement direction (e.g., via user input) and moving from the current environment image depicted for the spatial view 420 to the next environment image based on the movement direction. When moving from the current environment image to the next environment, the environment view construction engine 112 can determine the next environment image in the environment images forming the spatial view 420 as the specific environment image that is geographically closest to the current environment image along the movement direction.

[0048] In any such manner, the environment view construction engine 112 can support visualization of the environment via spatial view 420 and navigation via spatial view 420 through user commands. Since spatial view 420 is constructed from environment images 410 provided by a client device, a user (e.g., a remote technician) can navigate spatial view 420 independently of their current location or the use of the client device, which captures, selectively identifies, and transmits the environment images 410 used to form spatial view 420. Furthermore, client device-independent navigation of the spatial view 420 means that it is not necessary to instruct a local operator to position or orient the client device in a specific location for remote user viewing. Instead, the remote user can navigate to a specific portion of the environment within the spatial view 420 constructed for the environment by navigating to a specific environment image depicting that particular portion.

[0049] In addition to or as an alternative to a spatial view, the environment view construction engine 112 can construct a temporal view of the environment. For example, the environment view construction engine 112 can classify environment images 410 into different temporal sets based on the timestamp values ​​of each environment image. As an example, such temporal sets can be classified by the environment view construction engine 112 according to time range intervals (e.g., non-overlapping 10-second ranges or any other configurable time ranges), and such classification can be based on the timestamp values ​​of the environment images 410. The environment view construction engine 112 can construct different spatial views for each temporal set, based on the specific environment images classified into the respective temporal sets. Figure 4 In the example shown, the environment view building engine 112 builds temporal sets 421, 422, and 423, each of which may include a different subset of the environment image 410 transmitted from the client device.

[0050] The environment view construction engine 112 supports temporal navigation of the environment through temporal sets 421, 422, and 423. For example, the environment view construction engine 112 can support visualization of a specific location, pose, orientation, orientation, or portion of the environment across different time periods associated with temporal sets 421, 422, and 423 (and more). Because the environment view construction engine 112 can visualize the environment by depicting one environment image at a time, navigation along the depicted environment view can be supported. Along the timeline, the environment view construction engine 112 navigates backward or forward at specific locations in the environment by traversing the temporal sets and accessing relevant environment images for a given temporal set at a specific location.

[0051] Through any of the methods described herein, the environment view construction engine 112 can support the construction of environment views from received environment images 410, including multiple spatial views classified into different temporal sets. The environment construction techniques proposed herein also support navigation of the constructed environment views both spatially and temporally. With such construction and navigation features, the environment construction techniques described herein can provide a robust, efficient, and flexible mechanism through which users far removed from the environment can still view and evaluate the underlying components of the environment.

[0052] Figure 5 An example of logic 500 that a computing system can implement to construct an environment view from a transmitted environment image is shown. For example, computing system 100 may implement logic 500 as hardware, executable instructions stored on a machine-readable medium, or a combination of both. Server 104 of computing system 100 may implement logic 500 via environment view construction engine 112, through which computing system 100 may execute or implement logic 500 as a method supporting environment view construction from a received environment image. The following description of logic 500 is provided using environment view construction engine 112 as an example. However, various other implementation options for the system are possible.

[0053] When implementing logic 500, the environment view building engine 112 can receive an environment image (502) from the client device and build a spatial view of the environment (504) based on the included location data together with the environment image. When implementing logic 500, the environment view building engine 112 can also navigate the spatial view (506), including by receiving a movement direction (508) and moving from the current environment image depicted for the spatial view to the next environment image (510) based on the movement direction.

[0054] Figure 5The illustrated logic 500 provides an illustrative example of how server 104, according to the present invention, can support the construction of an environment view. Additional or alternative steps in logic 500 are contemplated herein, including any of the various features described herein for client device 102, environment capture engine 110, server 104, environment view construction engine 112, and any combination thereof.

[0055] Figure 6 An example of a computing system 600 supporting the construction of an environmental view from selectively determined environmental images is shown. The computing system 600 can be a distributed system comprising different components located in different geographical locations. For example, the computing system 600 can include client device 102 and server 104, which can be separate physical and logical entities within the computing system 600, each comprising individual hardware (e.g., processors and memory). Client device 602 and server 604 can share any characteristics and features as described herein with respect to client device 102 and server 104.

[0056] Client device 602 may include processor 610, and server 604 may include processor 612, either or both of which may take the form of a single or multiple processors. Processor 610 and processor 612 may respectively include a central processing unit (CPU), a microprocessor, or any hardware device adapted to execute instructions stored on a machine-readable medium. Client device 602 and server 604 of computing system 600 may each include machine-readable media, such as machine-readable medium 620 for client device 602 and machine-readable medium 622 for server 604. Machine-readable media 620 and 622 may take the form of any non-transient electronic, magnetic, optical, or other physical storage device storing executable instructions, such as environment capture instructions 630 stored on client device 602 and instructions stored on... Figure 6 The environment view construction instruction 632 in server 604. Therefore, machine-readable media 620 and 622 can be, for example, random access memory (RAM) (e.g., dynamic RAM (DRAM)), flash memory, spin torque memory, electrically erasable programmable read-only memory (EEPROM), storage drives, optical discs, etc.

[0057] The computing system 600 can execute instructions stored on machine-readable media 620 and 622 via processors 610 and 612. Execution of instructions (e.g., environment capture instruction 630 and / or environment view construction instruction 632) can cause the computing system 600 to perform any of the environment construction features described herein, including any feature based on environment capture engine 110, environment view construction engine 112, or a combination of both.

[0058] For example, the environment capture instruction 630 executed by the processor 610 can cause the client device 602 to access a stream of image frames (image frames depicting the environment) captured by the client device 602, determine an environment image that meets the selection criteria from the stream of image frames, and transmit the environment image to the server 604. The environment view construction instruction 632 executed by the processor 612 of the server 604 can cause the server 604 to receive the environment image from the client device 602, construct a spatial view of the environment based on the included location data together with the environment image, and navigate the spatial view, including by receiving a movement direction and moving from the current environment image depicted for the spatial view to the next environment image based on the movement direction.

[0059] Any additional or alternative environment building features as described herein may be implemented via environment capture instruction 630, environment view building instruction 632, or a combination of both.

[0060] The aforementioned systems, methods, devices, and logic, including client devices, servers, environment capture engine 110, and environment view building engine 112, can be implemented in many different ways using many different combinations of hardware, logic, circuitry, and executable instructions stored on a machine-readable medium. For example, environment capture engine 110, environment view building engine 112, or combinations thereof may include circuitry in a controller, microprocessor, or application-specific integrated circuit (ASIC), or may be implemented using discrete logic or components or combinations of other types of analog or digital circuitry combined on a single integrated circuit or distributed among multiple integrated circuits. For example, a computer program product may include a storage medium and machine-readable instructions stored on the medium that, when executed in a terminal, computer system, or other device, cause the device to perform operations according to any of the above descriptions (including any features of environment capture engine 110, environment view building engine 112, or combinations thereof).

[0061] The processing power of the systems, devices, and engines described herein (including client devices, servers, environment capture engine 110, and environment view building engine 112) can be distributed across multiple system components, such as across multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements. Parameters, databases, and other data structures can be stored and managed separately, can be combined into a single memory or database, can be logically and physically organized in many different ways, and can be implemented in many ways, including data structures such as linked lists, hash tables, or implicit storage mechanisms. Programs can be parts of a single program (e.g., subroutines), standalone programs, distributed across several memories and processors, or implemented in many different ways, such as in libraries (e.g., shared libraries).

[0062] While various examples have been described above, many more implementations are possible.

Claims

1. A method comprising: Through the calculation system: An environmental image (410) is received (502) from a client device (102, 602), wherein the environmental image (410) depicts an environment (200) and is selectively transmitted from the client device (102, 602) based on selection criteria applied by the client device (102, 602), the selection criteria including: Frequency criteria, which specify that the environmental image is transmitted at a rate below a threshold frequency (410); Fuzzy criteria, which specify a threshold image quality that the environmental image (410) must meet; and The difference criterion specifies that the content of a given environment image must differ from the content of a previously determined environment image by a threshold degree. Each of the environmental images (410) satisfies each of the selection criteria; Based on the included location data and the environmental image (410), a spatial view (420) of the environment (200) is constructed (504); and Navigation (506) of the spatial view (420), including via: Receive (508) the direction of movement; and Based on the direction of movement, move (510) from the current environment image depicted for the spatial view (420) to the next environment image.

2. The method according to claim 1, further comprising: The environmental images (410) are classified into different temporal sets (421, 422, 423) based on the timestamp values ​​of each environmental image (410).

3. The method according to claim 1, wherein, Moving from the current environment image (510) to the next environment image includes: determining the next environment image that is closest in position to the current environment image along the direction of movement from the environment images (410) in the spatial view (420).

4. The method according to claim 1, comprising: Construct a spatial view (420) of the environment (200) without stitching together any of the environment images (410).

5. The method according to claim 1, wherein, The environmental image (410) is selectively transmitted by the client devices (102, 602) based on the selection criteria applied to video frames of the video stream captured by the client devices (102, 602).

6. A system (100) comprising: Client devices (102, 602) include an environment capture engine (110), the environment capture engine being configured to: Access a stream of image frames (210) captured by the client devices (102, 602), the image frames (210) depicting the environment (200); An environmental image (410) satisfying selection criteria is determined from the stream of the image frames (210), wherein the selection criteria include: Frequency criteria, which specify that the environmental image is transmitted at a rate below a threshold frequency (410); Fuzzy criteria, which specify a threshold image quality that the environmental image (410) must meet; and The difference criterion specifies that the content of a given environment image must differ from the content of a previously determined environment image by a certain threshold. Each of the environmental images (410) satisfies each of the selection criteria; and Transmit the environmental image (410); and Servers (104, 604) include an environment view engine (112) configured to: Receive the environmental image (410) from the client devices (102, 602); A spatial view (420) of the environment (200) is constructed based on the included location data and the environment image (410); and Navigating the spatial view (420) includes: Receive the direction of movement; and Based on the direction of movement, the user moves from the current environment image depicted for the spatial view (420) to the next environment image.

7. The system according to claim 6, wherein, The environment view engine (112) is also configured to classify the environment images (410) into different temporal sets (421, 422, 423) based on the timestamp values ​​of each environment image (410).

8. The system according to claim 6, wherein, The environment view engine (112) is configured to move from the current environment image to the next environment image by determining the next environment image that is closest in position to the current environment image along the direction of movement from the environment image (410) in the spatial view (420).

9. The system according to claim 6, wherein, The environment view engine (112) is configured to construct a spatial view (420) of the environment (200) without stitching together any of the environment images (410).

10. A non-transient machine-readable medium (622) comprising instructions (632) that, when executed by a processor (612), cause a computing system to perform the method according to any one of claims 1 to 5.

Images