Optimizing image capture to train machine learning models for implicit representation

Abstract

Capture information is caused to be sent to a user computing device, where the capture information includes instructions for a user of the user computing device to capture a plurality of image sets of points of interest (POI). A plurality of image sets of the POI are received, wherein the plurality of image sets includes a calibration image set, a POI depiction image set, and a path image set. A machine-learned POI representation model is trained with the plurality of image sets of the POI, wherein the machine-learned POI representation model is trained to generate a representation of the POI. Video data is generated using a machine-learned POI representation model that depicts a path of a virtual camera traversing within the POI from a starting position to an ending position.

Description

[0001] Priority declaration

[0002] This application claims the benefit of and priority to U.S. Patent Application No. 18 / 392,908, filed on December 21, 2023, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to the collection and utilization of training data for machine learning models. More specifically, the present disclosure relates to optimizing image capture to train models to implicitly represent three-dimensional space. Background Art

[0004] Recently, machine learning models have been increasingly used for various generative tasks. In particular, models such as the Neural Radiance Field (NeRF) model have been introduced for implicitly representing a three-dimensional space based on a two-dimensional image depicting the space. For example, once trained on an image depicting a particular three-dimensional space, a NeRF model can generate radiance information for a particular point in the three-dimensional space given the spatial coordinates corresponding to that point. In this way, the NeRF model can be used to render images depicting novel views of the three-dimensional space.

[0005] Implicit representation models such as the NeRF model enable novel views of three-dimensional space to be rendered much more efficiently than conventional techniques, which typically require texture artists, animators, modelers, etc. However, conventional implicit representation models typically must be trained with very accurately captured images, and capturing such images is typically beyond the skill set of the average user. This barrier, in turn, prevents average users from taking advantage of the benefits of implicit representation models. Summary of the invention

[0006] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0007] An example aspect of the present disclosure relates to a computer-implemented method. The method includes causing, by a computing system including one or more computing devices, capture information to be sent to a user computing device, wherein the capture information includes instructions for capturing a plurality of image sets of a point of interest (POI) to a user of the user computing device. The method includes receiving, by the computing system, a plurality of image sets of a POI from a user computing device, wherein the plurality of image sets include: a calibration image set, the calibration image set including a plurality of calibration image pairs, each calibration image pair in the calibration image pairs including two calibration images captured at two different locations separated by a predefined distance within the POI; a POI depiction image set, the POI depiction image set including a plurality of depiction images, the plurality of depiction images depicting the POI from a corresponding plurality of angles and / or locations; and a path image set, the path image set including a plurality of path images, the plurality of path images collectively depicting a path within the POI from a starting location to an ending location. The method includes training, by the computing system, a machine-learned POI representation model using a plurality of image sets of the POI, wherein the machine-learned POI representation model is trained to generate a representation of the POI. The method includes generating, by a computing system, video data using a machine-learned POI representation model that depicts a virtual camera traversing a path within the POI from a starting location to an ending location.

[0008] Another example aspect of the present disclosure relates to a user computing device. The user computing device includes one or more processor devices. The user computing device includes one or more tangible non-transitory computer-readable media storing computer-readable instructions that, when executed by the one or more processor devices, cause the one or more processor devices to perform operations. The operations include obtaining capture information from a computing system, the capture information including instructions for a user of the user computing device to capture multiple sets of images of a POI. The operations include obtaining multiple sets of images of a POI in response to obtaining the capture information, wherein the multiple sets of images are captured by the user via an image capture device, and wherein the multiple sets of images include: a calibration image set, the calibration image set including multiple calibration image pairs, each calibration image pair in the calibration image pairs including two calibration images captured at two different locations separated by a predefined distance within the POI; a POI depiction image set, the POI depiction image set including multiple depiction images, the multiple depiction images depicting the POI from corresponding multiple angles and / or locations; and a path image set, the path image set including multiple path images, the multiple path images collectively depicting a path from a starting location to an ending location within the POI. The operations include performing a quality verification process for at least some of the plurality of image sets. The operations include transmitting capture information to a computing system in response to performing the quality verification process, wherein the capture information includes the plurality of image sets of the POI. The operations include obtaining, from the computing system, video data depicting a virtual camera traversing a path within the POI from a starting location to an ending location, wherein the video data is generated by the computing system using a machine-learned POI representation model that is trained to generate a representation of the POI based on the plurality of image sets.

[0009] Another example aspect of the present disclosure relates to one or more tangible non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processor devices, cause the one or more processor devices to perform operations. The operations include causing capture information to be sent to a user computing device, wherein the capture information includes instructions for capturing a plurality of image sets of a POI to a user of the user computing device. The operations include receiving a plurality of image sets of a POI from the user computing device, wherein the plurality of image sets include: a calibration image set including a plurality of calibration image pairs, each of the calibration image pairs including two calibration images captured at two different locations within the POI separated by a predefined distance; a POI depiction image set including a plurality of depiction images, the plurality of depiction images depicting the POI from a corresponding plurality of angles and / or locations; and a path image set including a plurality of path images, the plurality of path images collectively depicting a path within the POI from a starting location to an ending location. The operations include training a machine-learned POI representation model using the plurality of image sets of the POI, wherein the machine-learned POI representation model is trained to generate a representation of the POI. The operation includes using a machine-learned POI representation model to generate video data depicting a virtual camera traversing a path within the POI from a starting location to an ending location.

[0010] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0013] Figure 1 is an overview data flow diagram for optimizing image capture to train a machine learning model for implicit representation according to some implementations of the present disclosure.

[0014] Figure 2 Depicted are example capture instructions for capturing a set of calibration images in accordance with some implementations of the present disclosure.

[0015] Figure 3A Depicted are example capture instructions for capturing a front-facing depicted image of a point of interest (POI) depicted image set, according to some implementations of the present disclosure.

[0016] Figure 3B Depicted are example capture instructions for capturing a rear-facing depicted image of a set of POI depicted images, according to some implementations of the present disclosure.

[0017] Figure 3C Depicted are example capture instructions for capturing an inward-facing depiction image of a set of POI depiction images, according to some implementations of the present disclosure.

[0018] Figure 4 Depicted are example capture instructions for capturing a pathway image of a pathway image set, according to some implementations of the present disclosure.

[0019] Figure 5 Depicted are example capture instructions for executing an image capture mode when capturing images of various sets of images in accordance with some implementations of the present disclosure.

[0020] Figure 6 A flow chart depicting an example method for optimizing image capture to train a machine-learned POI representation model in accordance with some implementations of the present disclosure is depicted.

[0021] Figure 7 A flow chart depicting an example method for local optimization of image capture to train a machine-learned POI representation model, according to some implementations of the present disclosure.

[0022] Figure 8 Depicted is a block diagram of an example computing environment that performs image capture optimization to train a machine-learned POI representation model according to an example implementation of the present disclosure.

[0023] Reference numerals that are repeated across multiple drawings are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0024] The present disclosure generally relates to the collection and utilization of training data for machine learning models. More specifically, the present disclosure relates to optimizing image capture to train models to implicitly represent three-dimensional space. Implicit representation models enable novel views of three-dimensional space to be rendered much more efficiently than conventional techniques, which typically require texture artists, animators, modelers, etc. However, conventional implicit representation models typically must be trained with very accurately captured images, and capturing such images is typically beyond the skill set of the average user. This obstacle, in turn, prevents the average user from taking advantage of the benefits of implicit representation models.

[0025] Therefore, the implementation of the present disclosure proposes to optimize image capture to train an implicit representation model for machine learning. More specifically, a computing system (e.g., a server system, a cloud system, etc.) may receive a request from a user computing device to generate a video rendering of a specific point of interest (POI). In response, the computing system may provide capture instructions to the user computing device. The capture instructions may instruct the user how to capture multiple sets of images to generate an implicit representation of the POI.

[0026] The image set may include a calibration image set, a POI depiction image set, and a path image set. The instructions for capturing the calibration image set may instruct the user to capture a pair of images at locations separated by a predefined distance. For example, the instructions for the calibration image set may instruct the user to capture an image from a first location within the POI (e.g., a location within a room) while viewing a second location within the POI that is located a predefined distance (e.g., six feet) from the first location. The instructions may then instruct the user to capture a second image from the second location while viewing the first location. In this manner, the calibration image set may provide a reference frame for the spatial dimensions of the POI.

[0027] The instructions for capturing a set of POI depicting images may instruct the user to capture a plurality of depicting images depicting the POI from a variety of different angles and positions. For example, if the POI is a room, the instructions may instruct the user to collect a first set of images while walking in a forward direction around the periphery of the room. The instructions may further instruct the user to collect a second set of images while walking in a backward direction around the periphery of the room. The instructions may further instruct the user to collect a third set of images while walking in an inward direction (i.e., toward the center of the room) around the periphery of the room. In this manner, the instructions provided to the user may ensure that the set of POI depicting images depicts most or all of the surface of the POI.

[0028] For a path image set, the instructions for capturing the path image set may instruct the user to capture a series of images in sequence while following a path that the user wishes to depict. For example, if the user wishes to depict a straight path from a first point to a second point within a POI, the user may move along the path while capturing images in sequence (e.g., every three feet, etc.). For another example, the user may depict a circular path by navigating around the perimeter of a room in a clockwise or counterclockwise manner. In this manner, the path image set may be utilized to derive a path for a virtual camera within a three-dimensional representation of a POI.

[0029] The computing system may train a machine-learned representation model based on multiple image sets to represent the POI. In particular, the machine-learned representation model may be trained to generate a three-dimensional representation of the POI. For example, assume that the machine-learned representation model is a neural radiance field (NeRF) model. The computing system may first determine the spatial dimensions of the POI using a calibration image set. The computing system may then position the POI depiction images in space and train the NeRF model to generate a novel view of the three-dimensional representation of the POI given a certain viewing angle.

[0030] Once trained, the computing system can use the machine-learned representation model to generate video data that depicts a virtual camera traveling along a path indicated by a path image set. For example, a path image set may include multiple images captured sequentially as a user traverses a particular path through a POI from a starting location to an ending location. The computing system can generate a virtual path for the virtual camera corresponding to the path indicated by the path image set. The computing system can then render video data depicting the POI from the perspective of the virtual camera as the virtual camera traverses the path. In this way, the implementations described herein can enable users without specialized skill sets to efficiently generate virtual tours and similar representations of POIs (e.g., venues, businesses, entertainment areas, etc.).

[0031] Various aspects of the present disclosure provide a variety of technical effects and benefits. As an example technical effect and benefit, the implementation described herein can significantly reduce the cost of computing resources associated with exchanging and utilizing large sets of images to train machine-learned representation models. More specifically, by providing instructions for capturing specific types of image sets, the implementation described herein can significantly reduce the total number of images required to train machine-learned representation models. In turn, the reduced number of training images requires less bandwidth resources to transfer images to the training system, and less computing resources are required to train machine-learned representation models (e.g., power, memory, computing cycles, storage, network resources, etc.).

[0032] Referring now to the drawings, example implementations of the present disclosure will be discussed in further detail.

[0033] Figure 11 is an overview data flow diagram for optimizing image capture to train a machine learning model for implicit representation according to some implementations of the present disclosure. More specifically, the computing system 102 may cause the capture information 104 to be provided to the user computing device 106. The computing system 102 may be or include any number of physical and / or virtualized hardware resources. For example, the computing system 102 may be a distributed cloud computing system that includes both physical and virtualized hardware resources located at multiple physical locations. For another example, the computing system 102 may be a server computing system including one or more processor devices. The user computing device 106 may be any type or kind of computing device (e.g., a smartphone, a laptop computer, a desktop computer, a tablet computer, a wearable device, a digital single-lens reflex (DSLR) camera, etc.) that a user may use to capture and / or transmit images. For example, the computing system 102 may be a system associated with one or more application-based services (e.g., a map service, a visual search service, etc.).

[0034] As described above, the computing system 102 can cause the capture information 104 to be sent to the user computing device 106. In some implementations, the computing system 102 can transmit the capture information 104 directly to the user computing device 106. Alternatively, in some implementations, the computing system 102 can instruct another computing device or system to provide the capture information 104 to the user computing device 106 (e.g., a network node, a content delivery network (CDN), a networked computing device, etc.). Alternatively, in some implementations, the computing system 102 can instruct the user computing device 106 to retrieve the capture information 104 from the other computing device or system.

[0035] The capture information 104 may include instructions 108 to a user of the user computing device 106. The instructions 108 may instruct the user to capture a plurality of different sets of images for a particular POI (e.g., a room, a building, an outdoor space, a business, etc.). It should be noted that, as described herein, a "POI" may generally refer to any type or kind of room, building, outdoor area, geographic area, or any combination thereof. For example, a POI may refer to an open, enclosed, or semi-enclosed outdoor space, such as an amphitheater, a playground, an outdoor seating area, a rooftop, a hiking trail, etc. For another example, a POI may refer to one or more buildings, and / or one or more rooms within a building.

[0036] Typically, a user of the user computing device 106 may be associated with a POI. For example, if the POI is a restaurant, the user may be an employee or owner of the restaurant, a contractor employed by the restaurant, etc. For another example, if the POI is a recreational area provided by a government organization, the user may be a member of the government organization. For another example, if the POI is a public space, or a publicly accessible space, the user may be a member of the public. For yet another example, the user may be associated with one or more of the application-based services described above (e.g., an employee or contractor of a service, etc.).

[0037] In some implementations, the computing system 102 may cause the capture information 104 to be transmitted to the user computing device 106 in response to a request 110 from the user computing device 106. Specifically, the user computing device 106 may provide a request 110 for a three-dimensional representation of a particular POI. In response, the computing system 102 may cause the capture information 104 to be sent to the user computing device 106 with instructions 108 to train the model to generate a three-dimensional representation of the POI.

[0038] In some implementations, the instructions 108 in the capture information 104 may instruct the user of the user computing device 106 to capture a calibration image set 112. The calibration image set 112 may include a plurality of calibration image pairs 114A-114N (generally, calibration image pairs 114). Each calibration image pair in the calibration image pairs 114 may include two images captured at two corresponding locations within the POI separated by a predefined distance.

[0039] For example, go to Figure 2 , Figure 2 Depicted are example capture instructions for capturing a set of calibration images in accordance with some implementations of the present disclosure. Figure 2 Will combine Figure 1 108. In particular, calibration capture instructions 202 are example instruction images that may be included in instructions 108. To follow the depicted example, calibration capture instructions 202 instruct a user of user computing device 106 to capture a calibration image pair at chest height from a predefined distance of three meters while facing the pairing position. Based on calibration capture instructions 202, the user may capture a plurality of calibration image pairs, wherein each image pair includes two images captured at two corresponding positions separated by three meters.

[0040] To follow the example shown, POI 204 is a room in a building that a user of user computing device 106 wishes to represent in three dimensions. Based on calibration capture instructions 202, the user may first capture image A of calibration image pair 114A at location 206A within POI 204. The user may capture image B of calibration image pair 114A at location 206B within the POI that is three meters apart from location 206A. When capturing image A of calibration image pair 114A, the user may face location 206B, and when capturing image B of calibration image pair 114A, the user may face location 206A. The user may repeat the process to capture calibration image pair 114B at locations 208A and 208B, calibration image pair 114C at locations 210A and 210B, calibration image pair 114N at locations 212A and 212B, and so on. In this manner, instructions 108 may enable a user of user computing device 106 to capture a set of calibration images 112 sufficient to train a model for three-dimensional representations of POI 204 .

[0041] return Figure 1 In some implementations, the instructions 108 in the capture information 104 may instruct the user of the user computing device 106 to capture a POI depiction image set 116. The POI depiction image set 116 may include a plurality of POI depiction images. Each of the POI depiction images may depict a POI or a portion of a POI from a different angle and / or position.

[0042] In some implementations, the POI depicted image set 116 can include a plurality of front-facing depicted images 118A-118N (generally, front-facing depicted images 118). The front-facing depicted images 118 can be captured as a user traverses a POI in a forward direction, and each of the front-facing depicted images 118 can depict the POI from a viewpoint facing in the forward direction.

[0043] Additionally or alternatively, in some implementations, the POI depicted image set 116 can include a plurality of rearward-facing depicted images 120A-120N (generally, rearward-facing depicted images 120). The rearward-facing depicted images 120 can be captured while the user is traversing the POI in a rearward direction opposite to the forward direction, and each of the rearward-facing depicted images 120 can depict the POI from a viewpoint facing in the rearward direction.

[0044] Additionally or alternatively, in some implementations, the POI depiction image set 116 can include a plurality of inward-facing depiction images 122A-122N (generally inward-facing depiction images 122). The inward-facing depiction images 122 can be captured as the user traverses the POI, and each of the inward-facing depiction images 122 can depict the POI from a viewpoint facing perpendicular to the forward direction and the rearward direction.

[0045] For example, assume that the POI is an empty rectangular room. To capture the front-facing depicted image 118, the user may traverse clockwise in the forward direction around the outer periphery of the rectangular room while capturing the front-facing depicted image 118. To capture the rear-facing depicted image 120, the user may traverse counterclockwise in the rear-facing direction around the outer periphery of the rectangular room while capturing the rear-facing depicted image 120. To capture the inward-facing depicted image 122, the user may traverse clockwise in the forward direction or counterclockwise in the rear-facing direction around the outer periphery of the rectangular room while capturing the image.

[0046] For a more specific example, go to Figure 3A , Figure 3A Depicted are example capture instructions for capturing a front-facing depicted image of a set of POI depicted images, according to some implementations of the present disclosure. Figure 3A Will combine Figure 1 104 . To follow the example shown, forward-facing capture instructions 302A illustrate an example POI 303 (e.g., an area of ​​an airport) formed by surfaces 304A–304D (typically surface 304). Forward-facing capture instructions 302A further illustrate movement indicator 306A. Movement indicator 306A indicates a forward direction that the user is to move or traverse when capturing forward-facing depicted image 118. As shown, forward-facing capture instructions 302A instruct the user to traverse in a clockwise direction around the periphery of POI 303 while capturing images from a viewpoint facing in the forward direction. In other words, forward-facing capture instructions 302A instruct the user to begin moving clockwise around the outer edge of the room while capturing images in the same direction as they are moving.

[0047] The forward-facing capture instructions 302A depict a plurality of user capture representations 308-316. The user capture representations 308-316 correspond to specific locations or positions within the POI 303 at which the forward-facing depicted image 118 was captured. After capturing one of the forward-facing depicted images 118, the user may move from the location of one user capture representation to the location of a subsequent user capture representation. The direction of the movement is indicated by the movement indicator 306A. Each of the user capture representations 308-316 includes an arrow that indicates the direction the user was facing when the corresponding forward-facing depicted image 118 was captured. Figure 3A A subset of the front-facing depicted images 118 is depicted to illustrate the differences between the front-facing depicted images 118 as the user moves in a forward direction. However, it should be noted that the front-facing depicted images 118 may depict any type or kind of POI from any viewpoint or perspective.

[0048] For example, the forward facing capture instruction 302A depicts a user capture representation 308. The user capture representation 308 indicates the location at which the forward facing depicted image 118A was captured. The user capture representation 308 also indicates the direction the user was facing when the forward facing depicted image 118A was captured. To follow the previous example of the airport, the surface 304B of the POI 303 may be a wall including the entrance to the men's and women's restrooms within the airport. When located at the location of the user capture representation 308, the user may capture the forward facing depicted image 118A while facing the surface 304B.

[0049] The forward-facing capture instructions 302A depict a user capture representation 310. The user capture representation 310 indicates the location at which the forward-facing depicted image 118B is captured. The user capture representation 310 also indicates the direction in which the user was facing when the forward-facing depicted image 118B was captured. Since the user is still facing and moving in the same direction as when the forward-facing depicted image 118A was captured, the forward-facing depicted image 118B may depict the same portion of the surface 304B as the forward-facing depicted image 118A. However, since the user has moved closer to the surface 304B since the forward-facing depicted image 118A was captured, the viewpoint from which the forward-facing depicted image 118B depicts the surface 304B has also moved closer to the surface 304B. Similarly, the user capture representation 312 indicates that the user has moved closer to the surface 304B after the forward-facing depicted image 118B was captured, and the forward-facing depicted image 118C may depict the surface 304B from a viewpoint that is closer to the surface 304B than the forward-facing depicted image 118B.

[0050] The forward-facing capture instructions 302A depict a user capture representation 314. As with the user capture representations 308-312, the user capture representation 314 may indicate the location of the user, the direction the user is facing, and the direction the user is moving. Further, as with the user capture representations 308-312, the user capture representation 314 faces the same direction as indicated by the movement indicator 306A for the location of the user capture representation 314. However, unlike the user capture representations 308-312, the user capture representation 314 faces the surface 304C of the POI 303 instead of the surface 304B. In some implementations, this may be due to the user reaching a threshold distance from the surface 304B, at which point further movement toward the surface 304B is unnecessary.

[0051] Forward-facing capture instructions 302A depict user capture representation 316. User capture representation 316 indicates the location at which forward-facing depicted image 118E was captured. User capture representation 316 also indicates the direction the user was facing when forward-facing depicted image 118E was captured. Since the user is still facing and moving in the same direction as when forward-facing depicted image 118D was captured, forward-facing depicted image 118D may depict the same portion of surface 304C as forward-facing depicted image 118D. However, since the user has moved closer to surface 304C since forward-facing depicted image 118D was captured, the viewpoint of forward-facing depicted image 118E depicting surface 304C has also moved closer to surface 304C.

[0052] In such a manner, forward-facing capture instructions 302A may instruct the user to capture each of surfaces 304A- 304D from various positions and viewpoints while traversing POI 303 in the direction indicated by movement indicator 306A, thereby ensuring that forward-facing depicted images 118 are sufficient to train the model to generate a three-dimensional representation of the POI.

[0053] Go to Figure 3B , Figure 3B Depicted are example capture instructions for capturing a rear-facing depicted image of a set of POI depicted images, according to some implementations of the present disclosure. Figure 3B Will combine Figure 1 304A-304D. The backward-facing capture instructions 302B further illustrate a movement indicator 306B. The movement indicator 306B indicates a backward direction that the user is to move or traverse when capturing the backward-facing depicted image 120. As shown, the backward-facing capture instructions 302B instruct the user to traverse in a counterclockwise direction around the periphery of the POI while capturing images from a viewpoint facing the backward direction. In other words, the backward-facing capture instructions 302B instruct the user to begin moving counterclockwise around the outer edge of the POI 303 while capturing images in the same direction as the user is moving.

[0054] The rearward-facing capture instructions 302B depict a plurality of user capture representations 318-326. The user capture representations 318-326 correspond to specific locations or positions within the POI 303 at which the rearward-facing depicted images 120 are captured. After capturing one of the rearward-facing depicted images 120, the user may move from the location of one user capture representation to the location of a subsequent user capture representation. The direction of the movement is indicated by the movement indicator 306B. Here, the movement indicator 306B indicates the position of the user capture representation in relation to the rearward-facing depicted image 120. Figure 3A The movement indicator 306A moves in the opposite direction.

[0055] Each of the user-captured representations 318 - 326 includes an arrow that indicates the direction the user was facing when the corresponding rearward-facing depicted image 120 was captured. Figure 3B A subset of the rearward-facing depicted images 120 is depicted to illustrate the difference between the rearward-facing depicted images 120 as the user moves in a rearward direction. However, it should be noted that the rearward-facing depicted images 120 may depict any type or kind of POI from any viewpoint or perspective.

[0056] For example, the rearward-facing capture instruction 302B depicts a user capture representation 318. The user capture representation 318 indicates the location at which the rearward-facing depicted image 120A is captured. The user capture representation 318 also indicates the direction the user was facing when the rearward-facing depicted image 120A was captured. To follow the previous example of the airport, the surface 304A of the POI 303 may be a floor-to-ceiling glass window that provides a view of the airport runway. When located at the location of the user capture representation 318, the user may capture the rearward-facing depicted image 120A while facing the surface 304B.

[0057] The rear-facing capture instructions 302B depict a user capture representation 320. The user capture representation 320 indicates the position and direction that the user was facing when the rear-facing depicted image 120B was captured. Since the user is still facing and moving in the same direction as when the rear-facing depicted image 120A was captured, the rear-facing depicted image 120B may depict the same portion of the surface 304A as the rear-facing depicted image 120A. However, since the user has moved closer to the surface 304A since the rear-facing depicted image 120A was captured, the viewpoint of the rear-facing depicted image 120B depicting the surface 304A has also moved closer to the surface 304A.

[0058] The backward facing capture instructions 302B depict a user capture representation 322. As with the user capture representations 318 and 320, the user capture representation 322 may indicate the user's location, the direction the user is facing, and the direction the user is moving. Further, as with the user capture representations 318 and 320, the user capture representation 322 faces the same direction as indicated by the movement indicator 306B for the location of the user capture representation 322. However, unlike the user capture representations 318 and 320, the user capture representation 322 faces the surface 304D of the POI 303 instead of the surface 304A. For example, as the user has moved closer to the glass window, the airplane and objects depicted in the image appear larger, while other objects (such as a trash can) no longer appear in the frame.

[0059] Rearward-facing capture instructions 302B depict user capture representations 324 and 326. User capture representations 324 and 326 indicate the location and direction the user was facing when the rearward-facing depicted images 120D and 120E were captured, respectively. Since the user is still facing and moving in the same direction as when the rearward-facing depicted image 120C was captured, the rearward-facing depicted images 120D and 120E may depict the same portion of the surface 304D as the rearward-facing depicted image 120C. However, since the user has moved closer to the surface 304D since the rearward-facing depicted image 120C was captured, the viewpoints of the rearward-facing depicted images 120D and 120E depicting the surface 304D have also moved closer to the surface 304C.

[0060] In such a manner, backward-facing capture instructions 302B may instruct the user to capture each of surfaces 304A- 304D from various positions and viewpoints while traversing POI 303 in the direction indicated by movement indicator 306B, thereby ensuring that backward-facing depicted images 120 are sufficient to train the model to generate a three-dimensional representation of the POI.

[0061] Go to Figure 3C , Figure 3C Depicted are example capture instructions for capturing an inward-facing depiction image of a set of POI depiction images, according to some implementations of the present disclosure. Figure 3C Will combine Figure 1 304A-304D. The inward-facing capture instructions 302C further illustrate a movement indicator 306C. The movement indicator 306C indicates the direction the user is to move or traverse when capturing the inward-facing depicted image 122. As shown, the backward-facing capture instructions 302B instruct the user to traverse in a clockwise direction (i.e., a "forward" direction) around the periphery of the POI.

[0062] However, it should be noted that depicting movement indicator 306C as indicating forward movement is merely arbitrary. Rather, movement indicator 306C may indicate any type or kind of movement pattern (e.g., forward loop, backward loop, mixed movement, etc.) sufficient to traverse the perimeter of POI 303. Thus, if entering a room from a doorway (not depicted) located on surface 304D, a user capturing inwardly facing depicted image 122 may begin moving toward either surface 304A or surface 304B, and when moving within a threshold distance of either, change direction to move toward surface 304C.

[0063] When traversing the perimeter of POI 303, the user may capture images from a viewpoint that faces perpendicular to the forward and rearward directions. In other words, the user may face inward, toward the center of the room, when capturing the inward-facing depicted image 122. Inward-facing capture instructions 302C may indicate or describe the direction the user is to face when capturing the inward-facing depicted image 122 in a variety of ways. For example, inward-facing capture instructions 302C may indicate to the user that they should face opposite the wall to which they are closest. For another example, inward-facing capture instructions 302C may indicate that the user should hold their image capture device at chest height and shuffle sideways around the perimeter of the room.

[0064] Inward-facing capture instructions 302C depict a plurality of user capture representations 328-336. User capture representations 328-336 correspond to specific locations or positions within POI 303 at which inward-facing depicted images 122 were captured. After capturing one of inward-facing depicted images 122, a user may move from the location of one user capture representation to the location of a subsequent user capture representation. The direction of the movement is indicated by movement indicator 306C. Each of the user capture representations 328-336 includes an arrow that indicates the direction the user was facing when the corresponding inward-facing depicted image 122 was captured.

[0065] In some implementations, inward-facing capture instructions 302C may indicate that the user should aim their image capture device at the “center point” of POI 303 (eg, via a text prompt presented to the user, etc.).

[0066] Alternatively, in some implementations, inward-facing capture instructions 302C may indicate that the user should not focus on the "center" of POI 303, but should allow the viewpoint of inward-facing depicted image 122 to move "sideways" as the user also moves "sideways." For example, inward-facing capture instructions 302C depict the user capturing representation 329, as well as two possible vectors 338 and 340 along which the viewpoint of the captured image may be aligned. If the instructions collect images while aligning the viewpoint of the image capture device parallel to the surface the user is facing (i.e., moving "sideways"), the viewpoint of the captured image may be aligned with vector 338. Conversely, if the instructions collect images with the viewpoint focused toward the "center" of POI 303, the viewpoint of the captured image may be aligned with vector 340.

[0067] The rearward-facing capture instructions 302B depict a user capture representation 328. The user capture representation 328 indicates the location at which the inward-facing depicted image 122A was captured. The user capture representation 328 also indicates the direction the user was facing when the inward-facing depicted image 122A was captured. To follow the previous example of an airport, the inward-facing depicted image 122A may depict the POI 303 with respect to the POI 303. Figure 3B In particular, as depicted, because user-captured representation 328 is positioned closer to surface 304C than user-captured representations 322-326, the portion of surface 304D depicted by inward-facing depicted image 122A may be a different portion than the portion depicted by rearward-facing depicted images 120C-120E. Similarly, inward-facing depicted image 122B may include more of surface 304D depicted by rearward-facing depicted image 120C because the distance between user-captured representations 322 and 330 is smaller than the distance between user-captured representations 322 and 328.

[0068] Inward facing capture instructions 302C depict user capture representation 332. User capture representation 332 indicates the position and direction the user was facing when capturing inward facing depicted image 122C. Here, in order to maintain an inward viewpoint, the user has aligned the viewpoint of the image perpendicularly to the forward direction. Specifically, despite moving toward surface 304D, the viewpoint of inward facing depicted image 133C depicts surface 304B from an inward facing direction, which is perpendicular to the direction the user is moving toward surface 304D.

[0069] The user may continue to move toward surface 304D while capturing inwardly facing depicted images 122D and 122E at the locations where the user captured representations 334 and 336, respectively. Similar to inwardly facing depicted image 122C, inwardly facing depicted images 122D and 122E may depict surface 304B from a viewpoint that faces perpendicular to the forward and rearward directions (i.e., an "inward" direction). Thus, as the user moves toward surface 304D, the portion of surface 304B depicted by inwardly facing depicted images 122C-122E may "move" along surface 304B. In other words, the "movement" of the viewpoint depicted in inwardly facing depicted images 122C-122E may correspond to movement between user captured representations 332-336.

[0070] In such a manner, inward-facing capture instructions 302C may instruct the user to capture each of surfaces 304A- 304D from various positions and viewpoints while traversing POI 303 in the direction indicated by movement indicator 306B, thereby ensuring that inward-facing depicted images 122 are sufficient to train the model to generate a three-dimensional representation of the POI.

[0071] return Figure 1 In some implementations, the instructions 108 in the capture information 104 may instruct the user of the user computing device 106 to capture the path image set 124. For the path image set 124, the instructions for capturing the path image set 124 may instruct the user to capture a series of path images 126A-126N in sequence while following the path that the user wishes to depict. For example, if the user wishes to depict a straight path from a first point to a second point within the POI, the user may move along the path while capturing images in sequence (e.g., every three feet, etc.). For another example, the user may depict a circular path by navigating around the perimeter of a room in a clockwise or counterclockwise manner. In this manner, the path image set may be utilized to derive a path for a virtual camera in a three-dimensional representation of the POI 303.

[0072] For a specific example, go to Figure 4 , Figure 4 Depicted are example capture instructions for capturing a pathway image of a pathway image set, according to some implementations of the present disclosure. Figure 4 Will combine Figure 1 108. In particular, path image capturing instructions 402 are example instruction images that may be included in instructions 108. To follow the depicted example, path image capturing instructions 402 instruct a user of user computing device 106 to capture path images at regular intervals while traversing a desired path. Based on path image capturing instructions 402, the user may capture a plurality of path images 126A-126N (generally referred to as path images 126) at locations 404A-404N, respectively.

[0073] In some implementations, the path images may be captured at regular intervals and / or distances. Alternatively, in some implementations, the path images 126 may be captured at irregular intervals and / or distances. To follow the depicted example, the distance between locations 404E and 404F is greater than the distance between locations 404G and 404H. In this manner, by enabling a reduced capture density of areas within a POI that are less relevant than other areas of the POI, the implementations described herein may substantially reduce the computational resource expenditure associated with representing these areas.

[0074] return Figure 1 , the computing system 102 may receive representation information 128 from the user computing device 106. The representation information 128 may include the calibration image set 112, the POI depiction image set 116, and the path image set 124. In some implementations, the representation information 128 may include geographic location information 130. The geographic location information 130 may indicate the location of the POI for which the request 110 is provided. For example, if the POI is located in Times Square in New York City, the geographic location information 130 may indicate the precise coordinates of the POI, the geographic area of ​​the POI, etc.

[0075] Additionally or alternatively, in some implementations, the representation information 128 may include or otherwise indicate other characteristics of the POI. For example, if the POI is a business, the representation information 128 may indicate certain characteristics of the business (e.g., menu items for a restaurant, dimensional information of a building, products offered by the business, etc.). In some implementations, the representation information 128 may be used to supplement the POI depiction image set to generate a three-dimensional representation of the POI. For example, if the representation information 128 indicates the dimensions of the POI, the representation information 128 may be used instead of or in addition to the calibration image set 112.

[0076] In some implementations, the representation information 128 may include metadata 132. The metadata 132 may include application-specific information. For example, if the computing system 102 is associated with a map application, the metadata 132 may include map-specific information (e.g., information related to the entry of the POI within the map application, user-generated content submitted for the POI, etc.). For another example, if the computing system 102 is associated with a visual search application, the metadata may include search-specific information (e.g., search terms associated with the POI, etc.).

[0077] The computing system 102 may include a model trainer 134. The model trainer 134 may train a machine-learned POI representation model 136 based on the POI depiction image set 116. Specifically, the model trainer 134 may process the POI depiction image set 116 using the machine-learned POI representation model 136 to train the machine-learned POI representation model 136 to learn an implicit three-dimensional representation of the POI. Once trained, the machine-learned POI representation model 136 may receive the three-dimensional coordinates and viewing direction of a specific point in the POI as input, and output the corresponding color and / or light information of the specific point.

[0078] The computing system 102 may include a renderer 138. The renderer 138 may include a machine-learned POI representation model 136 and a waypoint determiner 140. The waypoint determiner may process the path image set 124, the representation information 128, and / or information derived from the path image set 124 to obtain coordinate information 142. The coordinate information 142 may include the three-dimensional coordinates and the viewing direction of a particular point within the POI. The machine-learned POI representation model 136 may process the coordinate information 142 to obtain radiance information (e.g., light, color, etc.) of the particular point. Iteratively, the renderer 138 may render each pixel of the rendered video data 144.

[0079] The rendered video data 144 may depict a three-dimensional representation of the POI depicted by the POI depiction image set 116. Specifically, the rendered video data 144 may depict the POI from the perspective of a virtual camera that traverses the path indicated by the path image set 124 through the POI. For example, if the POI is a hotel lobby, and the path image set 124 collectively depicts a path from the entrance of the hotel lobby to the reception desk, the rendered video data 144 may depict the virtual camera traversing from the entrance of the hotel to the reception desk. The computing system 102 may provide the rendered video data 144 to the user computing device 106.

[0080] Figure 5 Depicted are example capture instructions for executing an image capture mode when capturing images of various sets of images in accordance with some implementations of the present disclosure. Figure 4 Will combine Figure 1 504A-504E. In particular, spiral capture instructions 502 indicate a capture mode that the user is to perform when capturing images 504A-504E. To follow the depicted example, spiral capture instructions 502 instruct the user to move the image capture device in a spiral (e.g., circular, etc.) capture mode when capturing images of the previously described image set. In response, the user may capture images 504A-504E while performing the spiral capture mode, and images 504A-504E may depict different portions of the surface of the POI. In this manner, spiral capture instructions 502 may increase the likelihood that all portions of the surface of the POI are captured.

[0081] In some implementations, the image capture mode shown in capture instructions 502 may be some type of mode other than a spiral capture mode. For example, the capture mode may be a grid-based capture mode, where images are captured based on a grid. In some implementations, the image capture device may include multiple image capture sensors or devices arranged in a capture mode, rather than moving the image capture device in a capture mode. In this way, the extent of capture movement performed by the user may be reduced or eliminated.

[0082] Figure 6 A flow chart depicts an example method 600 for optimizing image capture to train a machine-learned POI representation model in accordance with some implementations of the present disclosure. Figure 6 The operations are depicted as being performed in a particular order for purposes of illustration and discussion, but the methods of the present disclosure are not limited to the particular illustrated order or arrangement. Various operations of method 600 may be omitted, rearranged in various ways, combined and / or adjusted without departing from the scope of the present disclosure.

[0083] At operation 602, the computing system may cause capture information to be sent to a user computing device. The capture information may include instructions to a user of the user computing device for capturing multiple sets of images of the POI. In some implementations, the instructions may be visual and / or textual instructions that instruct the user to capture multiple sets of images, each set of images including a different type of image. Additionally or alternatively, in some implementations, the instructions may instruct the user how to capture each type of image. In some implementations, the instructions may include example images for the user. For example, the instructions may include instructions that are respectively associated with the user. Figure 2 , Figure 3A – Figure 3C and Figure 4 The capture instructions 202, 302A-302C and 402 are similar or identical to the capture instructions in FIG.

[0084] At operation 604, the computing system may receive a plurality of image sets of the POI from the user computing device. The plurality of image sets may include a calibration image set. The calibration image set may include a plurality of calibration image pairs. Each calibration image pair in the calibration image pair may include two calibration images captured at two different locations separated by a predefined distance within the POI. The plurality of image sets may include a POI depiction image set, the POI depiction image set including a plurality of depiction images, the plurality of depiction images depicting the POI from a corresponding plurality of angles and / or locations. The plurality of image sets may include a path image set, the path image set including a plurality of path images, the plurality of path images collectively depicting a path from a starting location to an ending location within the POI.

[0085] In some implementations, the POI may be a building including multiple areas. The computing system may receive a path image set including multiple path images that collectively depict a path from a starting location to an ending location within the POI. The starting location may include a first area of ​​the multiple areas, and the ending location may include a second area of ​​the multiple areas that is different from the first area. For example, the area of ​​the POI may be a room, a corridor, a floor of a building, a building, a vehicle interior, etc.

[0086] In some implementations, the computing system may receive a second set of path images, the second set of path images comprising a second plurality of path images that collectively depict a second path within the POI from a second starting location to a second ending location. The second starting location may comprise a second area of ​​the plurality of areas, and the second ending location may comprise a third area of ​​the plurality of areas.

[0087] In some implementations, a computing system may receive a subset of forward-facing depicted images. The subset of forward-facing depicted images may include a plurality of forward-facing depicted images captured while a user traverses a POI in a forward direction. Each of the forward-facing depicted images may depict the POI from a viewpoint facing in a forward direction.

[0088] Additionally or alternatively, in some implementations, the computing system may receive a subset of rearward-facing depicted images, the subset of rearward-facing depicted images comprising a plurality of rearward-facing depicted images captured while the user traverses the POI in a rearward direction opposite to the forward direction. Each of the rearward-facing depicted images may depict the POI from a viewpoint facing in the rearward direction.

[0089] Additionally or alternatively, in some implementations, the computing system may receive an inward-facing depicted image subset comprising a plurality of inward-facing depicted images captured by a user while traversing a POI. Each of the inward-facing depicted images may depict the POI from a viewpoint facing perpendicular to the forward direction and the rearward direction.

[0090] In some implementations, each of the one or more subsets of depicted images is captured while moving the image capture device in a spiral motion. More specifically, the capture instructions provided to the user may instruct the user to capture each image to be included in the image set while moving the image capture device in a spiral pattern. For example, assume that the user is traversing the periphery of a POI in a forward direction to capture a forward-facing depicted image for a set of POI depicted images. While capturing the forward-facing depicted image, the user may move their image capture device in a circular motion. In this manner, more information may be obtained about the surface of a particular POI that the user was facing while capturing the forward-facing depicted image.

[0091] It should be noted that the capture instructions may instruct the user to perform some other type or kind of motion when capturing images. For example, the capture instructions may instruct the user to capture images while moving the image capture device in a square or rectangular pattern, a star pattern, a triangle pattern, a random pattern, etc. Alternatively, in some implementations, the user may utilize a type or kind of image capture device that avoids performing a moving pattern when capturing images.

[0092] In some implementations, each of the one or more subsets of depicted images is captured when an auto-exposure feature of the image capture device is activated. More specifically, if the images are captured using a particular type of image capture device, such as a DSLR, the capture instructions may instruct the user to set the value of an exposure parameter (e.g., International Organization for Standardization (ISO) sensitivity) to AUTO.

[0093] At operation 606, the computing system may train the machine-learned POI representation model using the plurality of image sets of the POI. The machine-learned POI representation model may be trained to generate representations of the POI. For example, the computing system may first position all POI depiction images in three-dimensional space based on the calibration image. The computing system may train the machine-learned POI representation model based on the positioned POI depiction images.

[0094] In some implementations, prior to training the model, the computing system may process multiple images from the multiple image sets of the POI to obtain a corresponding plurality of quality indicators. Based on the plurality of quality indicators, the computing system may identify one or more images of insufficient quality from the multiple images. In response to identifying the one or more images of insufficient quality, the computing system may cause remedial capture information to be sent to the user computing device. The remedial capture information may include instructions to a user of the user computing device to recapture one or more image sets including the one or more images of insufficient quality.

[0095] At operation 608 , the computing system may use the machine-learned POI representation model to generate video data depicting a virtual camera traversing a path within the POI from a starting location to an ending location.

[0096] In some implementations, a computing system may use a machine-learned POI representation model to generate multiple video datasets, the multiple video datasets including a first video dataset depicting a path traversed by a virtual camera from a first area to a second area, and a second video dataset depicting a second path traversed by the virtual camera from the second area to a third area.

[0097] In some implementations, to generate video data, a computing system may utilize a machine-learned POI representation model to process a plurality of inputs corresponding to a corresponding plurality of three-dimensional points within the POI to obtain color information. Each of the plurality of inputs may include a set of three-dimensional coordinates of a corresponding three-dimensional point in the plurality of points and a viewing direction value. The color information may indicate a predicted color at each of the plurality of three-dimensional points from a particular viewing direction. The computing system may render a frame of the plurality of frames of video data based on the predicted color at each of the plurality of points within the POI.

[0098] In some implementations, the machine-learned POI representation model may be a NeRF model, and the computing system may utilize the NeRF neural network to process a plurality of three-dimensional coordinates corresponding to a respective plurality of points within the POI to obtain radiation information indicating predicted radiation at each of the plurality of points within the POI.

[0099] In some implementations, before processing the plurality of three-dimensional coordinates using the NeRF neural network, the computing system may perform a ray casting process to obtain the plurality of three-dimensional coordinates corresponding to the plurality of points within the POI. The ray casting process may be derived from a viewpoint of a virtual camera. In some implementations, the viewpoint of the virtual camera may be based at least in part on the path image set.

[0100] Figure 7 A flow chart depicts an example method 700 for local optimization of image capture to train a machine-learned POI representation model according to some implementations of the present disclosure. Figure 7 The operations are depicted as being performed in a particular order for purposes of illustration and discussion, but the methods of the present disclosure are not limited to the particular illustrated order or arrangement. Various operations of method 700 may be omitted, rearranged in various ways, combined and / or adjusted without departing from the scope of the present disclosure.

[0101] At operation 702, a user computing device (e.g., a computing system, a laptop computer, a desktop computer, a smart phone, an image capture device, etc.) may obtain capture information from a computing system. The capture information may include instructions to a user of the user computing device to capture a plurality of sets of images of a POI.

[0102] At operation 704, the user computing device may obtain a plurality of image sets of the POI. The plurality of image sets may be captured by the user via an image capture device. For example, the user computing device may include a built-in image capture device, and the user may capture the image sets using the built-in image capture device. For another example, the user may capture images using a specific image capture device, such as a DSLR, and the image sets may be transmitted to the user computing device (e.g., via a physical storage medium, wireless transmission, etc.).

[0103] At operation 706, the user computing device may perform a quality verification process for at least some of the plurality of image sets. The quality verification process may be any type or kind of process or technique that analyzes the quality of an image and / or the content depicted by the image. In some implementations, the quality verification process may include processing the image using a machine learning model that is trained to detect defects within the image, such as blur, obstructions that impede image capture, pixelation, and the like. Additionally, or alternatively, in some implementations, the quality verification process may include processing the image using a machine learning model that is trained to detect sensitive content depicted within the image. For example, a machine learning model may be trained to detect whether inappropriate content is depicted in an image. For another example, a machine learning model may be trained to detect whether privacy-sensitive content is depicted in an image.

[0104] In some implementations, the user computing device may perform corrective actions based on the quality verification process. For example, if the quality verification process identifies an inappropriate or privacy-sensitive object depicted in the image, the quality user computing device may apply a masking effect to mask the object. Alternatively, the user computing device may transmit information indicating the location of the object within the image to the computing system.

[0105] In some implementations, the quality verification process may include receiving quality verification information from a computing system. More specifically, to perform the quality verification process, the user computing device may transmit the image to the computing system for evaluation. The computing system may evaluate the quality of the image based on the metrics described above and may communicate the results of the process to the user computing device.

[0106] Additionally or alternatively, in some implementations, based on the quality verification process, the user computing device may determine that additional images need to be captured. If necessary, the user computing device may indicate to the user that the user should recapture certain images within a particular image set, or recapture the entire image set.

[0107] Specifically, in some implementations, the user computing device may process multiple images from multiple image sets of the POI to obtain corresponding multiple quality indicators. Based on the multiple quality indicators, the user computing device may identify one or more images of insufficient quality from the multiple images.

[0108] In response to identifying one or more images of insufficient quality, the user computing device may cause instructions to be displayed to a user of the user computing device to recapture an image set including the one or more images of insufficient quality from the plurality of image sets. For example, the user computing device may display the instructions on a display device associated with the user computing device (e.g., a smartphone display, a mixed reality (MR) display device, etc.). The user computing device may obtain a replacement image set for the image set including the one or more images of insufficient quality.

[0109] For another example, a user computing device may receive remedial capture information from a computing system including instructions for display to a user of the user computing device. The instructions instruct the user to recapture one or more image sets including one or more images of insufficient quality. In response to receiving the remedial capture information, the user computing device may cause instructions to be displayed to the user of the user computing device to recapture the image set including the one or more images of insufficient quality. The user computing device may obtain a replacement image set for the image set including the one or more images of insufficient quality.

[0110] At operation 708, the user computing device may transmit the capture information to the computing system in response to performing the quality verification process. The capture information may include a plurality of sets of images of the POI.

[0111] At operation 710, a user computing device may obtain video data from a computing system that depicts a virtual camera traversing a path within a POI from a starting location to an ending location. The video data may be generated by the computing system using a machine-learned POI representation model that is trained to generate representations of POIs based on multiple image sets.

[0112] Figure 8 A block diagram of an example computing environment 800 that performs image capture optimization to train a machine-learned POI representation model is depicted according to an example implementation of the present disclosure. The computing environment 800 includes a user computing device 802 associated with a user of an application-based service such as a mapping service and / or a visual search service.

[0113] The user computing device 802 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device (e.g., a virtual / augmented reality device, etc.), an embedded computing device, a broadcast computing device (e.g., a webcam, etc.), etc.

[0114] In particular, in some implementations, the user computing device 802 may be a computing system for capturing images to train a machine learning model. For example, the user computing device 802 may be a smartphone that includes an image capture device sufficient to capture images for training a machine learning POI representation model. For another example, the user computing device may be a computing device (e.g., a laptop, a smartphone, etc.) that can receive images captured using a specific image capture device (e.g., a DSLR, etc.).

[0115] The user computing device 802 includes a processor 804 and a memory 806. The processor 804 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or an operatively connected processor. The memory 806 may include non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc. and combinations thereof. The memory 806 may store data 808 and instructions 810 executed by the processor 804 to cause the user computing device 802 to perform operations.

[0116] In particular, the memory 806 of the user computing device 802 may include an image capture system 812. The image capture system 812 may facilitate image capture for training and utilizing a machine-learned POI representation model. To facilitate image capture, the image capture system 812 may include a service module 814 that may collectively facilitate image capture and transmission by providing various services.

[0117] For example, the service module 814 may include a renderer 816. The renderer 816 may include a machine-learned POI representation model 818. The renderer 816 may utilize the machine-learned POI representation model 818 to generate video data, such as Figure 1 The renderer 138 is described.

[0118] The machine-learned POI representation model 818 may be or otherwise include various machine-learning models, such as a neural network (e.g., a deep neural network) or other types of machine-learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine-learning models may utilize attention mechanisms, such as self-attention. For example, some example machine-learning models may include a multi-head self-attention model (e.g., a transformer model).

[0119] For another example, the service module 814 may include an image quality assessor 820. The image quality assessor 820 may assess the quality of an image to train the machine-learned POI representation model 818, and may instruct the user to recapture the image if an image of insufficient quality is detected.

[0120] The user computing device 802 may also include an input device 830 that receives input from the user or otherwise captures data associated with the user. For example, the input device 830 may include a touch-sensitive device (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive device may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0121] In some implementations, the user computing device 802 may include or be communicatively coupled to an input device 830. For example, the input device 830 may include a camera device that can capture two-dimensional image data of a POI. In some implementations, the input device 830 may include multiple camera devices communicatively coupled to the user computing device 802, the multiple camera devices configured to capture image data from different perspectives to generate three-dimensional gesture data / representation (e.g., a representation of a POI, etc.).

[0122] In some implementations, the input device 830 may include a sensor device (e.g., an accelerometer, a global positioning satellite (GPS) sensor, a gyroscope, an infrared sensor, a head tracking sensor such as a magnetic capture system, an omnidirectional treadmill device, a sensor configured to track eye movements of a user, etc.) that is configured to capture sensor data indicative of movements of a user associated with the computing user device 802.

[0123] In particular, the input device 830 may include a specific image capture device 832. The specific image capture device 832 may be or otherwise include a device specifically configured to capture image data or video data. For example, the input device 832 may include a DSLR camera, an omnidirectional image capture device, a high-resolution video capture device, a wearable image capture device, etc.

[0124] In some implementations, the user computing device 802 may include or be communicatively coupled to an output device 834. The output device 834 may be or otherwise include a device configured to output audio data, image data, video data, etc. For example, the output device 834 may include a two-dimensional display device (e.g., a television, a projector, a smartphone display device, etc.). For another example, the output device 834 may include a display device for an augmented reality device or a virtual reality device.

[0125] The computing system 850 includes a memory 854 and a processor 852. The processor 852 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 854 may include non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 854 may store data 856 and instructions 858 executed by the processor 852 to cause the computing system 850 to perform operations.

[0126] In some implementations, computing system 850 may be or otherwise include a virtual machine or containerized unit of software instructions executed within a virtualized cloud computing environment (e.g., a distributed networked collection of processing devices) and may be instantiated upon request. Additionally or alternatively, in some implementations, computing system 850 may be or otherwise include a physical processing device, such as a processing node (e.g., a node of physical hardware resources) within a cloud computing network.

[0127] The computing system 850 may facilitate training of the machine-learned POI representation model 818. More specifically, the computing system 850 may utilize the representation model training system 860 to train the machine-learned POI representation model 818 and / or perform any necessary processing of the training data (e.g., images). For example, a loss function may be back-propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters in multiple training iterations. In some implementations, performing error back-propagation may include performing truncated back-propagation through time. The representation model training system 860 may perform a variety of generalization techniques (e.g., weight decay, temporary back-off, etc.) to improve the generalization ability of the model being trained.

[0128] The representation model training system 860 includes computer logic for providing the desired functionality. The representation model training system 860 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, the representation model training system 860 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the representation model training system 860 includes one or more computer executable instruction sets stored in a tangible computer-readable storage medium (such as RAM, a hard disk, or an optical or magnetic medium).

[0129] In some implementations, computing system 850 includes or is otherwise implemented by a server computing device. In instances where computing system 850 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0130] In some implementations, the computing system 850 may receive data from the user computing device 802 according to various encryption schemes (e.g., codecs, lossy compression schemes, lossless compression schemes, etc.). For example, the user computing device 802 may encode the audio data using an audio codec and then transmit the encoded audio data to the computing system 850. The computing system 850 may decode the encoded audio data using an audio codec. In some implementations, the user computing device 802 may dynamically select between a plurality of different codecs with different degrees of loss based on the conditions of the network 899, the user computing device 802, and / or the computing system 850 (e.g., available network bandwidth, accessibility of hardware / software resources, etc.). For example, the user computing device 802 may dynamically switch from audio data transmission according to a lossy encoding scheme to audio data transmission according to a lossless encoding scheme based on the signal strength between the user computing device 802 and the network 899.

[0131] The network 899 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications through the network 899 may be conducted via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0132] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0133] The following definitions provide a detailed description of the various terms discussed in this specification. Therefore, it should be noted that any previous references to the following terms in the specification should be understood in accordance with these definitions.

[0134] Cloud: As used herein, the term "cloud" or "cloud computing environment" generally refers to a network of interconnected computing devices (e.g., physical computing devices, virtualized computing devices, etc.) and associated storage media that interoperate to perform computing operations, such as data storage, transmission, and / or processing. In some implementations, a cloud computing environment may be implemented and managed by an information technology (IT) service provider. An IT service provider may provide access to a cloud computing environment as a service to various users, who may be referred to as "cloud customers" in some cases.

[0135] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0136] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, modifications, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet further embodiments. Therefore, the present disclosure is intended to cover such changes, changes, and equivalents.

Claims

1. A computer-implemented method comprising: causing, by a computing system including one or more computing devices, capture information to be sent to a user computing device, wherein the capture information includes instructions to a user of the user computing device to capture a plurality of sets of images of a point of interest (POI); The plurality of image sets of the POI are received by the computing system from the user computing device, wherein the plurality of image sets include: a calibration image set, the calibration image set comprising a plurality of calibration image pairs, each of the calibration image pairs comprising two calibration images captured at two different locations within the POI separated by a predefined distance; a POI depiction image set, the POI depiction image set comprising a plurality of depiction images, the plurality of depiction images depicting the POI from a corresponding plurality of angles and / or positions; and a path image set, the path image set comprising a plurality of path images, the plurality of path images collectively depicting a path from a starting position to an ending position within the POI; training, by the computing system, a machine-learned POI representation model using the plurality of image sets of the POI, wherein the machine-learned POI representation model is trained to generate a representation of the POI; and Video data is generated by the computing system using the machine-learned POI representation model, the video data depicting a virtual camera traversing the path within the POI from the starting location to the ending location.

2. The computer-implemented method of claim 1, wherein: The POI includes a building, the building includes a plurality of areas; and Receiving the plurality of image sets of the POI comprises: The path image set is received by the computing system from the user computing device, the path image set comprising the plurality of path images that collectively depict the path from the starting location to the ending location within the POI, wherein the starting location comprises a first area of ​​the plurality of areas, and wherein the ending location comprises a second area of ​​the plurality of areas that is different from the first area.

3. The computer-implemented method of claim 2, wherein the plurality of regions comprises at least one of: Room; corridor; Floors of buildings; a building; or Vehicle interior.

4. The computer-implemented method of claim 2, wherein receiving the plurality of image sets of the POI further comprises: receiving, by the computing system, from the user computing device, a second set of path images, the second set of path images comprising a second plurality of path images, the second plurality of path images collectively depicting a second path within the POI from a second starting location to a second ending location, wherein the second starting location comprises the second area of ​​the plurality of areas, and wherein the second ending location comprises a third area of ​​the plurality of areas; and The use of the machine-learned POI representation model to generate the video data includes: A plurality of video datasets are generated by the computing system using the machine-learned POI representation model, the plurality of video datasets comprising a first video dataset and a second video dataset, the first video dataset depicting the virtual camera traversing the path from the first area to the second area, the second video dataset depicting the virtual camera traversing the second path from the second area to the third area.

5. The computer-implemented method of claim 1 , wherein generating the video data using the machine-learned POI representation model comprises: processing, by the computing system, a plurality of inputs corresponding to a respective plurality of three-dimensional points within the POI using the machine-learned POI representation model to obtain color information, wherein each of the plurality of inputs comprises a set of three-dimensional coordinates of a corresponding three-dimensional point in the plurality of points and a viewing direction value, and wherein the color information indicates a predicted color at each of the plurality of three-dimensional points from a particular viewing direction; as well as A frame of the plurality of frames of the video data is rendered, by the computing system, based on the predicted color at each of the plurality of points within the POI.

6. The computer-implemented method of claim 5, wherein the machine-learned POI representation model comprises a Neural Radiance Field (NeRF) neural network, and wherein processing the plurality of three-dimensional coordinates comprises: The plurality of three-dimensional coordinates corresponding to the respective plurality of points within the POI are processed by the computing system using the NeRF neural network to obtain radiation information indicating predicted radiation at each of the plurality of points within the POI, and wherein the radiation information includes the color information.

7. The computer-implemented method of claim 6, wherein: Before processing the plurality of three-dimensional coordinates using the NeRF neural network, the method comprises: A ray casting process is performed by the computing system to obtain the plurality of three-dimensional coordinates corresponding to the plurality of points within the POI, wherein the ray casting process originates from a viewpoint of the virtual camera.

8. The computer-implemented method of claim 7, wherein the viewpoint of the virtual camera is based at least in part on the set of path images.

9. The computer-implemented method of claim 1 , wherein receiving the plurality of image sets of the POI comprises: The POI depiction image set is received by the computing system from the user computing device, wherein the POI depiction image set includes one or more depiction image subsets, the one or more depiction image subsets including at least one of the following: a forward-facing subset of depicted images, the forward-facing subset of depicted images comprising a plurality of forward-facing depicted images captured as the user traverses the POI in a forward direction, wherein each of the forward-facing depicted images depicts the POI from a viewpoint facing the forward direction; a rearward-facing subset of depicted images, the rearward-facing subset of depicted images comprising a plurality of rearward-facing depicted images captured as the user traverses the POI in a rearward direction opposite to the forward direction, wherein each of the rearward-facing depicted images depicts the POI from a viewpoint facing the rearward direction; or An inward-facing depicted image subset includes a plurality of inward-facing depicted images captured by the user while traversing the POI, wherein each of the inward-facing depicted images depicts the POI from a viewpoint facing perpendicular to the forward direction and the rearward direction.

10. The computer-implemented method of claim 9, wherein each of the one or more subsets of depicted images is captured while moving an image capture device in a spiral motion.

11. The computer-implemented method of claim 10, wherein each of the one or more subsets of depicted images is captured while an auto-exposure feature of the image capture device is activated.

12. The computer-implemented method of claim 1, wherein: Before training the machine-learned POI representation model using the plurality of image sets of the POI, the method comprises: processing, by the computing system, a plurality of images from the plurality of image sets of the POI to obtain a corresponding plurality of quality indicators; Based on the plurality of quality indicators, identifying, by the computing system, one or more images of insufficient quality from the plurality of images; and In response to identifying the one or more images of insufficient quality, the computing system causes remedial capture information to be sent to the user computing device, wherein the remedial capture information includes instructions to the user of the user computing device to recapture one or more sets of images including the one or more images of insufficient quality.

13. A user computing device comprising: one or more processor devices; as well as one or more tangible non-transitory computer-readable media storing computer-readable instructions that, when executed by the one or more processor devices, cause the one or more processor devices to perform operations comprising: obtaining capture information from a computing system, the capture information comprising instructions to a user of the user computing device to capture a plurality of sets of images of a point of interest (POI); In response to obtaining the capture information, obtaining the plurality of image sets of the POI, wherein the plurality of image sets are captured by the user via an image capture device, and wherein the plurality of image sets include: a calibration image set, the calibration image set comprising a plurality of calibration image pairs, each of the calibration image pairs comprising two calibration images captured at two different locations within the POI separated by a predefined distance; a POI depiction image set, the POI depiction image set comprising a plurality of depiction images, the plurality of depiction images depicting the POI from a corresponding plurality of angles and / or positions; and a path image set, the path image set comprising a plurality of path images, the plurality of path images collectively depicting a path from a starting position to an ending position within the POI; performing a quality verification process for at least some of the plurality of image sets; In response to performing the quality verification process, transmitting capture information to the computing system, wherein the capture information includes the plurality of image sets of the POI; and Obtaining from the computing system video data depicting a virtual camera traversing the path within the POI from the starting location to the ending location, wherein the video data is generated by the computing system using a machine-learned POI representation model that is trained to generate a representation of the POI based on the plurality of image sets.

14. The user computing device of claim 13, wherein: The POI includes a building, the building includes a plurality of areas; and Obtaining the plurality of image sets of the POI comprises: The path image set is obtained, the path image set comprising the plurality of path images, the plurality of path images collectively depicting the path from the starting location to the ending location within the POI, wherein the starting location comprises a first area among the plurality of areas, and wherein the ending location comprises a second area among the plurality of areas that is different from the first area.

15. The user computing device of claim 14, wherein obtaining the plurality of image sets of the POI further comprises: obtaining a second path image set, the second path image set comprising a second plurality of path images, the second plurality of path images collectively depicting a second path within the POI from a second starting position to a second ending position, wherein the second starting position comprises the second area of ​​the plurality of areas, and wherein the second ending position comprises a third area of ​​the plurality of areas; and Wherein obtaining the video data depicting the virtual camera traversing the path from the starting position to the ending position comprises: A plurality of video data sets are obtained from the computing system, the plurality of video data sets comprising a first video data set depicting a virtual camera traversing the path from the first area to the second area, and a second video data set depicting the virtual camera traversing the second path from the second area to the third area.

16. The user computing device of claim 13, wherein obtaining the plurality of image sets of the POI comprises: The POI image depiction set is obtained, wherein the POI image depiction set includes one or more depiction image subsets, and the one or more depiction image subsets include at least one of the following: a forward-facing subset of depicted images, the forward-facing subset of depicted images comprising a plurality of forward-facing depicted images captured as the user traverses the POI in a forward direction, wherein each of the forward-facing depicted images depicts the POI from a viewpoint facing the forward direction; a rearward-facing subset of depicted images, the rearward-facing subset of depicted images comprising a plurality of rearward-facing depicted images captured as the user traverses the POI in a rearward direction opposite to the forward direction, wherein each of the rearward-facing depicted images depicts the POI from a viewpoint facing the rearward direction; or An inward-facing depicted image subset includes a plurality of inward-facing depicted images captured by the user while traversing the POI, wherein each of the inward-facing depicted images depicts the POI from a viewpoint facing perpendicular to the forward direction and the rearward direction.

17. The user computing device of claim 16, wherein each of the one or more subsets of depicted images is captured while moving the image capture device in a spiral motion.

18. The user computing device of claim 13, wherein performing the quality verification process for the at least some of the plurality of image sets comprises: processing a plurality of images from the plurality of image sets of the POI to obtain a corresponding plurality of quality indicators; Based on the plurality of quality indicators, identifying one or more images of insufficient quality from the plurality of images; responsive to identifying the one or more images of insufficient quality, causing instructions to be displayed to the user of the user computing device to recapture a set of images from the plurality of sets of images that includes the one or more images of insufficient quality; as well as A replacement set of images is obtained for the set of images including the one or more images of insufficient quality.

19. The user computing device of claim 13, wherein performing the quality verification process for the at least some of the plurality of image sets comprises: receiving remedial capture information from the computing system, the remedial capture information comprising instructions for display to the user of the user computing device, wherein the instructions instruct the user to recapture one or more sets of images including the one or more images of insufficient quality; in response to receiving the remedial capture information, causing instructions to be displayed to the user of the user computing device to recapture the set of images including the one or more images of insufficient quality; as well as A replacement set of images is obtained for the set of images including the one or more images of insufficient quality.

20. One or more tangible non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processor devices, cause the one or more processor devices to perform operations comprising: causing capture information to be sent to a user computing device, wherein the capture information includes instructions to a user of the user computing device to capture a plurality of sets of images of a point of interest (POI); receiving the plurality of sets of images of the POI from the user computing device, wherein the plurality of sets of images comprises: a calibration image set, the calibration image set comprising a plurality of calibration image pairs, each of the calibration image pairs comprising two calibration images captured at two different locations within the POI separated by a predefined distance; a POI depiction image set, the POI depiction image set comprising a plurality of depiction images, the plurality of depiction images depicting the POI from a corresponding plurality of angles and / or positions; and a path image set, the path image set comprising a plurality of path images, the plurality of path images collectively depicting a path from a starting position to an ending position within the POI; training a machine-learned POI representation model using the plurality of image sets of the POI, wherein the machine-learned POI representation model is trained to generate a representation of the POI; and Video data is generated using the machine-learned POI representation model, the video data depicting a virtual camera traversing the path within the POI from the starting location to the ending location.

Images