System and method for 3D scene expansion and reconstruction

The system addresses the challenge of targeted content insertion in 2D/3D environments by using object identifiers and real-time bidding to ensure natural and effective placement of ads and media, enhancing audience engagement.

JP2024167218AInactive Publication Date: 2024-12-03RESONAI INC +2
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024126141
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-03-06
Filing Date
2024-08-01
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional methods struggle to efficiently identify and insert content into 2D or 3D environments in a targeted and natural manner, failing to account for audience preferences and accurately match replacement objects, and often result in incomplete or unnatural renderings.

Method used

A computer-implemented system that processes scenes to identify matching objects, inserts or replaces them in a natural way, and generates hybrid images or models, utilizing object image identifiers, real-time bidding, and scene segmentation to enhance content insertion.

Benefits of technology

Enables efficient, real-time, and audience-specific content insertion that appears natural within 2D or 3D environments, improving the effectiveness of advertising and media content delivery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024167218000001_ABST
    Figure 2024167218000001_ABST
Patent Text Reader

Abstract

To provide a visual input reconstruction system and method for generating, expanding and reconstructing a 2D or 3D scene.SOLUTION: In a visual input reconstruction system for enabling selective insertion of a content into the existing media content frame, a method accesses a memory storing an object image identifier associated with an object, transmits the object image identifier to a client device, receives a bid from a client device, determines a successful bid, receives image data of a successful bidder from a successful bid client device, stores the image data of the successful bidder in the memory, specifies an object insertion position within the existing media content frame, generates a media content frame processed by inserting rendition of the image data of the successful bidder into the object insertion position within the existing media content frame, and transmits the processed media content frame to a user device.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] I. CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 743,065, filed October 9, 2018, and U.S. Provisional Patent Application No. 62 / 814,513, filed March 6, 2019, both of which are incorporated herein by reference in their entireties.

[0002] II. Technical Fields The present disclosure generally relates to systems and methods for generating, augmenting, or reconstructing two-dimensional (2D) or three-dimensional (3D) scenes or images. More specifically, disclosed embodiments are directed to receiving a scene from an audiovisual environment and modifying, augmenting, or reconstructing one or more portions of the received scene. The scene may be from, for example, a virtual reality environment, an augmented reality environment, a mixed reality environment, a 2D or 3D video game environment, a 2D or 3D scan, a 2D or 3D still or video camera image or images, etc. [Background technology]

[0003] III. Background Information Advertisers and others may wish to place advertisements and other messages and images within 2D or 3D environments in a targeted and effective manner to reach an audience. Advertisers may include individuals or organizations seeking to sell products or services, political organizations with a message, religious organizations, representatives or agencies, non-profit organizations, and / or other individuals or organizations.

[0004] Media content providers may wish to offer advertisers competitive bidding opportunities. Media content providers may include game developers, video producers, streaming service providers, virtual reality providers, augmented reality or mixed reality providers, and / or any other media content providers. Additionally, media content providers and advertisers may wish to dynamically insert messages, images, and / or other data into content. Media content providers and advertisers may wish to insert content into media in real time for a targeted audience. In general, media content providers and advertisers may wish to place content in a natural way within a 2D or 3D environment, making the content appear to be part of the scene and not alien to it.

[0005] For example, a content provider may wish to offer competitive bidding opportunities to replace beverage containers displayed in a 2D or 3D environment with branded beverages. An advertiser may wish to place a particular brand of beverage in content delivered to a particular audience (e.g., diet soda for a health-conscious audience). An advertiser may wish to place a different brand of beverage in the same or different content when delivered to a different particular audience (e.g., energy drinks for a late-night audience). Summary of the Invention [Problem to be solved by the invention]

[0006] Conventional approaches for identifying and creating bidding opportunities and dynamically inserting content have several drawbacks. For example, conventional approaches may not be able to place advertisements based on their intended audience. Furthermore, conventional approaches may suffer from deficiencies when attempting to identify targeted media items to replace with matching alternatives from advertisers or other third parties. As another example, conventional approaches may not be able to fit advertising content into 2D or 3D environments in a natural way.

[0007] Therefore, given the shortcomings and problems of conventional approaches, there is a need for a flexible, non-traditional approach to efficiently, effectively, and in real time, identify and manage competitive bidding opportunities, and place content within 2D or 3D scenes for specific audiences and in a natural manner.

[0008] In some cases, a content provider (e.g., a game developer, advertiser, filmmaker, etc.) may wish to receive a scan of a real-world environment and develop a 3D model of that environment (i.e., a scene). The content provider may receive the scene from a memory device associated with the content provider or from a device associated with another party. In some cases, the content provider may wish to insert new objects into the scene. For example, the content provider may wish to create a scene that includes a living room based on a scan of the living room. The content provider may wish to insert a new chair into the scene or replace a scene chair with a new chair.

[0009] However, conventional methods of inserting content have drawbacks. For example, conventional approaches may not be able to accept an incomplete scan of an object (i.e., a scan that captures a partial representation) to generate a scene, modify a scene, or replace an object in a scene. Furthermore, conventional methods may not be able to combine aspects of one object with those of another object. For example, conventional methods may not be able to apply the texture of a new object (e.g., an object received from a memory device) to an object in a scene. Furthermore, conventional systems may not be able to identify which possible replacement object most closely matches an object in a scene and select the appropriate match. In general, conventional methods may not be able to render a scene modified to incorporate a new object in a natural-looking manner so that the incorporated object appears to be part of the scene.

[0010] Therefore, in view of the shortcomings and problems of conventional approaches, there is a need for a novel approach that can identify matching objects, insert the matching objects into a scene, and generate natural-looking modified scenes based on complete or incomplete scans.

[0011] A content provider or user may receive an image from a memory device associated with the content provider or user, or from a device associated with another party. In some cases, a content provider or user may receive a still image and wish to animate portions of the image to make the image appear more realistic or to draw attention to the image.

[0012] Conventional methods for inserting content, such as animation of an object, have drawbacks. Conventional methods may not be able to combine aspects of one object with another. For example, conventional methods may not be able to apply new object features (e.g., animation of moving parts of an object) to objects within an image or scene. In general, conventional methods may not be able to render a scene modified to incorporate a new object that includes moving parts in a natural-looking manner so that the incorporated object appears to be part of the image or scene.

[0013] Therefore, considering the shortcomings and problems of conventional approaches, there is a need for a non-conventional approach that may identify objects similar to an object in an image or scene, construct a movable version of the object using movement data associated with the similar objects, construct a hybrid image, and output the hybrid image for display.

[0014] In some cases, a content provider or user (e.g., a game developer, advertiser, filmmaker, game player, student, etc.) may receive a partial image of an object. Such rendering may be unusable. For example, a player may scan their room and want to play a 3D game based on the room in which they can move objects. Thus, the game needs to generate a room with objects similar to those in the player's room so that they can "see" views not available in the scan, the backsides of various moving objects. Thus, a content provider may desire the ability to complete the partial image to create a complete 3D model of the object for insertion into a scene within an augmented reality or virtual game environment.

[0015] However, conventional methods for creating 3D audiovisual content may suffer from deficiencies. For example, conventional approaches may be unable to transfer incomplete scans of an object (i.e., scans that capture a partial representation) into the 3D content. Furthermore, conventional methods may be unable to generate a complete image of an object based on an incomplete or partial image. More generally, conventional methods may be unable to render a scene modified to incorporate a partial image of an object in a natural-looking manner, such that the incorporated object appears complete and part of the scene.

[0016] Therefore, considering the shortcomings and problems of conventional approaches, there is a need for a non-conventional approach that can combine partial images of an object with additional information to construct a complete simulated 3D model and output the complete 3D model. [Means for solving the problem]

[0017] In many cases, a robot may interact with an environment to perform various operations. For example, a robotic vacuum cleaner may move around a room for cleaning purposes. As another example, a robotic lawn mower may move across a lawn or outdoor area to cut grass. In yet another example, an autonomous vehicle may be used to perform various operations on a workshop or industrial site, or a robot may be used on an automated assembly line to perform assembly, welding, etc. A robot may encounter, for example, one or more objects (e.g., chairs, tables, lamps, etc. in a room) or obstacles (e.g., rocks, faucets, etc. in a garden), or tools and fixtures (e.g., welding tools on an assembly or assembly line) in the environment associated with the robot.

[0018] It may be useful for a robot to recognize whether an object it encounters in its environment is movable or immovable. If the object is movable, it may also be useful for the robot to recognize the object's motion characteristics (distance moved, speed of movement, force of movement / force required for movement, acceleration characteristics, etc.). For example, it may be useful for a cleaning robot to recognize whether an object in a room is movable, and then the robot may be able to move the object from its current position so that the robot can clean the room more efficiently. The robot may recognize the amount or speed of the object's movement in the room in response to an applied stimulus, and the robot may adjust the magnitude and / or direction of the stimulus applied to the object to control the object's movement.

[0019] However, conventional methods of controlling robots in domestic, commercial, or industrial environments suffer from deficiencies. For example, conventional robots may be unable to determine whether an object is movable or the manner in which an object is movable in response to an external stimulus. As a result, the operation of such a robot may be limited to moving the robot around objects in the environment without interacting with the objects. This limitation may result in inefficient operation of the robot. For example, if a cleaning robot must constantly circle around an object, the time required to clean a room may be longer than if the robot moves the object to one side, making it easier for the robot to clean the room. As another example, if the robot cannot move the object, the cleaning robot may not be able to clean the area occupied by the object.

[0020] Therefore, in view of the shortcomings and problems of conventional approaches for controlling robots, there is a need for a non-conventional approach that can identify one or more objects in an environment associated with a robot, determine the movement characteristics of the identified objects, and control the robot to interact with the objects based on the movement characteristics.

[0021] In some cases, a media content provider may receive a scan of a real-world environment and wish to develop a 3D model of the environment (i.e., a scene). The media content provider may receive the scan of the scene from a device associated with the media content provider or from a device associated with another. The content provider may wish to insert new objects into the scene. For example, a content provider may receive a living room scene and wish to add to the scene additional complementary objects that typically form part of a living room (e.g., a vase, a coffee table, etc.). In other cases, the content provider may wish to replace objects that are already present in the room (e.g., replacing the scene chair with a new chair of a different type or design).

[0022] A consumer user may wish to create a scene using additional objects that complement objects that are in the scene. For example, a user may receive a scene of a portion of the user's home (e.g., a backyard). The user may want to receive suggestions of additional, complementary objects that are suitable for the current objects found in the scene. For example, a user may be shopping for garden furniture (e.g., a hammock, umbrella, etc.). The user may wish to receive suggestions of complementary objects and may also wish to preview the scene including the complementary objects, for example, before making a purchase.

[0023] However, traditional methods of generating 3D content may suffer from deficiencies. For example, traditional approaches may fail to correctly identify and suggest related objects that are typically associated with objects found in a particular scene. Traditional approaches may also fail to identify appropriate complementary objects that are typically found in a particular scene through user interaction.

[0024] Therefore, considering the shortcomings and problems of conventional approaches, there is a need for a non-conventional approach that can identify complementary objects, insert complementary objects into a scene, and generate a hybrid scene based on scanning the scene.

[0025] Advertisers and others may wish to place advertisements or other messages or images within 2D or 3D environments in a targeted and effective manner to reach audiences. Advertisers may include individuals or organizations seeking to sell products or services, political organizations with a message, religious organizations, representatives or agencies, non-profit organizations, and / or other individuals or organizations.

[0026] Media content providers may wish to offer advertisers competitive bidding opportunities. Media content providers may include game developers, video producers, streaming service providers, virtual reality providers, augmented reality or mixed reality providers, and / or other media content providers. Additionally, media content providers and advertisers may wish to dynamically insert messages, images, and / or other data into broadcast content. Media content providers and advertisers may wish to insert content into media in real time for a targeted audience. In general, media content providers and advertisers may want to place content in a natural way within a 3D environment, making the content appear to be part of the scene and not alien to it.

[0027] For example, a content provider may wish to offer competitive bidding opportunities to replace a beverage container displayed in a 3D broadcast scene with a branded beverage. An advertiser may wish to place a particular brand of beverage in a broadcast scene delivered to a particular audience (e.g., diet soda for a health-conscious audience). An advertiser may wish to place a different brand of beverage in the same or different broadcast scene when delivered to a different particular audience (e.g., energy drink for a late-night audience).

[0028] Conventional approaches for identifying and creating bidding opportunities and dynamically inserting content have several drawbacks. For example, conventional approaches may not be able to place advertisements based on the intended audience. Furthermore, conventional approaches may suffer from deficiencies when attempting to identify targeted media items to replace with matching alternatives from advertisers or other third parties. As another example, conventional approaches may not be able to naturally fit advertising content into the 3D environment of an existing broadcast 3D scene.

[0029] Therefore, given the shortcomings and problems of conventional approaches, a flexible, non-traditional approach is needed to efficiently and effectively identify and manage competitive bidding opportunities and place content in 3D scenes in a manner that is relevant to specific audiences and natural to them. [Brief explanation of the drawings]

[0030] IV. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In such drawings:

[0031] [Figure 1]FIG. 1 depicts an exemplary system for augmenting or reconstructing a 2D or 3D scene or image consistent with an embodiment of the present disclosure.

[0032] [Figure 2] FIG. 2 illustrates an exemplary computing device consistent with embodiments of the present disclosure.

[0033] [Figure 3] FIG. 3 depicts an exemplary system for selecting bids from advertisers and inserting images corresponding to winning bids into existing scenes from an audiovisual environment, consistent with embodiments of the present disclosure.

[0034] [Figure 4] FIG. 4 depicts an exemplary method for selecting and inserting advertising images into an existing scene from an audiovisual environment, consistent with an embodiment of this disclosure.

[0035] [Figure 5] FIG. 5 depicts an exemplary method for enabling selective insertion of content into an existing content frame, consistent with embodiments of the present disclosure.

[0036] [Figure 6] FIG. 6 depicts an exemplary method for placing a bid for inserting content into an existing content frame consistent with an embodiment of the present disclosure.

[0037] [Figure 7] FIG. 7 depicts an exemplary method for selecting a 3D model to replace a CAD object in a scene, consistent with an embodiment of this disclosure.

[0038] [Figure 8] FIG. 8 depicts an exemplary method for selecting a 3D model and replacing a CAD object in an existing scene with the selected 3D model, consistent with an embodiment of the present disclosure.

[0039] [Figure 9]FIG. 9 illustrates an exemplary method for generating a 3D scene consistent with embodiments of the present disclosure.

[0040] [Figure 10] FIG. 10 illustrates an exemplary system for augmenting, reconstructing, and providing animation to a 2D or 3D scene or image, consistent with embodiments of the present disclosure.

[0041] [Figure 11] FIG. 11 depicts an example object (eg, a fan) in an input 3D scene being viewed by a user, consistent with an embodiment of this disclosure.

[0042] [Figure 12] FIG. 12 depicts replacing one or more portions of an object from an input 3D scene with corresponding portions of an object in a data structure containing an animation, consistent with an embodiment of the present disclosure.

[0043] [Figure 13] FIG. 13 depicts a flowchart of a process for animating portions of a still image consistent with an embodiment of the present disclosure.

[0044] [Figure 14] FIG. 14 illustrates an example of meshing a partial image to simulate a complete 3D model, consistent with an embodiment of the present disclosure.

[0045] [Figure 15] FIG. 15 illustrates an exemplary method for meshing partial images to simulate a complete 3D model, consistent with embodiments of the present disclosure.

[0046] [Figure 16] FIG. 16 illustrates an exemplary system for controlling a robot consistent with embodiments of the present disclosure.

[0047] [Figure 17]FIG. 17 illustrates an exemplary method for controlling a robot based on properties of segmented objects from a scene, consistent with embodiments of the present disclosure.

[0048] [Figure 18] FIG. 18 illustrates an exemplary method for controlling a robot based on movement properties associated with objects in the robot's environment, consistent with embodiments of the present disclosure.

[0049] [Figure 19] FIG. 19 illustrates an exemplary system for generating 3D content consistent with embodiments of the present disclosure.

[0050] [Figure 20] FIG. 20 depicts an exemplary method for automating 3D content creation for adding complementary objects to a scene, consistent with embodiments of the present disclosure.

[0051] [Figure 21] FIG. 21 illustrates an exemplary method for selecting complementary objects and combining 3D representations of complementary objects in an existing scene, consistent with embodiments of this disclosure.

[0052] [Figure 22] FIG. 22 illustrates an exemplary method for identifying at least one complementary object, consistent with embodiments of the present disclosure.

[0053] [Figure 23] FIG. 23 depicts an exemplary method for selecting and inserting advertising images into a broadcast scene from an audiovisual environment, consistent with embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0054] V. Overview Some disclosed embodiments include a computer-implemented visual input reconfiguration system for enabling selective insertion of content into an existing content frame. The system may include at least one processor. The processor may be configured to access a memory storing a plurality of object image identifiers associated with a plurality of objects. The processor may be configured to transmit at least one object image identifier from the plurality of object image identifiers to one or more client devices. The processor may be configured to receive one or more bids associated with the at least one object image identifier from the one or more client devices. The processor may be configured to determine a winning bid from the one or more received bids, and the winning bid is associated with a winning client device from among the one or more client devices. The processor may be configured to receive image data of a winning bidder from a winning client device and store the image data of the winning bidder in memory. The processor may be configured to identify an object insertion location for an object corresponding to the at least one object image identifier in at least one existing media content frame. The processor may be configured to generate at least one processed media content frame by processing at least one existing media content frame to insert at least a rendition of the winning bidder's image data at the object insertion location. The processor may be configured to transmit the at least one processed media content frame to one or more user devices.

[0055] In some embodiments, the at least one object image identifier comprises at least one of a shape, a shape descriptor, a product, or a product descriptor.

[0056] In some embodiments, the existing media content frames include at least one of a still image, a series of video frames, a series of virtual three-dimensional content frames, or a hologram.

[0057] In some embodiments, the at least one processor is further configured to perform image processing on the image data of the winning bidder (winner) to make the image data of the winning bidder (winner) compatible with the format of existing media content frames.

[0058] In some embodiments, the at least one existing media content frame comprises a plurality of frames that make up a virtual reality field of view, and the insertion renders an object from the winning image data within the plurality of frames.

[0059] In some embodiments, the transmitting includes transmitting over a network.

[0060] In some embodiments, the transmitting includes transmitting the processed media content frame to a first user device of the one or more user devices, wherein at least one processor is configured to transmit at least one existing media content frame to a second user device in a manner that excludes image data of a winning bidder.

[0061] In some embodiments, the image data of the winning bidder (winner) is inserted into at least one existing media content frame, such that the image data of the winning bidder (winner) is overlaid on existing content within the at least one existing media content frame.

[0062] In some embodiments, the image data of the winning bidder (winner) is inserted into at least one existing media content frame such that the image data object of the winning bidder (winner) replaces existing content in the at least one existing media content frame.

[0063] In some embodiments, the image data of the winning bidder (winner) is inserted into the portion of the object corresponding to at least one object image identifier.

[0064] In some embodiments, the processor is further configured to receive instructions from a winning client device, the instructions including a size limit for the object corresponding to the at least one object image identifier, and the inserting of the at least one rendition of the winning client's image data is based on the instructions.

[0065] In some embodiments, the object corresponding to the at least one object image identifier comprises at least one of a wall, a billboard, a picture frame, or a window.

[0066] In some embodiments, the image data of the winning bidder displayed in the existing media content frame changes after a predetermined period of time.

[0067] In some embodiments, the processor is further configured to obtain at least one existing media content frame in real time and insert a rendition of the image data of the winning bidder into the at least one existing media content frame in real time.

[0068] Further disclosed embodiments include a computer-implemented method for enabling selective insertion of content into an existing content frame. The method may include accessing a memory storing a plurality of object image identifiers associated with a plurality of objects. The method may include transmitting at least one object image identifier of the plurality of object image identifiers to one or more client devices. The method may include receiving one or more bids associated with the at least one object image identifier from the one or more client devices. The method may include determining a winning bid from among the one or more received bids. The winning bid may be associated with a client device that wins from among the one or more client devices. The method may include receiving image data of a winning bidder from a winning client device and storing the image data of the winning bidder in a memory. The method may include identifying an object insertion location in at least one existing media content frame for an object corresponding to the at least one object image identifier. The method may include generating at least one processed media content frame by processing at least one existing media content frame to insert at least a rendition of the winning bidder's image data at an object insertion location. The method may include transmitting the at least one processed media content frame to one or more user devices.

[0069] Additional disclosed embodiments include a non-transitory computer-readable storage medium capable of storing program instructions that, when executed by at least one processor, cause the at least one processor to perform operations that enable selective insertion of content into an existing content frame. The operations may include accessing a memory that stores a plurality of object image identifiers associated with a plurality of objects. The operations may include transmitting at least one object image identifier of the plurality of object image identifiers to one or more client devices. The operations may include receiving one or more bids associated with the at least one object image identifier from the one or more client devices. The operations may include determining a winning bid from the one or more received bids. The winning bid may be associated with a winning client device from among the one or more client devices. The operations may include receiving image data of a winning bidder from a winning client device and storing the image data of the winning bidder in a memory. The operations may include identifying an object insertion location for an object corresponding to the at least one object image identifier in at least one existing media content frame. The operations may include generating at least one processed media content frame by processing at least one existing media content frame to insert at least a rendition of the winning bidder's image data at the object insertion location. The operations may include transmitting the at least one processed media content frame to one or more user devices.

[0070] An additional disclosed embodiment includes a system for generating a three-dimensional (3D) scene. The system may include at least one processor. The processor may be configured to receive a scene based on a scan. The scene may include at least one object. The processor may be configured to process image elements in the scene to segment the scene into scene components. The image elements may include at least one of voxels, points, or polygons. The processor may be configured to identify matched components from among the scene components based on a comparison of the scene components with stored image data. The matched components may correspond to components of the at least one object. The processor may be configured to identify image elements corresponding to the at least one object based on the matched components. The processor may be configured to retrieve a CAD model from a storage location based on the image elements corresponding to the at least one object. The processor may be configured to generate a modified scene by combining the CAD model of the object with the scene. The processor may be configured to output the modified scene for 3D display.

[0071] In some embodiments, the modified scene is a hybrid scene that includes at least a portion of the CAD model and at least a portion of the at least one object.

[0072] In some embodiments, the scan is an incomplete scan and the modified scene includes refinements of the scene based on the semantics of the CAD model.

[0073] In some embodiments, at least one processor is configured to access semantics associated with the CAD model, where the semantics include a script that describes a movable characteristic of at least one object, and to apply the script to the CAD model in the hybrid scene, where the script is configured to be executed to make the object movable in the hybrid scene.

[0074] In some embodiments, the hybrid scene containing the script is output for 3D viewing.

[0075] In some embodiments, the at least one processor is further configured to select another script associated with the object, where the another script represents an interaction between the object and at least one other object in the scene, and apply the script to the CAD model in the hybrid scene.

[0076] In some embodiments, the at least one processor is configured to extract material properties from the matched component and to apply the extracted material properties to the CAD model.

[0077] Additional disclosed embodiments include a computer-implemented method for generating a three-dimensional (3D) scene. The method may include receiving a scene based on a scan. The scene may include at least one object. The method may include processing image elements in the scene to segment the scene into scene components. The image elements may include at least one of voxels, points, or polygons. The method may include identifying matched components from the scene components based on a comparison of the scene components to stored image data. The matched components may correspond to components of the at least one object. The method may include identifying image elements corresponding to the at least one object based on the matched components. The method may include retrieving a CAD model from a storage location based on the image elements corresponding to the at least one object. The method may include generating a modified scene by combining the object and the CAD model of the scene. The method may include outputting the modified scene for 3D display.

[0078] In some embodiments, the method further includes accessing semantics associated with the CAD model, where the semantics include a script that describes a mobility characteristic of at least one object, and applying the script to the CAD model within the hybrid scene, where the script is configured to be executed to make the object movable within the hybrid scene.

[0079] In some embodiments, the method further includes selecting another script associated with the object, where the another script represents an interaction between the object and at least one other object in the scene, and applying the script to the CAD model of the hybrid scene.

[0080] In some embodiments, the method further includes extracting material properties from the matching component and applying the extracted material properties to the CAD model.

[0081] Additional disclosed embodiments include a non-transitory computer-readable storage medium capable of storing program instructions that, when executed by at least one processor, cause the at least one processor to perform operations that enable generation of a 3D scene. The operations may include receiving a scene based on a scan. The scene may include at least one object. The operations may include processing image elements in the scene to segment the scene into scene components. The image elements may include at least one of voxels, points, or polygons. The operations may include identifying matched components from the scene components based on a comparison of the scene components with stored image data. The matched components may correspond to components of the at least one object. The operations may include identifying image elements corresponding to the at least one object based on the matched components. The operations may include retrieving a CAD model from a storage location based on the image elements corresponding to the at least one object. The operations may include generating a modified scene by combining the object and the CAD model of the scene. The operations may include outputting the modified scene for 3D viewing.

[0082] An additional disclosed embodiment includes a computer-implemented system for animating portions of a still image. The system may include at least one processor. The processor may be configured to receive a still image of an object. The processor may be configured to perform a lookup to identify at least one image of a similar object stored in a memory. The memory may include segmentation data distinguishing moving portions from stationary portions in the stored images of the similar object, and movement data associated with the moving portions. The processor may be configured to perform an analysis of voxels in the received still image of the object to segment the still image into discrete components. The processor may be configured to compare the discrete components with moving portions of the at least one similar object to identify at least one static rendering of a moving discrete component in the received image that is distinct from the stationary components in the still image. The processor may be configured to extract a static rendering of the moving discrete component from the still image. The processor may be configured to construct a moving version of the static rendering of the moving component using the static rendering and the movement data. The processor may be configured to construct a hybrid image by combining a constructed movable version of the stationary rendering of the movable component with a stationary component of the still image such that the stationary component from the still image remains stationary while the movable discrete component is movable within the hybrid image.

[0083] In some embodiments, the still image includes a person's head, the discrete components include the person's head and hair, and the at least one processor is configured to cause the head to remain stationary and the hair to move in the hybrid image.

[0084] In some embodiments, the still image includes a body of water, the discrete components include waves and a coastline, and the at least one processor is configured to cause the coastline to remain stationary and the waves to move in the hybrid image.

[0085] In some embodiments, the still image includes a tree, the discrete components include a trunk and leaves, and the at least one processor is configured to leave the trunk stationary and move the leaves in the hybrid image.

[0086] In some embodiments, the still image includes a person, the discrete components include a body of the person and an item of clothing, and the at least one processor is configured to cause the body to remain stationary and the item of clothing to move in the hybrid image.

[0087] In some embodiments, the still image includes a clock and the discrete components include a face and a hand, and the at least one processor is configured to cause the clock to display different times in the hybrid image.

[0088] In some embodiments, the still image includes a pet, the discrete components include a body and fur, and the at least one processor is configured to keep the body stationary and move the fur in the hybrid image.

[0089] In some embodiments, the still image includes an animal, the discrete components include a body and a tail, and the at least one processor is configured to keep the body stationary and move the tail in the hybrid image.

[0090] In some embodiments, the moving parts in the stored image of the similar object include a plurality of moving parts, and the at least one processor is further configured to receive a selection of a moving part selected from among the moving parts, compare the discrete components to the selected moving parts to identify at least one of the static renderings of the moving discrete components, construct a hybrid image using the static rendering and movement data, a moving version of the static rendering of the selected moving component, and construct a hybrid image by combining the non-moving components of the static image with the moving version of the static rendering of the selected moving component.

[0091] In some embodiments, the at least one processor is configured to detect multiple moving parts and prompt a user for a selection.

[0092] In some embodiments, the movement data is configurable by the user.

[0093] Additional disclosed embodiments include a computer-implemented method for animating portions of a still image. The method may include receiving a still image of an object. The method may include performing a lookup to identify at least one image of a similar object stored in a memory. The memory may include segmentation data that distinguishes moving portions from stationary portions in the stored images of the similar object. The memory may include movement data associated with the moving portions. The method may include performing an analysis of voxels in the received still image of the object to segment the still image into discrete components. The method may include comparing the discrete components to the moving portions of the at least one similar object to identify at least one static rendering of a moving discrete component in the received image that is distinct from the stationary components of the still image. The method may include extracting a static rendering of the moving discrete component from the still image. The method may include constructing a moving version of the static rendering of the moving component using the static rendering and the movement data. The method may include constructing a hybrid image by combining a constructed movable version of the stationary rendering of the movable component with a non-moving component of the stationary image such that a movable version of the movable discrete component in the hybrid image can thereby move within the hybrid image, while the non-moving component from the stationary image remains stationary. The method may include outputting the hybrid image.

[0094] In some embodiments, outputting the hybrid image includes displaying the hybrid image.

[0095] In some embodiments, outputting the hybrid image includes saving the hybrid image.

[0096] In some embodiments, outputting the hybrid image includes transmitting the hybrid image.

[0097] In some embodiments, the first and second objects are similar.

[0098] In some embodiments, the first and second objects are substantially different.

[0099] In some embodiments, the first and second objects are selected by a user.

[0100] In some embodiments, the movement data is configurable by the user.

[0101] Additional disclosed embodiments include a non-transitory computer-readable storage medium capable of storing program instructions that, when executed by at least one processor, cause the at least one processor to perform operations that enable animating portions of a still image. The operations may include receiving a still image of an object. The operations may include performing a lookup to identify at least one image of a similar object stored in a memory. The memory may include segmentation data that distinguishes moving portions from stationary portions in the stored images of the similar object. The memory may include movement data associated with the moving portions. The operations may include performing an analysis of voxels in the received still image of the object to segment the still image into discrete components. The operations may include comparing the discrete components to moving portions of the at least one similar object to identify at least one stationary rendering of a moving discrete component in the received image that is distinct from the stationary components of the still image. The operations may include extracting the stationary rendering of the moving discrete component from the still image. The operations may include constructing, using the static rendering and the moving data, a movable version of the static rendering of the movable component. The operations may include constructing a hybrid image by combining the constructed movable version of the static rendering of the movable component with an immovable component of the static image such that the movable version of the movable individual component moves thereby in the hybrid image, while the immovable component from the static image remains immovable. The operations may include outputting the hybrid image.

[0102] Additional disclosed embodiments include a computer-implemented system for simulating a complete 3D model of an object from incomplete 3D data. The system may include at least one processor. The processor may be configured to receive a partial image of the object, the partial image being at least one of a 2D image or an incomplete 3D image. The processor may be configured to use the partial image to search at least one data structure for additional information corresponding to the partial image. The processor may be configured to determine that the data structure does not contain a corresponding 3D model of the object. The processor may be configured to search at least one data structure for a reference 3D model that is different from but similar to the object in the partial image. The processor may be configured to compare the partial image with the reference 3D model to determine portions of the 3D reference model that generally correspond to missing characteristics of the partial image. The processor may be configured to combine the partial image with the additional information to construct a simulated complete 3D model of the object. The processor may be configured to output the simulated complete 3D model for display on a display device.

[0103] In some embodiments, the additional information includes a 3D model corresponding to the received sub-image.

[0104] In some embodiments, the additional information includes information obtained from a partial scan of at least one object similar to the object in the partial image.

[0105] In some embodiments, combining includes meshing the partial image with the determined portion of the 3D reference model.

[0106] In some embodiments, the at least one processor is further configured to identify at least one of a color and a texture of the partial image and, when cutting into the mesh, apply the at least one of the color and the texture to the determined portion of the 3D reference model.

[0107] In some embodiments, the at least one processor is configured to export the complete simulated 3D model to a format compatible with the 3D consumable environment.

[0108] In some embodiments, the 3D consumable environment includes at least one of a virtual reality environment and an augmented reality environment.

[0109] In some embodiments, the at least one processor is further configured to receive an input for rotating the simulated complete 3D model by an angle within a range between about 0° and about 360°; rotate the simulated complete 3D model based on the input; and display the rotated simulated complete 3D model on a display device.

[0110] In some embodiments, the at least one processor is further configured to receive an input for scaling the simulated full 3D model, scale the simulated full 3D model based on the input, and display the scaled simulated full 3D model on a display device.

[0111] Additional disclosed embodiments include a computer-implemented method for simulating a complete 3D model of an object from incomplete 3D data. The method may include receiving a partial image of the object, the partial image being at least one of a 2D image or an incomplete 3D image. The method may include searching at least one data structure for additional information corresponding to the partial image. The method may include determining that the data structure does not contain a corresponding 3D model of the object. The method may include searching at least one data structure for a reference 3D model that is different from the object in the partial image but has similarity to the object in the partial image, the reference 3D model including the additional data. The method may include comparing the partial image to the reference 3D model to determine a portion of the 3D reference model that generally corresponds to the missing characteristic of the partial image. The method may include combining the partial image with the additional information, the additional data, or a combination of the additional information and the additional data to construct a simulated complete 3D model of the object. The method may include outputting the simulated complete 3D model for display on a display device.

[0112] In some embodiments, the method may further include receiving an input for rotating the simulated complete 3D model through an angle ranging between about 0° and about 360°; rotating the simulated complete 3D model based on the input; and displaying the rotated simulated complete 3D model on a display device; and displaying the rotated simulated complete 3D model on the display device.

[0113] In some embodiments, the method further includes receiving an input for scaling the simulated full 3D model, scaling the simulated full 3D model based on the input, and displaying the scaled simulated full 3D model on a display device.

[0114] Additional disclosed embodiments include a non-transitory computer-readable medium that may store program instructions that, when executed by at least one processor, may cause the at least one processor to perform operations that enable simulating a complete 3D model of an object from incomplete 3D data. The operations may include receiving a partial image of the object, the partial image being at least one of a 2D image or an incomplete 3D image. The operations may include searching at least one data structure for additional information corresponding to the partial image. The operations may include determining that the data structure does not include a corresponding 3D model of the object. The operations may include searching at least one data structure for a reference 3D model that is different from but similar to the object in the partial image, the reference 3D model including the additional data. The operations may include comparing the partial image to the reference 3D model to determine a portion of the 3D reference model that generally corresponds to the missing characteristic of the partial image. The operations may include combining the partial images with additional information, additional data, or a combination of additional information and additional data to construct a simulated complete 3D model of the object. The operations may include outputting the simulated complete 3D model for display on a display device.

[0115] An additional disclosed embodiment includes a control system for a robot. The system may include at least one processor. The processor may be configured to receive image information of a scene depicting an environment associated with the robot. The processor may be configured to segment the scene to extract an image of at least one object within the scene. The processor may be configured to access a data structure storing information about a plurality of objects. The processor may be configured to compare the extracted image with information in the data structure to identify corresponding information in the data structure regarding the at least one object. The corresponding information may include a script that describes mobility characteristics of the at least one object. The processor may be configured to control the robot by applying the script. Applying the script may cause the robot to interact with the at least one object based on the mobility characteristics defined in the script.

[0116] In some embodiments, the at least one processor is configured to segment the scene by processing image elements in the scene, the image elements comprising at least one of voxels, points, or polygons.

[0117] In some embodiments, the robot includes a camera configured to generate image information of the scene.

[0118] In some embodiments, the mobility characteristics include at least one rule that defines the movement of the at least one object based on an external stimulus.

[0119] In some embodiments, the at least one processor is configured to adjust an external stimulus exerted by the robot on the at least one object based on a movable characteristic of the at least one object.

[0120] In some embodiments, the at least one processor is configured to generate a modified scene based on an interaction of the robot with the at least one object.

[0121] In some embodiments, the at least one processor is configured to output the modified scene for display.

[0122] In some embodiments, the at least one processor is further configured to select another script associated with the at least one object, where the another script represents an interaction between the at least one object and at least one other object in the scene, and apply the script to the at least one object.

[0123] Additional disclosed embodiments include a computer-implemented method for controlling a robot. The method may include receiving image information of a scene depicting an environment associated with the robot. The method may include segmenting the scene to extract an image of at least one object in the scene. The method may include accessing a data structure storing information about a plurality of objects. The method may include comparing the extracted image with information in the data structure to identify corresponding information in the data structure regarding the at least one object. The corresponding information may include a script that describes mobility characteristics of the at least one object. The method may include controlling the robot by applying the script. Applying the script may cause the robot to interact with the at least one object based on mobility characteristics defined in the script.

[0124] In some embodiments, segmenting the scene comprises processing image elements within the scene, the image elements comprising at least one of voxels, points, or polygons.

[0125] In some embodiments, receiving the image information includes generating the image information of the scene using a camera associated with the robot.

[0126] In some embodiments, the method further includes adjusting an external stimulus exerted by the robot on the at least one object based on the movable characteristic of the at least one object.

[0127] In some embodiments, the method further includes generating a modified scene based on an interaction of the robot with the at least one object.

[0128] In some embodiments, the method further includes outputting the modified scene for display.

[0129] In some embodiments, the method further includes selecting another script associated with the at least one object, where the other script describes an interaction between the at least one object and at least one other object in the scene, and applying the script to the at least one object.

[0130] Additional disclosed embodiments include a non-transitory computer-readable storage medium capable of storing program instructions that, when executed by at least one processor, cause the at least one processor to perform operations for controlling a robot. The operations may include segmenting a scene to extract an image of at least one object in the scene. The operations may include accessing a data structure storing information about a plurality of objects. The operations may include comparing the extracted image with information in the data structure to identify corresponding information in the data structure regarding the at least one object. The corresponding information may include a script that describes mobility characteristics of the at least one object. The operations may include controlling the robot by applying the script. Applying the script may cause the robot to interact with the at least one object based on the mobility characteristics defined in the script.

[0131] An additional disclosed embodiment includes a system for automating three-dimensional (3D) content creation. The system may include at least one processor. The processor may be configured to receive a scan of a scene. The processor may be configured to segment the scan to identify at least one object in the scene. The processor may be configured to extract image data from the scan corresponding to the identified object. The processor may be configured to search at least one data structure using the extracted image data to identify at least one image of at least one complementary object to the identified object. The processor may be configured to retrieve a 3D representation of the at least one complementary object from the at least one data structure. The processor may be configured to generate a hybrid scene by combining the 3D representation of the at least one complementary object with portions of the scan of the scene other than those corresponding to the identified object. The processor may be configured to output the hybrid scene for presentation on a display device.

[0132] In some embodiments, the at least one image of the at least one complementary object includes a plurality of images of a plurality of complementary objects.

[0133] In some embodiments, the at least one processor is further configured to output for display an index of the plurality of images of the plurality of complementary objects.

[0134] In some embodiments, the at least one processor is further configured to receive a selection of at least one of the plurality of complementary objects from a user; and insert the selection into the scan of the scene.

[0135] In some embodiments, the extracted image data includes a classification of the identified object.

[0136] In some embodiments, the at least one processor identifies at least one complementary object based on the classification.

[0137] In some embodiments, the at least one processor is configured to generate a semantic tag for at least one identified object in the scene.

[0138] In some embodiments, the at least one processor is configured to compare semantic tags of the identified object with semantic tags of objects stored in the at least one data structure, and select at least one complementary object based on the comparison.

[0139] In some embodiments, the at least one data structure includes a 3D scene associated with the semantic tag.

[0140] Additional disclosed embodiments include a computer-implemented method for automating 3D content. The method may include receiving a scan of a scene. The method may include segmenting the scan to identify at least one object in the scene. The method may include extracting image data from the scan corresponding to the identified object. The method may include searching at least one data structure using the extracted image data to identify at least one image of at least one complementary object to the identified object. The method may include obtaining a 3D representation of the at least one complementary object from the at least one data structure. The method may include generating a hybrid scene by combining the 3D representation of the at least one complementary object with portions of the received scan other than those corresponding to the extracted object. The method may include outputting the hybrid scene for presentation on a display device.

[0141] In some embodiments, the method further includes outputting for display indices of the plurality of images of the plurality of complementary objects.

[0142] In some embodiments, the method includes receiving a selection of at least one of a plurality of complementary objects from a user and inserting the selection into a scan of the scene.

[0143] In some embodiments, the method further includes identifying at least one complementary object based on the classification.

[0144] In some embodiments, the method further comprises generating a semantic tag for at least one identified object in the scene.

[0145] In some embodiments, the method includes comparing semantic tags of the identified object with semantic tags of objects stored in at least one data structure, and selecting at least one complementary object based on the comparison.

[0146] In some embodiments, the operation further includes generating a semantic tag for at least one identified object in the scene.

[0147] Additional disclosed embodiments include a non-transitory computer-readable storage medium capable of storing program instructions that, when executed by at least one processor, cause the at least one processor to perform operations that enable automated 3D content creation. The operations may include segmenting the scan to identify at least one object within the scene. The operations may include extracting image data corresponding to the identified object from the scan. The operations may include searching at least one data structure using the extracted image data to identify at least one image of at least one complementary object to the identified object. The operations may include retrieving a 3D representation of the at least one complementary object from the at least one data structure. The operations may include generating a hybrid scene by combining the 3D representation of the at least one complementary object with portions of the received scan other than those corresponding to the extracted object. The operations may include outputting the hybrid scene for presentation on a display device.

[0148] An additional disclosed embodiment includes a computer-implemented system for adding 3D content to a 3D broadcast scene. The system may include at least one processor. The processor may be configured to display at least one 3D broadcast scene on a plurality of client devices. The processor may be configured to display at least one tag on the client device, the tag corresponding to at least one object in the 3D broadcast scene. The processor may also be configured to display instructions on the client device for placing at least one bid on the at least one tagged object. Further, the processor may be configured to receive one or more bids for the at least one tagged object from the client devices. The processor may be configured to determine a winning bid from among the one or more received bids, the winning bid being associated with a winning client device from among the client devices. The processor may also be configured to receive, from the winning client device, image data of a winning bidder corresponding to the at least one tagged image. The processor may be configured to isolate 3D image data corresponding to the at least one tagged object from the 3D broadcast scene. Further, the processor may be configured to generate a 3D hybrid rendering of the tagged object by combining the image data of the winning bidder with the extracted 3D image data, and the processor may be configured to insert the hybrid rendering into a hybrid 3D broadcast scene.

[0149] In some embodiments, the 3D broadcast scene is part of a video game.

[0150] In some embodiments, the 3D broadcast scene is part of a 3D movie.

[0151] In some embodiments, the 3D broadcast is part of an online advertisement.

[0152] In some embodiments, the at least one processor is further configured to perform image processing on the winning bidder image data to make the winning bidder image data compatible with a 3D broadcast scene format.

[0153] In some embodiments, the 3D broadcast scene includes multiple frames, and the insertion renders objects from the winning image data in the multiple frames.

[0154] In some embodiments, the image data of the winning bidder is inserted into the 3D broadcast scene such that the image data of the winning bidder is overlaid on existing content in the 3D broadcast scene.

[0155] In some embodiments, the at least one processor is configured to generate a spatial semantic graph for each scene, compare the generated spatial semantic graph with spatial semantic graphs of scenes stored in a data structure, identify scenes within the data structure that have a spatial semantic graph similar to the generated spatial semantic graph, and determine information about the 3D broadcast scene based on the scenes identified in the data structure.

[0156] Additional disclosed embodiments include a computer-implemented method for adding 3D content to a 3D broadcast scene. The method may include displaying at least one 3D broadcast scene on a plurality of client devices. The method may also include displaying, on the client devices, at least one tag corresponding to at least one object in the 3D broadcast scene. Further, the method may include displaying, on the client devices, instructions for placing at least one bid on the at least one tagged object. The method may include receiving one or more bids for the at least one tagged object from one or more client devices. The method may include determining a winning bid from among the bids, the winning bid being associated with a winning client device from the client devices. The method may also include receiving, from the winning client device, image data of a winning bidder corresponding to the at least one tagged image. The method may also include isolating, from the 3D broadcast scene, 3D image data corresponding to the at least one tagged object. The method may include generating a 3D hybrid rendering of the tagged object by combining the winner's image data with the extracted 3D image data. The method may also include inserting the hybrid rendering into a hybrid 3D broadcast scene. Further, the method may include broadcasting the 3D hybrid broadcast scene.

[0157] In some embodiments, the method further includes generating a spatial semantic graph for each scene; comparing the generated spatial semantic graph with spatial semantic graphs of scenes stored in the data structure; identifying scenes having spatial semantic graphs similar to the generated spatial semantic graph; and determining information about the 3D broadcast scene based on the identified scenes in the data structure.

[0158] Additional disclosed embodiments include a non-transitory computer-readable storage medium capable of storing program instructions that, when executed by at least one processor, cause the at least one processor to perform operations that enable selective insertion of content into a 3D broadcast scene. The operations may include displaying at least one 3D broadcast scene on a plurality of client devices. The method may also include displaying, on the client devices, at least one tag corresponding to at least one object in the 3D broadcast scene. Further, the operations may include displaying, on the client devices, instructions for placing at least one bid on the at least one tagged object. The operations may include receiving, from one or more client devices, one or more bids for the at least one tagged object. The operations may include determining a winning bid from among the bids, the winning bid being associated with a winning client device from the client devices. The method may also include receiving, from the winning client device, image data of a winning bidder corresponding to the at least one tagged image. The method may also include isolating 3D image data corresponding to at least one tagged object from the 3D broadcast scene. The operations may include generating a 3D hybrid rendering of the tagged object by combining image data of a winning bidder with the extracted 3D image data. The method may also include inserting the hybrid rendering into the hybrid 3D broadcast scene. Additionally, the operations may include broadcasting the 3D hybrid broadcast scene.

[0159] The disclosed systems and methods may be implemented using conventional hardware and software, as well as combinations of specialized hardware and software, such as machines specially constructed and / or programmed to perform the functions associated with the disclosed method steps. The foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit the scope of the claims.

[0160] VI. Detailed Description Exemplary embodiments are described with reference to the accompanying drawings. Figures are not necessarily drawn to scale. While examples and features of the disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. Additionally, the words "comprising," "having," "containing," "including," and other similar forms are equivalent in meaning, and do not imply an exclusive list of such one or more items, or that the following one or more items is limited to only the listed one or more items. It should also be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise.

[0161] term Voxel: A voxel may be a closed n-sided polygon (e.g., a cube, pyramid, or any closed n-sided polygon). Voxels in a scene may be uniform or non-uniform in size. Voxels may be consistently shaped within a scene, or they may vary within a scene.

[0162] Mesh: In embodiments of the present disclosure, a mesh may include a set of polygons (e.g., triangles, quadrilaterals, etc.) in 2D or 3D space that represent the exterior surfaces of objects in a scene. A mesh may be defined by a set of control points and a set of n-sided polygons (polygons) created by the control points. An n-sided polygon (polygon) may have any number of n edges (faces). The edges of a polygon may be of any length, and the edge lengths may be irregular (i.e., the edge lengths may vary within a scene). A mesh may include any number of control points or polygons, such as millions of control points. In some embodiments, a mesh may be created by establishing a set of control points and the relationships between them. In some embodiments, generating a mesh may include inserting previously generated objects. A point cloud may include a set of data points in a coordinate system such as an orthogonal plane (e.g., X, Y, and Z coordinates). A mesh may be configured to be presented to a user from one or more perspectives, including viewpoint location, orientation, and / or zoom. A mesh or point cloud may be created based on a scan. More generally, generating a scene based on a scan may involve processing an image to represent it as a mesh, point cloud, or other format. For example, a scan may include taking a snapshot of the real world and digitally representing it in one of the forms described above. A mesh may include multiple voxels representing a scene or a voxel-mapping of a subset of space. A mesh or point cloud may be created using software such as WEBVR, VRMESH, MESHROOMVR, FARO SCENE, MESHMAKER, and / or other programs or applications configured to generate and / or edit meshes.A mesh or point cloud may be associated with computing code (ie, scripts) that defines the object, boundary, animation, or other properties of the mesh.

[0163] Segmenting or Segmentation: Consistent with embodiments of the present disclosure, segmentation may include any calculation or process that enables the separation of an image into objects or portions that can be identified by a system. In some embodiments, a still image may be segmented or characterized into discrete components or parts. Discrete components or parts may include all or part of an object. A scene or image may be segmented, for example, by associating groups of image elements (e.g., pixels, points, polygons, voxels, etc.) with each other and identifying the groups as associated with separate objects or portions of objects. During segmentation, these groups of elements may be assigned to objects. Segmentation may include tagging, labeling, identifying, or otherwise classifying one or more primitive elements of a scanned scene (e.g., triangles in 3D or points in 3D) as belonging to objects in the scene. That is, segmentation may involve dividing (i.e., classifying) image elements of a scene into objects. Segmentation may involve mapping a large number of points or polygons (e.g., hundreds of thousands of polygons) to multiple discrete components in a still image. Segmentation may involve implementing an object recognition algorithm and / or a machine learning model to map basic image elements (e.g., voxels) to one or more discrete components, and may involve implementing a segmentation algorithm.Examples of image segmentation algorithms and methods that can be used include, but are not limited to, thresholding methods, clustering methods, compression methods, histogram-based methods, edge detection, dual clustering methods, region growing methods; partial differential equation-based methods such as parametric methods, level set methods, and fast marching methods; calculus of variations; graph partitioning methods; watershed transformations; model-based methods; multi-scale methods; semi-automatic segmentation; trainable segmentation; and deep learning algorithms. Segmentation may be based on color, shape, contrast, and / or other features of the scene. Segmentation may include assigning probabilities that image elements belong to particular individual components. As an example, the system may segment an image into multiple discrete components. Segmentation may include implementing a deep learning model (e.g., a convolutional neural network model, e.g., PointNet) to classify primitives in the image. In some embodiments, segmentation may include generating multiple 2D representations of the scanned scene (i.e., 2D snapshots), where the 2D representations include projections of the 3D scanned scene into two dimensions from various angles. The 2D snapshots may include multiple pixels. A pixel may correspond to multiple image elements. For example, a set of image elements along a line of sight may correspond to pixels in the 2D snapshot (i.e., generating the 2D snapshot by mapping a set of image elements to pixels). The 2D snapshots may overlap (i.e., may include one or more of the same objects).Segmentation may include tagging, labeling, identifying, or otherwise classifying one or more pixels of the 2D snapshot as belonging to a distinct component and mapping the one or more pixels to a set of 3D elements (e.g., by performing an inverse mapping of the mapping used to generate the 2D snapshot). Segmentation may also include identifying components corresponding to known and / or unknown classifications and may include implementing a classification model.

[0164] Comparing Objects and / or Image Data: Consistent with disclosed embodiments, comparing objects and / or image data (e.g., complete or partial image representations, image elements, 3D models, CAD models, text, or other information associated with the images) may include implementing a classification algorithm (e.g., a machine learning model). Comparing objects and / or image data may include generating one or more criteria, such as a similarity metric, as described herein. Comparing objects and / or image data may include segmenting objects within a scene, i.e., identifying components of an object, such as faces, lines, or surfaces of the object, or components that are themselves objects (e.g., identifying wheels as component objects of a car object). Comparing objects and / or image data may be based on image object identifiers. Comparing objects and / or images may include comparing semantic tags or spatial semantic graphs associated with the objects and / or images. Comparing objects and / or images may include comparing feature vectors associated with the objects and / or images. Comparing may include comparing based on text data, shape data, user data, and / or other data. Such comparison may include, by way of example only, a statistical analysis of similarities or an artificial intelligence-based approach to identifying similarities. In one example, the comparison may include determining a similarity metric indicating the degree of similarity between the compared objects and / or images. For example, the disclosed system may generate or retrieve feature vectors corresponding to the compared objects and / or images and then compare the feature vectors. In some embodiments, the disclosed system may generate the feature vectors using, for example, multi-view neural networks or other types of neural networks.The disclosed system can determine the similarity between the compared objects and / or images based on a similarity metric. The similarity metric may include a score that represents some measure of the degree of similarity. The similarity metric may be based on statistical similarity such as covariance, least squares distance, or Hausdorff distance between aligned objects.

[0165] Generating a Combined or Hybrid Image (by Inserting, Merging, or Substituting Images): Consistent with disclosed embodiments, generating a combined or hybrid image may include combining some or all of a first image with some or all of a second image. Thus, for example, a processor may combine some or all of the image elements associated with a first image with some or all of the image elements associated with a second image. Generating a combined or hybrid image may include placing objects in the first image with the same orientation (i.e., aligning the objects) and similar size (i.e., scaling the objects) as objects in the second image. By way of example, aligning an object in the first image with an object in the second image may include an affine transformation that transforms the (x, y, z) coordinates of an image element in the first image to the desired location T(x, y, z) of that element in the second image, or vice versa. In other embodiments, generating a combined or hybrid image may include combining some or all of two families of image elements. In yet another embodiment, generating a combined or hybrid image may include combining characteristics of some or all image elements of a first image and some or all image elements of a second image to obtain a blended image. For example, suppose the first image and the second image contain a family of polygons. Each polygon may be associated with a texture. The texture may be a 2D mapping from the image to the polygon, representing how the polygon will appear to a viewer (e.g., different parts of the polygon may have different colors). The alignment T of the first and second images may be used to determine matching of corresponding polygon families. For example, a polygon from the first image may be mapped to a polygon in the second image using a transformation T to find the closest first image polygon to the polygon in the first image. Using the matching, the system may match vertices of polygons in the first and second images.The disclosed system may also transfer color, texture, material properties, etc. from polygons in a first image to polygons in a second image, or vice versa. In some embodiments, aligning objects and / or scaling objects may include the use of principal component analysis (PCA). Generating a combined or hybrid image may include the use of image processing techniques (e.g., adjusting brightness, adjusting lighting, implementing gradient domain methods, etc.) consistent with disclosed embodiments. As one skilled in the art will recognize, gradient domain methods may include constructing a hybrid image by integrating gradients of image elements in a first image with gradients of image elements in a second image, or vice versa. Generating a combined or hybrid image may also include transferring one or more properties, such as texture, material, mobility, color, shading, etc., from some or all image elements in a first image to some or all image elements in a second image, or vice versa. In some embodiments, generating the combined or hybrid image may further include generating additional image elements in the second image based on image elements in the first image, or vice versa.

[0166] Data Structures: Data structures consistent with this disclosure may include any collection of data values ​​and the relationships between them. Data may be stored linearly, horizontally, hierarchically, relationally, non-relationally, one-dimensionally, multi-dimensionally, operationally, in an ordered manner, an unordered manner, an object-oriented manner, a centralized manner, a decentralized manner, a distributed manner, a custom manner, or any manner that enables data access. By way of non-limiting examples, data structures may include arrays, associative arrays, linked lists, binary trees, balanced trees, heaps, stacks, queues, sets, hash tables, records, tagged unions, entity relationship models, and graphs. For example, a data structure may include an XML database, an RDBMS database, an SQL database, or a NoSQL alternative for data storage / retrieval, such as, for example, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. A data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). Data within a data structure may be stored in contiguous or non-contiguous memory. Furthermore, a data structure as used herein does not require that the information be co-located. For example, it may be distributed across multiple servers that may be owned or operated by the same entity or different entities. Thus, the term "data structure" as used herein in the singular includes multiple data structures.

[0167] Processor: Consistent with disclosed embodiments, "at least one processor" may refer to any physical device or group of devices having electrical circuitry that performs logical operations on an input or multiple inputs. For example, the at least one processor may include one or more integrated circuits (ICs) that include an application-specific integrated circuit (ASIC), a microchip, a microprocessor, all or part of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a server, a virtual server, or other circuitry suitable for performing logical operations or executing instructions. The instructions executed by the at least one processor may be preloaded into, for example, a memory integrated with or embedded in the controller, or may be stored in a separate memory.

[0168] Memory: Memory may include random access memory (RAM), read-only memory (ROM), a hard disk, an optical disk, a magnetic medium, flash memory, other permanent, fixed, or volatile memory, or other mechanism capable of storing instructions. In some embodiments, the at least one processor may include two or more processors. Each processor may have a similar structure or may be of different structures that are electrically connected or disconnected from one another. For example, the processors may be separate circuits or integrated into a single circuit. When multiple processors are used, the processors may be configured to operate independently or in concert. The processors may be coupled electrically, magnetically, optically, acoustically, mechanically, or by other means that allow them to interact.

[0169] The present disclosure relates to computer-implemented advertising bidding systems for use in virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies and applications. The present disclosure provides a solution for new types of advertising within AR, VR, and MR technologies and applications to match real-time market advertisement inventory or real-time generated supply with real-time consumer intent and deliver precise and effective targeting of advertisements. While the present disclosure provides examples of AR, VR, and MR technologies and applications, it should be noted that aspects of the disclosure in their broadest sense are not limited to specific examples. Rather, it is contemplated that the principles described above may be applied to other computerized reality technologies and applications.

[0170] In the following description, numerous specific details are provided to provide a more thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without one or more of these details.

[0171] The present disclosure generally relates to bidding for inserting content into a computerized environment. A bid may include offering something of value (e.g., money) in exchange for inserting the content. A bid may include specifying aspects of the bid itself (e.g., an offer period during which the bid is accepted). A bid may include specifying aspects of the inserted content, such as timing, a target audience, a duration, a monetary value, or other aspects of the inserted content. In some examples, a bid may include an object (e.g., a car, a bottle, a chair, a door), a type of scene (e.g., an office, a living room, an art gallery), a time (e.g., between 1:00 PM and 2:00 PM), or / and a specific targeted location (e.g., New York City, 5th Avenue), or a class of user (i.e., a class of viewers).

[0172] Consistent with the present embodiments, bidding can be real-time. For example, bidding can occur during transmission or other broadcast to an online audience. The disclosed system can be configured to determine which objects, scenes, or elements within a scene match inventory or real-time market supply based on a scene analysis or analysis of objects within the scene. Consistent with the disclosed embodiments, matching can be performed based on multiple extracted data or features of the scene, inventory, or real-time market supply.

[0173] The terms VR, AR, and MR generally refer to computerized environments that simulate or are modeled after physical environments, such as three-dimensional (3D) environments. AR, VR, and MR are technologies that allow users to interact in an immersive and natural way with actual scenes and objects (in the case of AR and MR) within the environment, as well as with information and functionality associated with virtual objects. VR, AR, and MR environments may or may not be based on data describing real-world physical environments. VR in its broadest sense may refer to immersive multimedia or computer-simulated reality, including AR and MR environments. An immersive multimedia experience may present information (e.g., via a headset) in a way that simulates an interactive real-world experience in 3D and may capture user input that corresponds to real-world input, such as gestures, gaze, speech, etc. AR and MR may refer to partially computerized or hybrid environments that may blend sensor data from the real world with digital objects, allowing users to interact with the digital objects in real time. VR, AR, and MR technologies may involve a combination of hardware, such as a head-mounted display, a computer, or other hardware for generating and displaying computerized reality.

[0174] This disclosure also relates to advertising. Advertising, broadly speaking, includes the act of displaying information to an audience via a medium, which may include displaying specific information to a specific audience via a specific medium. Clickable advertisements—display banners, images, videos, 3D models, 3D filters, audio, or text—are all types of advertising units that target specific audience segments or are highly personalized for specific users. The advertising industry may invest significant efforts in researching specific audiences and their use of specific media for targeted advertising through various analytical techniques, such as incorporating big data analysis, user interaction analysis, and various types of machine learning optimization techniques.

[0175] This disclosure may relate to broadcasting of AR, VR, and MR technologies. Broadcasting, as used in this disclosure, may include transmission to multiple individuals over a network. For example, broadcasting may include transmission to many players playing a multiplayer game or many viewers watching a sporting event. In general, broadcasting may include transmission to viewers exposed to the same or similar view of a real or virtual scene. Broadcasting may include transmission over the Internet, cable television, or any other medium for transmitting data to many users simultaneously.

[0176] In some embodiments, advertiser objects may be inserted into a transmission to all individuals receiving the broadcast. In some embodiments, advertiser objects may be inserted into a transmission to a subset of individuals receiving the broadcast. For example, advertiser objects may be inserted into or excluded from a viewer's transmission based on characteristics of the viewer's environment, such as time of day, country, language, etc., or characteristics of the viewer, such as gender and age.

[0177] In the following description, numerous specific details are provided to provide a more thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without one or more of these details.

[0178] The present disclosure is intended to provide techniques for targeting advertisements to specific audiences or specific users. This application can be used in environments consumed via devices equipped with AR, MR, or VR applications. The present disclosure can match observed scenes or portions of scenes, or even single or partial objects within a scene, in this novel way with a real-time bidding system that associates value with given objects that can be advertised within the AR, MR, or VR consumed scene.

[0179] The bidding system allows advertisers or advertising agents to place bids or associate suggested values ​​with a given object. Associating a value may be accomplished with or without specific filters such as time, location, scene description, etc. The bidding system may push specific objects or portions of scenes to advertisers or advertising agents so that they can associate their advertising units (e.g., banners, images, videos, or 3D models) and associate suggested values ​​with those specific advertising units.

[0180] In some implementations, an advertiser or advertising agent can associate an advertising unit with one or more given objects. For example, an automobile manufacturer can associate a particular banner, 3D model, image, or video with an object identified as a "sports car." Based on the value the manufacturer assigns to the object "sports car," the matching system can determine when, where, and to which users the advertisement will be displayed (i.e., the matching system can determine the scene, object, time, or location for displaying the advertisement).

[0181] A typical use case may include adding a banner to an actual sports car in an AR or MR environment in an immersive manner. Another use case may include adding a banner to a digitized sports car model in a VR consumed scene. Yet another use case may involve an AR or MR environment and may include replacing an actual sports car in the AR or MR environment with an advertised sports car, so that a user experiencing the AR or MR environment will see the advertised sports car displayed in place of the actual sports car. A similar application may replace an original model in a VR scene with an advertised model.

[0182] In a VR constructed scene for some embodiments of the present disclosure, the designer or creator of the scene can pre-determine which objects are good candidates for replacement or embedding of ad units.

[0183] In some demonstrative examples, an owner of a real object, a content provider, or any other person, machine, or organization can predefine which objects can be replaced with advertised objects in an AR or MR environment or which objects can have advertisements embedded in them. For example, in an AR or MR environment that includes a real-world store, a store or manufacturer of a product can embed AR or MR-based advertisements to add content describing the product, its price, or usage.

[0184] The bidding system can select various parameters regarding which ad units should be incorporated into the AR / MR / VR consumer scene. Such parameters may include, but are not limited to, similarity of a real or digitized shape to an advertised shape; a value associated with the object assigned by an advertiser or advertising agent; or the likelihood of a user interacting with the ad unit, time zone, location, etc.

[0185] According to various exemplary embodiments of the present disclosure, novel scene augmentation and reconstruction concepts may enable advertisers to bid on shapes or objects (i.e., shape-based search) and then insert their products into still or video content. For example, in the context of a virtual gaming environment, an automobile manufacturer may bid on a car shape, and the winning bidder's car would be displayed in the gaming environment. Similarly, a beverage manufacturer may be allowed to bid on a bottle, and the winning bidder's beverage bottle may then appear in an online video. And, in a more general sense, anyone may bid on a wall, and the winning bidder's message may appear on the wall as part of the display of other online content.

[0186] An example of an implementation of an advertising system (or visual input reconstruction system) based on the present disclosure is shown below. In this example, a user can play a game using a VR headset, and within the game, the user can enter a room equipped with an office chair (i.e., a "gaming chair") generated by the game. In this example, the disclosed advertising system can exchange the gaming chair for another chair provided by the advertiser, such as a branded chair.

[0187] In one embodiment, a game may be programmed by a game developer using a 3D representation of the environment. An advertising system may analyze portions of the environment visible to a user in the game. The disclosed system may detect (i.e., recognize) objects by, for example, using scene segmentation to divide the visible environment into individual detected objects such as chairs, tables, beds, etc. Segmentation may include identifying components of a scene or object, such as an object's face, surface, or component that is itself an object (e.g., identifying a wheel as an object component of a car). Segmentation may additionally or alternatively be performed using techniques for segmentation discussed above, consistent with embodiments of the present disclosure. For example, the system may use an object recognition model, including a machine learning model. The advertising system may compare objects detected from the visible scene to objects in an object data structure. An exemplary data structure consistent with embodiments of the present disclosure is described above. Consistent with embodiments of the present disclosure, the advertising system may additionally or alternatively perform the comparison using techniques for comparing objects and / or image data discussed above.

[0188] The data structure may include text describing the object, and the disclosed system may determine a measure of similarity between the detected object and the data structure object. The measure of similarity may be based on the label, the text, user input, shape, or model output (e.g., classification model output or statistical model output). The disclosed system and / or a user may tag the object with text corresponding to the data structure label based on the measure of similarity and criteria (e.g., meeting a threshold). In this example, the disclosed system may identify scene objects in a scene that are similar to a data structure object tagged as "office chair."

[0189] Continuing with this example, the advertising system can receive and compare bids identified as being associated with "office chair" (e.g., bids containing the text "office chair"), which may be associated with the advertiser's chair. The system can determine the highest bid (e.g., the bid with the highest dollar value) based on the comparison and select that bid as the winning bid. Thus, the advertising system can remove the gaming chair object from the scene and insert the advertiser's chair into the scene with the correct orientation, size, and lighting to match the existing scene as naturally as possible.

[0190] In this example, the advertising system can analyze the received bids to determine the winning bid. For example, consistent with the disclosed embodiments, a bid may include an image of a chair, and the disclosed system can scan the image, detect the chair, and further determine that the chair is an office chair. The disclosed system can attach a banner to the back of the chair if it is visible to the user.

[0191] As one skilled in the art will recognize, in an exemplary embodiment, a game plus a VR headset is a special case of a visual input reconstruction system. The game environment as viewed by a user in a given case, designed by a game developer, is a special case of an existing content frame. In this example, the representation of an office chair object with the text "office chair" is a special case of an object image identifier. Other examples are possible consistent with this disclosure.

[0192] According to the present disclosure, a computer-implemented visual input reconstruction system is disclosed for enabling selective insertion of content into an existing content frame. The visual input reconstruction system may include a system capable of generating and / or displaying any 2D or 3D media, including VR, AR, or MR environments. For example, a special case of a visual input reconstruction system may include a system that generates content viewable on a VR headset, such as a software-based game played on a VR headset. The visual input reconstruction system may include a VR headset. Other exemplary visual input reconstruction systems may include or generate content compatible with a phone or tablet that provides an MR experience that adds elements to a camera view of a room; an MR headset that presents a 3D experience of the viewed room with additional elements added to the real environment; or any other device used by a user to interact with a real or virtual scene.

[0193] Consistent with disclosed embodiments, a pre-existing content frame may include an image viewed or intended to be viewed by a user at a particular time. The pre-existing content frame may include, for example, a representation of a game environment designed by a game developer. The pre-existing content frame may include an image of reality itself, as viewed through a phone, computer screen, mixed reality headset, or other device. In some embodiments, the pre-existing content frame may include images of real and virtual objects (i.e., AR / MR). The pre-existing content frame may include information including viewer (e.g., user playing a game) characteristics such as age or interests. Properties of reality may be included in the pre-existing content frame, such as date, time, or location. In some embodiments, properties of an experience may be included in the pre-existing content frame, such as the angular velocity of a user's field of view experiencing a game, image data (e.g., RGB data), and / or depth camera data. The pre-existing content frame may also include sensor data, such as GPS data, gyroscope data, or accelerometer data embedded in the user device to extract the position, translation, and rotation of the user device, and / or the velocity and acceleration of the user device. Some devices, such as certain head-mounted devices, capture eye movements and track the user's gaze in order to determine which elements of a scene are more relevant to the user at any given time.

[0194] The existing content frames may include at least one of a still image, a series of video frames, a series of virtual 3D content frames, or a hologram. The still image may include an image in any image format (e.g., .JPG). The series of video frames may include a series of frames in 2D or 3D that, when presented to a viewer at a certain speed, creates the appearance of movement. The series of video frames may be formatted in a known video format, such as .MP4, or other known formats. Consistent with disclosed embodiments, the series of virtual 3D content frames may include a series of 3D video frames configured for presentation in a VR, MR, or AR context. The hologram may include data configured for projection such that the resulting projected light has the appearance of a 3D object. For example, the hologram may include data that, when presented to a device (e.g., a laser) capable of emitting split coherent radiation beams, creates a three-dimensional image resulting from a pattern of interference by the split coherent radiation beams.

[0195] In some embodiments, the visual input reconstruction system may include at least one processor. The processor may be configured to access a memory. Exemplary descriptions of the processor and memory are described above and with reference to FIG. 2.

[0196] In some embodiments, the processor of the system may be configured to access a memory storing a plurality of object image identifiers. The object image identifier may include text representing the object image. For example, an object image of an office chair may be represented by the text "office chair." In some embodiments, the object image identifier may include at least one of a shape, a shape descriptor, a product, or a product descriptor. A shape may include shape data, including shape data, coordinates, vectors, a mesh or grid, a representation of the shape (e.g., a 2D or 3D model), or any other data related to the shape. A shape descriptor may include text data, a label, a classification, a tag, and / or any other data that describes or identifies the shape. A product may include shape data (e.g., shape data providing a representation of the product's physical surface, such as the surface of a sports car). A product descriptor may include text data, a label, a classification, a tag, and / or any other data that describes or identifies the product.

[0197] In some embodiments, the object image identifier may include the output of a shape similarity engine. To illustrate this example, an advertiser may run an image of a particular office chair through a shape similarity engine provided by the advertising system, comparing an image of the advertiser's chair with a gaming chair in the object data structure. The advertiser may approve or otherwise indicate search results that represent the advertiser's understanding of the office chair. In some exemplary embodiments, the advertising system may perform a similarity search during a game, comparing an image of the gaming chair with an image of the advertiser chair in the data structure. The system may identify results that satisfy a similarity metric between the gaming chair and the candidate advertiser chair. The similarity metric may include a score that represents some measure of the degree of similarity between the image data, object data, and / or shape data between the two object images. The similarity metric may be based on statistical similarities such as covariance, least-squares distance, distance between vectors associated with image elements (e.g., feature vectors), or Hausdorff distance between aligned objects. The similarity metric may be based on feature vectors, as described in more detail elsewhere herein. Based on the similarity between the gaming chair and the advertiser chair, the system can determine that the advertiser may be interested in placing an image of the advertiser chair in a gaming scene. The system may additionally or alternatively use one or more of the techniques for comparing objects and / or image data discussed above to compare the image of the gaming chair with the image of the advertiser chair in the data structure, consistent with embodiments of the present disclosure.

[0198] In some embodiments, the object image identifier may include text generated by an algorithm. For example, an advertiser may provide text describing an object, and a text analysis system may extract relevant keywords from the text for use in the object image identifier. In some embodiments, the object image identifier may include information based on the results of a classification model. For example, an advertiser may provide one or more images of a product, and a 3D or 2D matching algorithm may identify similar objects in a game scene. As an example of a 2D matching algorithm, a neural network may segment an image into one or more objects and classify the objects. In some embodiments, the neural network may include a deep learning model (e.g., a convolutional neural network model) trained to identify similar objects and / or classify objects in a 3D scene or 2D image. For example, the neural network may tag, label, identify, or otherwise classify one or more pixels of a 2D snapshot as belonging to an object.

[0199] In some embodiments, the matching algorithm may include mapping objects in the 3D scene to feature vectors (i.e., generating feature vectors). In some embodiments, the system may calculate feature vectors for scene components and / or for objects in the 3D scene. The feature vectors may include sequences of real numbers or other data. The feature vectors may include information related to the rotation and / or positional changes of the scene components or matched components. Generating the feature vectors may include the use of machine learning models such as multi-view convolutional neural networks. For example, a multi-view convolutional neural network may accept multiple 2D representations (i.e., snapshots) of a 3D shape, 2D representations including projections of the 3D shape onto 2D images from various viewpoints (e.g., photographs of the object).

[0200] Consistent with this disclosure, interactions between objects and other properties of the experience may be included in the object image identifier. For example, the object image identifier may include properties of the user playing the game (i.e., user data), such as age or interests. Other user data may include identification information, usage information, purchase history, interaction history, etc. In some embodiments, real-world properties, such as the date of the year and time of day, may be included in the object image identifier. The object image identifier may include properties of the experience, such as the angular velocity of the user's view experiencing the game.

[0201] According to the present disclosure, multiple object image identifiers may be associated with multiple objects. In some embodiments, an object image identifier may be associated with each object. Each object image identifier may be associated with a different object (i.e., the object image identifiers may be unique). In some embodiments, an object image identifier may be associated with multiple objects. For example, an object image identifier for "sports cars" may be associated with various sports car makes and models. In some embodiments, multiple object image identifiers may be associated with the same object. As a specific example, an object image identifier for "chair" and an object image identifier for "office furniture" may be associated with an object image representing an office chair. As one skilled in the art will recognize, the object corresponding to the object image identifier may include at least one of a wall, a billboard, a picture frame, a window, a computer screen, a book cover, a door, or any other object. For example, a VR scene may include a representation of a room having a wall (or other object onto which a 2D image may be projected), and the object image identifier may correspond to the wall. In some embodiments, the image object identifier may include a 2D image of the object, an abstracted image of the object (e.g., a generic bottle), 3D data of the object, and / or any other data representing the object.

[0202] In some example embodiments, the visual input reconstruction system can transmit an object image identifier of a plurality of object image identifiers to one or more client devices. Transmission can include transmission over any network, such as a TCP / IP network. The client devices can include phones, tablets, mobile devices, computers, servers, clusters of servers, cloud computing services, and / or other client devices. In some embodiments, the client devices can include or be components of an advertising system (i.e., a system managed by an advertiser, advertising agency, agent, etc.). Consistent with disclosed embodiments, the client devices can connect to the visual input reconstruction system via a network (e.g., network 140). In some embodiments, the client devices can connect to the visual input reconstruction system via short-range wireless technology (e.g., BLUETOOTH, WI-FI) or a wired connection (e.g., USB cable). In some embodiments, the client devices can be configured to send and receive information (e.g., via an interface) based on the image object identifier. In some embodiments, a client device may be configured to implement an algorithm (eg, an algorithm for generating or placing a bid) based on the image object identifier.

[0203] In some embodiments, transmitting the at least one object image identifier may cause the at least one object image identifier to be displayed by one or more client devices. Displaying the object image identifier may include displaying via a screen, projection, a light-emitting component, or other means of displaying information. Displaying may include playing an audio signal through a speaker.

[0204] In some example embodiments, an interface may be displayed by one or more client devices by transmitting at least one object image identifier. The interface may include a display, a VR headset, a touchscreen, a keyboard, a mouse, a game console, and / or any other input or output device capable of providing information to a user and receiving information from a user input. The interface may be dedicated to a particular use context (e.g., a kiosk). The interface may be configurable by the user.

[0205] The interface may be configured to place at least one bid for at least one object image identifier. Placing a bid may include receiving user input associating a value with the object image identifier. A bid may include a time period (e.g., a bid to run an advertisement at a specific time), a number of users (e.g., 1,000 game players), a rate (cost per unit of displayed time or cost per person receiving a broadcast), or any other information. As an example, a client device (e.g., a client device operated by an advertiser) may place a bid of $0.10 per broadcast recipient. As one skilled in the art will recognize, other examples of bidding are possible. A bid may include an update to a previously placed bid.

[0206] The interface may include an auction system for placing bids. For example, the interface may be configured to send and receive a price or a set of prices. The interface may be configured to include multiple bidding options. For example, the interface may include one option for adding a small image projected onto the object identifier, another option for a larger image, another option for replacing the object, and yet another option for replacing the object and changing a feature of the 3D environment (e.g., adjusting lighting, changing volume levels, etc.). The bids may be related to a similarity metric between the object identifier image and the advertiser object, as described herein (e.g., a lower price may be available for a metric indicating a lower degree of similarity).

[0207] The interface can present data on the object or object identifier, such as data that may provide information related to the compatibility of the potential advertiser object with the scene. For example, a toy office chair in the scene may be labeled to indicate that it is not a full-size office chair or is not compatible with a full-size office chair. Other properties of the object may also be displayed, including color, style, material, and any other properties of the object or its relationship to its surroundings that may be relevant to the bid.

[0208] According to the present disclosure, a visual input reconstruction system can receive one or more bids associated with at least one object image identifier from one or more client devices. In some embodiments, the bids can be based on user input received at a client device (e.g., a client device operated by an advertiser). In some embodiments, the bids can be based on an algorithm or other executable code on the client device that generates and submits bids with or without user input.

[0209] In some embodiments, the visual input reconstruction system may determine a winning bid from among one or more received bids. The winning bid may be associated with a winning client device from among one or more client devices. In some embodiments, the determination of the winning bid may be based on one or more criteria, such as value (i.e., monetary amount), compatibility of the advertiser object with the scene, audience-related information, and / or any other information. In some embodiments, the criteria for determining a winning bid may be based on the likelihood that a bid winner will submit a second bid after winning the initial bid. For example, the visual input reconstruction system may determine that an advertiser is likely to receive a positive outcome from winning the bid and that the advertiser is likely to submit a second bid in the future. Positive outcomes may include, for example, product purchases, increased website traffic, social media mentions, etc. The examples of criteria for determining a winning bid provided herein are not limiting, and other criteria are possible. Additionally, as one skilled in the art will appreciate, the visual input reconstruction system may determine multiple winning bids from among one or more received bids associated with each object image identifier.

[0210] More specifically, the criteria may include a numerical value such as a price and / or a similarity metric to the winning bidder's image data (e.g., a 3D model of the object) of the object in the scene. In some embodiments, the similarity metric may be based on the object's feature vectors. For example, the similarity metric may include a least-squares distance estimate between two feature vectors associated with each object. The least-squares distance may be normalized so that zero (0) is set as the average similarity of an object to multiple objects (e.g., multiple random objects in a data structure), and one (1) represents the similarity metric of the object to itself. In some embodiments, the criteria may include a measure of similarity between tags associated with the advertised object and tags associated with the scene object. The criteria may include spatial or other semantic criteria to be satisfied in the scene. For example, a spatial criterion may specify that the image data is "on the table." As another example, a semantic criterion may include a requirement that the image data be located so that a computer mouse is located "near the laptop." Additionally or alternatively, in some embodiments, the spatial criteria may include a spatial semantic graph, which is discussed in more detail below in this disclosure. In some embodiments, the criteria may be written in a computing language (e.g., code, script, etc.). For example, the criteria may be written in a predetermined computing language provided by the system, and a bidder may provide a script that defines the criteria, and the system may check whether the criteria, along with the proposed placement of objects, are satisfied in the scene.

[0211] In some embodiments, the visual input reconstruction system can receive image data of the winning bidder from the winning bidder's client device. The winning bidder's image data can include any image data consistent with disclosed embodiments. The winning bidder's image data can include a 2D or 3D image or model of an object. For example, the object image identifier can include an identifier associated with a soft drink can, and the winning bidder's image data can include a 2D logo of a beverage manufacturer suitable for display on the soft drink can. As another example, the winning bidder's image data can include a 3D model of the manufacturer's soft drink can. Consistent with disclosed embodiments, the image data can be in any format, including .JPG, .BMP, .GIF, .PNG, .SVG, 3D vector formats, computer-aided design (CAD) files, .FLV, .MP4, .AVI, .MPG, .MP3, .MOV, .F4V, .VR, or other image, video, or model formats. In some embodiments, the winning bidder image data includes text data (e.g., text data for projection onto objects in the scene) and / or any other modifications specified by the winning bidder client device (e.g., changes to the lighting levels or volume levels of the scene).

[0212] In some embodiments, the visual input reconstruction system may be configured to receive instructions from a winning bidder's client device. The instructions may include size constraints for the object corresponding to the at least one object image identifier. For example, the size constraints may include minimum or maximum object size, pixel density, font size, or other size constraints. The size constraints may include constraints related to scaling, rotation, or other transformations of the object.

[0213] The winning bidder's (winner's) image data may include data associated with one or more objects and configured to be inserted into a scene, consistent with disclosed embodiments. For example, a bid may be associated with an empty room, and the winning bidder's (winner's) image data may include text data for projection onto the walls of the room and image object data representing a mascot (or other spokesperson) conveying the message.

[0214] In some embodiments, the visual input reconstruction system may be configured to perform image processing on the winning image data to make it compatible with the format of the existing media content frame. For example, the winning image data may be in a first format, and the existing media content frame may be in a second format. The visual input reconstruction system may be configured to convert the winning image data from the first format to the second format, with or without intermediate conversion or processing. The format may include a broadcast format. The image processing of the winning image data may include any method of image processing consistent with disclosed embodiments. For example, the image processing may include adjusting brightness, shadows, ambient light, contrast, hue, saturation, scaling, cropping, rotating, stretching, filtering, smoothing, or otherwise transforming the image data.

[0215] The visual input reconstruction system can store the winning bidder's image data in memory. Storing in memory can include storing in a data structure, an index, data storage, local memory, remote memory, or any other method of storage. Storing the winning bidder's image data can include storing in any file format. Storing can include storing the converted winning bidder's image data.

[0216] According to the present disclosure, a visual input reconstruction system may identify an object insertion location for an object in at least one existing media content frame. The existing media content frame may include any image data, video data, VR data, AR data, MR data, etc. The existing media content frame may include multiple frames constituting a virtual reality field of view. The virtual reality field of view may include a field of view from one or more viewpoints of a VR, MR, or AR environment. For example, the VR environment may include a virtual room with a doorway, four walls, a ceiling, a floor, and furniture. The multiple frames constituting the virtual reality field of view may include multiple frames of the virtual room as seen by a person standing in the doorway. As another example, the multiple frames constituting the virtual reality field of view may include multiple frames of the virtual room as seen by a person sitting on furniture. The virtual reality field of view may change over time.

[0217] The object insertion location may include a position and / or orientation within a computerized environment. The object insertion location may correspond to at least one object image identifier (e.g., the object image identifier may include the text "door handle," and the object insertion location may correspond to a location on a door representing the door handle). The object insertion location may include a location associated with an object, such as the face, front, side, top, or location of a component of the object. The object insertion location may include a location within a VR, AR, or MR environment. For example, the object insertion location may include data identifying the center back of a chair in a VR scene (e.g., to insert a banner advertisement). The object insertion location may include relationships between objects within a scene. For example, the object insertion location may include information identifying proximity to another object (e.g., "near the table in the scene") or orientation relative to another object (e.g., in front of the table). More generally, 2D or 3D transformations between advertiser objects and game objects, with or without an action to erase the game objects, are all examples of object insertion locations.

[0218] In some embodiments, the visual input reconstruction system may generate at least one processed media content frame by processing at least one existing media content frame to insert at least a rendition of the winning bidder's image data at the object insertion location. Processing the existing media content frame may include inserting the winning bidder's image data or a rendition of the winning bidder's image data into the computerized environment. The rendition of the winning bidder's image data may include the winning bidder's image data itself. In some embodiments, the representation of the winning bidder's image data may include image data based on the winning bidder's image data. Processing may include using homography methods to scale or orient the winning bidder's image data. Processing may include gradient domain methods or other image processing techniques to blend the winning bidder's image data into the computerized environment. Processing may include adjusting brightness, shadows, ambient light, contrast, hue, saturation, or other image processing techniques. For example, the winning bidder's (winner's) rendition of the image data may include image data that has been scaled, cropped, rotated, stretched, filtered, smoothed, or otherwise transformed based on the winning bidder's (winner's) image data. Generating the processed media content frames may include the use of Principal Component Analysis (PCA).

[0219] Consistent with disclosed embodiments, processing the existing media content frames may include adding content to the scene based on the winning bidder's image object data. That is, the winning bidder's image object data may include additional content, such as interactive objects, that may be incorporated into the immersive experience. As an example, the winning bidder's object bid may include data related to a company mascot sitting in a chair. In this case, the presence of the mascot may change properties of the VR scene (e.g., a first-person game player may not be able to walk through the mascot without a virtual collision).

[0220] In some embodiments, the image object data may include code, text, or other instructions related to moving the winning bidder's image object, and processing the existing media content frame may include executing the instructions. For example, the instructions may include code to make a mascot jump off a table and sit in a chair. In some embodiments, the instructions may be provided in natural language that is translated by the advertising system (e.g., the instructions may be to "make the mascot jump," and the system may process the existing content frame accordingly). Additionally, as mentioned above, the winning bidder's image object data may include instructions to modify scene properties, such as adding lighting to focus a user's attention on the object.

[0221] According to the present disclosure, inserting at least a rendition of the winning bidder's image data may render an object from the winning object data in multiple frames. For example, the winning object data may be virtually displayed in a broadcast. As an example, the object may be a particular sports car, and the winning object image data may include an image of the particular sports car, and inserting a rendition of the winning bidder's image data may render the particular sports car in multiple frames. The winning bidder's image data may be inserted into at least one existing content frame, such that the winning bidder's image data may be overlaid on existing content in the at least one existing content frame. As an example, overlaying the winning bidder's image data on existing content may include, from the perspective of a viewer in the VR environment, superimposing the winning bidder's image data including an image of a logo on existing content including an image of a billboard. Overlaying the winning bidder's image data may include adding a banner to an object (e.g., adding a banner to a chair back or a bottle).

[0222] The insertion of the winning bidder's image data may include adding an element that interacts with an object (e.g., an element may include object data related to a mascot, and the mascot may sit in a chair in the scene). More generally, the insertion of the winning bidder's image data may include any addition to the scene that adds content, including content that may interact with other objects in the scene, regardless of whether the addition overlays or replaces the object.

[0223] For example, in some embodiments, the image data of the winning bidder may be inserted when one or more objects in a scene interact with one another. In some embodiments, physics-based engines (simulating objects jumping, falling, and pushing) and semantic-based engines (e.g., objects configured to move toward other objects that may be semantically tagged in a 3D environment) may be used to cause objects to interact with one another. An advertiser may define rules specifying that the image data of the winning bidder be inserted into a scene depending on the interaction between two objects in the scene. As an example, an advertiser may request that the scene include a floor, a table, and a laptop. Furthermore, the advertiser may specify interactions between objects in the scene, such as a small creature jumping onto the floor, jumping onto a chair, sitting for a while, then getting up, jumping onto the table, and then disappearing. The advertiser may specify that the winning bidder's (winner's) image data (e.g., advertising image) be inserted after the interaction has occurred, to ensure that the viewer's attention has been captured by the interaction prior to the insertion of the winning bidder's (winner's) image data.

[0224] The winning bidder's image data may be inserted into at least one existing content frame such that an image of an object represented by the winning bidder's image data replaces existing content in the at least one existing content frame. As an example, the existing content may include a virtual sports car moving through a VR game environment, and the winning bidder's image data including image data related to the particular sports car may replace the virtual sports car so that a viewer perceives the particular sports car rather than the virtual sports car. As another example, a house in an urban scene may be replaced with a fast food restaurant. In some embodiments, the winning bidder's image data may be inserted into a portion of an object corresponding to the at least one object image identifier. In some embodiments, inserting at least a rendition of the winning bidder's image data may be based on instructions including a size constraint. For example, the size constraint may include a maximum size, and inserting at least a rendition of the winning bidder's image data may include inserting a rendition of the winning bidder's image data that is less than or equal to the maximum size. Additional or alternative techniques for combining two images (e.g., a first image and a second image) discussed above may also be used, consistent with embodiments of the present disclosure, to generate at least one processed media content frame by replacing one or more objects or images within an existing content frame with the image data of the winning bidder, or by inserting and / or overlaying the image data of the winning bidder into at least one existing content frame.

[0225] In some embodiments, the visual input reconstruction system can transmit at least one processed media content frame to one or more user devices. The user devices may include any device configured to receive and / or display media content, such as a mobile device, a VR headset, a game console, a computer, a server, and / or any other user device. The transmitting may include broadcasting. As described above, broadcasting may include transmitting to multiple individuals over a network. In some embodiments, the transmitting may include transmitting the processed media content frame to a first user device of the one or more user devices. In some embodiments, the transmitting may further include transmitting at least one existing media content frame to a second user device in a manner that excludes image data of a winning bidder. In this manner, the media content frame may be targeted to a specific audience. For example, the processed media content frame may include an image of a diet soft drink that replaces a regular soft drink in the existing media content frame. In this case, the first user device may be a user device associated with a person identified as a health-conscious person, while the second user device may be associated with a person identified as a person who previously purchased regular soft drinks.

[0226] It should be understood that the foregoing steps and methods may be performed in real time. In some embodiments, the visual input reconstruction system may be configured to retrieve at least one existing content frame in real time and insert a rendition of the winning bidder's image data into the at least one existing content frame in real time. As one skilled in the art will recognize, the steps may be performed in various orders, and some or all steps may be repeated to modify a broadcast in real time. For example, in some embodiments, the winning bidder's image data displayed in an existing content frame may change after a predetermined period of time. To illustrate this example, a virtual billboard in a VR environment may display the winning bidder's image data, including a first logo, for 10 minutes and then display a second logo at the end of the 10 minutes. The predetermined period of time may be set by an advertising system (i.e., the visual input reconstruction system). In some embodiments, a bid may include a predetermined period of time (e.g., an advertiser may set a time for displaying an image object). In some embodiments, the predetermined period of time may be determined by a user (i.e., an audience member). As one skilled in the art will recognize, the predetermined period of time may include a scheduled time (e.g., changing the winning image data displayed after a predetermined period of time may include changing it at a set time, such as at 3:00 PM).

[0227] 1 illustrates an exemplary system 100 for augmenting or reconstructing a 2D or 3D scene or image consistent with embodiments of the present disclosure. As shown, system 100 may include a client device 110, a visual input reconstruction system 120, a data structure 130, and / or a user device 150. The components of system 100 may be connected to one another via a network 140. In some embodiments, aspects of system 100 may be implemented in one or more cloud services. In some embodiments, aspects of system 100 may be implemented on a computing device, including a mobile device, a computer, a server, a cluster of servers, or multiple server clusters.

[0228] As will be appreciated by those skilled in the art, the components of system 100 may be arranged in a variety of ways and, where applicable, implemented in any suitable combination of hardware, firmware, and / or software. For example, compared to the depiction of FIG. 1 , system 100 may include a greater or lesser number of client devices, visual input reconstruction systems, data structures, user devices, and / or networks. Additionally, system 100 may further include other components or devices not shown that perform or assist in the performance of one or more processes consistent with the disclosed embodiments. The exemplary components and arrangement shown in FIG. 1 are not intended to limit the disclosed embodiments.

[0229] In some embodiments, client device 110 may be associated with an advertiser, an advertising agency, and / or any other individual or organization. For example, client device 110 may be configured to execute software that enables advertisers to bid on inserting content into existing media content frames consistent with disclosed embodiments. Client device 110 may include one or more memory units and one or more processors configured to perform operations consistent with disclosed embodiments. In some embodiments, client device 110 may include hardware, software, and / or firmware modules. Client device 110 may include a mobile device, tablet, personal computer, terminal, kiosk, server, server cluster, cloud service, storage device, specialized device configured to perform methods according to disclosed embodiments, etc. Client device 110 may be configured to receive user input (e.g., at an interface), display information (e.g., images and / or text), communicate with other devices, and / or perform other functions consistent with disclosed embodiments. In some embodiments, the client device is configured to implement an algorithm that places a bid based on information received from another device (eg, from the visual input reconstruction system 120).

[0230] Visual input reconstruction system 120 may include a computing device, computer, server, server cluster, multiple server clusters, and / or cloud service consistent with disclosed embodiments. Visual input reconstruction system 120 may include one or more memory units and one or more processors configured to perform operations consistent with disclosed embodiments. Visual input reconstruction system 120 may be configured to receive, read, and / or transmit data from other components of system 100 and / or computing components external to system 100 (e.g., via network 140).

[0231] Data structure 130 may be hosted on one or more servers, one or more clusters of servers, or one or more cloud services. In some embodiments, data structure 130 may be a component of visual input reconstruction system 120 (not shown). Data structure 130 may include one or more data structures configured to store images, video data, image object information, image object identifiers, metadata, labels, and / or any other data. Data structure 130 may be configured to provide information about the data to another device or system. Data structure 130 may include a cloud-based data structure, a cloud-based bucket, or an on-premises data structure.

[0232] User device 150 may be any device configured to receive and / or display media content frames, including VR, AR, and / or MR data. For example, user device 150 may include a mobile device, a smartphone, a tablet, a computer, a headset, a gaming console, and / or any other user device. In some embodiments, user device 150 is configured to receive and / or display broadcasts. User device 150 may include one or more memory units and one or more processors configured to perform operations consistent with disclosed embodiments. In some embodiments, user device 150 may include hardware, software, and / or firmware modules.

[0233] One or more of client device 110, visual input reconstruction system 120, data structure 130, and / or user device 150 may be connected to network 140. Network 140 may be a public or private network and may include, for example, a local area network, a wide area network, a metropolitan area network, an IEEE 1002.11 wireless network (e.g., "Wi-Fi"), a network of networks (e.g., the Internet), a landline network, etc. Network 140 may be connected to other networks (not shown in FIG. 1 ) to connect various system components to each other and / or to external systems or devices. In some embodiments, network 140 may be a secure network and may require a password to access the network.

[0234] 2 illustrates an exemplary computing device 200 for implementing embodiments and features of the present disclosure. By way of example, computing device 200 or a similar computing device may be used to implement any of the devices or systems described herein, including client device 110, visual input reconstruction system 120, and / or user device 150. The components of computing device 200 are provided for illustrative purposes. Consistent with the present disclosure, it is contemplated that additional arrangements, number of components, and / or other modifications may be made to the disclosed computing device 200.

[0235] Computing device 200 may include one or more processors 202 for executing instructions. Processor 202 may include known computing processors, including microprocessors. Processor 202 may comprise a single-core or multi-core processor that simultaneously executes parallel processes. For example, processor 202 may be a single-core processor configured with virtual processing technology. In some embodiments, processor 202 may use a logical processor to simultaneously execute and control multiple processes. Processor 202 may implement virtual machine technology or other known technology to provide the ability to execute, control, perform, manipulate, store, etc., multiple software processes, applications, programs, etc. In another embodiment, processor 202 may include a multi-core processor configuration (e.g., dual-core, quad-core, etc.) configured to provide parallel processing functionality that allows multiple processes to be executed simultaneously. In particular embodiments, processor 202 may use a logical processor to simultaneously execute and control multiple processes. Those skilled in the art will appreciate that other types of processor configurations that provide the functionality disclosed herein may be implemented. The disclosed embodiments are not limited to any particular type of processor. Processor 202 may execute various instructions stored in memory 430 to perform various functions of the disclosed embodiments, which will be described in more detail below. Processor 202 may be configured to execute functions written in one or more known programming languages. In some embodiments, processor 202 may be based on a Reduced Instruction Set Computer (RISC) architecture, a Complex Instruction Set Computer (CISC) architecture, or any other computer instruction architecture known in the art. It is also contemplated that processor 202 may include one or more graphics or other digital signal processors.

[0236] The computing device 200 may also include one or more input / output (I / O) devices 204. By way of example, the I / O devices 204 may include a display (e.g., an LED display, a VR display), a headset, augmented glasses (e.g., Google Glass), a physical keyboard, a light-emitting component, a haptic feedback device, a touchpad, a mouse, a microphone, a printer, a scanner, a 3D scanner, a biometric device, a sensor, a motion sensor, a location sensor, a GPS sensor, an accelerometer, a magnetometer, a virtual touchscreen keyboard, a joystick, a stylus, a button, a switch, a dial, a knob, and / or any other I / O device.

[0237] As further shown in FIG. 2 , computing device 200 may include memory 206 configured to store data or one or more instructions and / or software programs that, when executed by one or more processors 202, perform functions or operations. Memory 206 may include volatile or nonvolatile, magnetic, semiconductor, optical, removable, non-removable, or other types of storage devices or tangible (i.e., non-transitory) computer-readable media, consistent with disclosed embodiments. Memory 206 may include encrypted and / or unencrypted data. By way of example, memory 206 may include random access memory (RAM) devices, NOR or NAD flash memory devices, read-only memory (ROM) devices, etc. Computing device 200 may also include storage medium 208 configured to store data or one or more instructions and / or software programs that, when executed by one or more processors 202, perform functions or operations. In some demonstrative embodiments, storage medium 208 may also be configured to store data or instructions. By way of example, storage medium 208 may include a hard drive, a solid state drive, a tape drive, a RAID array, a compact disc (CD), a digital video disc (DVD), a Blu-ray disc (BD), etc. Although Figure 2 shows only one memory 206 and one storage medium 208, computing device 200 may include any number of memories 206 and storage media 208. Furthermore, although Figure 2 shows memory 206 and storage medium 208 as part of computing device 200, memory 206 and / or storage medium 208 may be located remotely, and computing device 200 may access memory 206 and / or storage medium 208 via network 140.

[0238] Computing device 200 may include one or more displays 210 for displaying data and information. Display 210 may be implemented using devices or technologies such as a cathode ray tube (CRT) display, a liquid crystal display (LCD), a plasma display, a light emitting diode (LED) display, a touchscreen-type display, a projection system, virtual reality or augmented reality glasses or headsets, and / or any other type of display capable of displaying 2D or 3D audiovisual content as known in the art. The disclosed embodiments are not limited to any particular type of display configured in computing device 200.

[0239] Computing device 200 may also include one or more communications interfaces 212. Communications interface 212 may enable software and / or data to be transferred between computing device 200, network 140, client device 110, visual input reconstruction system 120, data structure 130, user device 150, and / or other components. Examples of communications interface 212 may include a modem, a network interface (e.g., an Ethernet card or a wireless network card), a communications port, a PCMCIA slot and card, a cellular network card, etc. Communications interface 212 may transfer software and / or data in the form of signals, which may be electronic, electromagnetic, optical, or other types of signals that may be transmitted or received by communications interface 212. Communications interface 212 may transmit or receive these signals using wire, cable, fiber optics, radio frequency (“RF”) link, and / or other communications channels. The communication interface 212 may be configured to communicate via WI-FI, BLUETOOTH, near field, LI-FI, and / or any other wireless transmission method.

[0240] Consistent with this disclosure, the disclosed system may include at least one processor that may be configured to execute one or more instructions, algorithms, etc. to perform the functions of the preview system. By way of example, as shown in Figures 1 and 2, system 100 may include one or more processors 202 included in one or more of client device 110 and visual input reconstruction system 120.

[0241] 3 illustrates an example system 300 for selecting bids from advertisers and inserting images corresponding to the winning bids into existing scenes from an audiovisual environment, consistent with embodiments of the present disclosure. System 300 may be an embodiment of system 100.

[0242] As shown, system 300 may include data including an existing scene, such as an existing 3D scene 302, which may be digitized. Consistent with disclosed embodiments, scene 302 may include existing media content frames. Scene 302 is not limited to 3D data, but may include VR data, AR data, MR data, image data, video data, and / or any other scene data. Scene 302 may include representations of image objects, such as chair 304, sofa 306, and / or table 308. The image objects may correspond to one or more image object identifiers, as described above.

[0243] The system 300 may be configured to receive advertiser bids 310. The advertiser bids may include identifying information that identifies an advertiser, account, individual, or other identifying information. For example, the identifying information may include a label such as "Advertiser 1," "Advertiser 2," or "Advertiser 3." The advertiser bids may include object information. The object information may include an object identifier, such as a product object identifier, such as "Chair 1," "Chair 2," or "Chair 3." The advertiser bids may be associated with a respective bid amount, represented by a dollar sign in the advertiser bids 310.

[0244] In some embodiments, the system 300 may be configured to identify a winning bid and replace an object in the scene 302 with an object associated with the winning bid (e.g., image data of the winning bidder). Identifying a winning bid may be based on criteria, consistent with disclosed embodiments. For example, the system 300 may be configured to replace (312) the scene chair with the chair associated with the highest bid (e.g., from advertiser 2).

[0245] The system 300 may be configured to perform rendering 314. Rendering may include processing existing media content frames to insert a rendition of the winning bidder's (winner's) image data at the object insertion location. The rendering 314 may include any image processing techniques as described herein or any other image processing techniques. The rendering 314 may be formatted for display by a VR device and / or screen (VR / screen 316). A user 318 may view the rendered scene through the VR / screen 316.

[0246] FIG. 4 illustrates an exemplary method 400 for selecting and inserting advertising images into an existing scene from an audiovisual environment, consistent with embodiments of the present disclosure. The order and arrangement of steps in process 400 are provided for illustrative purposes. As will be appreciated from this disclosure, modifications can be made to process 400 by, for example, adding, combining, removing, and / or rearranging steps for the process. The steps of method 400 can be performed by components of system 100, including, but not limited to, visual input reconstruction system 120. For example, while method 400 may be described as steps performed by visual input reconstruction system 120, it should be understood that client device 110 and / or user device 150 can perform any or all steps of method 400. As one skilled in the art will appreciate, method 400 can be implemented in conjunction with any other method described herein. For example, it should be understood that process 400 may include steps (not shown) of transmitting an object image identifier (e.g., step 504), receiving bids associated with the object image identifier (e.g., step 506), and / or any other actions consistent with disclosed embodiments. Process 400 may be performed in real time to modify the ongoing transmission of media content consistent with disclosed embodiments.

[0247] In step 402, the visual input reconstruction system 120 may receive or retrieve an input scene. Consistent with disclosed embodiments, the input scene may be received or retrieved from data storage. The input scene may be received from another component of the system 100 and / or another computing component external to the system 100 (e.g., via the network 140). The input scene may be retrieved from a memory (e.g., memory 206), a data structure (e.g., data structure 130), or any other computing component.

[0248] The input scene may be a VR, AR, and / or MR scene, consistent with disclosed embodiments. The input scene may be a 2D and / or 3D scene. The input scene may be in any format (e.g., F4V, .VR, etc.). The input scene may include existing content frames, consistent with disclosed embodiments. The input scene may include a previously modified scene, such as a scene including processed media content frames containing image data of a winning bidder, as described herein. In general, the input scene may include any visual media.

[0249] Consistent with disclosed embodiments, step 402 may include receiving or retrieving an image object identifier. Step 402 may include receiving user data. User data may include identification data (e.g., user ID, IP address, account number), usage data, purchase data, interaction data, etc. Interaction data may include, but is not limited to, data related to gestures, voice, gaze, touch, etc.

[0250] In step 404, the visual input reconstruction system 120 can scan the input scene for objects, consistent with disclosed embodiments. For example, the visual input reconstruction system 120 can scan to detect objects such as chairs, tables, or soft drink bottles. Other examples of objects are possible. Step 404 can include receiving or retrieving an image object identifier. The scanning can include an object recognition algorithm (e.g., a machine learning method). The scanning can include determining or detecting an object insertion location, consistent with disclosed embodiments.

[0251] In some embodiments, the input reconstruction system 120 can extract objects from a scene. Extraction can include generating or copying image object data associated with objects detected in the scene. The system can extract the 2D or 3D shape of an object, element, or portion of a scene. For example, the system can generate a model of a detected gaming chair. In some embodiments, the visual input reconstruction can transmit the extracted image objects and / or image object identifiers to a client device, consistent with disclosed embodiments.

[0252] In some embodiments, the scanning may include determining user interaction data in the VR, AR, or MR environment. For example, consistent with disclosed embodiments, the interaction data may be determined based on how frequently an object appears in the field of view. The interaction data may be based on gestures, gaze, or other user actions.

[0253] In step 406, the visual input reconstruction system 120 may compare the detected object to objects in a data structure, such as an advertiser object data structure, consistent with disclosed embodiments. Additionally or alternatively, step 406 may include receiving advertiser objects from a client device and comparing the advertiser objects to detect the object (e.g., receiving one or more bids including each advertiser object). Comparing may include implementing a classification algorithm (e.g., a machine learning model). Comparing may include generating one or more criteria, such as a similarity metric, as described herein. Comparing may include segmenting objects in the scene, i.e., identifying components of the object, such as faces, lines, surfaces, or components that are themselves objects (e.g., identifying a wheel as a component object of a car object). Comparing may be based on an image object identifier. Comparing may include comparing based on text data, shape data, user data, and / or other data.

[0254] At step 408, the input reconstruction system 120 may classify the input scene consistent with disclosed embodiments. For example, the scene may be classified as an indoor scene, an office scene, a reception hall scene, a sports venue scene, etc. Identifying or classifying the type of scene may be based on scene metadata that describes or otherwise labels the scene. Alternatively or additionally, the scan may include identifying or classifying the type of scene based on objects detected and identified within the scene (e.g., objects associated with a kitchen may be detected, and the scene may be identified accordingly as a kitchen scene).

[0255] At step 410, the input reconstruction system 120 may identify a user's interest, consistent with disclosed embodiments. The user's interest may include data indicating a correlation between the user and other data, such as objects in a scene, advertising objects, or other data. Identifying the user's interest may be based on user data, including interaction data, as described above. Identifying the user's interest may include performing a statistical analysis of the user data, object data, and / or scene classification. Identifying the user's interest may include receiving or retrieving additional user data based on the received user data and / or the inferred interaction data. Identifying the user's interest may be based on user data, including interaction data, as described above. Identifying the user's interest may include performing a statistical analysis of the user data, object data, and / or scene classification. Identifying the user's interest may include receiving or retrieving additional user data based on the received user data and / or the inferred interaction data.

[0256] At step 412, the visual input reconstruction system 120 may determine matching objects, consistent with disclosed embodiments. Determining matching objects may be based on the output of an algorithm or model (e.g., a machine learning model). Determining matches may be based on image object identifiers. Matching objects may include advertiser objects (i.e., objects associated with advertisers, products, etc.). Determining matches may be based on the highest bid and / or other criteria for determining winning bids, as described herein. For example, determining matching objects may be based on external variables such as time of day or properties of the user receiving the input scene or the changing scene. In some embodiments, determining matching objects may be based on an indication of user interest. User interest may be determined by user historical data, including interaction data, such as data indicating which parts of a scene the user interacts with. Interactions may include, but are not limited to, gestures, voice, gaze, touch, etc.

[0257] At step 414, the visual input reconstruction system 120 can provide an output scene, consistent with disclosed embodiments. In some embodiments, providing the output scene can include modifying the input scene and replacing objects with matching objects to output the scene modified with the matching objects, consistent with disclosed embodiments. That is, the output scene can be a modified scene. Replacing objects can include any image processing method, consistent with disclosed embodiments. For example, replacing objects can include inserting at least a rendition of the object into the scene.

[0258] As shown, step 414 can follow any step in process 400. For example, visual input reconstruction system 120 can determine that no match is found, that the input scene is classified (redacted) as not to contain advertising, or that the user would not be interested in the product. Thus, the output scene can be the same as the input scene. In this way, visual input reconstruction system 120 can provide targeted advertising by providing modified scenes to some users while providing existing content to other users.

[0259] Providing the output scene in step 414 may include storing and / or transmitting the output scene, consistent with disclosed embodiments. For example, step 414 may include broadcasting the output scene and / or storing the output scene in memory (e.g., memory 206, storage medium 208, and / or data structure 130).

[0260] 5 illustrates an exemplary method 500 for enabling selective insertion of content into an existing content frame, consistent with embodiments of the present disclosure. While the steps of process 500 may be described as being performed by visual input reconstruction system 120, one skilled in the art will understand that other components of system 100 and / or components external to system 100 may perform one or more steps of process 500. The order and arrangement of the steps of process 500 is provided for illustrative purposes. As will be appreciated from this disclosure, modifications can be made to process 500 by, for example, adding, combining, removing, and / or rearranging process steps.

[0261] In step 502, the visual input reconstruction system 120 may access a memory that stores a plurality of object image identifiers associated with a plurality of objects, consistent with disclosed embodiments.

[0262] In step 504, the visual input reconstruction system 120 may transmit at least one object image identifier of the plurality of object image identifiers to one or more client devices.

[0263] At step 506, the visual input reconstruction system 120 may receive one or more bids associated with at least one object image identifier from one or more client devices, consistent with disclosed embodiments.

[0264] At step 508, the visual input reconstruction system 120 may determine a winning bid from among the one or more received bids, consistent with disclosed embodiments. In some embodiments, the winning bid may be associated with a winning client device from among one or more client devices.

[0265] At step 510, the visual input reconstruction system 120 may receive winning bidder image data from the winning bidder client device, consistent with disclosed embodiments.

[0266] At step 512, the visual input reconstruction system 120 may store the winning bidder image data in memory, consistent with disclosed embodiments.

[0267] In step 514, the visual input reconstruction system 120 may, consistent with disclosed embodiments, identify an object insertion location in at least one existing media content frame for an object corresponding to at least one object image identifier.

[0268] In step 516, the visual input reconstruction system 120 may generate at least one processed media content frame by processing at least one existing media content frame to insert at least a rendition of the winning bidder image data at the object insertion location, consistent with the illustrated embodiment.

[0269] At step 518, the visual input reconstruction system 120 may transmit at least one processed media content frame to one or more user devices, consistent with disclosed embodiments.

[0270] 6 illustrates an example method 600 for placing a bid for inserting content into an existing content frame, consistent with embodiments of the present disclosure. While the steps of process 600 may be described as being performed by client device 110, one skilled in the art will recognize that other components of system 100 and / or components external to system 100 may perform one or more steps of process 600. The order and arrangement of the steps of process 600 is provided for illustrative purposes. As will be appreciated from this disclosure, modifications may be made to process 600 by, for example, adding, combining, removing, and / or rearranging steps of the process.

[0271] At step 602, the client device 110 may generate bidding rules consistent with disclosed embodiments. The bidding rules may be logical rules, algorithms, expressions, or models (e.g., machine learning models) that constrain or define characteristics of the bids. For example, the bidding rules may include rules for associating advertising image objects with a target audience. The bidding rules may be based on content media frames, object identifiers, and / or image data. For example, the bidding rules may be based on received image data, received object identifier data, advertising image data, and / or advertising object identifier data, as described herein. The bidding rules may include a maximum bid amount, a time associated with the bid, or other properties of the bid. The bidding rules may include rules for placing bids based on a similarity metric generated by a matching algorithm or a shape similarity engine. The generation of the bidding rules may be based on data received from another computing component, from user input, or based on data associated with previous bids (e.g., click-through rates of previous ads associated with previous bids). As one skilled in the art will recognize, the examples of bidding rules presented herein are not limiting, and other examples are possible consistent with the disclosed embodiments. Creating bidding rules may include copying or modifying previously created bidding rules.

[0272] At step 604, client device 110 may receive image data and / or image object identifier data consistent with disclosed embodiments. The image data may be in any format, including VR, AR, MR, 2D, 3D, and / or any other format. The image object identifier may include a label, text, a classification, and / or any other image object identifier consistent with disclosed embodiments.

[0273] At step 606, client device 110 may display the received image data and / or image object identifier data in an interface, consistent with disclosed embodiments. For example, client device 110 may generate the interface with a headset, an LED screen, a touchscreen, and / or any other screen. The interface may include input and output devices capable of receiving user input and providing information to a user, as described herein, or any other type of interface. The display may include providing visual data (e.g., via a projector or screen), audio data (e.g., by playing sound through speakers), and / or tactile data (e.g., via a haptic feedback device). For example, the received image data may include an image of a blue office chair with wheels, and the received object identifier data may include "office chair," "chair," "blue chair," and / or "wheelchair (wheeled chair)."

[0274] At step 608, client device 110 may retrieve (search) and / or identify (identify) advertiser image data and / or advertiser object identifier data, consistent with disclosed embodiments. In some embodiments, step 608 may include accessing or searching in data storage, such as data structure 130. The data storage may include advertiser image data and / or advertiser object identifier data. The advertiser image data may include images associated with a product, a message, a campaign, or any other image data, consistent with disclosed embodiments. In some embodiments, the advertiser image data may include text data, numeric data, and / or other data. The advertiser object identifier data may include any object identifier, consistent with disclosed embodiments.

[0275] The retrieval (retrieval) and / or identification (identification) of the advertiser image and / or advertiser object identifier data in step 608 may be based on the received image data and / or the received object identifier data. In some embodiments, the retrieval (retrieval) may include implementing a search algorithm and / or a matching algorithm, such as a shape similarity engine, as described above. In some embodiments, step 608 may be monitored and based on user input. In some embodiments, step 608 may be followed by step 610, as shown in FIG. 6.

[0276] At step 610, client device 110 may display the advertiser image data and / or advertiser object identifier data, consistent with disclosed embodiments. Step 610 may include displaying the advertiser image data and / or advertiser object identifier data in the same interface used in step 606 or in a separate interface. Step 610 may include displaying the received image data, advertiser image data, image object identifier data, and / or advertiser image object identifier data.

[0277] At step 612, the client device 110 may receive user input consistent with disclosed embodiments. The user input may be related to the received image data, advertiser image data, image object identifier data, and / or advertiser image object identifier data. The user input may include input for selecting at least one advertiser image object and / or advertiser object identifier. The user input may include input for generating a bid.

[0278] As shown in Figure 6, steps 608 through 612 may be repeated any number of times. For example, the user input received in step 612 may include instructions to retrieve and / or identify advertiser image data and / or advertiser object identifier data, allowing the user to repeat and refine the search to select advertiser image objects or advertiser object identifier data.

[0279] At step 614, client device 110 may generate a bid, consistent with disclosed embodiments. The bid may include a value, a value range, a period for which the bid remains open for acceptance, a period for which the advertisement is included, information about the target audience, a means of transmission or broadcast, and / or any other information related to the bid. The generation of the bid may be based on bidding rules (e.g., logic rules, algorithms, expressions, etc.) that limit or otherwise define the characteristics of the bid. The generation of the bid may also be based on user input (e.g., the user input at step 610). In some embodiments, as shown in FIG. 6 , generating the bid at step 614 may be based on the results of the search process performed at step 608. For example, process 600 may be automated, and the bid may be generated based on a matching algorithm and the received image data, the received object identifier data, and / or the bidding rules. The bid may include advertiser image data and / or advertiser object identifier data, consistent with disclosed embodiments.

[0280] At step 616, the client device 110 may submit a bid, consistent with disclosed embodiments. Step 616 may include submitting the bid to the input image reconstruction system 120. Submitting the bid may include submitting advertiser image data and / or advertiser object identifier data.

[0281] The present disclosure also relates to a computer-implemented system for processing scenes (e.g., scan-based 3D scenes) for use in virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies and applications. The present disclosure provides solutions to problems in technologies and applications for generating and modifying objects within a scene. While the present disclosure provides examples of AR, VR, and MR technologies and applications, it should be noted that aspects of the disclosure in their broadest sense are not limited to any particular examples. Rather, it is contemplated that the principles described above may be applied to other computer-based reality technologies and applications.

[0282] In accordance with the present disclosure, a computer-implemented system for generating 3D scenes is disclosed. FIG. 1 illustrates an exemplary system 100 for generating 3D scenes consistent with embodiments of the present disclosure. The system may include at least one processor. An exemplary description of the processor and memory is provided above and with reference to FIG. 2.

[0283] The processor may be configured to receive a scan of a scene. The scene may be received from another device (e.g., a client device, a user device). The scene may be retrieved (read) from remote or local data storage. The scene may include image data, consistent with disclosed embodiments. In some embodiments, the scene may be based on a scan, and the scan includes capturing image data from one or more cameras or scanners (e.g., a 3D scanner).

[0284] In some embodiments, a scan may be an incomplete scan (i.e., a scan that captures a partial representation of an object). For example, an incomplete scan may capture only a portion of an object, such as the front of a chair. In some cases, an incomplete scan may result when a scan acquires less information than necessary to represent the complete object in 3D. For example, a scan may receive data from fewer directions than necessary to capture all surfaces of a 3D object, or other objects may obstruct the scanner input (e.g., another object may be between the object and the camera). In some cases, an incomplete scan may only acquire two-dimensional (2D) data. An incomplete scan may occur when a scan does not capture properties of the actual object, such as color information. A scan may be incomplete due to limitations in the scan input method or other errors (e.g., a partially transparent object may not appear in the scan). In some cases, an incomplete scan may occur if the scan misses (fails to read) part of the scene data (e.g., a hole in the scene) or if other errors or distortions occur due to hardware limitations or user error made during the scan.

[0285] Consistent with disclosed embodiments, a scene may be configured to be displayed via a device such as a headset, a computer screen, a monitor, a projection, etc. Aspects of the scene may be encoded in known formats, such as 3D vector formats, computer-aided design files, .FLV, .MP4, .AVI, .MPG, .MP3, .MOV, .F4V, .VR, or image, video, or model formats. Embodiments consistent with the present disclosure may include a scene represented by a mesh, a point cloud, or any other representation that encodes the scene.

[0286] In an exemplary embodiment, a scene may include a 3D representation of a living room encoded as a mesh as described above. To illustrate this principle, a scene mesh may include establishing points that make up floors, walls, doorways, stairs, etc. Consistent with disclosed embodiments, the mesh may include multiple virtual objects (i.e., components) that represent real-world or fictional objects such as furniture items, buildings, people, animals, mythical creatures, plants, or any other object. A component may contain other components (e.g., a table may include legs). In general, a scene component (i.e., object) may include a group of points or polygons within the mesh based on relationships between the points or polygons. In a scene, objects may change location, rotate, resize, change shape, etc.

[0287] In some embodiments, a scene may include at least one object as described herein, which may be, for example, a chair, a car, a painting, a person, an animal, a mythical creature, and / or any other object.

[0288] In some embodiments, the system may generate image elements based on the received scan, and / or the scene may include image elements, consistent with disclosed embodiments. More generally, the scene may include image elements or multiple image elements, such as primitive 2D elements. For example, the image elements may include at least one of pixels, voxels, points, or polygons. In some embodiments, the system may generate a set of polygons or voxels, where each polygon or voxel is a primitive element.

[0289] In some cases, image elements may be further subdivided. For example, the system may generate a mesh consisting of multiple n-sided polygons as image elements, and one or more polygons may be subdivided into additional polygons to improve resolution or for other reasons. In some embodiments, a scene may include multiple pre-existing image elements. The pre-existing image elements may be received together with or separately from receiving the scene.

[0290] In some embodiments, the system may process image elements within a scene to segment the scene into scene-components consistent with the segmentation techniques discussed above. Consistent with embodiments of the present disclosure, segmentation may additionally or alternatively be performed using the techniques for segmentation discussed above. For example, the system may segment a scene into scene components of objects representing living room furniture, such as chairs, cups, tables, or other objects, consistent with disclosed embodiments. Objects may have one or more scene components, which may themselves be objects. For example, the armrests of an office chair may be separate, independent scene components or may be part of a scene component representing the entire chair. Thus, segmentation may identify the armrests as scene components and / or the entire chair as a scene component. Segmentation may include identifying scene components that correspond to known classifications (e.g., identifying a scene component and classifying it as an “armrest”) and / or unknown classifications (e.g., identifying a scene component and classifying it as an “unknown component”), as described below.

[0291] As discussed above, segmentation may include dividing (i.e., classifying) image elements of a scene into scene components. Classification may include types of scene components. For example, "furniture," "chair," and "office chair" may all be classes of objects that comprise the same class of objects. As will be apparent to one skilled in the art, classes may be defined in a hierarchy of broader or narrower classes relative to each other. For example, the "furniture" class may be broader than the "chair" class, and the "chair" class may be broader than the "office chair" class.

[0292] For example, a scene component may include points, voxels, or polygons associated with known or unknown objects, such as a table, table surface, or table legs. The system may segment a scene consisting of a scan of a living room into multiple scene components, such as chairs, doorknobs, handles, cups, utensils, shoes, walls, plant leaves, carpets, and televisions. The system may segment image elements to belong to a scene component and classify the scene component with a known or unknown classification. For example, during segmentation, a scene component may be labeled as a particular type of object (e.g., a chair), an unknown type of object, and / or a possible known object (e.g., a "likely" chair) based on a confidence or likelihood measure associated with the segmentation algorithm output. One or more image elements may remain unmapped (i.e., not assigned to an object or component of an object) after segmentation. Segmentation may also include mapping (i.e., assigning) a 3D element to one or more objects (e.g., the same 3D element may be assigned to "armrest" and "chair").

[0293] In an example embodiment, segmentation may include identifying scene components based on classification model output, including labels such as "unknown scene component," and implementing the classification model. For example, the "unknown scene component" may correspond to the armrest of a chair, but the classification model output may not recognize the scene component as an armrest. Other parts of the chair may be identified separately during segmentation, and the corresponding scene components may be labeled as "unknown," "headrest," "chair leg," etc.

[0294] The system may identify a matching component from among the scene components that corresponds to at least one object component based on a comparison of the scene component with the stored image data. Consistent with embodiments of the present disclosure, the system may additionally or alternatively compare the scene component with the stored image data using one or more of the techniques for comparing objects and / or image data discussed above. The matched component may be an improved classification of the scene component and / or may include additional information associated with the scene component. For example, a scene component may be classified as "unknown," and based on a comparison of the scene component with the stored image data, the scene component may be identified as a matched component labeled as "armrest." Consistent with disclosed embodiments, the matched component may include a model of at least one object and / or image object identifier. In some embodiments, the matched component may be a component that is similar but not identical to the scene component. For example, the scene component may be a black chair on a base with wheels and armrests, and the matched component may include office chairs with bases of various colors, with or without armrests.

[0295] In some embodiments, identifying a matched component from among the scene components may include mapping the 3D shape to a feature vector (i.e., generating a feature vector). In some embodiments, the system may calculate a feature vector for the scene component and / or a feature vector for the matched component. The feature vector may include a sequence of real numbers or other data. The feature vector may include information related to the rotation and / or position change of the scene component or the matched component. Generating the feature vector may include using a machine learning model such as a multi-view convolutional neural network. For example, the multi-view convolutional neural network may accept multiple 2D representations (i.e., snapshots) of the 3D shape, where the 2D representations include projections of the 3D shape into 2D from various angles (e.g., photographs of the object).

[0296] Comparing the components may include determining a similarity metric indicating the degree of similarity between the matched component and the stored image data. The similarity metric may be based on shape data, color data, and / or any other data. The similarity metric may be based on statistical similarity such as covariance, least-squares distance, distance between vectors associated with image elements (e.g., feature vectors), or Hausdorff distance between aligned objects. The similarity metric may be based on the feature vectors. In some embodiments, a model may generate the similarity metric, such as a machine learning model. In some embodiments, the comparison may include implementing a classification model (e.g., a random forest model) to classify the components of the object. Identifying the components corresponding to at least one object is based on the similarity metric. As an example, the object may be a chair, and the system may determine that the matched component, which is a chair armrest, is similar to the particular scene component based on a similarity metric between the image data and the particular scene component in a data structure containing a 3D model of the chair and the armrest. Consistent with embodiments of the present disclosure, an exemplary data structure is described above.

[0297] Comparing the scene components to the stored image data may include searching a data structure for objects based on the scene components and generating one or more search results corresponding to objects in the data structure (i.e., matches). The search results may include a percentage match, a likelihood, or another metric representing the similarity between the object corresponding to the image object identifier in the data structure or the object in the data structure and the detected object. The highest-ranking search result may, for example, define the narrowest class of components or objects in the data structure that match the detected object. In some embodiments, the system may identify similar objects based on previously performed searches of the data structure.

[0298] Consistent with the present disclosure, a data structure may include stored image data. The image data of the data structure may include a 2D or 3D model of an object. The image data may include scene data comprising tagged objects. A data structure consistent with the present disclosure may include one or more computer-aided design (CAD) models corresponding to one or more objects. The CAD models may be stored in one or more formats, such as a mesh, a point cloud, a voxel mapping of 3D space, and / or any other mapping that may be configured to present a graphical representation of the object. The CAD models may represent an object and / or a component of an object (e.g., a chair and / or chair arms).

[0299] A matching component in the data structure may correspond to one or more object image identifiers. The object image identifier may include text representing the object image (i.e., a label or tag, such as the name of the object). For example, the object image identifier for an office chair may include the label "office chair." In some embodiments, the object image identifier may include at least one of a shape, a shape descriptor (e.g., a label or text description of the shape, including a sentence), a product, or a product descriptor. A shape may include shape data, including shape data, coordinates, vectors, meshes, or grids, a representation of the shape (e.g., a 2D or 3D model), or any other data associated with the shape. A shape descriptor may include text data, a label, a classification, a tag, and / or any other data that describes or identifies the shape. A product may include shape data (e.g., shape data providing a representation of the product's physical surface, such as the surface of a sports car). A product descriptor may include text data, a label, a classification, a tag, and / or any other data that describes or identifies the product.

[0300] In an example embodiment, based on a comparison of the scene component labeled as an "unknown scene component" with the stored image data, the system may label the matching component as an "armrest." The object image identifier may include text representing the object image (i.e., a label or tag, such as the name of the object). For example, the object image identifier for an office chair may include the label "office chair." In some embodiments, the object image identifier may include at least one of a shape, a shape descriptor (e.g., a label or text description of the shape, including a sentence), a product, or a product descriptor. A shape may include shape data, which may include coordinates, vectors, a mesh or grid, a representation of the shape (e.g., a 2D or 3D model), or any other data associated with the shape. A shape descriptor may include text data, a label, a classification, a tag, and / or any other data that describes or identifies the shape. A product may include shape data (e.g., shape data providing a representation of the product's physical surface, such as the surface of a sports car). A product descriptor may include text data, labels, classifications, tags, and / or any other data that describes or identifies the product.

[0301] In some embodiments, the system may transmit the scene to a client device or display the scene in a system interface. Transmitting may include transmission over any network, such as a TCP / IP network, a broadband connection, a cellular data connection, and / or any other transmission method. The client device and / or system interface may include, without limitation, a mobile device, a headset, a computer, a display, an interface, and / or any other client device. In some embodiments, transmitting or displaying the scene may include transmitting or displaying the scene component as a highlighted scene component. For example, the highlighted scene component may be presented with an outline, a brightness change, a color change, etc. The system may transmit or display the scene such that when the client device performs an action (e.g., mouse over hover, gaze, etc.), the detected scene component is highlighted.

[0302] In some embodiments, using the interface, a client (e.g., a video game designer or advertiser) may select an object detected in a scene, such as an office chair. As one skilled in the art will appreciate, selecting may include performing one or more actions, such as clicking the object, entering an identifier in a text box, making a gesture to select the object, etc. The system may receive the selection from the interface or via a transmission from the client device. In some embodiments, the system may transmit data to the client device or display data in the interface, where the data is associated with one or more matched components. For example, the system may transmit image data associated with the matched component, a 3D model associated with the matched component, or an image object identifier associated with the matched component, consistent with disclosed embodiments. In some embodiments, the client may select data associated with the matched component. For example, the client may select an image of a blue office chair with a base but no armrests.

[0303] Consistent with this embodiment, the system may identify image elements corresponding to at least one object based on the matched components. For example, in some embodiments, the object may include the matching components (e.g., the matching components may be armrests and the at least one object may be a chair including armrests). In some embodiments, the at least one object may be the same as the matched components (e.g., the matched components may be soft drink bottles and the at least one object may be the same soft drink bottle). Identifying the image elements may include determining that 3D elements not included in the matched components correspond to objects, such as objects within a predetermined distance. For example, the matched component may be a beach umbrella and the at least one object may be a beach towel adjacent to the beach umbrella or a sandcastle within a predetermined distance of the beach umbrella. Consistent with the disclosed embodiment, identifying image elements corresponding to the at least one object may be based on image data in a data structure, such as a CAD model, an image object identifier, or other data.

[0304] In some embodiments, the system may retrieve a CAD model from a storage location based on graphical elements corresponding to at least one object. The CAD model may include scripts describing motion or material properties (i.e., texture characteristics) of the at least one object. The CAD model may include graphical elements, consistent with disclosed embodiments. As previously disclosed, the CAD model may represent an object and / or a component of the object (e.g., a chair and / or chair arms). The CAD model may be stored in one or more formats, such as a mesh, a point cloud, a voxel mapping of a 3D space, and / or any other mapping that may be configured to present a graphical depiction of the object.

[0305] In some embodiments, the system may access semantics associated with the CAD model, where the semantics include scripts that describe movable characteristics of at least one object. Accessing the semantics may include retrieving (reading) or receiving the semantics from data storage (e.g., a data structure), consistent with disclosed embodiments. In some embodiments, accessing the semantics may be based on an image object identifier associated with the 3D model and the semantics of the 3D model.

[0306] In embodiments consistent with the present disclosure, semantics associated with a CAD model may include object properties such as movement or material properties. For example, a chair may be able to swivel, recline, have adjustable armrest heights, or have other movement properties associated with its motion. In general, movement properties may include any property associated with the degree of freedom of movement of an object or a component of an object. As an example of material properties (i.e., texture properties), a chair seat may be hard and undeformable when another object is placed on it (e.g., when a person sits on a wooden chair). Alternatively, the texture properties of a chair seat may be plush and deformable when another object is placed on it (e.g., when a person sits on a padded chair). Material properties may include, without limitation, elasticity, hardness, roughness, smoothness, reflectivity, color, shape, deformability, or other material properties. In general, semantics associated with a CAD model may include movement properties or material properties of the CAD model.

[0307] Semantics may include scripts that represent or govern movement or material properties, and such scripts may be configurable by a designer or other client. For example, during animation of a scene, a client may change a chair's degrees of freedom, such as its ability to rotate. Scripts may include, for example, programmatic descriptions of how an object with degrees of freedom responds to external forces. For example, for a swivel chair, a program might provide a CAD model of the chair's top relative to the chair legs when a rotational force is applied to the top. This modification of the object may be rendered in the scene. As another example, a CAD model may replace a curtain, and a script may describe the curtain's reaction to wind, causing the replaced curtain in the scene to react accordingly. The script may be written in any code capable of animating objects (e.g., AutoCAD, Blender, Cinema 4D, and / or Autodesk Maya).

[0308] A CAD model may have scripts associated with an entire object (e.g., the elasticity of a ball when bouncing) or with parts of an object (e.g., the movement of rotors on a helicopter). The degrees of freedom may be complex, representing interactions between several objects, such as a lever that lifts a chair seat, a handle that opens a door, etc. Thus, a script may represent interactions between one object and at least one other object in a scene. While a scanned scene may be static or otherwise not include scripts that encode specific dynamic movements or material properties, the CAD model encodes the semantics, and the replaced scene may include programmatic descriptions of such movements and material properties.

[0309] In some embodiments, the system may generate a modified scene by combining CAD models of the scene and the object. Consistent with disclosed embodiments, combining may include replacing an object, replacing a portion of an object, moving an object, modifying material properties, etc. For example, generating a modified scene may include applying texture properties (e.g., reflectance or color) of the CAD model of the object to a scene component.

[0310] Consistent with the present disclosure, generating a modified scene may include combining object data with scene data. For example, the system may replace an original object (e.g., a black office chair) from a scene with another object (e.g., a blue office chair) selected by a client. Replacing an object may include placing the selected object in the same orientation (i.e., aligning the object) and similar size (i.e., scaling the object) as the original object. Aligning the object and / or scaling the object may include using principal component analysis (PCA). Replacing an object may include allowing a client to position, scale, or otherwise manipulate a selected object within a scene. Replacing an object may include using image processing techniques (e.g., adjusting brightness, adjusting lighting, implementing gradient domain methods, etc.) consistent with disclosed embodiments. As one skilled in the art will appreciate, gradient domain methods may include constructing a new image by integrating gradients of image elements. Replacement may include rendering a mesh, points, or other digitized representation of the object based on lighting, scene resolution, perspective, etc. Following object replacement, the resulting scene may be an example of a scene that has been modified by combining information obtained from the CAD model of the object and the original scene.

[0311] Consistent with disclosed embodiments, a modified scene may include a hybrid scene including at least a portion of a CAD model and at least a portion of at least one object. A hybrid scene may refer to a modified scene in which only a portion of an object (e.g., an armrest of an office chair) or a characteristic of an object (e.g., texture or color) is replaced with object data from a data structure. A scene comprising at least one object having characteristics obtained from both the original object and an object from another source (e.g., a data structure) is an example of a hybrid scene. A portion of the CAD model may include image elements corresponding to components of the CAD model (e.g., an armrest of a chair). A portion of at least one object may include image elements corresponding to components of the object (e.g., a leg of a chair). In some embodiments, the portion of the CAD model and the portion of the at least one object may be blended at their boundaries using image processing techniques to create a realistic appearance. For example, the image processing techniques may include gradient domain methods or other image merging methods.

[0312] In some embodiments, the modified scene may include scene refinement based on the semantics of the CAD model. In some embodiments, if the scanned scene is an incomplete scan, replacing may include scene refinement, i.e., the process of replacing partial information with additional information. Refinement may include replacing a portion of an object with a complete 3D object from the data structure. For example, the scan may include only a portion of an object (e.g., the front of a chair), and replacing the object may include replacing the portion of the object with the complete 3D object from the data structure. Another example of scene refinement from the semantics of the CAD model may include adding color and texture from an object in the data structure to a scanned object that has shape but no color or texture. As another example of scene refinement, the system may determine that a scanned chair is a metal chair, and the data structure may include material properties of metal or metal chairs, such as surface reflectance properties. The system may add lighting to the chair based on the surface reflectance properties to enhance the chair's realism. Scene refinement may be based on the semantics of the CAD model, as described in more detail below, including motion or material properties of the CAD model. Additional or alternative techniques for combining two images (e.g., a first image and a second image) discussed above may also be used to generate modified or hybrid scenes.

[0313] In some embodiments, the system may extract material properties from the matched component and apply the extracted material properties to the CAD model, consistent with disclosed embodiments. For example, the scan may extract object properties such as texture properties or color. In an exemplary embodiment, a scanned chair may include a metal surface, and the matching CAD chair in the data structure may have a different or non-specific material on its surface. Thus, substituting the object may include rendering the CAD chair as a metal chair. For example, the matched component may be a seat chair and may have material properties related to color, texture, hardness, etc. The system may extract material properties from the matched component and apply them to the CAD model of the chair. Additional or alternative techniques for combining two images (e.g., a first image and a second image) as discussed above may also be used to generate modified or hybrid scenes, consistent with embodiments of the present disclosure.

[0314] In some embodiments, the system may apply a script to a CAD model in the hybrid scene, and the script may be configured to execute to render objects within the hybrid scene. For example, the script may be configured to allow objects to translate, rotate, hinge, bend, or otherwise animate objects. The script may be written in any code capable of animating objects (e.g., Autocad, Blender, Cinema 4D, Autodesk Maya). In some embodiments, the hybrid scene including the script is output for 3D display. For example, the hybrid scene with the script may be formatted for a VR helmet or any other 3D display and sent to the display. Outputting a scene may be described in more detail below.

[0315] In some embodiments, the system may select another script associated with the object, where the another script represents an interaction between the object and at least one other object in the scene. Selecting the another script may be based on an image object identifier associated with the at least one object and the other object. For example, a first script may be associated with a first image object identifier corresponding to a ball, and a second script may be associated with the first image object identifier corresponding to the ball and a second image object identifier corresponding to a bat. The first script may include a script that enables the ball to move under gravity, and the second script may include a script that enables the bat to interact with the ball (e.g., the bat hits the ball). In some embodiments, the system may apply the script to a CAD model in the hybrid scene.

[0316] Consistent with disclosed embodiments, the system may output the modified scene for 3D display. Outputting the modified scene may include storing and / or transmitting the modified scene, consistent with disclosed embodiments. Transmitting may include transmitting over a network by any known method, consistent with disclosed embodiments. For example, the system may broadcast the modified scene (i.e., transmit over a network to multiple user devices), transmit the modified scene to a user device, and / or store (save) the modified scene in memory.

[0317] FIG. 7 illustrates an example method 700 for selecting a 3D model to replace a CAD object in a scene, consistent with embodiments of the present disclosure. The order and arrangement of steps in process 700 are provided for illustrative purposes. As will be appreciated from this disclosure, modifications to process 700 may be made, for example, by adding, combining, removing, and / or rearranging steps for the process. The steps of method 700 may be performed by components of system 100, including, but not limited to, 3D generator 120. For example, while method 700 may be described as steps performed by 3D generator 120, it should be understood that client device 110 and / or user device 150 may perform any or all steps of method 700. As one skilled in the art will recognize, method 700 may be performed in conjunction with any other method described herein. In some embodiments, process 700 may be performed in conjunction with steps of processes 800 and / or 900. Process 700 may be performed in real time to modify an ongoing transmission of media content (e.g., a broadcast of a scene), consistent with disclosed embodiments.

[0318] In step 702, the 3D generator 120 may receive a 3D scene, consistent with disclosed embodiments. The scene may be received or read from data storage, consistent with disclosed embodiments. The scene may be received from another component of the system 100 and / or another computing component external to the system 100 (e.g., via the network 140). The scene may be read from a memory (e.g., the memory 206), a data structure (e.g., the data structure 130), or any other computing component. The scene may be based on images captured by one or more cameras (i.e., scans), consistent with disclosed embodiments.

[0319] In step 704, the 3D generator 120 may segment the 3D scene consistent with disclosed embodiments. As described herein, segmentation may include dividing (i.e., classifying) image elements of the scene into scene components or objects, such as a table 706, a sofa 708, a chair 710, and / or other components or objects. In some embodiments, step 704 may include generating a mesh, a point cloud, or other representation of the scene.

[0320] In step 712, the 3D generator 120 may search the object data structure to identify one or more matching components, consistent with disclosed embodiments. The object data structure search may be based on scene components. The object data structure may include 3D models, image data, CAD models, image object identifiers, and / or any other data related to the components and / or objects.

[0321] At step 714, the 3D generator 120 may receive an object data structure result based on the search, consistent with disclosed embodiments. While at step 714 only two objects from the data structure are depicted as search results, more generally, the object data structure result may include any number of results. The search result from the data structure may include a 3D model, matched components, image object identifiers, and / or a similarity metric, consistent with disclosed embodiments. The similarity metric may include a "match score" or any other similarity metric, consistent with disclosed embodiments. The match score may represent a probability that a scene component is a data structure component or object. The match score may represent a similarity between the scene component and the data structure component. The match score may be based on the shape of the component and the shape of the data structure component. 7, "Chair 1" is an object in the data structure and is associated with a match score of 0.9, and "Chair 2" is an object in the data structure and is associated with a match score of 0.95. In the example of FIG. 7, the match scores may represent the degree of similarity between chair 1 and chair 710 in the data structure.

[0322] In step 716, the 3D generator 120 may identify a CAD model based on the resulting object data structure, consistent with disclosed embodiments. For example, the 3D generator 120 may identify the CAD model (e.g., "Chair 2") associated with the highest match score.

[0323] At step 718, the 3D generator 120 may render the modified scene consistent with disclosed embodiments. Rendering the modified scene may include rendering objects based on CAD models, replacing scene components with CAD models, combining the scene with a CAD model, or any other rendering method disclosed herein. The modified scene may be rendered to create a scene in which aspects (features) of the CAD model appear natural to the viewer compared to other components of the scene (e.g., a CAD model of a chair inserted into a scene appears to be part of the scene itself and has orientation, size, shadowing, highlights, etc. that resemble other aspects (features) of the scene). Rendering may include performing any image processing techniques consistent with disclosed embodiments.

[0324] At step 720, the 3D generator 120 may transmit the modified scene, consistent with disclosed embodiments. In some embodiments, the 3D generator 120 transmits the modified scene to a user device (e.g., user device 150). In some embodiments, step 720 includes broadcasting the modified scene. The transmitting at step 720 may include transmitting over a network by any known method, consistent with disclosed embodiments.

[0325] At step 722, the device may display the modified scene consistent with disclosed embodiments. In some embodiments, user device 150 or another device displays the modified scene consistent with disclosed embodiments. In this manner, the end user experiences the modified scene.

[0326] FIG. 8 illustrates an exemplary method 800 for selecting a 3D model and replacing CAD objects in an existing scene with the selected 3D model, consistent with embodiments of the present disclosure. The order and arrangement of steps in process 800 are provided for illustrative purposes. As will be appreciated from this disclosure, modifications may be made to process 800 by, for example, adding, combining, removing, and / or rearranging process steps. The steps of method 800 may be performed by components of system 100, including, but not limited to, 3D generator 120. For example, while method 800 may be described as steps performed by 3D generator 120, it should be understood that client device 110 and / or user device 150 may perform any or all steps of method 800. As one skilled in the art will appreciate, method 800 may be implemented in conjunction with any other method described herein. In some embodiments, process 800 may be performed in conjunction with steps of processes 700 and / or 900. Process 800 may be performed in real time to modify the ongoing transmission of media content (eg, a broadcast of a scene), consistent with disclosed embodiments.

[0327] In step 802, the 3D generator system may receive a scene, consistent with disclosed embodiments. The scene may be a 2D or 3D scene. The scene may be incomplete (i.e., based on a scan that captures a partial representation of an object), consistent with disclosed embodiments. The scene may be received from a client device, a data structure, memory, a user device, or any other computing component.

[0328] In step 804, the 3D generator system may segment the scene consistent with disclosed embodiments. As described herein, segmentation may include dividing (i.e., classifying) image elements of the scene into scene components or objects, such as a table 806, a sofa 808, a chair 810, and / or other components or objects. In some embodiments, step 804 may include generating a mesh, point cloud, or other representation of the scene. A scene component may include a complete object (e.g., a cup), a portion of an object (e.g., the handle of a cup), or a partial representation of an object (e.g., a cup viewed from one side).

[0329] In step 812, the 3D generator system may extract scene component characteristics consistent with disclosed embodiments. For example, the scan may extract motion characteristics and / or material characteristics (i.e., texture features), or any other characteristics, as described above.

[0330] In step 814, the 3D generator system may search an object data structure based on the scene components, consistent with disclosed embodiments. Searching the object data structure may be based on the segmented components. The object data structure may include 3D models, image data, CAD models, image object identifiers, and / or any other data related to the components and / or objects.

[0331] In step 816, the 3D generator system may receive the resulting object data structure, consistent with disclosed embodiments. As previously mentioned, the resulting object data structure may include a match score or other similarity metric.

[0332] At step 818, the 3D generator system may select a CAD model, consistent with disclosed embodiments. Selecting the CAD model may be based on input from the client. For example, step 818 may include displaying the object data structure results and / or a representation of the data structure object (e.g., a CAD model) in an interface and receiving input from the client. In some embodiments, step 818 may include sending the object data structure results and / or a representation of the data structure object (e.g., a CAD model) to a client device and receiving information from the client device. The 3D generator system may select the CAD model based on the received information. In some embodiments, selecting the CAD model is based on a similarity metric, a match score, or the like (e.g., selecting the highest match score).

[0333] At step 820, the 3D generator system may combine the scene component with the selected CAD model to generate a combined object model, consistent with disclosed embodiments. In some embodiments, combining the scene component with the selected CAD model may include merging aspects of the scene component with aspects of the CAD model. For example, the combined object model may include textures of the scene component as applied to the CAD model, or textures of the CAD model as applied to the scene component. As another example, the combined object model may include textures of the CAD model as applied to the scene component. In some embodiments, merging aspects of the scene component may include adding components of the CAD model to the scene component, or adding components of the scene component to the CAD model (e.g., adding armrests to a chair). In some embodiments, combining the scene component with the selected CAD model may include replacing the scene component with the CAD model (i.e., the combined model is the CAD model). Step 820 may include performing any image processing technique, consistent with disclosed embodiments.

[0334] In step 822, the 3D generator system may replace the scene components with the combined object model, consistent with disclosed embodiments. Step 822 may include generating a modified scene, including a hybrid scene, consistent with disclosed embodiments. Step 822 may include performing any image processing technique, consistent with disclosed embodiments.

[0335] FIG. 9 illustrates an example method 900 for generating a 3D scene consistent with embodiments of the present disclosure. The order and arrangement of steps in process 900 are provided for illustrative purposes. As will be understood from this disclosure, modifications may be made to process 900 by, for example, adding, combining, removing, and / or rearranging process steps. The steps of method 900 may be performed by components of system 100, including, but not limited to, 3D generator 120. For example, while method 900 may be described as steps performed by 3D generator 120, it should be understood that client device 110 and / or user device 150 may perform any or all steps of method 900. As one skilled in the art will appreciate, method 900 may be implemented in conjunction with any other method described herein. In some embodiments, process 900 may be performed in conjunction with steps of processes 700 and / or 800. Process 900 may be performed in real time to modify the ongoing transmission of media content (e.g., the broadcast of a scene) consistent with disclosed embodiments.

[0336] At step 902, the 3D generator may receive a scan of a scene, consistent with disclosed embodiments. The scan may include at least one object. As disclosed herein, the scan may be received from a memory, another computing component, etc. Receiving the scan may include receiving a mesh, a point cloud, or other representation of the scene.

[0337] In step 904, the 3D generator may process image elements in the scan to segment the scene into scene components, consistent with disclosed embodiments. As previously described, the image elements may include at least one of voxels, points, polygons, or other image elements. Segmentation may include generating a mesh, point cloud, or other representation of the scene.

[0338] In step 906, the 3D generator may identify one or more matched components, consistent with disclosed embodiments. The matched components may correspond to at least one scene component. The matched components may include any component or object described herein.

[0339] In step 908, the 3D generator may identify a graphical element corresponding to the object, consistent with disclosed embodiments. As disclosed, identifying the graphical element may include classifying the graphical element. Identifying the graphical element may include searching an object data structure and generating or receiving the results of the object data structure, consistent with disclosed embodiments.

[0340] At step 910, the 3D generator may obtain a CAD model based on image elements corresponding to the object, consistent with disclosed embodiments. Obtaining the CAD model may include searching an object data structure and generating or receiving a result in the object data structure, consistent with disclosed embodiments. Obtaining the CAD model may be based on input from a client device, consistent with disclosed embodiments. Obtaining the CAD model may include selecting a CAD model based on a similarity metric, consistent with disclosed embodiments.

[0341] In step 912, the 3D generator may generate the modified scene consistent with disclosed embodiments. Generating the modified scene may include combining and / or substituting scene components with the CAD model.

[0342] In step 914, the 3D generator may output the modified scene, consistent with disclosed embodiments. Outputting the modified scene may include transmitting and / or saving the modified scene, consistent with disclosed embodiments. Step 914 may include transmitting the modified scene to a user device.

[0343] The present disclosure also relates to a computer-implemented system for animating a portion of a still image. According to the present disclosure, a system for augmenting or reconstructing a 2D or 3D scene or image may include several components that may interact and cooperate with each other. By way of example, FIG. 10 illustrates an exemplary system 1000 consistent with the present disclosure. As illustrated in FIG. 10 , the system 1000 may include, for example, a user system 1010, a user 1012, a server 1020, a data structure 1030, and a network 1040. The components of the system 1000 may be connected to each other via the network 1040. In some embodiments, aspects of the system 1000 may be implemented on one or more cloud services. In some embodiments, aspects of the system 1000 may be implemented on a computing device, including a mobile device, a computer, a server, a cluster of servers, or multiple server clusters.

[0344] The user system 1010 may include one or more computing devices that may be used by a user 1012 to create, augment, or reconstruct audiovisual content. By way of example, the user system 1010 may include a computing device such as a personal computer, a laptop computer, a desktop computer, a tablet computer, a notebook, a mobile phone, a terminal, a kiosk, a specialized device configured to perform methods according to the disclosed embodiments, etc. The user system 1010 may be configured to execute an application or a set of instructions to generate, augment, or reconstruct audiovisual content, such as, for example, a 2D or 3D scene or image. The user system 1010 may be configured to be operated by one or more users 1012.

[0345] Server 1020 may include one or more computing devices that may be used to generate, enhance, or reconstruct audiovisual content. By way of example, server 1020 may be a general-purpose computer, a mainframe computer, or any combination of these components. In particular embodiments, server 1020 may be standalone, or it may be part of a subsystem that may be part of a larger system. For example, server 1020 may represent distributed servers that are remotely located and communicate over a network (e.g., network 1040) or a dedicated network such as a local area network (LAN). Additionally, consistent with disclosed embodiments, server 1020 may be implemented as a server, a server system including multiple servers, or a server farm including a load-balancing system and multiple servers.

[0346] The server 1020 may also be configured to interact with the data structure 1030 to store and / or retrieve (read) 3D model content data from the data structure 1030. The server 1020 may communicate with the data structure 1030 directly or via a network 1040. The user system 1010 and / or the server 1020 may be implemented using a computing device 200, as discussed above with respect to FIG. 2. Consistent with embodiments of the present disclosure, the data structure 1030 may have characteristics similar to those of the data structures described above.

[0347] The network 1040 may facilitate electronic communication and exchange of data and / or information between the user systems 1010, the server 1020, and / or the data structure 1030. The network 140 may include any combination of communication networks. For example, the network 1040 may include the Internet and / or another type of wide area network, an intranet, a metropolitan area network, a local area network, a wireless network, a cellular communication network, etc. Although FIG. 10 illustrates only one user system 1010, one server 1020, and one data structure 1030, it is contemplated that the content preview system 100 may include any number of user systems 1010, servers 1020, and / or data structures 1030.

[0348] According to the present disclosure, a computer-implemented system for animating portions of a still image is disclosed. The disclosed system may be used in an automated process for achieving photogrammetry by adding animation to still images, including frames captured from video. The system may be used to provide movable features to any type of still image and to provide perceived articulation, movement, or motion to all or portions of the still image. For example, in an image containing trees, a body of water, or a human head, the leaves, waves, and hair may be rendered movable while other parts of the image remain stationary.

[0349] The disclosed systems may be computer-implemented systems that include one or more computing devices, such as a mobile device, a computer, a server, a cluster of servers, a cluster of servers, a personal computer, a smart device, a tablet, a personal computer, a terminal, a kiosk, a cloud service, a storage device, a specialized apparatus configured to perform methods according to the disclosed embodiments, etc. While this disclosure provides examples of systems, devices, and methods, it should be noted that these disclosures are merely exemplary and are not intended to be limiting on the language of the claims.

[0350] The disclosed systems and processes may enable partial or complete animation of still images. A still image may include a static 2D or 3D image, a frame shot from a motion picture, a portion of a video, or any image that may be rendered still by data capture. A still image may be provided in any format, including .JPG, .BMP, .GIF, .PNG, .SVG, a 3D vector format, a computer-aided design file, or any other still image or model format. A still image may include a 3D mesh representation or a 3D point cloud representation. In some embodiments, an image may be referred to as a scene. In a 2D nature scene, a tree may be an example of a still image. A 3D mesh representing a fan may be a still image. A 3D point cloud representation of an object may be a still image. While this disclosure provides some example file formats, it should be noted that aspects of this disclosure in their broadest sense are not limited to the specific disclosed file formats. Any format in which an image or part of an image is rendered completely or partially motionless may underlie a still image according to the present disclosure.

[0351] In some embodiments, a system for animating still images may include at least one processor, an example description of which is described above and also with reference to FIG.

[0352] In some embodiments, the processor may be configured to receive a still image of an object. The object may be a real-world, fictional, or virtual element or feature, a combination of elements or features, or a portion of an element or feature present in the image. For example, in a still image of an apple, the apple may be considered an object along with its stem and leaf; the apple and stem may be considered an object, and the leaf may be considered a second, separate object; or the apple may be considered a first object, the stem may be considered a second object, and the leaf may be considered a third, separate object. An object may include a group of points or polygons within a mesh based on relationships between the points or polygons. While this disclosure provides examples of objects, it should be noted that, in their broadest sense, aspects of this disclosure are not limited to the disclosed examples.

[0353] In some embodiments, the processor may be configured to perform a search (lookup) to identify at least one image of a similar object stored in the memory. Similar objects may have at least partially similar appearances, properties, attributes, shapes, or characteristics. The similar objects may be based on a similarity metric, for example, based on shape data, color data, and / or data characterizing any other object. The similarity metric may be based on statistical similarity, such as covariance, least-squares distance, distance between vectors associated with image elements (e.g., feature vectors), or Hausdorff distance between aligned objects. The similarity metric may also be based on feature vectors. In some embodiments, a model, such as a machine learning model, may generate the similarity metric. In some embodiments, the comparing may include implementing a classification model (e.g., a random forest model) to classify components of the objects. Identifying objects corresponding to the at least one object may be based on the similarity metric. For example, if the object is a chair, the system may determine that an object in memory is similar to the chair based on a similarity metric between the chair in the scene and image data in a data structure containing 3D models of multiple objects. Exemplary data structures are described above, consistent with embodiments of the present disclosure. Performing a lookup may include searching a memory, a data structure, or other data acquisition system. Searching a data structure for similar objects may include, for example, performing a lookup. Performing a lookup may be performed locally or remotely over a network. While this disclosure provides examples of how to perform a lookup, it should be noted that aspects of this disclosure in their broadest sense are not limited to the disclosed examples.

[0354] In some embodiments, the memory may include segmentation data. The segmentation data may include related data grouped for different processing. For example, segmentation data associated with similar objects may distinguish moving portions of the object from immovable portions of the object. Moving portions allow for movement or perceived movement relative to immovable portions, while immovable portions of the object may remain in a fixed or stationary position or location and may not involve perceived movement. The segmentation data may be associated with all or a portion of the object or similar objects. A moving portion may be any part, part, or all of an object that is capable of movement, motion, translation, or rotation, including changes in speed, velocity, acceleration, or any type of change in position. This may be actual movement or movement, or merely the perception of movement or movement. An immovable portion may be any part, part, or all of an object that remains substantially stationary or cannot change position, speed, velocity, acceleration, or involve any change in position. This may include perceived immobility.

[0355] As an example, an image of a person's head may include moving parts (e.g., hair that moves with the wind) and immovable parts (e.g., other parts of the head that are different from the hair). As an example, an image of a tree may include moving parts (e.g., leaves that may move with the wind) and immovable parts (e.g., the bark of the tree). The segmentation data may include identification of one or more image elements (e.g., mesh points, polygons, pixels, voxels, etc.) that may be associated with moving parts versus immovable parts. The segmentation data may include labels or flags that may indicate that particular basic image elements (e.g., mesh points, polygons, pixels, voxels, etc.) belong to moving or immovable parts. In some embodiments, the segmentation data may identify (label or flag) basic image elements that define boundaries between moving and immovable parts of an image. While this disclosure provides examples of segmentation data, it should be noted that aspects of this disclosure in their broadest sense are not limited to the disclosed examples.

[0356] In some embodiments, the memory may include movement data associated with a moving part. The movement data may include any data related to the movement or perceived movement of an object. For example, the movement data may include any data associated with the degrees of freedom of movement of an object or a component or portion of an object. The movement data may include rotational movement about one or more axes, translational movement about one or more axes or along a line or plane, random movement, data related to a change in the object's orientation or a change in the perceived orientation, velocity data, speed data, or acceleration data. The movement data may be associated with the entire object or only a portion of the object. For example, movement data associated with a chair may include data related to pivoting about a first axis, reclining along a second axis, adjusting the seat height, adjusting the armrest height, or other movement parameters.

[0357] Movement data may also be based on physical laws. For example, a mechanical model of tree leaves may describe their response to wind. When a particular wind model (e.g., including direction, speed, etc.) is applied to a scene, a simulation may be performed to calculate the position of the leaves at multiple points in time. Here, the moving parts may include the leaves, starting from the stem connecting the leaf to the base.

[0358] Movement data may include parameterization of possible movement modes. For example, for a door and its hinges, there may be a rotation operator acting on the door, encoded by a single angle, where the angle varies from zero (the door is closed) to 90 degrees (the door is close to the wall and fully open). Movement data may include scripts that specify the position of an object, such as the function of the minute or second arrow on a manual clock, or scripts that specify the object's response to an external force, such as the motion of a swivel chair when pushed with a specific force from a specific point. Such scripts may be configurable by the system designer, user, or client. In some aspects, the script may specify the orientation, speed, acceleration, rotation, etc. of one or more primitive image elements associated with a movable part relative to one or more primitive image elements of a stationary part or relative to a predefined reference frame. For example, the system may allow changes in the orientation, speed, or degrees of freedom of an object or part of an object during animation by the designer, user, or client. The script may be written in any code capable of animating objects (e.g., AUTOCAD, BLENDER, CINEMA 4D, and / or AUTODESK MAYA). Note that while this disclosure provides examples of movement data, aspects of this disclosure in their broadest sense are not limited to the disclosed examples.

[0359] In some embodiments, the processor may be configured to perform an analysis of image elements within a received still image of an object to segment the still image into discrete components. Consistent with embodiments of the present disclosure, segmentation may additionally or alternatively be performed using the segmentation techniques discussed above. For example, the system may segment an image of a room into discrete components such as chairs, doorknobs, handles, cups, utensils, shoes, walls, plant leaves, carpets, televisions, fans, etc. The system may segment image elements that belong to the discrete components and may classify the components with a known or unknown classification. For example, during segmentation, the discrete components may be labeled as a particular type of object (e.g., a chair), an unknown type of object, and / or a possible known object (e.g., a "likely" chair) based on some measure of confidence or likelihood associated with the output of the segmentation algorithm.

[0360] Examples of segmentation that distinguish moving from non-moving parts in stored images of similar objects are a 3D model of a tree segmented into a stationary trunk and moving leaves, along with a physical model of how the leaves react to the wind, and a 2D image of a human head segmented into different facial parts such as the mouth, eyes, and ears, along with animation of the hair reacting to the wind and animation of the eyelids opening and closing. While this disclosure provides examples of how to segment images, it should be noted that in their broadest sense, aspects of this disclosure are not limited to the disclosed examples.

[0361] In some embodiments, the processor may be configured to compare the discrete components to moving portions of similar objects to identify, in the received image, at least one stationary rendering of a moving discrete component that differs from a stationary component in the still image. Comparing the discrete components to moving portions of similar objects may include any method that allows for comparing the discrete components, including, for example, one or more of the techniques for comparing objects and / or image data discussed above. Such comparison may include, by way of example only, a statistical analysis of similarity or an artificial intelligence-based approach for identifying similarity. In one example, the comparing may include determining a similarity metric indicative of the degree of similarity between the discrete component and the corresponding discrete component of the stored image. For example, the disclosed system may generate or retrieve a feature vector corresponding to the discrete component and compare the feature vector to a feature vector associated with the corresponding discrete component of the stored image or to a feature vector corresponding to the entire stored image. The disclosed system may determine the similarity between the discrete component of the still image and the discrete component of the stored image based on the similarity metric. The similarity metric may be based on statistical similarity such as covariance, least squares distance, or Hausdorff distance between the discrete components of the still image and the discrete components of the stored image.

[0362] Comparing components may include searching a data structure of objects based on the discrete components identified in the still image and generating one or more search results (i.e., matches) of objects within the data structure that may contain similar discrete components. In some aspects, the search results may include another metric, likelihood, or percentage match representing the degree of similarity between the discrete components of the still image and the discrete components of the images stored in the data structure. In some embodiments, the system may identify similar objects based on previously performed searches of the data structure. While this disclosure provides examples of methods for comparing discrete components, it should be noted that aspects of the disclosure in their broadest sense are not limited to the disclosed examples.

[0363] In some embodiments, the processor may be configured to extract a still rendering of the moving discrete component from the still image. As used herein, extracting a still rendering of the moving discrete component may include extracting moving from still or extracting still from moving. Once a portion of the image is identified as moving, any known mechanism may be used to perform such extraction. For example, the processor may extract or separate image elements (e.g., mesh points, polygons, pixels, voxels, etc.) within the still image associated with the discrete component determined to be movable. The extracting may include storing the basic image elements associated with the discrete component in a separate storage location. Additionally or alternatively, the extracting may include labeling or identifying the basic image elements associated with the discrete component in order to distinguish them from basic image elements associated with other portions of the still image. While this disclosure provides examples of methods for extracting a still rendering of the moving discrete component, it should be noted that aspects of the disclosure in its broadest sense are not limited to the disclosed examples.

[0364] In some embodiments, the processor may be configured to use the static rendering and movement data to construct a movable version of the static rendering of the movable component. The processor may also use the movement data to construct a movable version of the static rendering. As discussed above, in some embodiments, the movement data may include a script associated with the movable discrete component. The movement data may include a script that encodes specific dynamic movement or material properties and / or a programmatic description of such movement and material properties. The processor may apply the script to extracted basic graphical elements associated with the movable discrete component. For example, the processor may associate a programmatic description or script of movement with one or more of the basic graphical elements associated with the static rendering of the movable discrete component. The programmatic description may specify changes in color, lighting, orientation, scaling, texture, and / or other material properties over time for the basic graphical elements (e.g., mesh points, polygons, pixels, voxels, etc.) associated with the movable discrete component that may give the appearance of movement when viewed by a user. While this disclosure provides examples of how to build working versions of discrete components, it should be noted that aspects of the disclosure in its broadest sense are not limited to the disclosed examples.

[0365] For objects with moving parts, the processor may identify connecting elements between the stationary and moving parts of the object. For example, in a fan, the processor may identify a cylindrical element (e.g., a connecting rod) between the stationary and moving parts of the fan. The connecting element may have a distinct shape that can enable the processor to identify it in the image of the fan stored in the data structure and in the still image, along with its position relative to the entire object. The connecting elements may be matched between the scene fan and the database fan. Furthermore, using methods such as principal component analysis (PCA), the processor may align the connecting elements from the images in the data structure with the connecting elements in the still image.

[0366] In some embodiments, a processor may construct a hybrid image by combining a stationary component of a still image with a constructed movable version of a stationary rendering of the moving component, such that a movable version of the moving discrete component can move within the hybrid image while the stationary component from the still image remains stationary. The disclosed systems and processes may rely on the alignment of critical connection elements within the still image. For example, a rotatable fan may include a connecting member (e.g., a rod or tube) connecting a non-moving portion of the fan to a rotatable portion of the fan. During generation of the hybrid image, the disclosed systems and processes may help ensure that critical connection elements in the constructed movable version of the stationary rendering of the moving component align as closely as possible with corresponding critical connection elements in the still image. Such close alignment may help ensure that the inserted image of the moving component appears natural in the resulting hybrid image.

[0367] Consistent with disclosed embodiments, generating a hybrid image may include inserting a movable version of a stationary rendering of the movable component into the still image. Thus, for example, the processor may combine a primitive image element (including associated movement data) associated with the movable version of the stationary rendering of the movable component with a primitive image element associated with a stationary portion of the still image. Generating the hybrid image may include placing the movable version in the same orientation (i.e., aligning the objects) and similar size (i.e., scaling the objects) as the stationary rendering of the discrete component in the still image received by the processor. By way of example, aligning the movable version with the stationary portion of the still image may include an affine transformation that transforms the (x, y, z) coordinates of the primitive element of the movable version of the image to T(x, y, z), the desired position of the element in the stationary image coordinates. In other embodiments, generating a hybrid image may include combining a movable version of the image with a stationary portion of the still image by taking a union of the two families of image elements.

[0368] In yet another embodiment, generating a hybrid image may include combining characteristics of image elements of the movable version with image elements of the stationary portion of the still image to obtain a blended image. For example, the movable version and still image of a discrete component may include a family of polygons. Each polygon may be associated with a texture. The texture may be a 2D mapping from the image to the polygon, representing how the polygon will appear to a viewer (e.g., different parts of the polygon may have different colors). The alignment T of the stationary portion of the still image and the movable version may be used to determine a match for the corresponding polygon families. For example, a polygon from the movable version of the discrete component may be mapped to a polygon in the still image using a transformation T to find the location of the polygon in the movable version that is closest to the polygon in the still image. Using the match, the system may match vertices of polygons in the image data of the still image and the movable version. The disclosed system may also transfer color, texture, material properties, etc. from polygons in the image data of the movable version to polygons in the still image. In some embodiments, object alignment and / or object scaling may involve the use of principal component analysis (PCA).

[0369] Generating a hybrid image may also include using image processing techniques (e.g., adjusting brightness, adjusting lighting, implementing gradient domain methods, etc.) consistent with disclosed embodiments. As one skilled in the art will recognize, gradient domain methods may include constructing a hybrid image by integrating gradients of basic image elements of moving versions of discrete components with gradients of image elements of stationary portions of a still image. In a hybrid image, stationary portions of a still image included in the hybrid image may remain stationary, while moving versions of discrete components may move according to associated movement data. Additional or alternative techniques for combining two images (e.g., a first image and a second image) discussed above may also be used to construct a moving version of a stationary rendering of a moving component and / or generate a hybrid image consistent with embodiments of the present disclosure.

[0370] In some embodiments, the processor may be configured to output the hybrid image. Outputting the hybrid image may include displaying the image, storing and / or transmitting the image, consistent with disclosed embodiments. Transmitting may include transmitting over a network by any known method, consistent with disclosed embodiments. For example, the system may broadcast the hybrid image (i.e., transmit over a network to multiple user devices), transmit the hybrid image to a user device, and / or store the hybrid image in memory.

[0371] 11 illustrates an example object (e.g., a fan 1100) in an input 3D scene being viewed by a user consistent with embodiments of the present disclosure. The fan may include four legs 1104, a pole 1103, and a fan head including a movable fan element 1102 enclosed in a safety cage 1101. The disclosed system may be configured to segment an image of the fan 1100 into discrete components such as the four legs 1104, the pole 1103, the fan element 1102, and the safety cage 1101.

[0372] FIG. 12 illustrates example fans 1201 and 1202 in data structure 1030. Fans 1201 and 1202 may each include discrete components 1203 and 1205, which may be moving, and discrete components 1204 and 1206, which may be stationary. The disclosed system may compare discrete component 1102 of fan 1100 to images stored in the data structure. The disclosed system may identify one or more fans (e.g., 1201 and 1202) in the data structure that have similar moving discrete components 1203 and 1205 as discrete component 1102. For example, the disclosed system may compare segmented parts 1101, 1102, 1103, and 1104, respectively, to segmented parts 1203 and 1204 of fan 1202 or other fans identified in the data structure. The system may identify a match for each input fan component 1101, 1102, 1103, 1104 to a fan component stored in the data structure (e.g., leg for leg, pole for pole, etc.).

[0373] In the data structure, each fan segment is Mobility or immobilityThe motion of the fan 1201 may be described as a discrete component of the fan 1102, and the mode of motion of the mobile element may also be described. Thus, for example, in the fan 1202 illustrated in FIG. 12, the discrete fan component 1203 may be mobile, while the legs 1204 may be immobile. As discussed above, the disclosed system may extract a static rendering of the discrete fan component 1102 from the image of the fan 1100 (FIG. 11). The disclosed system may combine the extracted static rendering of the fan component 1102 with the motion data (e.g., script) associated with the matching discrete component 1203 of the fan 1202 or the matching discrete component 1205 of the fan 1201 stored in a data structure to create a mobile version of the static rendering of the fan component 1101. As discussed above, the disclosed system may generate a hybrid image by combining a moving version of a static rendering of fan component 1101 (including movement data for fan component 1203 or 1205) with a static rendering of the stationary portions of fan 1100 (e.g., 1103, 1104). The resulting hybrid image of fan 1100 will show fan component 1101 as moving according to the movement data for fan component 1203 or 1205, while the other components 1103, 1104 may remain stationary.

[0374] A comparison of each segment in the segmented input fan / tree with each mobile segment in the pre-segmented data structure fan / tree may be an example of a method for comparing discrete components to moving parts, where the discrete components are a segmentation of the input fan into parts, and the comparison may include a check for 2D or 3D object similarity. The system may determine that the image includes the input fan and further determine that the "top" of the fan (e.g., fan cage, blade control unit) may be an integrated segment. The system may identify similar integrated segments in the fan stored in the data structure. The system may also determine from the data structure that the identified integrated segment is mobile. Extracting the top of the input fan, along with determining that it is mobile, may be an example of a static rendering of a moving discrete component from a static image.

[0375] In some embodiments, the system may determine that the top of the leaf of the input fan / tree is mobile. The system may determine the mode of movement. In some exemplary embodiments, the system may determine additional information, such as a point-to-point matching between the top of the input fan and the top of the data structure fan (similarly for trees). The system may import the mode of movement from the data structure top to the input top by simply combining the mode of movement of the data structure fan with the point-to-point matching of the input fan. The top of the input fan is now mobile, with a known mode of movement. This may be an example of using a static rendering and movement data to build a movable version of a static rendering of a movable component.

[0376] In some embodiments, the system may not be able to determine a point-to-point match between the input upper fan and the data structure upper fan. However, the system may replace the input upper fan with the data structure upper fan by, for example, splicing an existing upper fan from the scene and inserting the data structure in the same position and orientation using the exemplary method for generating a hybrid image discussed above. The system may replace the input upper fan with the data structure upper fan. This case of splicing and insertion may be an example of constructing a hybrid image by combining an immovable component of a still image with a constructed movable version of a still rendering of the movable component.

[0377] In either case above, the resulting object may be partially animated by applying a motion script to the upper fan (where the script is either original or imported in the case of splicing) while keeping the lower fan fixed. This is an example of allowing a moving version of a moving discrete component to move within the hybrid image while an immobile component from the static image remains stationary.

[0378] In some embodiments, the still image may include a person's head, the discrete components may include the person's head and hair, and at least one processor may be configured to cause the head to remain stationary and the hair to move in the hybrid image. For example, the processor may receive a still image of a person's head or face. The processor may segment the still image into discrete components representing, for example, hair portions and remaining immobile portions of the head. The processor may compare the discrete components (e.g., hair and immobile head portions) to objects in the data structure. The processor may identify several images of the head in which the hair may be moving (e.g., due to wind). The processor may identify a best match between the received still image and an image in the data structure. The processor may extract portions of the received still image associated with the hair and apply movement data (e.g., a script or other programmatic description) to the extracted portions to create a rendering of the movable hair portion. The processor may combine renderings of the immobile head portion and the movable hair portion in the received still image to create a hybrid image. In the hybrid image, the hair pieces may exhibit movement according to the movement data retrieved from the data structure.

[0379] In some embodiments, the still image may include a body of water, the discrete components may include waves and a shoreline, and at least one processor may be configured to cause the shoreline to remain stationary and the waves to move in the hybrid image. For example, the processor may receive a still image of an ocean, lake, sea, or other body of water. The processor may divide the still image into discrete components representing, for example, waves and stationary portions of the shoreline. The processor may compare the discrete components (e.g., waves and stationary shoreline) to objects in the data structure. The processor may identify several images of the body of water in which waves may be moving. The processor may identify a best match between the received still image and the images in the data structure. The processor may extract portions of the received still image associated with waves and apply motion data (e.g., a script or other programmatic description) to the extracted portions to create a moving wave rendering. The processor may combine the moving wave rendering with the stationary shoreline portion in the received still image to create a hybrid image. In the hybrid image, the waves may exhibit movement according to movement data retrieved from the data structure. In this example, the critical connected components may include the ocean floor (or an abstract floor that may be deep enough that water below that level does not affect the animation). The disclosed system may simulate the movement of each image element in the object in the data structure corresponding to the waves and apply it to the corresponding image element in the static image.

[0380] In some embodiments, the still image may include a tree, the discrete components may include a trunk and leaves, and at least one processor may be configured to cause the trunk to remain stationary and the leaves to move in the hybrid image. For example, a processor may receive a still image of a tree. The processor may segment the still image into discrete components representing, for example, leaves and the remaining immovable trunk. The processor may compare the discrete components (e.g., leaves and immovable trunk) to objects in the data structure. The processor may identify several images of trees whose leaves may be moving (e.g., due to wind). The processor may identify the best match between the received still image and the images in the data structure. The processor may extract portions of the received still image associated with leaves and apply movement data (e.g., scripts or other programmatic descriptions) to the extracted portions to create a rendering of the movable leaves. The processor may combine the rendering of the movable leaves with the immovable trunk in the received still image to create a hybrid image. In the hybrid image, the leaves may exhibit movement according to movement data retrieved from the data structure. In another embodiment, the scene viewed by the user may be two-dimensional and may include, for example, a landscape nature image including a tree. The disclosed system may recognize trees in the scene as separate objects. The system may segment them (e.g., by color) into trunks, branches, and leaves. The system may search for similar tree images in the data structure. The system may be able to match leaves of an input tree with leaf segments of a tree image from the data structure (hereinafter referred to as a data structure tree). The system may determine that the data structure tree contains leaf animation. The system may import this animation into the leaves of the input tree, thereby creating a new "live" tree from a static tree.

[0381] In some embodiments, the still image may include a person, and the discrete components may include the person's body and clothing, and at least one processor may be configured to cause the body to remain stationary in the hybrid image while the clothing moves. For example, a processor may receive a still image of a person. The processor may segment the still image into discrete components representing, for example, the clothing and the person's remaining, immobile body. The processor may compare the discrete components (e.g., immobile body portions of the clothing) to objects in the data structure. The processor may identify several images of the body where the clothing may be moving (e.g., due to wind or human movement). The processor may identify a best match between the received still image and the images in the data structure. The processor may extract portions of the received still image associated with the clothing and apply movement data (e.g., a script or other programmatic description) to the extracted portions to create a rendering of the movable clothing. The processor may combine the rendering of the movable clothing with the immobile body portions in the received still image to create a hybrid image. In the hybrid image, the clothing may exhibit movement according to the movement data retrieved from the data structure, and the clothing may be a head cover, a shirt, a jacket, a scarf, pants, jeans, a dress, a skirt, or other clothing item.

[0382] In some embodiments, the still image may include a clock, the discrete components may include the clock face and hands, and at least one processor may be configured to cause the clock to display different times in the hybrid image. For example, the processor may receive a still image of a clock. The processor may segment the still image into discrete components representing, for example, the hands and the remaining, immovable dial portions. The processor may compare the discrete components (e.g., the hands and the immovable dial portions) to objects in the data structure. The processor may identify several images of the clock whose leaves may be moving. The processor may identify the best match between the received still image and the images in the data structure. The processor may extract portions of the received still image associated with the hands and apply movement data (e.g., a script or other programmatic description) to the extracted portions to create a rendering of the movable hands. The processor may combine the rendering of the movable hands with the immovable dial portions in the received still image to create the hybrid image. In the hybrid image, the hands may move according to movement data retrieved from the data structure and display different times. In this example, the critical connected component may be a point at the center of the hands around which the hands rotate. The coordinate system may include this center point along with a vector perpendicular to the plane of the clock that passes through this center point. The movable element may include an affine transformation associated with a rotation about this perpendicular vector. The disclosed system may match the plane and center of the clock to transfer the transformation from the image of the clock in the data structure to a static image of the clock.

[0383] In some embodiments, the still image may include a pet, the discrete components may include a body and fur, and at least one processor may be configured to cause the body to remain stationary and the fur to move in the hybrid image. For example, the processor may receive a still image of a pet. The pet may include a dog, cat, horse, mouse, or other pet. The processor may segment the still image into discrete components representing, for example, fur and immobile body parts. The processor may compare the discrete components (e.g., immobile body parts) to objects in a data structure. The processor may identify several images of the pet whose fur may be moving. The processor may identify a best match between the received still image and an image in the data structure. The processor may extract portions of the received still image associated with the fur and apply movement data (e.g., a script or other programmatic description) to the extracted portions to create a rendering of the movable fur. The processor may combine the rendering of the movable fur with the immobile body parts in the received still image to create a hybrid image. In the hybrid image, the fur may exhibit movement according to movement data retrieved from the data structure.

[0384] In some embodiments, the still image may include an animal, the discrete components may include a body and a tail, and at least one processor may be configured to cause the body to remain motionless in the hybrid image and the tail to move. For example, the processor may receive a still image of an animal. The animal may be a cat, a dog, a mouse, a horse, or any other animal. The processor may segment the still image into discrete components representing, for example, the tail and the immobile body parts. The processor may compare the discrete components (e.g., the tail and the immobile body parts) to objects in a data structure. The processor may identify several images of the animal whose tail may be moving. The processor may identify a best match between the received still image and the image in the data structure. The processor may extract a portion of the received still image associated with the tail and apply movement data (e.g., a script or other programmatic description) to the extracted portion to create a rendering of the movable tail. The processor may combine the rendering of the moving tail with the immobile body portion in the received still image to create a hybrid image in which the tail may exhibit movement according to the movement data retrieved from the data structure.

[0385] In another embodiment, the system may allow a user to select an object, parse the object into discrete components designated by the user, and configure motion data for each discrete component. Such user-defined discrete components would allow the user to separate an image into any combination of stationary and moving discrete components. The system may include a feature that allows the user to customize the motion data to create a desired scene or image. A user may capture a still image of an analog clock and parse the clock face into one discrete component, the clock's hour hand into a second discrete component, and the clock's minute hand into a third discrete component. The user may then customize the motion data by configuring the motion data for each discrete component. Such customization may include specifying the axis of rotation, the translation path, the direction of rotation or translation, the speed, velocity, or acceleration of a component, or any parameter associated with motion or perceived motion. This may allow the user to animate the clock, rotating the dial face in a first direction at a desired speed, the minute hand in a second direction at a desired speed, and the hour hand in a desired direction at a third speed.

[0386] In some embodiments, the system may allow a user to select whether to animate an object. The system may allow a user to adjust characteristics associated with the movement data. For example, the system may prompt the user to select or adjust a desired rotation speed, direction, range, or other parameter associated with the motion or perceived motion of a discrete component. For example, in the above example of a fan including movable fan elements 302 and a movable cage 301, the system may prompt the user to select or adjust the direction 408 of the fan elements 302 and the speed 408 of oscillation of the movable cage 301. Note that the system may prompt the user to select or adjust more than two characteristics associated with the movement data. The disclosed system may render a new 3D animation 407 with the fan rotating in a selected rotation mode based on the user's selection.

[0387] In another embodiment, the system may allow a user to select an object in an image from a plurality of objects. In some embodiments, the processor may be configured to detect multiple objects in a scene or image and prompt the user to select which object to animate. The system may search for object images in a data structure based on the user's selection. The object images in the data structure may be similar to or substantially different from the object selected by the user. The user may select at least one desired object in the data structure. The data structure includes segmentation data associated with moving portions of the selected object and immovable portions of the selected object. The system may allow the user to select a desired image and create a hybrid image based on the selection made by the user. For example, a user may select a shark in a still image. The system may allow the user to search for images of the object in a data structure, where the user selects a clown. The segmentation data associated with the clown may include movable arms and a head. The system may create an animated clown-shark hybrid image, where the body of the shark remains stationary but the head and arms move. The system may further allow the user to customize which parts of the object move within the hybrid image and define the variables associated with that movement.

[0388] FIG. 13 illustrates an example method 1300 for animating portions of a still image consistent with embodiments of the present disclosure. The steps of method 1300 may be performed by a component of system 1000 (e.g., user system 1010 or server 1020) consistent with the methods described above. In step 1301, system 1000 may receive a still image of an object. In step 1302, system 1000 may perform a lookup to identify at least one image of a similar object stored in memory. The memory may include segmentation data that distinguishes moving portions from stationary portions in the stored images of similar objects and may include movement data associated with the moving portions. In step 1303, system 1000 may perform an analysis of voxels in the received still image of the object to segment the still image into discrete components. In step 1304, the system 1000 may compare the discrete components with the moving portions of at least one similar object to identify in the received image at least one static rendering of the moving discrete component that is distinct from the static component of the static image. In step 1305, the system 1000 may extract the static rendering of the moving discrete component from the static image. In step 1306, the system 1000 may construct a movable version of the static rendering of the moving component using the static rendering and movement data. In step 1307, the system 1000 may construct a hybrid image by combining the static component of the static image with the constructed movable version of the static rendering of the moving component, thereby allowing the movable version of the moving discrete component to move in the hybrid image while the static component from the static image remains stationary. In step 1308, the system 1000 may output the hybrid image.

[0389] The present disclosure also relates to a computer-implemented system for generating a complete 3D model of one or more objects based on partial images of the objects. According to the present disclosure, a system for augmenting or reconstructing a 2D or 3D scene or image may include several components that may interact and cooperate with each other. By way of example, FIG. 10 illustrates an exemplary system 1000 consistent with the present disclosure. FIG. 2 illustrates an exemplary computing device 200 for implementing embodiments and features of the present disclosure.

[0390] According to the present disclosure, a computer-implemented system for simulating a complete 3D model of an object from incomplete 3D data is disclosed. The disclosed system may be a computer-implemented system including one or more computing devices, such as a mobile device, a computer, a server, a cluster of servers, a cluster of multiple servers, a personal computer, a smart device, a tablet, a terminal, a kiosk, a cloud service, a storage device, or a specialized device configured to perform methods according to the disclosed embodiments. The disclosed system may be used in a process for simulating or creating a complete 3D model of an object from incomplete data, such as a partial image or scene. While the present disclosure provides examples of systems or devices, it should be noted that aspects of the present disclosure in its broadest sense are not limited to the disclosed examples.

[0391] Disclosed embodiments include a computer-implemented system for processing images or scenes (e.g., scan-based 3D scenes) for use in virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies and applications. In some embodiments, the system may include at least one processor. An exemplary description of a processor and memory is described above and with reference to FIG. 2. By way of example, as shown in FIGS. 2 and 10, a system 1000 may include one or more processors 202 included in one or more user systems 1010 and a server 1020. As used in this disclosure, the term "processor" is used as a shorthand to refer to "at least one processor." The processor may be configured to access memory. Alternative and additional descriptions of the processor and memory are also described above with reference to FIG. 2.

[0392] In some embodiments, the processor may be configured to receive a partial image of an object, where the partial image is at least one of a 2D image or an incomplete 3D image. For example, the processor may receive the partial image from an image sensor, memory, a data structure, a look-up table, or by uploading from a user. Furthermore, the processor may receive the partial image via any data communication channel or using any data transfer protocol. The image, whether 2D or 3D, may be captured by any type of camera or imager. Non-exhaustive examples of 3D imagers include stereo cameras, range cameras, 3D scanners, laser range finders, lenticular devices, time-of-flight cameras, structured light cameras, and vectographic cameras. The 2D image may be captured by a CCD array or other digital camera technology. As used broadly herein, an image captured by any of the above or other image capture devices may be considered a “scan.”

[0393] An object may be a real-world, fictional, or virtual element or feature, a combination of elements or features, or a portion of elements or features present in an image. For example, in a still image of an apple, the apple may be an object together with its stem and leaf, the apple and stem may be one object and the leaf may be a second, separate object, or the apple may be a first object, the stem a second object, and the leaf a third, separate object.

[0394] A partial image may include only a portion of an object. A partial image may include less than the entire image of an object or scene. For example, a partial image may be the result of an incomplete image or scan, data loss due to data transfer, image noise, an object being obscured by another object, or other reasons that prevent a complete image of the object from being captured. For example, an incomplete scan may capture part of an object, such as the front of a chair but not the back of the chair. A scan of only the front of a chair is an example of incomplete 3D data. In some cases, an incomplete scan may result when a scan obtains less information than is necessary to represent the complete object in 3D. This partial scan of a chair is also an example of a partial image of an object. Another example of a process that can lead to a partial image of an object may include an object being scanned by a 3D scanner, but not scanned from all directions. Another example relates to an object being occluded by another object, or a portion of an object being occluded by another portion. In this example, the chair may be positioned behind a table during scanning. In another example, a segmentation algorithm may not accurately separate an object, portions of an object may be missing, or redundant parts may be present. In another example, an object may be scanned inaccurately due to low scanner resolution or poor lighting conditions. An incomplete scan may occur if the scan does not capture actual object characteristics such as color information, or due to limitations in the scan input method or other errors (e.g., a partially transparent object may not appear in the scan). In some cases, an incomplete scan may result if the scan misses parts of the scene data (e.g., a hole in the scene or an obscured part of the scene) or if other errors or distortions occur due to hardware limitations or user error during scanning.While this disclosure provides examples of partial images of objects, it should be noted that aspects of this disclosure in its broadest sense are not limited to the disclosed examples.

[0395] An image of all or a portion of an object may be represented as a mesh, a point cloud, or other representation that encodes the image or scene. In an exemplary embodiment, the image or scene may include a 3D representation encoded as a mesh as discussed above. In some embodiments, the scene or image may include multiple existing image elements. In some embodiments, the system may generate the image elements of the image based on the received scan, consistent with the disclosed embodiments. More generally, the image or scene may include multiple image elements, such as image elements (2D or 3D). For example, the image elements may include at least one of pixels, voxels, points, polygons, etc. In some embodiments, the system may generate a set of polygons, where each polygon is an image element. As another example, if the system generates a point cloud, each point may be an image element. For example, the system may generate a mesh composed of multiple n-sided polygons or voxels as image elements, where one or more polygons may be subdivided into additional polygons to improve resolution or for other reasons.

[0396] In some embodiments, the processor may use the partial image to search at least one data structure for additional information corresponding to the partial image. Consistent with embodiments of the present disclosure, exemplary data structures are described above. The data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). In some embodiments, the at least one data structure may include additional information corresponding to a 3D object and / or multiple 3D objects. The additional information and / or image data including the additional data in the data structure may include a 2D or 3D model of the object. A data structure consistent with the present disclosure may include one or more computer-aided design (CAD) models corresponding to one or more objects. The CAD models may be stored in one or more formats, such as a mesh, a point cloud, a voxel mapping of a 3D space, and / or any other mapping that can be configured to present a graphical representation of the object. The CAD models may represent the object and / or components of the object (e.g., a chair and / or chair arms).

[0397] In some embodiments, the additional information and / or data may include any information or data related to the inherent, extrinsic, or intrinsic characteristics of a real or fictitious object. For example, the additional information or data may include information or data related to color, texture, shape, contour, material, flexibility properties, mass properties, orientation, physical properties, or any other characteristic. The additional information or data may be entered into the system by a user as part of an information repository, via a neural network or self-learning system, or accumulated by the system over time by other methods enabling data collection. In some embodiments, the additional information or data may be object-specific, group-specific, category-specific, or even specified and defined by a user. The additional information or data may be linked to a particular image, scene, or model within a data structure. In some embodiments, the additional information may include a 3D model corresponding to the received partial image. In some embodiments, the additional information may include information derived from a partial scan of at least one object similar to the object in the partial image. The additional information or data may be a collection of partial image scans or image-based data associated with an object or set of objects. While this disclosure provides examples of additional information or additional data, it should be noted that aspects of the disclosure in its broadest sense are not limited to the disclosed examples.

[0398] Information stored in the data structure (e.g., a CAD model, additional information, etc.) may be associated with semantic tags or one or more spatial semantic graphs. For example, the spatial semantic graph and semantic tags associated with a 3D model stored in the data structure may represent the stored information in words and textual form. For example, textual expressions such as "table," "shelf," and "chair" may be semantic tags associated with the corresponding table, shelf, and chair object models stored in the data structure, respectively. In some embodiments, the semantic tags may include classifications of 3D objects. As will be apparent to those skilled in the art, classes may be defined in a hierarchy of broader or narrower classes relative to each other. For example, a "furniture" class may be broader than a "chair" class, which may be broader than an "office chair" class. In other embodiments, the semantic tags may represent environments or 3D scenes such as "office," "living room," or "kitchen."

[0399] The disclosed system may compare the semantic tags or spatial semantic graphs of the partial image with the semantic tags and / or spatial semantic graphs of models, objects, and other information stored in a data structure. Consistent with embodiments of the present disclosure, the disclosed system may additionally or alternatively compare the semantic tags or spatial semantic graphs using one or more techniques for comparing objects and / or images discussed above. In some embodiments, if desired, the disclosed system may segment one or more models or objects stored in the data structure before comparing the semantic tags or spatial semantic graphs. Consistent with embodiments of the present disclosure, segmentation may additionally or alternatively be performed using the segmentation techniques discussed above. The system may identify one or more models and / or other additional information associated with the partial image that have the closest or most similar semantic tags or spatial semantic graphs. The system may determine closeness or similarity based on statistical similarities such as covariance, least-squares distance, distance between vectors (e.g., feature vectors) associated with image elements, and Hausdorff distance between aligned objects. In some embodiments, the system may determine proximity or similarity based on a comparison of feature vectors associated with the subimage and additional information stored in a model, object, or data structure. In some embodiments, the system may identify similar objects based on a previously performed search of the data structure. While this disclosure provides examples of methods for searching at least one data structure, it should be noted that aspects of the disclosure in its broadest sense are not limited to the disclosed examples.

[0400] In some embodiments, one or more processors may be configured to determine that the data structure does not include a corresponding 3D model of the object. For example, the processor may determine that a measure of similarity is below a threshold, indicating that the partial image does not sufficiently match the additional information and / or model stored in the data structure, as discussed above. By way of example, if the partial image represents an object (e.g., glass) and the data structure includes a model of a furniture object, the processor may determine that the data structure does not include a 3D model corresponding to the glass object.

[0401] In some embodiments, one or more processors may be configured to search at least one data structure for a reference 3D model that is different from but has similarity to the object in the partial image. The processor may perform the search by comparing the partial image to the model and information stored in the data structure using a process similar to that discussed above. The processor may determine that the data structure contains a 3D model that is different from but similar to the object if, for example, a measure of similarity (e.g., between the semantic tags, the spatial semantic graph, and / or the feature vectors) exceeds a certain threshold. In some embodiments, the determination may include tagging, labeling, identifying, or otherwise classifying one or more elements of the image, scene, or object to indicate the similarity.

[0402] A 3D model stored in a data structure can be considered similar to an object in a subimage based on at least one characteristic. In some embodiments, similarity may include any element or portion of an image or scene that resembles another element or portion of the image or scene. Such similarity may be determined based on one or more of the size, shape, orientation, color, texture, pattern, position, or other features or characteristics associated with the subimage or portion thereof. For example, similarities to an object in a subimage include CAD or other 3D object representations that may include parts with a similar three-dimensional shape to the subimage, 3D object representations that may have textures similar to those appearing in the subimage, and 3D object representations of objects that may be in the same class as the subimage, such as a chair. As illustrated in FIG. 14 , a chair reference model 1410 has been identified for chair 1400. Reference model 1410 is not identical to chair 1400, but has similar features, such as a seat, legs 1411, and backrest 1412. While this disclosure provides examples of methods for identifying objects that have similarity to objects in a subimage, it should be noted that aspects of this disclosure in its broadest sense are not limited to the disclosed examples.

[0403] In some embodiments, the processor is configured to compare the partial image with the reference 3D model to determine portions of the reference 3D model that generally correspond to missing characteristics of the partial image. As previously discussed, missing characteristics may include missing data, features, or portions due to lack of data transfer, low-quality scans, noise, hidden objects, etc. By way of example, the system may detect (find) a reference 3D model in a data structure that has similarity to an object in the partial image. To understand which portions of the reference 3D model correspond to missing data from the scan, for example, the disclosed system may locate portions of the 3D reference image that generally correspond to missing characteristics of the partial image. One way to do this may be to determine the optimal orientation and scale at which portions of the reference 3D model best match the scanned object, and then consider all 3D object representations that do not have a direct match to parts in the scan as "missing" from the scan.

[0404] For example, the disclosed system may determine which object best matches the partial image and further determine which view of the data structure object matches the view of the partial image. The disclosed system may use this determination to obtain a rough alignment of the partial image and the object from the data structure. The disclosed system may use various computing algorithms (e.g., an Iterative Closest Point (ICP) algorithm) to obtain a point-to-point match between the two objects, keeping in mind that a significant portion of the 3D object representation obtained from the data structure will not match the partial image. The disclosed system may also determine a transformation that must be applied to the 3D object representation of the object from the data structure so that its position aligns with that of the existing partial image when inserted into the scanned scene. Based on this mapping between the partial image and the 3D object representation obtained from the data store, the disclosed system may determine the portion of the partial image that matches the 3D object representation. The disclosed system may additionally or alternatively use alignment and scaling techniques such as those discussed above in connection with combining two images.

[0405] The processor may perform the above operations by comparing image elements (e.g., pixels, voxels, polygons, mesh points, etc.) of the partial image with the 3D model retrieved from the data structure. Comparing image elements may include aligning the 3D model and the partial image in the same orientation (i.e., object alignment) and similar size. By way of example, aligning the 3D model from the data store and the partial image may include an affine transformation that transforms the (x, y, z) coordinates of an image element in the partial image to, or vice versa, T(x, y, z), the desired location of that element in the 3D model coordinates. After alignment and scaling, the processor may compare the image elements (e.g., pixels, voxels, polygons, mesh points, etc.) of the 3D model and the partial image to identify image elements in the 3D model that may be missing (i.e., not present) in the partial image. The processor may label, flag, or tag these missing image elements as corresponding to missing characteristics of the sub-image.

[0406] For example, as illustrated in Figure 14, a partial image 1400 of a chair is shown. The partial image 1400 is incomplete, missing data 1403 associated with the legs 1405 and the back of the chair at 1402. The partial image 1400 is also missing data as a result of noise or blurring at 1404. Additionally, the fourth leg 1405 of the image is missing from the image.

[0407] In some embodiments, the processor may combine the partial images with additional information or data to construct a simulated, complete 3D model of the object. The simulated, complete 3D model of the object may include features of the object from the partial images and features from the reference 3D model. Although referred to as "full," a simulated, complete 3D image may still be somewhat incomplete in that it may or may not include all details of the object. Even so, it may still be considered "full" within the context of this disclosure because it is fuller than a partial image. Generating the simulated, complete 3D model may include any manipulation, including replacing elements or features, blending elements or features, modifying elements or features, combining elements or features, or merging the partial image with the reference 3D model. Consistent with this disclosure, constructing the complete 3D model may include combining object data in the partial images with additional data associated with the 3D model of the object stored in a data store. For example, the system may replace a partial image with another object or part of another object. Such changes may be made automatically or at the direction of a user or client.

[0408] Replacing an object or portion of an object may include placing a selected object with the same orientation (i.e., aligning the object) and similar size (i.e., scaling the object) as the original object. In some embodiments, object alignment and / or object scaling may include affine transformation and / or principal component analysis (PCA) as discussed above. Replacing an object may include allowing a client to position, scale, or otherwise manipulate a selected object within a scene. Replacing an object may include image processing techniques (e.g., adjusting brightness, adjusting lighting, performing gradient domain methods, etc.) consistent with disclosed embodiments. As will be appreciated by those skilled in the art, gradient domain methods may include, for example, constructing a simulated complete 3D model by integrating gradients of image elements in a 3D model obtained from a data structure with gradients of image elements in a sub-image. Replacing may include rendering a mesh, points, or other digitized representation of the object based on lighting, scene resolution, perspective, etc. Following object replacement, the resulting scene may be an example of a scene that has been modified by combining information obtained from the original subimage and a 3D model of the object obtained from a data store.

[0409] In some embodiments, generating a simulated complete 3D model may include combining a partial image with a 3D model retrieved from a data structure, for example, by taking the union of the two families of image elements. In yet other embodiments, generating a hybrid image may include combining characteristics of image elements of a 3D model retrieved from a data store and image elements of a partial image to obtain a blended image. For example, the partial image and the 3D model of the data store may each include a family of polygons. Each polygon may be associated with a texture. The texture may be a 2D mapping from an image to a polygon, where the polygon appears to a viewer (e.g., various parts of the polygon may have different colors). An alignment T of the data store 3D model and the partial image (e.g., using an affine transformation) may be used to determine matching of corresponding polygon families. For example, a polygon from the partial image may be mapped to a polygon in the data store 3D model using the transformation T to find the closest polygon in the partial image to a polygon in the 3D model. Using matching, the system may match vertices of polygons in the 3D model and the subimage. The disclosed system may also transfer color, texture, material properties, etc. from polygons in the 3D model to polygons in the subimage, or vice versa.

[0410] In some embodiments, the combining may include meshing the partial image with the determined portion of the 3D reference model. As discussed above, the processor may compare image elements (e.g., pixels, voxels, polygons, mesh points, etc.) of the 3D model and the partial image obtained from a data structure (i.e., the 3D reference model) to identify image elements in the 3D model that may be missing (i.e., not present) in the partial image. In some embodiments, the processor may transfer these image elements to the partial image by, for example, copying properties such as coordinates, position, orientation, color, lighting, texture, material properties, etc., from the 3D model to the partial image. Thus, the processor may complete the mesh in the partial image by copying properties of the corresponding image elements from the 3D model, thereby meshing the partial image with the determined portion of the 3D reference model. Additional or alternative techniques for combining two images (e.g., a first image and a second image) discussed above may also be used to generate a simulated complete 3D model of the object.

[0411] For example, as illustrated in FIG. 14 , partial image 1400 is combined with additional information or data to construct a simulated complete or complete 3D model 1420. Missing characteristics 1402, 1403 of partial image 1400 have been replaced with additional information or additional information associated with reference 3D model 1410. In this example, complete 3D model 1420 includes a combination of features from both partial image 1400 and the reference 3D model. By comparing leg 1405 of partial image 1400 to leg 1411 of reference 3D model 1410, the system identified that entire leg 1405 and part of leg 1403 are missing from partial image 1400. Although the legs have different appearances, the system can utilize the additional information from reference 3D model 1410 to create an additional leg 1405 with the characteristics found in partial image 1400. As shown in the complete 3D model 1420, the system used additional data or information from the chair back 1412 to create the back 1412 within the complete 3D model 1420.

[0412] In some embodiments, the system may identify at least one of a texture and a color of the partial image and apply the at least one texture and color to the determined portion of the 3D reference model during meshing. As discussed above, the partial image may be represented by image elements (e.g., pixels, mesh points, polygons, voxels, etc.). The disclosed system may determine characteristics of the partial image based on characteristics associated with the image elements of the partial image. For example, the disclosed system may identify characteristics such as color, brightness, texture, material properties, etc. of the image elements of the partial image. Also, as discussed above, the system may copy information associated with image elements of the data store 3D model that are found to be missing in the partial image to the partial image. In some embodiments, the processor may apply the identified characteristics of the partial image to image elements copied from the data store 3D model to the partial image.

[0413] For example, as shown in FIG. 14 , color or texture 1421 from the partial image has been reconstructed in the complete 3D model 1420. In some embodiments, the system may automatically perform the merging of the partial image 1400 with the reference 3D model 1410. However, in some embodiments, the user may select which objects in the image or partial image can be replaced by a reference 3D model also selected by the user. Additionally, the user may be able to select and configure which specific portions of the image or model to replace, blend, or combine with other portions of the image or model. For example, in FIG. 14 , the user may select to replace all legs 1405 with legs 1411 while maintaining the original rectangular backrest in the complete 3D model 1420. In some embodiments, the user may also be able to select one or more properties (e.g., color, brightness, texture, material properties, etc.) to be applied to the image elements (e.g., pixels, mesh points, polygons, voxels, etc.) that are copied from the data store 3D model to the partial image.

[0414] In some embodiments, the processor may be configured to output the simulated complete 3D model for display on a display device. In some embodiments, the complete 3D model, image, or scene may be configured to be displayed via a device such as a headset, computer screen, monitor, projection, or the like. In some embodiments, outputting may include exporting the simulated complete 3D model to a format compatible with the 3D consumable environment. 3D consumable environments consistent with embodiments of the present disclosure may include virtual reality (VR), augmented reality (AR), and mixed reality (MR), 3D still images, video, or broadcast environments modeled after or simulating physical environments. The 3D consumable environment may include at least one of a virtual reality environment and an augmented reality environment, as discussed above. Models of objects, scenes, and image aspects may be encoded in known formats such as 3D vector formats, computer-aided design files, .FLV, .MP4, .AVI, .MPG, .MP3, .MOV, .F4V, .VR, or any other image, video, or model format.

[0415] In some embodiments, the system may output the model by transmitting it to a client device or display the scene in a system interface. Transmitting may include transmitting over any network, such as a TCP / IP network, a broadband connection, a cellular data connection, and / or any other transmission method. The client device and / or system interface may include, without limitation, a mobile device, a headset, a computer, a display, an interface, and / or any other client device. While this disclosure provides examples of methods for transmitting and devices for displaying, it should be noted that aspects of the disclosure in its broadest sense are not limited to the disclosed examples.

[0416] The partial object detection problem involves the problem of understanding a 3D object when the 3D data of the object is incomplete. The representation of the 3D object may be given in any 3D object format / modality, such as a point cloud, a mesh, a voxel representation, etc. There are several reasons why partial object detection is necessary, as discussed below.

[0417] 15 illustrates an exemplary method that may be performed by the system disclosed herein. As will be appreciated from this disclosure, modifications may be made to method 1500, for example, by adding, combining, deleting, and / or rearranging steps of method 1500. The steps of method 1500 may be performed by components of system 1000, including, but not limited to, 3D generator 1020. For example, while method 1500 may be described as steps performed by 3D generator 1020, it should be understood that user system 1010 and / or server 1020 may perform any or all steps of method 1500. As one skilled in the art will appreciate, method 1500 may be performed in conjunction with any other method described herein. The method may include receiving a partial scan and meshing the partially scanned image to generate a complete 3D model of the object.

[0418] In step 1501, the computing device 200 may receive a partial image of an object, where the partial image is at least one of a 2D image or an incomplete 3D image. As discussed herein, the computing device 200 may receive the partial image from a stored location such as a memory, from a data structure or multiple data structures, from user input, from any wired or wireless communication path, or from any other type of data transfer protocol.

[0419] In step 1502, the computing device 200 may search at least one data structure 1030 for additional information corresponding to the subimage. In some embodiments, the data structure 1030 may be local, stored in memory, or may be a remote data structure connected by a series of network or cloud storage devices.

[0420] In step 1503, computing device 200 may determine that data structure 1030 does not contain a corresponding 3D model of the object. In step 1504, computing device 200 may search at least one data structure 1030 for a reference 3D model that is different from the object in the partial image but has similarity to the object in the partial image, where the reference 3D model contains additional data. In step 1505, computing device 200 may compare the partial image with the reference 3D model to determine a portion of the 3D reference model that generally corresponds to the missing characteristic of the partial image.

[0421] At step 1506, the computing device 200 may combine the partial image with additional information, additional data, or a combination of additional information and additional data to construct a simulated complete 3D model of the object. At step 1507, the computing device 200 may output the simulated complete 3D model. In some embodiments, outputting includes saving, transmitting, or displaying the complete 3D model.

[0422] The following provides an example of an implementation of an object completion scheme according to the present disclosure. A user performs a 3D scan of a physical chair. The scan is of only a portion of the front of the chair; the back of the chair is not visible. Thus, the disclosed method may include scanning the object or receiving a partial scan of the object. The disclosed method may also include searching the resulting scan in an existing data structure of 3D objects. Furthermore, the disclosed system may not be able to find any chair in the data structure whose front face exactly matches the current chair. However, the disclosed system may be able to find a chair with similar proportions but different materials and textures. Thus, for example, the disclosed method may include selecting a CAD model of a selected object (e.g., a chair). The disclosed system may also create a new CAD model. For example, the disclosed method may include generating a new CAD model by combining the scan of the object with the CAD model of the selected object. The new model may inherit texture and materials from the scanned chair and 3D shape from the complete CAD model in the data structure. The new model may be, in some sense, an amalgamation (i.e., merger or combination) of the CAD model and the scan. In some embodiments, the disclosed method may identify the chair's materials based on tags in a data structure associated with the chair that are found to resemble a scan of the physical chair and receive material input from the user, for example. The data structure may store information correlating various materials with associated textures. The disclosed method may search the data structure for a texture corresponding to the material received as input from the user and apply that texture to the underlying graphical elements of the CAD model. The disclosed system may present the new model to the user via a 3D viewing device such as a VR headset, a regular screen, etc.Thus, for example, the disclosed method may include presenting a complete combined 3D CAD model of the object to a user, where the user may view the complete model on a 3D viewing device. It is contemplated that the user may view the fused 3D CAD model generated by the disclosed system.

[0423] The disclosed system may fuse two sources to create a 3D object that inherits properties from both. Thus, for example, the disclosed system may combine a partial image with additional information to build a simulated complete 3D model. Examples of such combining two sources may include replacing a scan with a best-fit CAD model; replacing a scan with a best-fit CAD model, where the proportions of the CAD model may be changed to match those found in the partial scan; and replacing a scan with a best-fit CAD model, where textures / materials may be transferred from the scanned data to the CAD model and extrapolated to the entire CAD model.

[0424] The new fused model may be created from the scanned model and an external CAD model, and may therefore contain more information than was contained in the original scan. For example, added information may come from CAD data in our data structure, and this additional information may be additional information corresponding to the subimage.

[0425] In some aspects, it may not be necessary to create a complete CAD chair from partial scans, since a CAD model of the chair parts visible in the scan may be sufficient. This may be useful, for example, for the calculation of more natural lighting on the model. Thus, for example, the additional information may include a 3D model corresponding to the received partial image.

[0426] In some aspects, the disclosed data structure may not include a CAD model, but may include a family of scanned objects, and these scans may be partial. In this case, the disclosed system may find a subset of scans in the data structure that are similar to the partial scan and further combine information from this subset. For example, the disclosed system may combine data from the data structure-scan to complete hidden parts in the current scan to create a fused partial image that is free of "holes" that were present in the current scan. Thus, for example, the additional information may include information obtained from a partial scan of at least one object similar to the object in the partial image.

[0427] In some aspects, the scanned data may be a point cloud, and it may be desirable to create a mesh that best represents this point cloud. This is a standard task associated with scanning 3D objects, and while there are methods for obtaining a mesh directly from a point cloud, these conventional methods may be sensitive to scanning errors. However, a search from the disclosed system, which is more global and therefore less sensitive to scanning errors, may find a 3D CAD model sufficiently similar to our scan, along with a mapping between the scanned data and the CAD model as a 3D manifold. If the system finds a high-quality mesh from the CAD model to the scan's point cloud, the system may transfer the existing high-quality mesh from the CAD model to the scan's point cloud. Thus, for example, in the disclosed systems and methods, combining may include meshing a partial image with a determined portion of the 3D reference model. In some embodiments, the disclosed system may use a hybrid approach, directly meshing the point cloud using standard meshing algorithms such as Poisson reconstruction. For other portions of the mesh, the disclosed system may use a mesh from the CAD model stored in a data structure.

[0428] In some aspects, the disclosed system may provide a CAD model that most accurately corresponds to the scanned sub-image. Thus, for example, the disclosed system may provide additional information that may include a 3D model that corresponds to the received sub-image.

[0429] In some aspects, the disclosed system may detect texture from the scanned data (e.g., for each point in the scan data, the disclosed system may infer from the scan data whether it is metallic, wooden, glass, etc.). The disclosed system may then merge this texture data with a CAD model (or a portion of th...

Claims

1. 1. A computer-implemented visual input reconfiguration system for enabling selective insertion of content into an existing media content frame, comprising: The visual input reconstruction system includes at least one processor, the at least one processor comprising: configured to access a memory storing a plurality of object image identifiers associated with a plurality of objects; configured to transmit at least one object image identifier of the plurality of object image identifiers to one or more client devices; configured to receive, from one or more client devices, one or more bids associated with the at least one object image identifier; configured to determine a winning bid from among the one or more received bids, where the winning bid is associated with a winning client device from among the one or more client devices; configured to receive image data of the winning bidder from the winning bid client device; configured to store image data of the successful bidder in a memory; configured to identify, in the at least one existing media content frame, an object insertion location for an object corresponding to the at least one object image identifier; configured to generate at least one processed media content frame by processing at least one existing media content frame to insert at least a rendition of the winning bidder's image data at the object insertion location; and configured to transmit at least one processed media content frame to one or more user devices; A computer-implemented visual input reconstruction system.

2. The computer-implemented visual input reconstruction system of claim 1 , wherein the at least one object image identifier comprises at least one of a shape, a shape descriptor, a product, or a product descriptor.

3. 2. The computer-implemented visual input reconstruction system of claim 1, wherein the existing media content frames include at least one of a still image, a series of video frames, a series of virtual three-dimensional content frames, or a hologram.

4. The computer-implemented visual input reconstruction system of claim 1 , wherein the at least one processor is further configured to perform image processing on the winning bidder's image data to render the winning bidder's image data compatible with existing media content frame formats.

5. The computer-implemented visual input reconstruction system of claim 1 , wherein the at least one pre-existing media content frame comprises a plurality of frames that constitute a virtual reality field of view, and the insertion renders objects from the winning image data within the plurality of frames.

6. The computer-implemented visual input reconstruction system of claim 1 , wherein transmitting comprises transmitting over a network.

7. 2. The computer-implemented visual input reconstruction system of claim 1, wherein transmitting includes transmitting the processed media content frame to a first user device of the one or more user devices, and the at least one processor is configured to transmit at least one existing media content frame to a second user device in a manner that excludes the image data of the winning bidder.

8. 2. The computer-implemented visual input reconstruction system of claim 1, wherein the image data of the winning bidder is inserted into at least one existing media content frame such that the image data of the winning bidder is overlaid on existing content in the at least one existing media content frame.

9. 2. The computer-implemented visual input reconstruction system of claim 1, wherein the image data of the winning bidder is inserted into at least one existing media content frame such that an object of the image data of the winning bidder replaces existing content in the at least one existing media content frame.

10. 10. The computer-implemented visual input reconstruction system of claim 1, wherein the image data of the winning bidder is inserted into a portion of the object corresponding to the at least one object image identifier.

11. 2. The computer-implemented visual input reconstruction system of claim 1, wherein the processor is further configured to receive instructions from a winning client device, the instructions including a size limit for the object corresponding to the at least one object image identifier, and the inserting of at least a rendition of the winning bidder's image data is based on the instructions.

12. The computer-implemented visual input reconstruction system of claim 1 , wherein the object corresponding to the at least one object image identifier comprises at least one of a wall, a billboard, a picture frame, or a window.

13. 2. The computer-implemented visual input reconstruction system of claim 1, wherein the winning bidder image data displayed in the existing media content frame changes after a predetermined period of time.

14. 2. The computer-implemented visual input reconstruction system of claim 1, wherein the processor is further configured to obtain at least one existing media content frame in real time and to insert a rendition of the winning bidder's image data into the at least one existing media content frame in real time.

15. 1. A computer-implemented method for enabling selective insertion of content into an existing media content frame, comprising: accessing a memory storing a plurality of object image identifiers associated with a plurality of objects; transmitting at least one object image identifier of the plurality of object image identifiers to one or more client devices; receiving one or more bids associated with at least one object image identifier from one or more client devices; determining a winning bid from among the one or more received bids, where the winning bid is associated with a winning client device from among the one or more client devices; receiving image data of the winning bidder from the winning client device; storing image data of the successful bidder in a memory; identifying an object insertion location in at least one existing media content frame for an object corresponding to the at least one object image identifier; generating at least one processed media content frame by processing at least one existing media content frame to insert at least a rendition of the winning bidder's image data at the object insertion location; and transmitting at least one processed media content frame to one or more user devices; The method includes:

16. The method of claim 15 , wherein the at least one object image identifier comprises at least one of a shape, a shape descriptor, a product, or a product descriptor.

17. The method of claim 15 , wherein the pre-existing media content frames include at least one of a still image, a series of video frames, a series of virtual three-dimensional content frames, or a hologram.

18. 16. The method of claim 15, wherein the at least one processor is further configured to perform image processing to render the winning bidder's image data to be compatible with an existing media content frame format.

19. 16. The method of claim 15, wherein the at least one pre-existing media content frame comprises a plurality of frames that constitute a virtual reality field of view, and the insertion renders objects from the winning image data within the plurality of frames.

20. 1. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations that enable selective insertion of content into an existing media content frame, the operations including: accessing a memory storing a plurality of object image identifiers associated with a plurality of objects; transmitting at least one object image identifier of the plurality of object image identifiers to one or more client devices; receiving one or more bids associated with at least one object image identifier from one or more client devices; determining a winning bid from among the one or more received bids, where the winning bid is associated with a winning client device from among the one or more client devices; receiving image data of the winning bidder from the winning bid client device; storing image data of the successful bidder in a memory; identifying an object insertion location within at least one existing media content frame for an object corresponding to the at least one object image identifier; generating at least one processed media content frame by processing at least one existing media content frame to insert at least a rendition of the winning bidder's image data at the object insertion location; and transmitting at least one processed media content frame to one or more user devices; A non-transitory computer readable medium comprising:

Citation Information

Patent Citations

  • Real estate claims in panoramic or 3D mapping environments for advertising purposes

    JP2011527483A

  • Content generation device, content generation method and content generation program

    JP2017191615A

  • Determining the Appearance of Objects in the Virtual World Based on Object Appearance Sponsorship

    JP2018522342A

  • Information processing device, information processing method, program, and storage medium

    WO2016067363A1