Flexible dynamic projection mapping system and method

Abstract

This document relates to systems and methods for dynamically tracking objects and projecting drawn 3D content onto said objects in real time. The methods described herein also include image data capture performed by various image capture devices, wherein said data is segmented into various components to identify one or more projectors for drawing and projecting the 3D content components onto one or more objects.

Description

[0001] This application claims the benefit and priority of U.S. Provisional Patent Application No. 62 / 784,960, filed December 26, 2018, entitled “RESILIENT DYNAMIC PROJECTION MAPPING SYSTEM AND METHODS,” the entire contents of which are expressly incorporated herein by reference for any and all non-limiting purposes. Background Technology

[0002] Certain projection mapping systems are limited by the types of physical objects on which content can be projected, and further, by the system's ability to accurately track physical objects and render content onto those objects in real time. In fact, previous projection systems (including those using marker-based methods to track objects) require more processing power, more setup time for pre-scanning objects (e.g., objects fixed with dot markers), and image capture devices with high fidelity and / or frame rates to capture the marked objects. Performing this resource-intensive object tracking and content rendering limits the scalability of existing projection mapping systems and further restricts the types and sizes of objects tracked and rendered by such systems.

[0003] To efficiently track and render 3D content onto various physical objects in real time under numerous scenarios and conditions with limited required processing resources, dedicated systems and processes are needed to minimize reliance on manually configuring and determining the pose and position of physical objects in 3D space. Therefore, an improved projection mapping system is needed that can efficiently acquire image data associated with objects and use such image data to perform detection and tracking techniques to project the rendered content onto the location of the tracked physical object. Thus, aspects of this disclosure aim to address this challenge by constructing a projection mapping system that does not require point markers to locate the tracked object and also identifies one or more projectors that optimally render and project 3D content onto specific areas of the physical object, thereby optimizing the image quality of the 3D content being rendered and projected onto each component of the physical object.

[0004] Therefore, improved systems and methods are needed to address these and other shortcomings in this field. Summary of the Invention

[0005] The following is a simplified overview to provide a basic understanding of some aspects of this disclosure. This overview is not an exhaustive summary of this disclosure. This overview is not intended to identify key or critically important elements of this disclosure or to define the scope of this disclosure. The following overview presents only some concepts of this disclosure in a simplified form as an introduction to the description below.

[0006] Various aspects of this disclosure relate to improved systems and methods for dynamically tracking physical objects and projecting drawn 3D content onto said physical objects in real time. Certain implementations can reduce the processing load on computing devices used to track physical objects and draw 3D content thereon. In some embodiments, image data captured by various image capture devices can be segmented into various components, allowing the system to identify the optimal (e.g., best) projection device for drawing and projecting each component of the 3D content onto the physical object. Additionally, the projection system described herein performs contour detection of the image data indicating the physical object, thereby enabling the system to track the position / angle of the physical object (in real time) and to filter out "false positives" in physical object detection by continuously tracking the detected contours for a specified time period to confirm their validity.

[0007] Example embodiments may relate to systems, methods, apparatuses, and computer-readable media configured to track, render, and project 3D content onto physical objects. Image capture devices may be used to collect image and depth data associated with the physical object, which the system uses to determine the position and rotation angle of the physical object within a coordinate system / space. Other aspects involve rendering 3D content to be projected onto various regions of the physical object. The system may select specific projection devices to project 3D content onto the physical object based on the position and angle of each projector relative to the various components of the 3D content to be displayed on the physical object.

[0008] These and other aspects of the embodiments are discussed in more detail in this disclosure (including the accompanying drawings). Attached Figure Description

[0009] Figure 1 The illustration depicts an example system, according to an example embodiment, that can be configured to provide a user with the ability to capture and transmit image data;

[0010] Figure 2 The diagram shows that it can be... Figure 1 part of the system or with Figure 1 Example computer devices for system communication;

[0011] Figure 3 A system environment for tracking and capturing images of objects, according to an example embodiment, is shown;

[0012] Figures 4A to 4B An example of image data indicating object and contour detection performance according to an example embodiment is shown;

[0013] Figures 5A to 5C An example of drawing and projecting content onto a physical object is shown, according to an exemplary embodiment.

[0014] Figures 6A to 6C An example of calibrating a projection device using a reference object according to an exemplary embodiment is shown;

[0015] Figure 7 A method for tracking objects according to an example embodiment is shown; and

[0016] Figure 8 A method for drawing and projecting content onto an object is shown according to an example embodiment. Detailed Implementation

[0017] The innovations disclosed herein further relate to systems and methods for dynamically tracking physical objects in coordinate space and for drawing and projecting content onto said objects using position and rotation angle information collected from an image capture device. Using improved processing, tracking, and drawing techniques, the projection mapping system described herein can selectively and dynamically identify the optimal projection device for drawing and projecting content onto physical objects in real time.

[0018] In the following description of various embodiments, reference is made to the accompanying drawings, which form part of this document, and which illustrate various embodiments in which aspects of the present disclosure may be practiced. It should be understood that other embodiments may be utilized, and structural and functional modifications may be made, without departing from the scope and spirit of the present disclosure. Furthermore, headings within this disclosure should not be construed as limiting the aspects of the disclosure, and exemplary embodiments are not limited to the exemplary headings.

[0019] I. Example User System

[0020] A. Descriptive Network

[0021] Various aspects of this disclosure relate to systems and methods that can be used on multiple networks. In this regard, some embodiments can be configured to adapt to dynamic network environments. Other embodiments can be used in different discrete network environments. Figure 1 An example of a user communication system 100 according to an exemplary embodiment is illustrated. The exemplary system 100 may include one or more interconnected networks, such as an illustrative body area network (BAN) 102, a local area network (LAN) 104, and a wide area network (WAN) 106. Figure 1As shown (and throughout this disclosure), one or more networks (e.g., BAN 102, LAN 104, and / or WAN 106) may overlap or otherwise include each other. Those skilled in the art will appreciate that illustrative networks 102 to 106 are logical networks that may each include one or more different communication protocols and / or network architectures but can be configured to have gateways to each other or other networks. For example, each of BAN 102, LAN 104, and / or WAN 106 may be operatively connected to the same physical network architecture, such as cellular network architecture 108 and / or WAN architecture 110. For example, a portable electronic device 112, which can be considered a component of both BAN 102 and LAN 104, may include a network adapter or network interface card (NIC) configured to convert data and control signals into network messages and convert data and control signals from network messages through one or more of architectures 108 and / or 110 according to one or more communication protocols, such as Transmission Control Protocol (TCP), Internet Protocol (IP), and User Datagram Protocol (UDP). These protocols are well known in the art and will not be discussed in more detail herein.

[0022] Network architectures 108 and 110 may include one or more information distribution networks of any type or topology, individually or in combination, such as cable, fiber optic, satellite, telephone, cellular, wireless, etc., and therefore, network architectures 108 and 110 may be configured in different ways, for example, having one or more wired or wireless communication channels (including but not limited to: Near Field Communication (NFC) and / or ANT technology). Therefore, in Figure 1 Any device within the network (such as portable electronic device 112 or any other device described herein) can be considered as including one or more of the different logical networks 102 to 106. With the foregoing idea in mind, example components of illustrative BANs and LANs (which can be connected to WAN 106) will be described.

[0023] LAN 104 may include one or more electronic devices, such as computer device 114. Computer device 114 or any other component of system 100 may include a mobile terminal, such as a telephone, music player, tablet, netbook, or any portable device. In other embodiments, computer device 114 may include a media player or recorder, desktop computer, server, game console, etc. XBOX Playstation and / or Wii game consoles. Those skilled in the art will understand that these are merely example devices for illustrative purposes, and this disclosure is not limited to any console or computing device.

[0024] Those skilled in the art will understand that the design and structure of computer device 114 can vary depending on several factors, such as the intended purpose of computer device 114. Figure 2 An example implementation of computer device 114 is provided below. Figure 2 A block diagram of computing device 200 is shown. Those skilled in the art will understand that... Figure 2 The disclosure herein applies to any apparatus disclosed herein. Apparatus 200 may include one or more processors, such as processors 202-1 and 202-2 (generally referred to herein as "processors" or "processor 202"). Processors 202 may communicate with each other or other components via an interconnect network or bus 204. Processors 202 may include one or more processing cores, such as cores 206-1 and 206-2 (referred herein as "cores 206" or generally referred to as "core 206"), which may be implemented on a single integrated circuit (IC) chip.

[0025] Core 206 may include a shared cache 208 and / or a private cache (e.g., caches 210-1 and 210-2, respectively). One or more caches 208 / 210 may locally cache data stored in system memory, such as memory 212, for faster access by components of processor 202. Memory 212 may communicate with processor 202 via chipset 216. In some embodiments, cache 208 may be part of system memory 212. Memory 212 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), and includes one or more of solid-state memory, optical or magnetic storage devices, and / or any other medium that can be used to store electronic information. In other embodiments, system memory 212 may be omitted.

[0026] System 200 may include one or more I / O devices (e.g., I / O devices 214-1 to 214-3, each generally referred to as I / O device 214). I / O data from one or more I / O devices 214 may be stored at one or more caches 208, 210 and / or system memory 212. Each I / O device 214 may be permanently or temporarily configured to operatively communicate with components of system 100 using any physical or wireless communication protocol.

[0027] Return to Figure 1 The illustration shows four example I / O devices (shown as elements 116 to 122) communicating with computer device 114. Those skilled in the art will appreciate that one or more of devices 116 to 122 can be standalone devices or can be associated with another device besides computer device 114. For example, one or more I / O devices can be associated with or interact with components of BAN 102 and / or WAN 106. I / O devices 116 to 122 can include, but are not limited to, activity data acquisition units, such as sensors. One or more I / O devices can be configured to sense, detect, and / or measure motion parameters of a user, such as user 124. Examples include, but are not limited to: accelerometers, gyroscopes, location determination devices (e.g., GPS), light (including invisible light) sensors, temperature sensors (including ambient temperature and / or body temperature), sleep pattern sensors, heart rate monitors, image capture sensors, humidity sensors, force sensors, compasses, angular rate sensors, and / or combinations thereof, etc.

[0028] In other embodiments, I / O devices 116 to 122 may be used to provide output (e.g., auditory, visual, or tactile cues) and / or receive input, such as user input from user 124. Examples of these illustrative I / O devices are provided below; however, those skilled in the art will understand that such discussions describe only a few of the many alternatives within the scope of this disclosure. Furthermore, any reference to any data acquisition unit, I / O device, or sensor should be interpreted as a disclosure that embodiments may have (individual or in combination) one or more I / O devices, data acquisition units, and / or sensors disclosed herein or known in the art.

[0029] System 100 can be configured to transmit and / or receive data, including image data captured by an image capture device within a portable device 112 (and / or a separate image capture device 118, such as a camera) collected within system 100 or otherwise provided to system 100. As an example, WAN 106 may include server 111. Server 111 may have… Figure 2The system 100 comprises one or more components of the system 200. In one embodiment, server 111 includes at least one processor and memory, such as processor 206 and memory 212. Server 111 may be configured to store computer-executable instructions on a non-transitory computer-readable medium. The instructions may include image data, such as raw or processed data collected within system 100. System 100 may be configured to transfer data, such as images and / or videos, to social networking sites or host such sites. Server 111 may be used to grant access to image data obtained by one or more users. Thus, server 111 may be configured to transmit and / or receive notifications based on image data or other information.

[0030] Returning to LAN 104, computer device 114 is shown in operative communication with display device 116, image capture device 118, and sensor 120, which are discussed sequentially below with reference to example embodiments. In one embodiment, display device 116 may provide audiovisual cues to user 124 to capture an image of a specific object. The audiovisual cues may be provided in response to computer-executable instructions executed on computer device 114 or any other device including BAN 102 and / or WAN. Display device 116 may be a touchscreen device or otherwise configured to receive user input.

[0031] Image data and / or depth data can be obtained from image capture device 118 (and / or portable device 112), which can be used to detect items within the image data (and / or train a system to detect items within the image data). Image capture device 118 may include transceiver devices. For example, image capture device 118 may transmit waveforms into the environment, including in a direction toward user 124, and receive "reflections" or otherwise detect changes in those emitted waveforms. Those skilled in the art will readily appreciate that signals corresponding to a wide variety of different data spectra can be utilized according to various embodiments. In this regard, device 118 may detect waveforms emitted from external sources (e.g., not system 100). For example, device 118 may detect heat emitted from user 124 and / or the surrounding environment. Therefore, image capture device 126 may include one or more thermal imaging devices. In one embodiment, image capture device 126 may include an IR device configured to perform distance phenomenology. In other embodiments, image capture device may include a depth sensor camera.

[0032] BAN 102 may include two or more means (including passive means) configured to receive, transmit, or otherwise facilitate the collection of image data. Exemplary means may include one or more data acquisition units or means known in the art or disclosed herein, including but not limited to I / O means 116 to 122. Two or more components of BAN 102 may communicate directly, but in other embodiments, communication may be implemented via a third means, which may be part of BAN 102, LAN 104, and / or WAN 106. One or more components of LAN 104 or WAN 106 may form part of BAN 102. In some embodiments, whether a means of portable device 112 is part of BAN 102, LAN 104, and / or WAN 106 may depend on the user’s proximity relative to an access point to allow communication with mobile cellular network architecture 108 and / or WAN architecture 110. User activity and / or preferences may also influence whether one or more components are used as part of BAN 102. Example embodiments are provided below.

[0033] User 124 can be associated with any number of devices, such as owning, carrying, wearing, and / or interacting with, including portable devices 112, shoe-mounted devices 126, and wrist-worn devices 128. One or more devices 112, 126, 128 may be specifically designed for image capture purposes. In practice, data from one or more devices can be used to collect and detect image data, and in some cases, motion data can be collected and detected. In some embodiments, one or more devices in BAN 102 (or any other network) may include image capture devices specifically designed to capture a particular type of image data.

[0034] The illustrative portable device 112 can be, for example, a multi-purpose electronic device including a telephone or digital music player, including those available from Apple Inc. in Cupertino, California. or Branded devices, or those available from Microsoft in Redmond, Washington. or Windows device. As is known in the art, a digital media player can be used as an output device, input device, and / or storage device for a computer. Device 112 can be configured as an input device to receive raw or processed data collected from one or more of the following devices: BAN 102, LAN 104, or WAN 106. In one or more embodiments, portable device 112 may include one or more components of computer device 114. For example, portable device 112 may include a display 116, an image capture device 118, and / or one or more data acquisition devices, such as any of the I / O devices 116 to 122 discussed above, and may or may not have additional components to include a mobile terminal.

[0035] In some embodiments, I / O devices may be formed within or otherwise associated with the clothing or accessories of user 124, including watches, armbands, wristbands, necklaces, shirts, shoes, etc. These devices may be configured to capture image data. It should be understood that they may detect movement based on one or more motions during user 124's interaction with computer device 114, and / or may operate independently of computer device 114 (or any other devices disclosed herein). For example, one or more devices in BAN 102 may be configured to function as an all-day activity monitor that measures activity regardless of the user's proximity to computer device 114 or whether or not the user is interacting with computer device 114. In some embodiments, Figure 1 The device 126 shown may include footwear that may include one or more sensors, including but not limited to those disclosed herein and / or known in the art.

[0036] Figure 1 Element 130 illustrates an example sensing location that can be associated with a physical device, such as a sensor, data acquisition unit, or other device. However, in other embodiments, it can be a body part, area, or specific location on a worn product, monitored, for example, via an image capture device (e.g., image capture device 118). In some embodiments, element 130 may include sensors, such that elements 130a and 130b can be sensors integrated into clothing, such as sportswear. Such sensors can be placed at any desired location on the user 124's body. Sensors 130a / b can communicate (e.g., wirelessly) with one or more devices (including other sensors) of BAN 102, LAN 104, and / or WAN 106.

[0037] II. General Description of Dynamic Projection Mapping Systems Based on Examples of This Disclosure

[0038] Generally speaking, as described above, some aspects of the present invention relate to systems and methods for performing contour detection on image data from multiple point clouds, including three-dimensional (3D) point clouds (e.g., data indicating two-dimensional (2D) images, depth information, etc.), and dynamically projecting the 3D content onto the position and rotation angle of a tracked object, including but not limited to consumer products such as footwear, clothing, sports equipment, etc.

[0039] The projection mapping system described herein can use a variety of different sensors and / or image capture devices to track various types of objects, including any desired type of consumer product. For example, the system can utilize one or more image capture devices (e.g., depth sensor cameras) to capture images and depth data of a physical object to determine the spatial coordinates or position of the physical object, and further determine the rotation angle of the physical object detected by the one or more image capture devices. The spatial coordinates and rotation angle can be determined relative to any suitable reference frame, such as the coordinate space of the imaging device or the tracking space in which the physical object to be tracked is located. The system can utilize one or more image capture devices that may be located near and / or around the physical object to capture one or more images, videos, and / or depth information (i.e., image data) of the object from multiple different locations and / or viewpoints. Using the captured image data obtained via the image capture devices, the system can also perform contour detection and tracking of the physical object; for example, the system can use the captured image data to detect one or more boundaries of the physical object.

[0040] This system achieves contour detection by processing image data captured by one or more image capture devices to identify one or more contours of a physical object, and the system also tracks the position and / or rotation angle of one or more contours. For example, the system can track the position and / or rotation angle of one or more contours relative to a specified area of ​​the physical object. Contours can include lines (e.g., straight lines or curves), such as lines representing the boundaries of a surface area of ​​the object. Contours can include shapes, such as the two-dimensional shape of the surface area of ​​the object, which can be a two-dimensional or three-dimensional shape of the surface area of ​​the object. For example, when the tracked physical object is footwear, the system can track the position and angle information within the foot area (e.g., the ground plane), the outsole area, the midsole area, the toe area, and / or the "ankle" area (e.g., a specified number of inches above the ground plane). The ankle area can be located at a specified number of inches above the "foot area," for example, 3, 4, 6, or 7 inches above a fixed foot area. Users or system administrators can specify the positions and / or areas attributed to the foot and ankle areas. As another example, the system can track positional and angular information within the "upper" region of footwear (e.g., the footwear portion including the upper). Additionally, the system can perform edge detection to find one or more boundaries of the tracked physical object in image data acquired from one or more image capture devices. When performing such edge detection, the system identifies data points in the image data acquired from one or more image capture devices where the brightness level of the image changes abruptly or exhibits discontinuity. For example, the system can compare changes (e.g., increments) in brightness and / or image contrast at one or more data points in the image data to a threshold. Additionally or alternatively, the system can compare a measured level of brightness and / or image contrast at one or more data points in the image data to a threshold or threshold range. Thus, the system typically organizes data points where image brightness changes abruptly (e.g., one or more data points in the image data satisfy one or more defined thresholds) into a set of curve segments or "edges."

[0041] This system can draw a 3D model of a tracked physical object using captured image data and contour detection information. The system can store the drawn 3D model of the object in a database (or other suitable storage) for later use. The drawn 3D model can be stored in an internal storage location and / or an external storage location. The system can determine one or more components (e.g., regions of interest) of the drawn 3D model of the object and store this information along with the 3D model in a separate database, server, or other suitable storage. Such components may include contours to be detected, thereby allowing the determination of position and / or angular information associated with the contours of specified components. Therefore, the component and contour information associated with the 3D model can be stored along with the 3D model in a database (e.g., a relational database that associates component and contour information with corresponding 3D models). The position and / or rotation angle information of each contour associated with the 3D model can also be stored in the database. For each of the one or more contours associated with the 3D model, the system can use the database to determine the position and / or rotation angle information of said contour. Additionally or alternatively, the system may use point cloud information associated with the 3D model to determine the position and / or rotation angle information of the contour. Furthermore, the system may utilize a drawn 3D model of the physical object to project the 3D model onto a position in the tracking space containing the physical object via one or more image projection devices (e.g., projectors). For example, in an instance where the tracked physical object is footwear, the system may utilize one or more projectors to project one or more different components of a drawn 3D model of the footwear onto the footwear. In this instance, the system may determine a specific projector (or projectors) to project specific components (or components) of the drawn 3D model based on the position and rotation angles of the components of the physical object and the 3D model, and the angle at which the projector projects the image of the drawn 3D model onto the physical object. For example, the system may determine one or more projectors having the most advantageous (e.g., optimal) projection angle for projecting the image of the 3D model onto the tracked object. In other words, for each component of the drawn 3D model, the system can determine which projector is positioned to optimally face the 3D model component and thus optimally project the 3D model component onto the physical object. As described above, the system can determine one or more projectors to project one or more components of the drawn 3D model based on the positions of the physical object and one or more components of the 3D model. In some cases, the system can utilize the current positions of the physical object and the components of the 3D model to determine one or more objects.

[0042] The system can project various content, such as images, photographs, and videos, onto different parts / segments of a physical object. Additionally, the system can display content on different parts of an object (e.g., footwear), such as: projecting color onto a portion of the footwear (e.g., various upper parts or components, various midsole parts or components, various outsole parts or components, etc.); projecting images or other graphic data onto the footwear; projecting logo data onto the shoe (e.g., the footwear manufacturer's logo, team or group logo, etc.); projecting texture data onto the shoe, etc. Although the above description is in conjunction with the projection of 3D content onto footwear, the system according to this disclosure can also be used to project content onto various other types of objects and consumer products (such as clothing, sports equipment, etc.). This content may not include the detected outline of the footwear for determining the position and / or angle information of the corresponding contour of the footwear.

[0043] In the footwear example described above, the projection mapping system may include multiple imaging devices to capture image data associated with a physical object from multiple different locations and / or multiple different perspectives. For example, the system may capture one or more images of the footwear from multiple different locations surrounding the physical object. The image data indicating the footwear may be stored in various locations, including the memory of the image capture devices, or the memory of one or more computing devices residing within (or outside) the projection mapping system. In some cases, a system administrator (or other user) may use a user interface provided via one or more software applications to view the captured image data for dynamically tracking 3D content and rendering it onto the physical object. Such software applications may execute on a user device, one or more computing devices within the projection mapping system, one or more computing devices outside the projection mapping system, or some combination thereof.

[0044] In some cases, users (e.g., system administrators) may use one or more computing devices within (and / or outside) the projection system to access tracking software applications (e.g., tracking engines) and / or rendering software applications (e.g., rendering engines). These one or more computing devices may establish communication channels within a network and communicate with a server system (including one or more server computers) that provides data processing for tracking physical objects and / or rendering 3D models onto physical objects. Any desired communication links and protocols may be used to provide and control data exchange between the computing devices and the system. System administrators (or other users) may use the computing devices to access the system via networks such as the Internet, Local Area Network (LAN), Wide Area Network (WAN), or as described above. Figure 1Other networks (as described) are connected to the projection mapping system. Users can connect their computing devices to the system via suitable communication channels, such as website portals and applications from various social networking websites linked to the website portals of the manufacturer and / or system administrator.

[0045] Without departing from the scope of this disclosure, various types of computing devices can be used to exchange data with projection mapping systems, such as computing devices capable of establishing networking and / or peer-to-peer connections and providing the necessary display, user interface, and input capabilities, as will be described in more detail below. Some more specific examples of computing devices that can be used in systems and methods according to at least some embodiments of the invention include, but are not limited to: desktop computers, personal computers, laptop computers, handheld computers, cellular phones, any other mobile devices or smartphones, personal digital assistants, computer workstations, televisions, etc.

[0046] A computing device that can be used in the systems and methods according to examples of the present invention may include one or more input devices and a data processing system (e.g., including one or more microprocessors). Examples of input devices that may be included in a computing device may include... Figure 1 and Figure 2 The device described herein includes one or more devices, and is not limited to conventional input devices, such as: a keyboard (hard keyboard or soft keyboard); a mouse, trackball, scroll ball, touchpad, or other pointing device; a stylus or other pen-type input device (e.g., for tablet PC-type computing devices); a disk drive; a USB port; a network connection; a joystick-type controller; a telephone connection; an Ethernet connection; voice recognition functionality; and so on. Furthermore, the computing device may have a "touchscreen" function, allowing a user to input data into the computing device by physically touching the screen of the display with their finger or a selection device such as a stylus. Additionally, any desired type of display device may be provided for use in conjunction with the computing device according to the system and method of the present invention, including display devices integrated with the computing device itself or display devices separate from but communicating with the computing device, such as projector displays, stand-alone monitor displays, etc.

[0047] The software used to generate the user interface described above may reside on a computing device or server system or on a computer-readable medium that can be used with a computing device or server system. The user interface will be provided and controlled by the user's computing device and / or server system, and data input through the user interface for generation, maintenance, and reception will be generated and provided via a computer-readable medium that is included as part of or associated with the computing device and / or server system. Examples of such computer-readable media include, but are not limited to, computer-readable storage devices that may be internal to the computer (e.g., hard disk drive) or separate from the computer (e.g., disk, solid-state or flash memory devices, data available via a network connection, etc.), including any type of computer-readable medium conventionally known and used in the field of computing. Additionally or alternatively, if desired, the software or at least some portions thereof may reside on more than one computing device or server system, which may be wholly or partially located within or outside the projection mapping system. The server system may be operated and maintained by the same organization or individuals that operate and maintain the computing device and / or network, or the server system may be operated, controlled, and maintained by a party separate from any or all of these entities. As some more specific examples, the server system may be operated and maintained by one or more entities that classify their products using the classification systems and methods described below (e.g., retailers, manufacturers, or suppliers selected by retailers, etc.) (and the user interface software may also be operated and maintained).

[0048] Image data can be captured from one or more image capture devices, such as cameras, camcorders, still image cameras, depth sensor cameras, and combinations thereof located on mobile terminal devices, and / or any device configured to detect wavelengths of energy including light, magnetic fields, and / or thermal energy. Image capture devices may include one or more of a camera (e.g., a camcorder, a depth sensor camera), a projector, and one or more processors (i.e., CPP units). As used herein, “image data” may include raw and / or compressed data, or data stored in a physical, tangible form or as electronic information on a computer-readable medium. Furthermore, multiple images may form part of video and / or depth sensor information. Therefore, references to image data, images, and / or pictures include video, depth sensor information, etc.

[0049] When a user is in a retail environment, image data can be captured by one or more image capture devices (e.g., a CPP unit). For example, a computer-readable medium may include computer-executable instructions that, when executed, cause the acquisition of multiple images / videos of footwear or other products worn by customers in a retail store. For example, portable electronic device 112 may include an application that allows user 124 (or other users, such as a system administrator) to use one or more image capture devices (part of the portable electronic device and / or external image capture devices) to capture image data associated with one or more physical objects.

[0050] When a user (e.g., a system administrator) activates the image capture function using a software application (e.g., a tracking and rendering engine), which may be a hard button or a soft button, one or more computing devices (e.g., computing devices 114 / 314) or some combination thereof, internal or external to the system, can process the captured image data to track one or more physical objects and / or render 3D models of objects, and further project the 3D models onto the physical objects. Additionally or alternatively, one or more steps of this data processing can be performed by one or more processors in one or more image capture devices (e.g., CPP units). Alternatively, the user can select the captured image data via a software application for performing tracking of physical objects and / or rendering / projection of the 3D model. Image data can be captured / determined, in whole or in part, based on image data obtained by the image capture device (e.g., depth sensor data).

[0051] As described above, the system can use one or more image capture devices, such as a CPP unit, to capture images of one or more physical objects within the tracking space. For example, such as Figure 3 As shown, the projection mapping system 300 may include a plurality of CPP units (e.g., CPP units 302, 304, 30, 308) for capturing images of one or more physical objects (e.g., footwear 320) within a tracking space (e.g., tracking space 321) surrounded by the CPP units. Figure 3 As shown, one or more CPP units can be operatively connected to and / or communicate with other CPP units. Various communication methods can be implemented within a system that enables a CPP unit to communicate with one or more other CPP units. For example, a CPP unit can be configured to communicate via one or more wired or wireless communication channels (including, but not limited to: Near Field Communication (NFC) and / or ANT technology communication. Additionally, although... Figure 3Four CPP units are illustrated, but system 300 may include fewer or additional CPP units for capturing image data, such as video or depth sensor data indicating footwear 320. The number of CPP units used to capture images of one or more physical objects (e.g., footwear 320) can be determined by the system administrator. One or more CPP units may use wired or wireless connections to transmit information (e.g., image data) to one or more computing devices (e.g., computing device 314) within the projection system, or to other CPP units. For example, as... Figure 3 As shown in element 318, CPP unit 304 may have a direct communication channel with computing device 314, for example, via a wired or wireless connection to computing device 314. Although in Figure 3 Not shown, but one or more other CPP units may be operatively communicated with computing device 314 (or other computing devices within the projection system) via a wired connection. Additionally or alternatively, one or more CPP units may be operatively communicated with computing device 314 (or other computing devices within the projection system) via a wireless connection (e.g., via network 301). Figure 3 (Other computing devices not shown in the diagram) communicate. Network 301 may include and / or have communication with the above-mentioned computing devices. Figure 1 The same functionality as one or more illustrative networks described herein (e.g., illustrative networks 102 to 106).

[0052] As described above, the projection mapping system can utilize the data processing capabilities of one or more CPP units to process image data captured by the CPP units. By performing some preprocessing functions locally within one or more CPP units, the system can provide increased scalability for processing large amounts of data, rather than performing such processing functions at locations outside the CPP units or other devices outside the projection mapping system. Furthermore, performing such preprocessing functions locally allows the system to reduce network load and latency caused by the transmission and external processing of image data captured by the CPP units. Additionally, by performing some preprocessing functions locally within the CPP units, performing system-driven data filtering (e.g., filtering out "false positives" associated with physical object detection by tracking detected contours over a specified time period to confirm their validity) is closer to the sensing and image capture performed by the system and CPP units. As discussed in more detail below, the system can, based on data obtained from multiple depth sensor cameras (e.g., CPP units), use tracked physical objects / models as a reference to calibrate the individual coordinate spaces associated with and captured by the respective CPP units into a consistent space over one or more time periods. To perform this calibration, the system can output a matching virtual 3D model of the tracked physical object for display on a display device via the tracking engine, to help users (e.g., system administrators) manually and / or visually align one or more point clouds (e.g., multiple data points / sets defined by a coordinate space) generated from image data obtained from one or more CPP units. Point clouds can be defined by various coordinate systems / spaces, such as a 3D Cartesian coordinate system. As described above, the system can generate one or more point clouds using data obtained from one or more CPP units and image / depth data collected by the CPP units. For example, after capturing images of footwear 320 at varying positions and angles using one or more CPP units, the system (e.g., CPP units and / or computing device 314) can generate and / or store several data points returned from (and covering) the surface of footwear 320.

[0053] III. General Description of Tracking and Drawing Performed by a Dynamic Projection Mapping System Based on Examples of This Disclosure

[0054] As described above, a projection mapping system can use one or more image capture devices (e.g., CPP units) to acquire image data (e.g., depth sensor data) corresponding to one or more tracked physical objects (e.g., footwear). One or more CPP units can be configured by a user (e.g., a system administrator) to acquire image data at a specified frame rate. For example, one or more CPP units can be configured to acquire image data at 30 frames per second (“fps”). CPP units can be configured by a user (e.g., a system administrator) to capture image data at other frame rates, such as 20fps, 25fps, 30fps, or 35fps. Figure 3 As shown, multiple image capture devices can acquire image data related to the tracked object (e.g., footwear 320). The system can use each of one or more image capture devices to capture image data related to the object (e.g., footwear 320) from various different positions, heights, angles, etc. The image data acquired by the CPP unit is transmitted, for example via network 301, to one or more computing devices (e.g., computing device 314) for 3D content tracking and / or rendering onto one or more objects (e.g., footwear 320).

[0055] As described above, the captured image data can be processed, in whole or in part, by one or more CPP units and / or one or more computing devices (e.g., computing device 314) to generate one or more point clouds of the tracked physical object. For the avoidance of doubt, a point cloud comprises multiple points defined in a spatial reference frame and corresponding to positions on the object's surface. As described above, the software application executed herein on one or more computing devices is referred to as a "tracking engine," which is used by the system to track the physical object and, as discussed in more detail below, to determine the effective position and angular information of the tracked object. As an example, Figure 4A The diagram illustrates the use of... Figure 3 The point cloud generated from image data captured by one or more CPP units is plotted 401. Figure 4A As shown, the system can use combined image data from one or more CPP units to generate a single point cloud of the tracked object (e.g., element 420) in a coordinate space (e.g., X / Y coordinate space) using a Cartesian coordinate system for the tracking space. In some cases, the system can generate one or more point clouds of the tracked object. The system can continuously capture image data using CPP units within a set time period (e.g., 10 seconds, 15 seconds, 30 seconds, etc.) to generate multiple point clouds associated with the tracked object. The user (e.g., a system administrator) can establish the time period for collecting image data indicating the physical object. The generated point cloud of the tracked object can be displayed in an exemplary coordinate space via a user interface, such as... Figure 4AComponent 401 is shown in the figure.

[0056] A projection system can combine one or more point clouds from multiple generated points, and the combined point cloud is drawn by the system from a fixed viewpoint to create an image of the tracking space. For example, the system can use a specified viewpoint to draw the combined point cloud to create a 2D image of the tracking space and / or physical objects. Users or system administrators can specify various viewpoints, including top-down, bottom-up, left-right, etc. In an example of a 2D top-down image, the system uses predetermined vision techniques (e.g., open libraries) to process the image data to identify and locate contours (e.g., pixel groups) within the image data, and further determine the X / Y coordinate positions of one or more identified contours and their rotation angles about a vertical axis (e.g., the Z-axis). For example, the system can... Figure 4A The one or more point clouds generated by the tracked object 420 shown perform contour detection, and thus the position and / or rotation angle of one or more contours detected by the system for the tracked physical object can be determined, such as... Figure 4B As shown. Alternatively, instead of tracking one or more objects by rotation about a vertical axis, the system can be configured to track one or more physical objects moving in three dimensions, for example, as a user moves or rotates the physical objects within the tracking space. The detected contours of one or more generated point clouds, along with the corresponding position and rotation angle information, can be displayed by the system via a user interface, such as... Figure 4B Component 411 is shown in the figure.

[0057] This system can track the position and / or rotation angle of one or more contours within a specified time period. For example, the system can track the position and / or angle of a contour for a tenth of a second, a quarter of a second, a half of a second, or other suitable time period. The system administrator or user can establish the time period for tracking one or more contours. The system can be configured to track the position and / or angle of contours within a time period, allowing the system to determine the validity of detected contours. For example, the system can track the position and / or angle of a contour within a time period, leading the system to consider the contour valid because it already exists in the captured image data for a predetermined time period (e.g., a quarter of a second). The time period used by the system to establish contour validity can be set by the user (e.g., the system administrator). By tracking contours (or multiple contours) within a specified time period to determine their validity, the system can filter out "false positives" that may be caused by 3D noise, which could otherwise be interpreted as valid objects (or their contours) in one or more image frames before disappearing. It can also exclude contours detected when an object moves within the tracking space, and this can improve the accuracy of the tracking system by avoiding contour detection when the object moves at a speed that prevents the image-based tracking system from accurately capturing image data of the object corresponding to a single position and orientation within the respective tracking space.

[0058] The system can transmit valid position and angle information of one or more tracked objects to the rendering engine for processing. Valid position and angle information of the tracked objects can be continuously transmitted to the rendering engine. For example, the system can transmit valid X / Y position data and rotation angle data for one or more tracked objects to the rendering engine. In some instances, as described above, the rendering engine can be executed wholly or partially by one or more computing devices within the projection mapping system and / or one or more computing devices external to the projection mapping system. In instances where the rendering engine executes on one or more remote computing devices, the system can transmit the determined position and angle information to the externally executed rendering engine for further data processing.

[0059] The system can utilize virtual 3D models (e.g., virtual designs) stored in one or more computing devices (e.g., computing device 314) or other suitable memory to project the rendering of the 3D model at the position and / or angle of one or more tracked physical objects determined by the tracking engine. The projection mapping system (or one or more computing devices external to the projection mapping system) can generate a virtual 3D model of an object by performing a 3D scan of one or more dimensions of the physical object. For example, the system can create a virtual 3D model of a footwear / shoe model by performing a 3D scan of one or more dimensions of a specific footwear / shoe model.

[0060] The system can use specially programmed computing devices (e.g., rotary scanners) to capture multiple images (e.g., 3D scans) of a specific product using one or more image capture devices (e.g., CPP units 302, 304, 306, and 308). For example, a rotary scanner can be used to collect discrete images (e.g., 120 images) of a specified number of objects, which may correspond to five (five) different image capture devices, each capturing images of 24 (twenty-four) objects. Each image capture device used to generate the 3D scan can be mounted at a specified angle as one or more objects rotate on a disk, allowing the image capture device to capture the object at multiple different angles. Additionally or alternatively, one or more objects can remain stationary while the one or more image capture devices rotate around the object to capture a specified number of images. The captured images and the 3D scans created using the rotary scanner can be used to develop and draw a 3D model of the tracked object. In other embodiments of this disclosure, one or more remote computing devices can create virtual 3D models of the objects and can also send data indicative of the 3D model to a projection system for storage and / or for subsequent use by the system to draw the received 3D model onto one or more physical objects.

[0061] After obtaining or creating a virtual 3D model of one or more tracked physical objects, the system can simplify the 3D model by splitting and / or segmenting the model into one or more regions of interest, or in other words, by breaking the model down into one or more object parts. For example, in an instance where the tracked object is footwear, one or more regions of interest may include various parts of the footwear, such as the heel, midsole, upper, etc. The system can split and / or segment the 3D model into various components using standard manual 3D modeling techniques and / or application software (e.g., "3D coating"). In some cases, the system can retrieve data indicating information and characteristics of the first object from a server, database, or other suitable storage, such as a pre-stored model of the footwear. The retrieved data may also include information indicating one or more predefined regions of interest associated with the first object. After retrieving the data, the system can determine one or more regions of interest in the virtual 3D model by comparing one or more parts of the 3D model with corresponding parts of the first object. For example, in the case where the first object is footwear, the system can compare a portion of the first object (e.g., the heel, midsole, upper, etc.) with the corresponding portion of the 3D model. During the comparison, the system can assign regions of interest to the 3D model based on the location of one or more predetermined regions of interest associated with the portion of the first object being compared with the 3D model.

[0062] In some aspects of this disclosure, the system may retrieve data from a server, database, or other suitable storage indicating contour information of multiple pre-stored (e.g., pre-built) models (e.g., the first object discussed in paragraph

[0057] above). This contour information may identify one or more contours associated with (or previously detected) each pre-stored model. The system may compare detected contours associated with the 3D model (e.g., as discussed above in paragraphs

[0036] to

[0039] and

[0052] to

[0053] ) with one or more associated contour information from the multiple pre-stored models to determine a particular pre-stored model for determining the location and orientation information of the object being tracked, as will be described in further detail below. When determining the location, position, and orientation of the tracked object, the system may compare the contour information of the pre-stored model with the detected contours of the 3D model. The multiple pre-stored models may include one or more model-scaled models corresponding to different sizes of a representative object (e.g., footwear). Therefore, by comparing the contour information of models of various sizes that are pre-stored, the system is configured to use the contour to determine the position and / or orientation of an object (e.g., footwear), even if the detected contour of the tracked object's precise dimensions is not included in the predetermined model geometry.

[0063] The system's scalability is improved by segmenting the 3D model into one or more regions of interest, allowing the system to render a wider range of object sizes (e.g., shoe sizes). Additionally, scalability is further improved by selectively scaling the geometry that best matches the object size, using a subset of the 3D model to render an even wider range of object sizes. Assuming that the geometry of objects such as footwear, clothing, and sports equipment typically changes with size in a fairly predictable manner, the system is able to selectively scale the geometry that best matches the object size.

[0064] The resulting 3D model is drawn in real time by the system at the tracked position and / or angle received from the tracking engine. For example... Figure 5A As illustrated in the example, the system can use a pre-built 3D model of the tracked object (e.g., component 522), and can also position the 3D model at an X / Y coordinate position and rotation angle specified by data obtained via the tracking engine. The pre-built models used by the system can be retrieved from a database, server, or other suitable storage that holds multiple pre-built models (e.g., the pre-stored models discussed above in paragraph

[0058] ). The drawn 3D model (aligned to match the position and rotation angle of the tracked physical object) can be displayed by the system via a user interface, such as... Figure 5AComponent 502 is shown in the diagram. The system can utilize virtual cameras to render the resulting / final scene of the tracked object from the perspective of each projector (e.g., a CPP unit), and the system uses these projectors to project the rendered / derived 3D model onto the tracked physical object. Figure 5B and Figure 5C As shown, the system can use one or more projectors to output a drawn 3D model onto a physical object, aligning the drawn 3D model with the tracked object by position and angle. Specifically, as... Figure 5B As shown, one or more CPP units (e.g., CPP units 302, 304, 306, 308) project a drawn 3D model (e.g., model 522) onto a tracked physical object, in this instance, a pair of white tennis shoes. Figure 5B The 3D model depicted by element 520, which is being projected onto the white tennis shoe, is misaligned, causing part of the projected 3D content to miss the physical object and spill onto other surfaces (e.g., the floor).

[0065] However, as mentioned above, assuming the system's tracking and rendering engines operate in real time, and further assuming the tracking engine tracks the position and rotation angle of the physical object over a predetermined time period, the system can determine (in real time) the most current position and angle information of the tracked physical object. Therefore, the real-time (or updated) angle and position information obtained by the tracking engine and sent to the rendering engine allows the system to appropriately project the rendered 3D model onto the tracked physical object via one or more projectors. For example, as... Figure 5C As illustrated by element 521, the system can dynamically and in real-time use updated position / angle information to project the drawn 3D model / content onto the tennis shoe, ensuring a closer alignment between the drawn 3D model / content and the tracked physical object, thereby minimizing any spillover effects (if any) and improving the quality of the projected image of the 3D model / content displayed on the physical object. Thus, the system can dynamically update the position and / or angle information of the object to reflect the extent to which the tracked physical object moves or is moved within the tracking space, with each projector projecting the drawn 3D model / content onto the object, ensuring alignment between the projected content and the physical object.

[0066] The system may not attempt to infer the size and / or other characteristics (e.g., shoe size, shoe model, etc.) of the physical object being tracked, because this information can be obtained from input devices (such as previously obtained information) by the user or system administrator. Figure 1 and 2The system may manually specify the characteristics of the tracked object (optionally including shoe size, shoe model, etc.) based entirely or in part on image data and other information collected / obtained by the image capture device and / or tracking engine.

[0067] In other embodiments of this disclosure, instead of drawing a virtual 3D model on the object by attempting to match the known geometry of the physical object as described above, the system can draw a projection of a large amorphous object that can completely and / or substantially cover the tracked physical object (e.g., cover 75% to 99% of the object). The system can then use 3D image data (including depth data obtained by one or more image capture devices) to dynamically draw a mask image to create a “green screen” effect, thereby removing any overflow from the large amorphous object drawn on the tracked object. As described above, assuming that this process of drawing content may not require the use of a pre-built 3D model that matches the physical object being tracked (e.g., one or more of the virtual 3D models discussed above with respect to paragraph

[55] ), the system can utilize fewer data processing resources to draw 3D content onto the tracked object via the use of a mask image, thereby improving the scalability of the system, making the system more flexible in terms of the types of physical objects that can be tracked / drawn, and improving the efficiency of the system in tracking / drawing these objects. Similarly, assuming the system may not require a pre-built 3D model to project 3D content onto the tracked object, it can dynamically adapt the way 3D content is projected onto different objects with various shapes, sizes, and configurations. Furthermore, by drawing the projected 3D content without relying on a pre-built 3D model, the system has greater flexibility to project 3D content onto a variety of object surfaces, including but not limited to clothing, sports equipment, and / or a customer's body. Therefore, this flexibility allows the system to operate more quickly and efficiently without the limitation of requiring a pre-built 3D model, and by obtaining a specific 3D model before the system projects 3D content onto an object, and then having to or attempting to map that model to the physical object, no additional time or computational resources are wasted.

[0068] For model-based or non-model-based methods of rendering 3D content onto one or more physical objects as described above, the system can use a "projector prioritization" technique to compare the average surface normal of the 3D geometry of one or more tracked objects with the angle of each projector (e.g., a CPP unit). In some cases, the system can compare the average surface normal of the 3D geometry of the object based on each component with the angle of each projector. This comparison can be performed for each component of the tracked object (e.g., region of interest). Thus, the system can display 3D content using only one or more projectors that have the most advantageous or optimal view of the 3D content from among multiple available projectors. In some cases, the 3D components of the content / model can be projected by the system using only one or more projectors that most directly face each region of the tracked object corresponding to the component being displayed / projected. The system can repeat this process for each frame (i.e., the frame of projection over time) such that the projector prioritization is updated as the position and angle of the tracked object change, thereby ensuring the best possible image quality and further ensuring that the rendered content is properly aligned with one or more tracked objects. To avoid confusion, by using a projector that most directly faces the corresponding area of ​​the tracked object (e.g., a surface orthogonally facing the corresponding area of ​​the tracked object), the effect of any lateral misalignment of the projected image (i.e., misalignment in a plane perpendicular to the axis between the tracked object and the corresponding projector) is reduced. Conversely, when the surface of the corresponding area becomes more tilted relative to the respective axes of the projector, for a given lateral misalignment in a plane perpendicular to that axis, the misalignment on the object will increase.

[0069] The system can store image data acquired by one or more image capture devices used for monitoring and / or tracking objects. Additionally or alternatively, the system can also use one or more of the system's projector and / or image capture devices (e.g., cameras) to capture images / videos of the obtained / drawn 3D content projected onto a physical object. The system can transmit stored and / or captured image data instructing the drawing of 3D content onto a physical object upon request from one or more users. In some instances, a user can associate the requested image information with one or more social network accounts or platforms (e.g., a NIKE+ account). By associating this image information instructing the drawing of 3D content onto a physical object with their social network account, the user can share their experience and the requested image information with other users on the social network platform.

[0070] For example, after a user has completed projecting the drawn 3D content onto a desired object (e.g., footwear), the user can be presented with the option to store the 3D drawing and projection image via a user interface. Additionally or alternatively, the user can be given the option to purchase consumer products associated with the tracked physical object and the 3D content / design drawn and projected onto it. For example, a user can select various features from the user interface and / or design website to view and / or purchase consumer products related to the tracked physical object and the 3D content (and / or design) that has been projected onto the object. For example, the option to purchase footwear resembling the tracked physical object and having a design, model type, or color scheme similar to the 3D model that has been drawn and projected onto the physical object can be presented to the user.

[0071] Users can select a "Save and Share" option from the user interface and / or design website, which can launch a new webpage / website or interface screen presenting users with various options, including one or more choices related to electronic storage and sharing a projected image of the captured 3D content onto the tracked object. For example, the "Save and Share" website / screen can simply allow the desired image / content to be electronically stored or "saved" into computer memory without additional screens or information. This image content can be stored on computer memory residing on a local computing device operated by the user, a remote computing system operated by the entity maintaining the design website, or any other suitable computing storage device. Image information of the 3D content projected onto the tracked object can be stored in the user's social network account and can be manipulated or viewed by other users in any suitable manner. Users can provide feedback on the stored image content and / or rank, rate, or otherwise comment on the content, which includes the drawn 3D content and the underlying 3D model used to draw the 3D content onto the physical object.

[0072] As mentioned above, user-stored image content can be published to social networking sites. Users with accounts or the ability to utilize social networking sites are able to "post" images of drawn 3D content on these sites. Many social networking sites have user profile pages that allow users to post personal items for other users to view. Users may wish to have their image content displayed on their user profile page provided by the social networking site. Such social networking sites can be maintained by the same entity that maintains the website design and / or user interface, such as the manufacturer of consumer products associated with the tracked individual, or they can be maintained by a third party, such as... Users may also want to publish their stored image / video content on personal websites or other web applications.

[0073] IV. General Description of Projector Alignment and Calibration Performed by a Dynamic Projection Mapping System Based on Examples of This Disclosure

[0074] As described above, the system can utilize one or more projection devices (e.g., projectors, CPP units) to project 3D content onto one or more tracked physical objects. In some instances, the projection mapping system can calibrate these one or more projectors and the drawing / projection coordinate space (as described above) against a physical reference object (e.g., a "calibration object") with a geometry known to the system. Figure 4A and Figure 4B (As discussed). This calibration is achieved by the system using a multi-data-point technique, which matches the projector's pose, position, and / or location with the projector's pose, position, and / or location of a known 3D model. For example, as Figure 6A As shown, the known calibration object is placed within a tracking space 621, which is surrounded and / or enclosed by multiple CPP units (e.g., CPP units 302, 304, 306, and 308). One or more CPP units can be calibrated by the system relative to the known geometry of the calibration object to establish a plotting / projecting coordinate space. This is achieved by the system visually aligning depth sensor data obtained by one or more depth sensors and / or CPP units with a 3D model of the calibration object.

[0075] For example, such as Figure 6B As shown, the system can provide a user interface (e.g., element 602) that allows a user (e.g., a system administrator) to align depth sensor data with a 3D model (e.g., element 620) of the calibration object. Figure 6B As shown, users can interact with various display interface elements (e.g., element 604) to access different views of depth sensor data (e.g., top, side, front) and a 3D model of the calibration object. Additionally, users (e.g., administrators) can selectively identify which of various types of depth data (e.g., front, right, left, etc.) should be displayed via the interface (e.g., element 606) and whether the calibration object should be displayed along with the depth data. Furthermore, the system allows users to align depth sensor data from multiple sources (e.g., multiple depth sensors, CPP units) with the 3D model of the calibration object via the user interface. For example, as... Figure 6CAs shown, users can interact with various display interface elements (e.g., element 604) to access different views (e.g., top, side, front) of depth sensor data obtained from multiple sources, and further align this depth sensor data with a 3D model of the calibration object, as shown in element 622. In some cases, the system can automatically align image data (e.g., depth sensor data) with a 3D model of the calibration object, instead of the user performing the alignment and calibration described above, thereby calibrating one or more CPP cells relative to the known geometry of the calibration object and establishing a drawing / projection coordinate space, which is used to track the physical object and project content onto it.

[0076] As referenced above Figure 6B and Figure 6C The system, for one or more projectors used to project rendered 3D content onto a tracked object, can calibrate these projectors relative to the known geometry of the calibrated object to establish a projection / rendering coordinate space. Additionally, the system can use data points obtained from one or more CPP units measuring the depth and / or position of the tracked object to visually align the image / depth data relative to the known geometry of the calibrated object, thereby establishing a tracking coordinate space. Therefore, after one or more projectors and / or image capture devices have been calibrated by the system, the real-world coordinate space will match the coordinate space of the aforementioned 3D sensors and 3D rendering pipeline. Thus, this calibration process produces a consistent coordinate space during the tracking and rendering processes performed by the system, enabling the rendered 3D content to be matched to one or more tracked physical objects and correctly projected onto them.

[0077] V. Description of processes and techniques for tracking and drawing content via a dynamic projection mapping system, according to examples of this disclosure.

[0078] Figure 7 The illustration depicts an example method for sensing and tracking one or more objects using dynamic projection mapping, which can be executed by one or more computing devices, according to an embodiment of the present disclosure. When describing the steps of the method, the term "system" may be used when referring to each component used in each step of the method. Such usage is merely for the convenience of discussing the method and should not be construed as limiting the precise embodiments in which the method can be performed.

[0079] In step 702, the system acquires image / depth data associated with the tracked object. As described above, the system may include one or more image capture devices (e.g., CPP units) that acquire image data including depth information of the tracked physical object. The one or more image capture devices may capture this image data on a per-frame basis at a specific frame rate. In some instances, the one or more image capture devices may be configured to capture image / depth data at a frame rate of approximately 30 frames per second, although, as described above, the image capture devices may be configured to capture image data at other frame rates. In some instances, the one or more image capture devices may be configured to capture image / depth data at a frame rate specified by the user or system administrator.

[0080] In step 704, the system obtains the image / depth data acquired in step 702. For example, the system may request the image / depth data from one or more image capture devices that acquired the image / depth data in step 702. In other instances, one or more image capture devices may transmit the acquired image / depth data to one or more centralized computing devices of the projection system, which are executing a rendering engine to further process the image / depth data.

[0081] In step 706, the system processes the image / depth data obtained in step 704 to draw one or more point clouds of the tracked object for further analysis. (See above regarding...) Figure 4A and Figure 4B The point cloud discussed can be defined by various coordinate systems / spaces (e.g., a 3D Cartesian coordinate system) and can be generated using image / depth data obtained from one or more CPP cells. For example, referencing... Figure 3 After capturing image / depth information related to footwear 320 at varying locations and angles using one or more CPP units, the system can generate and / or store several data points returned from (and covering) the surface of footwear 320. For example... Figure 4A As illustrated, the system can continuously capture image / depth data over a specified time period (e.g., 15 seconds) to generate one or more point clouds associated with the tracked object (e.g., footwear 320).

[0082] In step 708, the system performs contour detection on one or more point clouds drawn in step 706 to determine whether the tracked object corresponds to a specific object type. For example, the system may perform contour detection on one or more drawn point clouds to determine whether the tracked physical object corresponds to a footwear-like object. As mentioned above, the system is not limited to tracking only footwear or footwear-like objects, and can also be configured to track other objects, such as consumer products, including clothing and / or sports equipment. In step 708, the system may use the image / depth data obtained in step 704, such as... Figure 4A As shown, contour detection is performed on the drawn point cloud by detecting one or more boundaries of the tracked object, as represented by the drawn point cloud and the collected image data. (See reference above.) Figure 4A and Figure 4B The system discussed hereby performs contour detection by processing image / depth data captured (or obtained from) one or more image capture devices in steps 702 and 704 to identify one or more contours of the tracked object.

[0083] In step 710, the system tracks the coordinate position and / or rotation angle of the image / depth data, as well as one or more contours detected in step 708. The system can track position and angle information associated with the detected contours, which are related to a specified plane / region, such as the ground plane and / or within a specified proximity to the base on which the tracked object is located. The system can determine the X / Y position and rotation angle information of one or more detected contours based on the surface area of ​​the image / depth data found in a specific region of coordinate space. For example, when the tracked object is footwear, the system can analyze the surface area of ​​depth data within specified “foot region” (e.g., at the ground plane) and “ankle region” (e.g., above the ground plane).

[0084] In step 712, the system continues to track the position and / or rotation angle information of the image / depth data to identify specific objects within a specified time period and determine whether the expected object is continuously within the tracking space. For example, in the case of a tracked physical object being footwear, the system appears to identify shoe-shaped objects within the image / depth data over time to determine whether the footwear is continuously within the tracking space. As described above, the system can track the position and / or angle of the depth / image data, including one or more detected contours within a specified time period. The system administrator or user can establish / specify the time period during which the system attempts to track objects using the acquired image / depth data. In step 712, the system can continue to track the position and / or angle of the image / depth data within a time period such that the system considers the detected contour valid after a predetermined time period (e.g., a quarter second) has passed within the tracking space. As described above, ensuring the validity of detected contours allows the system to operate more effectively by eliminating false positives.

[0085] In some aspects of this disclosure, the system may use one or more point clouds drawn during step 706 and / or information obtained during steps 710 and 712, simultaneously with position and / or rotation angle information of the tracking image / depth data, to retrieve a predetermined 3D model corresponding to the object being tracked. The 3D model may be retrieved from a server, a database, or some other suitable storage containing multiple 3D models corresponding to various objects. For example, the database may include multiple different pre-stored footwear models, and each stored footwear model may include data describing various characteristics of each model, such as associated point clouds, contour information, regions of interest, etc. Various types of data describing model characteristics may be stored in the database. This database or other suitable storage may be the same as or similar to the database discussed above, for example, in paragraphs

[0038] and

[0057] . The system may use the point clouds and tracking information obtained during steps 706, 710, and / or 712 to compare the object being tracked with multiple stored 3D models and their associated feature data. Based on this comparison, the system may identify one or more pre-stored models having feature data similar to the object being tracked. For example, if the point cloud and tracking information of an object are within a threshold standard deviation of the feature data, the system can determine that the point cloud and tracking information of the object are similar to the feature data of a specific pre-stored model. In some cases, the system can compare the point cloud and tracking information of an object with the feature data of a pre-stored model to determine the level of similarity between the tracked object and the pre-stored model. Similarity can be based on how similar the object's point cloud and tracking information is to the relevant feature data of the pre-stored model. The system can either give higher weights to certain metrics indicating the point cloud and tracking information than to others, or it can give each metric equal weights. For example, the point cloud and tracking information can provide information about three different metrics, and the relevant feature data of the pre-stored model can be compared to each of the three metrics. The system can determine the ratio of one or more metrics of the tracked object to one or more metrics of the pre-stored model. If the ratio is higher than a threshold (e.g., greater than 60%, 70%, 80%, 90%, etc.), the system can determine a sufficient level of similarity.

[0086] In step 712, if the system determines that a specified object (e.g., a shoe-shaped object) has been identified within the image / depth data over a specified time period, the method may proceed to step 716, where the system transmits the position and angle information determined in step 710 to a software application (e.g., a rendering engine) executing on one or more computing devices for rendering 3D content. In step 716, the system can use the position and angle information determined in step 710 to begin rendering 3D content associated with the tracked physical object using the position / location and angle information determined by the tracking engine. If the system determines that the specified object (e.g., a shoe-shaped object) has not been identified in the image / depth data over the specified time period, the method may proceed to step 714, where the system may notify a system administrator or other user to stop rendering the tracked object.

[0087] In the exemplary embodiments described above, various features and steps can be combined, divided, omitted, rearranged, modified, and / or expanded in any desired manner, depending on the specific result and / or application. Various changes, modifications, and improvements will readily occur to those skilled in the art. Although not expressly stated herein, such changes, modifications, and improvements become apparent through this disclosure and are intended to be within the spirit and scope of this disclosure. Therefore, the foregoing description is merely exemplary and not restrictive.

[0088] Now for reference Figure 8 This figure illustrates an example method for drawing and projecting a virtual design (e.g., 3D content) onto a tracked physical object using dynamic projection mapping, which can be executed by one or more computing devices, according to an embodiment of this disclosure. When describing the steps of the following method, the term "system" may be used when referring to each component used in each step of the method. Such usage is merely for the convenience of discussing the method and should not be construed as limiting the precise embodiments in which the following methods can be performed. Furthermore, the following method can be referenced above... Figure 7 The methods described are executed together, or in addition to the methods mentioned above. Figure 7 In addition to the methods described, it also performs other tasks.

[0089] In step 802, the system obtains the coordinate position and rotation information of the image / depth data associated with the tracked object. As described above, the system can utilize a tracking engine (e.g., a software application) running on one or more computing devices to sense and track physical objects. As described above, regarding Figure 7In step 710 of the illustrated example method, the tracking engine further uses the image / depth data obtained from the CPP unit to determine the X / Y position and rotation angle information of the tracked object. In some instances, the tracking engine may execute on one or more computing devices located remotely from (and / or within) the projection mapping system. In instances where the tracking engine executes on a remote computing device, the system may request / obtain coordinate position and rotation angle information of the image / depth data associated with the tracked physical object from one or more remote computing devices.

[0090] In step 804, the system positions a virtual camera in the 3D scene from the perspective of one or more projection devices to project 3D content onto the tracked physical object. As described above, the system can obtain a pre-built 3D model of the tracked object (e.g., element 522) and can also position the 3D model at the X / Y position and angle received in step 802. The drawn 3D model matching the position and angle of the tracked object can be displayed by the system via a user interface, such as... Figure 5A Component 502 is illustrated. In step 804, the system can utilize a virtual camera to render the resulting / final scene of the tracked object from the perspective of each image projection device (e.g., projector, CPP unit) used by the system to project the drawn / derived 3D model onto the tracked object. Figure 5B and Figure 5C As shown, when 3D content is projected onto the tracked object, the system can use one or more projectors to output the drawn 3D model, and further align the drawn 3D content with the tracked physical object by position and rotation angle.

[0091] In step 806, the system determines which projection devices are available to project one or more 3D model / content components onto the tracked physical object. As described above, the system can use projector prioritization techniques to determine which projectors have the most advantageous view for projecting components (e.g., regions of interest) of the 3D content. In step 806, the system may determine one or more projection devices among a plurality of projection devices within the system that are available to project 3D content onto the tracked object. In step 806, the system may split / segment the 3D model obtained and / or generated by the system into one or more components (e.g., regions of interest) corresponding to different regions of the object. For example, in an instance where the tracked object is footwear, one or more regions of interest may include various parts / segments of the footwear, such as the heel, midsole, upper, etc.

[0092] In step 808, the system determines which of the projection devices(s) identified in step 806 is most suitable for projecting one or more specific 3D components onto the tracked physical object. For example, in step 808, for the first projection device among the one or more projection devices identified in step 806, the system can compare the average surface normal (e.g., perpendicular to the 3D geometry) of the tracked object with the angle of the first projector (e.g., a CPP unit). In step 808, the system can also compare the average surface normal of the 3D geometry based on each component with the angle of the first projector. In other words, although in Figure 8 Not shown, but this step can be performed for each component of the 3D content / model, each component corresponding to a different region of interest of the tracked object determined in step 806. In some instances, in step 808, the system may store in a database (or other suitable memory) indications about which projector the system has determined is best suited for displaying / projecting a particular 3D component. If the system determines that different projectors are better suited for displaying a 3D component, the system may modify the stored indications to assign / associate the 3D component to the more suitable projector. In step 808, the system may query the database to determine if a more suitable projector for displaying a particular 3D has already been determined.

[0093] In step 808, if the system determines that the current projector is not the optimal projector for displaying 3D components, the method may proceed to step 310, whereby the system may instruct the current projector not to render one or more 3D components considered by the system in step 808. As described above, in some cases, the system may consider multiple 3D components when determining whether the current projector is the optimal projector for rendering a particular 3D component among multiple 3D components. In this case, in step 310, the system may instruct the current projector not to render each of the one or more 3D components that the system determines (in step 808) the current projector will not render optimally.

[0094] In step 808, if the system determines that the current projector is the optimal projector for displaying the 3D components, the method may proceed to step 812, whereby the system may instruct the current projector to render one or more 3D components considered by the system in step 808. In step 812, the system may store an indication that the current projector is the optimal projector for rendering the 3D components. As described above, in some cases, when determining in step 808 whether the current projector is the optimal projector for rendering a particular 3D component among a plurality of 3D components, the system may consider multiple 3D components. In this case, in step 812, the system may instruct the current projector to render each of the one or more 3D components that the system determines (in step 808) the current projector would optimally render.

[0095] In step 314, the system determines whether each projector identified in step 806 has been considered by the system (in step 808) to determine if the projector is optimal for rendering one or more 3D model / content components onto the tracked object. If the system determines in step 314 that additional projectors still need to be considered, the method can return to step 808, where the system determines whether a subsequent projector is the optimal projector for displaying one or more 3D components. The system can execute steps 808 through 814 until each projector identified by the system in step 808 has been considered. In step 314, if the system determines that additional projectors do not need to be considered, in other words, the system has considered each projector identified in step 806, the method can proceed to step 816.

[0096] In step 816, the system outputs the drawn view of the 3D components using one or more projectors identified (in steps 810 and 812) as the optimal projectors. For example, in step 816, the system may instruct each projector identified / determined in step 812 to output its respective virtual camera from the 3D scene onto the physical object. In step 818, the system uses each projector identified / determined in step 812 to project one or more drawn 3D components onto the physically tracked object (e.g., a physical shoe, such as...). Figure 5B and Figure 5C As illustrated above, the system uses only those projectors identified as optimal and providing the best image quality to display / project 3D content; in other words, it uses only those projectors with the most advantageous view of the 3D content and / or those most directly facing each region of interest of the tracked object, which corresponds to one or more 3D content / model components. The system can repeat this process for each frame, updating the projector prioritization as the position and angle of the tracked object change, thereby ensuring the best possible image quality.

[0097] In the following text, various features will be highlighted in a set of numbered clauses or paragraphs. These features should not be construed as limiting the invention or inventive concept, but are provided merely as a highlighting of some of the features described herein, and not as implying a particular order of importance or relevance of these features.

[0098] Clause 1. A method comprising: acquiring image data associated with a first object via one or more image capture devices; generating one or more point clouds associated with the first object based on the image data; one or more contours of the first object; determining at least one of positional information and angle information of the contours of the first object based on the one or more point clouds; dynamically drawing a three-dimensional representation of the first object based on the positional information or the angle information; and using a plurality of projection devices to output the three-dimensional representation of the first object for display, wherein the three-dimensional representation is aligned with and projected onto the first object.

[0099] Clause 2. The method according to Clause 1 further includes: using the one or more point clouds to generate a two-dimensional representation of the tracking space including the first object.

[0100] Clause 3. The method according to Clause 2, wherein using the plurality of projection devices to output the three-dimensional representation of the first object for display further comprises: determining that the first object remains continuously within the tracking space during a threshold time period.

[0101] Clause 4. The method according to Clause 2, wherein using the plurality of projection devices to output the three-dimensional representation of the first object for display further comprises: determining that the first object has been monitored by the one or more image capture devices for at least a threshold time period.

[0102] Clause 5. The method according to any one of Clauses 1 to 4, wherein dynamically drawing the three-dimensional representation of the first object further comprises: sending the position information or the angle information of the one or more contours of the first object to the drawing engine.

[0103] Clause 6. The method according to any one of Clauses 1 to 5, wherein determining at least one of the position information or angle information of the contour of the first object further comprises: monitoring the position information or angle information of the contour of the first object via the one or more image capturing devices and during a first time period.

[0104] Clause 7. The method according to any one of Clauses 1 to 6, wherein the three-dimensional representation of the first object includes footwear design.

[0105] Clause 8. The method according to any one of Clauses 1 to 7 further comprises: determining a first set of contours based on a stored three-dimensional model; determining position information and angle information for each contour in the first set of contours; and comparing the one or more contours of the first object with the first set of contours based on the corresponding position information and angle information.

[0106] Clause 9. The method according to any one of Clauses 1 to 8, wherein outputting the three-dimensional representation of the first object for display further comprises: outputting one or more surface regions associated with the three-dimensional representation of the first object onto the first object for display via the plurality of projection devices.

[0107] Clause 10. The method according to any one of Clauses 1 to 9, wherein the one or more image capturing devices are positioned around the first object.

[0108] Clause 11. The method according to any one of Clauses 1 to 10 further includes: receiving input via an input device indicating one or more regions of the first object to be tracked by the one or more image capturing devices.

[0109] Clause 12. The method according to any one of Clauses 1 to 11 further comprises: determining position information of the first object; and selecting one or more of the plurality of projection devices based on the position information to output one or more components of the three-dimensional representation of the first object for display.

[0110] Clause 13. The method according to any one of Clauses 1 to 12 further comprises: determining projection angle information of the plurality of projection devices; and selecting one or more of the plurality of projection devices based on the projection angle information to output one or more components of the three-dimensional representation of the first object for display.

[0111] Clause 14. The method according to any one of Clauses 1 to 13 further comprises: determining rotation information of the first object; and selecting one or more of the plurality of projection devices based on the rotation information of the first object to output one or more components of the three-dimensional representation of the first object for display.

[0112] Clause 15. The method according to any one of Clauses 1 to 14 further comprises: for each of the plurality of projection devices, comparing the average surface normal of a corresponding portion of the first object with the projection angle of the projection device; and selecting a projection device based on the comparison to output at least a first component of the three-dimensional representation of the first object.

[0113] Clause 16. The method according to any one of Clauses 1 to 15 further includes: displaying the one or more associated point clouds in coordinate space via a user interface.

[0114] Clause 17. The method according to any one of Clauses 1 to 16, wherein outputting the three-dimensional representation of the first object for display further comprises: generating a representation of the tracking space of the first object for display.

[0115] Clause 18. The method according to Clause 17, wherein generating the representation of the tracking space for display further comprises: generating one or more views of the tracking space for display.

[0116] Clause 19. The method according to any one of Clauses 1 to 18 further includes: determining one or more regions of interest of the three-dimensional representation of the first object.

[0117] Clause 20. A computing device comprising: one or more processors; and a memory storing instructions, which, when executed, cause the computing device to perform any one of Clauses 1 to 19.

[0118] Clause 21. A system comprising: a first computing device configured to perform the method of any one of Clauses 1 to 19; one or more image capturing devices for acquiring an image associated with a first object; and a plurality of projection devices for outputting the three-dimensional representation of the first object for display.

[0119] Clause 22. A computer-readable medium storing instructions that, when executed, cause to perform the method described in any one of Clauses 1 to 19.

[0120] in conclusion

[0121] Although the subject matter has been described using language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms of implementing the claims. The foregoing description is intended to enable others skilled in the art to optimally utilize the features described in the various embodiments, and to have various modifications suitable for the particular intended use.

Claims

1. A method comprising: Image data associated with the first footwear product is obtained via one or more image capture devices; One or more point clouds associated with the first footwear product are generated based on the image data; For one or more contours of the first footwear article: Based on the one or more point clouds, determine at least one of the positional and angle information of the outline of the first footwear product; The three-dimensional representation of the first footwear product is dynamically drawn based on the location or angle information; as well as Multiple projection devices are used to output the three-dimensional representation of the first footwear for display, wherein the three-dimensional representation is aligned with the first footwear and projected onto the first footwear.

2. The method according to claim 1, further comprising: The one or more point clouds are used to generate a two-dimensional representation of the tracking space including the first footwear.

3. The method of claim 2, wherein using the plurality of projection devices to output the three-dimensional representation of the first footwear for display further comprises: It is determined that the first footwear product remains continuously within the tracking space during a threshold time period.

4. The method of claim 2, wherein using the plurality of projection devices to output the three-dimensional representation of the first footwear for display further comprises: It is determined that the first footwear product has been monitored by the one or more image capture devices for at least a threshold time period.

5. The method according to claim 1, wherein dynamically drawing the three-dimensional representation of the first footwear product further comprises: The position information or angle information of one or more contours of the first footwear product is sent to the rendering engine.

6. The method according to claim 1, wherein determining at least one of the positional information or angle information of the contour of the first footwear article further comprises: The positional information or angle information of the outline of the first footwear article is monitored via the one or more image capturing devices during a first time period.

7. The method of claim 1, wherein the three-dimensional representation of the first footwear article includes footwear design.

8. The method according to claim 1, further comprising: The first set of contours is determined based on the stored 3D model; Determine the position and angle information of each contour in the first group of contours; as well as Based on the corresponding position and angle information, one or more contours of the first footwear product are compared with the first set of contours.

9. The method of claim 1, wherein outputting the three-dimensional representation of the first footwear article for display further comprises: One or more surface areas associated with the three-dimensional representation of the first footwear are output onto the first footwear for display via the plurality of projection devices.

10. An apparatus comprising: One or more processors; as well as A memory operatively coupled to the one or more processors and storing computer-readable instructions that, when executed, cause the device to: Image data associated with the first footwear product is obtained via one or more image capture devices; One or more point clouds associated with the first footwear product are generated based on the image data; For one or more contours of the first footwear article: Based on the one or more point clouds, determine at least one of the positional and angle information of the outline of the first footwear product; The three-dimensional representation of the first footwear product is dynamically drawn based on the location or angle information; as well as Multiple projection devices are used to output the three-dimensional representation of the first footwear for display, wherein the three-dimensional representation is aligned with the first footwear and projected onto the first footwear.

11. The device of claim 10, wherein the instructions, when executed, further cause the device to: The one or more point clouds are used to generate a two-dimensional representation of the tracking space including the first footwear.

12. The device of claim 11, wherein the instructions, when executed, further cause the device to use the plurality of projection devices to output the three-dimensional representation of the first footwear for display: It is determined that the first footwear product remains continuously within the tracking space during a threshold time period.

13. The device of claim 11, wherein the instructions, when executed, further cause the device to use the plurality of projection devices to output the three-dimensional representation of the first footwear for display: It is determined that the first footwear product has been monitored by the one or more image capture devices for at least a threshold time period.

14. The device of claim 10, wherein the instructions, when executed, further cause the device to dynamically draw the three-dimensional representation of the first footwear article by: The position information or angle information of one or more contours of the first footwear product is sent to the rendering engine.

15. The device of claim 10, wherein the instructions, when executed, further cause the device to determine at least one of positional or angular information of the contour of the first footwear article by: The positional information or angle information of the outline of the first footwear article is monitored via the one or more image capturing devices during a first time period.

16. The apparatus of claim 10, wherein the three-dimensional representation of the first footwear article includes footwear design.

17. The device of claim 10, wherein the instructions, when executed, further cause the device to: The first set of predetermined contours is determined based on the stored 3D model; Determine the position and angle information of each of the first set of predetermined contours; Based on the corresponding position and angle information, the one or more contours of the first footwear product are compared with the first set of predetermined contours.

18. A system comprising: Computing device; One or more image capture devices; as well as Multiple projection devices; The computing device includes: One or more processors; and A memory storing first computer-readable instructions, which, when executed by the one or more processors, cause the computing device to: Image data associated with the first footwear product is obtained via one or more image capture devices; One or more point clouds associated with the first footwear product are generated based on the image data; For one or more contours of the first footwear article: Based on the one or more point clouds, determine at least one of the positional and angle information of the outline of the first footwear product; Dynamically draw a three-dimensional representation of the first footwear product based on the location or angle information; and Multiple projection devices are used to output the three-dimensional representation of the first footwear for display, wherein the three-dimensional representation is aligned with the first footwear and projected onto the first footwear.

19. The system of claim 18, wherein the instructions, when executed by the one or more processors, further cause the computing device to: The one or more point clouds are used to generate a two-dimensional representation of the tracking space including the first footwear.

20. The system of claim 19, wherein the instructions, when executed by the one or more processors, further cause the computing device to use the plurality of projection devices to output the three-dimensional representation of the first footwear for display: It is determined that the first footwear product remains continuously within the tracking space during a threshold time period.

21. The method of claim 1, wherein the one or more image capturing devices are positioned around the first footwear article.

22. The method according to claim 1, further comprising: The system receives input via an input device indicating one or more areas of the first footwear product to be tracked by the one or more image capture devices.

23. The method according to claim 1, further comprising: Determine the location information of the first footwear product; as well as Based on the location information, one or more of the plurality of projection devices are selected to output one or more components of the three-dimensional representation of the first footwear product for display.

24. The method according to claim 1, further comprising: Determine the projection angle information of the plurality of projection devices; as well as Based on the projection angle information, one or more of the plurality of projection devices are selected to output one or more components of the three-dimensional representation of the first footwear product for display.

25. The method according to claim 1, further comprising: Based on the angular information of the outline of the first footwear, one or more of the plurality of projection devices are selected to output one or more components of the three-dimensional representation of the first footwear for display.

26. The method according to claim 1, further comprising: Determine the rotation information of the first footwear product; as well as Based on the rotation information of the first footwear product, one or more of the plurality of projection devices are selected to output one or more components of the three-dimensional representation of the first footwear product for display.

27. The method according to claim 1, further comprising: For each of the plurality of projection devices, the average surface normal of the corresponding portion of the first footwear is compared with the projection angle of the projection device; as well as Based on the comparison, a projection device is selected to output at least a first component of the three-dimensional representation of the first footwear article.

28. The method according to claim 1, further comprising: The one or more associated point clouds are displayed in coordinate space via a user interface.

29. The method of claim 1, wherein outputting the three-dimensional representation of the first footwear article for display further comprises: A representation of the tracking space of the first footwear product is generated for display.

30. The method of claim 29, wherein generating the representation of the tracking space for display further comprises: One or more views of the tracking space are generated for display.

31. The method according to claim 1, further comprising: Determine one or more regions of interest in the three-dimensional representation of the first footwear article.

32. The device of claim 10, wherein the one or more image capturing devices are positioned around the first footwear article.

33. The device of claim 10, wherein the instructions, when executed, further cause the device to: The system receives input via an input device indicating one or more areas of the first footwear product to be tracked by the one or more image capture devices.

34. The device of claim 10, wherein the instructions, when executed, further cause the device to: Determine the location information of the first footwear product; and Based on the location information, one or more of the plurality of projection devices are selected to output one or more components of the three-dimensional representation of the first footwear product for display.

35. The device of claim 10, wherein the instructions, when executed, further cause the device to: Determine the projection angle information of the plurality of projection devices; and Based on the projection angle information, one or more of the plurality of projection devices are selected to output one or more components of the three-dimensional representation of the first footwear product for display.

36. The device of claim 10, wherein the instructions, when executed, further cause the device to: Based on the angular information of the outline of the first footwear, one or more of the plurality of projection devices are selected to output one or more components of the three-dimensional representation of the first footwear for display.

37. The device of claim 10, wherein the instructions, when executed, further cause the device to: Determine the rotation information of the first footwear product; and Based on the rotation information of the first footwear product, one or more of the plurality of projection devices are selected to output one or more components of the three-dimensional representation of the first footwear product for display.

38. The device of claim 10, wherein the instructions, when executed, further cause the device to: For each of the plurality of projection devices, the average surface normal of the corresponding portion of the first footwear is compared with the projection angle of the projection device; and Based on the comparison, a projection device is selected to output at least a first component of the three-dimensional representation of the first footwear article.

39. The device of claim 10, wherein the instructions, when executed, further cause the device to: The one or more associated point clouds are displayed in coordinate space via a user interface.

40. The device of claim 10, wherein the instructions, when executed, further cause the device to output the three-dimensional representation of the first footwear article for display by: A representation of the tracking space of the first footwear product is generated for display.

41. The device of claim 40, wherein the instructions, when executed, further cause the device to generate the representation of the tracking space for display by: One or more views of the tracking space are generated for display.

42. The device of claim 10, wherein the instructions, when executed, further cause the device to: Determine one or more regions of interest in the three-dimensional representation of the first footwear article.

Images