Depth-Based Projection of Image-Based Content

Abstract

A projection system and method for depth-based projection of image-based content is provided. The projection system receives a series of image frames to be projected onto a physical surface of a three-dimensional (3D) structure. The projection system controls a depth sensor to acquire depth data associated with the physical surface and provides a first input to a neural network-based model, the first input including the acquired depth data and a first image frame of the received series of image frames. The projection system further receives a set of geometric correction values ​​as a first output of the neural network-based model for the provided first input, and modifies the first image frame based on the received set of geometric correction values. The projection system further controls an illumination circuit to project the modified first image frame onto the physical surface.

Description

[Technical field]

[0001] CROSS REFERENCE TO RELATED APPLICATIONS / INCORPORATION BY REFERENCE none

[0002] Various embodiments of the present disclosure relate to image processing, machine learning, and image projection onto physical surfaces. In particular, various embodiments of the present disclosure relate to projection systems and methods for depth-based projection of image-based content. [Background technology]

[0003] Generally, light-based projectors (such as projectors based on digital light processing) are used to project content onto a physical surface (e.g., a projection screen). These projectors are widely used in environments such as conference rooms, cinemas, or classrooms. In some scenarios, if the physical surface is tilted relative to the projector, or if the physical surface is not smooth or flat, the shape of the projected content on the physical surface appears warped or distorted. For example, if the projector is not properly aligned with the physical surface (such as a wall), the projected content may appear trapezoidal compared to an expected rectangle. Such warping or distortion is undesirable and uncomfortable for people who view the projected content. Usually, the projector requires manual adjustment or alignment with the physical surface to properly project media content onto the physical surface. However, the manual adjustment process is very tedious and time-consuming. Summary of the Invention [Problem to be solved by the invention]

[0004] The limitations and disadvantages of the conventional approach will become apparent to one skilled in the art by comparing the described system with certain aspects of the present disclosure illustrated in the remaining portions of this application and with reference to the drawings. [Means for solving the problem]

[0005] A projection system and method for depth-based projection of image-based content is provided substantially as illustrated and / or described in connection with at least one of the figures and more fully as set forth in the claims.

[0006] These and other features and advantages of the present disclosure will become apparent from a consideration of the following detailed description of the disclosure when taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an environment for depth-based projection of image-based content, according to an embodiment of the present disclosure. [Diagram 2] 2 is an exemplary block diagram of the projection system of FIG. 1 in accordance with an embodiment of the present disclosure. [Figure 3A] 2 illustrates an example operation for calibration of the projection system 102 of FIG. 1 according to an embodiment of the present disclosure. [Figure 3B] 2 illustrates an example scenario of user guidance for calibrating the projection system 102 of FIG. 1 in accordance with an embodiment of the present disclosure. [Figure 4A] FIG. 2 illustrates an example operation for depth-based projection of image-based content, according to an embodiment of the present disclosure. [Figure 4B] FIG. 2 illustrates an example scenario of a first image frame projected onto a physical surface and a modified first image frame, according to an embodiment of the present disclosure. [Figure 5A] 5B and 5C together illustrate an example operation for depth-based projection of image-based content onto a non-flat surface according to an embodiment of the present disclosure. [Figure 5B] 5A and 5C together illustrate an example operation for depth-based projection of image-based content onto a non-flat surface according to an embodiment of the present disclosure. [Figure 5C]5A and 5B together illustrate an example operation for depth-based projection of image-based content onto a non-flat surface according to an embodiment of the present disclosure. [Figure 6A] FIG. 2 illustrates an example scenario for depth-based projection of image-based content, according to an embodiment of the present disclosure. [Figure 6B] FIG. 6B illustrates an exemplary surface cluster for the physical surface of FIG. 6A, according to an embodiment of the present disclosure. [Figure 6C] FIG. 6B illustrates an example scenario for the projection of the image frame of FIG. 6A according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates an architecture of a neural network-based model for depth-based projection of image-based content, according to an embodiment of the present disclosure. [Figure 8] 2 is an exemplary block diagram of an image projection subsystem of the projection system of FIG. 1 in accordance with an embodiment of the present disclosure. [Figure 9] FIG. 8 illustrates an example operation for training an artificial neural network of the neural network-based model of FIG. 7, according to an embodiment of the present disclosure. [Figure 10] 1 is a flowchart illustrating an example method for depth-based projection of image-based content, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The disclosed projection system and method for depth-based projection of image-based content may find implementation as described below. The disclosed projection system includes a circuit and an image projection subsystem that may be coupled to the circuit. The image projection subsystem may include illumination circuitry. The projection system receives a sequence of image frames to be projected onto a physical surface of a three-dimensional (3D) structure and controls a depth sensor to acquire depth data associated with the physical surface. The disclosed projection system provides inputs including the acquired depth data and a first image frame of the received sequence of image frames to a neural network-based model and receives a set of geometric correction values ​​as a first output of the neural network-based model for the provided input. The disclosed projection system modifies the first image frame based on the received set of geometric correction values ​​and controls the illumination circuit to project the modified first image frame onto the physical surface.

[0009] The disclosed projection system can project image-based content onto a physical surface while maintaining the correct shape of the projection, regardless of whether the projection system is properly aligned with the physical surface. For example, the disclosed projection system can modify an image frame based on a geometric correction such that the projected image frame retains the correct shape and original aspect ratio of the unmodified image frame. The disclosed projection system eliminates the requirement for manual adjustment or alignment of the projector system to the physical surface.

[0010] The disclosed projection system can also classify a physical surface into multiple surface portions based on the smoothness value of the physical surface. If the physical surface is not perfectly flat or smooth, such classification can be used to determine the smooth surface or the flattest surface portion of the physical surface. The projection system can project an image frame onto the smooth surface or the flattest surface portion of the physical surface. The projection system can use such projection to provide a pleasant projection experience on any type of surface.

[0011] Figure 1 illustrates an environment for depth-based projection of image-based content, according to an embodiment of the present disclosure. Figure 1 shows a diagram of a networked environment 100. The networked environment 100 includes a projection system 102 that includes an image projection subsystem 104 and a depth sensor 106. Figure 1 also shows a neural network-based model 108, a physical surface 110, a server 112, and a communication network 114. Also shown is a sequence of image frames 116 that can be projected onto the physical surface 110.

[0012] The projection system 102 may include suitable logic, circuitry, interfaces, and / or code that may be configured to control the image projection subsystem 104 to receive the sequence of image frames 116, modify each of the received sequence of image frames 116 based on the output of the neural network-based model 108, and project each of the modified image frames onto the physical surface 110. The output of the neural network-based model 108 may include geometric corrections (e.g., keystone correction values) that may be applied to the sequence of image frames 116. In an embodiment, the projection system 102 may be an electronic device that may be integrated into a light-based projector (such as a DLP projector) to eliminate the need for a tedious manual adjustment process of the light-based projector. In another embodiment, the projection system 102 may be external to the light-based projector and communicatively coupled to the light-based projector, and may be used by any user for calibration / alignment of the projection surface with the projection surface. Examples of projection system 102 include, but are not limited to, overhead or mountable projectors, pico projectors, digital light processing (DLP) projectors, mainframe machines, computer workstations, smart phones, cellular phones, mobile phones, wearable electronic devices, head-mounted devices, gaming devices, servers, consumer electronics (CE) devices, computing devices, and / or any other device with projection capabilities.

[0013] 1, the image projection subsystem 104 is shown as part of the projection system 102. However, the disclosure may not be so limited, and in some embodiments, the image projection subsystem 104 may be external to and communicatively coupled to the projection system 102.

[0014] The image projection subsystem 104 may include suitable logic, circuitry, interfaces, and / or code that may be configured to project each of the modified image frames (or each of the received sequence of image frames 116) onto the physical surface 110. In some embodiments, the image projection subsystem 104 may be separate from the projection system 102 and communicate with the projection system 102 via a communications network 114. Examples of the image projection subsystem 104 may include, but are not limited to, a cathode ray tube projection unit, a laser or (light emitting diode) LED based circuitry, a liquid projection unit, a laser or (light emitting diode) LED based circuitry, a liquid crystal based projection unit, an optical projection unit (including a lens assembly, an illumination unit (with a light source and a driver), and other ancillary devices / sensors), a digital light processing projection unit, a smart phone, a cellular phone, a mobile computer, a gaming device, a consumer electronics (CE) device, and / or any other device having image projection capabilities.

[0015] The depth sensor 106 may include suitable logic, circuitry, and interfaces that may be configured to capture depth data associated with the physical surface 110. The depth data may include depth values, each of which may be a measure of a distance (e.g., a relative distance) between the depth sensor 106 and a corresponding point on the physical surface 110.

[0016] The neural network-based model 108 may be a computational network or system in which nodes are artificial neurons arranged in multiple layers. The multiple layers of the neural network-based model 108 may include an input layer, one or more hidden layers, and an output layer. Each layer of the multiple layers may include one or more nodes (or artificial neurons, e.g., represented by circles). The output of every node in the input layer may be coupled to at least one node in the hidden layer(s). Similarly, the input of each hidden layer may be coupled to the output of at least one node in the other layers of the neural network-based model 108. The output of each hidden layer may be coupled to the input of at least one node in the other layers of the neural network-based model 108. The node(s) in the final layer may receive inputs from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer may be determined from hyperparameters of the neural network-based model 108. Such hyperparameters may be set before, during, or after training of the neural network-based model 108 based on a training dataset.

[0017] Each node of the neural network-based model 108 may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be adjusted during training of the network. The set of parameters may include, for example, weight parameters and regularization parameters. Each node may use a mathematical function to calculate an output based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of the neural network-based model 108. All or some of the nodes of the neural network-based model 108 may correspond to the same or different mathematical functions.

[0018] In training the neural network-based model 108, one or more parameters of each node of the neural network-based model 108 may be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result based on the loss function of the neural network-based model 108. The above process may be repeated for the same or different inputs until a minimum of the loss function is achieved to minimize the training error. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristic methods.

[0019] The neural network-based model 108 may include electronic data that may be implemented, for example, as a software component of an application executable on the projection system 102. The neural network-based model 108 may rely on libraries, external scripts, or other logic / instructions for execution by a processing device, such as a circuit. The neural network-based model 108 may include code and routines configured to enable a computing device, such as a circuit, to perform one or more operations to generate a set of geometric correction values. Additionally or alternatively, the neural network-based model 108 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural network-based model 108 may be implemented using hardware, software, or a combination of hardware and software.

[0020] Examples of the neural network-based model 108 may include, but are not limited to, a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), a fully connected neural network, and / or a combination of such networks. In an embodiment, the neural network-based model 108 may be a hybrid network of one or more variants of a DNN (e.g., a CNN-ANN).

[0021] The physical surface 110 may be a three-dimensional (3D) structured surface onto which a series of image frames 116 may be projected. The physical surface 110 may be fixed (i.e., permanently attached) or mobile (i.e., portable). Examples of the physical surface 110 may include, but are not limited to, a motorized projector screen, a manual projector screen, a tripod projector screen, a fixed frame screen, a floor-rising projector screen, a wall with paint, fabric, an extendable projector screen, projector screen paint, and any electronic device with projection capabilities.

[0022] The server 112 may include suitable logic, circuitry and interfaces, and / or code that may be configured to store the sequence of image frames 116. The server 112 may also be configured to train and store the neural network-based model 108. In one embodiment, the server 112 may be implemented as a cloud server and may perform operations through web applications, cloud applications, HTTP requests, repository operations, file transfers, and the like. Other implementations of the server 112 may include, but are not limited to, a database server, a file server, a web server, a media server, an application server, a mainframe server, or a cloud computing server. In an embodiment, the server 112 may be implemented as a number of distributed cloud-based resources using a number of techniques known to those skilled in the art.

[0023] The communication network 114 may include a communication medium that allows the projection system 102 and the server 112 to communicate with each other. The communication network 114 may be either a wired or wireless connection. Examples of the communication network 114 may include, but are not limited to, the Internet, a cloud network, a Wireless Fidelity (Wi-Fi) network, a personal area network (PAN), a fourth or fifth generation mobile wireless network, a local area network (LAN), or a metropolitan area network (MAN). The various devices in the network environment 100 may be configured to connect to the communication network 114 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE802.11, Light Fidelity (Li-Fi), 802.16, IEEE802.11s, IEEE802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0024] The projection system 102 may be configured to receive a series of image frames 116 during operation. In one embodiment, the projection system 102 may receive an analog video signal (e.g., YPbPr video) from an external data source (e.g., a high definition television (HDTV) tuner, a set-top box, a standard definition (SD) and HD disc, a personal computer, or a workstation) and digitize the analog video signal (which may be a digital video) to obtain the series of image frames 116. In another embodiment, the projection system 102 may receive the series of image frames 116 from the server 112 via the communication network 114. The received series of image frames 116 may include, for example, a first image frame 116A, a second image frame 116B, ..., an Nth image frame 116N.

[0025] The projection system 102 may be configured to control the depth sensor 106 to acquire depth data associated with the physical surface 110. The acquired depth data may include a depth value indicative of a distance of the physical surface 110 from the projection system 102 and / or the image projection subsystem 104.

[0026] The projection system 102 may be further configured to provide a first input to the neural network-based model 108. The first input may include, for example, the acquired depth data and a first image frame 116A of the series of image frames 116. The projection system 102 may receive a set of geometric correction values ​​as a first output of the neural network-based model 108 for the provided first input. The received set of geometric correction values ​​may include, for example, a set of digital keystone correction values. In an embodiment, the received set of geometric correction values ​​may also include coordinate values ​​that can be used for resizing the first image frame 116A.

[0027] The projection system 102 may be further configured to modify the first image frame 116A based on the received set of geometric corrections. For example, the projection system 102 may be configured to resize and / or warp the first image frame 116A such that when the modified first image frame 116A is projected onto the physical surface 110, it appears rectangular despite a misalignment between the projection system 102 and the physical surface 110. The projection system 102 may send the modified first image frame to the image projection subsystem 104 and control the image projection subsystem 104 to project the modified first image frame onto the physical surface 110.

[0028] 1 illustrates the neural network-based model 108 as part of the projection system 102, the disclosure may not be so limited, and in some embodiments, the neural network-based model 108 may be located on a cloud server separate from the projection system 102 without departing from the scope of the disclosure. Also, while FIG. 1 illustrates the image projection subsystem 104 as integrated within the projection system 102, the disclosure may not be so limited. In some embodiments, the image projection subsystem 104 may be separate from the projection system 102 without departing from the scope of the disclosure.

[0029] FIG. 2 is an example block diagram of the projection system of FIG. 1 according to an embodiment of the present disclosure. The description of FIG. 2 is provided with reference to the elements of FIG. 1. FIG. 2 illustrates a block diagram 200 of the projection system 102. The projection system 102 may include a circuit 202, a memory 204, an input / output (I / O) device 206, and a network interface 208. FIG. 2 also illustrates a depth sensor 106 and a neural network-based model 108. Also illustrated is an image projection subsystem 104 including an illumination circuit 210. The circuit 202 may be communicatively coupled to the memory 204, the I / O device 206, the network interface 208, and the image projection subsystem 104.

[0030] In one embodiment, the circuitry 202 can be external to and communicatively coupled to the image projection subsystem 104. In such a case, the image projection subsystem 104 can be external to (i.e., not integrated into) the projection system 102. In another embodiment, the circuitry 202 can be internal to and communicatively coupled to the image projection subsystem 104. In such a case, the image projection subsystem 104 can be internal to (i.e., integrated into) the projection system 102.

[0031] The circuitry 202 may include suitable logic, circuits, and interfaces that may be configured to execute instructions stored in the memory 204. The executed instructions may correspond to a series of image processing operations, for example, modifying the first image frame 116A and controlling the lighting circuitry 210 to project the modified first image frame onto the physical surface 110. The circuitry 202 may be implemented based on many processor technologies known in the art. Examples of the circuitry 202 may include, but are not limited to, a graphic processor unit (GPU), a co-processor, a central processing unit (CPU), an x86-based processor, a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, and combinations thereof.

[0032] The memory 204 may include suitable logic, circuitry, and / or interfaces that may be configured to store instructions that the circuit 202 may execute. The memory 204 may also store the sequence of image frames 116 and a set of geometric correction values. In at least one embodiment, the memory 204 may store the neural network-based model 108 and the acquired depth data. Example implementations of the memory 204 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0033] The I / O devices 206 may include suitable logic, circuitry, and / or interfaces that may be configured to act as an interface between a user of the projection system 102 and the projection system 102. The I / O devices 206 may be configured to receive a sequence of image frames 116. The I / O devices 206 may include a variety of input and output devices that may be configured to communicate with different operational components of the projection system 102. Examples of the I / O devices 206 may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, a microphone, and a display screen.

[0034] The network interface 208 may include suitable logic, circuitry, interfaces and / or code that may be configured to establish communications between the projection system 102 and the server 112 over the communications network 114. The network interface 208 may be configured to implement known technologies that support wired or wireless communications. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0035] The network interface 208 may be configured to communicate via offline and online wireless communications with networks such as the Internet, an intranet, and / or wireless networks such as a cellular telephone network, a wireless local area network (WLAN), a personal area network, and / or a metropolitan area network (MAN). The wireless communication may use any of a number of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), Fifth Generation New Radio (5G NR), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (WiFi) (such as IEEE 802.11, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or any other IEEE 802.11 protocol), Voice over Internet Protocol (VoIP), Wi-MAX, Internet of Things (IoT) technologies, Machine Type Communication (MTC) technologies, protocols for email, instant messaging, and / or Short Message Service (SMS).

[0036] The lighting circuitry 210 may include suitable logic, circuitry, and interfaces that may be configured to project each of the modified sequence of image frames 116 onto the physical surface 110. In an embodiment, the lighting circuitry 210 may include a light source (e.g., a laser or a set of light emitting diodes (LEDs), such as RGB LEDs) and a driver circuit coupled to the light source and to the circuitry 202. The circuitry 202 may control the driver circuit to operate the light source (based on the content that needs to be projected). In an embodiment, the lighting circuitry 210 may also include a spatial light modulator (such as a liquid crystal on silicon (LCoS) panel).

[0037] The functions or operations performed by the projection system 102 as described in Figure 1 may be performed by the circuitry 202. The operations performed by the circuitry 202 are described in detail, for example, in Figures 3A, 3B, 4A, 4B, 5A, 5B, 5C, 6A, 6B, 6C, 7, 8, 9 and 10.

[0038] FIG. 3A illustrates exemplary operations for calibration of the projection system 102 of FIG. 1 according to an embodiment of the present disclosure. The description of FIG. 3A is provided with reference to elements of FIG. 1 and FIG. 2. FIG. 3A illustrates a block diagram 300A illustrating exemplary operations 302A-302G described herein. The exemplary operations illustrated in block diagram 300A may begin at 302A and may be performed by any computer system, apparatus, or device, such as the projection system 102 of FIG. 1 or FIG. 2. Although the exemplary operations associated with one or more blocks of block diagram 300A are illustrated as discrete blocks, they may be divided into additional blocks, combined into fewer blocks, or eliminated depending on the particular implementation. The sequence of operations illustrated in block diagram 300A may be performed during a pre-initialization stage for calibration of the projection system 102 (e.g., before content can be projected onto the physical surface 110).

[0039] At 302A, a sample data acquisition operation may be performed in which the circuit 202 may be configured to receive a sample image frame for projection onto the physical surface 110. In one embodiment, the sample image frame may be extracted from a digital video signal generated by an analog-to-digital video converter or may be obtained from memory 204 of the projection system 102 or server 112.

[0040] At 302B, a sample data projection operation may be performed. In the sample data projection operation, circuitry 202 may be configured to project a sample image frame onto physical surface 110. In an embodiment, circuitry 202 may control illumination circuitry 210 to project the sample image frame onto physical surface 110.

[0041] At 302C, a projected view capture operation can be performed. In the projected view capture operation, the circuit 202 can be configured to capture a projected view of a sample image frame. In particular, the circuit 202 can be configured to determine corners of the projected sample image frame and store corresponding coordinates of each determined corner in the memory 204 of the projection system 102.

[0042] At 302D, a sample projection view search operation can be performed. In the sample projection view search operation, the circuit 202 can be configured to search for a sample projection view of a random image frame. The sample projection view can be an optimal (or even desirable) view for the projection of any image frame projected onto the physical surface 110. By way of example and not limitation, the sample projection view can be rectangular, and the center of the sample projection view can be the same as the center of the physical surface 110.

[0043] In 302E, it can be determined whether the captured projection view and the sample projection view are similar. In particular, it can be determined whether the captured projection view matches the sample projection view. The circuit 202 can compare the stored corner coordinates of the captured projection view with the corner coordinates of the sample projection view. If the captured projection view and the sample projection view are similar (or the same), control can proceed to 302F. Otherwise, control can proceed to 302G.

[0044] At 302F, a message rendering operation may be performed. In the message rendering operation, the circuitry 202 may be configured to render a message on the physical surface 110 and / or on a user device associated with a user of the projection system 102. The rendered message may indicate successful calibration of the projection system 102. After rendering the message, the pre-initialization phase of the projection system 102 may be considered complete, and the circuitry 202 may switch the projection system 102 from the pre-initialization phase to the operational phase.

[0045] In 302G, a user guidance operation can be performed. In the user guidance operation, the circuit 202 can render on the physical surface 110 and / or on the user device a set of instructions that assists the user in performing one or more operations to calibrate the projection system 102. For example, the circuit 202 can control the movement of the projection system 102 so that the projection surface of the projection system 102 is aligned with the physical surface 110. After the one or more operations are performed, the circuit 202 can receive a user input indicating the completion of the one or more operations by the user. The circuit 202 can be configured to perform a series of operations 302A-302F (or 302G) again based on the user input to check whether the captured projection view and the sample projection view are similar (or the same). This process can be repeated until the projection system 102 is successfully calibrated. Details of the user guidance operation are shown, for example, in FIG. 3B.

[0046] FIG. 3B illustrates an exemplary scenario of user guidance for calibrating the projection system 102 of FIG. 1, according to an embodiment of the present disclosure. FIG. 3B is described with reference to elements of FIG. 1, FIG. 2, and FIG. 3A. FIG. 3B illustrates an exemplary scenario 300B. Also illustrated is a physical surface 304 of a three-dimensional (3D) structure. FIG. 3B further illustrates a captured projection view 306 and a sample projection view 310 on the physical surface 304.

[0047] The circuit 202 can compare the captured projection view 306 to the sample projection view 310 to determine whether the corner coordinates of the captured projection view 306 are similar (or the same) as the corner coordinates of the sample projection view 310. If the captured projection view 306 is not similar (or the same) as the sample projection view 310, as described in 302E of FIG. 3A, the circuit 202 can render a set of instructions on the physical surface 304. In one embodiment, the set of instructions can include a first arrow 310A and a second arrow 310B.

[0048] As shown, the instruction set (including the first arrow 310A and the second arrow 310B) can assist the user in moving the projection system 102 to the left (i.e., the first action) and raising the height of the projection system 102 (i.e., the second action). After the projection system 102 has moved to the left and the height of the projection system 102 has increased (i.e., after the action has been performed), the circuit 202 can receive a user input that can indicate completion of the action sequence included in the instruction set. Based on the received user input, the circuit 202 can execute the action sequence 302A-302F (or 302G) to check whether the projection system 102 has been calibrated.

[0049] It should be noted that the first arrow 310A and the second arrow 310B shown in Figure 3B are merely an example of an instruction set. The present disclosure is also applicable to scenarios where the instruction set can be rendered as text instructions or audio instructions via an audio output device. A description of such instructions is omitted from this disclosure for the sake of brevity.

[0050] FIG. 4A illustrates an example operation for depth-based projection of image-based content, according to an embodiment of the present disclosure. FIG. 4A is described with reference to elements of FIG. 1, FIG. 2, FIG. 3A, and FIG. 3B. FIG. 4A illustrates a block diagram 400A illustrating example operations 402A-402D described herein. The example operations illustrated in block diagram 400A may begin at 402 and may be performed by any computer system, apparatus, or device, such as projection system 102 of FIG. 1 or FIG. 2. Although illustrated with discrete blocks, example operations associated with one or more blocks of block diagram 400A may be divided into additional blocks, combined into fewer blocks, or eliminated depending on the particular implementation.

[0051] At 402A, a data acquisition operation may be performed in which the circuitry 202 may receive a series of image frames 116 to be projected onto the physical surface 110 of a three-dimensional (3D) structure. The series of image frames 116 may be projected onto the physical surface 110 in the same sequence in which the series of image frames 116 are received.

[0052] In a data acquisition operation, the circuitry 202 can be further configured to control the depth sensor 106 to acquire depth data 404 associated with the physical surface 110. In one embodiment, the depth data 404 can include a depth map associated with the physical surface 110. The depth map can be an image or an image channel that includes information related to the distance of the physical surface 110 from the projection system 102.

[0053] At 402B, geometric correction values ​​can be generated. The circuit 202 can be configured to generate a set of geometric correction values ​​for the correction of the first image frame 116A. The circuit 202 can be configured to provide a first input to the neural network-based model 108 to generate the set of geometric correction values. The provided first input includes the acquired depth data 404 and the first image frame 116A of the received sequence of image frames 116. The acquired depth data 404 can be, for example, a depth map of the physical surface 110. In an embodiment, the neural network-based model 108 can be a pre-trained model that can be trained to output a geometric correction value for each image frame of the sequence of image frames 116. Details regarding the training of the neural network-based model 108 are shown, for example, in FIG. 9.

[0054] The circuit 202 can be configured to receive a first output from the neural network-based model 108 based on a first input provided thereto. The received first output can include a set of geometric correction values ​​for the first image frame 116A. In an embodiment, the set of geometric correction values ​​can include, for example, a keystone correction value or a geometric correction value for each side of the first image frame 116A.

[0055] At 402C, an image correction operation can be performed. In the image correction operation, the circuit 202 can be configured to correct the first image frame 116A based on the received set of geometric correction values. In an embodiment, the correction of the first image frame 116A can include applying an image warping operation to the first image frame 116A while maintaining the aspect ratio of the first image frame 116A. If the projection system 102 is at an angle to the physical surface 110, the image warping operation can be used to apply a geometry error correction to the first image frame 116A so that the first image frame 116A (after correction) appears correct on the physical surface 110 and preserves the aspect ratio of the first image frame 116A (before correction).

[0056] At 402D, a first image projection operation may be performed. In the first image projection operation, the circuitry 202 may be configured to control the illumination circuitry 210 to project the modified first image frame onto the physical surface 110. The projection of the modified first image frame may remain rectangular regardless of the projectable surface properties and the angle of the projection system 102 relative to the physical surface 110.

[0057] 4B is a diagram illustrating an example scenario of a first image frame projected onto a physical surface and a modified first image frame according to an embodiment of the present disclosure. An example scenario 400B is shown in FIG. 4B. Also shown is a three-dimensional (3D) structure physical surface 406, such as a wall. FIG. 4B further illustrates a first image frame 408 projected onto the physical surface 406 and a modified first image frame 410.

[0058] In an embodiment, the circuit 202 can receive a first image frame 408. After receiving the first image frame 408, the circuit 202 can control the lighting circuit 210 to project the received first image frame 408 onto the physical surface 406. As shown, the first image frame 408 can be trapezoidal when projected onto the physical surface 406. Such a projection of the first image frame 408 may be deemed undesirable or inconvenient to the audience. The circuit 202 can be configured to modify the first image frame 408 based on a set of geometric correction values ​​to modify the shape of the projection to generate a modified first image frame 410. The circuit 202 can further control the lighting circuit 210 to project the modified first image frame 410 onto the physical surface 406. As shown, the modified first image frame 410 can be rectangular when projected onto the physical surface 406. Such a projection may be deemed desirable to the audience. Details regarding the geometric correction sets and modifications are shown, for example, in FIGS. 4A, 5A, 5B, 5C, 7, 9 and 10.

[0059] 5A, 5B, and 5C collectively illustrate example operations for depth-based projection of image-based content on a non-flat surface, according to embodiments of the present disclosure. The description of FIGS. 5A, 5B, and 5C is provided with reference to elements of FIGS. 1, 2, 3A, 3B, 4A, and 4B. FIGS. 5A, 5B, and 5C illustrate a block diagram 500 illustrating example operations 502A-502K described herein. The example operations illustrated in block diagram 500 may begin at 502A and may be performed by any computer system, apparatus, or device, such as the projection system 102 of FIGS. 1 or 2. Although illustrated as discrete blocks, example operations associated with one or more blocks of block diagram 500 may be divided into additional blocks, combined into fewer blocks, or eliminated depending on the particular implementation.

[0060] At 502A, a data acquisition operation can be performed. In the data acquisition operation, the circuit 202 can be configured to acquire data required for projection onto the physical surface 504. The data can include depth data 506 related to the physical surface 504, and an image frame 508 (which can be part of the content to be projected). As shown, the physical surface 504 can be a non-flat surface, or a flat surface with some areas obscured by an object (such as a ladder or sofa set). Due to this occlusion, the depth data 506 of a flat surface (such as a wall) can show irregularities around the obscured areas even if the physical surface 504 is flat. The data acquisition operation can be divided into a depth data acquisition operation and an image reception operation.

[0061] Depth data acquisition can be performed at 502B. For the depth data acquisition, the circuitry 202 can be configured to control the depth sensor 106 to acquire depth data 506 associated with the physical surface 504. The depth data 506 can include information regarding a distance between the physical surface 504 and the projection system 102. In an embodiment, the depth data 506 can be a depth image of the physical surface 504.

[0062] At 502C, the circuitry 202 can receive an image frame 508 to be projected onto the physical surface 504. Details of receiving the image frame 508 are shown, for example, in Figures 1 and 4A.

[0063] At 502D, a flat surface determination operation may be performed. The circuit 202 may be configured to determine a flat surface portion 510 within the physical surface 504 onto which the image frame 508 is projected. To determine the flat surface, the circuit 202 may be configured to determine a first depth value associated with a corner of the physical surface 504. The circuit 202 may be configured to generate virtual depth data for the physical surface 504 based on the determination of the first depth value. In the virtual depth data, each depth value may be generated from a virtual plane that may be determined based on the corner value (i.e., the first depth value).

[0064] The circuit 202 may be further configured to compare the acquired depth data 506 with the generated virtual depth data to generate a comparison result. The circuit 202 may be further configured to apply a clustering method to the comparison result. The clustering method may be applied to divide the physical surface 504 into surface clusters. Each of the surface clusters may correspond to one or more surface portions of the physical surface 504. The circuit 202 may be further configured to determine a smoothness value for each surface cluster. The smoothness value may indicate a smoothness of the corresponding surface cluster.

[0065] The circuitry 202 may classify one or more portions of the physical surface 504 into different categories based on the calculated smoothness value. The different categories of surfaces may include, for example, flat surfaces, occluded surfaces, protruding surfaces, and hollow surfaces. The circuitry 202 may be configured to further determine a flat surface portion 510 of the physical surface 504 based on the classification. In an embodiment, one or more flat surface portions may be detected on the physical surface 504. In such a scenario, the determined flat surface portion 510 may have a maximum size (or maximum surface area) among the one or more detected flat surface portions.

[0066] In an embodiment, the circuit 202 can be configured to provide input to a convolutional neural network (CNN) 512 to determine flat surface portions 510 in the physical surface 504. The input can include acquired depth data 506 of the physical surface 504. The CNN 512 can be trained to determine the flat surface portions 510 from the depth data. The circuit 202 can receive an output of the CNN 512 based on the input. In an embodiment, the received output can include a clustering result that can indicate one or more flat surface portions of the physical surface 504 and one or more non-flat surface portions of the physical surface 504. The circuit 202 can be configured to determine a maximum size flat surface portion 510 from among the one or more flat surface portions.

[0067] At 502E, a coordinate value extraction operation can be performed. In the coordinate value extraction operation, the circuit 202 can be configured to extract coordinate values ​​associated with the determined corners 514 of the flat surface portion 510. In an embodiment, the circuit 202 can be configured to determine the corners of the flat surface portion 510. As shown, the determined corners 514 can include four corners. The circuit 202 can be further configured to extract coordinate values ​​associated with the determined corners 514 of the flat surface portion 510 based on the determination of the corners of the flat surface portion 510. The coordinate values ​​can be expressed in terms of image coordinates or physical measurements (e.g., in terms of a position in a room where the physical surface 504 is present or a position of the depth sensor 106). In some embodiments, the circuit 202 can receive the coordinate values ​​associated with the determined corners 514 of the flat surface portion 510 from the CNN 512.

[0068] At 502F, an image frame resize operation may be performed. It may be necessary to resize the image frame 508 because the image frame 508 (initially received (at 502C) for projection onto the physical surface 504) needs to be projected onto the determined flat surface portion 510 (smaller than the size of the physical surface 504). The circuit 202 may be configured to resize the image frame 508 to generate a resized image frame 516 that may be suitable for projection onto the flat surface portion 510 (the widest / largest of all flat surface portions). The image frame 508 may be resized based on the extracted coordinate values. For example, the corner coordinate values ​​of the resized image frame 508 may match the extracted coordinate values ​​(at 502E). In an embodiment, the resizing of the image frame 508 may correspond to applying an image scaling operation to the image frame 508 to generate the resized image frame 516 while maintaining the aspect ratio of the image frame 508.

[0069] At 502G, a projection length determination operation may be performed. In the projection length determination operation, the circuit 202 may be configured to determine a set of projection length values ​​associated with the determined corners 514 of the flat surface portion 510. Each projection length value in the set of projection length values ​​may correspond to a distance of the corresponding corner (from the corners 514) to the projection system 102. In an embodiment, the circuit 202 may be configured to determine the set of projection length values ​​associated with the determined corners of the flat surface portion 510 based on the depth data 506. In another embodiment, the set of projection length values ​​may be received from the CNN 512 or any other artificial neural network model used in the projection system 102.

[0070] At 502H, a surface length values ​​determination operation may be performed. The circuit 202 may be configured to determine a set of surface length values ​​associated with edges of the determined flat surface portion 510. Each surface length value of the set of surface length values ​​may correspond to a length of a corresponding edge of the determined flat surface portion 510. In one embodiment, the circuit 202 may be configured to determine the set of surface length values ​​based on extracted coordinate values ​​of the corners 514 of the flat surface portion 510. In another embodiment, the set of surface length values ​​may be received from the CNN 512.

[0071] At 502I, geometric correction values ​​may be received. The circuit 202 may be configured to receive a set of geometric correction values ​​for modification of the resized image frame 516 (or the image frame 508). The circuit 202 may be configured to provide inputs to an artificial neural network (ANN) 518. The inputs may include the determined projection length values ​​(and the determined surface length values).

[0072] The ANN 518 may be a pre-trained neural network that can output a set of geometric correction values ​​for modification of the resized image frame 516 (or image frame 508). In one embodiment, the set of geometric correction values ​​may include four values ​​(α, β, δ, Ω). More details regarding the ANN 518 are shown, for example, in FIG. 9.

[0073] At 502J, an image frame modification operation may be performed. In the image frame modification operation, the circuit 202 may be configured to modify the resized image frame 516 (or image frame 508) based on a set of geometric correction values ​​to generate a modified image frame 520. The modification may be performed to compensate for misalignment of the projection system 102 with respect to the physical surface 504. The modification of the resized image frame 516 (or image frame 508) may also include applying an image warping operation to the resized image frame 516 (or image frame 508) while maintaining an aspect ratio of the resized image frame 516 (or image frame 508).

[0074] At 502K, an image frame projection operation may be performed. In the image frame projection operation, the circuitry 202 may be configured to project a modified image frame 520 onto the determined flat surface portion 510. An image frame 522 projected onto the determined flat surface portion 510 is shown, for example, in FIG. 5C.

[0075] The circuitry 202 can be configured to control the illumination circuitry 210 of the image projection subsystem 104 to project the modified image frame 520 onto the flat surface portion 510. Details regarding the image projection subsystem 104 are shown, for example, in FIG.

[0076] FIG. 6A illustrates an exemplary scenario for depth-based projection of image-based content, according to an embodiment of the present disclosure. The description of FIG. 6A is provided with reference to elements of FIGS. 1, 2, 3A, 3B, 4A, 4B, 5A, 5B, and 5C. FIG. 6A illustrates an exemplary scenario 600 including a room 602 in which a projection system 604 is located. The room can include walls that can provide a physical surface 606 onto which an image frame 508 is projected. FIG. 6A further illustrates a seating structure 608 (such as a sofa or chaise lounge) on which a user associated with the projection system 604 can sit and view the image frame 508. The physical surface 606 can be obscured by object(s), have protrusions, or be hollow in some locations of the wall.

[0077] The circuitry 202 may receive an image frame 508 for projection onto the physical surface 606 at any time after the projection system 102 is operational. The circuitry 202 may control the depth sensor 106 to obtain depth data associated with the physical surface 606. As described in previous figures, the depth data may include a depth map of the physical surface 606.

[0078] The circuitry 202 can classify the physical surface 110 due to the presence of flat surface portions, occluded surface portions, or protrusions on the physical surface 606 into one or more protruding surface portions 610, one or more hollow surface portions 612, one or more occluded surface portions 614, and one or more flat surface portions 616. The circuitry 202 can then control the lighting circuitry 210 to project the image frame 508 onto a first of the one or more planar surface portions 616.

[0079] For classification, the circuit 202 can be configured to determine a first depth value associated with a corner 618 of the physical surface 606. It can be assumed that each corner 618 of the physical surface 606 has the same distance from the projection system 604. After determining the first depth values, the circuit 202 can be configured to generate virtual depth data for the physical surface 606 (also described in 502D of FIG. 5A ) based on the determined first depth values.

[0080] After generating the virtual depth data, the circuit 202 can be configured to compare the acquired depth data with the generated virtual depth data to generate a comparison result. Specifically, the circuit 202 can be configured to compare the depth values ​​associated with the physical surface 606 in the acquired depth data with the depth values ​​associated with the physical surface 606 in the generated virtual depth data. The depth values ​​associated with the one or more protruding surface portions 610 and the one or more occluded surface portions 614 in the acquired depth data can be smaller than the corresponding depth values ​​in the generated virtual depth data. Similarly, the depth values ​​associated with the one or more hollow surface portions 612 in the acquired depth data can be larger than the corresponding depth values ​​in the generated virtual depth data. The depth values ​​associated with the one or more flat surface portions 616 in the acquired depth data can be the same (or approximately equal) to the corresponding depth values ​​in the generated virtual depth data. Details regarding the comparison result are shown, for example, in FIG. 6B.

[0081] Figure 6B illustrates an example surface cluster on the physical surface of Figure 6A, according to an embodiment of the present disclosure. The description of Figure 6B is provided in conjunction with elements of Figures 1, 2, 3A, 3B, 4A, 4B, 5A, 5B, 5C, and 6A. Figure 6B illustrates an example scenario for the physical surface 606 of Figure 6A. The physical surface 606 can be divided into a first set of surface clusters 620, a second set of surface clusters 622, a third set of surface clusters 624, and a fourth set of surface clusters 626.

[0082] Based on the comparison results, the circuit 202 can be configured to apply a clustering method to divide the physical surface 606 into surface clusters. Specifically, the physical surface 606 can be divided into surface clusters based on depth values ​​associated with corresponding surface portions. By way of example and not limitation, a surface cluster can be generated for each of the one or more protruding surface portions 610, the one or more hollow surface portions 612, the one or more occluded surface portions 614, and the one or more flat surface portions 616. For example, a first set of surface clusters 620 can correspond to the one or more protruding surface portions 610. A second set of surface clusters 622 can correspond to the one or more hollow surface portions 612. A third set of surface clusters 624 can correspond to the one or more occluded surface portions 614. Similarly, a fourth set of surface clusters can correspond to the one or more flat surface portions 616.

[0083] In an embodiment, the circuit 202 can be configured to calculate for each cluster a percentage of depth values ​​where the acquired depth data and the generated virtual depth data are the same. The circuit 202 can further compare the calculated percentage to a minimum and / or maximum threshold value to determine a smoothness value for each cluster. The smoothness value for each cluster can indicate the smoothness or flatness of the corresponding surface portion.

[0084] The circuit 202 can classify one or more portions of the physical surface into different categories based on the calculated smoothness value. The different categories can include, but are not limited to, a flat surface category, a non-flat surface category, and a non-recommended surface category for projection. The flat surface category can include flat surface portions. The non-flat surface category can include occluded surface portions, protruding surface portions, and hollow surface portions.

[0085] By way of example and not limitation, if the minimum threshold is 30% and the maximum threshold is 60%, and the calculated percentage is greater than the maximum threshold (e.g., 70%), the corresponding surface cluster may be classified into a flat surface category. If the calculated percentage is between the minimum and maximum thresholds (e.g., 50%), the corresponding surface cluster may be classified into a non-flat surface category. If the calculated percentage is less than the minimum threshold (e.g., 20%), the corresponding surface cluster may be classified into a non-recommended surface category.

[0086] In an embodiment, the circuitry 202 can be configured to control the lighting circuitry 210 to project the image frame 508 onto the flat surface portion. If no flat surface portion is present and if the area of ​​the flat surface portion is below a threshold, the circuitry 202 can output an error (e.g., display a message on a display integrated or coupled to the projection system 102). The error can indicate that the physical surface 606 is not suitable for projecting the image frame 508.

[0087] Figure 6C illustrates an example scenario for the projection of the image frame of Figure 6A according to an embodiment of the present disclosure. The description of Figure 6C is provided with reference to elements of Figures 1, 2, 3A, 3B, 4A, 4B, 5A, 5B, 5C, 6A, and 6B. Figure 6C illustrates an example scenario 600C including an image frame 628 projected onto a first flat surface portion 626A.

[0088] The circuit 202 can be configured to determine a first flat surface portion 626A of the physical surface 606 for projecting the image frame 508 based on the classification of the one or more portions of the physical surface 606. If one or more flat surface portions 616 are present on the physical surface 606, the circuit 202 can be configured to determine a first flat surface portion 626A portion that can be the largest size (or largest area) from among the one or more flat surface portions 616 of the physical surface 606. After determining the first flat surface portion 626A, the circuit 202 can be further configured to control the lighting circuit 210 to project the image frame 508 onto the determined first flat surface portion 626A. For example, FIG. 6C shows a projected image frame 628.

[0089] FIG. 7 illustrates an architecture of a neural network-based model used for depth-based projection of image-based content, according to an embodiment of the present disclosure. FIG. 7 is described with reference to elements of FIG. 1, FIG. 2, FIG. 3A, FIG. 3B, FIG. 4A, FIG. 4B, FIG. 5A, FIG. 5B, FIG. 5C, FIG. 6A, FIG. 6B, and FIG. 6C. FIG. 7 illustrates an exemplary architecture 700. Also illustrated is a projection system 702 that includes a neural network-based model 704. The neural network-based model 704 can include a convolutional neural network (CNN) 706 and an artificial neural network (ANN) 708. FIG. 7 further illustrates an input layer 708A, a hidden layer 708B, and an output layer 708C of the ANN 708.

[0090] The projection system 702 may receive image frames 508 for projection onto the physical surface 504. The projection system 702 may further acquire depth data 506 associated with the physical surface 504. The received image frames 508 and the acquired depth data 506 are provided as inputs to an input layer of the CNN 706 of the neural network based model 704. As described above, the CNN 706 may be trained to determine flat surface portions 510 of the physical surface 504 and to calculate projection length values ​​associated with the flat surface portions 510. The output layer of the CNN 706 may be coupled to an input layer 708A of the ANN 708. The calculated projection length values ​​may then be provided as inputs to the input layer 708A of the ANN 708.

[0091] CNN706 may be a computational network or system in which nodes are artificial neurons arranged in multiple layers. The multiple layers of the neural network may include an input layer, one or more hidden layers, and an output layer. Each layer of the multiple layers may include one or more nodes (or artificial neurons). The output of every node in the input layer may be coupled to at least one node in the hidden layer(s). Similarly, the input of each hidden layer may be coupled to the output of at least one node in the other layers of CNN706. The output of each hidden layer may be coupled to the input of at least one node in the other layers of CNN706. The node(s) in the final layer may receive inputs from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer may be determined from hyperparameters of CNN706. Such hyperparameters may be set before, during, or after training of CNN706 based on a training dataset.

[0092] Each node of CNN 706 may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be adjusted during training of the network. The set of parameters may include, for example, weight parameters and regularization parameters. Each node may use a mathematical function to compute an output based on one or more inputs from nodes in other layer(s) of CNN 706 (e.g., previous layer(s). All or some of the nodes of CNN 706 may correspond to the same or different mathematical functions.

[0093] In training the CNN 706, one or more parameters of each node of the CNN 706 may be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result based on the loss function of the CNN 706. The above process may be repeated for the same or different inputs until a minimum of the loss function is achieved, minimizing the training error. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristic methods.

[0094] The CNN 706 may include electronic data that may be implemented, for example, as a software component of an application executable on the projection system 102. The CNN 706 may rely on libraries, external scripts, or other logic / instructions for execution by a processing device, such as the circuit 202. The CNN 706 may include code and routines configured to enable a processing device, such as the circuit 202, to perform one or more operations for determining the flat surface portion 510. Additionally or alternatively, the CNN 706 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, the CNN 706 may be implemented using a combination of hardware and software.

[0095] ANN 708 may be a computational network or system in which nodes are artificial neurons arranged in multiple layers (represented by circles). The multiple layers of ANN 708 may include an input layer 708A, a hidden layer 708B, and an output layer 708C. Each layer of the multiple layers may include one or more nodes (or artificial neurons, e.g., represented by circles). The output of every node in the input layer 708A may be coupled to at least one node in the hidden layer 708B. Similarly, the input of the hidden layer 708B may be coupled to the output of at least one node in another layer of ANN 708. The output of the hidden layer 708B may be coupled to the input of at least one node in another layer of ANN 708. The node(s) in the output layer 708C may receive inputs from the hidden layer 708B and output results as a set of geometric corrections 710. The number of layers and the number of nodes in each layer may be determined from hyperparameters of ANN 708. Such hyperparameters may be set before, during, or after training of ANN 708 based on a training data set.

[0096] Each node of ANN 708 may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be adjusted during training of ANN 708. The set of parameters may include, for example, weight parameters and regularization parameters. Each node may calculate an output using a mathematical function based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of ANN 708. All or some of the nodes of ANN 708 may correspond to the same or different mathematical functions.

[0097] ANN 708 may include electronic data that may be implemented, for example, as a software component of an application executable on projection system 102. ANN 708 may rely on libraries, external scripts, or other logic / instructions for execution by a processing device, such as circuit 202. ANN 708 may include code and routines configured to enable a computing device, such as circuit 202, to perform one or more operations to generate the set of geometric correction values ​​710. Additionally or alternatively, ANN 708 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, ANN 708 may be implemented using a combination of hardware and software.

[0098] The projection system 702 can receive a set of geometric correction values ​​710 from the output layer 708C of the ANN 708. The projection system 702 can then modify the image frame 508 based on the received set of geometric correction values ​​710 and project the modified image frame 520 onto the flat surface portion 510. Details regarding the modification of the image frame 508 and the projection of the modified image frame 520 onto the flat surface portion 510 are shown, for example, in Figures 5A, 5B, and 5C.

[0099] FIG. 8 is an example block diagram of an image projection subsystem of the projection system of FIG. 1 according to an embodiment of the present disclosure. FIG. 8 is described with reference to elements of FIGS. 1, 2, 3A, 3B, 4A, 4B, 5A, 5B, 5C, 6A, 6B, 6C, and 7. FIG. 8 illustrates a block diagram 800 of the image projection subsystem 104. The image projection subsystem 104 includes an illumination circuit 210, a direct current-to-direct current (DC-DC) converter module 802, one or more liquid crystal on silicon (LCoS) panels 804, a photosensor 806, and a projection lens 808. The image projection subsystem 104 may include other components, which are omitted from this disclosure for the sake of brevity. The illumination circuit 210 may be communicatively coupled to the DC-DC converter module 802, one or more LCoS panels 804, the photosensor 806, and the projection lens 808.

[0100] The DC-DC converter module 802 may include suitable logic, circuitry, or interfaces that may be configured to convert a direct current (DC) source from a first voltage level to a second voltage level. The second voltage level may be lower or higher than the first voltage level. Implementations of the DC-DC converter module 802 may be a switching-based DC-DC converter, a magnetic DC-DC converter, a non-isolated DC-DC converter, a step-down (buck) converter, a step-up (boost) converter, a buck-boost converter, an isolated converter, a flyback converter, and / or a forward converter.

[0101] Each of the one or more LCoS panels 804 may be a miniaturized reflective active-matrix liquid-crystal display that may use a liquid crystal layer on a silicon backplane. Specifically, each of the one or more LCoS panels 804 may have pixels in the liquid crystal layer that may be controlled using electrical currents and semiconductor silicon layers that allow the corresponding LCoS panel to reflect light. Light from the modified image frame 520 may be split into red, blue, and green beams, and each of these beams may be directed onto a separate LCoS panel of the one or more LCoS panels 804. For example, a red light beam may pass through a first LCoS panel of the one or more LCoS panels 804, a green light beam may pass through a second LCoS panel of the one or more LCoS panels 804, and a blue light beam may pass through a third LCoS panel of the one or more LCoS panels 804. The light beams passing through the one or more LCoS panels 804 may generate red, blue and green variants of the modified image frame 520. A prism may be used to combine the red, blue and green variants of the modified image frame 520 from the one or more LCoS panels 804 to generate a final image to be projected onto the physical surface 110. In an embodiment, the one or more LCoS panels 804 may include only one LCoS panel. In such an implementation, the LCoS panel may be based on a single panel architecture.

[0102] The photosensor 806 may include suitable logic, circuitry, and / or interfaces that may be configured to synchronize the series of image frames 116 with the projected image frames. In particular, the photosensor 806 may be configured to determine a synchronization latency of the projection system 102 and further synchronize the series of image frames 116 with the projected image frames based on the determined synchronization latency.

[0103] The projection lens 808 can be configured to magnify and focus the modified image frame 520 for projection onto the physical surface 110. The projection lens 808 can be a zoom projection lens or a fixed focus projection lens. A zoom projection lens can have multiple focal lengths, whereas a fixed focus projection lens can focus the image at a single fixed distance.

[0104] FIG. 9 illustrates an example operation for training an artificial neural network of the neural network-based model of FIG. 7 according to an embodiment of the present disclosure. The description of FIG. 9 is provided with reference to elements of FIG. 1, FIG. 2, FIG. 3A, FIG. 3B, FIG. 4A, FIG. 4B, FIG. 5A, FIG. 5B, FIG. 5C, FIG. 6A, FIG. 6B, FIG. 6C, FIG. 7, and FIG. 8. FIG. 9 illustrates a diagram 900 illustrating example operations 902-904. The example operations illustrated in diagram 900 may begin at 902 and may be performed by any computer system, apparatus, or device, such as the projection system 102 of FIG. 1 or FIG. 2.

[0105] At 902, training data can be input to an artificial neural network (ANN) 708. The training data can be multi-modal data and can be used to train the ANN 708. The training data can include, for example, a set of projection length values ​​902A, a set of surface length values ​​902B, and a set of geometric correction values ​​902C. For example, the training data can relate to generating geometric correction values ​​for various combinations of projection length values ​​and surface length values.

[0106] The training data may include a plurality of training samples. Each training sample of the plurality of training samples may include a first set of values ​​for a first parameter (i.e., a set of projection length values ​​902A), a second set of values ​​for a second parameter (i.e., a set of surface length values ​​902B), and a third set of values ​​for a third parameter (a set of geometric correction values ​​902C). The first parameter may correspond to a projection length. The second parameter may correspond to a surface length. The third parameter may correspond to a geometric correction. An example of training data is shown in Table 1 below. TIFF0007665132000001.tif31139 Table 1: Training data

[0107] Referring to Table 1, the first set of values ​​may include projection length values ​​associated with corners of the physical surface 110 (or the determined flat surface portion 510). The projection length values ​​may correspond to absolute distances between the projection system 102 and the corners of the physical surface 110 (or the determined flat surface portion 510). Thus, the first set of values ​​may include four values. By way of example and not limitation, if the absolute distance between the top left corner and the projection system 102 is "a1", the absolute distance between the top right corner and the projection system 102 is "b1", the absolute distance between the bottom left corner and the projection system 102 is "c1", and the absolute distance between the bottom right corner and the projection system 102 is "d1", then the first set of values ​​(or first projection length values) may be (a1, b1, c1, d1).

[0108] Referring to Table 1, the second set of values ​​may include surface length values ​​associated with the physical surface 110 (or the determined flat surface portion). The surface length values ​​may correspond to absolute distances between corners of the physical surface 110. Thus, the second set of values ​​may include four values. For example, if the absolute distance between the top left corner and the top right corner is "w1", the absolute distance between the bottom left corner and the bottom right corner is "x1", the absolute distance between the top left corner and the bottom left corner is "y1", and the absolute distance between the top right corner and the bottom right corner is "z1", then the second set of values ​​(or the first surface length values) may be (w1, x1, y1, z1).

[0109] Referring to Table 1, the third set of values ​​may include geometric correction values ​​for the modification of the image frame projected onto the physical surface 110 (or the determined flat surface portion). The geometric correction values ​​correspond to the dimensions of the modified first image frame. Thus, the third set of values ​​may also include four values ​​representing each side of the modified first image frame. If the first dimension between the top left corner and the top right corner is "α1", the second dimension between the bottom left corner and the bottom right corner is "β2", the third dimension between the top left corner and the bottom left corner is "δ3", and the fourth dimension between the top right corner and the bottom right corner is "Ω4", then the third set of values ​​(or the first geometric correction values) may be (α1, β1, δ1, Ω1).

[0110] It should be noted that each of the multiple training samples may include a corresponding first set of values, a second set of values, and a third set of values. For example, for the second training sample with serial number 2, the first set of values ​​may be (a2, b2, c2, d2), the second set of values ​​may be (w1, x1, y1, z1), and the third set of values ​​may be (α2, β2, δ2, Ω2). Similarly, for the Nth training sample, the first set of values ​​may be (an, bn, cn, dn), the second set of values ​​may be (wn, xn, yn, zn), and the third set of values ​​may be (αn, βn, δn, Ωn). For the sake of brevity, the description of the second training sample and the Nth training sample is omitted.

[0111] At 904, the ANN 708 can be trained based on the training data. Prior to training, a set of hyper-parameters can be selected based on user input, for example, from a software developer. For example, an initial training rate and random weight values ​​for the ANN 708 can be selected. Also, all layers of the ANN 708 can be initially updated to a fully connected configuration.

[0112] During training, these training data may be passed in sequence as input to the ANN 708. The ANN 708 may be trained to generate a set of geometric correction values ​​as output based on a set of projection length values ​​and a set of surface length values ​​as input. Parameters such as weights of the ANN 708 may be updated based on an optimization algorithm (such as stochastic gradient descent) that minimizes a loss function 906 of the ANN 708. Once trained, the ANN 708 may output predictions that may include a set of trial projection length values ​​and a set of final geometric correction values ​​for the trial set of surface length values. The trial set of projection length values ​​and the trial set of surface length values ​​may not be part of the training data set, but may be part of the test data set.

[0113] Although diagram 900 is shown as discrete operations, such as 902, 904, and 906, in some embodiments such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated depending on the particular embodiment without departing from the essence of the disclosed embodiments.

[0114] Figure 10 is a flow chart illustrating an exemplary method for depth-based projection of image-based content, according to an embodiment of the present disclosure. Figure 10 is described with reference to elements of Figures 1, 2, 3A, 3B, 4A, 4B, 5A, 5B, 5C, 6A, 6B, 6C, 7, 8, and 9. Figure 10 illustrates a flow chart 1000. Operations of the exemplary method may be performed by any computer system, such as, for example, the projection system 102 of Figure 1 or the circuit 202 of Figure 2. Operations of the flow chart 1000 may begin at 1002 and proceed to 1004.

[0115] At 1004, a series of image frames 116 may be received. The series of image frames 116 may be received for projection onto the physical surface 110 of the three dimensional (3D) structure. In at least one embodiment, the circuit 202 may receive the series of image frames 116 to be projected onto the physical surface 110 of the three dimensional (3D) structure. The series of image frames 116 may be received from the server 112 or the memory 204 of the projection system 102. Details regarding receiving the series of image frames 116 are shown, for example, in Figures 1, 4A (402) and 5A (502C).

[0116] At 1006, the depth sensor 106 can be controlled to obtain depth data associated with the physical surface 110. In at least one embodiment, the circuitry 202 can control the depth sensor 106 to obtain depth data associated with the physical surface 110. Details regarding the depth data obtained are shown, for example, in Figures 1, 4A, and 5A.

[0117] At 1008, a first input including the acquired depth data and a first image frame 116A of the received sequence of image frames 116 can be provided to the neural network-based model 108. In at least one embodiment, the circuit 202 can provide a first input that can include the acquired depth data and a first image frame 116A of the received sequence of image frames 116 to the neural network-based model 108. The neural network-based model 108 can include a CNN 706 and an ANN 708. An input layer of the ANN 708 can be coupled to an output layer of the CNN 706. More details regarding the neural network-based model 108 are provided in FIGS. 1, 4A, 4B, 5A, 5B, 5C, 6A, 6B, 6C, 7, and 9.

[0118] At 1010, a set of geometric corrections may be received as a first output from the neural network based model 108 for the first input provided. In at least one embodiment, the circuit 202 may receive the set of geometric corrections as a first output of the neural network based model 108 for the first input provided. Details regarding the set of geometric corrections are provided in FIGS. 1, 4A, 4B, 5A, 5B, 5C, 6A, 6B, 6C, 7, and 9.

[0119] At 1012, the first image frame 116A may be modified based on the received set of geometric correction values. The modification of the first image frame 116A may include applying an image warping operation to the first image frame 116A while maintaining an aspect ratio of the first image frame 116A. In at least one embodiment, the circuit 202 may modify the first image frame based on the received set of geometric correction values. Details regarding the modification of the first image frame 116A are provided in FIGS. 1, 4A, 4B, 5B, and 5C.

[0120] At 1014, the lighting circuitry 210 can be controlled to project the modified first image frame onto the physical surface 110. In at least one embodiment, the circuitry 202 can control the lighting circuitry 210 to project the modified first image frame onto the physical surface 110. More details regarding the image projection subsystem 104 are shown, for example, in FIG. 8. Control can proceed to an end.

[0121] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium having instructions stored thereon executable by a machine and / or computer to operate a projection system (e.g., projection system 102) for depth-based projection of image-based content. The instructions may cause the machine and / or computer to perform operations including receiving a series of image frames (e.g., series of image frames 116) to be projected onto a physical surface (e.g., physical surface 110) of a three-dimensional (3D) structure. The operations may further include controlling a depth sensor (e.g., depth sensor 106) to acquire depth data associated with the physical surface. The operations may further include providing a first input to a neural network-based model (e.g., neural network-based model 108) including the acquired depth data and a first image frame (e.g., first image frame 116A) of the received series of image frames. The operations may further include receiving a set of geometric corrections as a first output of the neural network-based model for the provided first input. The operations may further include modifying the first image frame based on the received set of geometric correction values ​​and controlling an illumination circuit (such as the illumination circuit 210) of an image projection subsystem (such as the image projection subsystem 104) communicatively coupled to the projection system to project the modified first image frame (e.g., the modified first image frame 410) onto the physical surface.

[0122] Some embodiments of the present disclosure can be found in a projection system and method for depth-based projection of image-based content. Various embodiments of the present disclosure can provide a projection system 102 that can include an image projection subsystem 104 and a circuit 202. The image projection subsystem 104 can be coupled to the circuit 202 and can include an illumination circuit 210. The circuit 202 can be configured to receive a sequence of image frames 116 to be projected onto a physical surface 504 of a three-dimensional (3D) structure. The circuit 202 can be further configured to control the depth sensor 106 to obtain depth data 506 related to the physical surface 504. The circuit 202 can be further configured to provide a first input to the neural network-based model 108, the first input including the obtained depth data 506 and an image frame 508 of the received sequence of image frames 116. The circuit 202 can be further configured to receive a set of geometric correction values ​​as a first output of the neural network-based model 108 for the provided first input. The circuitry 202 may be further configured to modify the image frame 508 based on the received set of geometric correction values. Modifying the image frame 508 may include applying an image warping operation to the image frame 508 while maintaining an aspect ratio of the image frame 508. The circuitry 202 may be further configured to control the illumination circuitry 210 of the image projection subsystem 104 to project the modified image frame 520 onto the physical surface 504.

[0123] According to an embodiment, the circuit 202 may be configured to determine a first depth value associated with a corner 514 of the physical surface 504. The circuit 202 may be further configured to generate virtual depth data of the physical surface 504 based on the determined first depth value. The circuit 202 may be further configured to compare the acquired depth data with the generated virtual depth data to generate a comparison result, and to apply a clustering method to the comparison result to divide the physical surface 504 into surface clusters. Each of the surface clusters corresponds to one or more surface portions of the physical surface 504. The circuit 202 may be further configured to calculate a smoothness value for each of the surface clusters, and classify the one or more portions of the physical surface into different categories based on the calculated smoothness values. The circuit 202 may be further configured to determine flat surface portions 510 of the physical surface 504 based on the classification. A modified image frame 520 may be projected onto the determined flat surface portions 510.

[0124] According to an embodiment, the neural network-based model 704 may include a CNN 706, and may initially provide a first input to the CNN 706. The circuit 202 may be configured to input the acquired depth data 506 to the CNN 706. The circuit 202 may receive an output of the CNN 706 for the input depth data 506. The received output includes a clustering result indicative of one or more flat surface portions of the physical surface 504 and one or more non-flat surface portions of the physical surface 504. The circuit 202 may be further configured to determine a maximum size flat surface portion 510 from among the one or more flat surface portions. The flat surface portion may be determined based on the clustering result.

[0125] According to an embodiment, the circuitry 202 may be configured to extract coordinate values ​​associated with corners of the determined flat surface portion 510. The circuitry 202 may be further configured to resize the image frame 508 based on the extracted coordinate values ​​and modify the resized image frame 516 based on the received set of geometric correction values. The modified image frame 516 may be projected onto the determined flat surface portion 510 of the physical surface 504.

[0126] According to an embodiment, the neural network-based model 704 may further include an ANN 708, where an input layer 708A of the ANN 708 may be coupled to an output layer of the CNN 706. The circuit 202 may be configured to determine a set of projection length values ​​associated with the determined corners of the flat surface portion 510 and input the determined set of projection length values ​​to the ANN 708. The ANN 708 generates a first output including a set of geometric correction values ​​based on the input set of projection length values.

[0127] The present disclosure can be realized in software, hardware, or a combination of hardware and software. The present disclosure can be realized in a centralized manner in at least one computer system, or in a distributed manner where different elements can be distributed across several interconnected computer systems. Any computer system or other device adapted to perform the methods described herein can be suitable. The combination of hardware and software can be a general-purpose computer system that includes a computer program that, when loaded and executed, can control a computer system to perform the methods described herein. The present disclosure can be realized in hardware, including parts of an integrated circuit that also perform other functions.

[0128] The present disclosure may also be embodied in a computer program product, which includes all features enabling the implementation of the methods described herein and which is capable of executing these methods when loaded into a computer system. A computer program in this context means any expression, in any language, code or notation, of a set of instructions intended to cause a system capable of processing information to execute a particular function, either directly, or after a) conversion into another language, code or notation, b) reproduction in a different content form, or both.

[0129] While the present disclosure has been described with reference to several embodiments, those skilled in the art will recognize that various modifications can be made and equivalents substituted without departing from the scope of the present disclosure. Many modifications can also be made to adapt a particular situation or material to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is not intended that the present disclosure be limited to the particular embodiments disclosed, but rather, it is intended to include all embodiments falling within the scope of the appended claims. [Explanation of symbols]

[0130] 108 Neural Network-Based Models A series of 116 image frames 116A First Image Frame 116B Second image frame 116N Nth image frame 400A Block Diagram 402A Data Acquisition 402B Geometric correction value reception 402C Image correction 402D Image Projection 404 Depth Data

Claims

1. 1. A projection system comprising: a circuit communicatively coupled to the image projection subsystem including an illumination circuit, the circuit comprising: receiving a sequence of image frames to be projected onto a physical surface of a three-dimensional (3D) structure; controlling a depth sensor to obtain depth data relating to the physical surface; providing a first input including the acquired depth data and a first image frame of the received sequence of image frames to a neural network based model; receiving a set of geometric corrections as a first output of the neural network based model for the provided first input; correcting the first image frame based on the received set of geometric corrections; controlling the lighting circuitry to project the modified first image frame onto the physical surface; It is configured as follows: The circuit comprises: determining a first depth value associated with a corner of the physical surface; generating virtual depth data for the physical surface based on the determined first depth value; comparing the acquired depth data with the generated virtual depth data to generate a comparison result; applying a clustering method to the comparison results to divide the physical surface into surface clusters, each cluster corresponding to one or more surface portions of the physical surface; calculating a smoothness value for each of said surface clusters; classifying one or more portions of the physical surface into different categories based on the calculated smoothness values; determining a flat portion of the physical surface based on the classification; wherein the modified first image frame is projected onto the determined flat surface portion.

1. A projection system comprising:

2. the neural network-based model includes a convolutional neural network, and the first input is initially provided to the convolutional neural network; 10. The projection system of claim 1 .

3. The circuit comprises: inputting the acquired depth data into the convolutional neural network; receiving an output of the convolutional neural network for the input depth data, the output including clustering results indicative of one or more planar surface portions of the physical surface and one or more non-planar surface portions of the physical surface; determining a maximum size flat surface portion from the one or more flat surface portions based on the clustering result; The projection system of claim 2 further configured as follows:

4. The circuit comprises: extracting coordinate values ​​associated with corners of said determined planar surface portion; resizing the first image frame based on the extracted coordinate values; modifying the resized first image frame based on the received set of geometric correction values; wherein the modified first image frame is projected onto the determined flat surface portion of the physical surface.

4. The projection system of claim 3.

5. the modified first image frame is projected onto the determined flat surface portion; 4. The projection system of claim 3.

6. The neural network-based model further comprises an artificial neural network, an input layer of the artificial neural network being coupled to an output layer of the convolutional neural network.

4. The projection system of claim 3.

7. The circuit comprises: determining a set of projected length values ​​associated with corners of the determined flat surface portion; inputting the determined set of projection length values ​​into the artificial neural network; wherein the artificial neural network generates the first output including the set of geometric correction values ​​based on the input set of projection length values.

7. The projection system of claim 6.

8. the modifying the first image frame includes applying an image warping operation to the first image frame while maintaining an aspect ratio of the first image frame.

10. The projection system of claim 1 .

9. further comprising the image projection subsystem including the illumination circuitry.

10. The projection system of claim 1 .

10. 1. A projection system communicatively coupled to an image projection subsystem, comprising: receiving a sequence of image frames to be projected onto a physical surface of a three-dimensional (3D) structure; controlling a depth sensor to obtain depth data associated with the physical surface; providing a first input including the acquired depth data and a first image frame of the received sequence of image frames to a neural network based model; receiving a set of geometric corrections as a first output of the neural network based model for the provided first input; correcting the first image frame based on the received set of geometric corrections; controlling illumination circuitry of the image projection subsystem to project the modified first image frame onto the physical surface; Including, determining a first depth value associated with a corner of the physical surface; generating virtual depth data for the physical surface based on the determined first depth value; comparing the acquired depth data with the generated virtual depth data to generate a comparison result; applying a clustering method to the comparison results to divide the physical surface into surface clusters, each cluster corresponding to one or more surface portions of the physical surface; calculating a smoothness value for each of said surface clusters; classifying one or more portions of the physical surface into different categories based on the calculated smoothness; and determining a flat portion of the physical surface based on the classification; and wherein the modified first image frame is projected onto the determined flat surface portion. A method comprising:

11. the neural network-based model includes a convolutional neural network, and the first input is initially provided to the convolutional neural network; The method of claim 10.

12. inputting the acquired depth data into the convolutional neural network; receiving an output of the convolutional neural network for the input depth data, the output including clustering results indicative of one or more planar surface portions of the physical surface and one or more non-planar surface portions of the physical surface; determining a maximum size flat surface portion from the one or more flat surface portions based on the clustering result; The method of claim 11 further comprising:

13. the modified first image frame is projected onto the determined flat surface portion; The method of claim 12.

14. extracting coordinate values ​​associated with corners of the determined planar surface portion; resizing the first image frame based on the extracted coordinate values; modifying the resized first image frame based on the received set of geometric correction values; projecting the modified first image frame onto the determined planar portion of the physical surface; The method of claim 12 further comprising:

15. The neural network-based model further comprises an artificial neural network, an input layer of the artificial neural network being coupled to an output layer of the convolutional neural network. The method of claim 12.

16. determining a set of projected length values ​​associated with corners of the determined flat surface portion; inputting the determined set of projection length values ​​into the artificial neural network; wherein the artificial neural network generates the first output including the set of geometric correction values ​​based on the input set of projection length values. The method of claim 15.

17. the modifying the first image frame includes applying an image warping operation to the first image frame while maintaining an aspect ratio of the first image frame. The method of claim 10.

18. 1. A non-transitory computer readable storage medium configured to store instructions that, in response to being executed, cause a projection system to: receiving a sequence of image frames to be projected onto a physical surface of a three-dimensional (3D) structure; controlling a depth sensor to obtain depth data associated with the physical surface; providing a first input including the acquired depth data and a first image frame of the received sequence of image frames to a neural network based model; receiving a set of geometric corrections as a first output of the neural network based model for the provided first input; correcting the first image frame based on the received set of geometric corrections; controlling illumination circuitry of an image projection subsystem communicatively coupled to the projection system to project the modified first image frame onto the physical surface; and causing the projection system to perform operations including: determining a first depth value associated with a corner of the physical surface; generating virtual depth data for the physical surface based on the determined first depth value; comparing the acquired depth data with the generated virtual depth data to generate a comparison result; applying a clustering method to the comparison results to divide the physical surface into surface clusters, each cluster corresponding to one or more surface portions of the physical surface; calculating a smoothness value for each of said surface clusters; classifying one or more portions of the physical surface into different categories based on the calculated smoothness; and determining a flat portion of the physical surface based on the classification; and wherein the modified first image frame is projected onto the determined flat surface portion. A non-transitory computer-readable storage medium comprising:

Images