Method and apparatus for real-time interactive performance

Abstract

Methods, apparatus, systems, and articles of manufacture for real-time interactive performance are disclosed. An example apparatus includes at least one memory; a machine readable instruction; and a processor circuit module to instantiate or execute at least one of the machine readable instructions, the method includes capturing an image of a performance area, detecting one or more performers in the performance area using the image, estimating a position of the one or more detected performers, smoothing the estimated position of the one or more detected performers based on a previously estimated position, and outputting the smoothed position of the one or more detected performers. And providing the smoothed estimated position to a display controller circuit module for generating an interaction effect based on the smoothed estimated position.

Description

Technical Field

[0001] The present disclosure relates generally to interactive performances, and more particularly, to methods and apparatus for real-time interactive performances. Background Art

[0002] In a theatrical performance, one or more performers move around a stage or other performance area in front of an audience. Such performances can range in size from small-scale performances involving only a few performers to large-scale performances involving dozens or hundreds of performers. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 It is a representation of large-scale interactive theatrical performance.

[0004] Figure 2 is a close-up representation of an alternative large-scale interactive performance.

[0005] Figure 3 yes Figure 2 A second alternative view of large-scale interactive performance.

[0006] Figure 4 is a block diagram representing an example usage environment of an interactive presentation system for real-time interactive performances.

[0007] Figure 5 Yes means Figure 4 A block diagram of an example implementation of an example position detection circuit module.

[0008] Figure 6 Yes means Figure 4 The example shows a block diagram of an example implementation of a controller circuit module.

[0009] Figure 7 is a representation that can be executed by an example processor circuit module to implement Figure 4 Example machine readable instructions and / or a flow chart of example operations of a display controller.

[0010] Figure 8 is a representation that can be executed by an example processor circuit module to implement Figure 4 A flowchart of example machine-readable instructions and / or example operations of a location detection circuit module.

[0011] Fig. 9 yes Figure 4 A plan view representation of an example system.

[0012] Fig.10 Depicted Figure 4 and / or Figure 5 Example of positioning of site detection circuit module.

[0013] Fig.11 is a diagram of a region of interest (ROI) mask.

[0014] Fig.12 Shows Figure 4 and / or Figure 5 An example position detection circuit module of an example vantage point relative to the stage.

[0015] Fig.13 is the representation of the homography matrix.

[0016] Fig.14 is a representation that can be executed by an example processor circuit module to implement Figure 4 A flowchart of example machine-readable instructions and / or example operations of a location detection circuit module.

[0017] Fig.15 is a representation of an interactive performance that includes person detection information.

[0018] Fig.16 It is a representation of the location detection of a person based on the bounding box locations.

[0019] Fig.17 is a representation that can be executed by an example processor circuit module to implement Figure 4 Example machine readable instructions and / or a flow chart of example operations of a display controller.

[0020] Fig.18 An example ROI definition for a first camera including a first overlap region is shown.

[0021] Fig.19 Shown in conjunction with Fig.18 An example ROI definition showing a first camera adjacent to a second camera, including a second overlap region.

[0022] Fig. 20 is a sample rendering of an interactive performance.

[0023] Fig.21 is a block diagram of an example processing platform including a processor circuit module configured to perform Figure 3 Example machine readable instructions and / or example operations to implement Figure 4 An example location detection circuit module and / or an example display controller circuit module.

[0024] Fig. 22 yes Fig.21 A block diagram of an example implementation of a processor circuit module.

[0025] Fig.23 yes Fig.21A block diagram of another example implementation of a processor circuit module.

[0026] Fig.24 is a block diagram of an example software distribution platform (eg, one or more servers) for distributing software (eg, Figure 7 , 8 , 14, 17) to client devices associated with end users and / or consumers (e.g., for licensing, sale, and / or use), retailers (e.g., for sale, resale, licensing, and / or sublicensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed to, for example, retailers and / or to other end users, such as direct-to-consumer consumers).

[0027] Generally, like reference numerals will be used throughout the drawing(s) and accompanying written description to refer to like or similar parts.The drawings are not to scale.

[0028] As used herein, unless otherwise specified, the term "above" describes the relationship of two parts relative to the earth. A first part is above a second part if the second part has at least one portion between the earth and the first part. Likewise, as used herein, a first part is "below" a second part when the first part is closer to the earth than the second part. As described above, the first part can be above or below the second part with one or more of the following: there are other parts between them, there are no other parts between them, the first part and the second part are in contact, or the first part and the second part are not in direct contact with each other.

[0029] As used in this patent, stating that any component (e.g., a layer, film, region, area, or plate) is on another component in any way (e.g., positioned on another component, located on another component, disposed on another component, or formed on another component, etc.) indicates that the referenced component is in contact with the other component, or that the referenced component is above the other component with one or more intermediate components located therebetween.

[0030] As used herein, connection references (e.g., attached, coupled, connected, and joined) may include intermediate members between elements referenced by the connection reference and / or relative movement between those elements, unless otherwise indicated. Thus, a connection reference does not necessarily imply that two elements are directly connected to each other and / or are connected to each other in a fixed relationship. As used herein, stating that any component is "in contact" with another component is defined to mean that there are no intermediate components between the two components.

[0031] Unless specifically stated otherwise, descriptors such as "first," "second," "third," etc., used herein do not attribute or otherwise indicate any meaning of priority, physical order, arrangement in a list, and / or ordering in any manner, but are merely used as labels and / or arbitrary names to distinguish elements to facilitate understanding of the disclosed examples. In some examples, the descriptor "first" may be used to refer to an element in the detailed description, while a different descriptor such as "second" or "third" may be used to refer to the same element in the claims. In such cases, it should be understood that such descriptors are only used to clearly identify those elements that may share the same name in other cases, for example.

[0032] As used herein, "approximately" and "approximately" modify their subject matter / values ​​to recognize the potential for variations that occur in real-world applications. For example, "approximately" and "approximately" may modify dimensions that may not be exact due to manufacturing tolerances and / or other real-world imperfections as would be understood by one of ordinary skill in the art. For example, "approximately" and "approximately" may indicate that such dimensions may be within a tolerance range of + / - 10%, unless otherwise indicated in the description below. "Substantially real time," as used herein, means occurring in a near instantaneous manner, recognizing that there may be real-world delays in computation time, transmission, etc. Thus, unless otherwise indicated, "substantially real time" means real time + / - 1 second.

[0033] As used herein, the phrase "in communication" (including variations thereof) encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or continuous communication, but additionally includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals and / or one-time events.

[0034] As used herein, a "processor circuit module" is defined as comprising (i) one or more dedicated electronic circuits that are configured to perform (one or more) specific operations and comprising one or more semiconductor-based logic devices (e.g., electronic hardware implemented by one or more transistors), and / or (ii) one or more general-purpose semiconductor-based electronic circuits that can be programmed with instructions to perform specific operations and comprising one or more semiconductor-based logic devices (e.g., electronic hardware implemented by one or more transistors). Examples of processor circuit modules include programmable microprocessors, field programmable gate arrays (FPGAs) that can instantiate instructions, central processor units (CPUs), graphics processor units (GPUs), digital signal processors (DSPs), XPUs, or microcontrollers and integrated circuits such as application-specific integrated circuits (ASICs). For example, an XPU can be implemented by a heterogeneous computing system that includes multiple types of processor circuit modules (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc. and / or combinations thereof) and (one or more) application programming interfaces (APIs) that can assign (one or more) computing tasks to whichever (or which) of the multiple types of processor circuit modules is most suitable for performing the (one or more) computing tasks. DETAILED DESCRIPTION

[0035] Theatrical performances can often benefit from interactivity. For example, as performers move around a stage or performance area, it is desirable for real-time interaction to occur between the performers and the electronic components of the performance area. For example, the stage (e.g., the floor of the performance area) may be able to display images or cause other interactive events. Such displays may be the result of a stage made of a light emitting diode screen, or may additionally or alternatively be the result of projected images (e.g., from an external light source such as a projector).

[0036] In large-scale performances, it is desirable to follow the performers with real-time graphics and / or images. Real-time and accurate character position tracking is necessary for such interactions. One possible method for detecting the character positions is to use a floor screen embedded with pressure sensors. However, such methods generally have extremely high production costs and are therefore undesirable. An alternative method is to use infrared markers or global positioning sensors worn by (one or more) performers. Such methods are also undesirable because they are limited in spatial range.

[0037] The example methods disclosed herein utilize cameras or video information and artificial intelligence to perform character tracking for very large performances and live broadcast scenarios. The example methods disclosed herein use computer vision and artificial intelligence algorithms for multi-person location tracking on large performance stages, achieving great advantages of low cost and ease of use. Although the example methods disclosed herein are described in the context of artistic performances, such methods may be equally applicable to other performances, such as, for example, sporting events. Alternatively, such methods may be additionally or alternatively used in non-performance events, for example, to provide interactive situations (e.g., on the street).

[0038] Figure 1 It is a representation of a large-scale interactive theatrical performance 100. Figure 1 The illustrated example shows a performance titled "Salute to the People" performed at the opening ceremony of the 2022 Winter Olympics in Beijing. In this performance, 24 performers walked around the performance area to "push away" the snow, and gradually revealed the trail behind the performers. The animation effects in this performance were driven by real-time tracking of the performers' positions.

[0039] Figure 2 A close-up representation of an alternative large-scale interactive performance 200. Figure 2 The illustrated example shows the "Snowflake" performance, which was also performed at the opening ceremony of the 2022 Winter Olympics in Beijing. In this performance, more than six hundred performers each held a toy pigeon and performed freely on the stage. Based on the detected positions of the performers, a snowflake effect was displayed on the stage.

[0040] Figure 3 yes Figure 2 A second alternative view 300 of the large-scale interactive performance. Figure 3 In the example shown in , six hundred performers are shown on a stage, each performer having a snow effect displayed near the performer.

[0041] Figure 4 4 is a block diagram of an example use environment for an interactive presentation system 407 for real-time interactive performances. The example use environment 400 includes a performance area 405 and an interactive presentation system. The example interactive presentation system 407 includes a plurality of location detection circuit modules 410, 411, 412, 413 and a display controller circuit module 430. In some examples, the interactive presentation system 407 includes a display circuit module for outputting graphics and / or interactive elements at the direction of the display controller circuit module 430.

[0042] exist Figure 4 In the example shown, the performance area 405 is a stage on which performers can stand. Figure 1 , 2In the examples of 404 and 405, the size of the stage is set to be more than ten thousand square meters. However, such large stage sizes need not be used for all performances. In addition, although the examples disclosed herein are described in the context of stage performances, other types of performances may also be used, such as, for example, sporting events. In some examples, performance area 405 includes a display circuit module that includes, for example, light emitting diodes (LEDs) to enable display of interactive elements. That is, the display circuit module can be a component of performance area 405, as opposed to a component of interactive presentation system 407. In some examples, a projection display circuit module may be used additionally or alternatively to facilitate the display of interactive elements.

[0043] Example location detection circuit modules 410, 411, 412, 413 represent cameras placed around the performance area 405. Each location detection circuit module will have a particular zone of the performance area that it will capture images of. The location detection circuit modules 410, 411, 412, 413 process the captured images to identify the location(s) of the performer(s) in their respective zones of the performance area and provide the location information to the display controller circuit module 430. Figure 4 In the example shown in FIG. 4 , four position detection circuit modules 410, 411, 412, 413 are shown. However, any number of position detection circuit modules 410, 411, 412, 413 may be used additionally or alternatively based on, for example, the layout availability of the cameras to adequately capture images of the entire performance area 405. For example, in a small theater, one or two position detection circuit modules may be used, while in a sports arena, three position detection circuit modules may be used to ensure that all performance areas are captured. In the following Figure 6 An example implementation of an example location detection circuit module is disclosed in .

[0044] The example display controller circuit module 430 receives the location information from the location detection circuit module(s) 410, 411, 412, 413, aggregates the location information, and generates a rendering effect for display. Figure 6 An example implementation of an example display controller circuit module 430 is disclosed.

[0045] exist Figure 4 In the example shown in FIG. 4 , a single display controller 430 is used and four position detection circuit modules 410, 411, 412, 413 are used. In some examples, it may be advantageous to use multiple display controllers and / or multiple position detection circuit modules operating on the same zone of the performance area. Such an approach provides redundancy in the event of a failure.

[0046] Figure 5 Yes means Figure 400 is a block diagram of an example implementation of an example location detection circuit module 500. Figure 5 The example location detection circuit module 500 of the illustrated example includes an image sensor 505, an image acquisition circuit module 510, a calibration data storage 520, a calibration display request circuit module 530, a homography matrix generator circuit module 540, an accuracy tester circuit module 550, a person detector circuit module 560, a location estimator circuit module 580, a smoothing circuit module 590 and a location provider circuit module 595. Figure 5 The location detection circuit module 500 can be instantiated (e.g., created as an instance, generated for any length of time, embodied, implemented, etc.) by a processor circuit module such as a central processing unit that executes instructions. Additionally or alternatively, Figure 5 The location detection circuit module 500 can be instantiated (e.g., create an instance, generate, materialize, implement, etc. for any length of time) by an ASIC or FPGA configured to perform operations corresponding to the instructions. It should be understood that Figure 5 Some or all of the circuit modules of may thus be instantiated at the same or different times. For example, some or all of the circuit modules may be instantiated in one or more threads that execute concurrently on hardware and / or serially on hardware. Furthermore, in some examples, Figure 5 Some or all of the circuit modules may be implemented by a microprocessor circuit module that executes instructions to implement one or more virtual machines and / or containers.

[0047] Figure 5 The example image sensor 505 of the illustrated example is implemented by one or more cameras. In the examples disclosed herein, the camera detects visible light and converts the visible light into an image. In some examples, the image sensor 505 includes one or more lenses that enable the capture of an area of ​​interest of the performance area.

[0048] Figure 5 The example image acquisition circuit module 510 of the illustrated example interacts with the image sensor 505 to acquire images. During the calibration process, the example image acquisition circuit module 510 identifies the location of the camera relative to the performance area. The example image acquisition circuit module 510 then identifies a region of interest in the image of the performance area. The image captured by the image acquisition circuit module 510 is cropped using the ROI mask, resized to an appropriate input size (e.g., 1280×960), and then used by the person detector circuit module 560 during operation to detect the location of a performer. In some examples, the image acquisition circuit module 510 is instantiated by a processor circuit module that executes image acquisition instructions and / or is configured to perform operations such as those performed by Figure 8 The flowcharts represent the operations of those operations.

[0049] In some examples, the site detection circuit module 500 includes components for acquisition. For example, the components for acquisition can be implemented by the example image acquisition circuit module 510. In some examples, the image acquisition circuit module 510 can be implemented by a processor circuit module (such as Fig.21 For example, the image acquisition circuit module 510 can be instantiated by Fig. 22 The image acquisition circuit module 510 may be instantiated by an example microprocessor 2200 that executes machine-executable instructions, such as at least those implemented by blocks 810, 820. In some examples, the image acquisition circuit module 510 may be instantiated by a hardware logic circuit module that may be implemented by an ASIC, an XPU, or a processor configured to perform operations corresponding to the machine-readable instructions. Fig.23 FPGA circuit module 2300. Additionally or alternatively, the image acquisition circuit module 510 can be instantiated by any other combination of hardware, software and / or firmware. For example, the image acquisition circuit module 510 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGA, ASIC, XPU, comparator, operational amplifier (op-amp), logic circuit, etc.), which are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0050] Figure 5 The example calibration data repository 520 of the illustrated example stores calibration information including, for example, the location of the position detection circuit module 410, a homography matrix, previously detected positions of performers, and the like. Figure 5 The example calibration data store 520 of the illustrated example is implemented by any memory, storage device, and / or storage disk for storing data, such as, for example, flash memory, magnetic media, optical media, solid-state memory, (one or more) hard drives, (one or more) thumb drives, etc. In addition, the data stored in the example calibration data store 520 can be in any data format, such as, for example, binary data, comma-delimited data, tab-delimited data, structured query language (SQL) structures, etc. Although in the illustrated example, the calibration data store 520 is shown as a single device, the example calibration data store 520 and / or any other data storage device described herein can be implemented by any number and / or type of memory.

[0051] Figure 5The example calibration display request circuit module 530 of the illustrated example transmits a request to display a calibration pattern to the display controller circuit module 430. The example display controller 430 causes the calibration pattern to be displayed on the performance area, which may then be captured by the image acquisition circuit module 510 for use in the calibration process.

[0052] In some examples, the calibration display request circuit module 530 is instantiated by a processor circuit module that executes calibration display request instructions and / or is configured to perform operations such as by Figure 8 The flowcharts represent the operations of those operations.

[0053] In some examples, the location detection circuit module 500 includes a component for requesting. For example, the component for requesting can be implemented by the example calibration display request circuit module 530. In some examples, the calibration display request circuit module 530 can be implemented by a processor circuit module (such as Fig.21 For example, the calibration display request circuit module 530 may be instantiated by executing machine executable instructions (such as those implemented by at least block 830). Fig. 22 In some examples, the calibration display request circuit module 530 may be instantiated by a hardware logic circuit module, which may be an ASIC, an XPU, or a processor configured to perform operations corresponding to machine-readable instructions. Fig.23 The calibration display request circuit module 530 may be implemented by any other combination of hardware, software, and / or firmware. Additionally or alternatively, the calibration display request circuit module 530 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the calibration display request circuit module 530 may be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0054] Figure 5 The example homography matrix generator circuit module 540 of the illustrated example generates a homography matrix that enables the detection locations of points in the image captured by the image capture circuit module 510 to be transformed into points in the physical space on the performance area 405. (Block 840). In some examples, the homography matrix generator circuit module 540 is instantiated by a processor circuit module that executes homography matrix generator instructions and / or is configured to perform operations such as by Figure 8 The flowcharts represent the operations of those operations.

[0055] In some examples, the location detection circuit module 500 includes components for generating. For example, the components for generating can be implemented by an example homography matrix generator circuit module 540. In some examples, the homography matrix generator circuit module 540 can be implemented by a processor circuit module (such as Fig.21 For example, the homography matrix generator circuit module 540 may be instantiated by executing machine executable instructions (such as those implemented by at least block 840). Fig. 22 In some examples, the homography matrix generator circuit module 540 may be instantiated by a hardware logic circuit module, which may be an ASIC, an XPU, or a processor configured to perform operations corresponding to machine-readable instructions. Fig.23 FPGA circuit module 2300. Additionally or alternatively, the homography matrix generator circuit module 540 can be instantiated by any other combination of hardware, software and / or firmware. For example, the homography matrix generator circuit module 540 can be implemented by at least one or more hardware circuit modules (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGA, ASIC, XPU, comparator, operational amplifier (op-amp), logic circuit, etc.), and the hardware circuit module is configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are also suitable.

[0056] Figure 5 The example accuracy tester circuit module 550 of the illustrated example tests the transformation of the second location detected using the homography matrix. The test is performed by attempting to transform the known location of the performer located on the performance area 405 and detecting the accuracy of the (one or more) transformations. If the accuracy tester circuit module 550 determines that the accuracy of the homography matrix is ​​insufficient (e.g., the location detection accuracy is inaccurate for at least a threshold number of locations (e.g., five locations) within a threshold distance (e.g., one tenth of a meter), the homography matrix generator circuit module 540 further improves the generated homography matrix. The example process is repeated until the accuracy of the homography matrix is ​​sufficient, at which point the example image acquisition circuit module 510 stores the homography matrix in the calibration data store 520.

[0057] In some examples, the accuracy tester circuit module 550 is instantiated by a processor circuit module that executes accuracy tester instructions and / or is configured to perform operations such as those performed by Figure 8The flowcharts represent the operations of those operations.

[0058] In some examples, the location detection circuit module 500 includes components for testing. For example, the components for testing can be implemented by an example accuracy tester circuit module 550. In some examples, the accuracy tester circuit module 550 can be implemented by a processor circuit module (such as Fig.21 For example, the accuracy tester circuit module 550 may be instantiated by executing machine executable instructions (such as those implemented by at least blocks 850, 860). Fig. 22 In some examples, the accuracy tester circuit module 550 may be instantiated by a hardware logic circuit module, which may be an ASIC, an XPU, or a processor configured to perform operations corresponding to machine-readable instructions. Fig.23 FPGA circuit module 2300. Additionally or alternatively, the accuracy tester circuit module 5500 can be instantiated by any other combination of hardware, software and / or firmware. For example, the accuracy tester circuit module 550 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGA, ASIC, XPU, comparator, operational amplifier (op-amp), logic circuit, etc.), which are configured to execute some or all of the machine-readable instructions and / or perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are also suitable.

[0059] Figure 5The example person detector circuit module 560 of the illustrated example detects one or more performers in the captured image. In the examples disclosed herein, the trained artificial intelligence model is executed by the person detector circuit module 560 to detect the performers. In the examples disclosed herein, the AI ​​model is trained using YouOnlyLookOnceX (YOLOX). However, other person or object detection methods may be used in addition or alternatively. In the examples disclosed herein, images are captured, annotated, and used for training of the AI ​​model. In some examples, convolutional neural network (CNN) training methods (such as data enhancement and hyperparameter adjustment) are used to achieve high detection rates while avoiding overfitting on the training data set. In some examples, the AI ​​model is modified, including, for example, pruning some feature layers, replacing the base convolutional layer with a depth-wise separated layer, converting the trained floating point 32 (float32) Pytorch model to an optimized integer 8 (int8) Openvino model, etc. As a result, when executing the model, the person detector circuit module 560 spends approximately 12ms for the person detection process, including post-processing of camera image cropping and resizing, model inference, and candidate selection. As an output of the person detection process performed by the person detector circuit module 560, a bounding box is created for each detected performer.

[0060] In some examples, person detector circuit module 560 is instantiated by a processor circuit module that executes person detector instructions and / or is configured to perform operations such as those performed by Fig.14 The flowcharts represent the operations of those operations.

[0061] In some examples, the location detection circuit module 500 includes components for detection. For example, the components for detection can be implemented by an example person detector circuit module 560. In some examples, the person detector circuit module 560 can be implemented by a processor circuit module (such as Fig.21 For example, the example person detector circuit module 560 may be instantiated by executing machine executable instructions (such as those implemented by at least block 1420). Fig. 22 In some examples, the person detector circuit module 560 may be instantiated by a hardware logic circuit module, which may be an ASIC, an XPU, or a processor configured to perform operations corresponding to machine-readable instructions. Fig.23FPGA circuit module 2300. Additionally or alternatively, the person detector circuit module 560 may be instantiated by any other combination of hardware, software and / or firmware. For example, the person detector circuit module 560 may be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGA, ASIC, XPU, comparator, operational amplifier (op-amp), logic circuit, etc.), which are configured to execute some or all of the machine-readable instructions and / or perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0062] Figure 5 The example location estimator circuit module 580 of the illustrated example estimates the location of the performer based on a bounding box corresponding to the detected performer. In order to estimate the actual location of the person on the ground plane, the pixel location in the camera image where the performer's feet touch the ground is identified. If the location of the performer's feet can be identified, then that point can be used as the location of the performer within the image. Alternatively, if the location of the performer's feet cannot be identified, a point representing the bottom center of the bounding box is used as the performer's location within the image. In practice, the difference between the two possible points is typically low, with an average value of less than three tenths of a meter, which is generally acceptable in real-world applications. The example location estimator circuit module 580 then converts the performer's location in the image into a location in physical space on the performance area 405 using the homography matrix stored in the calibration data store 520.

[0063] In some examples, the location estimator circuit module 580 is instantiated by a processor circuit module that executes location estimator instructions and / or is configured to perform operations such as those performed by Fig.14 The flowcharts represent the operations of those operations.

[0064] In some examples, the location detection circuit module 500 includes components for estimating. For example, the components for estimating can be implemented by an example location estimator circuit module 580. In some examples, the location estimator circuit module 580 can be implemented by a processor circuit module (such as Fig.21 For example, the location estimator circuit module 580 may be instantiated by executing machine executable instructions (such as those implemented by at least block 1440). Fig. 22 In some examples, the location estimator circuit module 580 may be instantiated by a hardware logic circuit module, which may be implemented by an ASIC, XPU, or processor configured to perform operations corresponding to machine-readable instructions. Fig.23 The FPGA circuit module 2300 of the present invention is implemented. Additionally or alternatively, the site estimator circuit module 580 can be instantiated by any other combination of hardware, software and / or firmware. For example, the site estimator circuit module 580 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to execute some or all of the machine-readable instructions and / or perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are also suitable.

[0065] Figure 5 The example smoothing circuit module 590 of the illustrated example smoothes the performer's compensation position. Smoothing of the performer's position helps reduce jitter in the detected position of the performer. In the examples disclosed herein, the smoothing circuit module 590 implements a one euro filter. However, any other type of filter may additionally or alternatively be used, such as, for example, a low pass filter. In some examples, the smoothing circuit module 590 is instantiated by a processor circuit module that executes smoothing instructions and / or is configured to execute instructions such as those provided by Fig.14 The flowcharts represent the operations of those operations.

[0066] In some examples, the location detection circuit module 500 includes components for smoothing. For example, the components for smoothing can be implemented by an example smoothing circuit module 590. In some examples, the smoothing circuit module 590 can be implemented by a processor circuit module (such as Fig.21 For example, the smoothing circuit module 590 may be instantiated by executing machine executable instructions (such as those implemented by at least block 1460). Fig. 22 In some examples, the smoothing circuit module 590 may be instantiated by a hardware logic circuit module, which may be an ASIC, an XPU, or a processor configured to perform operations corresponding to machine-readable instructions. Fig.23In addition or alternatively, the smoothing circuit module 590 may be instantiated by any other combination of hardware, software and / or firmware. For example, the smoothing circuit module 590 may be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0067] Figure 5 The example location provider circuit module 595 of the illustrated example provides the smooth location to the display controller circuit module 430. In some examples, the location provider circuit module 595 is instantiated by a processor circuit module that executes location provider instructions and / or is configured to perform operations such as by Fig.14 The flowcharts represent the operations of those operations.

[0068] In some examples, the location detection circuit module 500 includes components for providing. For example, the components for providing can be implemented by an example location provider circuit module 595. In some examples, the location provider circuit module 595 can be implemented by a processor circuit module (such as Fig.21 For example, the location provider circuit module 595 may be instantiated by executing machine executable instructions (such as those implemented by at least block 1470). Fig. 22 In some examples, the location estimator circuit module 595 may be instantiated by a hardware logic circuit module, which may be implemented by an ASIC, XPU, or processor configured to perform operations corresponding to machine-readable instructions. Fig.23 FPGA circuit module 2300. Additionally or alternatively, the site provider circuit module 595 can be instantiated by any other combination of hardware, software and / or firmware. For example, the site provider module 595 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGA, ASIC, XPU, comparator, operational amplifier (op-amp), logic circuit, etc.), which are configured to execute some or all of the machine-readable instructions and / or perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are also suitable.

[0069] Although in Figure 5 An example method of implementing the site detection circuit module 500 is shown in FIG. Figure 5 One or more of the elements, processes and / or devices shown in the drawings may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. Figure 5 The example image acquisition circuit module 510, the example calibration display request circuit module 530, the example homography matrix generator circuit module 540, the example accuracy tester circuit module 550, the example person detector circuit module 560, the example location estimator circuit module 580, the example smoothing circuit module 590, the example location provider circuit module 595 and / or more generally, the example location detection circuit module 500 can be implemented by hardware alone or by hardware combined with software and / or firmware. Thus, for example, Figure 5 Any of the example image acquisition circuit module 510, the example calibration display request circuit module 530, the example homography matrix generator circuit module 540, the example accuracy tester circuit module 550, the example person detector circuit module 560, the example location estimator circuit module 580, the example smoothing circuit module 590, the example location provider circuit module 595 and / or more generally the example location detection circuit module 500 can be implemented by a processor circuit module, (one or more) analog circuits, (one or more) digital circuits, (one or more) logic circuits, (one or more) programmable processors, (one or more) programmable microcontrollers, (one or more) graphics processing units (GPUs), (one or more) digital signal processors (DSPs), (one or more) application specific integrated circuits (ASICs), (one or more) programmable logic devices (PLDs) and / or (one or more) field programmable logic devices (FPLDs) (such as field programmable gate arrays (FPGAs)). In addition, the example location detection circuit module 500 may include in addition to or instead of Figure 5 One or more elements, processes and / or devices other than those shown, and / or may include more than one of any or all of the elements, processes and devices shown.

[0070] Figure 6 Yes means Figure 4 0048 is a block diagram of an example implementation of the controller circuit module 430. Figure 6 The display controller circuit module 430 may be instantiated (e.g., created as an instance, generated for any length of time, materialized, realized, etc.) by a processor circuit module such as a central processing unit executing instructions. Additionally or alternatively, Figure 6The display controller circuit module 430 may be instantiated (e.g., created, generated for any length of time, embodied, implemented, etc.) by an ASIC or FPGA configured to perform operations corresponding to the instructions. It should be understood that Figure 6 Some or all of the circuit modules of may thus be instantiated at the same or different times. For example, some or all of the circuit modules may be instantiated in one or more threads that are executed concurrently on the hardware and / or serially on the hardware. Furthermore, in some examples, Figure 6 Some or all of the circuit modules may be implemented by a microprocessor circuit module that executes instructions to implement one or more virtual machines and / or containers. Figure 6 The example display controller circuit module 430 of the illustrated example includes a site receiver circuit module 610 , an aggregation circuit module 620 , a redundancy checker circuit module 630 , an effect generation circuit module 640 , and an effect outputter circuit module 650 .

[0071] Figure 6 The example location receiver circuit module 610 of the illustrated example obtains location information from the location detection circuit module. In some examples, the location information is received via a network (such as a local area network). However, the location information may be received via any (one or more) other communication technologies. In some examples, the location receiver circuit module 610 is instantiated by a processor circuit module that executes location receiver instructions and / or is configured to perform operations such as those performed by Fig.17 The flowcharts represent the operations of those operations.

[0072] In some examples, the display controller circuit module 430 includes components for accessing. For example, the components for accessing can be implemented by the example location receiver circuit module 610. In some examples, the location receiver circuit module 610 can be implemented by a processor circuit module (such as Fig.21 For example, the location receiver circuit module 610 may be instantiated by executing machine executable instructions (such as those implemented by at least block 1710). Fig. 22 In some examples, the location receiver circuit module 610 may be instantiated by a hardware logic circuit module, which may be an ASIC, XPU, or other processor configured to perform operations corresponding to machine-readable instructions. Fig.23The FPGA circuit module 2300 of the present invention is implemented. Additionally or alternatively, the location receiver circuit module 610 can be instantiated by any other combination of hardware, software and / or firmware. For example, the location receiver circuit module 610 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to execute some or all of the machine-readable instructions and / or perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are also suitable.

[0073] Figure 6 The example aggregation circuit module 620 of the illustrated example aggregates the location information received from the various location detection circuit modules (received via the location receiver circuit module 610). In some examples, the aggregation circuit module 620 is instantiated by a processor circuit module that executes aggregation instructions and / or is configured to perform operations such as by Fig.17 The flowcharts represent the operations of those operations.

[0074] In some examples, the display controller circuit module 430 includes components for aggregation. For example, the components for aggregation can be implemented by the aggregation circuit module 620. In some examples, the aggregation circuit module 620 can be implemented by a processor circuit module (such as Fig.21 For example, the aggregation circuit module 620 may be instantiated by executing machine executable instructions (such as those implemented by at least block 1720). Fig. 22 In some examples, the aggregation circuit module 620 may be instantiated by a hardware logic circuit module, which may be an ASIC, an XPU, or a processor configured to perform operations corresponding to machine-readable instructions. Fig.23 The FPGA circuit module 2300 is implemented. Additionally or alternatively, the aggregation circuit module 620 can be instantiated by any other combination of hardware, software and / or firmware. For example, the aggregation circuit module 620 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to execute some or all of the machine-readable instructions and / or perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0075] Figure 6The example redundancy checker circuit module 630 of the illustrated example identifies sites within the overlap region(s) of aggregated site information to eliminate any redundantly identified performers. When a performer stands in the physical space represented by the overlap region, there may be two detection results for the same performer. In the examples disclosed herein, four site detection circuit modules are used to identify performers in their own corresponding ROIs. The example redundancy checker circuit module 630 enables redundancy checks to be performed to, for example, identify whether two sites have been identified for the same performer.

[0076] For example, if a performer is identified by two site detection circuit modules responsible for adjacent zones of a performance area, one of the site identifications should be omitted to avoid multiple interactive elements being displayed for the performer and / or other rendering irregularities. The example redundancy checker circuit module 630 determines whether any two sites are within a threshold distance from each other. In the example disclosed herein, the threshold distance is three tenths of a meter. If two sites are detected within a threshold distance from each other, the redundancy checker circuit module 630 removes one of the sites. That is, two identifications for the same performer are reduced to a single identification for a single performer. In some examples, a site from one of the site detection circuit modules is superior to another. That is, when two sites are detected within a threshold distance from each other, the site originating from the first of the site detection circuit modules is retained, while the site originating from the second of the site detection circuit modules is deleted.

[0077] In some examples, the redundancy checker circuit module 630 is instantiated by a processor circuit module that executes redundancy checker instructions and / or is configured to perform operations such as those performed by Fig.17 The flowcharts represent the operations of those operations.

[0078] In some examples, the display controller circuit module 430 includes a component for checking. For example, the component for checking can be implemented by a redundancy checker circuit module 630. In some examples, the redundancy checker circuit module 630 can be implemented by a processor circuit module (such as Fig.21 For example, the redundancy checker circuit module 630 may be instantiated by executing machine executable instructions (such as those implemented by at least blocks 1730, 1740, 1750). Fig. 22 In some examples, the redundancy checker circuit module 630 may be instantiated by a hardware logic circuit module, which may be an ASIC, an XPU, or a processor configured to perform operations corresponding to machine-readable instructions. Fig.23The redundancy checker circuit module 630 may be implemented by any other combination of hardware, software, and / or firmware. Additionally or alternatively, the redundancy checker circuit module 630 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the redundancy checker circuit module 630 may be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0079] Figure 6 The example effect generation circuit module 640 of the illustrated example generates a rendering effect based on the detected position of the performer. In some examples, the effect is generated using a rendering engine (such as, for example, Unity or Unreal). However, any other rendering engine and / or technology for generating interactive effects may be used in addition or alternatively. In some examples, the effect generation circuit module 640 is instantiated by a processor circuit module that executes effect generation instructions and / or is configured to perform operations such as by Fig.17 The flowcharts represent the operations of those operations.

[0080] In some examples, the display controller circuit module 430 includes components for creating. For example, the components for creating can be implemented by the effect generation circuit module 640. In some examples, the effect generation circuit module 640 can be implemented by a processor circuit module (such as Fig.21 For example, the effect generation circuit module 640 may be instantiated by executing machine executable instructions (such as those implemented by at least block 1760). Fig. 22 In some examples, the effect generation circuit module 640 may be instantiated by a hardware logic circuit module, which may be an ASIC, an XPU, or a processor configured to perform operations corresponding to machine-readable instructions. Fig.23The effect generation circuit module 640 may be implemented by any other combination of hardware, software, and / or firmware. Additionally or alternatively, the effect generation circuit module 640 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the effect generation circuit module 640 may be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0081] Figure 6 The example effect output circuit module 650 of the illustrated example causes the display of the generated (one or more) effects. In some examples, the effect output circuit module 650 is implemented by a display driver. However, any other method of implementing the effect output circuit module 650 may be used in addition or alternatively.

[0082] In some examples, the effect outputter circuit module 650 is instantiated by a processor circuit module that executes effect outputter instructions and / or is configured to perform operations such as those performed by Fig.17 The flowcharts represent the operations of those operations.

[0083] In some examples, the display controller circuit module 430 includes components for output. For example, the components for output can be implemented by the effect output circuit module 650. In some examples, the effect output circuit module 650 can be implemented by a processor circuit module (such as Fig.21 For example, the effect output circuit module 650 may be instantiated by executing machine executable instructions (such as those implemented by at least block 1760). Fig. 22 In some examples, the effect output circuit module 650 may be instantiated by a hardware logic circuit module, which may be an ASIC, an XPU, or a processor configured to perform operations corresponding to machine-readable instructions. Fig.23The effect output circuit module 650 may be implemented by any other combination of hardware, software and / or firmware. For example, the effect output circuit module 650 may be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0084] Although in Figure 6 An example method of implementing the display controller circuit module 430 is shown in FIG. Figure 6 One or more of the elements, processes and / or devices shown in the drawings may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. Figure 6 The example site receiver circuit module 610, the example aggregation circuit module 620, the example redundancy checker circuit module 630, the example effect generation circuit module 640, the example effect output circuit module 650 and / or more generally, the example display controller circuit module 430 can be implemented by hardware alone or by hardware combined with software and / or firmware. Thus, for example, Figure 6 Any of the example site receiver circuit module 610, the example aggregation circuit module 620, the example redundancy checker circuit module 630, the example effect generation circuit module 640, the example effect outputter circuit module 650, and / or the more general example display controller circuit module 430 may be implemented by a processor circuit module, (one or more) analog circuits, (one or more) digital circuits, (one or more) logic circuits, (one or more) programmable processors, (one or more) programmable microcontrollers, (one or more) graphics processing units (GPUs), (one or more) digital signal processors (DSPs), (one or more) application specific integrated circuits (ASICs), (one or more) programmable logic devices (PLDs), and / or (one or more) field programmable logic devices (FPLDs) (such as field programmable gate arrays (FPGAs)). In addition, Figure 6 The example display controller circuit module 430 may include in addition to or in place of Figure 6 One or more elements, processes and / or devices other than those shown, and / or may include more than one of any or all of the elements, processes and devices shown.

[0085] Figure 7 , Figure 8 , Fig.15 and Fig.17 A flowchart representing example machine readable instructions that may be executed to configure a processor circuit module to implement Figure 4 The machine-readable instructions may be provided to the processor circuit module (such as the following in combination with Fig.21 The processor circuit module 2112 shown in the example processor platform 2100 and / or the following combination Fig. 22 and / or Fig.23 The program may be implemented in software, which is stored on one or more non-transitory computer-readable storage media such as a compact disk (CD), a floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a digital versatile disk (DVD), a Blu-ray disk, a volatile memory (e.g., any type of random access memory (RAM), etc.) or a non-volatile memory (e.g., an electrically erasable programmable read-only memory (EEPROM), FLASH memory, HDD, SSD, etc.) associated with a processor circuit module located in one or more hardware devices, but the complete program and / or its portion may alternatively be executed by one or more hardware devices other than the processor circuit module and / or implemented in firmware or dedicated hardware. The machine-readable instructions may be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, the client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that may facilitate communications between a server and an endpoint client hardware device). Similarly, a non-transitory computer-readable storage medium may include one or more media located in one or more hardware devices. In addition, although reference is made to Figure 7 , Figure 8 , Fig.15 and Fig.17The flowchart shown in describes an example program, but many other methods of implementing the example site detection circuit module 410 and / or the example display controller circuit module 430 may alternatively be used. For example, the execution order of the boxes may be changed, and / or some of the boxes described may be changed, eliminated, or combined. Additionally or alternatively, any one or all of the boxes may be implemented by one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to perform corresponding operations without executing software or firmware. The processor circuit modules may be distributed in different network locations and / or local to one or more hardware devices (e.g., single-core processors (e.g., single-core central processing units (CPUs)), multi-core processors in a single machine (e.g., multi-core CPUs, XPUs, etc.), multiple processors distributed across multiple servers of a server rack, multiple processors distributed across one or more server racks, CPUs and / or FPGAs located in the same package (e.g., the same integrated circuit (IC) package) or in two or more separate housings, etc.).

[0086] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions as described herein may be stored as data or data structures (e.g., as part of instructions, code, code representations, etc.) that can be used to create, manufacture, and / or produce machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located in the same or different locations of a network or collection of networks (e.g., in the cloud, in an edge device, etc.). The machine-readable instructions may require one or more of installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, reassignment, compilation, etc., so that they can be directly read, interpreted, and / or executed by a computing device and / or other machine. For example, machine-readable instructions may be stored in multiple parts that are separately compressed, encrypted, and / or stored on separate computing devices, where the parts, when decrypted, decompressed, and / or combined, form a set of machine-executable instructions that implement one or more operations that together may form a program (such as the programs described herein).

[0087] In another example, the machine-readable instructions may be stored in a state in which they are readable by a processor circuit module, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., in order to execute the machine-readable instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., settings are stored, data are entered, network addresses are recorded, etc.) before the machine-readable instructions and / or corresponding program(s) can be executed in whole or in part. Therefore, as used herein, a machine-readable medium may contain machine-readable instructions and / or program(s) regardless of the particular format or state of the machine-readable instructions and / or program(s) when stored or otherwise at rest or in transit.

[0088] The machine-readable instructions described herein may be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, the machine-readable instructions may be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0089] As mentioned above, Figure 7 , 8The example operations of 14 and / or 17 can be implemented using executable instructions (e.g., computer and / or machine-readable instructions) stored on one or more non-transitory computer and / or machine-readable media, such as optical storage devices, magnetic storage devices, HDDs, flash memories, read-only memories (ROMs), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage devices or storage disks, wherein information is stored for any duration (e.g., for an extended period of time, permanently, for transient instances, for temporary buffering, and / or for caching of information). As used herein, the terms non-transitory computer-readable media, non-transitory computer-readable storage media, non-transitory machine-readable media, and non-transitory machine-readable storage media are explicitly defined to include any type of computer-readable storage device and / or storage disk, and do not include propagation signals and do not include transmission media. As used herein, the terms "computer-readable storage device" and "machine-readable storage device" are defined to include any physical (mechanical and / or electronic) structure that stores information, but do not include propagation signals and do not include transmission media. Examples of computer-readable storage devices and machine-readable storage devices include any type of random access memory, any type of read-only memory, solid-state memory, flash memory, optical disks, magnetic disks, disk drives, and / or redundant array of independent disks (RAID) systems. As used herein, the term "device" refers to a physical structure, such as a mechanical and / or electrical device, hardware, and / or circuit module, which may or may not be configured by computer-readable instructions, machine-readable instructions, etc. and / or manufactured to execute computer-readable instructions, machine-readable instructions, etc.

[0090] "Including" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim adopts any form of "includes" or "comprising" (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or in any type of claim recitation, it is to be understood that additional elements, terms, etc. may be present without exceeding the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transitional term, such as in the preamble of a claim, it is open-ended in the same manner as the terms "includes" and "comprising" are open-ended. When used in a form such as A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, and C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, or (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in the context of describing the performance or execution of processes, instructions, acts, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing the implementation or execution of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" is intended to refer to an implementation that includes any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0091] As used herein, singular references (e.g., "a," "an," "first," "second," etc.) do not exclude a plurality. As used herein, the term "a" (a or an) object refers to one or more of the object. The terms "a" (a or an), "one or more," and "at least one" are used interchangeably herein. In addition, although listed separately, multiple components, elements, or method actions may be implemented by, for example, the same entity or object. In addition, although individual features may be included in different examples or claims, these features may be combined, and inclusion in different examples or claims does not imply that a combination of features is not feasible and / or disadvantageous.

[0092] Figure 7 is a representation that can be executed by an example processor circuit module to implement Figure 4 The interactive presentation system 407 is a flowchart of example machine-readable instructions and / or example operations for creating an interactive performance. Figure 7 The machine readable instructions and / or operations 700 of FIG. 70 begin at block 710 where each location detection circuit module 500 performs a calibration process. (Block 710). Figure 7 In the illustrated example of , the calibration process is performed once, for example, during the setup of the interactive presentation system 407. However, in some examples, the calibration process may be re-performed periodically (e.g., daily) and / or aperiodically (e.g., before a performance event). Calibration of the position detection circuit module 500 ensures that the position detection circuit module 500 accurately provides information about the position of the performer to the display controller 430. Figure 7 In the illustrated example of FIG. 5 , multiple iterations of block 710 represent performing a calibration process for each location detection circuit module 500. In some examples, the calibration process is performed serially for each of the location detection circuit modules. However, performing such calibration in parallel may also be useful. Figure 8 Provide further details of the calibration process.

[0093] After calibration, the example position detection circuit module 500 detects the position of the performer and reports the position(s) to the display controller circuit module 430. (Block 720). Such detection of the performer is based on the calibration process performed at block 710. Figure 7 In the illustrated example of FIG, multiple iterations of block 720 represent performing the site identification process for each site detection circuit module 500. Although the calibration process can be performed serially for each of the site detection circuit modules, the site identification process is ideally performed in parallel by each of the site detection circuit modules. Fig.14 Provides further details of the site detection process.

[0094] The example display controller circuit module 430 receives location information from each of the location detection circuit modules and generates one or more effects based on the identified locations. (Block 730). In some examples, the display controller circuit module 430 aggregates the location information from each of the location detection circuit modules to create an understanding of the location (one or more) of each of the performers in the performance area 405. Fig.17 Further details of the effect generation process are provided. The process of blocks 720 and 730 is then repeated to continue the interactive performance. Such a process may be terminated, for example, at the request of a user (e.g., the director of the performance) and / or at the end of the definition of the performance.

[0095] Figure 8 is a representation that can be executed by an example processor circuit module to implement Figure 4 A flowchart of example machine readable instructions and / or example operations for calibrating a location detection circuit module of the device. Figure 8 The machine readable instructions and / or operations 710 of begin at block 810, where the image acquisition circuit module 510 identifies the location of the camera relative to the performance area. (Block 810).

[0096] Fig. 9 yes Figure 4 A plan view of an example system is shown in 900. Fig. 9 In the example shown in FIG. 4 , each location detection circuit module 410, 411, 412, 413 covers a corresponding zone. The location of the location detection circuit module 410 is known (eg, fixed), such as Fig.10 As shown, it depicts Figure 4 and / or the positioning of the example site detection circuit module of 5. For example, Fig.10 The position of the position detection circuit module 1010 (corresponding to the position detection circuit module 410) relative to a point on the stage (eg, the center point of the stage) is known. Fig.10 In the illustrated example of FIG. 1 , an auxiliary site detection circuit module 1020 is shown. In some examples, the auxiliary site detection circuit module 1020 uses the same site as the site detection circuit module 1010. Fig.10 In the illustrated example of , the location detection circuit modules 1010, 1020 are positioned in an upper region 1030 of the venue surrounding the performance area 405. Such positioning ensures a clear vantage point of the zone of the performance area where the location detection circuit modules 1010, 1020 are to detect performer positions.

[0097] Back to Figure 8 The example image acquisition circuit module 510 then identifies a region of interest in the image of the performance area. (Block 820). Fig.11is a graphical representation of the region of interest (ROI) mask. Fig.11 In the illustrated example of , four portions 1110, 1120, 1130, 1140 are shown, each corresponding to a respective person detection circuit module 500. In the examples disclosed herein, each location detection circuit module 500 uses its own region of interest (ROI) mask 1115, 1125, 1135, 1145, which identifies areas in the captured image where performers can be identified. In the examples disclosed herein, the ROI mask is manually defined. However, in some examples, the ROI can be automatically detected from the captured (one or more) images. Any performers outside the ROI mask area will not be detected. The ROI mask is used to crop the image captured by the image acquisition circuit module 510, resize it to the appropriate input size (e.g., 1280×960), and then used by the person detector circuit module 560 during operation to detect the position of the performers.

[0098] Back to Figure 8 The example calibration display request circuit module 530 then transmits a request to display the calibration pattern to the display controller circuit module 430. (Block 830). The example display controller 430 causes the display of the calibration pattern on the performance area, which can then be captured by the image acquisition circuit module 510. Fig.12 Shows Figure 4 and / or Figure 5 An example location detection circuit module is provided with respect to an example vantage point of the stage. Fig.12 In the illustrated example of , a calibration pattern (eg, a checkerboard pattern) is displayed in the performance area. However, any other calibration pattern may be used in addition or alternatively, including, for example, a grid pattern.

[0099] The captured image of the calibration pattern is used by the homography matrix generator circuit module 540 to generate a homography matrix. (Block 840). Fig.13 is a representation of the homography matrix 1300. In order to track the real-world position of the performer on the stage, a homography estimation method is used to compute the transformation between the camera imaging plane and the stage plane. Fig.13 The homography matrix H shown is obtained by finding the i ,v i ) to the ground screen position (X i ,Y i ) is estimated by mapping.

[0100] To estimate the matrix H, a calibration pattern displayed in response to the request of block 830 is captured. In the examples disclosed herein, the calibration pattern has a known grid size on the performance area. A first number (e.g., four) of points are marked on the image and used to generate a 3×3 homography matrix by finding a mapping from pixel points to ground points. A second number of sites on the performance area are also selected and used to test the accuracy of the homography matrix.

[0101] The example accuracy tester circuit module 550 then tests the transition of the second location detected using the homography matrix. (Block 850). If the accuracy tester circuit module 550 determines that the accuracy of the homography matrix is ​​insufficient (block 860 returns a negative result) (e.g., the location detection accuracy is inaccurate for at least a threshold number of locations (e.g., five locations) within a threshold distance (e.g., one tenth of a meter), the example process returns to block 840, where the homography matrix generator circuit module 540 further improves the generated homography matrix. The example process of blocks 840-860 is repeated until the accuracy of the homography matrix is ​​sufficient, at which point the example image acquisition circuit module 510 stores the homography matrix in the calibration data store 520. (Block 870).

[0102] Fig.14 is a representation that can be executed by an example processor circuit module to implement Figure 4 The example machine readable instructions and / or example operation flow chart of the location detection circuit module to locate the performer. When the example image acquisition circuit module 510 captures an image via the image sensor 505, Fig.14. (Block 1410). The example person detector circuit module 560 detects one or more performers in the captured image. (Block 1420). In the examples disclosed herein, the trained artificial intelligence model is executed by the person detector circuit module 560 to detect the performers. In the examples disclosed herein, YouOnlyLookOnceX (YOLOX) is used to train the AI ​​model. However, other person or object detection methods may be used in addition or alternatively. In the examples disclosed herein, images are captured, annotated, and used for training the AI ​​model. In some examples, CNN training methods such as data augmentation and hyperparameter adjustment are used to achieve high detection rates while avoiding overfitting of the training data set. In some examples, the AI ​​model is modified, including, for example, pruning some feature layers, replacing base convolutional layers with layers separated by depth, converting a trained floating point 32 (float32) Pytorch model to an optimized integer 8 (int8) Openvino model, etc. As a result, when executing the model, the person detector circuit module 560 spends approximately 12ms for the person detection process, including post-processing of camera image cropping and resizing, model inference, and candidate selection. As an output of the person detection process performed by the person detector circuit module 560, a bounding box is created for each detected performer.

[0103] Fig.15 is a representation 1500 of an interactive performance, including person detection information. Fig.15 In the illustrated example of , each of the performers is overlaid with a bounding box. For example, bounding box 1510 represents one of the detected performers. In some examples, additional information 1520 related to the identity of the performer represented by bounding box 1510 is also overlaid.

[0104] Back to Fig.14 , the example location estimator circuit module 580 estimates the location of the performer based on the bounding box corresponding to the detected performer. (Block 1440) To estimate the true location of the person on the ground plane, the pixel locations in the camera image where the performer's feet touch the ground are identified. Fig.16 is a representation of the position detection of a person based on the bounding box locations 1610. If the position of the performer's feet can be identified (e.g., Fig.16 1620), then that point can be used as the position of the performer within the image. Alternatively, if the position of the performer's feet cannot be identified, then a point representing the bottom center of the bounding box is used as the performer's position within the image (e.g., Fig.161625 in the performance area 405). In practice, the difference between the two possible points is typically low, with an average value of less than three tenths of a meter, which is generally acceptable in real-world applications. The example location estimator circuit module 580 then converts the performer's location in the image to a location in physical space on the performance area 405 using the homography matrix stored in the calibration data store 520.

[0105] The example location estimator circuit module 580 compensates for the location to account for system latency. (Block 1450). For example, although location detection may be performed every hundred milliseconds, if the performer is moving at a high rate of speed, the detected location may lag behind the performer. To account for this, the example location estimator circuit module 580 uses the performer's previously detected location to compensate by predicting a new location along the direction of movement of the location. In other words, a small offset is added to the current location. In some examples, the small offset is obtained by multiplying the speed and the short time.

[0106] The example smoothing circuit module 590 smoothes the compensated position of the performer. (Block 1460) Smoothing of the performer's position helps reduce jitter in the detected position of the performer.

[0107] The example location provider circuit module 595 provides the smoothed location to the display controller circuit module 430. (Block 1470). The Fig.14 The example process 720 enables sequential detection of the performers' positions.

[0108] Fig.17 is an example processor circuit module that can be executed to implement Figure 4 A flow chart of example machine readable instructions and / or example operations for a display controller to generate interactive elements based on a performer's position. Fig.17 The machine readable instructions and / or operations 730 of the example begin at block 1710 where the location receiver circuit module 610 obtains location information from the location detection circuit module (block 1710). The example aggregation circuit module 620 aggregates the location information. (block 1720).

[0109] The example redundancy checker circuit module 630 identifies locations within the overlap region(s). (Block 1730). Fig.18 An example ROI definition 1810 for a first camera is shown, including a first overlap region 1820, and Fig.19 Shows that for the combination with Fig.1819. Example ROI definition 1910 of a second camera adjacent to a first camera, including a second overlap region 1920. When a performer stands in the physical space represented by the overlap region, there may be two detection results for the same performer. In the examples disclosed herein, four location detection circuit modules are used to identify performers in their own corresponding ROIs. The example redundancy checker circuit module 630 enables redundancy checks to be performed, for example, to identify whether two locations have been identified for the same performer.

[0110] For example, if a performer is identified by two site detection circuit modules responsible for adjacent zones of the performance area, one of the site identifications should be omitted to avoid multiple interactive elements being displayed for the performer and / or other rendering irregularities. The example redundancy checker circuit module 630 determines whether any two sites are within a threshold distance from each other. (Box 1740). In the example disclosed herein, the threshold distance is three tenths of a meter. If two sites are detected within a threshold distance from each other, the redundancy checker circuit module 630 removes one of the sites. (Box 1750). That is, two identifications for the same performer are reduced to a single identification for a single performer. In some examples, a site from one of the site detection circuit modules is superior to the other. That is, when two sites are detected within a threshold distance from each other, the site originating from the first of the site detection circuit modules is retained, while the site originating from the second of the site detection circuit modules is deleted.

[0111] The example effect generation circuit module 640 generates a rendering effect based on the detected position of the performer. (Block 1760). In some examples, the effect is generated using a rendering engine (such as, for example, Unity or Unreal). However, any other rendering engine and / or technology for generating interactive effects may additionally or alternatively be used. The example effect output circuit module 650 then causes display of the generated effect. (Block 1770).

[0112] Fig. 20 Is Figure 4 An example of an interactive performance 2000 output by the interactive presentation system 407.

[0113] Fig.21 is constructed to execute and / or instantiate Figure 7 , 8 , 14 and / or 17 machine readable instructions and / or operations to implement Figure 4 The processor platform 2100 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, an iPad, etc.).TM ), a tablet computer such as a , a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, headsets (e.g., an augmented reality (AR) headset, a virtual reality (VR) headset, etc.) or other wearable device, or any other type of computing device.

[0114] The processor platform 2100 of the illustrated example includes a processor circuit module 2112. The processor circuit module 2112 of the illustrated example is hardware. For example, the processor circuit module 2112 can be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs and / or microcontrollers from any desired series or manufacturer. The processor circuit module 2112 can be implemented by one or more semiconductor-based (e.g., silicon-based) devices. In this example, the processor circuit module 2112 implements the example site detection circuit module 410 and / or the example display controller circuit module 430. In the examples disclosed herein, the site detection circuit modules 410, 411, 412, 413 and the display controller circuit module 430 are implemented by separate processor platforms. However, in some examples, the site detection circuit modules 410, 411, 412, 413 and / or the example display controller circuit module 430 can be implemented on the same processor platform. In such examples, the image sensor corresponding to each of the location detector circuit modules can provide location information to a center location including one or more instances of location detection circuit modules 410, 411, 412, 413. In some examples, display controller circuit module 430 can be implemented at the same processor platform as one of the location detector circuit modules.

[0115] The processor circuit module 2112 of the illustrated example includes a local memory 2113 (e.g., cache, register, etc.). The processor circuit module 2112 of the illustrated example communicates with a main memory (including a volatile memory 2114 and a non-volatile memory 2116) via a bus 2118. The volatile memory 2114 may be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 2116 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 2114, 2116 of the illustrated example is controlled by a memory controller 2117.

[0116] The processor platform 2100 of the illustrated example further includes an interface circuit module 2120. The interface circuit module 2120 may be implemented by hardware according to any type of interface standard, such as an Ethernet interface, a universal serial bus (USB) interface, interface, a near field communication (NFC) interface, a peripheral component interconnect (PCI) interface, and / or a peripheral component interconnect express (PCIe) interface.

[0117] In the example shown, one or more input devices 2122 are connected to the interface circuit module 2120. The input device(s) 2122 allow a user to input data and / or commands into the processor circuit module 2112. The input device(s) 2122 may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a track pad, a trackball, an isopoint device, and / or a voice recognition system.

[0118] One or more output devices 2124 are also connected to the interface circuit module 2120 of the illustrated example. The output device(s) 2124 may be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-situ switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit module 2120 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics processor circuit module such as a GPU.

[0119] The interface circuit module 2120 of the illustrated example also includes a communication device, such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and / or a network interface to facilitate the exchange of data with an external machine (e.g., any kind of computing device) via the network 2126. The communication may be through, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a field wireless system, a cellular telephone system, an optical connection, etc.

[0120] The processor platform 1000 of the illustrated example also includes one or more mass storage devices 2128 to store software and / or data. Examples of such mass storage devices 2128 include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, CDs, Blu-ray disk drives, redundant array of independent disks (RAID) systems, solid-state storage devices such as flash memory devices and / or SSDs, and DVD drives.

[0121] Can be Figure 7 , 8The machine-readable instructions 2132 implemented by the machine-readable instructions of 14 and / or 17 may be stored in the mass storage device 2128, in the volatile memory 2114, in the non-volatile memory 2116, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0122] Fig. 22 yes Fig.21 2112 of the processor circuit module. In this example, Fig.21 The processor circuit module 2112 is implemented by the microprocessor 2200. For example, the microprocessor 2200 may be a general-purpose microprocessor (eg, a general-purpose microprocessor circuit module). The microprocessor 2200 executes Figure 7 , 8 , 14 and / or 17 of the machine readable instructions to effectively Figure 4 , 5 The position detection circuit module 410 and / or the display controller circuit module 430 of the and / or 6 are instantiated as logic circuits to perform operations corresponding to those machine-readable instructions. In some such examples, Figure 4 , 5 The position detection circuit module 410 and / or the display controller circuit module 430 of and / or 6 are instantiated by the hardware circuit of the microprocessor 2200 in combination with instructions. For example, the microprocessor 2200 can be implemented by a multi-core hardware circuit module, such as a CPU, a DSP, a GPU, an XPU, etc. Although it may include any number of example cores 2202 (for example, 1 core), the microprocessor 2200 of this example is a multi-core semiconductor device including N cores. The cores 2202 of the microprocessor 2200 can operate independently, or can cooperate to execute machine-readable instructions. For example, the machine code corresponding to a firmware program, an embedded software program, or a software program can be executed by one of the cores 2202, or can be executed by multiple cores in the core 2202 at the same or different times. In some examples, the machine code corresponding to the firmware program, the embedded software program, or the software program is divided into threads and executed in parallel by two or more cores in the core 2202. The software program may correspond to a program executed by Figure 7 , 8 , 14, and 17 are flowcharts representing part or all of the machine-readable instructions and / or operations.

[0123] The core 2202 can communicate via a first example bus 2204. In some examples, the first bus 2204 can be implemented by a communication bus to implement communications associated with (one or more) cores in the core 2202. For example, the first bus 2204 can be implemented by at least one of an inter-integrated circuit (I2C) bus, a serial peripheral interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the first bus 2204 can be implemented by any other type of computing or electrical bus. The core 2202 can obtain data, instructions, and / or signals from one or more external devices via an example interface circuit module 2206. The core 2202 can output data, instructions, and / or signals to the one or more external devices via the interface circuit module 2206. Although the core 2202 of this example includes an example local memory 2220 (e.g., a level 1 (L1) cache that may be divided into a L1 instruction cache and a L1 data cache), the microprocessor 2200 also includes an example shared memory 2210 (e.g., a level 2 (L2 cache)) that may be shared by the cores for high-speed access to data and / or instructions. Data and / or instructions may be transferred (e.g., shared) by writing to and / or reading from the shared memory 2210. The local memory 2220 and the shared memory 2210 of each core in the core 2202 may be a memory including multiple levels of cache memory and main memory (e.g., Fig.21 14, 2116). Typically, memories at higher levels in the hierarchy exhibit lower access times and have smaller storage capacities than memories at lower levels. Changes in the various levels of the cache hierarchy are managed (e.g., coordinated) by cache coherence policies.

[0124] Each core 2202 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit module. Each core 2202 includes a control unit circuit module 2214, an arithmetic and logic (AL) circuit module (sometimes referred to as an ALU) 2216, a plurality of registers 2218, a local memory 2220, and a second example bus 2222. Other structures may exist. For example, each core 2202 may include a vector unit circuit module, a single instruction multiple data (SIMD) unit circuit module, a load / store unit (LSU) circuit module, a branch / jump unit circuit module, a floating point unit (FPU) circuit module, etc. The control unit circuit module 2214 includes a semiconductor-based circuit configured to control (e.g., coordinate) data movement within the corresponding core 2202. The AL circuit module 2216 includes a semiconductor-based circuit configured to perform one or more mathematical and / or logical operations on data within the corresponding core 2202. The AL circuit module 2216 of some examples performs integer-based operations. In other examples, the AL circuit module 2216 also performs floating-point operations. In still other examples, the ALU circuit module 2216 may include a first AL circuit module that performs integer-based operations and a second AL circuit module that performs floating-point operations. In some examples, the AL circuit module 2216 may be referred to as an arithmetic logic unit (ALU). The register 2218 is a semiconductor-based structure for storing data and / or instructions (such as the results of one or more of the operations performed by the AL circuit module 2216 of the corresponding core 2202). For example, the register 2218 may include (one or more) vector registers, (one or more) SIMD registers, (one or more) general registers, (one or more) flag registers, (one or more) segment registers, (one or more) machine-specific registers, (one or more) instruction pointer registers, (one or more) control registers, (one or more) debug registers, (one or more) memory management registers, (one or more) machine check registers, etc. The register 2218 may be arranged as Fig. 22 2202. Alternatively, registers 2218 may be organized in any other arrangement, format, or structure, including being distributed throughout core 2202 to reduce access time. Second bus 2222 may be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0125] Each core 2202 and / or more generally the microprocessor 2200 may include additional and / or alternative structures to those shown and described above. For example, one or more clock circuits, one or more power supply devices, one or more power gates, one or more cache home agents (CHA), one or more convergence / common grid stops (CMS), one or more shifters (e.g., (one or more) barrel shifters) and / or other circuit modules may exist. The microprocessor 2200 is a semiconductor device made to include many transistors, which are interconnected to implement the structure described above in one or more integrated circuits (ICs) contained in one or more packages. The processor circuit module may include one or more accelerators and / or cooperate with one or more accelerators. In some examples, the accelerator is implemented by a logic circuit module to perform certain tasks faster and / or more efficiently than what can be done by a general-purpose processor. Examples of accelerators include ASICs and FPGAs, such as those discussed herein. GPUs or other programmable devices may also be accelerators. The accelerator may be on the board of the processor circuit module, in the same chip package as the processor circuit module, and / or in one or more packages separated from the processor circuit module.

[0126] Fig.23 yes Fig.21 2 is a block diagram of another example implementation of a processor circuit module 2112 of FIG. 2. In this example, the processor circuit module 2112 is implemented by an FPGA circuit module 2300. For example, the FPGA circuit module 2300 may be implemented by an FPGA. The FPGA circuit module 2300 can be used, for example, to perform operations that may otherwise be performed by a processor executing corresponding machine-readable instructions. Fig. 22 However, once configured, the FPGA circuit module 2300 instantiates the machine-readable instructions in hardware and can therefore often perform operations faster than if the operations could be performed by a general-purpose microprocessor executing corresponding software.

[0127] More specifically, as described above Fig. 22 The microprocessor 2200 (which is programmable to execute Figure 7 , 8 , 14 and / or 17, but its interconnections and logic circuit modules are fixed once they are made), compared to a general purpose device that contains some or all of the machine-readable instructions represented by the flowcharts of 1, 14 and / or 17, but whose interconnections and logic circuit modules are fixed once they are made. Fig.23 The example FPGA circuit module 2300 includes a plurality of circuits that can be configured and / or interconnected in different ways after fabrication to instantiate, for example, Figure 7 , 8, 14 and / or 17. In particular, FPGA circuit module 2300 can be thought of as an array of logic gates, interconnections, and switches. The switches can be programmed to change how the logic gates are interconnected by the interconnections, effectively forming one or more dedicated logic circuits (unless and until FPGA circuit module 2300 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on the data received by the input circuit module. Those operations may correspond to the operations performed by Figure 7 , 8 , 14 and / or 17. Therefore, the FPGA circuit module 2300 can be configured to Figure 7 , 8 , 14 and / or 17 are effectively instantiated as dedicated logic circuits to perform operations corresponding to those software instructions in a dedicated manner similar to an ASIC. Thus, a general-purpose microprocessor can perform operations corresponding to Fig.21 FPGA circuit module 2300 can execute the corresponding instructions faster than some or all of the machine-readable instructions. Fig.21 The operations of some or all of the machine-readable instructions.

[0128] exist Fig.23 In the example of , FPGA circuit module 2300 is configured to be programmed (and / or reprogrammed one or more times) by an end user via a hardware description language (HDL) such as Verilog. Fig.23 The FPGA circuit module 2300 includes an example input / output (I / O) circuit module 2302 to obtain data from and / or output data to the example configuration circuit module 2304 and / or external hardware 2306. For example, the configuration circuit module 2304 may be implemented by an interface circuit module, which may obtain machine-readable instructions to configure the FPGA circuit module 2300 or (one or more) parts thereof. In some such examples, the configuration circuit module 2304 may obtain machine-readable instructions from a user, a machine (e.g., a hardware circuit module (e.g., a programmed or dedicated circuit module) that may implement an artificial intelligence / machine learning (AI / ML) model to generate instructions), etc. In some examples, the external hardware 2306 may be implemented by an external hardware circuit module. For example, the external hardware 2306 may be implemented by Fig. 22The FPGA circuit module 2300 also includes an array of example logic gate circuit modules 2308, multiple example configurable interconnects 2310, and example storage circuit modules 2312. The logic gate circuit modules 2308 and the configurable interconnects 2310 are configurable to instantiate one or more operations that may correspond to at least some of the machine-readable instructions of FIG. _ and / or other desired operations. Fig.23 The logic gate circuit modules 2308 shown in are made in groups or blocks. Each block contains a semiconductor-based electronic structure that can be configured into a logic circuit. In some examples, the electronic structure includes a logic gate (e.g., an AND gate, an OR gate, a NOR gate, etc.) that provides a basic building block for the logic circuit. An electrically controllable switch (e.g., a transistor) is present in each of the logic gate circuit modules 2308 so that the configuration of the logic gate and / or electronic structure can form a circuit that performs the desired operation. The logic gate circuit module 2308 may include other electronic structures, such as a lookup table (LUT), a register (e.g., a flip-flop or a latch), a multiplexer, etc.

[0129] The configurable interconnect 2310 of the illustrated example is a conductive path, trace, via, or the like that may include electrically controlled switches (e.g., transistors), the state of which may be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more logic gate circuit modules in the logic gate circuit module 2308, thereby programming the desired logic circuit.

[0130] The storage circuit module 2312 of the illustrated example is configured to store the result(s) of one or more operations performed by the corresponding logic gates. The storage circuit module 2312 may be implemented by a register or the like. In the illustrated example, the storage circuit module 2312 is distributed in the logic gate circuit module 2308 to facilitate access and increase execution speed.

[0131] Fig.23The example FPGA circuit module 2300 also includes an example dedicated operation circuit module 2314. In this example, the dedicated operation circuit module 2314 includes a dedicated circuit module 2316, which can be called to implement common functions to avoid the need to program those functions on site. Examples of such dedicated circuit modules 2316 include memory (e.g., DRAM) controller circuit modules, PCIe controller circuit modules, clock circuit modules, transceiver circuit modules, memory, and multiplier-accumulator circuit modules. Other types of dedicated circuit modules may exist. In some examples, the FPGA circuit module 2300 may also include an example general-purpose programmable circuit module 2318, such as an example CPU 2320 and / or an example DSP 2322. Additionally or alternatively, other general-purpose programmable circuit modules 2318 may exist, such as GPUs, XPUs, etc., which can be programmed to perform other operations.

[0132] Although Fig. 22 and Fig.23 Show Fig.21 These are two example implementations of the processor circuit module 2112, but many other implementations are contemplated. For example, as mentioned above, a modern FPGA circuit module may include an onboard CPU, such as Fig.23 One or more of the example CPUs 2320. Thus, Fig.21 The processor circuit module 2112 can additionally be combined by Fig. 22 An example microprocessor 2200 and Fig.23 In some such hybrid examples, the Figure 7 , 8 , 14 and / or 17 may be represented by a first portion of the machine readable instructions. Fig. 22 Executed by one or more of the cores 2202, Figure 7 , 8 , 14 and / or 17 may be represented by a second portion of the machine readable instructions represented by the flowcharts of Fig.23 FPGA circuit module 2300 to execute, and / or by Figure 7 , 8 , 14 and / or 17, the third portion of the machine-readable instructions represented by the flowcharts may be executed by an ASIC. It should be understood that Fig.23 Some or all of the circuit modules of can therefore be instantiated at the same or different times. For example, some or all of the circuit modules can be instantiated in one or more threads that execute concurrently and / or serially. In addition, in some examples, Fig.23 Some or all of the circuit modules may be implemented within one or more virtual machines and / or containers executed on the microprocessor.

[0133] In some examples, Fig.21 The processor circuit module 2112 may be in one or more packages. For example, Fig. 22 The processor circuit module 2200 and / or Fig.23 The FPGA circuit module 2300 may be in one or more packages. In some examples, the XPU may be composed of a Fig.21 The XPU may be implemented by a processor circuit module 2112. For example, the XPU may include a CPU in one package, a DSP in another package, a GPU in still another package, and an FPGA in yet another package.

[0134] Shown is a method for converting Fig.21 A block diagram of an example software distribution platform 2405 of example machine readable instructions 2132 for distributing software to a hardware device owned and / or operated by a third party is provided in Fig.24 The example software distribution platform 2405 may be implemented by any computer server, data facility, cloud service, etc. that can store software and transfer software to other computing devices. A third party may be a consumer of an entity that owns and / or operates the software distribution platform 2405. For example, an entity that owns and / or operates the software distribution platform 2405 may be a Fig.21 The third party may be a consumer, user, retailer, OEM, etc. who purchases the software and / or licenses the software for use and / or resale and / or sublicense. In the example shown, the software distribution platform 2405 includes one or more servers and one or more storage devices. The storage device stores the machine-readable instructions 2132, which may correspond to Figure 7 , 8 , 14 and / or 17 (as described above). One or more servers of the example software distribution platform 2405 communicate with the example network 2410, which may correspond to any one or more of the Internet and / or any of the example networks 2126 described above. In some examples, as part of a business transaction, the one or more servers respond to a request to transfer software to a requesting party. The delivery, sale and / or payment of the license of the software may be processed by the one or more servers of the software distribution platform and / or by a third-party payment entity. The server enables a purchaser and / or licensor to download the machine-readable instructions 2132 from the software distribution platform 2405. For example, it may correspond to Figure 7 , 8, 14 and / or 17 can be downloaded to the example processor platform 2100, which is to execute the machine-readable instructions 2132 to implement the location detection circuit module 410 and / or the display controller circuit module 430. In some examples, one or more servers of the software distribution platform 2405 periodically supply, transmit and / or force the software (e.g., Fig.21 Example machine readable instructions 2132) to ensure that improvements, patches, updates, etc. are distributed and applied to the software at the end-user device.

[0135] From the foregoing, it will be appreciated that example systems, methods, apparatus, and articles of manufacture have been disclosed that enable large-scale interactive performances. The disclosed systems, methods, apparatus, and articles of manufacture improve the efficiency of using computing devices by using an artificial intelligence model to detect the position of performers using image sensors, as opposed to more complex approaches such as pressure sensors embedded in the performance area. Thus, the disclosed systems, methods, apparatus, and articles of manufacture are directed to one or more improvements in the operation of a machine such as a computer or other electronic and / or mechanical device.

[0136] This document discloses example methods, devices, systems, and articles for real-time interactive performances. Further examples and combinations thereof include the following examples:

[0137] Example 1 includes a device for real-time interactive performances, the device comprising: at least one memory; machine-readable instructions; and a processor circuit module, the processor circuit module being used to instantiate or execute at least one of the machine-readable instructions to: capture an image of a performance area; use the captured image to detect one or more performers in the performance area; estimate positions of the one or more detected performers; smooth the estimated positions of the one or more detected performers based on previously estimated positions; and provide the smoothed estimated positions to a display controller circuit module for generating an interactive effect based on the smoothed estimated positions.

[0138] Example 2 includes the apparatus of Example 1, wherein the processor circuit module is further configured to compensate the estimated position to account for delay.

[0139] Example 3 includes an apparatus as described in Example 1, wherein, to estimate the position of the one or more detected performers, the processor circuit module is used to apply a homography matrix to transform from pixel positions within the image to physical positions in the performance area.

[0140] Example 4 includes the apparatus of Example 1, wherein, to estimate the position of the one or more detected performers, the processor circuit module is to generate a bounding box corresponding to each of the one or more detected performers.

[0141] Example 5 includes the apparatus of Example 4, wherein the estimate of the position of the one or more detected performers is based on a position of a midpoint of a lower edge of the bounding box.

[0142] Example 6 includes a system for real-time interactive performances, the system comprising: a first location detection circuit module for estimating the location of a performer in a first zone of a performance area; a second location detection circuit module for estimating the location of the performer in a second zone of the performance area; and a display controller circuit module for aggregating the locations of the performers estimated by the first location circuit module and the second location detection circuit module, and generating an interactive effect based on the aggregated locations.

[0143] Example 7 includes the system of Example 6, wherein the first zone of the performance area is adjacent to the second zone of the performance area.

[0144] Example 8 includes the system of Example 7, wherein the display controller circuit module is to remove one of the two estimated locations in response to determining that the two estimated locations are within a threshold distance of each other.

[0145] Example 9 includes the system of Example 8, wherein the display controller circuit module is to remove the one of the two estimated locations only if the one location is within an overlapping region.

[0146] Example 10 includes a non-transitory machine-readable storage medium comprising instructions that, when executed, cause a processor circuit module to at least: capture an image of a performance area; detect one or more performers in the performance area using the captured image; estimate locations of the one or more detected performers; smooth the estimated locations of the one or more detected performers; and provide the smoothed estimated locations to a display controller circuit module for generating an interactive effect based on the smoothed estimated locations.

[0147] Example 11 includes the non-transitory machine-readable storage medium of Example 10, wherein the processor circuit module is further configured to compensate the estimated location to account for latency.

[0148] Example 12 includes a non-transitory machine-readable storage medium as described in Example 10, wherein, in order to estimate the locations of the one or more detected performers, the processor circuit module is used to apply a homography matrix to transform from pixel locations within the image to physical locations in the performance area.

[0149] Example 13 includes the non-transitory machine-readable storage medium of Example 10, wherein, to estimate the location of the one or more detected performers, the processor circuit module is used to generate a bounding box corresponding to each of the one or more detected performers.

[0150] Example 14 includes the non-transitory machine-readable storage medium of Example 13, wherein the estimate of the location of the one or more detected performers is based on a position of a midpoint of a lower edge of the bounding box.

[0151] Example 15 includes a method for real-time interactive performances, the method comprising: capturing an image of a performance area; using the captured image to detect one or more performers in the performance area by executing instructions with at least one processor; estimating locations of the one or more detected performers; smoothing the estimated locations of the one or more detected performers based on previously estimated locations; and providing the smoothed estimated locations to a display controller circuit module for generating an interactive effect based on the smoothed estimated locations.

[0152] Example 16 includes the method of Example 15, further comprising compensating the estimated location to account for delay.

[0153] Example 17 includes the method of Example 15, further comprising: to estimate the location of the one or more detected performers, applying a homography matrix to transform from pixel locations within the image to physical locations in the performance area.

[0154] Example 18 includes the method of Example 15, further comprising: generating a bounding box corresponding to each of the one or more detected performers to estimate the location of the one or more detected performers.

[0155] Example 19 includes the method of Example 18, wherein the estimate of the location of the one or more detected performers is based on a position of a midpoint of a lower edge of the bounding box.

[0156] Example 20 includes a device for real-time interactive performances, the device comprising: a component for capturing an image of a performance area; a component for using the captured image to detect one or more performers in the performance area; a component for estimating the positions of the one or more detected performers; a component for smoothing the estimated positions of the one or more detected performers based on previously estimated positions; and a component for providing the smoothed estimated positions to a display controller circuit module for generating an interactive effect based on the smoothed estimated positions.

[0157] Example 21 includes the apparatus of Example 20, further comprising means for compensating the estimated location to account for delay.

[0158] Example 22 includes the apparatus of Example 20, wherein the means for estimating is further for applying a homography matrix to transform from pixel locations within the image to physical locations in the performance area.

[0159] Example 23 includes the apparatus of Example 20, wherein the means for estimating is further for generating a bounding box corresponding to each of the one or more detected performers.

[0160] Example 24 includes the apparatus of Example 23, wherein the estimate of the location of the one or more detected performers is based on a position of a midpoint of a lower edge of the bounding box.

[0161] The attached claims are hereby incorporated into this detailed description by this reference. Although certain example systems, methods, devices, and articles have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all systems, methods, devices, and articles that fall completely within the scope of the claims of this patent.

Claims

1. A device for real-time interactive performance, the device comprising: at least one memory; machine-readable instructions; as well as A processor circuit module configured to instantiate or execute at least one of the machine-readable instructions to: capturing images of the performance area; using the captured images to detect one or more performers in the performance area; estimating the position of the one or more detected performers; smoothing the estimated positions of the one or more detected performers based on previously estimated positions; as well as The smoothed estimated position is provided to a display controller circuit module for use in generating an interactive effect based on the smoothed estimated position.

2. The device according to claim 1, wherein: The processor circuit module is also used to compensate the estimated position to account for time delay.

3. The device of claim 1, wherein: To estimate the position of the one or more detected performers, the processor circuit module is used to apply a homography matrix to transform from pixel positions within the image to physical positions in the performance area.

4. The device of claim 1, wherein: To estimate the position of the one or more detected performers, the processor circuit module is used to generate a bounding box corresponding to each of the one or more detected performers.

5. The device of claim 4, wherein: The estimate of the position of the one or more detected performers is based on the position of a midpoint of a lower edge of the bounding box.

6. A system for real-time interactive performance, the system comprising: A first location detection circuit module, used for estimating the location of the performer in the first zone of the performance area; A second location detection circuit module, used for estimating the location of the performer in a second zone of the performance area; as well as The display controller circuit module is used to aggregate the positions of the performer estimated by the first position circuit module and the second position detection circuit module, and generate an interactive effect based on the aggregated positions.

7. The system of claim 6, wherein: The first zone of the performance area is adjacent to the second zone of the performance area.

8. The system of claim 7, wherein: The display controller circuit module is used to remove one of the two estimated locations in response to determining that the two estimated locations are within a threshold distance of each other.

9. The system of claim 8, wherein: The display controller circuit module is used to remove the one of the two estimated locations only if the one location is within the overlap region.

10. A non-transitory machine-readable storage medium comprising instructions that, when executed, cause a processor circuit module to at least: capturing images of the performance area; using the captured images to detect one or more performers in the performance area; estimating locations of the one or more detected performers; smoothing the estimated positions of the one or more detected performers; as well as The smoothed estimated position is provided to a display controller circuit module for use in generating an interactive effect based on the smoothed estimated position.

11. The non-transitory machine-readable storage medium of claim 10, wherein: The processor circuit module is also used to compensate the estimated position to account for delay.

12. The non-transitory machine-readable storage medium of claim 10, wherein: To estimate the locations of the one or more detected performers, the processor circuit module is used to apply a homography matrix to transform from pixel locations within the image to physical locations in the performance area.

13. The non-transitory machine-readable storage medium of claim 10, wherein: To estimate the location of the one or more detected performers, the processor circuit module is used to generate a bounding box corresponding to each of the one or more detected performers.

14. The non-transitory machine-readable storage medium of claim 13, wherein: The estimate of the location of the one or more detected performers is based on the position of the midpoint of the lower edge of the bounding box.

15. A method for real-time interactive performance, the method comprising: capturing images of the performance area; detecting one or more performers in the performance area using the captured images by executing instructions with at least one processor; estimating locations of the one or more detected performers; smoothing the estimated positions of the one or more detected performers based on previously estimated positions; as well as The smoothed estimated position is provided to a display controller circuit module for use in generating an interactive effect based on the smoothed estimated position.

16. The method of claim 15, further comprising compensating the estimated location to account for delay.

17. The method of claim 15, further comprising: applying a homography matrix to transform from pixel locations within the image to physical locations in the performance area in order to estimate the locations of the one or more detected performers.

18. The method of claim 15, further comprising generating a bounding box corresponding to each of the one or more detected performers in order to estimate the location of the one or more detected performers.

19. The method of claim 18, wherein: The estimate of the location of the one or more detected performers is based on the position of the midpoint of the lower edge of the bounding box.

20. A device for real-time interactive performance, the device comprising: means for capturing images of a performance area; means for detecting one or more performers in the performance area using the captured images; means for estimating the location of said one or more detected performers; means for smoothing the estimated positions of the one or more detected performers based on previously estimated positions; as well as Means for providing the smoothed estimated location to a display controller circuit module for generating an interactive effect based on the smoothed estimated location.

21. The apparatus of claim 20, further comprising means for compensating the estimated location to account for time delay.

22. The apparatus of claim 20, wherein: The means for estimating is further for applying a homography matrix to transform from pixel locations within the image to physical locations in the performance area.

23. The apparatus of claim 20, wherein: The means for estimating is further for generating a bounding box corresponding to each of the one or more detected performers.

24. The apparatus of claim 23, wherein: The estimate of the location of the one or more detected performers is based on the position of the midpoint of the lower edge of the bounding box.

Images