Systems and methods for determining projected target location of a handheld object

Abstract

A projected target position of the handheld object is determined based on applying a translation factor, a zoom factor, and an offset to a position of a reference element of the handheld object detected by the camera on a two-dimensional plane. The translation factor is determined based on a difference between a calibration position on the plane and an initial position of the reference element corresponding to the calibration position, and is used to shift the position of the reference element to generate a projection target position. A scaling factor is determined based on an estimated length of an arm of a user holding the handheld object and is used to scale a position of the reference element to generate a projected target position. The offset is determined based on a polynomial equation and is used to extend the distance between the projection target position and the calibration position.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511529659.0 (22) Application Date 2020.09.21 (30) Priority Data 62 / 905901 2019.09.25 US 16 / 749865 2020.01.22 US (62) Divisional Application Data 202080067328.9 2020.09.21 (71) Applicant Universal City Cinema LLC Address California, USA (72) Inventors YJ Lin M Beavers S Burkert J D Dench K P Hanley H Kan A Mello (74) Patent Agency China Patent Agency (Hong Kong) Limited 72001 Patent Attorney Wang Bin Liu Chunyuan (51) Int.Cl. A63F 13 / 213(2014.01) A63F 13 / 219(2014.01) A63F 13 / 22(2014.01) A63F 13 / 27(2014.01) A63F 13 / 426(2014.01) A63H 30 / 04(2006.01) G06F 3 / 01(2006.01) G06F 3 / 03(2006.01) G06T 7 / 80(2017.01) G08C 17 / 02(2006.01) (54) Invention Title System and Method for Determining Projected Target Position of a Handheld Object (57) Abstract The projected target position of a handheld object is determined based on the position of a reference element of the handheld object detected by a camera on a two-dimensional plane by applying a translation factor, a scaling factor, and an offset. A translation factor is determined based on the difference between a calibration position on a plane and an initial position of a reference element corresponding to the calibration position, and is used to transfer the position of the reference element to generate a projected target position. A scaling factor is determined based on an estimated length of the arm of a user holding the handheld object, and is used to scale the position of the reference element to generate a projected target position. An offset is determined based on a polynomial equation and is used to extend the distance between the projected target position and the calibration position. Claims 5 pages, Description 14 pages, Drawings 9 pages, CN 121490366 A 2026.02.10 CN 1 21 49 03 66 A 1. A method comprising: receiving an initial position of a reference element of a handheld object projected onto a two-dimensional plane in a three-dimensional space; determining a first offset along a first axis of the two-dimensional plane, the first offset compensating for distortion along the first axis associated with an arcuate movement of the handheld object in the three-dimensional space; determining a second offset along a second axis of the two-dimensional plane, the second offset compensating for distortion along the first axis associated with the arcuate movement of the handheld object in the three-dimensional space.The method of claim 1, wherein determining the projection target position comprises shifting the initial position of the reference element along the first axis by the first offset. 3. The method of claim 1, wherein the first offset along the first axis comprises a horizontal offset in the two-dimensional plane. 4. The method of claim 1, wherein determining the projection target position comprises shifting the initial position of the reference element along the second axis by the second offset. 5. The method of claim 1, wherein the second offset along the second axis comprises a vertical offset in the two-dimensional plane. 6. The method of claim 1, wherein the arc is at least partially defined by the user's arm length when the handheld object is moved through the three-dimensional space. 7. The method of claim 1, wherein determining the first offset comprises performing regression analysis to fit test data to a first polynomial equation, and wherein determining the second offset comprises performing regression analysis to fit the test data to a second polynomial equation. 8. The method of claim 7, wherein the first polynomial equation and the second polynomial equation are each of order three. 9. A theme park attraction system, comprising: an output device configured to output a user interactive experience; a controller having one or more processors and a memory storing machine-readable instructions configured to cause the one or more processors to: determine a first offset along a first axis of a two-dimensional plane, the first offset compensating for distortion along the first axis associated with an arcuate movement of a handheld object in three-dimensional space; determine a second offset along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arcuate movement of the handheld object in the three-dimensional space; determine a projected target position of the handheld object based on an initial position of a reference element of the handheld object in the two-dimensional plane, the first offset, and the second offset; and, in response to determining that the projected target position corresponds to a target position, cause the output device to output the user interactive experience. 10. The theme park attraction system of claim 9, wherein the first offset is determined based on a distance along the first axis from the initial position of the reference element of the projected target position of the handheld object. 11. The theme park attraction system according to claim 9, wherein the second offset is based on the handheld object. (Claim 1 / 5, page 2, CN 121490366 A)The projected target position is determined by the distance along the second axis away from the initial position of the reference element. 12. The theme park attraction system of claim 9, comprising a camera configured to capture an image of the reference element of the handheld object on the two-dimensional plane. 13. One or more tangible non-transitory computer-readable media, comprising instructions that, when executed by at least one processor, cause the at least one processor to: receive an initial position of a reference element of a handheld object projected onto a two-dimensional plane; determine a first offset along a first axis of the two-dimensional plane, the first offset compensating for distortion along the first axis associated with an arcuate movement of the handheld object in three-dimensional space; determine a second offset along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arcuate movement of the handheld object in three-dimensional space; determine a projected target position of the handheld object based on the initial position of the reference element, the first offset, and the second offset; and output a user interactive experience in response to determining that the projected target position corresponds to a target position. 14. One or more tangible non-transitory computer-readable media of claim 13, wherein the instructions cause the at least one processor to divide the arc into a plurality of reference element regions, and wherein each of the plurality of reference element regions is associated with a corresponding target region among a plurality of target regions on the two-dimensional plane. 15. One or more tangible non-transitory computer-readable media of claim 14, wherein each of the plurality of reference element regions is associated with: a first equation that compensates for the distortion along the first axis associated with the corresponding target region and the arc on the two-dimensional plane; and a second equation that compensates for the distortion along the second axis associated with the corresponding target region and the arc on the two-dimensional plane. 16. One or more tangible non-transitory computer-readable media of claim 15, wherein the first equation and the second equation are third-order polynomial equations. 17. One or more tangible non-transitory computer-readable media of claim 14, wherein each of the plurality of reference element regions is of the same size, and wherein each of the plurality of target regions is of a different size. 18. The one or more tangible non-transitory computer-readable media of claim 14, wherein a first target region of the plurality of target regions is farther away from the reference element than a second target region of the plurality of target regions, and wherein the first target region is larger in size than the second target region. 19. The one or more tangible non-transitory computer-readable media of claim 14, wherein each of the plurality of reference element regions is of a different size, and wherein each of the plurality of target regions...They are the same size. 20. One or more tangible non-transitory computer-readable media of claim 14, wherein a first reference element region of the plurality of reference element regions is farther away from the two-dimensional plane than a second reference element region of the plurality of reference element regions, and wherein the first reference element region is smaller in size than the second reference element region. 21. A method comprising: determining a first offset along a first axis of a two-dimensional plane, the first offset compensating for distortion along the first axis associated with the arc by dividing an arc of movement of a reference element of a handheld object in three-dimensional space into a plurality of reference element regions, and wherein each of the plurality of reference element regions is associated with a corresponding target region of a plurality of target regions on the two-dimensional plane; determining a projection target position of the handheld object based on an initial position of the reference element of the handheld object on the two-dimensional plane and the first offset; and outputting a user interaction experience in response to determining that the projection target position corresponds to a target position. 22. The method of claim 21, further comprising receiving the initial position of the reference element of the handheld object projected onto the two-dimensional plane. 23. The method of claim 22, further comprising: determining a second offset along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arc by dividing the arc in the three-dimensional space into a plurality of additional reference element regions, and wherein each of the plurality of additional reference element regions is associated with an additional corresponding target region among a plurality of additional target regions on the two-dimensional plane; and determining the projected target position of the handheld object based on the initial position of the reference element of the handheld object, the first offset, and the second offset. 24. The method of claim 23, wherein each of the plurality of additional reference element regions is of the same size, and wherein each of the plurality of additional target regions is of a different size. 25. The method of claim 23, wherein a first additional target region among the plurality of additional target regions is farther from the reference element than a second additional target region among the plurality of additional target regions, and wherein the first additional target region is larger in size than the second additional target region. 26. The method of claim 23, wherein each of the plurality of additional reference element regions is of a different size, and wherein each of the plurality of additional target regions is of the same size. 27. The method of claim 23, wherein a first additional reference element region of the plurality of additional reference element regions is farther away from the two-dimensional plane than a second additional reference element region of the plurality of additional reference element regions, and its28. The method of claim 23, wherein determining the projection target position comprises shifting the initial position of the reference element along the first axis by the first offset, shifting the initial position of the reference element along the second axis by the second offset, or both. 29. The method of claim 23, wherein the first offset along the first axis comprises a horizontal offset in the two-dimensional plane, and the second offset along the second axis comprises a vertical offset in the two-dimensional plane. 30. The method of claim 23, wherein determining the first offset comprises performing regression analysis to fit test data to a first equation, and wherein determining the second offset comprises performing additional regression analysis to fit the test data to a second equation. 31. The method of claim 30, wherein the first equation and the second equation are each third-order polynomial equations. 32. The method of claim 31, wherein the first polynomial equation compensates for the distortion along the first axis associated with the corresponding target area and the arc in the two-dimensional plane, and wherein the second polynomial equation compensates for the distortion along the second axis associated with the corresponding additional target area and the arc in the two-dimensional plane. 33. The method of claim 21, wherein each of the plurality of reference element areas is of the same size, and wherein each of the plurality of target areas is of a different size. 34. The method of claim 21, wherein a first target area of ​​the plurality of target areas is farther from the reference element than a second target area of ​​the plurality of target areas, and wherein the first target area is larger in size than the second target area. 35. The method of claim 21, wherein each of the plurality of reference element areas is of a different size, and wherein each of the plurality of target areas is of the same size. 36. The method of claim 21, wherein a first reference element region of the plurality of reference element regions is farther from the two-dimensional plane than a second reference element region of the plurality of reference element regions, and wherein the first reference element region is smaller in size than the second reference element region. 37. A system comprising: an output device configured to output a user interactive experience; a controller having one or more processors and a memory storing machine-readable instructions configured to cause the one or more processors to: determine a first offset along a first axis of a two-dimensional plane, the first offset compensating for a displacement along the first axis associated with the arc by dividing an arc of movement of a reference element of a handheld object in three-dimensional space into a plurality of reference element regions.Distortion, and wherein each of the plurality of reference element regions is associated with a corresponding target region among a plurality of target regions on the two-dimensional plane; determining a projection target position of the handheld object based on an initial position of the reference element of the handheld object on the two-dimensional plane and a first offset; and causing the output device to output a user interaction experience in response to determining that the projection target position corresponds to a target position. 38. The system of claim 37, wherein the machine-readable instructions are configured to cause the one or more processors to receive the initial position of the reference element of the handheld object projected on the two-dimensional plane. 39. The system of claim 38, wherein the machine-readable instructions are configured to cause the one or more processors to: determine a second offset along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arc in the three-dimensional space by dividing the arc into a plurality of additional reference element regions, and wherein each of the plurality of additional reference element regions is associated with an additional corresponding target region among a plurality of additional target regions on the two-dimensional plane; and determining the projection target position of the handheld object based on the initial position of the reference element of the handheld object on the two-dimensional plane, the first offset, and the second offset. 40. One or more tangible non-transitory computer-readable media, comprising instructions that, when executed by at least one processor, cause the at least one processor to: receive an initial position of a reference element of a handheld object projected onto a two-dimensional plane; determine a first offset along a first axis of the two-dimensional plane, the first offset compensating for distortion along the first axis associated with the arc by dividing an arc of movement of the reference element of the handheld object in three-dimensional space into a plurality of reference element regions; determine a second offset along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arc by dividing the arc in three-dimensional space into a plurality of additional reference element regions, and wherein each of the plurality of additional reference element regions is associated with an additional corresponding target region in a plurality of additional target regions in the two-dimensional plane; determine a projection target position of the handheld object based on the initial position of the reference element of the handheld object on the two-dimensional plane, the first offset, and the second offset; and output a user interaction experience in response to determining that the projection target position corresponds to a target position. Claims 5 / 5, page 6, CN 121490366 A: System and method for determining the projected target position of a handheld object

[0001] Cross-reference to related applicationsThis application is a divisional application of Chinese Patent Application No. 202080067328.9, entitled "System and Method for Determining Projected Target Position of a Handheld Object," which entered the Chinese national phase on March 25, 2022. This Chinese patent application claims the benefit of U.S. Provisional Application No. 62 / 905901, entitled "System and Method for Determining Projected Target Position of a Handheld Object," filed on September 25, 2019. Both the Chinese patent application and the provisional application are hereby incorporated in their entirety by reference for all purposes. Background Art

[0002] This disclosure generally relates to a handheld object for aiming, and more particularly to determining the projected target position of a handheld object.

[0003] This section is intended to introduce the reader to various technical aspects that may relate to various aspects of this disclosure, which are described and / or claimed below. This discussion is intended to help provide the reader with background information to facilitate a better understanding of the various aspects of this disclosure. Therefore, it is understood that these statements will be read from this perspective and not as an admission of prior art.

[0004] A handheld object can be used to aim at or select a target. For example, in a theme park setting, a customer can use a handheld object to aim at an animated character at an attraction, and in response to detecting this, the system may cause the animated character to output a user interactive experience (e.g., wag its tail). However, it is now recognized that certain physical characteristics related to the user's body can make it difficult to accurately determine when the user is aiming at a target.

[0005] These and other features, aspects, and advantages of the present disclosure will become more apparent when the following detailed description is read with reference to the accompanying drawings, in which the same characters denote the same parts throughout the drawings, wherein: FIG1 is a diagram of a user aiming a handheld object at a target according to an embodiment of the present disclosure; FIG2 is a block diagram of a theme park attraction system according to an embodiment of the present disclosure; FIG3 is a diagram of a user aiming a handheld object at a calibration position according to an embodiment of the present disclosure; FIG4 is a diagram of an example of applying one or more translation factors to the subsequently detected position of a reference element of the handheld object of FIG3 according to an embodiment of the present disclosure; FIG5 is a diagram of an example of applying a scaling factor to the subsequently detected position of a reference element of the handheld object of FIG3 according to an embodiment of the present disclosure; FIG6 is a diagram of a user aiming a handheld object at different targets of the system according to an embodiment of the present disclosure; FIG7 is a diagram of multiple reference element areas of different sizes and multiple projection target areas of uniform size according to an embodiment of the present disclosure; FIG8 is a diagram of multiple reference element areas of uniform size and multiple projection target areas of different sizes according to an embodiment of the present disclosure; Figure 9 is a flowchart of a process for determining the projection target position of the handheld object of Figure 3 according to an embodiment of the present disclosure; and page 1 / 14 of the specification 7 CNFigure 10 is a flowchart of a process for compensating for distortion caused by the difference in shape between the arcuate nature of the user's arm movement and a two-dimensional plane, according to an embodiment of the present disclosure. Detailed Description

[0006] One or more specific embodiments will be described below. To provide a concise description of these embodiments, not all features of the actual implementation are described in the specification. It should be appreciated that, as in any engineering or design project, many implementation-specific decisions must be made to achieve the developer's specific goals, which may vary depending on the implementation, such as compliance with system-related constraints and business-related constraints. Furthermore, it should be appreciated that such development efforts may be complex and time-consuming, but for those skilled in the art who benefit from the present disclosure, such development efforts will be nothing more than routine tasks of design, fabrication, and manufacturing.

[0007] When introducing elements of the various embodiments of the present disclosure, the articles “a,” “an,” and “the” are intended to mean the presence of one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements may exist in addition to those listed. Furthermore, it should be understood that references to “one embodiment” or “an embodiment” in this disclosure are not intended to be construed as excluding the existence of additional embodiments that also incorporate the described features.

[0008] This disclosure generally relates to a handheld object for aiming, and more particularly, to determining the projected target position of the handheld object. In particular, a reference element can provide an indication of where the handheld object is being aimed. For example, in a theme park setting, a user can use the handheld object to aim at an animated object (e.g., a robot, or in other cases, an animated character) at an attraction, and, in response to detecting the position of the reference element, the animated object can output a user interactive experience (e.g., wagging its tail). As another example, a user can aim at a word on a poster, and, in response to detecting the position of the reference element, a nearby speaker can output the speech of that word. As yet another example, a user can aim at an image of a person on an electronic display, and, in response to detecting the position of the reference element, the display can play a video showing the person moving in the image.

[0009] Currently disclosed systems and methods involve using a camera to determine the position of a reference element in a two-dimensional plane perpendicular to the camera's direction. The camera can detect the reference element of the handheld object, which can be made of a material that is more easily detected by the camera (e.g., a reflective material). The position of the reference element can be used to determine the target location where the user was aiming the handheld object. However, in some systems, the user's sense of where they are aiming the handheld object is limited.It is known that the projected position where the user is aiming may not match the view determined by the camera. This may be due to a variety of factors, including dominance of one eye over the other (e.g., right eye dominance or left eye dominance), head tilt, weight shift, tilting to one side or the other, etc. Any combination of these factors may cause the user to perceive where they are planning to move when their hand is aiming the handheld object in the same position. It should be noted that the camera is an example of various light detectors that can be used according to the present embodiments. Therefore, reference to the camera indicates other light detectors that can be used by embodiments of this disclosure.

[0010] The currently disclosed systems and methods include providing a calibration point on a two-dimensional plane where the user can aim the handheld object. The position of a reference element relative to the two-dimensional plane can be determined as an initial position, and one or more translation factors can be determined based on the difference between the initial position and the calibration point. That is, the calibration point can be associated with the position where the user perceives they are aiming the handheld object, and the initial position of the reference element can be associated with the position of the reference element on the two-dimensional plane as seen from the viewpoint of the camera. The difference between the two can be used to translate the subsequently detected position of the reference element in the two-dimensional plane from the camera's viewpoint to the projected target position (e.g., corresponding to the position where the user perceives they are aiming at or intend to aim at). That is, one or more translation factors can compensate for the difference between the user's perception of where they are aiming the handheld object and the camera's determination of where the reference element is located in the two-dimensional plane.

[0011] Furthermore, users use their arms to move and aim the handheld object, and the arm can act as the radius or spherical segment of a sphere in the interaction model, where their shoulder is considered the center of the sphere. When a user moves the handheld object or aims it at different targets, the corresponding position of the reference element of the handheld object may differ between users, even when aiming at the same target. This may be due to the different arm lengths of the users.

[0012] Therefore, the currently disclosed systems and methods determine the height of a reference element (e.g., above the ground) based on its initial position, and estimate the user's height based on the height of the reference element. The user's arm length can be estimated from the user's height, and this arm length can be used to determine one or more scaling factors. These scaling factors can scale or multiply the subsequently detected position of the reference element in a two-dimensional plane as seen from the camera's viewpoint to more accurately determine the position of the projected target (e.g., corresponding to the position where the user perceives they are aiming at or intend to aim at). In this way, the one or more scaling factors can compensate for differences in user arm lengths.

[0013] When the position of a subsequent reference element is detected by the camera, one or more translation factors and one or more scaling factors can be applied to the subsequent reference element position to determine the projected target position relative to a two-dimensional plane. Current embodiments may include a processor that operates to analyze data captured and transmitted by the camera to provide relevant data such as translation factors, scaling factors, the projected target position relative to a two-dimensional plane, etc.

[0014] Additionally, since a user's arm acts as the radius or cross section of a sphere, with their shoulder as the center, a user can move a handheld object in an arc or circular manner. However, a camera determining the position of a reference element of a handheld object on a flat two-dimensional plane may distort the determined position of the reference element due to the shape difference between the arcuate movement of the handheld object in space and the flat two-dimensional plane, which can be detected by the camera.

[0015] Therefore, currently disclosed systems and methods can determine one or more offsets to be applied to the projected target position to compensate for this distortion. These one or more offsets can shift the projected target position to increase or lengthen the distance between the projected target position and the initial position in order to compensate for the shape difference between the arcuate nature of the user's arm movement and the flat two-dimensional plane. For example, the one or more offsets can be determined using polynomial regression, which fits test data to one or more polynomial equations (e.g., third-order polynomial equations).

[0016] In some embodiments, multiple reference element regions (e.g., where reference elements are positioned along an arc based on the user's arm) can be determined to correspond to multiple projection target regions (e.g., projected onto a two-dimensional plane). Each projection target region can correspond to a corresponding set of polynomial equations that can accurately compensate for distortions applicable to that projection target region. Accordingly, a camera can detect reference elements in the reference element regions, determine that a corresponding projection target region corresponds to a reference element region, and the corresponding set of polynomial equations corresponding to the corresponding projection target region can be used to determine the position to be applied to the reference element to compensate for one or more offsets of the distortion. In such embodiments, the multiple reference element regions can be of different sizes (e.g., the reference element regions decrease in size the farther they are from the two-dimensional plane), while the multiple projection target regions are of the same size, or the multiple reference element regions can be of the same size, while the multiple projection target regions are of different sizes (e.g., the projection target regions increase in size the farther they are from the reference elements).

[0017] By way of description, FIG1 is a diagram of a user 10 aiming a handheld object 12 at a target 14 according to an embodiment of the present disclosure. The target 14 may be a physical object, a drawing, a photograph, a graphic, etc. In some cases, the target 14 may be a physical object, a drawing, a photograph, a graphic, etc.The image output by the display. Target 14 may be printed, etched, written, projected, attached, or otherwise displayed on structure 15. The user's perception is indicated by the first dashed line 16. That is, the user 10 perceives that they are aiming the handheld object 12 at target 14, and specifically at target location 17. However, due to certain human factors (such as dominance of one eye over the other, head tilt, weight shift, tilting to one side or the other, etc.), regardless of the user's perception or intention, the user 10 actually aims the handheld object 12 at the actual target location 18 as indicated by dashed line 19.

[0018] The handheld object 12 may represent or include any suitable object that the user 10 can use to aim or point at target 14, such as a stick, pencil, toy or model of a gun or weapon, wand, etc. The handheld object 12 may include a reference element 20, which may facilitate the determination of where the user 10 is aiming. Specifically, camera 22 can detect the position of reference element 20, and reference element 20 can be made of a material or device that makes it easier for camera 22 to detect reference element 20. For example, reference element 20 can be made of a reflective material (e.g., reflective glass beads, microprisms, or encapsulated lenses sealed to a fabric or plastic substrate), a metal strip, etc. In another example, reference element 20 can include an identifier (e.g., a unique graphic design, barcode, quick response (QR) code, etc.) that enables camera 22 to identify reference element 20. As illustrated, reference element 20 can be located at end 24 of the handheld object 12 opposite to end 26, at which end 26 the user's hand 28 is holding the handheld object 12. This can facilitate determining that the user is holding the handheld object 12 in the direction they are aiming; however, reference element 20 can be positioned on any part of the handheld object 12 or even the user 10.

[0019] Camera 22 can detect the position 30 of reference element 20 relative to two-dimensional plane 32. Position 30 can be used to determine the target position 17 that the user 10 perceives they are aiming at or intend to aim at by applying one or more translation factors. As illustrated, the two-dimensional plane 32 may share the same plane as the structure 15; however, in some embodiments, the two-dimensional plane 32 and the structure 15 may not share the same plane. For example, the two-dimensional plane 32 and the structure 15 may be parallel to each other. Furthermore, in order for the camera 22 to detect the position 30 of the reference element 20, the structure 15 may be made translucent, transparent, or include any other suitable properties that enable the camera 22 to detect the position 30 of the reference element 20.

[0020] In particular, one or more translation factors may be applied to the position 30 of the reference element 20 to compensate for the user's perception of the target position 17.The difference between their perception of where the handheld object 12 is aimed and the camera's determination of where the reference element 20 is located on the two-dimensional plane 32. One or more translation factors may be determined during a calibration process in which the user 10 aims their handheld object 12 at a calibration point, and the camera 22 detects this initial position of the reference element 20 on the two-dimensional plane 32. One or more translation factors may represent one or more distances by which the initial position is shifted to result in the calibration point (e.g., relative to the two-dimensional plane 32). Additionally, one or more translation factors may mitigate or compensate for one eye dominance over another (e.g., right eye dominance or left eye dominance), head tilt, weight shift, tilting to one side or the other, etc.

[0021] Furthermore, one or more scaling factors may be applied to the position 30 of the reference element 20 to account for or compensate for differences in the user's arm length. That is, the user uses their arm to move and aim the handheld object 12, and the arm may act as the radius or cross section of a sphere, with their shoulder as the center of the sphere. When a user moves the handheld object 12 or aims it at different targets, the corresponding position of the reference element 20 of the handheld object 12 may differ between users, even when aiming at the same target, due to differences in arm length.

[0022] Therefore, the height of the reference element 20 (e.g., above the ground) can be determined based on the initial position of the reference element 20, and the user's height can be estimated based on the height of the reference element 20. The user's arm length can be estimated from the user's height, and the user's arm length can be used to determine one or more scaling factors. One or more scaling factors can be used to scale or multiply the position 30 of the reference element 20 detected by the camera 22 on the two-dimensional plane 32.

[0023] In addition, one or more offsets can be applied to the position 30 of the reference element 20 to generate a projected target position of the handheld object 12 to compensate for distortion caused by the arcuate or circular movement of the user's arm. That is, this distortion may be caused by the difference in shape between the arcuate movement and the camera's detection of the position 30 of the reference element 20 on the flat two-dimensional plane 32. One or more offsets can shift the projected target position to increase or lengthen the distance between the projected target position and the initial position in order to compensate for the difference in shape between the arcuate nature of the user's arm movement and the flat two-dimensional plane. For example, one or more offsets can be determined using polynomial regression that fits test data to a polynomial equation (such as a third-order polynomial equation).

[0024] In this way, a projected target position of the handheld object 12 can be generated that can highly match the target position 17 where the user 10 perceives that they are aiming the handheld object 12 at. Advantageously, unlike some otherThe system uses only one calibration point to determine the translation factor, scaling factor, and offset, and accurately determines the projected target position of the handheld object 12. However, in other applications (e.g., aiming devices used in performances), reducing calibration time may be less important because calibration can occur before the actual performance (e.g., during the preparation phase) and will not be observed by the audience or customers. However, in the current situation (e.g., at a theme park attraction), it may be important to create an immersive user experience by hiding the calibration process or preventing the user 10 from noticing that calibration is in progress. Accordingly, reducing the calibration process to a single point (e.g., aiming the handheld object 12 at a single calibration point) can be used to enhance or improve the user experience.

[0025] With this in mind, FIG2 is a block diagram of a theme park attraction system 40 according to an embodiment of the present disclosure. The theme park attraction system 40 enables the user 10 to aim the handheld object 12 at various targets 14 and outputs a user interaction experience based on determining that the user 10 is aiming the handheld object 12 at the target 14. For example, the theme park attraction system 40 may include settings with characters popular with children, settings themed around television or movies, shooting ranges, sets of targets, etc.

[0026] The theme park attraction system 40 may include a handheld object 12 with a reference element 20, such as one held and manipulated by the user 10. The theme park attraction system 40 may also include a user interaction system 42, which includes a camera 22 that detects the position of the reference element on a two-dimensional plane 32. The theme park attraction system 40 may also include a projection position determination system 44 that determines the projection target position of the handheld object 12. In particular, the projection target position may represent the position on the two-dimensional plane 32 where the user 10 perceives that they are aiming at or intend to aim at. In fact, the closer the projection target position is to the target position 17, the more accurate the projection target position.

[0027] The projection position determination system 44 may include a controller 46 having one or more processors (described as a single processor 48) and one or more memories or storage devices (described as a single memory device 50). Processor 48 can execute software programs and / or instructions stored in memory device 50, which facilitate the determination of the projection target position of handheld object 12. Furthermore, processor 48 may include multiple microprocessors, one or more "general-purpose" microprocessors, one or more application-specific microprocessors, and / or one or more application-specific integrated circuits (ASICs). For example, processor 48 may include one or more Reduced Instruction Set Computing (RISC) processors. Memory device 50 may store information such as control software, lookup tables, configuration data, etc. Memory device 50 may include tangible, non-transitory machine-readable media, such as volatile memory.Storage devices (e.g., random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM)), flash memory, one or more hard disk drives and / or any other suitable optical storage medium, magnetic storage medium or solid-state storage medium. The memory device 50 can store a wide variety of information and can be used for a variety of purposes, such as facilitating instructions for the projection target position of the handheld object 12.

[0028] The projection position determination system 44 may also include reference element position detection logic 52 for determining the position of the reference element 20 on the two-dimensional plane 32. In particular, the projection position determination system 44 can be communicatively coupled to the user interaction system 42 by any suitable means, such as via wired communication or through a communication network using wireless communication protocols or technologies (e.g., radio, Bluetooth, WiFi, infrared, Ethernet, Thread, ZigBee, Z-Wave, KNX, mobile and / or microwave). The reference element position detection logic 52 can thus receive captured images (e.g., videos) from the camera 22 showing the reference element 20 on the two-dimensional plane 32. Reference element position detection logic 52 can determine the position of reference element 20 on two-dimensional plane 32, as expressed by, for example, a two-dimensional coordinate system (e.g., x and y).

[0029] The projection position determination system 44 may also include transformation logic 54, which transforms the position of reference element 20, as determined by reference element position detection logic 52, into a projection target position relative to two-dimensional plane 32. Transformation logic 54 includes translation logic 56, which determines one or more translation factors that compensate for the difference between the user's perception of where they are aiming the handheld object 12 and the camera's determination of where reference element 20 is located on two-dimensional plane 32.

[0030] In particular, translation logic 56 may determine one or more translation factors by performing a single-point calibration process. The process includes: receiving a calibration position on a two-dimensional plane 32; receiving the position of a reference element 20 on the two-dimensional plane 32 (e.g., corresponding to when the user 10 aims the handheld object 12 at the calibration position); and determining one or more translation factors based on the positional difference between the calibration position and the position of the reference element 20.

[0031] FIG3 is a diagram of the user 10 aiming the handheld object 12 at a calibration position 80 according to an embodiment of the present disclosure. The calibration position 80 may correspond to a physical object, a drawing, a photograph, a graphic, etc. In some cases, the calibration position 80 may correspond to an image output by a display. The user 10 may be prompted by instructions provided in any suitable format (e.g., written, etched, printed, attached, or displayed on structure 15). The calibration position 80 may be provided so that the user can similarly aim at their handheld object.The arm is positioned to enable controlled detection of the user's height, while also allowing the projection position determination system 44 of FIG2 to determine the difference between the user's perception of where they are aiming the handheld object 12 and where the user 10 is actually aiming the handheld object 12. For example, the calibration position 80 can be positioned so that the user 10 can extend their arm 82 as close as possible to parallel to the ground 84, at an angle relative to a plane parallel to the ground, etc. In some embodiments, the calibration position 80 can be customized for the user's height. That is, in some embodiments, the calibration position 80 can be positioned lower on the structure 15 for users sitting in a vehicle (such as a wheelchair, personal electric vehicle, stroller, etc.). As another example, the calibration position 80 can be positioned higher on the structure 15 for adults than for children, the calibration position 80 can be positioned higher on the structure 15 for male users than for female users, etc.

[0032] Accordingly, the calibration position 80 can be predetermined and known by the projection position determination system 44. When prompted, user 10 can extend their arm 82 and aim the handheld object 12 at calibration position 80. However, due to distortion effects caused by the human body (such as one eye dominance over the other, head tilt, weight shift, tilting to one side or the other, the user's choice of hand to hold the object 12 (e.g., right hand vs. left hand, physical limitations (e.g., physical limitations affecting range of motion), whether the user's movement may change due to obstruction (e.g., carrying a backpack or holding a child), etc.), although the user perceives or intends to aim the object 12 at the calibration position 80 as indicated by dashed line 85, the user 10 may actually aim the object 12 at another position (e.g., actual calibration position 86) as indicated by dashed line 88.

[0033] Camera 22 detects the position 90 of reference element 20 on two-dimensional plane 32 and sends an indication of position 90 to projection position determination system 44. Translation logic 56 (which may be part of a human interaction model) can then determine the positional difference between the position 90 of reference element 20 and the predetermined calibration position 80, which can be expressed in two dimensions (e.g., x). The coordinates are expressed as y and y). Translation logic 56 can use this difference to generate one or more translation factors, which can be applied to the subsequently detected position of reference element 20 to shift the subsequently detected position of reference element 20 and determine the subsequent projected target position of the handheld object 12 corresponding to the position where user 10 intends to aim the handheld object 12. The translation factors can be provided in the form of a transformation matrix, which can be applied to the subsequently detected position of reference element 20 to generate the projected target position of reference element 20, as shown below: Specification 6 / 14Page 12 CN 121490366 A Equation 1 Where: x = horizontal component of the position 90 of the reference element 20 on the two-dimensional plane 32; y = vertical component of the position 90 of the reference element 20 on the two-dimensional plane 32; X = horizontal difference between the reference element 20 and the calibration position 80 on the two-dimensional plane 32; Y = vertical difference between the reference element 20 and the calibration position 80 on the two-dimensional plane 32; x' = horizontal component of the projected target position of the handheld object 12 on the two-dimensional plane 32; and y' = vertical component of the projected target position of the handheld object 12 on the two-dimensional plane 32.

[0034] For example, FIG4 is a diagram of an example of applying one or more translation factors to the subsequently detected position 120 of the reference element 20 according to an embodiment of the present disclosure. As illustrated, during calibration, the position 90 of the reference element 20 is 2 units (e.g., cm) to the right of the calibration position 80 and 1 unit (e.g., cm) upward from the calibration position 80. Accordingly, the translation factor may include +2 in the horizontal direction and +1 in the vertical direction. Thus, in the transformation matrix, X may be set to +2, and Y may be set to +1. Translation logic 56 may apply the transformation matrix to the subsequently detected position 120 of the reference element 20 (e.g., [4, 2]) to shift the subsequently detected position 120 to the right by 2 units and upward by 1 unit to generate a projected target position 122 at 6 units to the right of the calibration position 80 and 3 units upward (e.g., [6, 3]). Thus, translation logic 56 may compensate for the difference between the user's perception of where they are aiming the handheld object 12 and the camera's determination of where the reference element 20 is located on the two-dimensional plane 32.

[0035] Returning to FIG2, transformation logic 54 may also include scaling logic 58, which determines one or more scaling factors to compensate for the difference between the user's arm length. That is, as shown in FIG3, users 10 use their arms 82 to move and aim the handheld object 12, the arms 82 serving as the radius or cross section 92 of a sphere, with their shoulders serving as the center 94 of the sphere. When users 10 move the handheld object 12 to aim at different targets, the corresponding position of the reference element 20 of the handheld object 12 may differ between users 10, even when aiming at the same target, due to the different arm lengths of users 10.

[0036] In particular, scaling logic 58 may determine one or more scaling factors based on the position 90 of the reference element 20 detected by camera 22 during the calibration process. The height 96 of camera 22 above ground 84 can be predetermined and known by scaling logic 58. Thus, scaling logic 58 may determine the reference element 20 based on the position 90 of the reference element 20 and the predetermined height 96.The height 98 of element 20 above the ground 84. Based on the height 98 of reference element 20, the user height estimation logic 60 of scaling logic 58 can determine the user's height 100. Specifically, test or sample data of the position 90 of reference element 20 and the height of user 10 when user 10 aims handheld object 12 at calibration position 80 can be collected. The height 102 of the position 90 of reference element 20 can be correlated with the height of user 10, and scaling logic 58 can estimate the user's height 100 based on this predetermined correlation and the height 98 of reference element 20. The model used to identify the correlation can be populated with a table of standard correlations between height and reach (e.g., the ratio between height and arm length for various body types in a population).

[0037] The user arm length estimation logic 62 of scaling logic 58 can then estimate the user's arm length 104 based on the user height 100. The estimation can be based on a predetermined correlation between arm length 104 and user height 100 (e.g., an algorithm or table based on empirical data). The predetermined correlation can be determined based on test or sample data, scientific data related to human proportions, and / or any other suitable source. Specification 7 / 14 page 13 CN 121490366 A

[0038] The scaling logic 58 can determine one or more scaling factors based on the user's arm length 104. For example, when pointing away from the initial position (e.g., calibration position 80), the camera 22 can detect the position of the reference element 20 closer to the initial position compared to a user 10 with a shorter arm length 104. Accordingly, the scaling logic 58 can determine a larger scaling factor for a user 10 with a longer arm length 104 compared to a user 10 with a shorter arm length 104. The scaling logic 58 can apply one or more scaling factors to subsequently detected positions of the reference element 20 to scale (e.g., shrink or enlarge) that position to generate a projected target position for the reference element 20. The scaling factor may include horizontal and vertical components, provided in the form of a transformation matrix, and interpolated into a transformation matrix that includes the translation factor from Equation 1 above, as shown below: Equation 2 Where: k1 = horizontal scaling factor generated based on user arm length 104; and k2 = vertical scaling factor generated based on user arm length 104.

[0039] The values ​​of scaling factors k1 and k2 may be determined based on relevant test or sample data collected from users 10 who aim the handheld object 12 at various targets and the arm length 104 of those users 10. For example, scaling logic 58 may determine the height of reference element 20 above ground 84 based on image data received from camera 22 (e.g., a first image or calibration image of the image).The height 98 of the reference element 20 is 1.25 meters. The user height estimation logic 60 can determine the user's height 100 as approximately 1.8 meters based on the height 98 of the reference element 20. The user arm length estimation logic 62 can determine the user's arm length 104 as 0.6 meters based on the user's height 100. The scaling logic 58 can then determine a horizontal scaling factor k1 of 1.5 and a vertical scaling factor k2 of 1.75 based on the user's arm length 104. Therefore, the scaling logic 58 can generate a transformation matrix in Equation 2, where k1 = 1.5 and k2 = 1.75, and the projection position determination system 44 can apply the transformation matrix to the subsequently detected position of the reference element 20 to generate a projected target position where the user 10 intends to aim the handheld object 12, which compensates for the difference in the aspect of the user's arm length 104.

[0040] For example, FIG5 is a diagram of an example of applying scaling factors to the subsequently detected position 120 of the reference element 20 according to an embodiment of the present disclosure. As illustrated, the subsequently detected position 120 of reference element 20 is 4 units (e.g., cm) to the right of calibration position 80 and 4 units (e.g., cm) upward from calibration position 80 (e.g., [4, 4]). Applying the transformation matrix of Equation 2, which has a horizontal scaling factor k1 = 1.5 and a vertical scaling factor k2 = 1.75, to the subsequently detected position 120 results in the subsequently detected position 120 being scaled horizontally by a factor of 1.5, thus generating a projected target position 130 6 units to the right of calibration position 80, and vertically scaled by a factor of 1.7, thus generating a projected target position 130 7 units (e.g., cm) upward (e.g., [6, 7]). Thus, scaling logic 58 can compensate for the difference in the length of the user's arm 104.

[0041] Returning to FIG2, the projection position determination system 44 may include arc distortion compensation logic 64, which compensates for the difference in shape between the arcuate nature 92 of the user's arm movement and the flat two-dimensional plane 32. For example, Figure 6 shows a user 10 aiming a handheld object 12 at different targets. As explained, the angle ϴ formed between the first position 140 and the second position 142 of the user's arm 82 is the same as that between the third position 144 and the fourth position 146 of the user's arm 82. However, when viewed and captured by the camera 22 on the two-dimensional plane 32, the distance h0 between the first reference element position 148 corresponding to the first position 140 of the user's arm 82 and the second reference element position 150 corresponding to the second position 142 of the user's arm 82 is different from (e.g., greater than) the distance h0 between the third reference element position 152 corresponding to the third position 144 of the user's arm 82 and the distance h0 between the third reference element position 152 corresponding to the third position 144 of the user's arm 82 and the distance h0 between the third reference element position 152 corresponding to the fourth position 146 of the user's arm 82. (Page 8 / 14 of the specification)CN 121490366 A Distance h1 between four reference element positions 154.

[0042] Accordingly, the arc distortion compensation logic 64 can determine one or more offsets to be applied to the projected target position to compensate for the distortion. The one or more offsets can shift the projected target position to increase or lengthen the distance between the projected target position and the initial position (e.g., calibration position 80) to compensate for the difference in shape between the arcuate nature 92 of the user's arm movement and the flat two-dimensional plane 32. For example, the one or more offsets can be determined using regression analysis, which fits test or sample data from the user 10 aiming the handheld object 12 at various targets (e.g., where the reference element 20 is along arc 92) to an equation. In some embodiments, the arc distortion compensation logic 64 can fit the test data to a polynomial equation (e.g., a third-order polynomial equation); however, any suitable order or type of equation can be used. For example, the first third-order polynomial equations (Equations 3 and 4 below) can be used to determine the horizontal offset to compensate for the distortion in the horizontal direction applied to the projected target position, and the second third-order polynomial equations (Equations 5 and 6 below) can be used to determine the vertical offset to compensate for the distortion in the vertical direction applied to the projected target position: Equation 3 (which may also be expressed as: Equation 4 Equation 5 (which may also be expressed as: Equation 6, where: x offset = horizontal offset to be applied to the projected target position; y offset = vertical offset to be applied to the projected target position; x = horizontal component of the projected target position; y = vertical component of the projected target position; and ai, bi, ci, a, b, c, d, e, f, g, h, k and l = constants determined using regression analysis, wherein each constant may vary from equation to equation (e.g., the constant a in Equation 4 may be different from the constant a in Equation 6).

[0043] The horizontal component of the projected target position can be measured as the horizontal distance away from the initial position (e.g., corresponding to calibration position 80 and / or when user 10 aims the handheld object 12 directly at camera 22), while the vertical component of the projected target position can be measured as the vertical distance away from the initial position. As previously mentioned, for any of the polynomial equations 3-6, the constants ai, bi, ci, a, b, c, d, e, f, g, h, k, and l can be determined by fitting the test or sample data to the polynomial equations using polynomial regression analysis (and may differ between the equations). Accordingly, one or more offsets can be determined for each projected target position as user 10 moves and aims the handheld object 12.

[0044] However, for each projected target position, when user 10 moves and aims the handheld object 12, the offsets can be determined for each projected target position.Applying any of Equations 3-6 to determine horizontal and vertical offsets can be time-consuming and consume excessive computational resources (e.g., processing, memory, storage devices, or network resources). Accordingly, to more efficiently compensate for the shape difference between the arcuate nature 92 of the user's arm movement and the flat two-dimensional plane 32, in some embodiments, the arcuate distortion compensation logic 64 can divide the arc 92 where the reference element 20 can be located into multiple reference element regions, each of which can correspond to a corresponding projection target region (e.g., projected onto a two-dimensional plane). Each projection target region can correspond to a corresponding set of polynomial equations that can accurately compensate for distortions applicable to that projection target region. Accordingly, the camera 22 can detect the reference element 20 within the reference element region, the arcuate distortion compensation logic 64 can determine the corresponding projection target region corresponding to the reference element region, and the arcuate distortion compensation logic 64 can apply the corresponding set of polynomial equations corresponding to the corresponding projection target region to the position of the reference element to determine one or more offsets to be applied to the reference element to compensate for the distortion. In such embodiments, the multiple reference element regions may be of different sizes (e.g., the farther the reference element region is from the two-dimensional plane 32, the smaller the reference element region becomes), while the multiple projection target regions are of the same size, or the multiple reference element regions may be of the same size, while the multiple projection target regions are of different sizes (e.g., the farther the projection target region is from the reference element 20, the larger the projection target region becomes).

[0045] FIG7 is a diagram of multiple reference element regions 170 of different sizes and multiple projection target regions 172 of uniform size according to an embodiment of the present disclosure. As illustrated, the first reference element region 174, which is closest to the two-dimensional plane 32, is the largest in size, followed by the second reference element region 176, which is also closest to the two-dimensional plane 32, and is the largest in size (but smaller than the first reference element region 174), followed by the third reference element region 178, which is also closest to the two-dimensional plane 32, and is the largest in size (but smaller than the second reference element region 176), and the fourth reference element region 180, which is also closest to the two-dimensional plane 32, is the largest in size (but smaller than the third reference element region 178). Although four reference element regions 170 are illustrated in Figure 7, it should be understood that any suitable number of reference element regions 170 of any suitable size can be imagined, wherein the farther the reference element region 170 is from the two-dimensional plane 32, the smaller its size becomes. Furthermore, each projection target region 172 is the same size as the other projection target regions 172, corresponding to the corresponding reference element region 170, and corresponding to the corresponding system of polynomial equations that generate the corresponding offsets (e.g., horizontal and vertical offsets) that can be applied to the position of the reference element 20. Specifically, with the correspondingEach set of polynomial equations corresponding to the projection target area 172 can have different sets of values ​​for the constants ai, bi, ci, a, b, c, d, e, f, g, h, k, and l, as provided in any of equations 3-6 (and may differ between equations). The further the reference element area 170 is from the two-dimensional plane 32, the smaller the size of the reference element area 170 becomes, while maintaining the same size of the projection target area 172, so that the arc distortion compensation logic 64 can compensate for the shape difference between the arcuate nature 92 of the user's arm movement and the flat two-dimensional plane 32 in an efficient and resource-saving manner.

[0046] FIG8 is a diagram of a plurality of reference element areas 190 of uniform size and a plurality of projection target areas 192 of different sizes according to an embodiment of the present disclosure. As illustrated, each reference element area 190 is of the same size. However, the first projection target area 194, closest to the reference element 20, is the smallest in size, followed by the second projection target area 196, which is also closest to the reference element 20 and is the smallest in size (but larger than the first projection target area 194). The third projection target area 198, also closest to the reference element 20, is the smallest in size (but larger than the second projection target area 196), and the fourth projection target area 200, also closest to the reference element 20, is the smallest in size (but larger than the third projection target area 198). Although four projection target areas 192 are illustrated in Figure 8, it should be understood that any suitable number of projection target areas 192 of any suitable size can be imagined, wherein the farther the projection target area 192 is from the reference element 20, the larger it becomes in size. Each projection target area 192 corresponds to a corresponding reference element area 190 and also corresponds to a corresponding system of polynomial equations for generating a corresponding offset (e.g., horizontal and vertical offset) that can be applied to the position of the reference element 20. Specifically, each set of polynomial equations corresponding to the corresponding projection target area 192 may have different sets of values ​​for the constants ai, bi, ci, a, b, c, d, e, f, g, h, j, k, and l, as provided in any of equations 3-6 (and may differ between equations). The further the projection target area 192 is from the reference element 20, the larger the size of the projection target area 192 becomes, while maintaining the same size of the reference element area 190, so that the arc distortion compensation logic 64 can compensate for the shape difference between the arcuate nature 92 of the user's arm movement and the flat two-dimensional plane 32 in an efficient and resource-saving manner.

[0047] It should be noted that, for simplicity, Figures 6-8 illustrate only the distortion caused by the shape difference between the arcuate nature 92 of the user's arm movement and the flat two-dimensional plane 32 in the vertical (e.g., y) direction. However, currentlyThe disclosed systems and methods envision compensating for distortion along any suitable direction: including the horizontal (e.g., x) direction, as indicated by Equation 3 (page 10 / 14, CN 121490366 A) and Equation 4, which provide a horizontal offset to compensate for distortion along the horizontal direction; and the vertical (e.g., y) direction, as indicated by Equations 5 and 6, which provide a vertical offset to compensate for distortion along the vertical direction.

[0048] Returning to FIG2, if the projection position determination system 44 determines that the projection target position corresponds to a target 14 printed, etched, written, attached, or otherwise displayed on the structure 15, then the output device 66 of the user interaction system 42 can output a user interaction experience. The output device 66 can be any suitable device capable of outputting the desired user interaction experience, such as an electronic display, speaker, virtual reality device, augmented reality device, actuator, and / or animation device (e.g., robot figure). Target 14 may be part of, fixed to, attached to, or include output device 66, or target 14 may be detachable from output device 66. For example, in a theme park setting, target 14 and output device 66 may both be animated objects of the attraction, and the animated object may output a user interactive experience (e.g., wagging its tail) in response to determining that the projected target position corresponds to the animated object. As another example, target 14 may be a word printed on a poster, and output device 66 may be a nearby speaker, and the nearby speaker may output the speech of the word in response to determining that the projected target position corresponds to the word printed on the poster. As yet another example, target 14 may be an image of a person on an electronic display, and output device 66 may be an electronic display, and the electronic display may play a video showing the person in the image performing a signature gesture in response to determining that the projected target position corresponds to the image of the person.

[0049] In view of this, FIG9 is a flowchart of a process 210 for determining the projected target position of a handheld object 12 according to an embodiment of the present disclosure. Process 210 can be performed by any suitable means capable of determining the projected target position of the handheld object 12 (such as any component of the projection position determination system 44, including controller 46, processor 48, reference element position detection logic 52, transformation logic 54, translation logic 56, scaling logic 58, user height estimation logic 60, and / or user arm length logic 62). While process 210 is described using steps performed in a specific sequence, it should be understood that the steps described herein are contemplated to be performed in a different sequence than that described, and that some described steps may be omitted or not all performed together. In some embodiments, process 210 can be performed using a processor (such as processor 48).This is implemented by instructions stored in a tangible, non-transitory computer-readable medium (such as memory device 50).

[0050] As illustrated, in process block 212, processor 48 receives an instruction to calibrate handheld object 12. This instruction may be in the form of an image captured by camera 22 (e.g., a first image of a video or a calibration image) that includes the presence of reference element 20 of handheld object 12. In some embodiments, a motion sensor or other suitable sensor capable of indicating that user 10 has entered the viewing area of ​​camera 22 with handheld object 12 having reference element 20 may provide this instruction.

[0051] In process block 214, processor 48 receives calibration position 80. In particular, calibration position 80 may be predetermined and known to processor 48, as calibration position 80 may be fixed to structure 15 or displayed by processor 48 on structure 15.

[0052] In process block 216, processor 48 receives the position of reference element 20 of handheld object 12. For example, camera 22 can provide an image of reference element 20 (e.g., a second or subsequent image of the image captured by camera 22). Processor 48 can then instruct reference element position detection logic 52 to determine the position of reference element 20 on two-dimensional plane 32.

[0053] In process block 218, processor 48 instructs translation logic 56 to determine one or more translation factors based on the position of reference element 20 and calibration position 80. One or more translation factors can compensate for the difference between the user's perception of where they are aiming the handheld object 12 and the camera's determination of where the reference element 20 is located on two-dimensional plane 32. In particular, translation logic 56 can determine one or more translation factors by performing a single-point calibration procedure. This procedure includes: receiving a calibration position on two-dimensional plane 32; receiving the position of reference element 20 on two-dimensional plane 32 (e.g., corresponding to when user 10 aims the handheld object 12 at the calibration position); and determining one or more translation factors based on the positional difference between the calibration position and the position of reference element 20. Specification 11 / 14 pages 17 CN 121490366 A

[0054] Translation logic 56 can use this difference to generate one or more translation factors, which can be applied to the subsequently detected position of reference element 20 to shift the subsequently detected position of reference element 20 and determine the subsequent projected target position of handheld object 12 corresponding to the position where user 10 intends to aim handheld object 12. The translation factor can be provided in the form of a transformation matrix, which can be applied to the subsequently detected position of reference element 20 to generate the projected target position of reference element 20, as shown in Equation 1.

[0055] In process block 220, processor 48 instructs user height estimation logic 60 to determine the position of reference element 20 based on the position of reference element 20.The height of user 10 is set to 100. In process block 222, processor 48 instructs user arm length estimation logic 62 to determine the arm length 104 of user 10 based on the height 100 of user 10.

[0056] In process block 224, processor 48 instructs scaling logic 58 to determine one or more scaling factors based on the arm length 104 of user 10. Scaling logic 58 may provide scaling factors as a transformation matrix of Equation 2 as shown above. Scaling factors may compensate for differences in the arm length 104 by scaling (e.g., multiplication) relative to the initial position (e.g., calibration position 80) at the position of reference element 20.

[0057] In process block 226, processor 48 instructs transformation logic 54 to determine the projection target position of handheld object 12 based on the position of reference element 20, one or more translation factors, and one or more scaling factors. In particular, transformation logic 54 may apply the transformation matrix of Equation 2, which includes one or more translation factors and one or more scaling factors, to the position of reference element 20 to generate the projection target position. That is, the projection target location can correspond to a location where the user 10 perceives that they are aiming at or intend to aim at.

[0058] In decision block 228, processor 48 determines whether the projection target location is associated with a user interaction element. The user interaction element can be any suitable target used as a trigger to perform a user interaction experience. For example, the user interaction element can include any feature of interest that the user 10 can expect to cause a user interaction experience to be performed when aiming with the handheld object 12.

[0059] If processor 48 determines that the projection target location is associated with a user interaction element, then in process block 230, processor 48 instructs user interaction system 42 to perform the corresponding user interaction experience using an appropriate output device 66. For example, output device 66 can be an animated object of a spot, and user interaction system 42 can cause the animated object to bark, meow, speak, move, blink, etc. As another example, output device 66 can be a speaker, and user interaction system 42 can cause the speaker to output sound, speech, music, etc. As another example, the output device 66 may be an electronic display, and the user interaction system 42 may cause the electronic display to display images, play videos, etc.

[0060] If the processor 48 determines that the projection target position is not related to the user interaction element, then in decision block 232, the processor 48 determines whether the next position of the reference element 20 has been received. If so, the processor 48 repeats process block 226 and determines the projection target position of the handheld object 12 based on the next position of the reference element 20 and the translation and scaling factors already determined from process blocks 218 and 224.

[0061] If the processor 48 determines that the next position of the reference element 20 has not yet been received, the processor 48 repeats process block 212 to receive the next instruction to calibrate the handheld object 12 (e.g., from the next user 10). In this way, process 210 can use single-point calibration to determine the projection target position of the handheld object 12 (e.g., without requiring the user 10 to aim the handheld object 12 at more than one point for calibrating the projection position determination system 44), which compensates for two differences: the difference between the user's perception of where they are aiming the handheld object 12 and the camera's determination of where the reference element 20 is located on the two-dimensional plane 32, and the difference in the aspect of the user's arm length 104.

[0062] Furthermore, the projection position determination system 44 can also compensate for distortion caused by the difference in shape between the arcuate nature 92 of the user's arm movement and the flat two-dimensional plane 32, as illustrated in FIG6. FIG10 is a flowchart of process 240 for compensating for this distortion according to an embodiment of this disclosure, pages 12 / 14, 18 CN 121490366 A. Process 240 can be performed by any suitable means capable of compensating for the distortion, such as any component of the projection position determination system 44, including controller 46, processor 48, and / or arc distortion compensation logic 64. While process 240 is described using steps performed in a specific sequence, it should be understood that the steps described herein are contemplated to be performed in a different sequence than described, and that some described steps may be omitted or not all performed together. In some embodiments, process 240 can be implemented by using a processor (such as processor 48) to execute instructions stored in a tangible, non-transitory computer-readable medium (such as memory device 50).

[0063] As illustrated, in process block 242, processor 48 receives the position of reference element 20 of the handheld object 12. In some embodiments, processor 48 may receive the projection target position of the handheld object 12.

[0064] In process block 244, processor 48 determines a horizontal offset based on the position of reference element 20 and a first polynomial equation. Specifically, processor 48 can receive the projected target position of the handheld object 12 or determine the projected target position using process 210 of FIG. 9. Processor 48 can then instruct arc distortion compensation logic 64 to apply polynomial equation 3 or 4 to the projected target position of the handheld object 12 to determine the horizontal offset.

[0065] In process block 246, processor 48 determines the vertical offset based on the position of reference element 20 and a second polynomial equation. Specifically, processor 48 can instruct arc distortion compensation logic 64 to apply polynomial equation 5 or 6 to the projected target position of the handheld object 12 to determine the vertical offset.

[0066] In process block 248, processor 48 determines the projection target position of the handheld object 12 based on the position, horizontal offset, and vertical offset of reference element 20. Specifically, processor 48 may instruct arc distortion compensation logic 64 to apply (e.g., add) a horizontal offset to the horizontal component (e.g., x-coordinate) of the projection target position and apply (e.g., add) a vertical offset to the vertical component (e.g., y-coordinate) of the projection target position to generate the projection target position.

[0067] In some embodiments, to more efficiently compensate for the shape difference between the arcuate nature 92 of the user's arm movement and the flat two-dimensional plane 32, arc distortion compensation logic 64 may divide the arc 92 where reference element 20 may be located into a plurality of reference element regions, each of which may correspond to a corresponding projection target region (e.g., projected onto a two-dimensional plane). Each projection target region may correspond to a corresponding set of polynomial equations that can accurately compensate for distortions applicable to that projection target region. Accordingly, camera 22 can detect reference element 20 in reference element area, arc distortion compensation logic 64 can determine the corresponding projection target area corresponding to the reference element area, and arc distortion compensation logic 64 can apply the corresponding polynomial equation system corresponding to the corresponding projection target area to the position of the reference element to determine the position to be applied to the reference element to compensate for one or more offsets of the distortion. In such an embodiment, multiple reference element areas may be of different sizes (e.g., the reference element area decreases in size as it is farther away from the two-dimensional plane 32), while multiple projection target areas are of the same size, as shown in FIG. 7, or multiple reference element areas may be of the same size, while multiple projection target areas are of different sizes (e.g., the projection target area increases in size as it is farther away from the reference element 20), as shown in FIG. 8.

[0068] In this way, process 240 can compensate for the arcuate nature 92 of the user's arm movement and the flat two-dimensional plane 32. Furthermore, to compensate for the differences between the user's perception of where they are aiming the handheld object 12 and the camera's determination of where the reference element 20 is located on the two-dimensional plane 32, the differences in the length of the user's arm 104, and the differences in shape between the arcuate nature 92 of the user's arm movement and the flat two-dimensional plane 32, process 240 of FIG. 10 may be performed before, after, or as part of process 210 of FIG. 9.

[0069] While the embodiments set forth in this disclosure may be susceptible to various modifications and alternatives, specific embodiments have been shown by way of example in the accompanying drawings and have been described in detail herein. However, it should be understood that this disclosure is not intended to be limited to the particular forms disclosed. This disclosure will cover the specification falling within the scope of this disclosure as defined by the following appended claims. (Pages 13 / 14, 19)CN 121490366 A All modifications, equivalents, and alternatives within the spirit and scope.

[0070] The techniques set forth and claimed herein are referenced and applied to substantial objects and specific examples of practical nature, which can arguably improve the technical field and are therefore not abstract, intangible, or purely theoretical. Moreover, if any claim appended to this specification contains one or more elements designated as “component for [implementing]...[function]” or “step for [implementing]...[function]”, such elements are intended to be interpreted according to 35 USC 112(f). However, for any claim containing elements designated in any other manner, such elements are intended not to be interpreted according to 35 USC 112(f). Instruction manual, page 14 / 14, 20 CN 121490366 A, Figure 1; Instruction manual, Figure 1 / 9, page 21 CN 121490366 A, Figure 2; Instruction manual, Figure 2 / 9, page 22 CN 121490366 A, Figure 3; Instruction manual, Figure 3 / 9, page 23 CN 121490366 A, Figure 4; Instruction manual, Figure 4 / 9, page 24 CN 121490366 A, Figure 6; Instruction manual, Figure 5 / 9, page 25 CN 121490366 A, Figure 7; Instruction manual, Figure 6 / 9, page 26 CN 121490366 A, Figure 8; Instruction manual, Figure 7 / 9, page 27 CN 121490366 A, Figure 9; Instruction manual, Figure 8 / 9, page 28 CN 121490366 A, Figure 10; Instruction manual, Figure 9 / 9, page 29 CN 121490366 A, Abstract: A projected target location of a handheld object is determined based on... on applying translation factors, scaling factors, and offsets to a location of a reference element of the handheld object detected by a camera on a two-dimensional plane. The translation factors are determined based on a difference between acalibration location on the plane and an initial location of the reference element corresponding to the calibration location, and serve to shift the location of the reference element to generate the projected target location. The scaling factors are determined based on an estimated length of a user's arm holding the handheld object, and serve to scale the location of the reference element to generate the projected target location. The offsets are determined based on polynomial equations, and serve to extend the distance between the projected target location and the calibration location.

Claims

1. A method comprising: Receive the initial position of the reference element of the handheld object projected onto the two-dimensional plane in three-dimensional space; A first offset is determined along a first axis of the two-dimensional plane, the first offset compensating for distortion along the first axis associated with the arcuate movement of the handheld object in the three-dimensional space; A second offset is determined along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arc; The projection target position of the handheld object is determined based on the initial position of the reference element, the first offset, and the second offset; as well as In response to determining that the projected target position corresponds to the target position, the user interaction experience is output.

2. The method according to claim 1, wherein, Determining the projection target position includes shifting the initial position of the reference element along the first axis by the first offset.

3. The method according to claim 1, wherein, The first offset along the first axis includes a horizontal offset on the two-dimensional plane.

4. The method according to claim 1, wherein, Determining the projection target position includes shifting the initial position of the reference element along the second axis by the second offset.

5. The method according to claim 1, wherein, The second offset along the second axis includes a vertical offset in the two-dimensional plane.

6. The method according to claim 1, wherein, When the handheld object is moved across the three-dimensional space, the arc is at least partially defined by the length of the user's arm.

7. The method according to claim 1, wherein, Determining the first offset includes performing regression analysis to fit the test data to a first polynomial equation, and wherein determining the second offset includes performing regression analysis to fit the test data to a second polynomial equation.

8. The method according to claim 7, wherein, The first polynomial equation and the second polynomial equation are both of the third order.

9. A theme park attraction system, comprising: An output device configured to output the user's interactive experience; A controller having one or more processors and a memory storing machine-readable instructions configured to cause the one or more processors to: Determine a first offset along a first axis of a two-dimensional plane, the first offset compensating for distortion along the first axis associated with the arc of movement of the handheld object in three-dimensional space; A second offset is determined along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arc of the movement of the handheld object in the three-dimensional space; The projection target position of the handheld object is determined based on the initial position of the reference element of the handheld object on the two-dimensional plane, the first offset, and the second offset; as well as In response to determining that the projected target position corresponds to the target position, the output device outputs the user interaction experience.

10. The theme park attraction system according to claim 9, wherein, The first offset is determined based on the distance along the first axis from the initial position of the reference element of the projected target position of the handheld object.

11. The theme park attraction system according to claim 9, wherein, The second offset is determined based on the distance along the second axis from the initial position of the reference element of the projected target position of the handheld object.

12. The theme park attraction system of claim 9, comprising a camera configured to capture an image of the reference element of the handheld object on the two-dimensional plane.

13. One or more tangible, non-transitory computer-readable media, comprising instructions that, when executed by at least one processor, cause the at least one processor to: Receive the initial position of the reference element of the handheld object projected onto the two-dimensional plane; A first offset is determined along a first axis of the two-dimensional plane, the first offset compensating for distortion along the first axis associated with the arcuate movement of the handheld object in three-dimensional space; A second offset is determined along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arc of the movement of the handheld object in the three-dimensional space; The projection target position of the handheld object is determined based on the initial position of the reference element, the first offset, and the second offset; as well as In response to determining that the projected target position corresponds to the target position, the user interaction experience is output.

14. One or more tangible non-transitory computer-readable media according to claim 13, wherein, The instruction causes the at least one processor to divide the arc into a plurality of reference element regions, wherein each of the plurality of reference element regions is associated with a corresponding target region among a plurality of target regions on the two-dimensional plane.

15. One or more tangible non-transitory computer-readable media according to claim 14, wherein, Each of the plurality of reference element regions is associated with the following: A first equation that compensates for the distortion along the first axis associated with the corresponding target area and the arc in the two-dimensional plane; and The second equation compensates for the distortion along the second axis associated with the corresponding target area and the arc on the two-dimensional plane.

16. One or more tangible non-transitory computer-readable media according to claim 15, wherein, The first equation and the second equation are the third-order polynomial equations.

17. One or more tangible non-transitory computer-readable media according to claim 14, wherein, Each of the plurality of reference element regions is of the same size, and each of the plurality of target regions is of a different size.

18. One or more tangible non-transitory computer-readable media according to claim 14, wherein, The first target region of the plurality of target regions is farther away from the reference element than the second target region of the plurality of target regions, and wherein the first target region is larger in size than the second target region.

19. One or more tangible non-transitory computer-readable media according to claim 14, wherein, Each of the plurality of reference element regions is of a different size, and each of the plurality of target regions is of the same size.

20. One or more tangible non-transitory computer-readable media according to claim 14, wherein, The first reference element region of the plurality of reference element regions is farther away from the two-dimensional plane than the second reference element region of the plurality of reference element regions, and wherein the first reference element region is smaller in size than the second reference element region.

21. A method comprising: A first offset is determined along a first axis of a two-dimensional plane, the first offset compensating for distortion along the first axis associated with the arc by dividing the arc of movement of a reference element of a handheld object in three-dimensional space into a plurality of reference element regions, and wherein each of the plurality of reference element regions is associated with a corresponding target region among a plurality of target regions on the two-dimensional plane; The projection target position of the handheld object is determined based on the initial position of the reference element of the handheld object on the two-dimensional plane and the first offset; and In response to determining that the projected target position corresponds to the target position, the user interaction experience is output.

22. The method of claim 21, further comprising receiving the initial position of the reference element of the handheld object projected onto the two-dimensional plane.

23. The method according to claim 22, characterized in that, include: A second offset is determined along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arc by dividing the arc in the three-dimensional space into a plurality of additional reference element regions, wherein each of the plurality of additional reference element regions is associated with an additional corresponding target region among a plurality of additional target regions on the two-dimensional plane; as well as The projection target position of the handheld object is determined based on the initial position of the reference element of the handheld object, the first offset, and the second offset.

24. The method according to claim 23, wherein, Each of the plurality of additional reference element regions is of the same size, and each of the plurality of additional target regions is of a different size.

25. The method according to claim 23, wherein, The first additional target region of the plurality of additional target regions is farther away from the reference element than the second additional target region of the plurality of additional target regions, and wherein the first additional target region is larger in size than the second additional target region.

26. The method according to claim 23, wherein, Each of the plurality of additional reference element regions is of a different size, and each of the plurality of additional target regions is of the same size.

27. The method according to claim 23, wherein, The first additional reference element region of the plurality of additional reference element regions is farther away from the two-dimensional plane than the second additional reference element region of the plurality of additional reference element regions, and wherein the size of the first additional reference element region is smaller than that of the second additional reference element region.

28. The method according to claim 23, wherein, Determining the projection target position includes shifting the initial position of the reference element along the first axis by the first offset, shifting the initial position of the reference element along the second axis by the second offset, or both.

29. The method according to claim 23, wherein, The first offset along the first axis includes a horizontal offset in the two-dimensional plane, and the second offset along the second axis includes a vertical offset in the two-dimensional plane.

30. The method according to claim 23, wherein, Determining the first offset includes performing regression analysis to fit the test data to a first equation, and wherein determining the second offset includes performing additional regression analysis to fit the test data to a second equation.

31. The method according to claim 30, wherein, The first equation and the second equation are each a third-order polynomial equation.

32. The method according to claim 31, wherein, The first polynomial equation compensates for the distortion along the first axis associated with the corresponding target area and the arc shape on the two-dimensional plane, and wherein the second polynomial equation compensates for the distortion along the second axis associated with the corresponding additional target area and the arc shape on the two-dimensional plane.

33. The method according to claim 21, wherein, Each of the plurality of reference element regions is of the same size, and each of the plurality of target regions is of a different size.

34. The method according to claim 21, wherein, The first target region of the plurality of target regions is farther away from the reference element than the second target region of the plurality of target regions, and wherein the first target region is larger in size than the second target region.

35. The method according to claim 21, wherein, Each of the plurality of reference element regions is of a different size, and each of the plurality of target regions is of the same size.

36. The method according to claim 21, wherein, The first reference element region of the plurality of reference element regions is farther away from the two-dimensional plane than the second reference element region of the plurality of reference element regions, and wherein the first reference element region is smaller in size than the second reference element region.

37. A system comprising: The output device is configured to output the user interaction experience. A controller having one or more processors and memory, the memory storing machine-readable instructions configured to cause the one or more processors to: A first offset is determined along a first axis of a two-dimensional plane, the first offset compensating for distortion along the first axis associated with the arc by dividing the arc of movement of a reference element of a handheld object in three-dimensional space into a plurality of reference element regions, and wherein each of the plurality of reference element regions is associated with a corresponding target region among a plurality of target regions on the two-dimensional plane; The projection target position of the handheld object is determined based on the initial position of the reference element of the handheld object on the two-dimensional plane and the first offset; and In response to determining that the projected target position corresponds to the target position, the output device outputs a user interactive experience.

38. The system according to claim 37, wherein, The machine-readable instructions are configured to cause the one or more processors to receive the initial position of the reference element of the handheld object projected onto the two-dimensional plane.

39. The system according to claim 38, wherein, The machine-readable instructions are configured to cause the one or more processors to: A second offset is determined along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arc by dividing the arc in the three-dimensional space into a plurality of additional reference element regions, wherein each of the plurality of additional reference element regions is associated with an additional corresponding target region among a plurality of additional target regions on the two-dimensional plane; and The projection target position of the handheld object is determined based on the initial position of the reference element of the handheld object on the two-dimensional plane, the first offset, and the second offset.

40. One or more tangible, non-transitory computer-readable media, comprising instructions that, when executed by at least one processor, cause the at least one processor to: Receive the initial position of the reference element of the handheld object projected onto the two-dimensional plane; A first offset is determined along a first axis of the two-dimensional plane, the first offset compensating for distortion along the first axis associated with the arc by dividing the arc of the movement of the reference element of the handheld object in three-dimensional space into a plurality of reference element regions; A second offset is determined along a second axis of the two-dimensional plane, the second offset compensating for distortion along the second axis associated with the arc by dividing the arc in the three-dimensional space into a plurality of additional reference element regions, wherein each of the plurality of additional reference element regions is associated with an additional corresponding target region among a plurality of additional target regions on the two-dimensional plane; The projection target position of the handheld object is determined based on the initial position of the reference element of the handheld object on the two-dimensional plane, the first offset, and the second offset; and In response to determining that the projected target position corresponds to the target position, the user interaction experience is output.

Images