A system for simulating physical systems of interconnected elements

Abstract

A simulation system comprising: a plurality of physical components, each physical component corresponding to one of a plurality of physical component types; an attachment panel comprising an arrangement of attachment locations such that one or more physical components can be attached to the attachment panel; a display system configured to provide visualization content on or proximate one or both of the attachment panel and the plurality of physical components; a capture device configured to capture image data of a current state of the attachment panel and the plurality of physical components; and a controller configured to display a display on at least one physical component. The present invention relates to a method for manufacturing a virtual component that is configured to: a) determine a configuration of one or more physical components coupled to the attachment panel; b) create a configuration of virtual components by assigning a virtual component to each physical component in the configuration of physical components; c) identify a user's physical interaction with the configuration of physical components and determine a corresponding change to a behavioral property of the configuration of virtual components; d) apply a selected modeling template to the current arrangement of virtual components to generate a visualization including indications of physical impacts associated with the changes in the behavioral properties; and e) cause the display system to display the visualization.

Description

Technical Field

[0001] The present invention generally relates to a system and method for simulating a physical system of interconnected elements or components. The present invention is applicable to simulating physical systems in which elements are mechanically coupled to each other, as well as other systems including electrical, mechanical, etc. Background Art

[0002] Many digital and physical aids exist for teaching scientific and engineering principles. Physical aids such as foam blocks or hanging chains can help students develop an intuition for structural behavior through tactile responses, while computer-based aids can effectively develop an intuition for internal force distributions and can support the teaching of theoretical principles. However, physical aids are limited to set demonstrator examples (i.e., limited or no customization capabilities), such as buckling columns, portal frames, or foam blocks, which means that they do not allow users to have an exploratory or design-oriented experience. Conversely, digital tools that use play-based games or software models to supplement theory are often overly playful or too complex, and invariably ignore tactile interactions that are known to be very valuable in teaching (especially for mechanical or structural systems whose overall properties are defined by deformation).

[0003] There is an urgent need to provide a digital or physical teaching aid to improve or overcome one or more shortcomings of known teaching aids, or at least provide an alternative to known teaching aids. Summary of the Invention

[0004] According to one aspect of the present invention, there is provided a simulation system comprising:

[0005] a plurality of physical components, each physical component corresponding to one of a plurality of physical component types,

[0006] an attachment panel comprising an arrangement of attachment locations such that one or more of the physical components can be attached to the attachment panel;

[0007] a display system configured to provide visual content on or proximate one or both of the attachment panel and the plurality of physical components;

[0008] a capture device configured to capture image data of a current state of the attachment panel and the plurality of physical components; and

[0009] a controller configured to, when at least one physical component is coupled to the attachment panel:

[0010] a) determining a configuration of one or more physical components coupled to the attachment panel;

[0011] b) creating a configuration of virtual components by assigning a virtual component to each physical component in the configuration of physical components;

[0012] c) identifying a user's physical interaction with the configuration of the physical component and determining corresponding changes to behavioral attributes of the configuration of the virtual component;

[0013] d) applying the selected modeling template to the current arrangement of virtual components to generate a visualization including an indication of physical effects associated with the change in the behavioral properties; and

[0014] e) enabling the display system to display the visual content.

[0015] In an embodiment, the at least one physical component corresponds to a structural member, and wherein the or each physical component is mechanically coupleable to at least one other physical component.

[0016] The at least one physical component may be a linear portion selected from the group consisting of one or more of: a beam, a strut, a cable, a rod, and a rope linear portion.

[0017] Alternatively, the at least one physical component may be an anchor portion selected from the group consisting of one or more of: a pin anchor, a rigid anchor, a pulley anchor, a tile anchor, and a turntable anchor.

[0018] Alternatively, the at least one physical component may be a joint portion selected from the group consisting of one or more of: a pin anchor, a rigid anchor, a pulley anchor, a tile anchor, and a turntable anchor.

[0019] Alternatively, the at least one physical component may be a docking portion selected from the group consisting of one or more of: a connection dock, a tiled dock, and a tracking beacon.

[0020] In some versions of the system,

[0021] Step c) comprises identifying a distortion applied to the configuration of the physical component and determining a corresponding distortion to be applied to the configuration of the virtual component, and

[0022] Step d) comprises applying the selected modeling template to the current arrangement of virtual components to generate a visualization including an indication of a physical effect associated with the distortion.

[0023] In some versions of the system,

[0024] Determining a distortion to the configuration of the virtual component is based at least in part on the controller performing the following operations:

[0025] receiving image data from a capture device;

[0026] identifying a distortion of the physical component based on analyzing the received image data; and

[0027] Each corresponding virtual component is assigned an equivalent distortion.

[0028] In some versions of the system,

[0029] Determining a distortion to the configuration of the virtual component is based at least in part on the controller performing the following operations:

[0030] receiving image data from a capture device;

[0031] determining a current position of an interaction device, wherein the interaction device is pressed against one or more physical components to cause a distortion in the configuration of the physical components; and

[0032] Based on the current position of the interaction device, a distortion to the configuration of the virtual component is determined.

[0033] In some versions of the system, the interactive device is a handheld device.

[0034] In other versions of the system, the interaction device is coupled to one or more interconnected support members, one support member being coupled to the attachment panel.

[0035] In other embodiments, the at least one physical component corresponds to a circuit element.

[0036] In this case, step c) may comprise identifying a change in the user selection to at least one circuit element and determining a corresponding change to the electrical properties of the virtual component configuration, and

[0037] Step d) may comprise applying the selected modeling template to the current arrangement of virtual components to generate a visualization including an indication of an electrical effect associated with the user-selected change.

[0038] Determining a change to one or more electrical properties of a configuration of a virtual component is based at least in part on the controller performing the following operations:

[0039] receiving image data from a capture device;

[0040] determining a current position of an interactive device coupled to the attachment panel; and

[0041] Based on the current position of the interaction device, a corresponding change to an electrical property of the configuration of the virtual component is determined.

[0042] The capture device may include an infrared sensor and / or an RGB sensor.

[0043] The controller may be configured to identify individual physical components coupled to the attachment panel and record the determined type for each physical component.

[0044] Visualizations may be generated based on component type, shape, and position of the current arrangement of physical components.

[0045] These attachment locations may include recesses that are configured to prevent rotation of the physical component relative to the plane of the attachment surface when the physical component is attached to the attachment location.

[0046] These attachment locations may define a periodic rectangular array.

[0047] Another aspect of the present invention provides a method implemented by a controller for generating visualization content in one or both of an attached panel and a plurality of physical components for simulation, the method comprising the following steps:

[0048] a) identifying a configuration of virtual components based on the received image data, wherein the configuration of the virtual components corresponds to a configuration of physical components, wherein each physical component is associated with a physical component type, wherein the physical components are arranged to be coupled to the attachment panel, and wherein the physical components include at least one physical component attachable to an attachment location of the attachment panel;

[0049] b) identifying a user's physical interaction with the physical configuration of the component and determining corresponding changes to behavioral attributes of the configuration of the virtual component;

[0050] c) applying the selected modeling template to the current arrangement of virtual components;

[0051] d) generating a visualization including an indication of a physical effect associated with the change made to the behavioral attribute; and

[0052] e) enabling the display system to display the visual content.

[0053] Yet another aspect of the present invention provides a computer program, which is configured to cause a processor to implement the above method when the program is executed by the processor.

[0054] As used herein, the word "include" or its variants "comprises" or "contains" are used in an inclusive sense, that is, the above words indicate the presence of the stated features in various embodiments of the invention, but do not exclude the presence or addition of further features in various embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order that the present invention may be more clearly understood, preferred embodiments thereof will now be described in detail by way of example with reference to the accompanying drawings, in which:

[0056] Figure 1 is a block diagram of a simulation system according to one embodiment of the present invention;

[0057] Figure 2 is formed Figure 1 a front view of an attachment panel as part of a simulation system;

[0058] Figure 3 is formed Figure 1 Isometric view of a typical physical component of a part of the simulated system, in this case a linear section;

[0059] Figure 4 is formed Figure 1 Isometric view of a typical physical component of a part of the simulated system, in this case the anchoring part;

[0060] Figure 5 is formed Figure 1 Isometric view of a typical physical component of a part of the simulated system, in this case a joint;

[0061] Figure 6 yes Figure 3 An enlarged isometric view of the linear portion of the beam shown;

[0062] Figure 7 yes Figure 3 an isometric enlarged view of the linear portion of the strut shown;

[0063] Figure 8 is an isometric view of yet another physical component, in this case a docking section, showing docking to beams, rods, and cable linear sections;

[0064] Figure 9 is formed Figure 1 an isometric view of the configuration of the physical components of a portion of the simulated system shown and illustrating their attachment to the attachment panel;

[0065] Figure 10A and Figure 10B Shown to form Figure 1 Two isometric views of a user interaction device ("pointer stick") that is part of the simulation system shown;

[0066] Figures 11 to 13 Indicates Figure 10A 、 Figure 10B The user interaction device is shown as having a configuration of physical components and attached to an attachment panel, the user interaction device forming Figure 1 A portion of the simulation system is shown;

[0067] Figure 14 and 15 is formed Figure 1 A flow chart of steps executed by a controller of a portion of a simulation system as shown in ;

[0068] Figure 16 Shows how to make Figure 1 Physical components that are part of a simulated system are shown with general symbols that approximate corresponding features;

[0069] Figure 17 and 18 Indicates that the display system is Figure 2 The template of the attached panels shown in the figure shows that the display system forms Figure 1 A portion of the simulation system is shown;

[0070] Figure 19 and 20 isometric views of embodiments of simulation systems using 2-dimensional and 3-dimensional display systems, respectively;

[0071] Figure 21 A front view of another embodiment of a physical component attached to an attachment panel is shown, the physical component forming a Figure 1 Part of the simulated system shown, in this case the tile anchor;

[0072] Figure 22 and 23 Two further embodiments are shown with front views of physical components attached to an attachment panel, the physical components forming Figure 1 As part of the simulated system shown, these physical components include a turntable anchor; and

[0073] Figure 24 and Figure 25 A front view of one or more embodiments of a tracking beacon and an attachment member are shown, each docked into an opening in an attachment panel for use with the simulation system described herein. DETAILED DESCRIPTION

[0074] Figure 1 A simulation system 10 is shown in one embodiment for visualizing dynamic graphical information onto a structural simulation. The system 10 includes an attachment panel 11, a display system 12, a capture device 13, and a controller 14. The display system 12 and the capture device 13 are connected to the controller 14.

[0075] The controller 14 may generally correspond to a programmable computing device. The functionality of the controller 14 is generally defined in software that, when executed by the controller 14, implements the methods described herein.

[0076] Reference Figure 2According to one embodiment, the attachment panel 11 is a planar surface, for example having a rectangular profile as shown. The attachment panel 11 further includes an arrangement of attachment locations 20 (one attachment location 20 is labeled in the figure) on a front surface 21 of the attachment panel 11. In the embodiment shown, the attachment locations 20 constitute recessed portions of the attachment panel 11. Of course, it is contemplated that the attachment locations 20 may also constitute raised portions (at least partially constituted as raised portions).

[0077] In one embodiment, as shown, the arrangement of the attachment locations 20 is a regular rectangular array - that is, adjacent attachment locations 20 are equally spaced horizontally and vertically. Of course, other arrangements are also contemplated, including irregular and / or non-rectangular arrangements.

[0078] The attachment panel 11 can have a size suitable for a particular application. In some cases, each side of the attachment panel 11 can be less than one meter long—for example, when used as an auxiliary tool for a desk or tabletop. In other cases, a side of the attachment panel 11 can be longer than one meter—for example, when used as a teaching aid for a seminar.

[0079] Figures 3 to 5 Examples of physical components for attaching to an attachment panel and / or coupling to each other are each shown according to various embodiments.

[0080] In this embodiment, the physical components are categorized into linear parts (beams, rods, struts, cables, and ropes) 120 to 128, anchoring parts (rigid anchors, pin anchors, roller anchors, pulley anchors, turntable anchors, and tile anchors) 130 to 142, and joint parts (rigid joints, pin joints, pulley joints, counterweight joints, and expansion joints) 144 to 164.

[0081] Each physical component is configured to simulate a structural member—a beam, strut, or cable. These physical components are designed to withstand manipulation consistent with the structural component being simulated—for example, bending and axial motion. Certain types of physical components are also designed to be removably attached to other types of physical components. For example, anchors and connectors can be configured to simulate connections between linear sections.

[0082] Generally, since the purpose of the physical components in this embodiment is to simulate structural members, a variety of different physical components (or "toolkits") can be provided. The embodiments can be considered to include one or more specific examples described, but should not be considered limited to these examples.

[0083] At least a portion of the physical component may be formed using known techniques - for example, injection molding. It is also contemplated that a 3D printing data file may be provided, comprising instructions for causing a 3D printer to manufacture the various physical components described herein.

[0084] Combine Figure 4 The term "anchor" means that the anchor portions 130 to 142 are respectively provided with anchor members 130a to 142a so that the anchor portions can be attached to the attachment panel 11. The anchor portions 130 to 142 are configured to have a complementary profile to the attachment location 20 (e.g. Figure 2 142a) such that the anchoring members 130a to 142a can be secured in place by placing (e.g., pushing) the anchoring portions 130a to 142a into the complementary attachment locations 20. In the example shown, the anchoring members 130a to 142a include one or more protrusions (e.g., the protrusions labeled 136b) that are configured to securely fit within the recesses defined by the attachment locations 20. Relatedly, the recesses and the anchoring members are shaped to limit (preferably prevent) rotation of the anchoring portions 130 to 142. In the example shown, this is achieved by a cross-shaped cross-section.

[0085] The attachment locations 20 can each include a cross-shaped recess - each attachment location 20 is symmetrical with respect to a 90-degree rotation. These protrusions are also configured to have a cross-shaped recessed section that is complementary to the cross-shaped recessed section of the attachment location 20. Therefore, each anchoring portion 130 to 142 can be attached to the attachment panel 11 in any of four orientations. In other embodiments, different shapes can be used. For example, a hexagonal or 6-pointed star-shaped recessed and protruding cross-section arrangement can achieve attachment in six orientations. A hexagonal or 8-pointed star will enable eight orientations. Generally, the shape of the recessed and protruding sections can be selected as needed. It is conceivable that the attachment panel 11 can include two or more differently shaped attachment locations 20 with different rotation options.

[0086] Anchor portions 130 to 142 may be considered to represent locations where the structural component being modeled is fixed to resist some movement—for example, representing a beam fixed to a ground support.

[0087] For example, the rigid anchor portion 130 includes one or more fixed orientation couplers 42a (one shown) configured to receive the connection portions at the ends of the linear portions 120 to 128. The fixed orientation couplers 42a are fixed relative to the body of the rigid anchor portion 130—thus, the fixed orientation couplers 42a do not change orientation when attached to the attachment panel 11.

[0088] As another example, the pin anchoring portion 132 includes one or more variable-direction couplers 42b configured to accommodate the connection portions at the ends of the linear portions 120 to 128. In this figure, one variable-direction coupler 42b is shown. The one or more variable-direction couplers 42b are rotatably mounted to the body of the pin anchoring portion 132, for example, via an axis 402.

[0089] The pin anchoring portion 132 can include a block 403 (as shown) to limit the available rotation angle of one or more variable direction couplers 42b. In the figure, the block 403 is formed as a body part of the pin anchoring portion 132 and effectively blocks approximately 120 degrees of rotation. Different sizes of blocks 403 can be used as needed - for example, to block rotations greater than 120 degrees or less than 120 degrees. Advantageously, the block 403 can be configured to provide a physical representation of the graphic symbols used in the field (e.g., building science / building design).

[0090] Roll anchor 136 is a modification of pin anchor 132, and thus includes one or more variable-direction couplers (two in the example). Roll anchor 136 is configured for limited movement relative to attachment panel 11—in the illustrated example, limited movement constitutes a degree of freedom of short-range movement in a single dimension. To enable movement, roll anchor 136 includes a mounting portion 404 to which is attached a variable-direction coupler 42b, which is slidably connected to a body 400c, into which the body is secured when inserted into attachment panel 11.

[0091] In contrast to anchor portions 130-142, joint portions 144-164 do not include anchoring members and are therefore able to move relative to the plane of attachment panel 11. Joint portions 144-164 are "floating" and support a connection between two or more physical components without securing the connection to attachment panel 11.

[0092] For example, the joint portion 152 includes three variable-direction couplers 42b, which are rotatably coupled to each other around an axis 402. Optionally, the joint portion 152 may include a blocking portion (not shown) to limit relative rotation of the variable-direction couplers 42b. Thus, the physical components coupled to the joint portion 152 can be effectively rotationally coupled to each other at the joint portion 152.

[0093] Another example is a connector portion 144 that is configured to couple to two or more physical components. In this case, the connector portion 144 includes two fixed-direction couplers 42a. The fixed-direction couplers 42a are fixedly coupled to each other (and typically will be integrally formed). Thus, the physical components coupled to the attachment module 31e are effectively fixedly coupled to each other in a linear or docking portion.

[0094] Figure 6 An example of a beam linear portion 120 is shown in greater detail. The beam linear portion 120 is configured to simulate a bendable structural component. The term "bendable" does not necessarily refer to a structural component that is designed to bend significantly—rather, bendable refers to a structural component that may be subjected to bending forces. An example of such a structural component is a beam.

[0095] Beam linear section 120 includes a first end member 300 and a second end member 301. Integrally formed with end members 300, 301 are joints 303a-303d and one or more (in this example, three) structural modules 302a-302c. Joint 303 is configured to enable in-plane bending of beam linear section 120 while providing high rigidity against out-of-plane bending. In this embodiment, joint 303 has a flat cross-section, thereby allowing in-plane bending about axis 304. Structural modules 302 are configured to provide rigidity against bending in all directions.

[0096] The joint 303 can be designed based on the desired bending properties of the beam linear portion 120. That is, the thickness and / or length of the joint 303, for example, can be configured based on the desired bending properties. For example, increasing the thickness of the joint 303 can reduce the ease of bending compared to reducing the thickness. Similarly, a shorter length generally results in reduced bendability, while a longer length increases bendability. In one embodiment, the physical element 30a is made of a material selected to provide the desired bendability.

[0097] The configuration of the structural modules 302 can also be selected based on the desired bending properties. For example, considering different bendable physical elements 30a of the same length, the number of structural modules 302 can have an effect on the bending properties - it should be noted that, in general, for the same physical element length, increasing the number of structural modules 302 reduces the length of the structural modules 302.

[0098] The beam linear portion 120 is configured to have an appearance designed to look like a typical structural I-beam - this appearance can help enhance the visual appeal of the model and the educational value of the element 30a.

[0099] Figure 7 An example of a strut linear portion 124 is shown in greater detail. The strut linear portion 124 is configured to simulate the effects of an axial force that acts to compress (and expand) the strut linear portion 124 along an axis. For example, the strut linear portion 124 can simulate a structural component, such as a strut, that is subjected to compression or tension along an axis.

[0100] The support linear portion 124 includes a first end piece 310 and a second end piece 311. The physical element 30b also includes a first structural module 312a coupled to (optionally integrally formed with) the first end piece 310, and a second structural module 312b coupled to (optionally integrally formed with) the second end piece 311. The first structural module 312a is slidably coupled to the second structural module 312b. As shown, the first structural module 312a forms a piston that enters a cylindrical portion within the second structural module 312b. The first structural module 312a includes a pin 315 extending from its piston portion, which is configured to move within an elongated aperture 316 of the cylindrical portion of the second structural module 312b so as to, for example, limit relative sliding of the first structural module 312a and the second structural module 312b. Figure 7 Also shown in FIG, there is a spring 317 coupled to the first end piece 310 and the second end piece 311 and configured to provide tension and compression between the end pieces 310 and 311 .

[0101] The support linear portion 124 can be designed to simulate different structural components by the change of configuration and design. For example, different responses (for example, different stiffnesses) to axial force can be simulated by springs 316 of different strengths. In addition, the length of the aperture 315 can be selected to set the maximum range of relative motion between the first structural module 312a and the second structural module 312b. In one embodiment, the spring 316 is interchangeable with springs 316 having different spring constants.

[0102] Figure 8 An example of a cable linear portion 31f is shown in greater detail. The cable linear portion 31f includes one or more cable couplers 42c (one shown). Each cable coupler 42c can be coupled to a cable physical element 30c. The cable linear portion 31f can be floating (as shown) or anchored (not shown). The coupling function can be flexible, non-flexible, etc.

[0103] The cable linear portion is configured to simulate a structural member that transmits force by tension. Typically, force is transmitted only when the cable linear portion 30c is taut. The cable linear portion 30c includes a first end 320 and a second end (not shown) and defines a string, rope, or other cable-like object 322. A semi-inflexible cable linear portion 30c can be used that is configured to allow force to be transmitted in a direction different from the tensile force. In the embodiment shown, the first end 320 is attached to the cable coupler 42c of the cable linear portion 31f.

[0104] Figure 8 The docking portion 31f is shown, which is used Figure 3Standard length "wire sections" of the type designated 120 to 128 are provided to facilitate diagonal connections. In this exemplary scenario, the beam wire section 120 is connected to the rod wire section 122 by a pin joint 150. The pin joint 150 includes an aperture which, when in use, extends perpendicular to the plane of the assembled panel 1.

[0105] The docking component 31f includes a central elongated body 450, a docking head 452, and a coupling tail 454. The docking head 452 includes a male docking member 456 adapted to be inserted into an aperture extending through the pin connector 150. The coupling tail 454 includes a direct coupler 42b for engaging another linear portion, in this case, a cable linear portion 126, which extends diagonally across the configuration of the physical component mounted to the attachment panel. The male docking member is preferably rotatable within the aperture so that the angle at which the cable linear portion 126, or a similar linear portion, extends across the configuration of the physical component mounted to the attachment panel can be varied as desired.

[0106] Figure 9 A collection of physical components and an attachment panel 11 with attachment locations 20 is shown, collectively referred to as a "configuration." Specifically, the figure illustrates how the physical components are attached to each other, and more notably, how the anchor portion is attached to the attachment panel 11. As shown, there are three flexible physical components 30a-1, 30a-2, and 30a-3. There are also anchor portions 31a and 31b. There is also a floating joint portion 31d (which allows rotational movement) and a floating joint portion 31e (which does not allow rotational movement).

[0107] The anchor portion 31a is positioned upward above the fixed direction coupler 42a-1. The fixed direction coupler 42a-1 receives the first end piece 300 of the physical component 30a-1. The anchor 31a is received by the attachment panel 11 at the attachment location 20a. The second end piece 301 of the physical component 30a-1 is configured to be received by the variable direction coupler 42b-1 of the floating joint portion 31d. Note that the floating joint portion 31d is not attached to the attachment panel 11. The variable direction coupler 42b-2 of the attachment module 31d receives the first end piece 300 of the physical component 30a-2. The second end piece 301 of the physical component 30a-2 is received by the fixed direction coupler 42a-2 of the floating joint portion 31e. The fixed direction coupler 42a-3 of the floating joint portion 31e receives the first end piece 300 of the physical element 30a-3. The second end piece 301 of the physical element 30a-3 is received by the variable direction coupler 42b-3 of the anchor portion 31b. Note that the anchor portion 31b allows it to move the variable direction coupler 42b-3 about the axis 402. The anchor portion 31a is received by the attachment panel 11 at the attachment location 20b. The structure can be deformed by applying a force at one or more points on the components 30, 31. The deformation is constrained by a variety of factors, including, in this example, the location of the attachment locations 20a, 20b, the rotatability (or non-rotatability) of the component 31, and the bending properties of the physical component 30.

[0108] Reference again Figure 1 According to one embodiment, the display system 12 includes a projector 50. The projector 50 is configured to project an image onto the attachment panel 11—in the example shown, the projector 50 projects onto the front surface 21 of the attachment panel 11. However, it is contemplated that in another embodiment, the projector 50 projects onto the rear surface of the attachment panel 11—in this case, the attachment panel 11 should be sufficiently translucent to enable a user to view the projected image on the front surface 21. Depending on the specific embodiment, the display system 12 may take other forms. For example, an LCD or OLED screen may be formed as a feature of the attachment panel 11.

[0109] According to an embodiment, the display system 12 is controlled by the controller 14. Thus, the display system 12 is in data communication with the controller 14, thereby enabling the controller 14 to transmit display data to the display system 12 to provide an image for display. This data communication can be of a known type, for example, using a digital standard such as HDMI, DVI, DisplayPort, or USB, or an analog standard.

[0110] The display system 12 is configured to present image visualization content (it should be understood that "visualization content" includes rear visualization content) to the attachment panel 11. The visualization content will be currently coupled (e.g., attached) to any physical components 30, 31 of the attachment panel 11, thereby providing an integrated experience with the visualization of the components 30, 31.

[0111] Still refer to Figure 1 , the capture device 13 interfaces with the controller 14 - in the example shown, the capture device 13 includes an infrared camera, an RGB camera, a monochrome camera, or a depth-of-field camera. The capture device 13 is configured to capture image data representing the current state of the attachment panel 11 and depth. The capture device 13 communicates the image data to the controller 14. The controller 14 is typically configured to receive user input and present display content to the user (e.g., via a human interface device (HID)).

[0112] refer to Figure 10A 、 Figure 10B 、 Figure 11 、 Figure 12 and Figure 13 Optionally, an interaction device 51 (also referred to herein as a "pointer stick 51") is provided. The pointer stick 51 has one (or more) defined visual attributes known to the controller 14 - typically, information enabling the controller to visually identify the position and typical orientation of the pointer stick 51 is stored in a memory of the controller 14.

[0113] It will be appreciated that in other embodiments, the user may interact with the physical component manually, such as using their hands.

[0114] Specifically, refer to Figure 10A and Figure 10B In this embodiment, the indicator wand 51 includes one or more electromagnetic sources 56 - in the example shown, the indicator wand 51 includes an infrared LED. The electromagnetic source 56 should generally be visible to the capture device 13 - in this embodiment, the capture device 13 is an infrared camera. Figure 15Two modes are shown - a "beam push" mode, in which the indicator rod 51a is configured to push into the component 30, 31 (particularly, a bendable physical component). In this mode, the indicator rod 51a includes a roller 59 that is configured to roll and thereby minimize friction between the indicator rod 51 and the physical component. Another mode is a "node pull" mode, in which the indicator rod 51b is configured to pull the physical component (again, such as a bendable physical component) via a hook member 60. One (or more) electromagnetic sources 56 can further be configured to transmit information to the controller 14, such as by pulse operation. This information can be responsive to user input, such as via a trigger 61. For example, a pulse can indicate that a user has "selected" a displayed option, whereby the indicator rod is placed in a position corresponding to the selection and the user activates the trigger, thereby selecting a particular displayed option.

[0115] Alternatively (or in addition), the indicator rod 51 may include features that are easily visible under infrared light, for example, when the capture device 13 includes an infrared camera. In one example, the indicator rod 51 has a defined shape in two or three dimensions, and the controller 14 is configured to recognize the shape from the image data. The defined shape enables, for example, the controller 14 to determine the orientation of the indicator rod 51.

[0116] According to one embodiment, Figure 14 As shown, the controller 14 is configured to receive image data from a capture device at step 100 in response to determining that there is an initial configuration of components 30, 31 attached to the attachment panel 11. For example, this may be determined in response to user input indicating a current initial configuration.

[0117] The controller 14 is also configured to determine a virtual representation of the initial configuration at step 101. The virtual representation assigns a virtual component to each physical component 30, 31 - so there may be, for example, a virtual beam and a virtual strut. Typically, the controller 14 is configured to simulate applying distortion to the virtual components in a manner consistent with the distortion applied to the components 30, 31 (explained below).

[0118] In one embodiment, reference Figure 17 and Figure 18, the controller 14 is configured to cause the display system 12 to display a template 57 of a specific configuration of the physical components 30, 31 on the attachment panel 11. The display template 57 provides an indication of which physical components 30, 31 should be attached to the attachment panel 11, an indication of where the physical components 30, 31 should be attached, and an indication of how the various physical components 30, 31 should be attached to each other. In the illustrated embodiment, the template 57 includes a component symbol 58 for each physical component 30, 31, wherein each component symbol 58 is selected to have an appearance similar to that of the physical component 30, 31 (e.g., a similar outline or 2-dimensional representation). The user can then attach the corresponding physical component 30, 31 according to the template 57. Because the template 57 is displayed on the attachment panel 11, it is relatively easy for the user to set the correct configuration of the physical components 30, 31-the user only needs to attach the relevant physical components 30, 31 in the correct position.

[0119] In one embodiment, the user is enabled to provide input to the controller 14 that specifies which virtual components to use, and the relationship between the virtual components - thereby, the user is responsible for configuring the arrangement of the virtual components so that the arrangement is consistent with the arrangement of the components 30, 31 on the attachment panel 11.

[0120] In another embodiment, the image data is processed by the controller 14 to identify one or more virtual components corresponding to the components 30, 31 currently coupled to the attachment surface 11. Generally, this processing can employ known imaging processing algorithms and techniques. The process of identifying one or more individual physical components 30, 31 also includes determining the current shape and position of the physical components 30, 31, which are stored in data variables. Generally, the current shape and position can be represented using different methods as desired, such as using a bitmap or vector method.

[0121] According to one embodiment, the configuration of the virtual component is recorded with reference to the attachment surface 11 - ie the attachment surface 11 is considered to define a reference plane and the current virtual configuration is recorded relative to its position on the attachment surface 11 .

[0122] For each identified physical component 30, 31, at step 102, the type of component 30, 31 is determined. Identification (of each component) includes determining whether the component 30, 31 is a physical element 30 or an attachment module 31. The controller 14 also determines the type of physical element 30 or attachment module 31 (if applicable). This information is stored in the data store in association with the current shape and position. In a related embodiment, the identification and type of the component 30, 31 is provided by the selected template 57.

[0123] As a result of steps 101 and 102, the controller has information in memory indicating a current set of one or more virtual components corresponding to the physical components 30, 31 present on the attachment surface 11. These virtual components are associated with information indicating the type of each physical component 30, 31 and its original shape and position on the attachment panel 11. Controller 14 can be configured to identify component parts using imaging analysis techniques. Additionally or alternatively, controller 14 can be provided with depth information as part of the image data—for example, this information can be obtained using a time-of-flight sensor (e.g., an infrared time-of-flight sensor)—and based on this depth information, controller 14 can be configured to identify individual component parts. Such image and / or depth analysis can also be used to identify specific types of component parts, for example, based on templates pre-provided to controller 14 (e.g., stored in controller 14's memory). Other methods related to steps 101 and 102 can also be incorporated—for example, each type of component part can be uniquely color-coded, allowing the type of the component part to be determined based at least in part on its color. Image processing may be implemented using algorithms determined using machine learning, where the machine learning is taught to recognize the various physical components 30 , 31 .

[0124] The controller 14 then determines the selected modeling template at step 103. The current modeling template may be selected by a user from a set of one or more modeling templates. Each modeling template defines a physical model to be applied to the current arrangement of components 30, 31, for example, to determine display data to be transmitted to the display system 12. For example, the modeling template includes information indicating how the virtual component deforms in response to deformations applied to the corresponding configuration of the physical components 30, 31. The modeling template may also include information that can determine simulated physical properties associated with the physical components 30, 31—for example, information indicating the strains and / or forces simulated by the physical components 30, 31.

[0125] The controller 14 is then configured to determine a distortion of the virtual component at step 104 that corresponds to the distortion resulting from the configuration of the components 30, 31. The distortion is caused by one or more forces being applied to the physical components 30, 31, resulting in a change in the shape defined by the arrangement of the physical components 30, 31. The distortion is constrained by the actual physical components 30, 31 used and their positions—for example, the anchor attachment modules 31a, 31b, 31c cannot move relative to their positions on the attachment panel 11. The bendable physical component 30a can bend but not compress or expand, while the compressible physical component 30b cannot bend. The floating physical components 31d, 31e are able to move relative to the attachment panel 11, and therefore the ends 300, 301, 310, 311 of the physical component 30 coupled to the floating physical components 31d, 31e are able to move relative to the attachment panel 11. As a corollary, the ends 300,301,310,311 coupled to the anchoring physical components 31a,31b,31c cannot move relative to the attachment panel 11. However, the physical component 31 with the variable direction coupler 42b enables rotational movement of the coupled ends 300,301,310,311.

[0126] refer to Figure 15 In one embodiment, the controller 14 is configured to determine a distortion of the virtual component based on the current position, and in one embodiment, based on the orientation of the indicator wand 51. The controller 14 is configured to determine the current position (and optionally the orientation) of the indicator wand 51 at step 200. The controller 14 is then configured to determine a corresponding displacement of at least one virtual component at step 201 by assuming that the indicator wand 51 acquires its current position by contacting one or more physical components 30, 31 and forcing them to distort so that the indicator wand 51 can reach its current position. The controller 14 may employ a collision model to determine when the indicator wand 51 interacts with a component 30, 31 - the controller 14 does this by determining whether the indicator wand position has caused a collision with a virtual component.

[0127] The controller 14 is also configured to determine the distortion of each virtual component based on the properties of each virtual component (e.g., fixed in place, bendable, etc.) by applying the modeling template at step 202. Thus, the controller 14 is configured to calculate the current shape defined by the distorted virtual component. Steps 201 and 202 can be performed simultaneously because the distortion caused by the indicator stick 51 can affect the entire shape of the virtual component.

[0128] In another embodiment, the controller 14 is configured to continuously receive image data showing the current shape of the physical component 30, 31. Based on this data, the controller 14 is configured to determine the corresponding shape of the virtual component.

[0129] Reference again Figure 14 The controller 14 then applies the selected modeling template to the current arrangement of virtual components (corresponding to the current arrangement of physical components 30 , 31 ) at step 104 to generate display data that is sent to the display system 12 at step 105 .

[0130] In response to step 105 , the display system 12 displays the display data as image visualization content on the attachment panel 11 , as previously described.

[0131] For example, the modeling template interprets the current arrangement of physical components 30, 31 as a model of a physical system. For example, the arrangement of struts (i.e., physical elements 30) with a specific type of connection (i.e., attachment modules 31). The selected modeling template also defines the visual response to the arrangement of physical components 30, 31—for example, a visualization showing specific forces and / or strains depending on the physical system currently being modeled.

[0132] Figures 11 to 13 Several features are illustrated. For example, in each figure, a selection interface 59 is shown for selecting one or more modeling templates. Figure 11 In the , the modeling templates "Axial force" and "Reaction force" are selected. Figure 19 In the , the modeling template "Bending Moment" is selected. Figure 20 , the modeling templates "Shear Force" and "Reaction Force" are selected. These figures also show different visualizations of the modeled physical properties (e.g., axial force, reaction force, bending moment, shear force)—these physical properties are displayed in different ways. The visualizations are dynamic and change as the position of the indicator stick changes—and thus correlate with the distortion of the configuration of components 30, 31.

[0133] For reference, the inventors describe modeling of physical systems in the following publications:

[0134] Quinn, G: “Augmented and Virtual Reality Architectures”, Proceedings of the 2004 LASS Annual Symposium, September 25-28, 2017, Hamburg, Germany.

[0135] Quinn, G: “StructVR: Structures for Virtual Reality”, Proceedings of the 2018 LASS Workshop, July 16–20, 2018, MIT, Boston, USA.

[0136] The disclosure of the present application includes many features not defined in these publications—for example, the controller 14 can be configured to identify the types of physical elements 30 and attachment modules 31 present and include that information when applying the model. The techniques disclosed in the prior art require the controller 14 to know in advance which physical elements 30 and attachment modules 31 are present—for example, by predefining them in programming.

[0137] Thus, the embodiments described herein provide for greater flexibility in simulation and are easier to configure—in effect, enabling a user to configure a modeling template by coupling a variety of physical components 30 and 31 to the attachment panel 11. For example, the physical component 30a is discretized by the structure module 302 to advantageously represent the discretization of the finite elements of the simulation defined by the modeling template—this advantageously allows for a representation of forces that matches the actual distortion of the physical component 30a.

[0138] In an advantageous embodiment, reference Figure 9 , the physical components 30, 31 are configured, for example, to resemble common symbols of corresponding features. For example, Figure 16 Schematic symbol 800 in includes visually identifiable features for a component of configuration 900, and schematic symbol 801 includes visually identifiable features for a component of configuration 901. In one embodiment, these standardized symbols are projected by display system 12 as part of the visualization.

[0139] Further modifications may be implemented without departing from the spirit and scope of this specification. Figure 19 As shown, the embodiment described above includes an assembly of an attachment panel 300 and physical components 302, one or more of which are attached to the attachment panel 300. In the example described above, the visualization 304 includes an indication of a physical effect provided by the projector 50 on the attachment panel 304, which is associated with a change in a behavioral attribute of a configuration of a virtual component created by a controller forming part of the system.

[0140] However, if Figure 20 As shown, in other embodiments of the present invention, the display system may include an augmented reality helmet 400 that is operably connected to the controller 14 in the same manner as the projector 50. This arrangement is particularly suitable for embodiments of the present invention in which the assembly 400 of physical components attached to the attachment panel 402 is a 3-dimensional structure (relative to the 3-dimensional structure). Figure 19 ). In this case, the augmented reality headset 400 (displaced) provides visualizations 406 and 408 on or proximate to one or both of the attachment panel 404 and the assembly 402 of the physical component.

[0141] It should be understood that the foregoing embodiments are examples of simulation systems, which can be more broadly characterized as including

[0142] a plurality of physical components, each physical component corresponding to one of a plurality of component types,

[0143] an attachment panel comprising an arrangement of attachment locations such that one or more physical components can be attached to the attachment panel;

[0144] a display system configured to provide visual content on or proximate one or both of the attachment panel and the plurality of physical components;

[0145] a capture device configured to capture image data of a current state of the attachment panel and the plurality of physical components; and

[0146] a controller configured to, when at least one component is coupled to the attachment panel:

[0147] a) determining a configuration of one or more physical components coupled to the attachment panel;

[0148] b) creating a configuration of virtual components by assigning a virtual component to each physical component in the configuration of multiple components;

[0149] c) identifying a user's physical interaction with the configuration of the physical component and determining a corresponding change to a behavioral attribute of the configuration of the virtual component;

[0150] d) applying the selected modeling template to the current arrangement of virtual components to generate a visualization including an indication of physical effects associated with the change in the behavioral properties; and

[0151] e) enabling the display system to display the visual content.

[0152] Although described with respect to components 30 and 31 corresponding to physical structures, the simulation system can be adapted to represent other physical systems, such as chemical, electrical, and quantum systems. However, these representations can include additional layers of abstraction, relying on the specific physical elements 30 and attached modules 31 that represent specific aspects of the system. Furthermore, in these adaptations, the controller 14 is configured to identify the characteristics of the system being represented by identifying the specific type of physical element 30 or attached module 31.

[0153] Figure 21An alternative embodiment is described. In this case, one or more tile anchors 500 to 508 are configured to simulate circuit elements. In this case, a controller recognizes that a user has made a selectable change to one or more electronic components. The controller determines a corresponding change to the electrical properties of the configuration of the virtual components and then applies the selected modeling template to the current arrangement of the virtual components to generate a visualization that includes an indication of the electrical effects associated with the user-selected change.

[0154] For example, the user selected changes may include replacing elements, where each element represents a different resistance, capacitance, inductance, etc. Optionally, the capture device 13 may be configured to detect the rotational position of the turntable anchor, e.g. Figure 4 Turntable anchor 140 is shown.

[0155] The user-selected changes may include rotating the dial anchor 140 to increase or decrease the desired electrical properties of the circuit components corresponding to the tile anchors 500 through 508. Accordingly, the controller determines changes to one or more electrical properties of the configuration of the virtual components based at least in part on receiving image data from the capture device, determining a current position of an interaction device (e.g., the dial anchor 140) coupled to the attachment panel 510, and then determining corresponding changes to the electrical properties of the configuration of the virtual components based on the current position of the interaction device.

[0156] The turntable anchor 140 can be used in a variety of scenarios. The turntable anchor 140 is a device that is pre-set to dock anywhere on the assembly panel, and its function can be assigned and reassigned according to the required functions in the specific use scenario. Examples include:

[0157] Temperature control of simulated fluids or gases

[0158] Wind speed for simulating fluids on custom assemblies of physical components

[0159] Display metrics of the projected physical behavior (e.g., the displayed size of the forces acting in a beam)

[0160] Properties of electrical components (e.g., the resistance of a resistor or the voltage of a battery)

[0161] The direction of solar radiation on the physical components of the custom assembly

[0162] In one or more embodiments, the turntable anchor has a finite number (eg, 120) of "click-and-stop" radial positions so that the turntable can be turned by a user and held in place.

[0163] Another embodiment of the present invention is to combine Figure 22 550 is depicted, which shows an assembly of structural components attached to an attachment panel 552. The turntable anchor 554 is Figure 4 Turntable anchor 140 is shown similarly, also attached to attachment panel 552. Rather than a user applying distortion to a component's configuration using, for example, indicator wand 51, in this embodiment, a user can rotate turntable anchor 554 to a position indicating a desired wind speed affecting the assembly of component 550. Visualization 556 can display changes in the behavioral properties of assembly 550 caused by an increase or decrease in wind speed. For example, a visualization of air turbulence or other physical effects associated with changes in wind speed can be displayed to the user.

[0164] In yet another embodiment, Figure 23 As shown, Figure 4 A turntable anchor 600, similar to the illustrated turntable anchor 140, can be attached to an attachment panel 602 to allow a user to physically interact with an assembly 604 of physical components attached to the attachment panel 602. In this example, the user can rotate the turntable anchor 604 to a desired position corresponding to a temperature. Based on the position of the turntable anchor 600, a controller can determine a corresponding change to a behavioral attribute of the configuration of the assembly 604. For example, the assembly 604 of physical components can simulate a closed system containing a gas. The physical effect associated with a user-defined increase in the temperature of the gas within the assembly 604 can be a visualization corresponding to an increase in heat, pressure, or other physical effect associated with a change in the temperature of the gas.

[0165] Figure 24 and Figure 25 Tracking beacons 700 and 702 are shown, each docked in an opening or aperture in one or more physical components. Tracking beacons 700 and 702 are Figure 1 The capture device 13 is shown tracking, and the visualizations at 708 and 710 depict the time course path of the beacon's position. This can be applied to any user generated assembly, but is particularly effective for physical applications.

Claims

1. A simulation system comprising: a plurality of physical components, each of the physical components corresponding to one of a plurality of physical component types; an attachment panel comprising an arrangement of attachment locations such that one or more of the physical components can be attached to the attachment panel; a display system configured to provide visual content on or proximate one or both of the attachment panel and the plurality of physical components; a capture device configured to capture image data of a current state of the attachment panel and the plurality of physical components; as well as a controller configured to, when at least one physical component is coupled to the attachment panel: a) determining a configuration of one or more physical components coupled to the attachment panel; b) creating a configuration of virtual components by assigning a virtual component to each physical component in the configuration of physical components; c) identifying a user's physical interaction with the configuration of the physical component and determining a corresponding change to a behavioral attribute of the configuration of the virtual component; d) applying the selected modeling template to the current arrangement of virtual components to generate a visualization including an indication of a physical effect associated with the change in the behavioral attribute; as well as e) causing the display system to display the visual content; Wherein, the at least one physical component corresponds to a circuit element.

2. The system according to claim 1, wherein: The at least one physical component corresponds to a structural member, and wherein the or each physical component is mechanically coupleable to at least one other physical component.

3. The system according to claim 2, wherein: The physical component is a linear portion selected from the group consisting of one or more of: a beam, a strut, a cable, a rod, and a rope linear portion.

4. The system according to claim 2, wherein: The physical component is an anchor portion selected from the group consisting of one or more of: a pin anchor, a rigid anchor, a pulley anchor, a tile anchor, and a turntable anchor.

5. The system according to claim 2, wherein: The physical component is a joint portion selected from the group consisting of one or more of: a pin anchor, a rigid anchor, a pulley anchor, a tile anchor, and a turntable anchor.

6. The system according to claim 2, wherein: The physical component is a docking portion selected from the group consisting of one or more of: a docking station, a tiled dock, and a tracking beacon.

7. A system according to any one of the preceding claims, wherein: Step c) comprises identifying a distortion applied to the configuration of the physical component and determining a corresponding distortion to be applied to the configuration of the virtual component, and Step d) comprises applying the selected modeling template to the current arrangement of the virtual components to generate a visualization including an indication of a physical effect associated with the distortion.

8. The system according to claim 7, wherein: The controller at least partially determines the distortion of the configuration of the virtual component by performing the following operations: receiving image data from the capture device; identifying a distortion of the physical component based on analyzing the received image data; and Each corresponding virtual component is assigned an equivalent distortion.

9. The system according to claim 7, wherein: The controller at least partially determines the distortion of the configuration of the virtual component by performing the following operations: receiving image data from the capture device; determining a current position of an interaction device, wherein the interaction device is pressed against one or more physical components to cause a distortion in the configuration of the physical components; as well as Based on the current position of the interaction device, a distortion to the configuration of the virtual component is determined.

10. The system according to claim 9, wherein: The interactive device is a handheld device.

11. The system according to any one of claims 9 or 10, wherein: The interaction device is coupled to one or more interconnected support members, one support member being coupled to the attachment panel.

12. The system according to claim 1, wherein: Step c) comprises identifying a change of the user selection to at least one of said circuit elements and determining a corresponding change to the electrical properties of the virtual component configuration, and Step d) comprises applying the selected modeling template to the current arrangement of the virtual components to generate a visualization including an indication of an electrical effect associated with the user-selected change.

13. The system according to claim 12, wherein: Determining a change to one or more electrical properties of a configuration of the virtual component is based at least in part on the controller performing the following operations: receiving image data from the capture device; determining a current position of an interaction device coupled to the attachment panel; as well as Based on the current position of the interaction device, a corresponding change to an electrical property of the configuration of the virtual component is determined.

14. The system according to any one of claims 1-6, 8-10, 12-13, wherein: The display system includes a projector configured to project an image onto a front surface of the attachment panel.

15. The system according to any one of claims 1-6, 8-10, 12-13, wherein: The display system includes an augmented reality helmet, which is used to display the image of the visualization content to the user.

16. The system according to any one of claims 1-6, 8-10, 12-13, wherein: The capturing device includes an infrared sensor and / or an RGB sensor.

17. The system according to any one of claims 1-6, 8-10, 12-13, wherein: The controller is configured to identify individual physical components coupled to the attachment panel and record the determined type for each physical component.

18. The system according to any one of claims 1-6, 8-10, 12-13, wherein: The visualization is generated based on component type, shape, and position of the current arrangement of physical components.

19. The system according to any one of claims 1-6, 8-10, 12-13, wherein: The attachment location includes a recess configured such that when the physical component is attached to the attachment location, the physical component is prevented from rotating relative to the plane of the attachment panel.

20. The system according to any one of claims 1-6, 8-10, 12-13, wherein: The attachment locations define a periodic rectangular array.

21. A method implemented by a controller for generating visualization content on one or both of an attached panel and a plurality of physical components for simulation, the method comprising the steps of: a) identifying the configuration of the virtual components based on the received image data, wherein the configuration of the virtual component corresponds to the configuration of the physical component, Where each physical component is associated with a physical component type, wherein the physical component is arranged to be coupled to an attachment panel, and wherein said physical component comprises at least one physical component, said at least one physical component being attachable to an attachment location of said attachment panel; b) identifying a user's physical interaction with the physical configuration of a component and determining corresponding changes to behavioral attributes of the configuration of the virtual component; c) applying the selected modeling template to the current arrangement of virtual components; d) generating a visualization including an indication of a physical effect associated with a change made to a behavioral attribute; and e) causing a display system to display the visual content; Wherein, the at least one physical component corresponds to a circuit element.

22. The method according to claim 21, wherein Step c) comprises identifying a distortion applied to the configuration of the physical component and determining a corresponding distortion to be applied to the configuration of the virtual component, and Step d) comprises applying the selected modeling template to the current arrangement of virtual components to generate a visualization including an indication of physical effects associated with the distortion.

23. The method according to claim 22, wherein Determining a distortion to a configuration of a virtual component is based at least in part on the controller performing the following operations: receiving image data from a capture device; identifying distortions to the physical component based on analyzing the received image data; and Each corresponding virtual component is assigned an equivalent distortion.

24. The method according to any one of claims 22 or 23, wherein Determining a distortion to the configuration of the virtual component is based at least in part on the controller performing the following operations: receiving image data from a capture device; determining a current position of an interaction device, wherein the interaction device is pressed against one or more physical components to cause a distortion in the configuration of the physical components; as well as Based on the current position of the interaction device, a distortion to the configuration of the virtual component is determined.

25. The method according to claim 21, wherein Step c) comprises identifying a user-selected modification to one or more circuit elements and determining a corresponding change to the electrical properties of the configuration of the virtual component, and Step d) comprises applying the selected modeling template to the current arrangement of the virtual components to generate a visualization including an indication of an electrical effect associated with the user-selected change.

26. The method according to claim 25, wherein Determining a change to one or more electrical properties of a configuration of the virtual component is based at least in part on the controller performing the following operations: receiving image data from a capture device; determining a current position of an interaction device coupled to the attachment panel; as well as Based on the current position of the interaction device, a corresponding change to an electrical property of the configuration of the virtual component is determined.

27. A computer program configured to cause a processor to implement the method according to any one of claims 21 to 26 when the program is executed by the processor.

Images