Information processing system, its control method, and program

Abstract

This information processing system provides a mechanism that allows operation to continue after movement of a portable template component on which multiple physical pieces are placed, even if the position associated with contact with the physical pieces is not continuously acquired. This mechanism maintains the association of information with registered positions. [Solution] A portable template member 120 that can move on a contact detection surface is provided with a plurality of physical pieces 130 that can move and detach individually within the plane, and at least one tactile reference section 122. A storage unit 142 stores information associated with registered positions within the template member 120. A control unit 141 recognizes the movement of the template member 120 from the change in the position corresponding to the reference section 122, and maintains the association of information with registered positions even if contact positions with the physical pieces 130 are not continuously acquired during movement. The arrangement state is determined from the reference section position after movement, the new contact positions are converted to calculated positions within the template member 120, and a registered position is identified from the positional relationship with the calculated position and updated.

Description

Technical Field

[0001] The present invention relates to an information processing system that presents or manipulates information using a contact detection surface such as a touch panel, and more particularly to a technology for performing information processing while a user grasps the arrangement, affiliation, order, hierarchy, or operation state of information through tactile sensation, using a plurality of physical pieces arranged on the contact detection surface and a portable template member on which the plurality of physical pieces can be placed.

[0002] More specifically, the present invention relates to a technology for distinguishing and processing movement of the portable template member and relative movement of the physical pieces with respect to the portable template member, in a configuration where the portable template member itself is movable on the contact detection surface, and the plurality of physical pieces on the portable template member are individually movable relative to the portable template member. The present invention also relates to a tactile user interface that holds, moves, edits, hierarchizes, saves, or redeploys a plurality of information objects as physical positional relationships without relying on vision.

Background Art

[0003] Conventionally, there has been known an information processing technology that guides digital information by presenting sound, vibration, or other sensory stimuli according to the contact position on a touch panel or the like for users who have difficulty directly viewing the screen, such as visually impaired persons. There is also known a technology in which a tactile structure or a physical member is arranged on a screen or an operation surface to allow a user to easily grasp the operation position through tactile sensation.

[0004] Furthermore, there has also been proposed a technology in which a physical object or a physical token is arranged on a contact detection surface, and information processing is performed based on the contact position when a user touches the physical object, the position or orientation of the physical object, or the positional relationship between the physical objects.

[0005] Furthermore, there is a known technique for acquiring the position or orientation of a movable overlay or template placed on a contact detection surface, and associating the user interface with that position or orientation.

[0006] In configurations using physical objects and movable templates, it is conceivable to place multiple physical objects in a single workspace and treat that workspace as a single unit. For example, by associating multiple information objects with multiple physical objects and placing them on a single template, users can understand the interrelationships between multiple information objects as physical arrangement relationships.

[0007] However, if the template that serves as the work area is movable, and the physical objects on that template are also individually movable relative to the template, then it becomes necessary to distinguish and process whether the positional change on the contact detection surface is due to the movement of the template itself, the relative movement of the physical object to the template, or both.

[0008] Furthermore, in configurations where the position of a physical object is acquired only when it is touched by a user, or in configurations where the position of physical objects is not constantly tracked, it is not always possible to continuously acquire the current positions of all physical objects on the template while the template is being moved. Even in this case, it is desirable to be able to continue operations after moving the template without losing the correspondence between the template and each information target.

[0009] Furthermore, when multiple physical pieces are held and manipulated on a template, it is desirable to suppress the unintentional detachment of the physical pieces from the periphery of the template, while also enabling smooth transfer of the physical pieces between the template and the area outside the template on the contact detection surface when necessary. In addition, even if the representative detection position of a physical piece is within the logical area on the template, the bottom surface or part of the outer shape of the physical piece may not be sufficiently on the template, preventing stable tracking during template movement.

[0010] Furthermore, when mapping a haptic workspace using multiple physical pieces to a hierarchical information structure, it is useful to treat the workspace itself as a child element of another workspace, or to save the workspace state containing multiple physical pieces as a single physical piece and then redeploy it onto another workspace. [Prior art documents] [Patent Documents]

[0011] [Patent Document 1] U.S. Patent No. 9965116

[0012] [Patent Document 2] European Patent No. 0499633

[0013] [Patent Document 3] U.S. Patent No. 9240268

[0014] [Patent Document 4] Patent No. 6355795

[0015] [Patent Document 5] U.S. Patent Application Publication No. 2013 / 0124654 [Non-patent literature]

[0016] [Non-Patent Document 1] Brygg Ullmer, Hiroshi Ishii and Robert J. K. Jacob, "Token+Constraint Systems for Tangible Interaction with Digital Information," ACM Transactions on Computer-Human Interaction, Vol.12, No.1, pp.81-118, 2005, DOI:10.1145 / 1057237.1057242.

[0017] Non-Patent Document 2 Javier Marco, Eva Cerezo and Sandra Baldassarri, "Playing with Toys on a Tabletop Active Surface," Proceedings of the 9th International Conference on Interaction Design and Children (IDC 2010), pp.296-299, 2010, DOI:10.1145 / 1810543.1810596.

[0018] Non-Patent Document 3 Yvonne Jansen, Pierre Dragicevic and Jean-Daniel Fekete, "Tangible Remote Controllers for Wall-Size Displays," Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2012), pp.2865-2874, 2012, DOI:10.1145 / 2207676.2208691.

[0019] Non-Patent Document 4 Martin Kaltenbrunner and Jens Vetter, "Tquencer: A Tangible Musical Sequencer Using Overlays," Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction (TEI 2018), pp.429-434, 2018, DOI:10.1145 / 3173225.3173233. SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0020] One of the main objects of the present invention is, in a configuration using a portable template member movable on a contact detection surface and a plurality of physical pieces individually movable thereon, even when the position associated with contact with the physical pieces is not continuously acquired while the portable template member is being moved, to maintain the correspondence between the registered positions in the portable template member where the physical pieces were arranged and the information, and provide an information processing system capable of continuing the operation after the movement.

[0021] Further, one of the other objects of the present invention is not to require continuous tracking of the physical individual identity of the physical pieces, to determine the position within the template member from the newly acquired contact position after the movement of the portable template member, identify one registered position based on the positional relationship between that position and the registered position held before the movement, and update the registered position, thereby enabling the continuity of the information associated with the registered position to be maintained.

[0022] Further, one of the other objects of the present invention is to provide at least one reference part on the portable template member that is recognizable by a user through tactile sensation and allows acquisition of the position on the contact detection surface, so that the arrangement state for converting the position on the contact detection surface into the position within the portable template member can be determined from the position corresponding to the reference part.

[0023] Furthermore, the present invention also aims to achieve, as a secondary objective, the suppression of deviation of physical pieces on a portable template member, the transfer of physical pieces between the contact detection surface and the area outside the template, notification of unstable states near boundaries, hierarchical arrangement of multiple templates, saving and redeploying workspace states, operation of magnetically held physical pieces, and separation of tactile confirmation and input confirmation. [Means for solving the problem]

[0024] To solve the above problems, an information processing system according to one aspect of the present invention comprises: a contact detection surface capable of detecting a contact position; a portable template member placed on the contact detection surface and movable along the contact detection surface; a plurality of physical pieces that can be arranged within a two-dimensional region on the portable template member, are movable in the in-plane direction relative to the portable template member, and can be detached from the portable template member and transferred onto the contact detection surface; at least one reference part fixed to the portable template member and identifiable by touch by a user; storage means; and control means.

[0025] The storage means registers at least a portion of the positions within the portable template member where each of the plurality of physical pieces is arranged as a registration position, and stores information associated with each of the registration positions. Here, the registration position is different from the individual identification information of the physical piece itself, and is a storage position for holding information associated with a position within the portable template member.

[0026] The position on the contact detection surface corresponding to the reference part that the user has contacted among the at least one of the reference parts can be acquired in connection with the contact. Furthermore, when any one of the plurality of physical pieces is placed on the portable template member, the position on the contact detection surface in connection with the user's contact with that physical piece can be acquired.

[0027] The control means recognizes the movement of the portable template member based on a change in position on the contact detection surface corresponding to at least one reference portion. During the movement, even if the position on the contact detection surface associated with the user's contact with one or more of the registered positions is not continuously acquired, the control means maintains the association of information to each of the one or more registered positions.

[0028] After the movement, the control means determines the arrangement state of the portable template member for converting the position on the contact detection surface to a position within the portable template member, based on the position on the contact detection surface corresponding to the at least one reference portion. If, after the movement, a new position on the contact detection surface is acquired due to the user's contact with one of the plurality of physical pieces arranged on the portable template member, the control means converts the newly acquired position to a position within the portable template member based on the arrangement state to determine the calculated position.

[0029] The control means, based on the positional relationship between the calculated position and the one or more registered positions, identifies one of the one or more registered positions and updates the registered position stored in the storage means based on the calculated position. This process does not require the identification or tracking of the same physical piece before and after movement, and updates the registered position based on the currently acquired position while maintaining the correspondence between the registered position and the information.

[0030] The portable template member may be made of paper, resin sheet, fiber sheet, thin film, thin plate, or other thin layer material. In the main embodiment, the thickness and material configuration of the portable template member are such that the contact position associated with the user's contact with the physical piece can be acquired by the contact detection surface through the template member.

[0031] The physical piece may be positioned at any two-dimensional location on the portable template member. Furthermore, the physical piece is not limited to a slider or knob mechanically integrated with the portable template member, but may be an independent object that can be removed from the portable template member and moved to another location on the contact detection surface.

[0032] The control means may recognize the movement of the portable template member based on the fact that at least two contact positions, including the reference portion, have moved in coordination. Alternatively, it may determine that the movement of the portable template member maintains its orientation based on the fact that the relative direction or positional relationship of the at least two contact positions has been maintained within a predetermined range. In the following context, when explaining the distinction from the relative movement of physical pieces, the movement of the portable template member itself may be referred to as "parent movement" or "parent movement operation," the arrangement state of the portable template member may be referred to as "parent arrangement," and the relative movement of physical pieces with respect to the portable template member may be referred to as "child relative movement."

[0033] The parent movement operation may be initiated by an explicit transition to a movement mode, a predetermined contact with a reference unit, simultaneous contact with multiple reference positions, a voice command, or a combination thereof. During the parent movement operation, changes in contact positions acquired as the portable template member moves may not be determined as individual edits of information corresponding to the physical piece.

[0034] At least a portion of the periphery of the portable template member may be provided with a deviation constraint and tactile boundary portion that restricts the physical piece from unintentionally moving outside the area of ​​the portable template member, and allows the user to identify the boundary of the area by touch.

[0035] Another part of the periphery may be provided with an opening for transporting the physical piece between the contact detection surface and the portable template member. The opening may have a thin-walled portion, an inclined portion, a curved portion, an open end, or a combination thereof, which allows the physical piece to be transported without lifting it or substantially releasing the holding force.

[0036] The control means may determine, based on the relationship between the representative position obtained by the contact detection surface for the physical piece and the contact surface shape or outer shape of the physical piece, whether the representative position is within the area of ​​the portable template member, and whether the physical piece belongs to a dangerous area where there is a risk that the physical piece may not stably follow the portable template member when the portable template member is moved.

[0037] If it is determined that the physical piece is located within the hazardous area, the control means may guide the user to move the physical piece further inside the portable template member using voice, vibration, sound image, warning sound, or other non-visual feedback.

[0038] The physical piece may be held to the contact detection surface or the portable template member by magnetic force. In one example, the physical piece has a magnet, and a magnetic material is placed on the back side of the contact detection surface. In another example, the physical piece has a magnetic material, and a magnet or magnetic layer is placed on the back side of the contact detection surface.

[0039] If a physical piece has a magnet, a non-magnetic outer periphery may be provided around the magnet, and a predetermined separation distance may be maintained between the magnets when the outer peripheries of adjacent physical pieces are in contact with each other. This may suppress at least one of the following: tipping over, inversion, attraction between magnetic surfaces, or unintended displacement caused by the magnetic action of adjacent physical pieces.

[0040] The physical piece may have a tactile recognition section that allows the user to identify the type, value, or corresponding information of the physical piece through touch. The control means may be configured so that the input of information corresponding to the physical piece or the target of the operation is not determined solely by touching the tactile recognition section.

[0041] The control means may receive a declaration input representing a value, type, or information determined by the user based on the tactile knowledge unit, and may store the contact position acquired in response to a predetermined confirmation operation in association with the information represented by the declaration input. This allows the information represented by the physical piece to be reassociated with the new position based on the new contact position and the user's declaration input, even after a period of time has passed during which the physical piece is not detected by the contact detection surface.

[0042] A second portable template member may be placed on the first template member, which is the portable template member. The control means may associate the second template member as a subgroup of the information group corresponding to the first template member.

[0043] In this case, the control means may distinguish and process the movement of the first template member, the relative movement of the second template member with respect to the first template member, and the relative movement of the physical piece on the second template member with respect to the second template member. Also, if the second template member is moved to another parent template member, the information groups below the second template member may be reassigned as parent members together.

[0044] The control means may store information corresponding to multiple physical pieces on a portable template member, and the workspace state including their arrangement relationships, as information corresponding to a single physical piece specified by the user. Hereinafter, this process of abstracting and associating the workspace state with a single physical piece may be referred to as "pieceization."

[0045] If one physical piece that has been pieced up and another portable template member are specified, the control means may, based on the workspace state, redeploy the information included in the workspace state onto the other portable template member, associating it with multiple physical pieces.

[0046] In the redeployment described above, the control means may determine the number of physical pieces to be used for redeployment, the information division unit, the presentation order, or the arrangement rules based on the semantic structure of the information included in the workspace state, the available area of ​​the other portable template member, the number of available physical pieces, and at least one of user settings or editing purposes.

[0047] The contact detection surface or the portable template member may have reference lines, periphery, grooves, ribs, or other arrangement guides that can be identified by touch by the user. The control means may set the presentation order, logical order, or editing order of information corresponding to a plurality of physical pieces based on the physical arrangement order of the plurality of physical pieces arranged along the arrangement guide.

[0048] The information processing system may further include a function to store and later restore multiple physical pieces, piece positions, corresponding information, workspace state, operation history, or processing history by artificial intelligence (AI). Furthermore, based on the stored state, the system may guide the user to the repositioning location of the physical pieces using voice or other non-visual feedback.

[0049] The information processing system may associate physical pieces or portable template members with visual attributes that can be identified visually by sighted individuals and tactile attributes that can be identified by touch by visually impaired individuals or other tactile users. Furthermore, these attributes may be associated with the same logical object on the system, enabling shared referencing with third-party terminals.

[0050] The information processing system may convert a physical object specified on a third-party terminal into voice, directional guidance, vibration, or other perceptual stimuli to guide the user. It may also determine whether the object the user is touching matches or does not match the object specified by the third party, and provide corrective guidance if there is a mismatch.

[0051] The aforementioned information processing system may acquire the functional meaning of an object on an external visual user interface, regardless of its visual position, color, shape, or display hierarchy, and rearrange the object corresponding to that functional meaning to a tactile position easily accessible to the user. Furthermore, it may save the operations instructed from the outside and their context as a history and use it to rearrange objects to tactile positions in similar contexts.

[0052] The aforementioned information processing system may associate teaching materials, forms, personal information, products, shopping carts, business partners, payment information, and other semantic objects with physical pieces or portable template components. This allows for tactile operations such as referencing and answering tactile teaching materials, entering information into forms based on semantic items, selectively providing only necessary information, and confirming and finalizing each product.

[0053] The information processing system may assign mutually identifiable perceptual attributes such as left / right sound, virtual sound source direction, pitch, timbre, speaker, playback speed, rhythm, sound length, volume, or vibration to multiple contact locations or multiple information objects. Furthermore, the assignment of these perceptual attributes may be changed based on the user's unique auditory discrimination characteristics.

[0054] The information processing system may use contact with the left and right hands or multiple fingers to select the target of operation or the operation mode based on the type of fingers, finger combination, or body reference position of one hand, and continuously change parameters such as the audio playback speed, information granularity, hierarchical depth, comparison target, interpretation depth, or acoustic ratio based on the rotational operation, amount of movement, or relative position change of the other hand.

[0055] Each of the above configurations can be implemented individually or in any combination. Furthermore, the contact detection surface is not limited to a capacitive touch sensor, but may also be a physical piece or other method capable of detecting user contact or proximity. The holding of the physical piece is not limited to magnetic force, but may also be friction, walls, grooves, adhesion, mechanical engagement, or other holding means.

[0056] The present invention may be implemented as an information processing system, as a control method for the information processing system, or as a program that causes a computer to execute each of the above processes. The control method or program may include, for example, a process of associating and storing information with registered positions within a portable template member, a process of acquiring a position on a contact detection surface corresponding to a reference part, a process of recognizing the movement of the portable template member based on the change in the position, a process of maintaining the association of information with registered positions even when the position associated with contact with a physical piece is not continuously acquired during movement, a process of determining the arrangement state from the position corresponding to the reference part after movement, a process of converting a new contact position to a calculated position based on the arrangement state, a process of identifying a registered position from the positional relationship with the calculated position, and a process of updating the registered position based on the calculated position. [Effects of the Invention]

[0057] According to the present invention, multiple independent physical pieces can be individually manipulated on a portable template member, while the portable template member itself can be moved on a contact detection surface. Furthermore, by associating information with registered positions within the portable template member and storing it therein, the arrangement of information can be managed as registered positions, separate from the physical individual identity of the physical pieces.

[0058] Furthermore, according to the present invention, even if the position associated with contact with one or more registered positions of a physical piece is not continuously acquired during the movement of the portable template member, the association of information with the registered position can be maintained. Therefore, it is possible to maintain continuity in information processing without requiring the continuous tracking of all physical pieces individually or the continuous identification of the same physical piece before and after movement.

[0059] Furthermore, the system can determine the placement state after movement from the position on the contact detection surface corresponding to a reference part that the user can recognize by touch, convert the newly acquired contact position to a calculated position based on that placement state, and identify and update a registered position from its positional relationship with the calculated position. This makes it easier to continue operations even after the portable template member has been moved, by using the association with the retained information.

[0060] Furthermore, by combining deviation constraints and tactile boundaries, entrances and exits, hazard zone determination, magnetic retention, separation of tactile confirmation and input confirmation, nesting of parent and child templates, and pieceification and redeployment of workspace states, it is possible to configure a haptic user interface that allows multiple information objects to be moved, edited, saved, reconfigured, or co-referenced while maintaining their physical location, order, and hierarchy. [Brief explanation of the drawing]

[0061] [Figure 1] This is a schematic diagram showing the overall configuration of an information processing system according to one embodiment of the present invention.

[0062] [Figure 2] This is a plan view showing an example of the arrangement of a portable template member, a reference section, and multiple physical pieces.

[0063] [Figure 3] This is an explanatory diagram showing the relationship between the portable template component before movement, after movement, and re-contact with the physical piece.

[0064] [Figure 4] This is an explanatory diagram showing the state transitions of the normal operation state S1, template movement recognition state S2, mapping hold state S3, post-movement placement state calculation state S4, re-contact / calculation position verification state S5, and registered position update state S6.

[0065] [Figure 5] This is an explanatory diagram showing an example of parent movement and attitude determination using two or more contact points, including a reference point.

[0066] [Figure 6] These are plan and cross-sectional views showing examples of deviation constraint / tactile boundary areas, entrances / exits, and hazardous areas.

[0067] [Figure 7] This is a cross-sectional view showing an example of a magnetic retention structure in which a magnet is provided on the physical piece side and a magnetic material is placed on the back side of the contact detection surface.

[0068] [Figure 8] This is a cross-sectional view showing an example of a magnetic retention structure in which a magnetic material is provided on the physical piece side and a magnet or magnetic layer is placed on the back side of the contact detection surface.

[0069] [Figure 9] This is an explanatory diagram showing an example of a physical piece equipped with a non-magnetic outer periphery and a tactile non-magnetic portion.

[0070] [Figure 10] This is an explanatory diagram illustrating an example of a process that separates tactile confirmation from input confirmation.

[0071] [Figure 11] This is an explanatory diagram showing an example of a nested tactile workspace in which a second template member is placed on a first template member.

[0072] [Figure 12] This is an explanatory diagram illustrating an example of the process of moving parent and child templates hierarchically and re-parenting a child template component to a different parent template component.

[0073] [Figure 13] This is an explanatory diagram illustrating the process of consolidating a workspace state containing multiple physical pieces into a single physical piece.

[0074] [Figure 14] This is an explanatory diagram illustrating the process of redeploying a modularized workspace state to another portable template component.

[0075] [Figure 15] This is an explanatory diagram illustrating an embodiment in which the document structure of claims, etc., is broken down into multiple physical pieces, edited, and then reintegrated.

[0076] [Figure 16] This is an explanatory diagram showing an example of an editing lane that utilizes a contact detection surface or the periphery of a portable template member.

[0077] [Figure 17] This is an explanatory diagram illustrating the saving and restoration of workspace states and guidance for repositioning physical pieces.

[0078] [Figure 18] This is an explanatory diagram illustrating an example of a learning system that combines tactile learning sheets with a workspace for answering questions.

[0079] [Figure 19] This is an explanatory diagram illustrating an example of a co-reference system that associates the visual, tactile, and logical attributes of physical game pieces.

[0080] [Figure 20] This is an explanatory diagram illustrating an example of bidirectional collaborative pointing, where a third-party terminal guides a physical object to a haptic user.

[0081] [Figure 21] This is an explanatory diagram illustrating an example of the process of redistributing functional meaning from an external visual user interface onto a haptic user interface.

[0082] [Figure 22] This is an explanatory diagram illustrating an example of form input based on semantic items and the selective provision of personal information.

[0083] [Figure 23] This is an explanatory diagram illustrating an example of product verification, shopping cart, and payment authorization using a haptic workspace.

[0084] [Figure 24] This is an explanatory diagram illustrating an example of a process for assigning different acoustic attributes to multiple tactile locations or information targets.

[0085] [Figure 25] This is an explanatory diagram illustrating an example of controlling an object and continuous parameters using one or more fingers of both hands.

[0086] [Figure 26] This is an explanatory diagram showing an example of the data structure for workspace status information, template placement information, and piece correspondence information in one embodiment of the present invention. [Modes for carrying out the invention]

[0087] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments are merely examples for realizing the present invention, and the present invention is not limited to these configurations. Furthermore, the configurations described in each embodiment can be combined with each other to the extent that they do not contradict the technical standards. <Embodiment 1: Physical configuration of the system and contact position detection>

[0088] The information processing system 100 shown in Figure 1 comprises a contact detection surface 110 and a contact detection device 111, a portable template member 120 placed on the contact detection surface 110, a plurality of physical pieces 130A, 130B, 130C, etc. arranged on the portable template member 120, and a control device 140. In this embodiment, before describing the registered position or arrangement state in terms of information processing, these physical configurations and the configuration in which the position is acquired from the user's contact will be described first.

[0089] The contact detection surface 110 is a surface capable of acquiring a position corresponding to contact or proximity by the user's fingers, hands, or other body parts, and the contact detection device 111 outputs coordinates or position information representing that position to the control device 140. In this embodiment, a capacitive touch sensor is used as an example, but as will be described later, other detection methods capable of acquiring a position corresponding to contact or proximity may be used.

[0090] The portable template member 120 shown in Figure 2 is a member that is placed on the contact detection surface 110 and is movable along the contact detection surface 110. The portable template member 120 can be made of, for example, paper, resin sheet, fiber sheet, thin film, thin plate, or a composite material thereof. In this embodiment, the thickness and material configuration of the portable template member 120 are such that the contact position when the user touches the physical piece 130 or the reference part 122 can be acquired on the contact detection surface 110 via the portable template member 120. A two-dimensional arrangement area for arranging multiple physical pieces 130 can be provided on the portable template member 120.

[0091] The physical piece 130 is a physical object that a user can grasp or touch with their fingers, etc., and that can be placed on the portable template member 120 or on the contact detection surface 110. The physical piece 130 is not required to be a slider or knob, etc., mechanically integrated with the portable template member 120, and can move independently in the in-plane direction on the portable template member 120, and can be removed from the portable template member 120 and moved to another location as needed. The physical piece 130 may have an individual unique ID, but in this embodiment, an individual unique ID is not required.

[0092] The portable template member 120 is provided with at least one reference part 122. The reference part 122 is a physical part whose position is fixed relative to the portable template member 120 and which the user can distinguish from other parts by touch and touch. The reference part 122 can be made up of a protrusion, recess, rib, notch, fixed piece, magnet, conductive part, or other shape or material.

[0093] When a user touches the reference section 122, the contact detection surface 110 and the contact detection device 111 acquire the position on the contact detection surface 110 corresponding to that contact. Similarly, when a user touches a physical piece 130 on the portable template member 120, the position on the contact detection surface 110 corresponding to that contact is also acquired. In this embodiment, when a capacitive contact detection surface 110 is used, the capacitance change originating from the human body may be transmitted to the contact detection surface 110 via a part of the physical piece 130, a conductive part, a human body contact part, the portable template member 120, or other structures. The position acquired here corresponds to the user's contact and does not mean that the physical piece 130 itself is constantly being tracked when there is no contact.

[0094] In the two-dimensional arrangement area shown by the dashed line in Figure 2, multiple physical pieces 130A, 130B, 130C, etc., can be placed. As shown in "Parent Movement" in Figure 2, the portable template member 120 itself can be moved along the contact detection surface 110, while as shown in "Child Relative Movement," a single physical piece 130 can be moved individually in-plane relative to the portable template member 120. Furthermore, as shown in physical piece 130D and "Detachment" in Figure 2, the physical piece 130 can be detached from the portable template member 120 and transferred to an area outside the template on the contact detection surface 110 or onto another portable template member.

[0095] Even after the user releases their hand, the physical piece 130 can be used as a tactile cue based on its physical position. On the other hand, in an embodiment where the contact detection surface 110 acquires its position upon contact by the user, the current position of the physical piece 130 after the user releases their hand is not necessarily continuously acquired. Therefore, the fact that the position has not been acquired should be treated as a separate issue from the fact that the physical piece 130 has not physically moved.

[0096] The control device 140 shown in Figure 1 may include a control unit 141, a storage unit 142, an audio input / output unit 143, and a communication unit 144. The contact position acquired from the contact detection device 111 is input to the control unit 141, which processes the information by referring to the information stored in the storage unit 142 and can present the information to the user via the audio input / output unit 143, etc. Information may also be sent and received between the control device 140, a user terminal 180, or a third-party terminal 190 via the communication unit 144.

[0097] In this specification, "acquisition of contact position" means obtaining the position on the contact detection surface 110 corresponding to the user's contact. The acquisition of a contact position alone does not automatically determine what information the contact specified, what operation was confirmed, or the individual identity of the physical piece 130 present at that position. The specification of the target or the confirmation of the operation may be combined with the registration position, placement status, operation status, user instructions, and other information described later.

[0098] Hereinafter, the movement of the portable template member 120 itself will be referred to as "parent movement" or "parent movement operation" in the context of explaining the distinction from the relative movement of the physical piece 130, the arrangement state of the portable template member 120 will be referred to as "parent arrangement," and the relative movement of the physical piece 130 with respect to the portable template member 120 will be referred to as "child relative movement." Based on the above physical configuration and contact position detection, the information model for maintaining the position in association with information will be explained next. <Embodiment 2: Information Model, Registration Location, and Arrangement State>

[0099] In this specification, "registered position" refers to a position within the portable template member 120 that is stored in association with the information. "calculated position" refers to a position obtained by converting a newly acquired position on the contact detection surface 110 into a position within the portable template member 120 based on the arrangement state of the portable template member 120 at that time. "arrangement state" is the state used to convert a position on the contact detection surface 110 into a position within the portable template member 120, and includes, for example, a reference position on the contact detection surface 110 and, if necessary, an in-plane rotation angle or orientation.

[0100] As shown in Figure 26, the storage unit 142 may store workspace status information 150. The workspace status information 150 may include template placement information 151 representing the arrangement status of the portable template members 120, piece correspondence information 152 including the registration position and information associated with the registration position, and relationships between registration information, confirmation status, or history information 154, etc. Information obtained from the contact detection device 111, external information processing device 170, and user terminal 180 shown in the lower part of Figure 26 may be used to generate or update this status information.

[0101] As a concrete example of the data structure shown in Figure 26, the template placement information 151 can store "Template 1, Reference position = (500, 300), Rotation angle = 0 degrees," and the piece correspondence information 152 can store "Registered information A, Parent = Template 1, Registration position = (40, 30), Information = Information 1" and "Registered information B, Parent = Template 1, Registration position = (90, 30), Information = Information 2." Here, Information 1 and Information 2 may be text, items, functions, values, or other information objects or references to such information objects.

[0102] The aforementioned registration information A and registration information B are internal identifiers or names for distinguishing records within the computer, and do not mean that the physical piece 130 itself is assigned a unique individual ID, A or B. In embodiments where the physical piece 130 has a unique individual ID, that ID may be retained as additional information, but the unique individual ID of the physical piece 130 is not required to maintain the correspondence between the registration location and the information.

[0103] A local coordinate system may be set within the portable template member 120. For example, the local position corresponding to the reference unit 122 is set to (0,0), and other positions within the portable template member 120 are expressed in coordinates relative to this reference. This allows the registered position to be maintained as a position within the template member 120 even if the absolute placement position of the portable template member 120 on the contact detection surface 110 changes.

[0104] If the orientation of the portable template member 120 is known and the rotation angle is 0 degrees, when the position (500,300) on the contact detection surface 110 is acquired upon contact with the reference part 122, the position (x,y) on the contact detection surface 110 can be converted to a position (x-500,y-300) inside the portable template member 120. In this case, the information that the reference position = (500,300) and the rotation angle = 0 degrees represents the arrangement state for converting the position on the contact detection surface 110 to a position inside the template member 120.

[0105] As an example of a generalized coordinate transformation, if the local position within the portable template member 120 is u, the position on the contact detection surface 110 is p, the position of the local coordinate origin on the contact detection surface 110 is t, and the transformation representing in-plane rotation is R, then the arrangement can be represented as p = R(u) + t. Therefore, the calculated position within the template member 120 can be obtained from the newly acquired position p by the inverse transformation (pt) of u = R. When the orientation of the coordinate axes coincides with only translation, this corresponds to the process of subtracting the origin position from the position on the contact detection surface 110.

[0106] When acquiring a single position for the reference section 122, if the orientation of the portable template member 120 is known, if the orientation can be determined from other information, or if the condition that the orientation does not change is met, the arrangement state can be determined using that single position. On the other hand, if free in-plane rotation is allowed and the orientation cannot be acquired separately, it is not treated as if the rotation angle has been determined from only one position.

[0107] When dealing with free in-plane rotation, the rotation angle or orientation may be determined using a second reference unit 123 whose positional relationship is known or other attitude information. A specific example of using two reference units will be described in Embodiment 4, which will be described later. Thus, the arrangement state includes components necessary to convert the position on the contact detection surface 110 to a position within the portable template member 120, depending on the detection information used.

[0108] When a user places a physical piece 130 on a portable template member 120, and the position (540, 330) on the contact detection surface 110 is acquired upon contact with the physical piece 130, the position (40, 30) within the template member 120 can be determined based on the placement state. When information 1 is specified as the registration target, the control unit 141 can store this position as the registration position (40, 30) in the piece correspondence information 152, associating it with information 1. Similarly, the registration position (90, 30) can be determined from the contact position (590, 330) and stored in association with information 2.

[0109] The registered position is a memory location used to store information associated with a position within the portable template member 120, and is not the result of continuous measurement of the current position of the physical piece 130 when it is not in contact. Therefore, even if there is a period during which the contact position to the physical piece 130 is not acquired, the registered position or its association with the information is not deleted solely on the grounds that the position is not acquired. Furthermore, the physical position of the physical piece 130 is not deemed to have not changed solely on the grounds that the position is not acquired.

[0110] In the information model described above, instead of the individual identity of the physical piece 130, the correspondence between the registered position within the portable template member 120 and the information can be used as the basis for continuity in information processing. Below, using Figures 3 and 4, a series of state controls that update the registered position while maintaining this correspondence when the portable template member 120 is moved will be explained. <Embodiment 3: State control for registration position update and template movement>

[0111] Figures 3 and 4 illustrate the recognition of movement of the portable template member 120, the maintenance of mapping during movement, the calculation of the placement state after movement, the comparison of the calculated position with the registered position, and the updating of the registered position. Figure 3 shows the relationship from "before parent movement" on the left, through "parent movement operation," to "after parent movement" on the right, and then "re-contact" with the physical piece 130. Figure 4 represents this process as a state transition, with S1 being the normal operation state, S2 being the template movement recognition state, S3 being the mapping maintenance state, S4 being the post-movement placement state calculation state, S5 being the re-contact / calculated position comparison state, and S6 being the registered position update state. Figure 4 shows a typical state transition and does not cover all inputs or exceptions.

[0112] Before initiating the state control, the control unit 141 identifies the portable template member 120 to be used and, upon contact by the user with at least one reference section 122, acquires the position on the contact detection surface 110 corresponding to the reference section 122. Based on this position and information regarding the required orientation or posture, the control unit 141 sets the template placement information 151.

[0113] In the above specific example, the position on the contact detection surface 110 corresponding to the reference section 122 is set to (500, 300), and the rotation angle is set to 0 degrees. Under this arrangement, the local position within the portable template member 120 can be determined by subtracting (500, 300) from the position on the contact detection surface 110. When dealing with free rotation, necessary reference position or orientation information is used in combination.

[0114] Next, the user places multiple physical pieces 130 on the portable template member 120. The control unit 141 converts the position on the contact detection surface 110 acquired upon contact with each physical piece 130 to a position within the template member 120 based on the arrangement state at that time, and stores at least a portion of the converted positions as registered positions, associating the information with each registered position and storing it in the piece correspondence information 152.

[0115] For example, if information 1 is held as the target for registration and a contact position (540,330) to a physical piece 130 is obtained, the control unit 141 determines the position (40,30) within the template member 120 and associates it with information 1 as the registration position (40,30). Subsequently, it determines the registration position (90,30) from another contact position (590,330) and associates it with information 2. The following describes a specific example of state transition using these two registration positions.

[0116] When comparing the calculated location obtained after movement with the registered location, predetermined conditions may be set. For example, a reception area may be defined as a predetermined distance from each registered location, and one registered location that includes the calculated location within that reception area may be identified. Alternatively, one registered location that is closest to the calculated location among the registered locations within a predetermined upper distance limit may be identified. If there are no candidates or if one cannot be identified from multiple candidates, the registered location will not be updated, and the user will be asked to re-specify or make additional contact.

[0117] The correspondence between the registration location and the information is not the same as the individual unique ID of the physical piece 130. Even when internal identifiers such as registration information A or registration information B are used, they are for distinguishing the registration information in memory. Therefore, in the post-movement update described later, individual unique IDs, tactile attributes, or user declarations can be used as additional conditions as needed, but it is not necessary to identify that the physical piece is the same individual before and after the move.

[0118] Assuming the initial setup and registration described above are complete, the processes from S1 to S6 in Figure 4 will be explained below.

[0119] and

[0120] This is S1, paragraph

[0121] This is S2, paragraph

[0122] and

[0123] This is S3, paragraph

[0124] This is S4, paragraph

[0125] This is S5, paragraph

[0126] These correspond to S6 respectively.

[0119] S1 in Figure 4 shows the normal operating state. In S1, when the position corresponding to a single registered piece of information is to be changed individually, the control unit 141 identifies the single registered piece of information to be updated from the contact position, current arrangement state, operating state, or user instruction, and determines the new position within the template member 120 from the position acquired after the movement. With the target identified, the arrangement of the information within the template member 120 can be changed by updating the new position as the registered position of that registered piece of information.

[0120] When updating individual registration locations in S1, check for overlaps with other registration locations, candidate competition, or changes in affiliation as needed. If the update target cannot be uniquely identified, do not arbitrarily assign it to other information based solely on the location difference; instead, request user re-designation, additional contact, or registration procedures. When only updating the location, retain the information content associated with the registration location.

[0121] S2 in Figure 4 shows the state in which the movement of the portable template member 120 is recognized. When the user moves the portable template member 120 itself, as shown in "Parent Movement Operation" in Figure 3, the control unit 141 recognizes the movement of the portable template member 120 based on the change in position on the contact detection surface 110 corresponding to at least one reference part 122. For example, if the position corresponding to the reference part 122 changes from (500,300) to (700,400), under the condition that the orientation is maintained, the template member 120 can be treated as having moved only (200,100) on the contact detection surface 110.

[0122] S3 in Figure 4 shows the state in which the association of information to the registration position is maintained during template movement. When movement of the portable template member 120 is recognized, the control unit 141 maintains the association of information to each of the one or more registration positions stored before the movement. In the above specific example, the association between registration position (40,30) and information 1, and the association between registration position (90,30) and information 2 are maintained.

[0123] In S3, the position on the contact detection surface 110 associated with the user's contact with the physical piece 130, which was positioned at one or more registered locations, does not need to be continuously acquired during the movement. The control unit 141 does not release the association with the registered location or information solely because the position is not acquired. Furthermore, the fact that the position is not acquired does not mean that the physical piece 130 has remained physically in the same position or that it has moved in exactly the same way as the template member 120.

[0124] S4 in Figure 4 shows the state for determining the template placement after the "parent movement" in Figure 3. After the movement, the control unit 141 determines the placement state after the movement based on the position on the contact detection surface 110 corresponding to at least one reference part 122. In the above specific example, the placement state is determined where the reference position after the movement is (700, 400) and the rotation angle is 0 degrees. If a valid reference position is continuously acquired until the end of the movement, that position may be used, and if position acquisition is interrupted, the position may be acquired by re-contacting the reference part 122.

[0125] S5 in Figure 4 corresponds to "re-contact" in Figure 3, and shows the state in which a calculated position is determined from the contact position obtained after movement and compared with the held registered position. For example, if a new position (742,431) on the contact detection surface 110 is obtained due to contact with the physical piece 130 after movement, the control unit 141 determines the calculated position (42,31) based on the arrangement state. Then, the positional relationship between the calculated position (42,31) and the registered positions (40,30) and (90,30) is compared according to the predetermined conditions. "Has the relative position within the parent changed?" in Figure 4 represents an example in which this comparison determines whether or not there is a substantial positional change that warrants changing the value of the registered position for a specific candidate registered position. If a registered position cannot be identified, the process does not proceed to S6, and re-designation or additional contact may be requested.

[0126] S6 in Figure 4 shows the registration position update state. In the above specific example, if the calculated position (42,31) is within the reception area of ​​the registration position (40,30) and can be identified by conditions such as being clearly closer than the registration position (90,30), the control unit 141 updates the registration position (40,30) of the registration information A stored in the storage unit 142 to (42,31) based on the calculated position (42,31). At this time, the association with information 1 is maintained, and after the update, it becomes "Registered information A, Parent = Template 1, Registration position = (42,31), Information = Information 1". This update does not require identification that the physical piece 130 before and after the move are the same individual. After processing, it is possible to return to S1.

[0127] While the portable template member 120 is moving, the control unit 141 does not determine that the positional change that occurs on the contact detection surface 110 due to the movement is an individual movement of the information associated with the registered position. This prevents the child registered information from being incorrectly processed as having moved individually simply because the entire template member 120 has been moved.

[0128] As another embodiment not shown in Figure 4, parent movement operations and child relative movement may be processed simultaneously. In this case, the control unit 141 acquires the parent's reference position and required orientation, and the contact position of the child whose target has been identified, as information for the corresponding time points, and converts the child's contact position to a relative position within the parent at each time point. Then, for the same child information whose target has been continuously identified, the movement of the entire parent and the relative movement of the child may be processed separately by comparing the temporal change in the relative position within the parent. If rotation is involved, the placement information including the orientation at each time point is used instead of simply subtracting the displacement on the contact detection surface 110. In sections where the parent placement, the time of contact, or the child's target correspondence is not determined, the child relative movement in that section may not be determined, and the parent placement may be reacquired or the child re-specified. It is not assumed that the movement path in sections where the child's contact position is not acquired can be reconstructed solely from those inputs. <Embodiment 4: Two-point movement and posture maintenance>

[0129] Figure 5 shows an example of determining the movement and orientation of the template member 120 using the positions of the first reference part 122 and the second reference part 123, which are fixed to the portable template member 120 on the contact detection surface 110, before and after their movement, and their respective displacement vectors.

[0130] The portable template member 120 may be provided with a first reference section 122 and a second reference section 123. Both reference sections 122 and 123 are fixed to the portable template member 120.

[0131] The control unit 141 distinguishes and registers which reference unit each position corresponds to for the first reference unit 122 and the second reference unit 123. For example, the user may be instructed to touch both reference units in a predetermined order during initial registration or reacquisition, and the order of contact may be associated with the role of each reference unit. As long as contact is continuously identified, the positions of the two points can be acquired while maintaining the association, and it is not necessary to repeat the registration operation with each movement. If the roles of the two points are unknown, such as after the acquisition of positions has been interrupted, they are re-identified using a shape that can be distinguished by touch and the registration operation, or other identification information. Reference units with the same physical shape may be used, but two points whose roles are unknown are not always treated as being associated in the same order based only on left / right on the contact detection surface or contact number. When two points that have been sequentially contacted are used to calculate a single posture, they are acquired under the condition that the arrangement of the template does not change during that time.

[0132] As shown in Figure 5, when a user makes contact with the first reference unit 122 and the second reference unit 123, and the positions of the two points whose roles have been identified move in a coordinated manner, the control unit 141 can treat this as a parent movement operation of the portable template member 120. For example, in determining a parallel movement that maintains posture, the displacement vectors of the two points over the same period may be compared, and it may be determined that the difference is within a predetermined range, and that the change in the length and direction of the line segment connecting the two points is within a predetermined range. To distinguish minute fluctuations in the contact position from the start of movement, determination conditions such as the amount of displacement or duration may be set. In parent movements involving rotation, the two points do not necessarily move in the same direction by the same distance, so in addition to determining a parallel movement, changes in arrangement based on the positional relationship of the two points, as described below, may also be used.

[0133] In one example of determining an arrangement that includes in-plane rotation, the control unit 141 uses the local coordinates of the first reference unit 122 and the second reference unit 123, whose roles have been identified, and the positions of two points on the contact detection surface 110 corresponding to them. The difference between the direction of the vector from the first reference position to the second reference position in the local coordinates and the direction of the detected vector from the first reference position to the second reference position is determined as the in-plane rotation angle. Next, the position of the local coordinate origin on the contact detection surface 110 is determined by subtracting the local coordinates of the first reference position transformed by this rotation angle from the detected first reference position. The distance between the two points may be used to confirm consistency with the distance between the stored reference positions. If the distance between the two points is too short for the detected position fluctuation, if the correspondence of roles is unclear, or if the inconsistency with the stored distance exceeds a predetermined range, this calculation result may not be used as the final arrangement, and the system may request reacquisition of the reference positions.

[0134] For example, if the change in the orientation of the line segment connecting two points before and after the movement is within a predetermined range, and the change in the distance between the two points is also within a predetermined range, the control unit 141 may determine that the portable template member 120 has moved in parallel while substantially maintaining its orientation. If the change in orientation exceeds the range, and the roles of the two points and the distance between the reference positions are consistent, the placement information may be updated as a parent movement including rotation. Each allowable range can be set according to the input portion of the reference unit, the distance between the two points, the detection variation of the contact position, or the required operating resolution. Here, the orientation determined from the two points is the in-plane direction in an example where the target area is treated as maintaining its shape in a plane, and the mere coincidence of the two points does not mean that the presence or absence of deformation of the entire template or the out-of-plane orientation has been confirmed.

[0135] One of the two reference parts may be positioned in a location that is easy for the user to grasp or touch during movement. The other may be positioned in a location that allows the user to confirm the orientation of the portable template member 120 by touch.

[0136] Furthermore, one of the two points may be replaced with another tactile or operating part fixed in position relative to the portable template member 120, instead of the second reference part 123. It is also possible to use at least some of the multiple contact positions as a reference representing the movement of the parent template, and the remaining contact positions as conditions for the success of the movement operation or as auxiliary information. However, in the example of determining the rotation angle from the two points, it is used under the condition that the positional relationship of both points within the template is known and that this positional relationship is maintained during the operation. When using a contact point on a child piece that can move freely relative to the template, it is not assumed that the point is fixed relative to the parent, but rather that it is combined with operating conditions or other attitude information that allows the use of the point. The mere fact that two points, a parent fixed reference and a freely moving point, have been obtained does not mean that the rotation of the parent and the relative movement of the child can always be uniquely separated.

[0137] If one of the positions is lost while the parent's position is being updated using two points, the control unit 141 retains the last valid position information but does not treat the current rotation angle as having been determined from the remaining point. If the orientation is acquired separately or the condition that it will not change is maintained, the update of available position components may continue, but if it is unclear whether that condition is met, the state requiring reacquisition of orientation is maintained. For example, it may instruct the user to "touch the other reference part as well," and after reacquiring the roles and current positions of both reference parts, it resumes interpreting the contact position of the child that depends on the parent's position. Registered information is retained while the position is unconfirmed, and the determination of target identification or editing that depends on the position is suspended, but it is not necessary to uniformly stop operations until a target is identified in another valid work area. <Embodiment 5: Deviation restriction and tactile boundary, entrance / exit and hazardous area>

[0138] Figure 6 shows a plan view example of the peripheral structure of the portable template member 120 on the left, and a cross-sectional view example of the same structure on the right. The plan view example shows the deviation constraint / tactile boundary section 124, the entrance / exit 125, the thin-walled transport section 126, the hazardous area, the logical area, the representative positions of the physical piece 130, and the direction of "guidance further inward".

[0139] A deviation constraint and tactile boundary section 124 may be provided on at least a portion of the periphery of the portable template member 120. The deviation constraint and tactile boundary section 124 physically restricts the physical piece 130 from unintentionally moving outside the area of ​​the portable template member 120, and also functions as a cue for the user to recognize the boundary of the portable template member 120 by touch.

[0140] The deviation constraint and tactile boundary section 124 can be composed of ribs, walls, bumps, steps, grooves, thickened sections, or other shapes. The height and shape should be set so as not to unnecessarily hinder the operation of the physical piece 130, and so that the user can recognize the boundary.

[0141] An entrance / exit 125 may be provided in another part of the periphery. The entrance / exit 125 is a part for transferring the physical piece 130 from the contact detection surface 110 onto the portable template member 120, or vice versa.

[0142] As shown in the cross-sectional example on the right side of Figure 6, the entrance / exit 125 may be provided with a thin-walled transfer section 126, an inclined surface, a curved surface, an open end, or a combination thereof. This allows the physical piece 130 to be slid and transferred between the template outer area on the contact detection surface 110 and the portable template member 120 without lifting the physical piece 130 or substantially releasing magnetic or other holding forces.

[0143] When transferring the physical piece 130 from the area outside the template on the contact detection surface to the portable template member 120, even if the representative position detected by the contact detection surface 110 enters the logical area of ​​the portable template member 120, the bottom surface or the entire outer shape of the physical piece 130 may not be sufficiently placed on it.

[0144] The hazardous area shown in the plan view example on the left side of Figure 6 may be defined based on the relationship between the representative position of the physical piece 130, the contact surface shape or outer shape of the physical piece 130, and the boundary of the portable template member 120. In one example, the range occupied by the bottom surface or outer shape of the piece is obtained based on the representative position, using the specified type of piece, a pre-registered shape, or the user's dimension input, and a margin is added to this range according to the deviation in position detection and the uncertainty of the boundary position. If the representative position does not coincide with the center of the piece, the range is defined including that positional relationship. If the orientation of the piece cannot be obtained, a range that encompasses the outer shape within the range of acceptable orientations may be used. When the template placement is valid, the obtained representative position is converted to a relative position within the parent, and the inclusion relationship between the range and the template area is determined.

[0145] A hazardous area refers to a range of positions where, even if the representative position of the physical piece 130 is within the logical area of ​​the portable template member 120, there is a risk that the physical piece 130 may not stably follow when the portable template member 120 is moved. For example, the outer shape of the piece can be represented as a range within a predetermined radius from the representative position, and if the shortest distance from the representative position to the template boundary is smaller than the distance obtained by adding a margin for positional deviation to that radius, it can be treated as belonging to the hazardous area. In another example, the bottom shape according to the orientation is used to determine whether the required mounting range fits within the template's area. The required range or margin is determined according to the piece, template, and movement conditions used, and fixed dimensions common to all structures are not required.

[0146] As shown in the left side of Figure 6, "Guided further inward," if the control unit 141 determines that the physical piece 130 is in a dangerous area, it may guide the user to move the physical piece 130 further inward using voice, warning sounds, vibrations, or sound images. For example, it may select a position within the template that encompasses the previously described placement range as a candidate, and present the direction from the currently acquired representative position to the candidate using coordinate directions shared with the user. The movement candidate may be selected considering corners, reception areas for other pieces, or entrances / exits, etc., and if there are no suitable candidates for placement, it will guide the user to use another area or piece. For objects whose position has not been acquired or whose placement information is invalid, it will not be treated as meeting the placement conditions simply because no warning has been issued, and will guide the user to make the necessary contact or reacquire the reference position.

[0147] This allows us to distinguish between the logical determination of being "within the system's area" and the determination of the placement conditions necessary for the physical piece 130 to stably follow the parent's movement. The determination that it does not belong to the danger area indicates that the conditions of shape, arrangement, and margin used are met, but this alone does not measure whether the piece will actually follow all speeds, holding forces, friction, or template deformations. For unknown deviations that occur during the non-detection period, the aforementioned handling for cases where the correspondence is unknown may be applied.

[0148] In examples using magnetic retention, the attraction of the contact detection surface 110 to the back side and the tracking of the physical piece 130 when the portable template member 120 is moved in the in-plane direction are confirmed as separate actions. For example, by combining the friction of the contact surface between the piece and the template, the mass and bottom shape of the piece, the thickness, rigidity and surface condition of the template, the magnetic force, and the acceleration / deceleration during movement, conditions are selected that allow the user to slide the piece independently and maintain the required relative position during normal parent movement. Tracking to the template is not assumed to be achieved solely by a large magnetic force; the structure used is also checked, including the resistance to parent movement due to the holding force or the deformation of the template. <Embodiment 6: Magnetic Retaining A / B and Tactile Piece>

[0149] Figure 7 shows a partial cross-sectional view of a magnetic holding example on the left and a plan view on the right showing multiple physical pieces 130 arranged in close proximity. In the example on the left, the physical piece 130 has a magnet 131, and a magnetic material 132 is placed on the back side of the portable template member 120 and the contact detection surface 110. The physical piece 130 is slidable in the in-plane direction while being held.

[0150] The magnetic force between the magnet 131 and the magnetic material 132 holds the physical piece 130 toward the contact detection surface 110 or the portable template member 120. The holding force may be set to a range that allows the user to slide the physical piece 130 in the in-plane direction and prevents unintended movement after releasing the hand.

[0151] When multiple magnetic physical pieces 130 are placed in close proximity, magnetic attraction, repulsion, tipping, or inversion may occur between the pieces. For this reason, a non-magnetic outer periphery 133 may be provided around the magnet 131.

[0152] The non-magnetic outer periphery 133 can be configured so that when the outer peripheries of adjacent physical pieces 130 come into contact, the internal magnets 131 do not come into contact with each other beyond a predetermined distance. This suppresses at least tipping or inversion due to magnetic action. However, this does not rule out configurations in which some degree of repulsion or misalignment remains.

[0153] As shown in the example of close-proximity arrangement on the right side of Figure 7, a distance is maintained between the magnets 131 when the non-magnetic outer peripheral portions 133 are in contact with each other. If notches or the like are provided on the non-magnetic outer peripheral portions 133 as contact-recognition parts 134, the shape may be determined so that the magnets 131 do not come into excessive proximity, not only in terms of the position of the notches but also including the outer peripheral portion used for contact with other physical pieces. For example, the distance between the magnets when the outer peripheral portions are in contact with each other in the in-plane direction and relative orientation allowed during use is checked, and the remaining width of the contact portion or the depth of the notch is set so that the necessary distance is not lost due to the notch portion interlocking, etc. In examples aimed at suppressing tipping or inversion, the behavior at and near contact is checked for combinations of magnets used, back-side members, and piece shapes. The width, number, or arrangement for tactile sensation may be set within a range that is compatible with maintaining this distance.

[0154] Figure 8 shows a partial cross-sectional view of the second example of magnetic retention on the left and an example of the arrangement of multiple physical pieces on the right. In this example, a magnet or magnetic layer is placed on the back side of the contact detection surface 110, and iron, steel, or other magnetic material 132 is provided on the physical piece 130 side. A non-magnetic outer periphery 133 may be provided on the physical piece 130, and it is slidable in the in-plane direction while being held.

[0155] In this configuration, since permanent magnets are not provided on the physical piece 130 side, magnetic attraction or repulsion between multiple physical pieces 130 can be reduced. For example, a passive member such as an iron washer can be used as the magnetic material 132.

[0156] In this embodiment, the capacitive coupling path for acquiring the contact position associated with human body contact with the physical piece 130 is not limited to one that goes through the magnet 131 or the magnetic material 132 itself. It may also be transmitted to the contact detection surface 110 via a part of the physical piece 130, a conductive part, a human body contact part, or other structure.

[0157] As shown in the example of the top view on the left and the AA cross-section on the right in Figure 9, a tactile identification section 134 may be provided on the outer periphery or top surface of the physical piece 130. In embodiments that include a non-magnetic outer periphery 133, the tactile identification section 134 may be formed on the non-magnetic outer periphery 133 itself, so that the same outer periphery has both the function of ensuring the separation of the magnets 131 and the function of allowing the user to identify the type, value, or meaning of the physical piece 130 by touch. The tactile identification section 134 can be made up of grooves, notches, unevenness, outer shape, height, width, surface pattern, and other tactile features. Figure 9 also shows an example in which opposing edges are touched with the index finger and thumb, etc.

[0158] For example, the physical piece 130 shown in Figure 9 may be formed in a roughly rectangular shape, and by touching two opposing sides with two fingers, the user can identify a numerical value from the combination of the width, number, or arrangement of notches formed on those two sides. For example, a wide notch may correspond to "5," and one narrow notch to "1," and the sum of these combinations appearing on opposite sides may represent a number from 0 to 9. For example, a combination of one wide notch and two narrow notches can be tactilely identified as "7." For 0, there may be no notch, or another predetermined pattern may be used to correspond to 0. Alternatively, by providing two pairs of orthogonal opposite sides of the roughly rectangular shape with notch combinations representing the same numerical value, the user can identify the same numerical value by touching either pair of opposite sides with two fingers, even if the physical piece 130 rotates in 90-degree increments on the contact detection surface 110. Furthermore, by forming the notch or tactile recognition section 134 substantially symmetrically with respect to the thickness direction of the physical piece 130, the same tactile code can be identified even when the front and back of the physical piece 130 are reversed. By using such tactile codes, the identification result does not change easily with respect to rotation or reversal, the value or type can be determined without visually confirming the orientation of the physical piece 130.

[0159] In the "tactile confirmation" shown on the left side of Figure 10, the user confirms the type, value, or meaning of the physical piece 130 by touching the tactile knowledge identification unit 134. On the other hand, in the "input confirmation" shown on the right side of Figure 10, the input is confirmed by touching the input contact unit 135 provided on the contact detection surface 110. Therefore, the control unit 141 does not confirm the input of information corresponding to the physical piece 130 simply by touching the tactile knowledge identification unit 134, thereby separating the "operation of confirming what kind of piece it is by touching it" from the "operation of confirming the value or meaning of that piece as input."

[0160] Confirmation may involve contact with the input contact area 135, contact for a predetermined period of time, contact with another finger, voice command, gesture, or a combination thereof.

[0161] Alternatively, after the user recognizes "7" from the tactile knowledge unit 134, they may make a declaration input such as "I've decided on 7 here," and the position where they touched the input contact unit 135 at the same or nearby time may be associated with the declared value "7."

[0162] In this case, even if the physical piece 130 is temporarily removed from the contact detection surface 110 and the system is unable to track it for a period of time, the user can still identify it as "7" using the tactile knowledge unit 134. Then, after repositioning it to a different location, the user can reassociate the value "7" with the new contact location by making a declaration such as "Place a 7 here." <Embodiment 7: Nested tactile workspace>

[0163] Figure 11 shows an example of a child group in which a second portable template member 120B is placed on a first portable template member 120A, and physical pieces 130D and 130E are placed on the second template member 120B. Figure 12 shows three processing examples for this nested configuration: "parent template movement," "child template relative movement," and "re-parenting to another parent."

[0164] The second template member 120B may be smaller than the first template member 120A, and can be treated as a single physical information group on the first template member 120A.

[0165] The control unit 141, for example, identifies the registration information of the first template member 120A and the second template member 120B, and accepts an operation to associate the second template member 120B with the first template member 120A as a child. Identifying each template may be done by contacting a reference part when placement is valid, specifying a name, or contacting a reference part according to guidance in an operation state where an object is specified. The control unit 141 stores in the registration information of the second template member 120B a reference to the registration information of the first template member 120A and its relative placement relative to the first template member 120A. The registration information corresponding to physical pieces 130D, 130E, etc. on the second template member 120B stores the position relative to the second template member 120B and a reference to corresponding information. If the same contact position is included in the area of ​​multiple templates, the operation target is identified by the already placed parent-child relationship, the specified object, or additional instructions, and the affiliation or operation hierarchy is not treated as being determined solely by overlapping on a plane.

[0166] In one example where the arrangement is represented by position and rotation in a plane, the control unit 141 includes and stores the coordinate transformation from the first template member 120A to the contact detection surface 110 and the coordinate transformation from the second template member 120B to the first template member 120A in the workspace state information 150. By applying the coordinate transformation in the order of child to parent, and then parent to contact detection surface, to the local position corresponding to the physical piece 130 on the second template member 120B, the estimated position on the contact detection surface 110 can be determined. Conversely, by applying the parent transformation and the inverse transformation of the child transformation in order to the contact position where the target and operation hierarchy are specified, the relative position of the physical piece 130 within the child template can be determined. This allows the movement of the first template member 120A, the relative movement of the second template member 120B with respect to its parent, and the relative movement of the physical piece 130 with respect to its child to be processed by distinguishing between the registration information or coordinate transformation hierarchy to be updated. The coordinate transformation uses information that is effective in representing the arrangement of the target, and if the information is insufficient, the reference position, etc., is reacquired.

[0167] As shown in "Parent Template Movement" on the left side of Figure 12, in an example where the first template member 120A moves and the second template member 120B and the physical pieces 130D, 130E, etc. on it follow while maintaining their relative positions to their respective parents, the control unit 141 updates the positioning information of the first template member 120A while maintaining the relative position of the second template member 120B within its parent, the relative positions of the physical pieces 130D, 130E, etc. within its children, and the corresponding information and parent-child relationships. In this case, the subordinate positions on the board are estimated positions calculated from the updated parent positions, and are not the result of remeasuring all uncontacted child templates or physical pieces. If a discrepancy in position or correspondence is detected during subsequent contact, the target and its affiliation are confirmed, and the relative movement is registered or re-specified. It is not necessary to rewrite the content of subordinate text, etc., solely because of the parent's movement.

[0168] As shown in "Child Template Relative Movement" in the center of Figure 12, when only the second template member 120B is moved relative to the first template member 120A, the control unit 141 holds the second template member 120B as the target of movement and converts the arrangement obtained from the reference position of the second template member 120B into a parent relative arrangement using the valid arrangement of the first template member 120A. Then, it updates the relative arrangement held for the second template member 120B, and the child relative positions and corresponding information of the physical pieces 130 under it are held unless individual relative movements are recognized. In examples where both the parent and child move, the relative arrangement is determined using the valid arrangement information at the corresponding time. Furthermore, when moving the physical pieces 130 on the child individually, the target corresponding to the physical piece 130 is identified, and then the relative position relative to the child template member 120B is updated. If arrangement information for multiple layers is insufficient, the layer to be changed is inferred from the change in position alone and not determined.

[0169] As shown in "Re-parenting to a Different Parent" on the right side of Figure 12, in an example of re-parenting the second template member 120B to a different parent template member, the control unit 141 identifies the second template member 120B to be transferred and the parent template member to which it will be transferred based on the role of the source object and the role of the destination object, respectively. After placement by the user, the control unit 141 obtains the reference position of the second template member 120B and uses the valid placement information of the destination parent template member to determine the relative placement of the second template member 120B with respect to the new parent. In response to the re-parenting instruction, the control unit 141 changes the parent referenced by the registration information of the second template member 120B to the new parent and registers its relative placement. At this time, the registration information corresponding to the information elements and physical pieces of the second template member 120B and below may be handled while maintaining the internal parent-child relationships and local positions of the children and below. It is not necessary to confirm re-parenting based solely on placement or overlapping areas.

[0170] In reparenting, in examples using a hierarchical structure that references a single parent, the control unit 141 verifies that the new parent is not the object being transported or one of its subordinate elements, thus avoiding the creation of circular parent-child relationships. If the object is being relocated to the same parent, the parent reference is not changed, and it can be treated as an update to the relative placement. If the object, destination, or required placement information is not yet determined, the new parent-child relationship is not determined, and the object is kept in a state of being transported or placement unconfirmed. In implementations that store the child lists of the old and new parents separately, the parent reference and both lists are determined as a single update that is consistent, and a state where only a part has been changed is not treated as completed. Even if the operation is canceled midway, it is not assumed that the actual object has returned to its original position, and a reconfirmation of the placement may be requested. It is not necessary to stop operations in unrelated work areas for these confirmations.

[0171] For example, a second template member 120B representing a museum plan can be placed on top of a first template member 120A representing a travel plan, and multiple physical pieces representing exhibited works can be placed on top of that. By moving the second template member 120B to a different travel plan, the entire set of information under the museum plan can be moved to a different context.

[0172] In the prototype according to this embodiment, it was confirmed that magnetic retention and contact detection are achieved even in a two-layer state consisting of a parent template member and a child template member. However, the conditions for achieving these functions may vary depending on the materials used, thickness, magnetic force, sensor characteristics, etc. <Embodiment 8: Mapping (pieceification) and redeployment of workspace state to a physical piece>

[0173] Figure 13 shows the process of abstracting and associating a workspace state containing multiple physical pieces with a single physical piece 130X or information corresponding to that physical piece 130X. In this embodiment, this process is hereinafter referred to as "pieceization". Figure 14 shows the process of redeploying the pieceized workspace state as multiple physical pieces on another portable template member 120C.

[0174] As shown in "Before Pieceification" on the left side of Figure 13, the workspace state to be pieceified may include a portable template member 120, a reference unit 122, and multiple physical pieces 130A, 130B, 130C, etc. The control unit 141 holds a group of registered information identified via the reference unit 122, etc., or a set of registered information identified by a predetermined multiple selection operation, as the source for pieceification. The control unit 141 reads at least part of the content of the information included in the set, a reference to a saved version, the local position, order, parent-child relationship, editing state, or history of each element. If the selection range includes a child template, whether or not to include its subordinates is determined according to the rules of operation or the user's specification. The control unit 141 may distinguish and maintain whether the position to be saved is the last confirmed position, an estimated position obtained from the parent's placement update, or an unconfirmed position, and does not treat the current position of all uncontacted physical pieces as having been measured for pieceification.

[0175] In the example where the physical piece 130X shown as "After Piece Creation" on the right side of Figure 13 is used as the storage destination, the control unit 141, while retaining the set of pieces from which the pieces were created, accepts contact with the physical piece 130X in the operation state for specifying the storage destination, and identifies the registered information or newly created registered information corresponding to the contact location as the storage destination. In response to an instruction to consolidate the workspace state, the control unit 141 saves the information and arrangement relationships corresponding to the set as the storage state in the storage unit 142, etc., and sets information that references the storage state in the registration information of the storage destination. Contacts specifying the storage destination are processed according to their role, and the physical piece 130X is not added to the set of pieces from which the pieces were created solely because of such contact. If other information is already associated with the storage destination, it may be possible to select replacement, use of a different storage destination, or cancellation by predetermined instructions.

[0176] As described above, the piece creation process in this embodiment is a process that abstracts and associates multiple physical pieces and their corresponding workspace states with a single physical piece or the information corresponding to that physical piece. If the source or destination for piece creation is not specified, the user is requested to specify it again, and the process is completed after confirming the consistency between saving the saved state as a usable state and changing the correspondence of the save destination.

[0177] In one example of reproducing the state at the time of saving, the control unit 141 duplicates the content of the information to be saved, or stores a reference to a version from which the content at the time of saving can be read. This method does not mean that the content at the time of saving is preserved, and is distinct from a method that references the latest information shared by others. If the set of source pieces and their contents are determined before creating the registration information for the piece-making destination, even if the physical pieces that will be saved are within the original range, it is possible to avoid incorporating the reference to the saved state to be set into the saved state itself. Piece-making can be performed as saving the state and associating it with the save destination, and does not necessarily erase the original multiple pieces or the original information. If the content of the target is changed between the acquisition of the content to be saved and the confirmation of saving, the version to be saved may be checked or reacquired.

[0178] The physical piece 130X, which has been pieced up, may be removed from the original portable template member 120. The control unit 141 retains registration information that references the storage state even during periods when the position is not acquired, and does not delete the storage state solely on the grounds of the loss of contact. When redeploying, the control unit 141 identifies the storage state to be read using the correspondence between one registration information retained as the target for transport and an instruction specifying the redeployment destination, the user's selection of the name or identification information of the storage state, a declaration regarding a pre-registered tactile code, or obtainable individual identification information. If multiple storage states correspond to the same numerical value or the same tactile code, one is not determined from that alone, but is identified by name, storage time, or additional selection. The reference to the identified storage state can be associated with the registration information of the newly acquired contact location. This process does not mean that the physical individual was tracked during the non-detection period or that it was automatically confirmed to be the same physical individual.

[0179] As shown in "Before Redeployment" on the left side of Figure 14, if the saved state to be read is identified using one pieced-up physical piece 130X as a clue, and another portable template member 120C shown in "After Redeployment" on the right side of Figure 14 is specified as the deployment destination, the control unit 141 holds the saved state and the registration information of the deployment destination as the source and destination for the redeployment process. If the placement of the deployment destination is not confirmed, it guides the user to contact the reference part of the deployment destination to obtain placement information. The control unit 141 reads the saved state and creates a correspondence between the deployment items for each information element or group thereof and the placement candidates based on the local coordinates of the deployment destination. On the right side of Figure 14, the deployed physical pieces 130A, 130B, 130C, 130D, and 130E are shown as examples. The creation of placement candidates does not mean that a physical piece already exists at that location.

[0180] During redeployment, the number of physical pieces to use, the division unit, the presentation order, or the arrangement rules may be determined based on the saved information structure, the available area of ​​the deployment destination, the number of available physical pieces, the editing purpose, or the semantic structure. For example, the control unit 141 creates deployment items using the saved divisions of chapters, sections, claims, or paragraphs and their original order, and divides or aggregates them in light of the number of available pieces declared by the user or the number obtained from valid registration information, and the number that can be placed in the deployment destination area considering the size and spacing of the pieces. If there are many elements, consecutive elements are grouped together and associated with a single higher-level item, and their breakdown is maintained, or they are deployed in multiple stages. If there are few elements, it is not necessary to use all of the available number. If the original information is not structured, candidates may be created from sentence divisions, etc., and the division range may be determined based on the user's specification or the analysis result. If the available area or number of pieces is insufficient, another area or deployment method is guided, and information that does not fit is not arbitrarily discarded.

[0181] For example, the control unit 141 can create an expansion plan using five physical pieces for a saved state including a group of claims, treating claim 1 as a single item and the other claims individually or in groups as separate items. The plan retains the original information elements referenced by each expansion item, the order of the items, and the candidate placements within the expansion destination. The control unit 141 may also provide voice guidance such as "Use five physical pieces," "Arrange from left to right," and "Place the main item at the front and the dependent items to the right." The reference direction for the "from left to right" guidance is determined according to the orientation of the expansion destination or the arrangement direction shared with the user. Even if the number of physical pieces changes, the original claim numbers or informational relationships such as dependency relationships can be retained separately without altering the arrangement.

[0182] In one example where a user places the required number of physical pieces 130 onto another portable template member 120C according to instructions, the control unit 141 selects one deployment item to be registered next and guides the user to contact the physical piece to be used for that item. If the contact position is acquired in response to the guidance and the predetermined registration conditions are met, the control unit 141 determines the local position using the currently valid deployment destination arrangement and registers the reference to the deployment item and the local position in the piece correspondence information 152. If the correspondence between the item and the contact is unclear, if the position overlaps with another registration target, or if there is a risk of mistakenly assigning one contact to multiple items, the control unit 141 requests re-specification before proceeding to the next item. Registered items and unregistered items are distinguished, and the correspondence with the planned items is established by the user sequentially making contacts. If the template moves midway, the registered local positions are retained, and the guidance positions for unregistered items are determined from the updated arrangement.

[0183] During redeployment, the control unit 141 may maintain the saved state and reference to the deployment destination, the planned items, whether each item has been registered for placement, and the next item to be guided. For example, if the process is interrupted after registering 2 out of 5 items, the control unit can distinguish between the 2 registered items and the 3 unregistered items and resume the process, preventing the duplicate generation of already completed items by simply re-accepting the same input. An embodiment may be used in which confirmed items can be used even before all items are registered, but unregistered items are not treated as completed placement. In this example, a procedure is used in which the piece registered by the user is left at the placement position and the next piece is placed, and replacement or removal during non-detection after registration is not unconditionally detected. If the validity of the parent placement or the position of a registered piece is lost when resuming, the necessary reference position or target is re-confirmed. Maintaining the saved state of information when canceling is separate from automatically removing the actual placed object.

[0184] In this redeployment, it is not necessary to reproduce the same physical layout as before the pieceization. For example, even if the original workspace consists of three physical pieces, the sub-elements of the saved information may be further divided into five or six physical pieces and redeployed depending on the information structure or editing purpose. The correspondence between each item in the redeployment destination and the original information elements may be maintained separately from the number of physical pieces, and if correspondence is intended for multiple locations representing the same element, the duplication may be distinguished and recorded. In an example where the redeployment destination is an independent editing work state, the work state can be created by referring to the content at the time of saving or its version, so that changes in the redeployment destination do not unconditionally affect the original saved state or the original workspace. A distinction may be made between a method of updating the original information in response to an explicit reflection instruction and a reference method of sharing information.

[0185] When a physical piece 130 representing a redeployed information item is retrieved and edited in an area outside the template on the contact detection surface 110, the control unit 141 identifies the item to be retrieved and the destination work area, and sets the correspondence in that work area by acquiring the position or registering the repositioning while retaining the object to be moved. As an example, an editing work state is created based on the content of the original item or its version, and while retaining information that identifies the original workspace and the original item, the work state is broken down into multiple physical pieces for editing. Since a piece may not exist at the original position, that position may not be unconditionally accepted as a placed input position, and may be distinguished from a state of editing or unconfirmed placement. After integrating the editing result back into a single information unit, it can be reflected in the original item or a separately specified item, and placement guidance and contact position registration can be performed to return the physical piece to the higher-level workspace.

[0186] The editing work state retains, for example, the identification information of the original workspace, the identification information of the original information item, the original position within the parent or in the array, and the version of the item's content at the start of editing. These are records that distinguish information within the computer and do not require the individual ID of a physical piece. The return destination is not recorded solely by the absolute coordinates on the contact detection surface at the start of editing, but is determined by referring to the current parent arrangement and the arrangement relationship of the item. Even if the position or orientation of the original workspace changes, if the content of the original information item has not changed, it is not necessary to refuse to reflect the editing result solely on the grounds of an arrangement update. If the original item is deleted, its affiliation is changed, or its placement destination is used by another object, unconditional write-back or overwriting to the original position can be avoided, and a different reflection destination or placement destination can be specified.

[0187] By repeatedly performing this pieceification and redeployment process, it becomes possible to recursively edit deeper information hierarchies logically without excessively increasing the number of physical layers that users simultaneously perceive through touch. For example, an item that references a certain saved state can be saved as an element of another workspace state, and the referenced item can be retrieved and redeployed when needed. In examples where processing is done as a finite hierarchy, references are managed using identification information that distinguishes the saved states, preventing the redeployment from becoming unfinished due to self-referencing or circular references. Even when grouping into a higher state, the hierarchy can be reused without linking everything into a single document by maintaining references to lower-level information elements or saved states.

[0188] For example, it is possible to create a workspace representing a group of patent claims as a single physical piece, then rearrange that physical piece into multiple claim pieces on another template member, and further decompose the physical piece representing claim 1 into multiple constituent element pieces for editing, before integrating them back into a single claim 1 piece.

[0189] Embodiments 1 to 8 described above illustrate the core physical structure, state control, and hierarchical processing of the present invention. Subsequent embodiments will describe application examples to editing lanes, saving and restoring, educational materials, collaborative work, heterogeneous UIs, forms, purchasing and payment, sound, and bodily input. <Embodiment 9: Editing Lane and Text Editing>

[0190] An embodiment of using multiple physical pieces 130 for sequential editing of information units will be explained using Figures 15 and 16. Figure 15 shows three stages from left to right: "before decomposition," "decomposition and editing," and "reintegration." Figure 16 shows three arrangement examples from left to right: "single editing lane," "sequence change," and "multiple columns representing hierarchy."

[0191] As shown in the "single row of editing lanes" on the left side of Figure 16, the contact detection surface 110 or the portable template member 120 may be provided with editing lanes that can be identified by touch. The editing lanes can be formed by the periphery of the contact detection surface 110, the periphery of the portable template member 120, linear ribs, grooves, steps, arrangement guides 127, or a combination thereof. Figure 16 shows an example in which multiple physical pieces 130 are arranged in a predetermined direction along the arrangement guides 127.

[0192] Multiple physical pieces 130 are arranged sequentially in a predetermined direction on the editing lane. The control unit 141 registers the position, direction, and, if necessary, width of the editing lane based on the contact detection surface 110 or the portable template member 120. As an example, the contact position of each piece whose correspondence with the target information has been identified is converted to a position in the reference coordinate system of the editing lane, and the presentation order, logical order, or editing order is determined by arranging them in order of position components along a predetermined direction from the starting point of the lane. On the template lanes, the orientation of the registered lanes is used as the reference so that the order is not reversed by translation or rotation of the parent only. If the positional relationship is indistinguishable or the correspondence of the target is unconfirmed, additional specifications can be requested.

[0193] As shown in the "Order Change" section in the center of Figure 16, when editing text, each physical piece 130A, 130B, 130C, etc., can be associated with an information unit such as a sentence, clause, paragraph, proposition, constituent elements, or phrase. In one example where the user rearranges the order of pieces, the control unit 141 holds the information unit identified at the start of the movement as the target of the operation, and obtains the position after the movement from the movement for which position acquisition is ongoing or from the re-arrangement registration that holds the target. Next, it recalculates the order of positions based on the editing lane and changes the order of the information unit. After rearranging multiple pieces without acquiring positions, it is not always possible to determine which information unit has moved simply by acquiring the same group of positions again, and the user is asked to re-specify the correspondence between information units and positions as needed.

[0194] When removing a physical piece 130 from an array is treated as deleting, holding, or moving information to another workspace, the control unit 141 identifies the information unit to be removed and determines the processing type based on the operation state or user instruction that accepts the action. For example, if a delete instruction is received for the identified target, the reference to it is removed from the array; if a hold instruction is received, it is removed from its placement in the array and stored separately. In the case of transfer to another area, the destination and new registration position are specified. The removal of the piece or deletion of information is not determined solely by the loss of contact. Furthermore, when inserting another unit between two known information units, the insertion target and its new position or insertion order are identified, and a reference to the target is added to the sequence information.

[0195] As shown in the "Multiple Columns Representing Hierarchy" on the right side of Figure 16, the editing lane is not limited to a single column, but may have multiple columns, staggered columns, indentation positions, or hierarchical areas. For example, when placing the main item in the first column and its subordinate items in the second column, the control unit 141 maintains the correspondence between the registration position of each column or area and the hierarchical role accepted there. In response to instructions for movement or change of affiliation where the target information is specified, a parent-child relationship may be set using the role and the specified higher-level item. On the other hand, in operations that simply change the display column, the original logical hierarchy can also be maintained. If multiple main items are candidates, higher-level items are specified, and the parent is not selected and associated solely based on depth or horizontal displacement.

[0196] In the example of claim editing shown in "Before Disassembly" on the left side of Figure 15, physical pieces 130 corresponding to claims 1, 2, 3, etc., are placed on the portable template member 120, and the control unit 141 identifies the information for claim 1 by contacting and removing the physical piece 130 representing claim 1. As indicated by the arrow "Remove from template" in Figure 15, the physical piece 130 corresponding to claim 1 is moved to the area outside the template on the contact detection surface 110, and a reference to the original workspace and original item, as well as the version of the information content at the time of removal, are maintained. It is not necessary to fix the arrangement of the original workspace to the state at the start of editing, and the user can move the original portable template member 120 even during editing.

[0197] As shown in "Decomposition and Editing" in the center of Figure 15, the control unit 141 or the external information processing device 170 may analyze the document structure of claim 1 and divide it into constituent elements, clauses, phrases, or other units. Figure 15 shows an example in which configuration 1, configuration 2, and configuration 3 are each associated with physical pieces 130 and placed on the editing lane. Using existing delimiter information, user-specified ranges, or delimiter candidates obtained through analysis, multiple editing items are generated that retain references to the original information, the range or content of each part, the original order, and delimiters. If the division range cannot be determined by analysis alone, candidates are presented and the user's specification is accepted.

[0198] When a user touches each physical piece 130, the control unit 141 identifies the target editing item based on the position verification and operation status described above, and may present its title, summary, or full text by voice or other means. When in a state where editing instructions are accepted, the control unit 141 holds the identified editing item and accepts input for that item by voice input, external terminal input, or dialogue with the generating AI. For example, the replacement content from voice or terminal may be stored as the work content for that item, and if a modification proposal obtained from the generating AI is used, it may be presented as a candidate associated with the target item and the content at the time of processing, and the work content may be updated according to a predetermined adoption operation. If the content of the target changes while waiting for a response, the new content will not be automatically overwritten with a response to the old content. Information viewing and confirmation of editing content can be distinguished according to the operation status.

[0199] After editing is completed in the center of Figure 15, the control unit 141 identifies the set of information units to be integrated and obtains their edited content and confirmed order information. The order may be determined from the position on the editing lane where correspondence has been confirmed, the order specified by the user, or the retained document structure. When the user specifies the physical piece 130 to be integrated and gives an instruction to confirm the integration, the control unit 141 connects the edited content according to the order and necessary delimiters, or reconstructs it according to the retained document structure, to generate the integrated information unit shown in "Reintegration" on the right side of Figure 15. Then, it associates the information unit or its reference with the registered information of the integration destination.

[0200] The right side of Figure 15 shows an example where the reintegrated physical piece 130 corresponds to the edited claim 1. For example, in response to an instruction such as "put claim 1 back on this piece," the control unit 141 generates the edited claim 1 from the selected information units and order of constituent elements, and associates it with the registration information corresponding to the physical piece 130 identified as the integration destination. In this case, what is integrated is not the physical piece 130 itself, but the information specified using them as clues. When returning the result to the original workspace, the destination is identified based on the reference to the original workspace and original items held at the time of retrieval, or based on the user's new specification. Even if the original template has moved during editing, the placement can be guided based on the currently valid template placement information 151 and the parent position of the return destination.

[0201] In one example of reflecting editing results back to the original item, the control unit 141 compares the version or content itself of the content held at the start of editing the item with the content of the item immediately before the reflection. If they match, only that item may be updated, and any additions, edits, or changes to the placement of other items in the original workspace may be preserved. If they do not match, both the current content and the editing results may be preserved, and the reflection may be postponed, allowing the user to choose to adopt the changes after comparison, re-edit, or save them to a different item. Information that manages updates to the parent placement may be distinguished from information that manages the version of the information content. The same check is also performed if the content or affiliation of the target changes between the time the content update is prepared and confirmed. If the parent moves during placement guidance, the guidance and contact position registration should be redone using the updated criteria, and the old absolute coordinates should not be used to map to a different target.

[0202] In this way, by breaking down information into multiple physical units, physically rearranging them, editing them as needed, and then reintegrating them, users can tactilely edit text and other sequential structures without relying on their vision. <Embodiment 10: Saving, restoring, and guiding the rearrangement of workspace state>

[0203] Figure 17 shows three stages: "Saving" on the left, "Storage Location / Restore Control" in the center, and "Restore / Repositioning Guidance" on the right. On the left, the state of physical pieces 130A, 130B, and 130C on the portable template member 120 is saved; in the center, the control unit 141 controls saving or restoring using the storage unit 142, external information processing device 170, user terminal 180, cloud storage, or other storage locations; and on the right, an example is shown of guiding the repositioning location of the physical pieces based on the saved state.

[0204] As shown in "Save" on the left side of Figure 17, the control unit 141 may read the workspace state information 150 for the range specified in response to an operation to specify a portable template member 120 or a set of selected items, and store it as the saved state. What is saved is not the physical pieces 130A, 130B, and 130C themselves, but rather registration information that uses each physical piece as a clue, the content of the corresponding information or a reference to the version at the time of saving, the relative position to the parent, the arrangement order, the hierarchy, the editing state, the parent-child relationship, the operation history or the AI ​​processing history, etc.

[0205] As shown in the "Storage Destination / Restore Control" in the center of Figure 17, the storage destination may be the storage unit 142, the external information processing device 170, the user terminal 180, cloud storage, or other storage destinations. An external storage area associated with the identification medium 128 may also be used. The control unit 141 maintains the correspondence between the identification information of the storage status and the read destination. When saving to an external location, the system distinguishes between the transmission of a request, confirmation of completion of saving externally, and incomplete saving, and does not treat data as saved solely due to communication interruption.

[0206] When restoring a saved state later, the control unit 141 identifies the saved state to be read from the name of the saved state selected by the user, the save date, and reference information or a reference to the pieced state corresponding to the identification medium 128. The information elements and relationships of the state are read for work, and a rearrangement plan is created based on the effective arrangement of the template or work area specified as the restoration destination, and the number or type of available pieces. When restoring to an area that holds existing information, the control unit distinguishes between deployment to a different area, co-location with existing information, or replacement of a specified range, and does not erase irrelevant information simply by instructing restoration. The number, type, and placement location of the necessary physical pieces 130 are guided according to the plan.

[0207] As shown in "Restoration and Repositioning Guidance" on the right side of Figure 17, if in the saved state physical piece 130A corresponds to the upper left, physical piece 130B to the center, and physical piece 130C to the lower right of the parent template, the control unit 141 converts the saved parent positions to the currently valid template arrangement and finds repositioning candidates corresponding to "upper left," "center," and "lower right" in the figure. The user may be guided to the placement positions of each physical piece 130A, 130B, and 130C by voice, direction guidance, distance guidance, or sound image, etc.

[0208] The control unit 141, similar to the sequential registration of redeployment described above, guides the user in placing and touching pieces while holding the information element to be placed next. When a contact position corresponding to the guidance is acquired and the role and placement conditions of the target are confirmed, the actually acquired local position is set in the registration information corresponding to that element. Planned positions and registered positions may be distinguished, and the registered or unregistered status may be maintained for each item. When resuming after interruption, the saved registration progress is read, but it is not taken for granted that the same physical arrangement remains after a restart or non-detection period, and necessary arrangement checks are performed. To avoid duplicate processing of the re-reception of the same arrangement input as the addition of a new item, work and item identification information may be used.

[0209] The physical arrangement during restoration does not have to be exactly the same as at the time of saving. Depending on the size of the currently used portable template component 120, the number of available physical pieces, or the user's preference, the arrangement may be converted to a different layout while maintaining the identification and semantic relationships of the information elements, by determining which arrangement relationships, order, or hierarchy to maintain. For example, the orientation or spacing of columns may be changed, elements may be aggregated into groups that retain their breakdown if there are many elements, or they may be arranged in multiple stages. If it is not possible to maintain all distances, order, and relationships simultaneously, the conditions to be maintained and the arrangement to be changed will be guided, and elements that do not physically fit will not be omitted without notice.

[0210] Workspace status information 150 may retain editing history by the user or AI. For example, it may record whether the operation involved referencing, adding, deleting, splitting, merging, or moving the information being manipulated, the content or version before and after the operation, and the result of the operation. After restoration, this history can be read to present the previous changes. The process of presenting the history is distinguished from the process of re-executing the operation; simply reading the history of external transmission, purchase, or payment does not mean re-executing them.

[0211] In embodiments where saving or registration can be interrupted and resumed, the control unit 141 records the work identification information, the saving status, the registration progress for each item, and the final status in a non-volatile memory area, making the state in which all the necessary information is available as a single saved result. The result of an interruption before the record of saving completion is not read out as a completed state, and the process may be resumed from the last saved state or resume record that was confirmed to be completed. When resuming, the success of reading the content and logical relationships and the confirmation of the current physical placement of the parent and pieces are treated separately, and the placements that require confirmation are re-registered. Approvals for external transmissions etc. that have been received in the past are not repurposed for new execution approvals solely for the purpose of restoration. <Embodiment 11: Tactile teaching material and learning system>

[0212] Figure 18 shows an example of a learning system in which a "teaching material presentation area" is provided on the left side of the contact detection surface 110 and an "answer or operation area" is provided on the right side. The left side shows an input area where a tactile teaching material sheet 210 and an identification medium 211 are placed, and the right side shows an input area where a portable template member 120 for answers and multiple physical pieces 130 are placed.

[0213] The material presentation area and the answer or operation area shown in Figure 18 may be arranged side by side on the same contact detection surface 110. For example, the tactile material sheet 210 may be placed on the left side of the user, and a portable template member 120 for answers or an input area on the contact detection surface 110 may be provided on the right side.

[0214] The tactile teaching material sheet 210 on the left side of Figure 18 may have diagrams, boundaries, letters, mathematical formulas, symbols, guide lines, or explanatory positions that can be recognized by touch. Figure 18 shows rectangular, circular, and linear tactile elements as examples. In addition, RFID, NFC, two-dimensional codes, or other identification means may be provided as the identification medium 211.

[0215] The control unit 141 may read the corresponding questions, explanations, input formats, valid areas, invalid areas, scoring rules, or the following operations based on the identification information of the tactile teaching material sheet 210. For example, it may obtain a correspondence table between the identification information of elements or questions within the teaching material, the tactile area based on the local coordinates of the teaching material, and references to explanations and answer items. The identification of the teaching material is distinguished from its current placement, and the coordinates within the teaching material are mapped to positions on the contact detection surface 110 by a method of aligning to a predetermined placement position, a method of registering tactile reference positions in the order of guidance, or another position detection method. If the placement or orientation is not confirmed, it is not treated that the contact on the teaching material corresponds to which question based solely on its identification information.

[0216] In the input area on the right side of Figure 18, multiple physical pieces 130 on the portable template member 120 may be associated with input items such as numerator, denominator, operator, or answer. For example, if the teaching material sheet 210 on the left side is for fraction calculations, the control unit 141 associates the registration position of each input area or piece with item information indicating the role of the input, and records the numerical value or operator obtained from the user's declaration, etc., in the corresponding item in response to the confirmation operation.

[0217] When a user touches a specific location on the teaching material sheet 210, the control unit 141 converts that location into coordinates within the teaching material using the valid placement of the teaching material, and identifies an element or question of the teaching material by comparing it with the area in the correspondence table. It reads the answer item corresponding to that element, finds the target location from the valid placement on the answer side, and guides the user with sound, directional guidance, vibration, or other perceptual stimuli. Even if there are multiple items with the same name, they are distinguished using the correspondence between the teaching material, question, and item. If there are multiple candidates due to overlapping areas, etc., or if the placement is unknown, the control unit requests additional specification or reacquisition of the reference position.

[0218] The teaching material sheet 210 may be configured as loose-leaf paper, cards, pages, or other interchangeable media. By separating multiple teaching material sheets 210 into physical bundles such as unprocessed, in processing, processed, or for today's use, users can physically grasp their learning progress and remaining quantity.

[0219] The control unit 141 may associate and store the identification information of each teaching material sheet 210 and the identification information of the problem or item with the user's confirmed answer, answer result, time taken, incorrect answer details, number of retries, or support history. For example, it may record the operation of starting a problem, confirming the answer, and interrupting the problem to determine the elapsed time or number of attempts. The electronic progress status is not automatically updated simply by moving the teaching materials into a physical bundle; the progress may be changed based on reading from the identification medium, completion instructions, or other obtainable inputs. Records of unprocessed, in processing, and processed materials can be used to assist in physical bundle sorting.

[0220] Number pieces, operator pieces, fraction templates, or arrangement guides may be used to represent calculations, fractions, matrices, equations, chemical formulas, document structures, or program structures as arrangements of physical pieces 130. In one example, the declared value or registered meaning of each piece is associated with the roles of rows, columns, numerators, denominators, operators, etc., and structural data is created from the position or arrangement order. The control unit 141 checks whether the necessary items are present and their relationships according to the structural rules corresponding to the selected teaching material or application, and processes them. The meaning or correct answer for all applications is not automatically determined from the arrangement alone, and if the roles, values, or rules necessary for interpretation are missing, the system accepts their specification.

[0221] Thus, in this embodiment, the content of the teaching materials, the answer position, the progress, and the order of operations can be presented by combining interchangeable tactile teaching materials and physical operations on a contact detection surface. <Embodiment 12: Cross-modal collaboration>

[0222] Using Figures 19 and 20, an embodiment in which a haptic user, a sighted person, a supporter, an instructor, or an AI jointly references the same physical information space will be explained. The left side of Figure 19 shows "three-layer attributes of the same physical piece," and the right side shows "joint reference." Figure 20 shows the relationship between the third-party terminal 190 on the left, the bidirectional control by the control unit 141 in the center, and the contact detection surface 110 and portable template member 120 on the haptic user side on the right.

[0223] As shown on the left side of Figure 19, a single physical piece 130 may be associated with a visual attribute 301 that can be visually identified by sighted individuals, a tactile attribute 302 that can be tactilely identified by tactile users, and a logical attribute 303 managed by the system. For example, each attribute can be set in the registration information of the target of operation using a known attribute table corresponding to the type of piece, a declaration from the user or supporter, or another detection means that can read the attributes.

[0224] Visual attributes 301 may include colors, numbers, letters, shapes, or patterns. Tactile attributes 302 may include notches, bumps, outlines, heights, grooves, or tactile symbols. Logical attributes 303 may include piece type, role, correspondence information, affiliated group, or hierarchy.

[0225] For example, a three-layer identification system could be set up for a single physical piece 130, where it appears as "red, number 3" to a third party, is recognized as a predetermined notch pattern to a haptic user, and corresponds to TYPE_4 or "greeting message" in the system.

[0226] As shown in "Joint Reference" on the right side of Figure 19, the third-party terminal 190 may display the location of the registration target, visual attributes 301, tactile attributes 302, logical attributes 303, correspondence information, contact status, selection status, the portable template member 120 to which it belongs, or the hierarchy, based on the physical piece 130 handled by the haptic user. The control unit 141 associates the information necessary for joint reference between the haptic user and the third-party terminal 190.

[0227] If a tactile user touches a physical piece 130B or the like as instructed on the right side of Figure 20, and the registered information of the target to be operated is identified from that touch, the control unit 141 may notify the third-party terminal 190 of information corresponding to the "touch result" and "highlighting" shown from the center to the left of Figure 20, and highlight the corresponding display. If the target is not identified, it is displayed as an unidentified touch or candidate to distinguish it from touching an identified target.

[0228] On the left side of Figure 20, an example is shown where targets A, B, and C are displayed in the physical information space on the third-party terminal 190, and the third party selects target B. This selection is sent to the control unit 141 in the center of Figure 20 as "target selection," and the control unit 141 identifies the target to be guided from that information and uses the corresponding parent placement and registration position to determine the guidance position on the haptic user side on the right side of Figure 20.

[0229] The "guidance" shown from the center to the right in Figure 20 may be provided by non-visual methods such as voice instructions like "5cm to the right," sound images representing left / right or front / back directions, pitch changes according to distance, vibrations, tactile stimuli from the surroundings, etc. In one example, the user is asked to specify the contact position where the guidance will begin, and the amount of movement in the shared direction is calculated from the difference between that position and the guidance position.

[0230] The control unit 141 may compare the identification information of the target designated by a third party with the identification information of the target identified from the user's contact and determine whether they match or not. Whether the unit of comparison is the registered information of the piece, an information element, or a group is determined according to the purpose of the guidance. If the target the user has contacted cannot be identified, the user is asked to re-specify, separate from the confirmation of a mismatch. If there is a mismatch and the guidance positions of both targets are valid, a corrected guidance such as "The designated target is to the right" is presented based on the difference in position. It is not necessary to determine whether individual targets match based solely on whether their name, color, or value is the same.

[0231] If the user, a third party, and the AI ​​each have different objects of attention, each may be maintained as an independent logical cursor or attention state. The third-party terminal 190 may display each object of attention distinctly, and the haptic user may be presented with distinctions using sound or perceptual stimuli.

[0232] The control unit 141 may store the distinction between the user, a third party, or the AI, the time, the reference to the designated object, the explanatory content, the object identified from the contact, and the modification content or processing result as a joint cognitive history. In examples using independent logical cursors or attention states, it distinguishes whose designation was received by whom, and does not change the user's editing target or input confirmation status simply because the third party's attention target has changed.

[0233] Using a shared cognitive history, it may be possible to estimate tactile codes that users tend to confuse, concepts that took time to understand, effective explanation methods, or next guidance methods. For example, mismatches between designated targets and contact targets can be aggregated by attribute or explanation method, and guidance with fewer re-designations or shorter required times can be prioritized as candidates. If the number of data points to aggregate is small or the differences are unclear, existing settings can be maintained or the user's preferences can be confirmed. Trends obtained from these records may be used in selecting guidance candidates, but not treated as results of directly measuring the user's understanding or ability. <Embodiment 13: Semantic retargeting between heterogeneous UIs>

[0234] Figure 21 shows the three stages: "External Visual UI" on the left, "Semantic Retargeting" in the center, and "Relocation to Haptic UI" on the right. This is an example in which the functional meaning of the object being operated on is obtained from the external visual user interface 500, and the control unit 141 relocations that meaning to the portable template member 120 and physical piece 130 on the haptic side.

[0235] The visual user interface 500 shown on the left side of Figure 21 may display operation targets such as "Screen Settings," "Back," and "Confirm." In Figure 21, an example is shown where "Screen Settings" is displayed as a green operation target in the upper right corner of the screen, but in this embodiment, the functional meaning is used rather than the screen position or color itself.

[0236] As shown in "Semantic Retargeting" in the center of Figure 21, the control unit 141 or the external information processing device 170 acquires not only screen coordinates, colors, or shapes, but also their functional meaning 501 for the target of operation on the visual user interface 500. In one example, a pair of identification information referring to the target of operation and its functional meaning 501 is created using the name, role, state, and operation type of screen elements provided by the external device, or a pre-registered function correspondence table.

[0237] For example, if a green button located in the upper right corner of the screen means "screen settings," the control unit 141 may acquire the functional meaning 501 of "screen settings" without copying the visual characteristics themselves, such as "upper right," "green," or a specific icon, to the haptic user interface.

[0238] The control unit 141 determines the placement candidates for haptic operation targets corresponding to the functional meaning 501, based on the current contact position indicated by the user, the registered body orientation, the operation history, the available area on the portable template member 120, or the required number of options. For example, it selects candidates from the available area that does not overlap with the existing reception area, in order of proximity to the reference contact position. The control unit 141 associates the reference to the original screen element and the operation to be performed with the haptic registration information. It distinguishes between cases where only the information assignment is changed and cases where the user is guided to the placement of a new piece and contact registration is performed, and it does not treat the physical piece 130 as having moved or been placed simply because placement candidates have been created.

[0239] Figure 21, on the right, shows an example of "rearranging to the haptic UI" where functions such as "back," "current position," and "screen settings" are associated with physical pieces 130 on the portable template member 120. When the user selects "screen settings" through a predetermined selection operation, the control unit 141 outputs an operation request for that element to an external device according to the reference to the original screen element held in the registration information. Based on the screen state or execution result obtained from the external device, subordinate functions such as "brightness" and "font size" may be re-acquired and arranged around the current position.

[0240] Thus, instead of directly replicating the entire complex visual user interface into the haptic space, it may be possible to sequentially reconstruct the functions necessary for the current operation context into the local haptic space.

[0241] Furthermore, if a third party or AI instructs a user on how to operate the system, the context at that time, the functional meaning 501 that was instructed, the operations actually performed by the user, and the results may be saved as history information 154.

[0242] If a similar context arises later, the control unit 141 may compare the screen or function state, previous operation, and purpose recorded in the history information 154 with the current context and rank functions selected under the same or similar conditions as candidates. For example, based on the number of times it has been selected from the same higher-level functions and its recency, "Screen Settings" may be ranked as a higher candidate, and after confirming that it is available in the current external UI, it may be pre-positioned in an easily accessible haptic location. Predictive presentation is separate from execution instructions, and instead of directly manipulating the saved old screen coordinates, it utilizes the correspondence to the current operation target.

[0243] The third-party terminal 190 may simultaneously display the original visual user interface 500 and the haptic user interface currently presented to the user. When displaying the correspondence between the two, the same operation target is indicated by lines, numbers, colors, or highlights, using references to the original screen elements retained along with the functional meaning 501 and the registration information on the haptic side. Even if there are multiple functions with the same name, it is not necessary to display different screen elements as a single correspondence based solely on string matching.

[0244] This allows a third party to provide support while confirming which manipulated object on the visual screen corresponds to which position or physical piece is presented to the haptic user. <Embodiment 14: Meaning Form and Minimal Disclosure of Personal Information>

[0245] Figure 22 shows the three stages: "External Form" on the left, "Semantic Correspondence and Minimal Disclosure" in the center, and "Haptic Presentation and Provision Confirmation" on the right. This is an example in which meaning is obtained from input field 401 of the external form 400, the control unit 141 presents only the necessary items to the haptic side, and provides them only after an explicit confirmation operation.

[0246] The form 400 shown on the left side of Figure 22 may include input fields 401 such as "Name," "Address," and "Telephone Number." The same applies to application forms, delivery forms, administrative forms, reservation forms, and other forms.

[0247] As shown in the "Semantic Correspondence and Minimal Disclosure" section in the center of Figure 22, the control unit 141 or the external information processing device 170 acquires the semantic 402 of each input item 401, rather than its absolute position on the screen. For example, it reads the item's identification information, label, input type, whether it is required or optional, input conditions, and the form to which it belongs, which can be obtained from the form, and associates it with the semantic 402 based on a pre-registered semantic correspondence table or user confirmation.

[0248] As shown in "Haptic Presentation / Provision Confirmation" on the right side of Figure 22, haptic input targets corresponding to each meaning 402 may be presented as physical pieces 130 on the portable template member 120 or as operating positions on the contact detection surface 110. Figure 22 shows physical pieces 130 corresponding to "Address," "Name," and "Telephone Number" as examples. The information to be provided may be provided externally in accordance with the condition "Transmit only after explicit confirmation operation" shown at the bottom of the same figure.

[0249] When a user selects an item such as "address," the control unit 141 obtains a reference to the form item corresponding to the selection and its meaning 402, and compares it with the personal information items stored in advance. For example, if both a home address and a work address are candidates, one is determined based on its purpose or the user's selection. If the value is not stored, the user is prompted to enter it, and the system guesses a candidate without confirming it as the user's information. The selected value may be stored as working information before transmission, with the correspondence to the item being retained.

[0250] Personal information may be retained by dividing it into item units or semantic units, including name, address, telephone number, email address, shipping address, payment-related information, and other information.

[0251] When multiple information items are requested from an external service, the control unit 141 compares the requested meaning 402 and specific input destination with the meaning of the personal information held by the user. Then, it selects candidate items and values ​​to provide in response to the request, and may also be implemented in a way that avoids providing all personal information at once, including items that have not been selected. It distinguishes between what is requested and what the user has approved to provide, notifies the user if essential items are missing, and allows the user to choose not to provide optional items. If format conversion is necessary, it maintains the correspondence between the converted value and the original value, and does not confirm changes that alter the meaning or scope of provision based solely on inference.

[0252] Before provision, the recipient, items to be provided, purpose of provision, or scope of provision may be confirmed to the user by voice, physical piece, or tactile placement on a portable template member 120. In one example, the control unit 141 maintains the service and form to be provided, the specific items and values ​​selected, and the version of the confirmation content as a single provision plan, and accepts an explicit confirmation operation for its contents. If the purpose of provision, etc., cannot be obtained from an external source, it is treated as known and not explained, and is shown as unconfirmed. In embodiments where values ​​are not sent externally for candidate selection, the candidates are held on the user's device until confirmation, and unapproved values ​​are not written to forms, etc., that send content during input.

[0253] The control unit 141 may send the items and values ​​to be provided corresponding to the operation to the external service only when the user performs an explicit confirmation operation. In this example, if the recipient, selected items, or values ​​are changed after approval, the old approval is not reused for the changed content, and confirmation of the changed provision is requested. The information to be sent consists of the set held as the subject of approval, and does not include the entire personal information stored in advance or any additional items. The system distinguishes between acceptance of the transmission request, confirmation of the transmission result, and unknown result, and if no response is received, it does not determine whether it has been received or not. <Embodiment 15: Tactile purchasing and payment>

[0254] Figure 23 shows the three stages: "Product Selection / Cart Management" on the left, "Synchronization / Transaction Context" in the center, and "Purchase Confirmation / Payment Approval" on the right. This is an example where the states of products A, B, and C on the haptic cart 420 and physical pieces 130 on the left are "synchronized" to the control unit 141, and the process proceeds to the purchase process on the right based on that transaction context.

[0255] As shown in "Product Selection / Cart Management" on the left side of Figure 23, a single portable template component 120 may be used as a tactile cart 420 corresponding to a shopping cart of an external service, and product details such as product A, product B, product C, etc. may be associated with the physical pieces 130 on it.

[0256] Each physical piece 130 on the haptic cart 420 can be associated with information that identifies the product details of an external service and the cart, as well as information such as the product name, options, quantity, or price. Details of the same product or name but with different options, etc., are stored as separate items as needed. Users can confirm the product name, price, quantity, description, and other information of the product details identified by touching the physical piece 130 via voice or other means. The corresponding details and physical placement are stored separately, and it is not necessary to limit the quantity represented by one piece to 1, or to determine the purchase quantity solely from the number of pieces detected.

[0257] In one example of removing a product item from the tactile cart 420 to an area outside the template for detailed inspection, the control unit 141 identifies the product details based on the selection made at the start of the movement and holds those details as the item to be moved. The contact position at the destination is acquired using a predetermined transfer procedure and associated with the work state for detailed inspection. In this example, temporary removal for display or operation is distinguished from removal from the purchase target of the external cart, and detailed information, options, quantity, or comparison information can be expanded outside the template while the product details remain in the cart. By maintaining a reference to the original tactile cart 420, the target to be returned can be identified even if the original template moves during detailed inspection.

[0258] When a user returns an item to the tactile cart 420, the control unit 141 associates the held item details with the contact position based on the current cart arrangement and can terminate the temporary retrieval state. Items held in the external cart during temporary retrieval are not duplicated by this return. In the case of deleting an item, a predetermined exclusion area or deletion instruction is provided separately, and when the movement of the target item to that area is accepted as a specified movement, a request is made to the external system to delete or change the quantity of the corresponding item. Deletion is not considered merely the loss of contact. If no response is received from the external system, the system may distinguish between deletion request in progress and deletion completion and maintain these states.

[0259] As shown in the "Synchronization / Transaction Context" in the center of Figure 23, the control unit 141 may use information identifying the product details to compare the registration information of the haptic cart 420 with the actual cart status obtained from an external service and synchronize the status. If the quantity, options, price, or inventory are changed externally, the control unit 141 updates the information of the relevant details and notifies the user, and if the original position can be preserved, it changes the information corresponding to that position.

[0260] A transaction context may be established by selecting physical or logical objects representing a business partner or service, and objects representing the user's personal information or authentication information, specifying their respective roles. The control unit 141 maintains associations between references to business partners and shopping carts and references to the accounts or information sets used. Contacting these objects itself is not treated as successful authentication or payment authorization for external services; in configurations requiring authentication on an external terminal, the results are obtained separately.

[0261] When a user proceeds to the purchase process in response to "Proceed to Purchase" shown from the center to the right in Figure 23, the control unit 141 acquires the product details, quantity, total amount, shipping address, personal information to be provided, and payment method from the target cart status and stores them as the transaction details to be confirmed in "Purchase Confirmation / Payment Approval" on the right side of Figure 23. Transaction conditions such as available shipping costs, currency, and options may be presented sequentially or all at once via tactile or audio.

[0262] In "Purchase Confirmation and Payment Approval" on the right side of Figure 23, payment or purchase may be executed on the condition of an explicit final approval operation that is distinguished from the operation of touching a product, the operation of confirming personal information, or the search operation. The control unit 141, in the operation state of accepting final approval, records the approval for the transaction content held as the confirmation target, and outputs a purchase request corresponding to the content to an external service.

[0263] The final approval operation may be contact at a specific position, simultaneous contact at a plurality of positions, voice confirmation, contact for a predetermined time, authentication at an external terminal, or other methods.

[0264] In the example of outputting a purchase request to the outside, the control unit 141 may hold the confirmed transaction content, approval, request identification information, and execution result in association with each other. When the external service accepts identification information that prevents duplicate execution of the same request, that information can be used for re-inquiry and the like. If the response is lost, the transaction result is held as unconfirmed, and collation with confirmed order information or result inquiry is performed. If duplicate prevention or result inquiry cannot be used, the new purchase request is not unconditionally repeated, and confirmation by the user or the external service is requested. It is not necessary to notify that the purchase is completed only by sending the request, or to treat that the external order is also canceled only by canceling the local state.

[0265] Thereby, product search, product confirmation, personal information confirmation, and payment approval are managed as different states, and it is possible to prevent a transaction from being confirmed only by contact during tactile search. <Embodiment 16: Body Input, Acoustic Multiplexing and Personal Adaptation>

[0266] Embodiments of acoustic attribute assignment and two-hand cooperative input will be described using Figure 24 and Figure 25. Figure 24 shows "Tactile Position and Information Target" on the left, "Acoustic Attribute Assignment" in the center, and "Acoustic Presentation" on the right. Figure 25 shows "Left Hand: Select Operation Target or Adjustment Mode" on the left, the control unit 141 in the center, and "Right Hand: Operate Continuous Parameters" on the right.

[0267] As shown in "Assignment of Acoustic Attributes" in the center of Figure 24, the control unit 141 may assign different acoustic attributes to multiple contact locations, physical pieces 130, or information targets. In one example, it maintains a correspondence between the registered information or information elements of the target, data representing its location, hierarchy, category, or role, and the acoustic attributes to be used. The target information is read out in response to contact or selection and presented as "Acoustic Presentation" on the right side of Figure 24 from the audio input / output unit 143, etc.

[0268] Acoustic attributes may include left / right localization, virtual sound source direction, pitch, timbre, speaker, playback speed, rhythm, tone length, volume, reverberation, or other auditory characteristics.

[0269] As shown in "Tactile Position and Information Target" on the left side of Figure 24, spatial relationships can be audibly represented by presenting information corresponding to the physical piece 130A on the left, positioned to the left, and information corresponding to the physical piece 130B on the right, positioned to the right. The control unit 141 may assign acoustic attributes such as left / right localization according to the position of the object and reflect them in the acoustic presentation on the right side of Figure 24.

[0270] When presenting multiple pieces of information simultaneously or in close proximity, the control unit 141 may assign the hierarchy, importance, category, or role of the information to the tone or speaker, etc., using a registered correspondence table, etc. If the same combination of acoustic attributes overlaps and is difficult to distinguish, a different combination is selected, the number of presentations is reduced, or the presentation is switched to sequential. Even after the object is moved, the attributes corresponding to the position and the attributes corresponding to the category, etc., can be updated separately, and it is not necessary to change all attributes at once.

[0271] The control unit 141 may store the user's unique auditory discrimination characteristics and prioritize assigning combinations of acoustic attributes that are easy to distinguish. For example, it may obtain the user's identification results or preferred specifications for pre-prepared sets of different presentations such as left / right localization, timbre, or pitch, and associate the conditions with the results. Candidates can be ranked using the identification rate, number of re-presentations, or response time. If there is insufficient data, the standard settings or settings selected by the user will be used, and if the user reports difficulty in distinguishing, the system will switch to a different attribute or sequential presentation. This information is for operational support under the given usage conditions and does not require a diagnosis of auditory function.

[0272] For example, for users who can easily distinguish between left and right localization but have difficulty distinguishing between pitch differences, left and right localization may be used as the primary distinguishing attribute, and pitch differences as a supplementary one.

[0273] As shown on the left side of Figure 25, the user may select an operation target or adjustment mode such as "Selection 1," "Selection 2," or "Selection 3" using the index, middle, and ring fingers of the left hand, or a combination thereof, and input that selection to the control unit 141. On the other hand, on the right side of Figure 25, the continuous parameters are manipulated by circling, distance changes, or direction changes with the right hand.

[0274] The "circling operation" or "distance change or direction change" shown on the right side of Figure 25 may be used as a continuous quantity for the selected target. For example, the orientation of the right hand's contact position relative to the registered operation center is determined, and for a series of contacts that can be continuously acquired, the orientation difference between adjacent acquisition points is accumulated along the circling direction. Continuous values ​​such as audio playback speed are changed according to the accumulated amount from the initial value and limited to the usable range.

[0275] Finger type identification may be performed not only based on the absolute position on the contact surface, but also on the relative relationship with the palm, thumb side, base of the hand, and other body reference positions. In one example, the user is asked to indicate the position of the base of the hand and the thumb side in the order indicated, and after specifying which hand it is, the position of each finger is registered sequentially. This stores the correspondence between the relative position or arrangement from the body reference and the operation performed by the finger type and finger combination. During use, contact candidates are compared with the registered relationships based on the body reference positions that can be acquired, and the corresponding selection target or mode is determined. In configurations where the palm, etc., is excluded or cannot be acquired by the contact detection device 111, a reference position etc. indicated separately by the user is used, and body parts that are not detected are naturally not used as input. If there is only one reference and the orientation is not determined, if fingers cross and candidates overlap, or if the hand is repositioned and the correspondence becomes unclear, re-registration or explicit selection may be switched.

[0276] This allows for the creation of a two-handed cooperative interface where one hand selects "what to adjust" and the other hand continuously controls "how much to adjust." <Embodiment 17: Modification, History Utilization, and Contextual Adaptation>

[0277] In the embodiments described above, examples in which a capacitive touch sensor is used as the contact detection surface 110 have been mainly explained, but the present invention is not limited thereto. Resistive, optical, pressure, magnetic, or other methods may be used as long as they can detect contact or proximity to the user or physical piece 130.

[0278] The portable template member 120 is not limited to paper, but may be made of resin, cloth, fiber, rubber, thin film, thin plate, composite material, etc. Furthermore, the entire member does not need to have a uniform thickness; the thickness or material may be varied locally depending on factors such as contact detection, rigidity, gripping ability, or transportability.

[0279] The means for holding the physical piece 130 is not limited to magnetic retention. Friction, adhesion, walls, grooves, recesses, mechanical engagement, electrostatic force, and other retention methods may be used.

[0280] To acquire information about the portable template member 120, in addition to contact with the reference unit 122, an identification medium 128 such as RFID or NFC, an external camera, a magnetic position sensor, an optical marker, wireless positioning, or a combination thereof may be used. Distinguish whether the information obtained by each means is identification information, proximity or presence information, coordinates, or orientation, and in processes where the current position and orientation are required, combine the information that can acquire them. A method that simply reads the identification information from an identification medium is not treated as a method that continuously measures the coordinates or orientation of an arbitrary position.

[0281] For example, the type or logical identification information of the portable template member 120 may be acquired by the identification medium 128, and the current position on the contact detection surface 110 may be associated with the identification information based on the initial contact position with the reference unit 122 thereafter.

[0282] The movement of the portable template member 120 is not limited to translation, but may also involve rotation. Template placement information 151, including the rotation angle or orientation, may be acquired by using multiple reference units or attitude detection means.

[0283] Physical piece 130 may or may not have an individual unique ID. Even if it does not have an individual unique ID, correspondence information may be identified or reassigned based on user declaration, tactile attributes, location, order, history, or operation context, etc.

[0284] The physical piece 130 does not need to be tracked continuously when not in contact, but in another embodiment, it may be continuously tracked using a camera, RFID, magnetic sensor, or other means. For example, the correspondence between information that identifies an individual and location information from a means that can acquire its location is registered, and the coordinate systems and acquisition times between detection means are matched as needed. For means that only perform identification, it is not assumed that they can acquire a continuous position on their own. Even when performing continuous tracking, the distinction between parent movement and child relative movement can be used in combination, and the system may be configured to reconfirm objects for which the acquired information is inconsistent or for which tracking has been lost.

[0285] The control unit 141 may estimate information that is highly likely to be used next, using operation history, AI processing history, information selected by the user in the past, statistical information of other users, usage time, location, application, or the current work context, and the like. In one example, high-level items, immediately previously selected information, operation types, and the like are recorded as a context, and candidates are ranked based on the frequency or recency of information selected next in the same or similar context. For time, location, or information of other users, available items are used, and unacquired conditions are not supplemented as if they have been measured. Even when AI is used, candidates corresponding to the input context may be acquired, and after confirming the correspondence to targets present in the current information, the candidates may be used for presentation.

[0286] The estimated information may be preferentially assigned to placement candidates that can be reached with a small movement from the contact position indicated by the user or the registered work range. In an example where only the next required physical piece 130 or information is expanded to the area outside the template while maintaining the existing arrangement on the portable template member 120, existing registration targets and candidates are distinguished, information is assigned to available positions, and if necessary, the user is guided to perform piece placement and contact registration. Existing targets will not be replaced without permission or external operations will not be executed solely on the grounds of pre-placement of candidates.

[0287] The tactile attribute, acoustic attribute, placement position, area within the template, or guidance method may be changed according to the importance level, hierarchy, centrality, urgency, or usage frequency of information, and the like. For example, the acoustic attribute or guidance order is changed according to a correspondence table between importance or usage frequency and presentation methods. When changing the tactile attribute of the physical piece 130 with a fixed shape, the user may be guided to replace it with another piece having required notches or outer shapes, and a method may be adopted in which a reference to the information held as the replacement target is associated with the contact position or declared attribute acquired after replacement. A change in registered attribute or meaning is distinguished from a change in the shape of the actual object, and this does not mean that the resin shape of a piece without a deformation mechanism can be deformed only by information processing.

[0288] The user's actions and their results or evaluations are retained as a history and may be reused later in similar contexts. For example, the type of guidance, applied presentation settings, and whether or not retries were performed are recorded, and the ranking of candidates or presentation settings are updated according to the user's preferences or certain evaluations. The system may also be configured to retain the original settings and allow for the cancellation of changes, separating candidate generation using history from confirmation of user input, editing, or external execution. This allows for adaptation based on verifiable input and processing results, according to the user's operation method, preferences, tendency to misidentify, or learning history.

[0289] Some or all of the above processes may be executed locally on the control device 140, or on an external information processing device 170, a user terminal 180, a server, or a combination thereof. In the case of requesting processing from an external source, the system maintains the request identification information, a reference to the target of the operation, and the version of the target's content or arrangement, and associates the response with the request and the target. If the target has changed after the request, the response is not unconditionally applied to the current different target, but the target or scope of application is checked. In the event of a communication interruption, processes for which the necessary input or results cannot be obtained may be kept as incomplete and distinguished from operations that can be continued with locally valid information. For processes that use location information and processes that use text content, the version or scope of validity to be checked can be separated.

[0290] Furthermore, users are not limited to those with visual impairments. It can be applied to situations where tactile information placement or manipulation is useful, such as in dark places, high-load work environments, situations where both eyes are used for other tasks, education, training, collaborative editing, accessibility support, and other applications.

[0291] In each of the embodiments described above, the terms "parent" and "child" are relative, and one portable template member 120 may be a child with respect to a higher workspace, but a parent with respect to a physical piece 130 or another portable template member placed on top of it.

[0292] Therefore, parent-child relationships can be recursively established, and the information under the child workspace can be moved, pieced, redeployed, or re-parented together.

[0293] Furthermore, the number of physical layers that a user can simultaneously perceive through direct touch may be limited. For example, a user may directly manipulate two layers, a parent template and a child template, and handle deeper information layers by repeatedly pieceifying and redeploying them to other templates.

[0294] The configurations described in each of the above embodiments may be implemented individually or combined as arbitrarily as long as they do not contradict each other technically. Furthermore, a configuration described in one embodiment may be applied to another embodiment. When changing the detection method, holding method, or presentation method, etc., it should be confirmed that the necessary contact position, correspondence with the target, or presentation information can be acquired or output with the configuration, and for information that cannot be acquired, a different input or re-specification should be used. It is not assumed that attributes, posture, or authentication results that can be acquired in one embodiment can naturally be acquired in a different configuration.

[0295] The present invention is not limited to the uses or structures shown in each of the embodiments described above, but can be broadly applied to information processing systems that haptatively handle the arrangement, belonging, order, hierarchy, editing state, or operation context of information using multiple physical objects and a contact detection surface. [Explanation of symbols]

[0296] 100 Information Processing Systems 110 Contact detection surface 111 Contact detection device 120 Portable template components 120A First Portable Template Member 120B Second Portable Template Component 120C Another portable template component 122 Reference section 123 2nd standard section 124 Deviation constraint and tactile boundary 125 Entrance / Exit 126 Thin-walled transfer section 127 Sequence Guide 128 Identification medium 130 Physical pieces 130A~130E Physical Pieces 130X Physical pieces corresponding to the pieced workspace state 131 Magnets 132 Magnetic material 133 Non-magnetic outer periphery 134. Tactile Knowledge Section 135 Input contact area 140 Control device 141 Control Unit 142 Storage section 143 Audio Input / Output Section 144 Communications Department 150 Workspace Status Information 151 Template Placement Information 152 Piece Correspondence Information 154 History Information 170 External Information Processing Device 180 User terminals 190 Third-party terminal 210 Tactile Teaching Material Sheets 211 Identification medium 301 Visual Attributes 302 Tactile Attributes 303 Boolean Attributes 400 forms 401 Input fields 402 Meaning 420 Tactile Cart 500 Visual User Interfaces 501 Functional meaning

Claims

1. A contact detection surface capable of detecting the contact position, A portable template member that is placed on the contact detection surface and is movable along the contact detection surface, A plurality of physical pieces, each capable of being positioned within a two-dimensional region on the portable template member, movable in the in-plane direction relative to the portable template member, and capable of detaching from the portable template member and being transferred onto the contact detection surface, Fixed to the aforementioned portable template member, and comprising at least one reference part that can be identified by touch by the user, A storage means that registers at least a portion of the positions within the portable template member on which each of the plurality of physical pieces is arranged as a registration position, and stores information associated with each of the registration positions, Control means and Equipped with, The position on the contact detection surface corresponding to the reference part that the user has contacted among the at least one of the reference parts can be acquired in connection with the contact. With one of the plurality of physical pieces positioned on the portable template member, the position on the contact detection surface can be acquired upon contact by the user with the physical piece. The control means is Based on the change in position on the contact detection surface corresponding to at least one of the reference parts, the movement of the portable template member is recognized. Even if, during the aforementioned movement, the position on the contact detection surface resulting from the user's contact with one or more of the registered positions is not continuously acquired, the association of information to each of the one or more registered positions is maintained. After the aforementioned movement, the arrangement state of the portable template member is determined based on the position on the contact detection surface corresponding to the at least one reference portion, in order to convert the position on the contact detection surface to a position within the portable template member. If, after the aforementioned movement, a new position on the contact detection surface is acquired due to the user's contact with one of the multiple physical pieces placed on the portable template member, Based on the arrangement state, the newly acquired position on the contact detection surface is converted to a position within the portable template member to determine the calculated position. Based on the positional relationship between the calculation position and the one or more registration positions, when one of the one or more registration positions is identified, The registered position stored in the storage means is updated based on the calculated position. Information processing system.

2. The control means recognizes the movement of the portable template member based on the coordinated movement of at least two positions on the contact detection surface, including a position on the contact detection surface corresponding to one of the at least one reference portion. The information processing system according to claim 1.

3. The control means determines that the movement of the portable template member maintains its orientation, based on the fact that the distance between two of the at least two positions fixed relative to the portable template member and the change in the direction connecting the two positions are within a predetermined range. The information processing system according to claim 2.

4. The control means does not determine the change in position on the contact detection surface acquired in conjunction with the movement of the portable template member as an individual movement of the information associated with the registered position while the movement of the portable template member continues. The information processing system according to claim 1.

5. The arrangement state of the portable template member includes its position on the contact detection surface and its in-plane rotation angle. The control means determines the calculated position by converting the newly acquired position on the contact detection surface into coordinates within the portable template member based on the arrangement state. The information processing system according to claim 1.

6. The aforementioned physical piece has a magnet, A magnetic material is placed on the back side of the contact detection surface. The physical piece is held toward the contact detection surface by the magnetic force between the magnet and the magnetic material. The information processing system according to claim 1.

7. The aforementioned physical piece has a non-magnetic outer periphery around the magnet that can come into contact with other physical pieces. The non-magnetic outer periphery ensures a predetermined minimum separation distance between the magnets when in contact with other physical pieces, thereby suppressing at least one of tipping over and inversion caused by the magnetic action of adjacent physical pieces. The information processing system according to claim 6.

8. The non-magnetic outer periphery has a tactile identification structure that allows the user to identify the type, value, or information represented by the physical piece through touch. The information processing system according to claim 7.

9. The aforementioned physical piece has a magnetic material, A magnet or magnetic layer is arranged on the back side of the contact detection surface. The physical piece is held toward the contact detection surface by the magnetic force between the magnetic material and the magnet or magnetic layer. The information processing system according to claim 1.

10. The aforementioned physical piece has a tactile component that allows the user to identify the type, value, or information represented by the physical piece through touch. The control means does not determine the input of information corresponding to the physical piece or the target of the operation based solely on the operation of touching the tactile information separation part. The information processing system according to claim 1.

11. The control means is The system accepts declaration inputs from users that represent values, types, or information identified by the user based on the aforementioned tactile knowledge section. The system stores the contact position obtained in response to a predetermined confirmation operation in association with the value, type, or information represented by the declaration input. The information processing system according to claim 10.

12. The control means is After a period in which the position of the physical piece is not acquired by the contact detection surface, the new contact position resulting from contact with the physical piece and the declared input received from the user are used to reassociate the value, type, or information represented by the declared input with the new contact position. The information processing system according to claim 11.

13. At least a portion of the periphery of the aforementioned portable template member is This restricts the physical piece from unintentionally moving outside the area of ​​the portable template member, The region has a deviation constraint and tactile boundary section that allows the user to identify the boundary of the region by touch. The information processing system according to claim 1.

14. Of the periphery of the portable template member, in a portion different from the deviation constraint and tactile boundary portion, An opening is formed that allows the physical piece to be moved between an area outside the portable template member on the contact detection surface and the portable template member. The information processing system according to claim 13.

15. The aforementioned entrance / exit is, Having at least one of a thin-walled portion, an inclined portion, a curved portion, or an open end, which allows the physical piece to be transported without lifting it or without substantially releasing the holding force on the physical piece. The information processing system according to claim 14.

16. The control means is Based on the relationship between the representative position obtained by the contact detection surface of the physical piece, the contact surface shape or outer shape of the physical piece, and the boundary of the portable template member, Even if the aforementioned representative position is within the area of ​​the portable template member, it is determined whether or not it belongs to a dangerous area where the physical piece may not stably follow the portable template member when the portable template member is moved. If it is determined that the user is in the aforementioned danger zone, non-visual feedback will be presented to the user. The information processing system according to claim 1.

17. A second portable template member can be placed on the first template member, which is the aforementioned portable template member. The control means associates the second template member as a subgroup of the information group corresponding to the first template member. The information processing system according to claim 1.

18. The control means is The movement of the first template member, The relative movement of the second template member with respect to the first template member, The relative movement of the physical piece on the second template member with respect to the second template member, Distinguish and process them. The information processing system according to claim 17.

19. The control means is Information corresponding to multiple physical pieces on the portable template member, and workspace state information including their arrangement relationships, It stores information corresponding to a single physical piece specified by the user. The information processing system according to claim 1.

20. The control means is When the aforementioned workspace state information is associated with one physical piece and another portable template member, Based on the workspace state information, the information contained in the workspace state information is re-deployed onto the other portable template member, associating it with multiple physical pieces. The information processing system according to claim 19.

21. The control means is Based on the semantic structure of the information included in the workspace state information, the usable area of ​​the other portable template member, or at least one of the number of available physical pieces, The number of physical pieces or information division units to be used for the redeployment is determined. The information processing system according to claim 20.

22. The contact detection surface or the portable template member has a reference line, periphery, or arrangement guide that can be identified by touch by the user. The control means is Based on the physical arrangement order of the multiple physical pieces arranged along the aforementioned reference line, periphery, or arrangement guide, To set the presentation order, logical order, or editing order of multiple pieces of information corresponding to the multiple physical pieces. The information processing system according to claim 1.

23. A contact detection surface capable of detecting the contact position, A portable template member that is placed on the contact detection surface and is movable along the contact detection surface, A plurality of physical pieces, each capable of being positioned within a two-dimensional region on the portable template member, movable in the in-plane direction relative to the portable template member, and capable of detaching from the portable template member and being transferred onto the contact detection surface, Fixed to the aforementioned portable template member, and comprising at least one reference part that can be identified by touch by the user, A control method for an information processing system using the following: The process involves designating at least a portion of the positions within the portable template member on which each of the plurality of physical pieces is arranged as a registration position, and associating and storing information with each of the registration positions. A step of acquiring the position on the contact detection surface corresponding to the reference part that the user has contacted among the at least one of the reference parts, With one of the plurality of physical pieces positioned on the portable template member, the process involves acquiring the position on the contact detection surface in response to the user's contact with the physical piece. A step of recognizing the movement of the portable template member based on a change in position on the contact detection surface corresponding to at least one of the reference parts, Even if, during the aforementioned movement, the position on the contact detection surface resulting from the user's contact with one or more of the registered positions is not continuously acquired, the process of maintaining the association of information to each of the one or more registered positions, After the aforementioned movement, the process involves determining the arrangement state of the portable template member for converting the position on the contact detection surface to a position within the portable template member, based on the position on the contact detection surface corresponding to the at least one reference portion. After the aforementioned movement, if a new position on the contact detection surface is acquired due to the user's contact with one of the plurality of physical pieces placed on the portable template member, the process of calculating the position is performed by converting the newly acquired position on the contact detection surface to a position within the portable template member based on the arrangement state. A step of updating the stored registered position based on the calculation position when one of the registered positions is identified based on the positional relationship between the calculation position and the one or more registered positions, including, A method for controlling an information processing system.

24. A contact detection surface capable of detecting the contact position, A portable template member that is placed on the contact detection surface and is movable along the contact detection surface, A plurality of physical pieces, each capable of being positioned within a two-dimensional region on the portable template member, movable in the in-plane direction relative to the portable template member, and capable of detaching from the portable template member and being transferred onto the contact detection surface, Fixed to the aforementioned portable template member, and comprising at least one reference part that can be identified by touch by the user, A computer used in an information processing system equipped with the following features: The process involves designating at least a portion of the positions within the portable template member on which each of the plurality of physical pieces is placed as a registration position, and associating and storing information with each of the registration positions. A process to acquire the position on the contact detection surface corresponding to the reference part that the user has touched among the at least one of the reference parts, in connection with the contact, With one of the plurality of physical pieces placed on the portable template member, the process of acquiring the position on the contact detection surface in response to the user's contact with the physical piece, A process for recognizing the movement of the portable template member based on a change in position on the contact detection surface corresponding to at least one of the reference parts, Even if, during the aforementioned movement, the position on the contact detection surface resulting from the user's contact with one or more of the registered positions is not continuously acquired, the process maintains the association of information with each of the one or more registered positions. After the aforementioned movement, a process is performed to determine the arrangement state of the portable template member for converting the position on the contact detection surface to a position within the portable template member, based on the position on the contact detection surface corresponding to the at least one reference portion. After the aforementioned movement, if a new position on the contact detection surface is acquired due to the user's contact with one of the multiple physical pieces placed on the portable template member, the process of converting the newly acquired position on the contact detection surface to a position within the portable template member based on the arrangement state and calculating the position, Based on the positional relationship between the calculation position and the one or more registration positions, when one of the one or more registration positions is identified, the stored registration position is updated based on the calculation position. A program to execute.

Images