Input method, device and equipment for obtaining tactile feedback in expanded reality space

Abstract

The invention relates to an input method, device and equipment for obtaining tactile feedback in an expanded reality space. Comprising the following steps: taking one or more surfaces of a user body or one or more surfaces of an object as a preset input surface, and taking a body part of the user as an input source; tracking the input source and a preset input surface, and when the relationship between the input source and the preset input surface meets the requirement, taking the preset input surface as a target input surface and generating a virtual keyboard on the target input surface; and the user completes the input of the corresponding virtual keys on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or the object surface corresponding to the target input surface. According to the method, the virtual keyboard is not limited at a specified position in an expanded reality space any more, the size of the virtual keyboard and the number of body parts serving as input sources are not limited any more, so that ten-finger interaction is supported, interaction is carried out through the body parts and the body surface or the object surface, a user can obtain real physical tactile feedback during interaction, and the user experience is improved. And the input is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to an input method, device and equipment for obtaining tactile feedback in an expanded real space. Background Art

[0002] In the extended reality space, input is a major demand, but existing input technologies hinder the advancement of XR devices into productivity tools. Currently, commonly used input methods include input based on an external keyboard, input based on a handle, input based on a touch screen or a touchpad, and input based on hand motion tracking or eye tracking. Among them, the input method based on an external keyboard has the disadvantage of poor portability and requires a stable and flat surface. Based on traditional 3Dof or 6Dof handles (direct interaction or ray), the input speed is slow and easy to cause fatigue. Input based on a touch screen or a touchpad needs to rely on the device, and the accuracy is limited by the touchpad. The input method based on hand motion tracking will be affected by large tracking errors, and long-term suspended operations will easily cause user fatigue. In addition to being limited by the accuracy of tracking recognition, the input method based on eye tracking is more expensive and has low input efficiency. Summary of the invention

[0003] To solve the problems existing in the prior art, the embodiments of this specification provide an input method, device and equipment for obtaining tactile feedback in an augmented reality space, wherein one or more surfaces of a user's body or one or more surfaces of an object in the augmented reality space are used as preset input surfaces, and any part of the user's body is used as an input source. The relationship between the input source and each preset input surface is tracked, and it is determined whether to enter an input mode and determine a target input surface. When the input mode is entered, a virtual keyboard is rendered on the target input surface, and the user completes the input of the corresponding virtual keys on the virtual keyboard through the interaction of the body part corresponding to the input source on the target input surface.

[0004] In order to solve any of the above technical problems, the specific technical solutions of this specification are as follows:

[0005] On the one hand, an embodiment of the present specification provides an input method for obtaining tactile feedback in an augmented reality space, the method comprising:

[0006] One or more surfaces of the user's body or one or more surfaces of an object are used as preset input surfaces, and at least one body part of the user is used as an input source;

[0007] Tracking the input source and each of the preset input surfaces, and when the relationship between the input source and the preset input surface meets a predetermined requirement, taking the preset input surface as a target input surface and generating a virtual keyboard on the target input surface;

[0008] The user completes input of corresponding virtual keys on the virtual keyboard through interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface.

[0009] Furthermore, the process of the user completing the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface includes:

[0010] When the user moves the body part corresponding to the input source close to the target input surface, a virtual key corresponding to the position of the input source is pre-selected. Then, the user moves the body part corresponding to the input source away from the target input surface, and the pre-selected virtual key is input.

[0011] Furthermore, when the user brings the body part corresponding to the input source close to the target input surface, the preselected virtual key corresponding to the position of the input source further includes:

[0012] Determining whether the distance between the input source and the target input surface is less than a preselected distance threshold;

[0013] If so, a virtual key corresponding to the position of the input source is pre-selected.

[0014] Furthermore, when the user brings the body part corresponding to the input source close to the target input surface, the preselected virtual key corresponding to the position of the input source further includes:

[0015] When the user contacts the target input surface with the body part corresponding to the input source, the virtual key corresponding to the position where the input source contacts the target input surface is pre-selected.

[0016] Furthermore, after the user preselects a virtual key corresponding to the position of the input source, the method further includes:

[0017] When the body part corresponding to the input source is kept close to the target input surface, the user moves the body part corresponding to the input source to switch the pre-selected virtual button.

[0018] Furthermore, when the relationship between the input source and the preset input surface meets a predetermined requirement, taking the preset input surface as a target input surface and generating a virtual keyboard on the target input surface further comprises:

[0019] When the position of the input source falls into the input space of the preset input surface and the duration exceeds a preset time length, the preset input surface is used as the target input surface and a virtual keyboard is generated on the target input surface.

[0020] Furthermore, generating a virtual keyboard on the target input surface further comprises:

[0021] Determining attribute information of the target input surface when the relationship between the input source and the target input surface meets a predetermined requirement;

[0022] The virtual keyboard is generated according to the attribute information.

[0023] Furthermore, the attribute information includes one or more of the size, orientation, shape and position of the target input surface.

[0024] Furthermore, generating the virtual keyboard according to the attribute information further includes:

[0025] If the attribute information includes the size of the target input surface, determining the size of the virtual keyboard according to the size of the target input surface;

[0026] If the attribute information includes the orientation of the target input surface, determining the orientation of the virtual keyboard according to the orientation of the target input surface;

[0027] If the attribute information includes the shape of the target input surface, determining the distribution mode of the virtual keyboard according to the shape of the target input surface;

[0028] If the attribute information includes the position of the target input surface, determining the position of the virtual keyboard according to the position of the target input surface;

[0029] The virtual keyboard is generated according to one or more of the size, orientation, distribution mode and position of the virtual keyboard.

[0030] Furthermore, in the process of the user completing the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface, the method further includes:

[0031] Provide input feedback to the user according to the feedback mode corresponding to the target input surface.

[0032] Furthermore, the user completing the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface further includes:

[0033] The user interacts with the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface based on the input aiming mode corresponding to the target input surface, thereby completing the input of the corresponding virtual key on the virtual keyboard.

[0034] Furthermore, the input aiming mode is a line of sight intersection mode, an input source projection mode or an input source nearest interaction point mode.

[0035] Furthermore, after generating the virtual keyboard on the target input surface, the method further includes:

[0036] Determine whether the target input surface is out of the user's field of view, and if so, end the input of the virtual keyboard; or,

[0037] It is determined whether the distance between the target input surface and the user's viewpoint exceeds a preset distance threshold, and if so, the input of the virtual keyboard is terminated.

[0038] Furthermore, after generating the virtual keyboard on the target input surface, the method further includes:

[0039] In the case where the target input surface is the user's body surface, it is determined whether the body movement change of the user's body surface corresponding to the target input surface meets the preset input termination condition, and if so, the input of the virtual keyboard is terminated.

[0040] On the other hand, the embodiments of this specification also provide an input device for obtaining tactile feedback in an augmented reality space, including:

[0041] An input setting unit, used to use one or more surfaces of a user's body or one or more surfaces of an object as preset input surfaces and at least one body part of the user as an input source;

[0042] an input determination unit, configured to track the input source and each of the preset input surfaces, and when the relationship between the input source and the preset input surface meets a predetermined requirement, use the preset input surface as a target input surface and generate a virtual keyboard on the target input surface;

[0043] The input unit is used to process the user's interaction on the body surface or object surface corresponding to the target input surface through the body part corresponding to the input source, so as to complete the input of the corresponding virtual key on the virtual keyboard.

[0044] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0045] Finally, an embodiment of this specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0046] Through the method of the embodiment of this specification, the XR device scans the user's body and the objects in the real space where the user is located, and renders the user's body and the objects in the space where the user is located in the extended real space. In order to improve the convenience of user input and user experience, the embodiment of this specification uses one or more surfaces of the user's body or one or more surfaces of an object as preset input surfaces, and uses at least one body part of the user as an input source, so that when the user interacts in the extended real space, the virtual keyboard can be aroused for input anytime and anywhere, and the virtual keyboard is no longer limited to a specified position in the extended real space. There is no need for the user to perform additional operations to open the virtual keyboard, and the size of the virtual keyboard and the number of body parts as input sources are no longer limited, thereby supporting users to interact more conveniently and naturally in the extended real space, such as ten-finger interaction, etc., to improve the user's operational convenience. Then, according to the input source and each preset input surface, when the relationship between the input source and the preset input surface meets the predetermined requirements, the preset input surface is used as the target input surface, and a virtual keyboard is generated on the target input surface rendered in the extended real space. The user completes the input of the corresponding virtual keys on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface. Because the user interacts with the body surface or object surface through body parts, the user can get real physical tactile feedback during the interaction, making the input more accurate. The user's body surface or object surface with tactile feedback can also serve as support for the user during input, reducing fatigue. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flow chart of an input method for obtaining tactile feedback in an extended real space according to an embodiment of the present specification;

[0049] Figure 2 Shown is a schematic diagram of an example of a preset input surface of the index finger in an embodiment of this specification;

[0050] Figure 3 Shown is a schematic diagram of a preset input space in an embodiment of this specification;

[0051] Figure 4 The figure is a schematic diagram of the process of generating a virtual keyboard on the target input surface in an embodiment of the present specification;

[0052] Figure 5The figure is a schematic diagram of a process of generating the virtual keyboard according to the attribute information in an embodiment of the present specification;

[0053] Figure 6 Shown is an example schematic diagram of the display positions of two virtual keyboards in the embodiments of this specification;

[0054] Figure 7 It is a schematic diagram showing a virtual key being tapped / activated / inputted in an embodiment of this specification;

[0055] Figure 8 Shown is a schematic diagram of input aiming modes of three input modes in the embodiments of this specification;

[0056] Fig. 9 The figure shows an input example schematic diagram in the embodiment of this specification;

[0057] Fig.10 The figure shows a schematic diagram of the structure of an input device for obtaining tactile feedback in an extended real space in an embodiment of the present specification;

[0058] Fig.11 The figure is a schematic diagram of the structure of a computer device in an embodiment of the present specification.

[0059]

Description of reference numerals

[0060] 1001. Input setting unit;

[0061] 1002, input determination unit;

[0062] 1003, input unit;

[0063] 1102. Computer equipment;

[0064] 1104. Processing equipment;

[0065] 1106. Storage resources;

[0066] 1108. Driving mechanism;

[0067] 1110, input / output module;

[0068] 1112. Input device;

[0069] 1114. Output device;

[0070] 1116. Presentation equipment;

[0071] 1118. Graphical user interface;

[0072] 1120. Network interface;

[0073] 1122. Communication link;

[0074] 1124. Communication bus. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0076] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0077] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0078] In order to solve the problems existing in the prior art, an embodiment of the present specification provides an input method for obtaining tactile feedback in an extended real space, taking one or more surfaces of a user's body or one or more surfaces of an object in the extended real space as preset input surfaces, and taking any part of the user's body as an input source, tracking the relationship between the input source and each preset input surface, judging whether to enter an input mode and determining a target input surface, and when entering the input mode, rendering a virtual keyboard on the target input surface, and the user completes the input of the corresponding virtual keys on the virtual keyboard through the interaction of the body part corresponding to the input source on the target input surface. Figure 1The flowchart of an input method for obtaining tactile feedback in an extended reality space according to an embodiment of the present specification is shown. This figure describes the input process of obtaining tactile feedback in an extended reality space, but it may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the system or device product is executed in practice, it can be executed in the order or in parallel according to the method shown in the embodiment or the accompanying drawings. Specifically, Figure 1 As shown, the method may include:

[0079] Step 101: using one or more surfaces of a user's body or one or more surfaces of an object as preset input surfaces, and using at least one body part of the user as an input source;

[0080] Step 102: Track the input source and each of the preset input surfaces, and when the relationship between the input source and the preset input surface meets a predetermined requirement, use the preset input surface as a target input surface and generate a virtual keyboard on the target input surface;

[0081] Step 103: The user completes input of corresponding virtual keys on the virtual keyboard through interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface.

[0082] Through the method of the embodiment of this specification, the XR device scans the user's body and the objects in the space where the user is located, and renders the user's body and the objects in the space where the user is located in the extended reality space. In order to improve the convenience of user input and user experience, the embodiment of this specification uses one or more surfaces of the user's body or one or more surfaces of an object as preset input surfaces, and uses at least one body part of the user as an input source, so that when the user interacts in the extended reality space, the virtual keyboard can be aroused for input anytime and anywhere, and the virtual keyboard is no longer limited to a specified position in the extended reality space. There is no need for the user to perform additional operations to open the virtual keyboard, and the size of the virtual keyboard and the number of body parts as input sources are no longer limited, thereby supporting users to interact more conveniently and naturally in the extended reality space, such as ten-finger interaction, etc., to improve the user's operational convenience. Then, according to the input source and each preset input surface, when the relationship between the input source and the preset input surface meets the predetermined requirements, the preset input surface is used as the target input surface, and a virtual keyboard is generated on the target input surface rendered in the extended reality space. The user completes the input of the corresponding virtual keys on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface. Because the user interacts with the body surface or object surface through body parts, the user can get real physical tactile feedback during the interaction, making the input more accurate. The user's body surface or object surface with tactile feedback can also serve as support for the user during input, reducing fatigue.

[0083] In the embodiments of this specification, one or more surfaces are selected from the surface of the user's body rendered by the XR device and the surface of the object in the real space as the preset input surface. The preset input surface in the embodiments of this specification refers to a real surface on which the user can obtain continuous tactile feedback when completing the input. This embodiment of the specification does not limit which surface is selected. It can be a real plane in the physical space, such as the surface of a table, a wall, a book, etc., or it can be a part of the user's body, such as the curved surface on one side of the index finger, the curved surface on the upper side of the thigh, etc. Different XR devices can be equipped with different preset input surfaces according to their own capabilities (such as whether the XR device can detect the (flat / curved) surface in the user's real space in real time, whether it can detect the thigh surface, etc.). There can be multiple preset input surfaces at the same time, but at least one preset input surface is required. Because mainstream XR devices can recognize the surfaces of both hands, the embodiments of this specification preferably use the preset surfaces parallel to the fingertips of the left and right index fingers or a part of the radial surface of the index finger as the preset input surface (such as Figure 2 ), whether to configure more preset input surfaces is not limited in the embodiments of this specification.

[0084] In the embodiments of this specification, the preset input source may be a joint of the user's hand. In the embodiments of this specification, one body part of the user may be used as the input source, such as the tip of the index finger, or multiple body parts of the user may be used as the input source, such as the tips of ten fingers of both hands, so as to realize ten-finger interaction, that is, the user can input through the tips of ten fingers. The embodiments of this specification do not limit which part is selected as the input source.

[0085] The XR device can track the input source and each preset input surface, and render it in the extended reality space. When the relationship between the input source and one of the input surfaces meets the predetermined requirements, it enters "input mode", takes the preset input surface as the target input surface, and generates a virtual keyboard on the target input surface.

[0086] When it is recognized that the relative position of a certain input source P_input of the user's arms / hands and a certain preset input surface meets the preset rules, the "input mode" begins. The preset input surface can be the side of the index finger, the upper thigh (sitting on the sofa); it can also be a real plane that actually exists in the space, such as the top of the table. When the "input mode" is turned on, the virtual keyboard for input is displayed to the user, and the movement of the user's hands / fingers or the movement and operation of other interactive input sources (such as an external mouse, head control input, etc.) no longer interact with targets outside the virtual keyboard to ensure the stability and accuracy of the input interaction. The input source P_input of the user's arms / hands does not meet the preset rules with all the corresponding preset input surfaces. If it is far away from all preset input surfaces, it is determined that the "input mode" is over. Optionally, when the "input mode" ends, the virtual keyboard for input is hidden to release this interactive space.

[0087] In order to improve the convenience of entering the "input mode", according to an embodiment of the present specification, when the relationship between the input source and the preset input surface meets the predetermined requirements, taking the preset input surface as the target input surface and generating a virtual keyboard on the target input surface further includes:

[0088] When the position of the input source falls into the input space of the preset input surface and the duration exceeds a preset time length, the preset input surface is used as the target input surface and a virtual keyboard is generated on the target input surface.

[0089] In the embodiment of the present specification, the input space of the preset input surface is pre-set. When the relationship between any one (there can be multiple) input source P_input and at least one preset input surface Surface_i meets the following conditions, it is an "input mode", otherwise it is other interaction modes in the system (such as direct interaction mode, indirect interaction mode, etc.). The movement and operation of the interaction source in the "input mode" will not affect the interaction results of other modes. Because there can be multiple input sources P_input and preset input surfaces, and multiple input sources P_input can correspond to the same preset input surface. When determining the interaction mode, the embodiment of the present specification can monitor the information of each input source P_input to determine whether the conditions for entering the "input mode" are met. When at least one input source P_input meets the following two conditions at the same time, it is regarded as entering / switching to the "input mode".

[0090] Condition 1: If the plane set is represented by: P = {Surface_i | Surface_i is a plane}; the surface set is represented by: C = {Surface_i | Surface_i is a surface}. The input source is represented by p[i] (Vector3). The difference in the distance of the preset surface in the axial direction of the preset direction is represented by r. The preset input space is the space surrounded by two surfaces with a distance r from the preset input surface in the preset direction and the opposite direction, represented by R. The surface can be a plane or a surface, such as Figure 3 , taking the surface as an example. Any input source entering its corresponding preset input space is recorded as p[i] ∈ R.

[0091]

[0092] Condition 2: The duration for which k = true is maintained is greater than the preset time length T_mode.

[0093]

[0094] When entering the "input mode", a virtual keyboard is generated on the target input surface. Specifically, Figure 4 As shown, generating a virtual keyboard on the target input surface further includes:

[0095] Step 401: Determine attribute information of the target input surface when the relationship between the input source and the target input surface meets a predetermined requirement;

[0096] Step 402: Generate the virtual keyboard according to the attribute information.

[0097] In the embodiment of the present specification, when entering the "input mode", the attribute information of the target input surface of the first frame F0 of the "input mode" is recorded, and the attribute information includes one or more of the size, orientation, shape and position of the target input surface.

[0098] Furthermore, if Figure 5 As shown, generating the virtual keyboard according to the attribute information further includes:

[0099] Step 501: if the attribute information includes the size of the target input surface, determining the size of the virtual keyboard according to the size of the target input surface;

[0100] In this step, the corresponding relationship between the size of the target input surface and the size of the virtual keyboard may be predefined, and the size of the virtual keyboard may be determined according to the size of the target input surface in the attribute information.

[0101] Step 502: If the attribute information includes the orientation of the target input surface, determining the orientation of the virtual keyboard according to the orientation of the target input surface;

[0102] In this step, the orientation of the virtual keyboard and the orientation of the target input surface may be set to be the same, which is not limited in the embodiments of this specification.

[0103] Step 503: if the attribute information includes the shape of the target input surface, determining the distribution mode of the virtual keyboard according to the shape of the target input surface;

[0104] In this step, the correspondence between the shape of the target input surface and the distribution of the virtual keyboard can be predefined, and the virtual keyboard and the virtual keys on the virtual keyboard can be laid out according to the user's thumb or other input source's habitual movement on the real contact surface (such as movement range, position, path, etc.). The method of the embodiment of this specification can provide a default initial virtual keyboard, and the user can also define it as needed. For example, a split virtual keyboard with left and right halves, a virtual keyboard with a more ergonomic arc distribution, a virtual keyboard arranged within a fixed range (such as on the upper thigh), etc. The embodiment of this specification does not limit the size, form, layout of virtual keys, etc. of the virtual keyboard.

[0105] Step 504: If the attribute information includes the position of the target input surface, determining the position of the virtual keyboard according to the position of the target input surface;

[0106] In this step, a virtual keyboard is generated at the position of the target input surface in the augmented reality space.

[0107] Step 505: Generate the virtual keyboard according to one or more of the size, orientation, distribution mode and position of the virtual keyboard.

[0108] Figure 6 The following are examples of the display positions of two virtual keyboards in the embodiment of this specification. Figure 6 In the example, the inner side of the index finger can be used as the preset input surface, the thumb tip of the same hand can be used as the input source, the thumb tip and the inner side of the index finger of this hand can be tracked, and when the distance between the thumb tip and the inner side of the index finger is less than the threshold value D1, a virtual keyboard that fits the inner side of the index finger is displayed on the inner side of the index finger. The desktop can also be used as the preset input surface, and the three preset joints of the hand can be used as input sources respectively. When the distance between at least one input source and the desktop is less than the threshold value D2, a virtual keyboard that fits the desktop is displayed on the desktop.

[0109] Furthermore, in the process of the user completing the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface, the method further includes:

[0110] Provide input feedback to the user according to the feedback mode corresponding to the target input surface.

[0111] In the embodiments of the present specification, the feedback method can be sound feedback and / or key color change, that is, the sound feedback and / or key color change of the virtual keyboard input corresponding to each preset input surface is pre-defined (it can also be customized by the user, and the embodiments of the present specification do not limit this), thereby improving the user experience.

[0112] In order to enable the user to obtain physical tactile feedback when inputting, the embodiment of this specification adopts that the input source approaches the preset input surface as pre-selection (Hover), and then leaves as input (Tap).

[0113] Specifically, the process of the user completing the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or the object surface corresponding to the target input surface includes:

[0114] When the user moves the body part corresponding to the input source close to the target input surface, a virtual key corresponding to the position of the input source is pre-selected. Then, the user moves the body part corresponding to the input source away from the target input surface, and the pre-selected virtual key is input.

[0115] The user moves the body part corresponding to the input source close to the target input surface, and the pre-selection of the virtual key corresponding to the position of the input source further includes:

[0116] Determining whether the distance between the input source and the target input surface is less than a preselected distance threshold;

[0117] If so, a virtual key corresponding to the position of the input source is pre-selected.

[0118] In the embodiments of this specification, the preselected distance threshold may be set based on experience, or the user may customize it based on input sensitivity, which is not limited in the embodiments of this specification.

[0119] In the embodiment of the present specification, the input reference point of the virtual key can be preset. When the distance between the input source and the target input surface is less than the preselected distance threshold, the virtual key closest to the reference point position is preselected as the preselected virtual key.

[0120] Preferably, in the embodiment of the present specification, when the user contacts the target input surface with the body part corresponding to the input source, the virtual key corresponding to the position where the input source contacts the target input surface is pre-selected.

[0121] The method in which the user contacts the body part corresponding to the input source on the target input surface for inputting is essentially to determine whether the distance between the body part corresponding to the input source and the target input surface is less than the pre-selected distance threshold, and the preset threshold can be set to be smaller. For the user, there is no need to care about the size of the pre-selected distance threshold when inputting, but directly contact the body part corresponding to the input source on a corresponding virtual key in the virtual keyboard on the target input surface. Therefore, when the body part corresponding to the input source contacts the target input surface, the distance between the two is the smallest, and usually it must be less than the pre-selected distance threshold. When the user contacts the body part corresponding to the input source on the target input surface, the body part corresponding to the input source can obtain real physical tactile feedback, and the virtual key at the corresponding position of the input source is pre-selected. Then the user leaves (for example, lifts) the body part corresponding to the input source from the target input surface, and then inputs the pre-selected virtual key. The input method of the embodiments of this specification is different from the traditional solution of hovering switching pre-selection. The method of approaching as pre-selection (Hover) and then leaving as input (Tap) in the embodiments of this specification can obtain real physical tactile feedback, making the pre-selection switching more accurate, and the user's body surface or object surface of the tactile feedback can also serve as support for the user when inputting, reducing fatigue.

[0122] For example, Figure 7 As shown, the 6Dof information of the specified joint point of the user's hand and the 6Dof information of the virtual keyboard (which can be the left and right parts) are obtained. According to the predetermined distance judgment algorithm, it is determined whether the current virtual key A is pre-selected (Hover) or activated / input (Tap).

[0123] In addition, the embodiment of the present specification can also perform more accurate pre-selection switching by using the method that when the input source approaches the preset input surface, it is pre-selected (Hover), and then leaves it as input (Tap). Specifically, after the user pre-selects the virtual key corresponding to the position of the input source, the method further includes:

[0124] When the body part corresponding to the input source remains close to the target input surface, the user moves the body part corresponding to the input source, and switches the pre-selected virtual button according to the movement of the input source.

[0125] It can be understood that after the user touches the target input surface with the body part corresponding to the input source, if the user does not move the body part corresponding to the input source away from the target input surface, the pre-selected virtual key will not be input. At this time, if the user wants to pre-select other virtual keys, he only needs to slide the body part corresponding to the input source on the target input surface. When input is needed, he only needs to move the body part corresponding to the input source away from the target input surface.

[0126] In addition, to avoid accidental touches by the user, the embodiment of this specification can also exit the pre-selection by a specified operation mode after the user pre-selects the virtual key, without inputting the pre-selected virtual key. The embodiment of this specification does not limit the specific operation mode for exiting the pre-selection.

[0127] According to an embodiment of the present specification, the user completing the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface further includes:

[0128] The user interacts with the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface based on the input aiming mode corresponding to the target input surface, thereby completing the input of the corresponding virtual key on the virtual keyboard.

[0129] Because the method of the embodiment of this specification supports multiple preset input surfaces, such as the index finger surface (curved surface) and the physical surface (plane), it supports one input aiming mode corresponding to each preset input surface Surface_i.

[0130] In the embodiments of this specification, there are three calculation methods for the input reference point of the pre-selection (Hover) process, corresponding to the input aiming modes of the three input modes. The user can customize the input aiming mode of his input mode, and the input aiming mode includes the line of sight intersection mode, the input source projection mode and the input source nearest interaction point mode.

[0131] like Figure 8Shown are schematic diagrams of input aiming modes for three input modes, where the default (when not set by the user) is the “line of sight intersection” mode (the XR system may select an aiming mode based on the characteristics of the device, and the embodiments of this specification do not limit the default mode).

[0132] Line of sight intersection mode: The reference point is the intersection of the line of sight starting point (e.g., the Center Eye Anchor of glasses) and the position of the preset input source (e.g., the tip of the index finger) in the first frame during the same Hover interaction on the virtual keyboard. It should be noted that the embodiment of this specification allows that the point does not appear on the virtual keyboard, but on a preset plane (e.g., a physical plane), and the calculation method is the same, that is, the virtual key closest to the point is still calculated.

[0133] Preset input source projection mode: The reference point is the position of the projection point of the preset input source (such as the tip of the index finger) in the first frame of a Hover interaction on the virtual keyboard / preset plane.

[0134] Closest interaction point mode: The reference point is the position of the point where the preset input source is closest to the virtual keyboard during a hover process (from the first frame to the last frame).

[0135] The embodiments of this specification pre-set (or are customized by the user) the input aiming mode of each preset input surface, so that a higher input accuracy rate can be achieved in different usage scenarios.

[0136] Optionally, the input aiming mode can also be calibrated to allow the user to complete a typing game using the virtual keyboard of the embodiment of this specification. Collect the preset input sources and reference point positions in different input aiming modes during the user's typing process, such as the position of the thumb tip (input source) and the thumb belly surface (target input surface) and the position of the pressed virtual key. Calculate the distance between these points and the center position of the user's target key, such as the point P1 where the thumb tip / thumb belly is closest to the virtual keyboard; the intersection P2 of the line of sight and the thumb tip with the virtual keyboard; the projection point P3 of the thumb tip to the virtual keyboard in the F0 frame. Calculate the mean of the distance between these points and the center position of the real target key in the virtual keyboard, and select the mode corresponding to the reference point with the smallest mean as the user's preferred input aiming mode. Note: The target key is the virtual key that prompts the user to input in this game.

[0137] According to an embodiment of the present specification, after generating the virtual keyboard on the target input surface, the method further includes:

[0138] Determine whether the target input surface is out of the user's field of view, and if so, end the input of the virtual keyboard; or,

[0139] It is determined whether the distance between the target input surface and the user's viewpoint exceeds a preset distance threshold, and if so, the input of the virtual keyboard is terminated.

[0140] In the embodiments of the present specification, when the target input surface moves out of the user's field of view or the distance between the target input surface and the user's viewpoint exceeds a preset distance threshold, the user can be prompted to exit the "input mode" or to delay for a certain period of time before exiting the "input mode" or to exit the "input mode" immediately, and the embodiments of the present specification do not impose any restrictions.

[0141] Furthermore, after generating the virtual keyboard on the target input surface, the method further includes:

[0142] In the case where the target input surface is the user's body surface, it is determined whether the body movement change of the user's body surface corresponding to the target input surface meets the preset input termination condition, and if so, the input of the virtual keyboard is terminated.

[0143] In the embodiments of the present specification, changes in body movements on the user's body surface may include changes in the amplitude of body surface movement, changes in posture, etc. For example, the user's palm is the target input surface, and when entering "input mode", the palm is open, and a virtual keyboard is generated on the palm. If it is detected that the palm is clenched, it can be considered that the preset input end condition is met, and the input of the virtual keyboard is ended.

[0144] For example, Fig. 9 As shown, the user's body part P_hand corresponding to the input source P_input is the thumb pad of the left hand, and the preset input surface is a part of the radial surface of the left index finger. According to the preset information and the tracking of the 6Dof information of P_hand and the preset space, when the mode determination conditions 1 and 2 are met, that is, the thumb pad is maintained in the preset input space where the preset surface of the left index finger is located for a time period exceeding the time threshold T (such as 2s), the "input mode" is entered.

[0145] When entering the "input mode", the style, size, position and orientation of the virtual keyboard are determined according to the attribute information of the preset input surface at that moment: here, the space near the radial surface of the left index finger. The virtual keyboard in this example includes three virtual keys arranged horizontally occupying the preset input surface (such as Fig. 9 The virtual keyboard is oriented toward the starting point of the user's line of sight. When the position of the preset input surface or the angle relative to the user's line of sight is detected to have changed beyond the position or angle threshold, the position and orientation of the virtual keyboard are updated. If the "input mode" end information is received from the mode judgment module, the virtual keyboard is folded and no longer displayed.

[0146] When the input mode is turned on, the 6Dof information of P_input is monitored, that is, the position of the left thumb. The virtual key "A" closest to the reference point of P_input is in the Hover state. If P_input continues to move and does not leave the preset input space, the Hovered virtual key is updated according to the position of its reference point, such as switching to the virtual key "S"; if it moves out of the preset input space, the previous Hovered virtual key is input.

[0147] Finally, complete the visual and / or auditory feedback for the virtual key being hovered and the virtual key being tapped. Fig. 9 The visual effect of the virtual button "A / S" when hovering is that the front panel moves to the rear panel. When tapped, the front panel returns to its initial position from the rear panel position.

[0148] Based on the same inventive concept, the embodiments of this specification also provide an input device for obtaining tactile feedback in an expanded real space, such as Fig.10 As shown, the device comprises:

[0149] An input setting unit 1001, configured to use one or more surfaces of a user's body or one or more surfaces of an object as preset input surfaces, and use at least one body part of the user as an input source;

[0150] An input determination unit 1002 is used to track the input source and each of the preset input surfaces, and when the relationship between the input source and the preset input surface meets a predetermined requirement, use the preset input surface as a target input surface and generate a virtual keyboard on the target input surface;

[0151] The input unit 1003 is used to process the user's interaction on the body surface or object surface corresponding to the target input surface through the body part corresponding to the input source, so as to complete the input of the corresponding virtual key on the virtual keyboard.

[0152] Furthermore, the process of the user completing the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface includes:

[0153] When the user moves the body part corresponding to the input source close to the target input surface, a virtual key corresponding to the position of the input source is pre-selected. Then, the user moves the body part corresponding to the input source away from the target input surface, and the pre-selected virtual key is input.

[0154] Furthermore, when the user brings the body part corresponding to the input source close to the target input surface, the preselected virtual key corresponding to the position of the input source further includes:

[0155] Determining whether the distance between the input source and the target input surface is less than a preselected distance threshold;

[0156] If so, a virtual key corresponding to the position of the input source is pre-selected.

[0157] Furthermore, when the user brings the body part corresponding to the input source close to the target input surface, the preselected virtual key corresponding to the position of the input source further includes:

[0158] When the user contacts the target input surface with the body part corresponding to the input source, the virtual key corresponding to the position where the input source contacts the target input surface is pre-selected.

[0159] Furthermore, the input unit 1003 is also used to, after the user pre-selects a virtual key corresponding to the position of the input source, move the body part corresponding to the input source while keeping the body part corresponding to the input source close to the target input surface to switch the pre-selected virtual key.

[0160] Furthermore, when the relationship between the input source and the preset input surface meets a predetermined requirement, taking the preset input surface as a target input surface and generating a virtual keyboard on the target input surface further comprises:

[0161] When the position of the input source falls into the input space of the preset input surface and the duration exceeds a preset time length, the preset input surface is used as the target input surface and a virtual keyboard is generated on the target input surface.

[0162] Furthermore, generating a virtual keyboard on the target input surface further comprises:

[0163] Determining attribute information of the target input surface when the relationship between the input source and the target input surface meets a predetermined requirement;

[0164] The virtual keyboard is generated according to the attribute information.

[0165] Furthermore, the attribute information includes one or more of the size, orientation, shape and position of the target input surface.

[0166] Furthermore, generating the virtual keyboard according to the attribute information further includes:

[0167] If the attribute information includes the size of the target input surface, determining the size of the virtual keyboard according to the size of the target input surface;

[0168] If the attribute information includes the orientation of the target input surface, determining the orientation of the virtual keyboard according to the orientation of the target input surface;

[0169] If the attribute information includes the shape of the target input surface, determining the distribution mode of the virtual keyboard according to the shape of the target input surface;

[0170] If the attribute information includes the position of the target input surface, determining the position of the virtual keyboard according to the position of the target input surface;

[0171] The virtual keyboard is generated according to one or more of the size, orientation, distribution mode and position of the virtual keyboard.

[0172] Furthermore, the input unit 1003 is also used to provide input feedback to the user in accordance with the feedback method corresponding to the target input surface when the user completes the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface.

[0173] Furthermore, the user completing the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface further includes:

[0174] The user interacts with the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface based on the input aiming mode corresponding to the target input surface, thereby completing the input of the corresponding virtual key on the virtual keyboard.

[0175] Furthermore, the input aiming mode is a line of sight intersection mode, an input source projection mode or an input source nearest interaction point mode.

[0176] Furthermore, the input determination unit 1002 is further configured to determine whether the target input surface is out of the user's field of view after the virtual keyboard is generated on the target input surface, and if so, terminate the input of the virtual keyboard; or,

[0177] It is determined whether the distance between the target input surface and the user's viewpoint exceeds a preset distance threshold, and if so, the input of the virtual keyboard is terminated.

[0178] Furthermore, the input determination unit 1002 is also used to determine whether the changes in body movements of the user's body surface corresponding to the target input surface meet a preset input termination condition after the virtual keyboard is generated on the target input surface, when the target input surface is the user's body surface; if so, terminating the input of the virtual keyboard.

[0179] The beneficial effects obtained by the above-mentioned device are consistent with the beneficial effects obtained by the above-mentioned method, and will not be described in detail in the embodiments of this specification.

[0180] like Fig.11 The diagram shows the structure of a computer device in an embodiment of the present specification. The apparatus in the present specification may be the computer device in the embodiment, executing the method in the present specification.

[0181] Computer device 1102 may include one or more processing devices 1104, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads.

[0182] The computer device 1102 may also include any storage resources 1106 for storing any kind of information such as code, settings, data, etc. For example, without limitation, the storage resources 1106 may include any one or a combination of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc.

[0183] More generally, any storage resource may use any technology to store information.

[0184] Further, any storage resource may provide volatile or non-volatile retention of information.

[0185] Further, any storage resources may represent fixed or removable components of computer device 1102 .

[0186] In one case, when the processing device 1104 executes the associated instructions stored in any storage resource or combination of storage resources, the computer device 1102 may perform any of the operations of the associated instructions.

[0187] The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any storage resources, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like.

[0188] The computer device 1102 may also include an input / output module 1110 (I / O) for receiving various inputs (via input devices 1112 ) and for providing various outputs (via output devices 1114 ).

[0189] One particular output mechanism may include a presentation device 1116 and an associated graphical user interface (GUI) 1118 .

[0190] In other embodiments, the input / output module 1110 (I / O), the input device 1112 and the output device 1114 may not be included, and the computer only serves as a computer device in the network.

[0191] The computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.

[0192] The communication link 1122 may be implemented in any manner, such as through a local area network, a wide area network (eg, the Internet), a point-to-point connection, etc., or any combination thereof.

[0193] The communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0194] The embodiments of the present specification also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0195] The embodiments of the present specification also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the above method.

[0196] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0197] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0198] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0200] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0201] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0202] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0203] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0204] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. An input method for obtaining tactile feedback in an extended real space, characterized in that: The method comprises: One or more surfaces of a user's body or one or more surfaces of an object are used as preset input surfaces, and at least one body part of the user is used as an input source; Tracking the input source and each of the preset input surfaces, and when the relationship between the input source and the preset input surface meets a predetermined requirement, taking the preset input surface as a target input surface and generating a virtual keyboard on the target input surface; The user completes input of corresponding virtual keys on the virtual keyboard through interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface.

2. The method according to claim 1, characterized in that The process of the user completing the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or the object surface corresponding to the target input surface includes: When the user moves the body part corresponding to the input source close to the target input surface, a virtual key corresponding to the position of the input source is pre-selected. Then, the user moves the body part corresponding to the input source away from the target input surface, and the pre-selected virtual key is input.

3. The method according to claim 2, characterized in that The user moves the body part corresponding to the input source close to the target input surface, and the pre-selection of the virtual key corresponding to the position of the input source further includes: Determining whether the distance between the input source and the target input surface is less than a preselected distance threshold; If so, a virtual key corresponding to the position of the input source is pre-selected.

4. The method according to claim 2, characterized in that: The user moves the body part corresponding to the input source close to the target input surface, and the pre-selection of the virtual key corresponding to the position of the input source further includes: When the user contacts the target input surface with the body part corresponding to the input source, the virtual key corresponding to the position where the input source contacts the target input surface is pre-selected.

5. The method according to claim 2, characterized in that: After the user preselects a virtual key corresponding to the position of the input source, the method further includes: When the body part corresponding to the input source is kept close to the target input surface, the user moves the body part corresponding to the input source to switch the pre-selected virtual button.

6. The method according to claim 1, characterized in that When the relationship between the input source and the preset input surface meets a predetermined requirement, taking the preset input surface as a target input surface and generating a virtual keyboard on the target input surface further comprises: When the position of the input source falls into the input space of the preset input surface and the duration exceeds a preset time length, the preset input surface is used as the target input surface and a virtual keyboard is generated on the target input surface.

7. The method according to claim 1, characterized in that Generating a virtual keyboard on the target input surface further comprises: Determining attribute information of the target input surface when the relationship between the input source and the target input surface meets a predetermined requirement; The virtual keyboard is generated according to the attribute information.

8. The method according to claim 7, characterized in that The attribute information includes one or more of the size, orientation, shape, and position of the target input surface.

9. The method according to claim 8, characterized in that Generating the virtual keyboard according to the attribute information further comprises: If the attribute information includes the size of the target input surface, determining the size of the virtual keyboard according to the size of the target input surface; If the attribute information includes the orientation of the target input surface, determining the orientation of the virtual keyboard according to the orientation of the target input surface; If the attribute information includes the shape of the target input surface, determining the distribution mode of the virtual keyboard according to the shape of the target input surface; If the attribute information includes the position of the target input surface, determining the position of the virtual keyboard according to the position of the target input surface; The virtual keyboard is generated according to one or more of the size, orientation, distribution mode and position of the virtual keyboard.

10. The method according to claim 1, characterized in that In the process in which the user completes inputting corresponding virtual keys on the virtual keyboard through interaction of the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface, the method further includes: Provide input feedback to the user according to the feedback mode corresponding to the target input surface.

11. The method according to claim 1, characterized in that: The user completes the input of the corresponding virtual key on the virtual keyboard through the interaction of the body part corresponding to the input source on the body surface or the object surface corresponding to the target input surface, further comprising: The user interacts with the body part corresponding to the input source on the body surface or object surface corresponding to the target input surface based on the input aiming mode corresponding to the target input surface, thereby completing the input of the corresponding virtual key on the virtual keyboard.

12. The method according to claim 11, characterized in that The input aiming mode is a line of sight intersection mode, an input source projection mode, or an input source nearest interaction point mode.

13. The method according to claim 1, characterized in that After generating the virtual keyboard on the target input surface, the method further includes: Determine whether the target input surface is out of the user's field of view, and if so, end the input of the virtual keyboard; or, It is determined whether the distance between the target input surface and the user's viewpoint exceeds a preset distance threshold, and if so, the input of the virtual keyboard is terminated.

14. The method according to claim 1, characterized in that After generating the virtual keyboard on the target input surface, the method further includes: In the case where the target input surface is the user's body surface, it is determined whether the body movement change of the user's body surface corresponding to the target input surface meets the preset input termination condition, and if so, the input of the virtual keyboard is terminated.

15. An input device for obtaining tactile feedback in an extended real space, characterized in that: The device comprises: An input setting unit, used to use one or more surfaces of a user's body or one or more surfaces of an object as preset input surfaces and at least one body part of the user as an input source; an input determination unit, configured to track the input source and each of the preset input surfaces, and when the relationship between the input source and the preset input surface meets a predetermined requirement, use the preset input surface as a target input surface and generate a virtual keyboard on the target input surface; The input unit is used to process the user's interaction on the body surface or object surface corresponding to the target input surface through the body part corresponding to the input source, so as to complete the input of the corresponding virtual key on the virtual keyboard.

16. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 14 is implemented.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images