Electronic device for acquiring object contact information and control method thereof

Abstract

The electronic device includes a contact portion including a plurality of marks, and a camera disposed on the contact portion when one surface of the contact portion is in contact with an object. Identifying at least one displacement of the plurality of marks based on a first image obtained by photographing a first surface of the contact portion facing the at least one surface by the camera, and identifying at least one of an intensity of a force applied to the contact portion and a direction of the force applied to the contact portion based on the at least one displacement, in which a first mark of the plurality of marks may include at least one of a shape and a color different from a shape and a color of a second mark, and a second mark of the plurality of marks may include at least one of an intensity of the force applied to the contact portion and a direction of the force applied to the contact portion. The second marker is in the plurality of markers and adjacent to the first marker.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device for acquiring contact information related to an object and a control method thereof. Background Art

[0002] As electronic technology develops, various types of electronic devices are being developed. Recently, robots with forms and functions similar to those of humans are being developed. Summary of the invention

[0003] Provided are an electronic device for improving the sensing performance of contact information of a robot hand and a control method thereof.

[0004] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0005] According to one aspect of the present disclosure, an electronic device may include a contact portion, a camera, and at least one processor, the contact portion including a plurality of marks, the at least one processor being configured to identify at least one displacement of the plurality of marks based on a first image based on at least one surface of the contact portion of a contact object, and to identify at least one of the intensity of a force applied to the contact portion and the direction of the force applied to the contact portion based on the at least one displacement, wherein in the first image, a first surface of the contact portion opposite to the at least one surface is captured by the camera, wherein a first mark among the plurality of marks may include at least one of a form and a color that is different from a second mark among the plurality of marks that is adjacent to the first mark.

[0006] Each of the multiple marks may include at least one first pattern of a first form and at least one second pattern of a second form arranged on one side of the at least one first pattern, and the at least one processor may also be configured to identify at least one of the intensity of the force and the direction of the force based on the at least one first pattern and the at least one second pattern.

[0007] The first form may include a circular pattern, and the second form may include a linear pattern.

[0008] The at least one processor may be further configured to identify a rotational force parallel to the contact portion according to a displacement of the linear pattern based on the circular pattern.

[0009] The at least one second pattern may include a plurality of second patterns, and adjacent patterns of the plurality of second patterns may vary in color.

[0010] The first mark may include a circular first pattern and a linear first pattern oriented from the circular first pattern toward a first direction, the second mark may be adjacent to the first mark in a direction opposite to the first direction, the second mark includes a circular second pattern and a linear second pattern oriented from the circular second pattern toward a second direction perpendicular to the first direction, the multiple marks may include a third mark and a fourth mark, the third mark may be adjacent to the second mark in a direction opposite to the second direction, the third mark includes a circular third pattern and a linear third pattern oriented from the circular third pattern toward a third direction perpendicular to the second direction, and the fourth mark may be adjacent to the third mark in a direction opposite to the third direction, the fourth mark includes a circular fourth pattern and a linear fourth pattern oriented from the circular fourth pattern toward a fourth direction perpendicular to the third direction.

[0011] The electronic device may include a lens array arranged at a relative surface of the contact portion, the lens array including multiple lenses corresponding to the multiple marks respectively, wherein the at least one processor may also be configured to identify a size change of each of the multiple marks through each of the multiple lenses, and identify a force in a direction perpendicular to the contact portion based on the identified size change.

[0012] Each of the plurality of markings may include a first pattern of a first form or a first pattern of a first form and at least one second pattern of a second form adjacent to the first pattern of the first form.

[0013] The electronic device may include a memory configured to store a second image in which a surface of the contact portion is captured by a camera before the contact portion contacts the object, and the at least one processor may be further configured to identify at least one displacement based on the first image and the second image.

[0014] The contact portion may include a transparent gel and a coating layer formed at upper ends of the plurality of marks, and the plurality of marks may be provided at one surface of the transparent gel.

[0015] According to one aspect of the present disclosure, a control method of an electronic device includes: bringing at least one surface of a contact part of the electronic device into contact with an object, identifying at least one displacement of multiple marks set in the contact part based on a first image, and identifying at least one of the intensity of the force applied to the contact part or the direction of the force applied to the contact part based on the identified at least one displacement, wherein in the first image, a first surface of the contact part opposite to the at least one surface is captured by a camera of the electronic device, wherein the first mark may include at least one of a form and a color that is different from a second mark adjacent to the first mark among the multiple marks.

[0016] Each of the multiple marks may include at least one first pattern of a first form and at least one second pattern of a second form arranged on one side of the at least one first pattern of the first form, and may identify at least one of the intensity of the force and the direction of the force based on the at least one first pattern of the first form and the at least one second pattern of the second form.

[0017] The at least one first pattern of the first form may include a circular pattern, and the at least one second pattern of the second form may include a linear pattern.

[0018] Identifying at least one of the strength of the force and the direction of the force may include identifying a rotational force parallel to the contact portion according to a displacement of the linear pattern based on the circular pattern.

[0019] The at least one second pattern may include a plurality of second patterns, and adjacent patterns of the plurality of second patterns may vary in color.

[0020] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium may store instructions that, when executed by at least one processor, cause the at least one processor to perform the following operations: based on at least one surface of a contact part of an electronic device contacting an object, identify at least one displacement of a plurality of marks set in the contact part based on a first image, identify at least one of the intensity of a force applied to the contact part or the direction of the force applied to the contact part based on the identified at least one displacement, wherein in the first image, a first surface of the contact part opposite to the at least one surface is captured by a camera of the electronic device, wherein the first mark may include at least one of a form and a color that is different from a second mark of the plurality of marks adjacent to the first mark.

[0021] Each of the multiple marks may include at least one first pattern of a first form and at least one second pattern of a second form arranged on one side of the at least one first pattern, and the instructions, when executed, may also enable the at least one processor to identify at least one of the intensity of the force and the direction of the force based on the at least one first pattern and the at least one second pattern.

[0022] The first form may include a circular pattern, and the second form may include a linear pattern.

[0023] The instructions, when executed, may further cause the at least one processor to identify a rotational force parallel to the contact portion based on a displacement of the linear pattern based on the circular pattern.

[0024] The at least one second pattern may include a plurality of second patterns, and wherein adjacent patterns in the plurality of second patterns vary in color. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which: Figure 1 is a diagram illustrating a method of sensing contact information; Figure 2 is a block diagram showing a configuration of an electronic device according to one or more embodiments of the present disclosure; Figure 3 is a block diagram showing a detailed configuration of an electronic device according to one or more embodiments of the present disclosure; Figure 4 is a diagram showing a plurality of indicia according to one or more embodiments of the present disclosure; Figure 5 is a diagram illustrating measurement of rotational force according to one or more embodiments of the present disclosure; Figure 6 is a diagram showing a plurality of marks including colors according to one or more embodiments of the present disclosure; Figure 7 and Figure 8 is a diagram showing a basic unit of a mark according to one or more embodiments of the present disclosure; Fig. 9 , Fig.10 and Fig.11 is a diagram showing an example of a contact portion implemented in a circular pattern of different colors according to one or more embodiments of the present disclosure; Fig.12 , Fig.13 and Fig.14 is a diagram showing a lens array according to one or more embodiments of the present disclosure; and Fig.15 is a flowchart illustrating a control method of an electronic device according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] Hereinafter, example embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the accompanying drawings, and their redundant descriptions will be omitted. The embodiments described herein are example embodiments, and therefore, the present disclosure is not limited thereto, and may be implemented in various other forms. It should be understood that unless the context clearly dictates otherwise, singular forms include plural referents. Terms including technical or scientific terms used in the present disclosure may have the same meanings as those generally understood by those skilled in the art.

[0027] The terms used in describing one or more embodiments of the present disclosure are currently widely used general terms selected in consideration of their functions herein. However, the terms may be changed according to the intentions of technicians in the relevant fields, legal or technical interpretations, the emergence of new technologies, etc. Further, in some cases, there may be arbitrarily selected terms, and in this case, the meaning of the terms will be disclosed in more detail in the corresponding description. Therefore, the terms used herein should not be simply understood as their names, but based on the meaning of the terms and the overall context of the present disclosure.

[0028] In the present disclosure, expressions such as “having”, “may have”, “including”, “may include”, etc. are used to specify the existence of corresponding characteristics (for example, elements such as values, functions, operations, or components) without excluding the existence or possibility of additional characteristics.

[0029] As used herein, expressions such as "at least one of..." when preceding a list of elements modify the entire list of elements rather than modifying the individual elements of the list. For example, the expression "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0030] Expressions such as "first", "second", "1st", "2nd", etc. used herein may be used to refer to various elements, regardless of order and / or importance. In addition, it should be noted that these expressions are only used to distinguish an element from another element, rather than limiting the relevant elements.

[0031] Unless otherwise specified, singular expressions may include plural expressions. It should be understood that terms such as "formed" or "comprising" are used herein to specify the presence of a characteristic, quantity, step, operation, element, component, or a combination thereof, and do not exclude the presence or possibility of adding one or more other characteristics, quantities, steps, operations, elements, components, or a combination thereof.

[0032] In the present disclosure, the term “user” may refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

[0033] Figure 1 It is a diagram for explaining a method of sensing contact information.

[0034] An example of a contact portion of a robot hand is shown at 10. The contact portion may include a plurality of markings, such as marking 11, such as Figure 1 10. Each of the marks may have a circular pattern.

[0035] The contact portion may be implemented with gel, and the robot may obtain contact information based on a plurality of markers included in the contact portion. The contact information may include the strength, direction, etc. of the force applied at the contact portion.

[0036] For example, as shown at 20, when a shear force is generated from the contact portion in the left direction, the shape of the contact portion implemented with the gel can be changed by the shear force, so that the marker 1 can move to the position 1', the marker 2 can move to the position 2', and the marker 3 can move to the position 3'. The robot can recognize the strength, direction, etc. of the force applied to the contact portion based on the displacement of the marker 1, the marker 2, and the marker 3.

[0037] However, when the strength of the force applied to the contact portion is increased, problems such as shown at 30 may occur. Specifically, marker 1 may move to position 1', marker 2 may move to position 2', and marker 3 may move to position 3'. Since the strength of the force at 30 is greater than at 20 at this time, position 2' may move closer to position 1 than position 2, and position 3' may move closer to position 2 than position 3. In this case, the robot may recognize that marker 1 has moved to position 2' and marker 2 has moved to position 3'. That is, the force in the right direction may be recognized as a force in the left direction, and an error may occur. Therefore, it is necessary to reduce the error as described above.

[0038] Figure 2 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure.

[0039] The electronic device 100 may be a device for obtaining contact information. For example, the electronic device 100 may be a robot. Alternatively, the electronic device 100 may be a robot hand or a finger portion of a robot hand. Alternatively, the electronic device 100 may be a type of optical tactile sensor.

[0040] However, the above is not limited thereto, and the electronic device 100 may be any device that obtains contact information.

[0041] refer to Figure 2 , the electronic device 100 may include a contact portion 110 , a camera 120 , and a processor 130 .

[0042] The contact portion 110 may include a plurality of marks. A first mark among the plurality of marks may be configured such that at least one of a form or a color is different from a second mark adjacent to the first mark.

[0043] For example, each of the marks may include a pattern of a first form and a pattern of a second form disposed (e.g., placed) at one side of the pattern of the first form. The pattern of the first form and the pattern of the second form may be different. In an example, the pattern of the first form may be a circular pattern, and the pattern of the second form may be a linear pattern. However, the above is not limited to this, and the pattern of the first form may be a quadrilateral pattern, and the pattern of the second form may be a linear pattern. The pattern of the first form and the pattern of the second form may be implemented in any various forms. In addition, each of the marks may be implemented not only as a pattern of the first form and a pattern of the second form, but also as a pattern of at least three forms.

[0044] Alternatively, adjacent patterns of the second form from among a plurality of patterns of the second form included in the plurality of indicia may vary in color.

[0045] Alternatively, a first mark among the plurality of marks may include a circular first pattern and a linear first pattern oriented from the circular first pattern toward a first direction, a second mark among the plurality of marks may be adjacent to the first mark in a direction opposite to the first direction and include a circular second pattern and a linear second pattern oriented from the circular second pattern toward a second direction perpendicular to the first direction, a third mark among the plurality of marks may be adjacent to the second mark in a direction opposite to the second direction and include a circular third pattern and a linear third pattern oriented from the circular third pattern toward a third direction perpendicular to the second direction, and a fourth mark among the plurality of marks may be adjacent to the third mark in a direction opposite to the third direction and include a circular fourth pattern and a linear fourth pattern oriented from the circular fourth pattern toward a fourth direction perpendicular to the third direction.

[0046] Alternatively, each of the marks may include a pattern of the first form, or include a pattern of the first form and at least one pattern of the second form formed adjacent to the pattern of the first form.

[0047] The plurality of markers may be implemented in any of various forms, and the above will be described in more detail through the accompanying drawings.

[0048] The contact portion 110 may include a transparent gel, a plurality of marks printed at one surface of the gel, and a coating formed at the upper ends of the plurality of marks. The coating may prevent the plurality of marks from being worn. However, the above is not limited thereto, and the gel may be translucent and may be any material whose degree of transparency of the plurality of marks can be recognized by the camera 120. In addition, the above may be implemented with another material other than the gel.

[0049] The camera 120 may be a configuration for capturing a still image or a moving image. The camera 120 may capture a still image at a specific point in time, but may also capture the still image continuously.

[0050] The camera 120 may include a lens, a shutter, an aperture, a solid-state imaging device, an analog front end (AFE), and a timing generator (TG). The shutter may be configured to adjust the time when light reflected from an object enters the camera 120, and the aperture may be configured to adjust the amount of light incident on the lens by mechanically increasing or decreasing the size of the opening portion through which the light enters. The solid-state imaging device may be configured to output an image generated by the photocharge as an electrical signal based on the light reflected from the object being accumulated as photocharge. The TG may be configured to output a timing signal for reading out pixel data of the solid-state imaging device, and the AFE may be configured to digitize the electrical signal output from the solid-state imaging device by sampling.

[0051] The camera 120 may capture the contact portion 110 according to the control of the processor 130. The processor 130 may recognize the positions of the plurality of markers from the captured image.

[0052] The processor 130 may be provided as a single processor or multiple processors, and may be implemented in a single device or multiple devices. The processor 130 may be referred to herein as at least one processor 130. The processor 130 may control the overall operation of the electronic device 100. Specifically, the processor 130 may control the overall operation of the electronic device 100 by connecting to each configuration of the electronic device 100. For example, the processor 130 may control the operation of the electronic device 100 by connecting to configurations such as, for example but not limited to, the contact portion 110, the camera 120, a memory, and the like.

[0053] At least one processor 130 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a multi-integrated core (MIC), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. At least one processor 130 may control one or a random combination of other elements of the electronic device 100 and perform operations associated with communication or data processing. At least one processor 130 may execute at least one program or instruction stored in a memory. For example, at least one processor 130 may execute a method according to one or more embodiments of the present disclosure by executing at least one instruction stored in a memory.

[0054] When the method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor, or by multiple processors. For example, when the first operation, the second operation, and the third operation are performed by the method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor), and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0055] At least one processor 130 may be implemented as a single-core processor including one core, or may be implemented as at least one multi-core processor including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). If at least one processor 130 is implemented as a multi-core processor, each of the cores included in the multi-core processor may include a processor internal memory, such as a cache memory and an on-chip memory, and a common cache shared by multiple cores may be included in the multi-core processor. In addition, each core (or a portion of the multiple cores) included in the multi-core processor may independently read and execute a program command for implementing a method according to one or more embodiments, or read and execute a program command for implementing a method according to one or more embodiments of the present disclosure due to the interconnection of all (or a portion) of the multiple cores.

[0056] When the method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one of the multiple cores, or by the multiple cores included in the multi-core processor. For example, when the first operation, the second operation, and the third operation are performed by the method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first core included in the multi-core processor, or the first operation and the second operation may be performed by the first core included in the multi-core processor, and the third operation may be performed by the second core included in the multi-core processor.

[0057] According to one or more embodiments, at least one processor 130 may refer to a system on chip (SoC) in which at least one processor and other electronic components are integrated, a single-core processor or a multi-core processor, or a core included in a single-core processor or a multi-core processor, and the core herein may be implemented as a CPU, a GPU, an APU, a MIC, an NPU, a hardware accelerator, a machine learning accelerator, etc., but the embodiments of the present disclosure are not limited thereto. However, for ease of description, the expression "processor 130" will be used below to describe the operation of the electronic device 100.

[0058] The processor 130 may identify at least one displacement of a plurality of marks based on a first image of a relative surface of the contact portion 110 being captured by the camera 120 based on one surface of the contact portion 110 contacting an object, and identify at least one of the intensity of a force or the direction of the force applied to the contact portion 110 based on the identified displacement.

[0059] For example, when each mark includes a pattern of a first form and a pattern of a second form disposed at one side of the pattern of the first form, the processor 130 may identify at least one of the strength of the force or the direction of the force based on the pattern of the first form and the pattern of the second form. In an example, if the pattern of the first form is a circular pattern and the pattern of the second form is a linear pattern, the processor 130 may identify a rotational force parallel to the contact portion based on the circular pattern according to the displacement of the linear pattern disposed at one side of the circular pattern.

[0060] The electronic device 100 may further include a memory that stores a second image of the opposite surface of the contact portion 110 captured by the camera 120 before one surface of the contact portion 110 contacts the object, and the processor 130 may identify at least one displacement of the plurality of markers based on the first image and the second image. That is, the processor 130 may identify at least one displacement of the plurality of markers by comparing the second image with the currently captured first image.

[0061] In addition, the processor 130 may capture a plurality of images at a preset time interval, and identify at least one change in the strength or direction of the force by comparing the second image with each of the plurality of images. For example, the processor 130 may identify that the direction of the force is toward the left side by comparing the image captured at time point T1 with the second image, and identify that the direction of the force is toward the right side by comparing the image captured at time point T2 after time point T1 with the second image. In this case, the processor 130 may identify that the direction of the force is gradually changing.

[0062] Alternatively, the processor 130 may capture a plurality of images at a preset time interval, identify a change in the strength of the force by comparing the second image with each of the plurality of images, predict the strength of the force at a future time point based on the change, and change the contact portion 110 to be contacted based on the predicted strength of the force. For example, the processor 130 may capture a plurality of images at a preset time interval, and stop the contact of the object with the contact portion 110 based on identifying that the strength of the force is gradually increasing by comparing the second image with each of the plurality of images, and that the contact portion 110 will be damaged after a determined period of time (e.g., three seconds) according to the increasing trend. However, the above is not limited thereto, and the processor 130 may stop the contact of the object with the contact portion 110 before a preset time from the time point when the contact portion 110 is predicted to be damaged. In an example, the processor 130 may stop the contact of the object with the contact portion 110 after two seconds, which is one second before the time point when the contact portion 110 is predicted to be damaged, based on the contact portion 110 being identified as damaged after a determined amount of time (e.g., three seconds).

[0063] The processor 130 may provide a damage message to the user based on the contact portion 110 being damaged. For example, the processor 130 may capture the contact portion 110 through the camera 120 when the contact portion 110 is not in contact with the object, and recognize the contact portion 110 as damaged based on the captured image changing from the second image by an amount greater than or equal to a preset level. The processor 130 may display a damage message or provide the damage message as a sound based on the contact portion 110 being recognized as damaged.

[0064] When the preset use time has passed, the processor 130 may update the second image. For example, the contact portion 110 may deform the gel due to the application of physical force, and therefore, even when the contact portion 110 is not contacted by the object, the positions of the plurality of marks may be different from those of the second image. Therefore, when the preset use time has passed, the processor 130 may capture the contact portion 110 through the camera 120 when the contact portion 110 is not in contact with the object, and update the second image using the captured image.

[0065] The electronic device 100 may further include a lens array formed at the opposite surface of the contact portion 110, the lens array including a plurality of lenses corresponding to each mark. The processor 130 may recognize the size change of the plurality of marks through the lens array, and recognize the force in the direction perpendicular to the contact portion 110 based on the recognized size change.

[0066] For example, the processor 130 may recognize that a force in a direction perpendicular to the contact portion 110 is applied based on an increase in the size of at least one mark among the plurality of marks.

[0067] However, the above is not limited thereto, and the lens array may be implemented so that one lens magnifies a portion of a plurality of marks. In an example, the lens array may be implemented so that one lens corresponds to four marks. Alternatively, the lens array may be implemented in the form of a lens including a portion of the plurality of marks. For example, a lens may be provided at a center mark among 3×3 marks, and a lens may not be provided at the remaining marks among the 3×3 marks. The processor 130 may identify a force in a direction perpendicular to the contact portion 110 based on a change in the size of the mark to which the lens is provided.

[0068] Alternatively, even without a lens array, the processor 130 may identify a force in a direction perpendicular to the contact portion 110. For example, when a force is applied in a direction perpendicular to the contact portion 110, the displacement increases according to a mark arranged closer to the center point of the force in the perpendicular direction, and the displacement of the mark around the center point may face the center point. The processor 130 may detect the displacement and identify the force in a direction perpendicular to the contact portion 110 based on the mark displacement rather than the size change of the mark.

[0069] The memory may store the neural network model. The processor 130 may recognize at least one of the strength of the force or the direction of the force by inputting the first image and the second image into the neural network model.

[0070] Functions associated with artificial intelligence according to the present disclosure may be operated by the processor 130 and the memory.

[0071] The processor 130 may be formed of one or more processors. The one or more processors may be general-purpose processors (such as CPU, application processor (AP), digital signal processor (DSP), etc.), graphics-specific processors (such as GPU and visual processing unit (VPU)), or artificial intelligence-specific processors (such as NPU).

[0072] One or more processors can control the input data to be processed according to predefined operating rules or artificial intelligence models stored in the memory. Alternatively, if one or more processors are artificial intelligence dedicated processors, the artificial intelligence dedicated processor can be designed as a hardware structure dedicated to the processing of a specific artificial intelligence model. The predefined operating rules or artificial intelligence models can be characterized by being created through learning.

[0073] Learning may refer to a predefined set of operating rules or artificial intelligence models to perform a desired feature (or purpose) trained by a learning algorithm using a plurality of learning data. Learning may be performed in the device itself that performs artificial intelligence according to the present disclosure, or by a separate server and / or system. Examples of learning algorithms may include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above examples.

[0074] The artificial intelligence model may be formed by a plurality of neural network layers. Each neural network layer may include a plurality of weight values, and the neural network calculation is performed by calculation between the calculation result of the previous layer and the plurality of weight values. The plurality of weight values ​​included in the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, the plurality of weight values ​​may be updated for the loss value or cost value obtained by the artificial intelligence model during the learning process to be reduced or minimized.

[0075] Artificial neural networks may include deep neural networks (DNNs), and examples thereof may include convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), deep Q networks, etc., but are not limited thereto.

[0076] Figure 3 1 is a block diagram showing a detailed configuration of an electronic device according to one or more embodiments of the present disclosure. The electronic device 100 may include a contact portion 110, a camera 120, and a processor 130. Figure 3 , the electronic device 100 may further include a memory 140, a communication interface 150, a display 160, a user interface 170, a speaker 180 and a microphone 190. Figure 3 The components shown in Figure 2 A detailed description of the overlapping parts of the elements shown in FIG.

[0077] The memory 140 may refer to hardware that stores information such as data in an electrical or magnetic form for access by the processor 130, etc. To this end, the memory 140 may be implemented as at least one hardware of a nonvolatile memory, a volatile memory, a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), a random access memory (RAM), a read-only memory (ROM), etc.

[0078] In the memory 140, at least one instruction for the operation of the electronic device 100 or the processor 130 may be stored. The instruction may be a code unit that instructs the operation of the electronic device 100 or the processor 130, and may be prepared in a machine language that is understood by a computer. Alternatively, the memory 140 may store a plurality of instructions for performing a specific work of the electronic device 100 or the processor 130 as an instruction set.

[0079] The memory 140 may store data, which is information in bit or byte units that may represent characters, numbers, images, etc. For example, the memory 140 may store a second image, a neural network model, and the like.

[0080] The memory 140 may be accessed by the processor 130 , and the reading, writing, modification, deletion, updating, etc. of instructions, instruction sets, or data may be executed by the processor 130 .

[0081] The communication interface 150 may be a configuration for performing communication with various types of external devices according to various types of communication methods. For example, the electronic device 100 may perform communication with a server, another electronic device, a sensor of another electronic device, etc. through the communication interface 150.

[0082] The communication interface 150 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Each communication module may be implemented in the form of at least one hardware chip.

[0083] The Wi-Fi module and the Bluetooth module can perform communication in the Wi-Fi method and the Bluetooth method, respectively. When using the Wi-Fi module or the Bluetooth module, various connection information such as a service set identifier (SSID) and a session key can be first sent and received, and after being connected to the communication using the connection information, various information can be sent and received. The infrared communication module can perform communication according to infrared communication (Infrared Data Association (IrDA)) technology, which wirelessly transmits data within a short distance by using infrared rays that exist between visible rays and millimeter waves.

[0084] In addition to the above-mentioned communication methods, the wireless communication module may also include at least one communication chip for performing communication according to various wireless communication standards, such as, for example but not limited to, ZigBee, third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), advanced LTE (LTE-A), fourth generation (4G), fifth generation (5G), etc.

[0085] Optionally, the communication interface 150 may include a wired communication interface, such as, for example but not limited to, a High-Definition Multimedia Interface (HDMI), a Display Port (DP), a Thunderbolt interface, a Universal Serial Bus (USB), RGB, D-SUB, a Digital Video Interface (DVI), etc.

[0086] Besides, the communication interface 150 may include at least one of a local area network (LAN) module, an Ethernet module, or a wired communication module that performs communication using a twisted pair, a coaxial cable, an optical fiber cable, or the like.

[0087] The display 160 may be a configuration for displaying an image, and may be implemented as various forms of displays, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, and a plasma display panel (PDP). In the display 160, a driving circuit, a backlight unit, etc., which may be implemented in the form of an a-si thin film transistor (TFT), a low temperature polysilicon (LTPS) TFT, an organic TFT (OTFT), etc., may be included. The display 160 may be implemented as a touch screen coupled with a touch sensor, a flexible display, a three-dimensional (3D) display, etc.

[0088] The processor 130 may control the display 160 to display at least one of the strength of the force or the direction of the force. Alternatively, the processor 130 may control the display 160 to warn of damage to the contact portion 110, notify of damage, and the like.

[0089] The user interface 170 may be implemented with buttons, a touch pad, a mouse, and a keyboard, or may be implemented as a touch screen capable of performing a display function and an operation input function together therewith. The buttons may be various types of buttons, such as mechanical buttons, touch pads, or wheels formed at random areas of the front surface portion or side surface portion, rear surface portion, etc., of the exterior of the body of the electronic device 100.

[0090] The speaker 180 may be an element that outputs not only various audio data processed in the processor 130 but also various notification sounds, voice messages, and the like.

[0091] The processor 130 may control the speaker 180 to output a sound guiding the state of the contact portion 110 .

[0092] The microphone 190 may be a configuration for receiving sound and converting into an audio signal. The microphone 190 may be electrically connected to the processor 130 and may receive sound by the control of the processor 130.

[0093] For example, the microphone 190 may be formed as an integrated type integrated into the upper side or the front surface direction, the side surface direction, etc. of the electronic device 100. Alternatively, the microphone 190 may be provided in a remote controller or the like separated from the electronic device 100. In this case, the remote controller may receive sound through the microphone 190 and provide the received sound to the electronic device 100.

[0094] The microphone 190 may include various configurations, such as a microphone for collecting sounds in analog form, an amplifier circuit for amplifying the collected sounds, an analog-to-digital (A / D) converter circuit for sampling the amplified sounds and converting them into digital signals, a filter circuit for removing noise components from the converted digital signals, and the like.

[0095] The microphone 190 may be implemented in the form of a sound sensor, and may be of any method as long as it is a configuration that can collect sound.

[0096] Processor 130 may also receive user commands through microphone 190 .

[0097] As described above, the electronic device 100 can use the contact portion 110 including multiple marks implemented in various forms and colors to identify various directions of force and reduce sensing errors even if the displacement is slightly large. In addition, the electronic device 100 can include a lens array and more accurately identify forces perpendicular to the contact portion 110.

[0098] exist Figure 2 and Figure 3 In the embodiment, the contact portion 110, the camera 120 and the processor 130 have been described as separate elements, but are not limited thereto. For example, one processor may control five contact portions and five cameras. Alternatively, one camera may capture two contact portions, and one processor may be used to sense the force applied to the two contact portions.

[0099] In addition, the operation of the electronic device 100 will be described in more detail below, and a separate embodiment will be described. However, it can be implemented in any combination state. Figures 4 to 14 A separate embodiment of .

[0100] Figure 4 is a diagram showing a plurality of indicia according to one or more embodiments of the present disclosure.

[0101] like Figure 4 As shown, a first mark 405 among a plurality of marks (such as mark 40) included in the contact portion 110 may include a circular first pattern 410-1 and a linear first pattern 410-2 oriented from the circular first pattern 410-1 toward a first direction. A second mark 406 among the plurality of marks may be adjacent to the first mark 405 in a direction opposite to the first direction, and include a circular second pattern 420-1 and a linear second pattern 420-2 oriented from the circular second pattern 420-1 toward a second direction perpendicular to the first direction. A third mark 407 among the plurality of marks may be adjacent to the second mark 406 in a direction opposite to the second direction, and include a circular third pattern 430-1 and a linear third pattern 430-2 oriented from the circular third pattern 430-1 toward a third direction perpendicular to the second direction. A fourth mark 408 of the plurality of marks may be adjacent to the third mark in a direction opposite to the third direction and include a circular fourth pattern 440-1 and a linear fourth pattern 440-2 oriented from the circular fourth pattern 440-1 toward a fourth direction perpendicular to the third direction.

[0102] In this case, even if the first mark 405 moves to the position of the second mark 406, the processor 130 may not confuse the first mark 405 with the second mark 406 due to the linear first pattern 410-2 and the linear second pattern 420-2, and thus, the problem caused by interference can be solved.

[0103] exist Figure 4 In the figure, for the convenience of description, four marks may be the basic unit 45, and the contact portion 110 has been shown as being formed with marks in the form of four marks of the basic unit 45 being repeated, but is not limited thereto. For example, nine marks in different forms from each other may be the basic unit, and the contact portion 110 may be formed with marks in the form of repeating nine marks, and may be implemented in any various forms.

[0104] Figure 5 is a diagram illustrating measurement of rotational force according to one or more embodiments of the present disclosure. Figure 5 In the description, for convenience, multiple marks (such as mark 50) are described as follows Figure 4 in the form of realization.

[0105] The first mark 500 is shown and includes a circular pattern 500-1 and a linear pattern 500-2. The processor 130 may recognize a rotation force parallel to the contact portion 110 according to a displacement of the linear pattern 500-2 formed at one side of the circular pattern 500-1 based on the circular pattern 500-1.

[0106] For example, Figure 5 As shown in FIG. , based on the linear pattern 500-2 formed at one side of the circular pattern 500-1 moving from position 510 to position 520, the processor 130 can identify the rotation angle of the linear pattern 500-2 based on the circular pattern 500-1, and identify the rotation force parallel to the contact portion 110 based on the rotation angle.

[0107] Figure 6 is a diagram illustrating a plurality of markers including colors according to one or more embodiments of the present disclosure.

[0108] As shown in 60 , when the linear pattern is rotated 90 degrees from the mark 600 - 1 , the linear pattern may move from the position 610 to the position 620 , and it may be difficult to distinguish the mark 600 - 1 from the mark 600 - 2 .

[0109] According to one or more embodiments, the contact portion 110 may include a color added to the linear pattern. In this case, as shown in FIG61, the first linear pattern 630, the second linear pattern 640, the third linear pattern 650, and the fourth linear pattern 660 may all have different colors, and even if the first linear pattern 630 is rotated 90 degrees and the form becomes the same as the adjacent mark on the right, because the color is different from the second linear pattern 640, the distinction between the marks is possible. Therefore, the sensing error of the processor 130 can be reduced.

[0110] exist Figure 6 In the embodiment, for the sake of convenience, the use of four colors has been described, but the above is not limited thereto. For example, the above can be implemented in the form that the colors of the first linear pattern 630 and the third linear pattern 650 are the same, and the colors of the second linear pattern 640 and the fourth linear pattern 660 are the same. In addition, the colors of the circular patterns can be different and can be implemented in any of various methods.

[0111] Figure 7 and Figure 8 is a diagram illustrating a basic unit of a mark according to one or more embodiments of the present disclosure.

[0112] like Figure 7 As shown, six marks may be a basic unit 700, and the contact portion 110 may be formed with marks in a form of repeating the six marks. The six marks of the basic unit may all be in different forms.

[0113] However, the above is not limited thereto, and the six marks of the basic unit may be implemented such that at least one of the form or the color is implemented in a different form.

[0114] Alternatively, if Figure 8 As shown, sixteen marks may be a basic unit 800, and the contact portion 110 may be formed with marks in a form of repeating sixteen marks. The sixteen marks of the basic unit may all be in different forms.

[0115] However, the above is not limited thereto, and the sixteen marks of the basic unit may be implemented such that at least one of the form or the color is implemented in a different form.

[0116] The displacement measurable by the processor 130 may increase as the number of markers included in the base unit increases, but the manufacturing cost may increase.

[0117] Fig. 9 , Fig.10 and Fig.11 is a diagram illustrating an example of a contact portion implemented in a circular pattern of different colors according to one or more embodiments of the present disclosure.

[0118] like Fig. 9 As shown, the contact portion 110 may include multiple marks having a single form pattern, such as mark 900, and the multiple marks herein may be implemented in two colors. For example, the first mark 910 may be red, and the second mark 920 may be black. In this case, because the forms of the multiple marks are all the same, the manufacturing cost can be reduced.

[0119] In addition, since the colors between horizontally or vertically adjacent marks are different, the sensing error of the processor 130 can be reduced. Fig.10 As shown in 1001, when the colors of the horizontally or vertically adjacent marks are the same and the magnitude of the force is relatively small, no interference occurs between the adjacent marks because the displacement of the marks is not as large as in 1010 and 1020. However, as Fig.10 As shown in 1002, when the colors of the horizontally or vertically adjacent marks are the same and the magnitude of the force is relatively large, the displacement of the marks may become larger and interference between the adjacent marks may occur as in 1030. That is, Fig.10 As shown in 1002 of FIG. 1 , even when a force is applied in the left direction, the processor 130 may recognize that the mark has moved in the right direction and recognize that a force has been applied in the right direction. Fig.10 As shown in 1003, when the colors between horizontally and vertically adjacent marks are different and the magnitude of the force is relatively large, even if the displacement of the mark becomes larger, the processor 130 can recognize that 1040 is a different mark and identify at least one of the strength of the force or the direction of the force based on 1050 and 1060.

[0120] That is to say, Fig.11 As shown in 1101, when the colors between the horizontally or vertically adjacent marks are the same, if the displacement is less than d, the processor 130 can sense the force without error. However, when the colors between the horizontally or vertically adjacent marks are different, if the displacement is less than about 1.4d, the force can be sensed without error, as shown in FIG. Fig.11 1102. That is, when the colors between the horizontally or vertically adjacent marks are different, the possibility of error occurrence can be reduced compared to when the colors between the horizontally or vertically adjacent marks are the same.

[0121] Fig.12 , Fig.13 and Fig.14 is a diagram illustrating a lens array according to one or more embodiments of the present disclosure.

[0122] The electronic device 100 may include a contact portion 110, a camera 120, a housing 1210 for accommodating the contact portion 110 and the camera 120, and a lens array 1220. Fig.12 As shown. Fig.12 In the figure, for the convenience of description, the processor 130 is omitted.

[0123] The contact portion 110 may include a gel 110 - 1 , a plurality of marks 110 - 2 printed at one surface of the gel 110 - 1 , and a coating layer 110 - 3 formed at upper ends of the plurality of marks 110 - 2 .

[0124] The lens array 1220 may be formed at the opposite surface of the gel 110-1 and include a plurality of lenses corresponding to each of the markers 110-2. Fig.13 As shown, the lens array 1220 may include a plurality of lenses 1310-1, 1310-2, ..., 1310-N corresponding to each of the markers 110-2. The plurality of lenses 1310-1, 1310-2, ..., 1310-N may respectively magnify the plurality of markers 110-2 (ie, the plurality of lenses may be used to generate images magnifying the plurality of markers).

[0125] The processor 130 may recognize a size change of each of the marks 110-2 through each of the lenses, and recognize a force in a direction perpendicular to the contact portion 110 based on the recognized size change. Fig.14 As shown, the processor 130 can identify the size of each mark in the mark 110-2 from the smaller magnification state shown in 1401 to the larger magnification state shown in 1402 through each lens, thereby more accurately identifying the size change of each mark in the mark 110-2.

[0126] Alternatively, by the structure as described, a plurality of marks 110 - 2 can be realized in a reduced size. In this case, there is an effect of increasing the durability of the contact portion 110. Alternatively, by the structure as described, the manufacturing cost can be reduced by reducing the resolution of the camera 120.

[0127] Fig.15 is a flowchart illustrating a control method of an electronic device according to one or more embodiments of the present disclosure.

[0128] In operation S1510, one surface of a contact portion included in an electronic device may contact an object. In operation S1520, a displacement of at least one of a plurality of marks included in the contact portion may be identified. The displacement may be identified based on a first image of an opposite surface of the contact portion captured by a camera included in the electronic device. In operation S1530, at least one of the strength of a force or the direction of the force applied to the contact portion may be identified based on the identified displacement. A first mark among the plurality of marks may be made different from a second mark adjacent to the first mark in at least one of a form or a color.

[0129] Each mark may include a first form pattern and a second form pattern set at one side of the first form pattern, and identifying at least one of the strength of the force or the direction of the force (operation S1530) may include identifying at least one of the strength of the force or the direction of the force based on the first form pattern and the second form pattern.

[0130] Furthermore, the pattern of the first form may be a circular pattern, and the pattern of the second form may be a linear pattern.

[0131] Identifying at least one of the strength of the force or the direction of the force (operation S1530 ) may include identifying a rotational force parallel to the contact portion according to a displacement of the linear pattern based on the circular pattern.

[0132] Furthermore, adjacent patterns of the second form may vary in color.

[0133] The first mark may include a circular first pattern and a linear first pattern oriented from the circular first pattern toward a first direction, the second mark may be adjacent to the first mark in a direction opposite to the first direction, the second mark includes a circular second pattern and a linear second pattern oriented from the circular second pattern toward a second direction perpendicular to the first direction, the multiple marks may include a third mark and a fourth mark, the third mark may be adjacent to the second mark in a direction opposite to the second direction, the third mark includes a circular third pattern and a linear third pattern oriented from the circular third pattern toward a third direction perpendicular to the second direction, and the fourth mark may be adjacent to the third mark in a direction opposite to the third direction, the fourth mark includes a circular fourth pattern and a linear fourth pattern oriented from the circular fourth pattern toward a fourth direction perpendicular to the third direction.

[0134] In addition, the electronic device may further include a lens array formed at an opposite surface of the contact portion and including a plurality of lenses corresponding to each mark. Identifying at least one of the strength of the force or the direction of the force (operation S1530) may include identifying a size change of each mark through each lens, and identifying a force in a direction perpendicular to the contact portion based on the identified size change.

[0135] Each indicia may include a pattern of the first form, or include a pattern of the first form and at least one pattern of the second form disposed adjacent to the pattern of the first form.

[0136] In addition, recognizing the displacement (operation S1520 ) may include recognizing at least one displacement of the plurality of markers based on capturing first and second images of an opposite surface of the contact portion through a camera before the surface of the contact portion contacts the object.

[0137] The contact portion may include the transparent gel, the plurality of marks may be printed at one surface of the gel, and a coating layer is formed at upper ends of the plurality of marks.

[0138] According to various embodiments of the present disclosure as described above, the electronic device can recognize various directions of force using a contact portion including a plurality of marks implemented in various forms and colors, and reduce a sensing error even if the displacement is slightly large.

[0139] In addition, the electronic device may include a lens array and more accurately recognize a force perpendicular to the contact portion.

[0140] According to one or more embodiments of the present disclosure, the various embodiments described above may be implemented with software including instructions stored in a machine-readable storage medium (e.g., a computer). The machine may call instructions stored in a storage medium, and as a device operable according to the called instructions, may include an electronic device according to the above embodiments. Based on the instructions executed by the processor, the processor may perform functions corresponding to the instructions directly or using other elements under the control of the processor. The instructions may include code generated by a compiler or executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. In this article, "non-transitory" may only mean that the storage medium is tangible and does not include signals, and the term does not distinguish whether data is semi-permanently stored in the storage medium or temporarily stored in the storage medium.

[0141] According to one or more embodiments, a method according to the various embodiments described above including a computer program product may be provided. The computer program product may be exchanged between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or distributed online through an application store (e.g., PLAYSTORE™). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be at least temporarily stored in a storage medium such as a memory of a manufacturer's server, an application store's server, or a relay server, or may be temporarily generated.

[0142] According to one or more embodiments of the present disclosure, the various embodiments described above can be implemented in a recordable medium, which can be read by a computer or a computer-like device using software, hardware, or a combination of software and hardware. In some cases, the embodiments described herein can be implemented by the processor itself. According to software implementation, embodiments such as the processes and functions described herein can be implemented with separate software. The corresponding software can perform one or more functions and operations described herein.

[0143] Computer instructions for performing processing operations in the apparatus according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in the non-transitory computer-readable medium may cause the specific apparatus to perform processing operations from the apparatus according to the various embodiments described above when executed by the processor of the specific apparatus. Non-transitory computer-readable medium may refer to a medium that stores data semi-permanently rather than for a very short period of time, such as a register, a cache, a memory, etc., and can be read by the apparatus. Specific examples of non-transitory computer-readable media may include, for example, but not limited to, a compact disc (CD), a digital versatile disc (DVD), a hard disk, a Blu-ray disc, a USB, a memory card, a ROM, etc.

[0144] In addition, the various elements (e.g., modules or programs) according to the various embodiments described above may be formed by a single entity or multiple entities, and some of the sub-elements described above may be omitted, or other sub-elements may be further included in various embodiments. Alternatively or additionally, some elements (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions performed by the corresponding corresponding elements before integration. According to various embodiments, the operations performed by modules, programs or other elements may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or at least some operations may be performed in a different order, omitted, or different operations may be added.

[0145] The embodiments of the present disclosure disclosed in the specification and the drawings only provide specific examples to easily describe the technical content according to the embodiments of the present disclosure and help understand the embodiments of the present disclosure, and are not intended to limit the scope of the embodiments of the present disclosure. Therefore, in addition to the embodiments disclosed herein, the scope of the various embodiments of the present disclosure should be interpreted as covering all modifications or changes derived from the technical spirit of the various embodiments of the present disclosure.

Claims

1. An electronic device, comprising: a contact portion, including a plurality of markings; camera; as well as At least one processor configured to: identifying at least one displacement of the plurality of markers based on contact of at least one surface of the contact portion with an object based on a first image in which a first surface of the contact portion opposite to the at least one surface is captured by the camera, and identifying at least one of a strength of a force applied to the contact portion and a direction of the force applied to the contact portion based on the at least one displacement, Wherein, at least one of the form and the color of a first mark among the plurality of marks is different from that of a second mark among the plurality of marks adjacent to the first mark.

2. The electronic device according to claim 1, wherein: Each of the plurality of marks includes at least one first pattern of a first form and at least one second pattern of a second form disposed at one side of the at least one first pattern, and Wherein, the at least one processor is further configured to identify at least one of the intensity of the force and the direction of the force based on the at least one first pattern and the at least one second pattern.

3. The electronic device according to claim 2, wherein: The first form comprises a circular pattern, and Wherein, the second form comprises a linear pattern.

4. The electronic device according to claim 3, wherein: The at least one processor is further configured to identify a rotational force parallel to the contact portion based on the circular pattern according to a displacement of the linear pattern.

5. The electronic device according to claim 2, wherein: The at least one second pattern includes a plurality of second patterns, and Wherein, adjacent patterns in the plurality of second patterns vary in color.

6. The electronic device according to claim 1, wherein: The first mark includes a circular first pattern and a linear first pattern oriented toward a first direction from the circular first pattern, wherein the second mark is adjacent to the first mark in a direction opposite to the first direction, and the second mark includes a circular second pattern and a linear second pattern oriented from the circular second pattern toward a second direction perpendicular to the first direction, Wherein, the plurality of marks further include a third mark and a fourth mark, wherein the third mark is adjacent to the second mark in a direction opposite to the second direction, and the third mark includes a circular third pattern and a linear third pattern oriented from the circular third pattern toward a third direction perpendicular to the second direction, and The fourth mark is adjacent to the third mark in a direction opposite to the third direction, and the fourth mark includes a circular fourth pattern and a linear fourth pattern oriented from the circular fourth pattern toward a fourth direction perpendicular to the third direction.

7. The electronic device according to claim 1, further comprising: a lens array disposed at an opposite surface of the contact portion, the lens array comprising a plurality of lenses corresponding to the plurality of marks, Wherein, the at least one processor is further configured to: identifying a size change of each of the plurality of marks through each of the plurality of lenses, and A force in a direction perpendicular to the contact portion is identified based on the identified dimensional change.

8. The electronic device according to claim 1, wherein: Each of the plurality of tags comprises: the first pattern of the first form, or A first pattern of the first form and at least one second pattern of the second form adjacent to the first pattern of the first form.

9. The electronic device according to claim 1, further comprising: a memory configured to store a second image in which a surface of the contact portion is captured by the camera before the contact portion contacts the object, Wherein, the at least one processor is further configured to identify the at least one displacement based on the first image and the second image.

10. The electronic device according to claim 1, wherein: The contact portion comprises: Clear gel; and forming a coating layer at upper ends of the plurality of markings, and Wherein, the plurality of marks are arranged on a surface of the transparent gel.

11. A method for controlling an electronic device, the method comprising: bringing at least one surface of a contact portion of the electronic device into contact with an object; identifying at least one displacement of a plurality of markings disposed in the contact portion based on a first image in which a first surface of the contact portion opposite the at least one surface is captured by a camera of the electronic device; as well as identifying at least one of a strength of a force applied to the contact portion and a direction of the force applied to the contact portion based on the identified at least one displacement, Wherein, at least one of the form and the color of a first mark among the plurality of marks is different from that of a second mark among the plurality of marks adjacent to the first mark.

12. The control method according to claim 11, wherein: Each of the plurality of tags comprises: at least one first pattern of a first form, and at least one second pattern of the second form is disposed at one side of the at least one first pattern of the first form, and wherein at least one of the intensity of the force and the direction of the force is identified based on the at least one first pattern of the first form and the at least one second pattern of the second form.

13. The control method according to claim 12, wherein: The at least one first pattern of the first form comprises a circular pattern, and Wherein, the at least one second pattern of the second form comprises a linear pattern.

14. The control method according to claim 13, wherein: Identifying at least one of the magnitude of the force and the direction of the force comprises: Based on the circular pattern, a rotational force parallel to the contact portion is identified according to a displacement of the linear pattern.

15. The control method according to claim 12, wherein: The at least one second pattern includes a plurality of second patterns, and Wherein, adjacent patterns in the plurality of second patterns vary in color.

Images