Devices with touchscreen displays and their control methods

Abstract

This invention provides a device with a touchscreen display that performs desired processing on objects within an image through simple operation, as well as a control method and program thereof. The object, including at least a portion of the contact area, is determined relative to the object area occupied by the image based on the contact area of ​​the pinch action. When transitioning from a pinch action to an action other than a pinch action, the image displayed on the touchscreen display is subjected to actions related to the object area.

Description

Technical Field

[0001] This invention relates to a device with a touch screen display and a control method thereof, and more particularly to a technique for performing desired processing on images through intuitive actions. Background Technology

[0002] In recent years, touchscreen displays, which feature touch panels on the display for detecting touch-based input, have become widespread and are used in mobile devices such as smartphones.

[0003] In devices that use touchscreen displays, changes are made to the images displayed on the display by the user through input from the touch panel.

[0004] For example, Patent Document 1 discloses a technique that involves picking up and saving any object corresponding to a first multi-touch input from the displayed objects, and calling the saved object in accordance with a second multi-touch input.

[0005] Furthermore, Patent Document 2 discloses a technique for correcting the position of a selected region within an image by dragging the selected region.

[0006] In addition, non-patent document 1 discloses a technique for automatically identifying and cutting object regions.

[0007] Previous technical documents

[0008] Patent documents

[0009] Patent Document 1: Japanese Patent No. 5731979

[0010] Patent Document 2: US Patent No. 9,575,641

[0011] Non-patent literature

[0012] Non-patent literature 1: Paint 3D (Microsoft Corporation) Magic Select <URL:https: / / www.microsoft.com / ja-jp / p / %E3%83%9A%E3%82%A4%E3%83%B3%E3%83%88-3d / 9nblggh5fv99?activetab=pivot%3aoverviewtab> Summary of the Invention

[0013] The technical problem to be solved by the invention

[0014] In the technology described in Patent Document 1, the objects that can be picked up are limited to those that have been identified as objects in the system, and objects that have not been identified as objects cannot be picked up.

[0015] Furthermore, in the technology described in Patent Document 2, the user has to go through multiple steps to select the desired area, making it impossible to achieve a simple operation.

[0016] Furthermore, in the technology of non-patent document 1, it is necessary to specify the rectangular range used to determine the seed point, which makes it impossible to perform a simple operation.

[0017] The present invention was made in view of this situation, and its object is to provide a device with a touch screen display and a control method and program thereof for performing desired processing on objects within an image with simple operation.

[0018] means for solving technical problems

[0019] To achieve the above objectives, one embodiment of a device with a touchscreen display includes: a touchscreen display; an image display unit for displaying an image on the touchscreen display; a pinch action recognition unit for recognizing a user's pinch action on the touchscreen display; a contact area recognition unit for recognizing a contact area that comes into contact with the touchscreen display through the pinch action; an object area determination unit for determining, based on the contact area recognized by the contact area recognition unit, the object area occupied by an object, including at least a portion of the contact area, relative to the image; an action transition recognition unit for recognizing a transition from a pinch action to an action other than a pinch action; and an output behavior unit for performing an action related to the object area on the image displayed on the touchscreen display when a transition to an action other than a pinch action is recognized.

[0020] According to this method, the object area occupied by the image is determined relative to the contact area of ​​the pinch action, which includes at least a portion of the contact area. When transitioning from a pinch action to an action other than a pinch action, the image displayed on the touch screen is subjected to actions related to the object area. Therefore, the desired processing of the object within the image can be performed with simple operation.

[0021] Preferably, the action is one of the following: a pinch-open action other than a pinch-locking action; an action to set the contact with the touch screen display to non-contact; a dragging action; and an action to apply greater pressure to the touch screen display than the pinch-locking action. Thus, starting from the pinch-locking action, the desired processing of the object within the image can be performed in a series of actions.

[0022] The preferred actions related to the object region are at least one of the following: copying or moving the object region, recording the object region in memory, changing the orientation of the object region, and scaling the object region. This allows for appropriate processing of objects within the image.

[0023] The image is preferably a photographic image obtained by an imaging element. According to this method, even for images where the object region has not been extracted, the desired processing of the objects within the image can be performed.

[0024] To achieve the above objectives, one method for controlling a device with a touchscreen display includes: an image display step for displaying an image on the touchscreen display; a pinch action recognition step for recognizing a user's pinch action on the touchscreen display; a contact area recognition step for recognizing a contact area that comes into contact with the touchscreen display through the pinch action; an object area determination step for determining, based on the contact area recognized by the contact area recognition step, an object area occupied by an object, including at least a portion of the contact area, relative to the image; an action transition recognition step for recognizing a transition from the pinch action to an action other than the pinch action; and an output behavior step for performing an action related to the object area on the image displayed on the touchscreen display when a transition to an action other than the pinch action is recognized.

[0025] According to this method, desired processing of objects within an image can be performed with simple operation. The program for controlling the computer to execute the aforementioned control method for a device with a touchscreen display is also included in this method.

[0026] Invention Effects

[0027] According to the present invention, desired processing of objects within an image can be performed with simple operations. Attached Figure Description

[0028] Figure 1 This is a front-view stereoscopic image of a tablet computer.

[0029] Figure 2 This is a rear-view stereoscopic view of a tablet computer.

[0030] Figure 3 It is a block diagram representing the system architecture of a tablet computer.

[0031] Figure 4 This is a block diagram showing the system structure of the touchscreen display control unit.

[0032] Figure 5 It is a diagram used to illustrate the contact area.

[0033] Figure 6 This is a flowchart illustrating the control method of a tablet computer.

[0034] Figure 7 This is a diagram showing an example of an image displayed on a touchscreen display.

[0035] Figure 8 This is a diagram used to illustrate the pinching action on a touchscreen display.

[0036] Figure 9 It is a diagram used to illustrate the transition from pinching action to actions other than pinching action and the behavior related to the object area.

[0037] Figure 10 It is a diagram that represents the image displayed on the touchscreen display at the point in time when a series of actions are completed.

[0038] Figure 11 It is a diagram used to illustrate copying objects between different images.

[0039] Figure 12 It is a diagram that represents the image displayed on the touchscreen display at the point in time when a series of actions are completed.

[0040] Figure 13 It is a diagram used to illustrate the copying of objects.

[0041] Figure 14 It is a diagram that represents the image displayed on the touchscreen display at the point in time when a series of actions are completed.

[0042] Figure 15 It is a diagram used to illustrate the rotation of an object.

[0043] Figure 16 A diagram showing the image displayed on a touchscreen display at the point in time when a series of actions are completed.

[0044] Figure 17 It is a magnified diagram used to illustrate an object.

[0045] Figure 18 It is a diagram that represents the image displayed on the touchscreen display at the point in time when a series of actions are completed. Detailed Implementation

[0046] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0047] Devices with touchscreen displays

[0048] Figure 1 This is a front-view perspective view of a tablet computer terminal 10 (hereinafter referred to as tablet computer 10; PC is short for Personal Computer), which is an example of a device with a touch screen display according to this embodiment. Figure 2 It is a rear-view stereoscopic view. For example... Figure 1 and Figure 2 As shown, the tablet computer 10 has a flat frame 12 with a rectangular outline. The frame 12 is configured to include a touch screen display 14, operation buttons 16, a speaker 18, a built-in camera 20, and an external connection terminal 24.

[0049] Figure 3 This is a block diagram representing the system architecture of the tablet computer 10. For example... Figure 3 As shown, the tablet computer 10 includes a CPU (Central Processing Unit) 30 that controls the overall operation of the tablet computer 10. The CPU 30 is connected via a system bus 32 to a main memory 34, a non-volatile memory 36, a mobile communication unit 38, a wireless LAN (Local Area Network) communication unit 40, a short-range wireless communication unit 42, a wired communication unit 44, a key input unit 50, a voice processing unit 52, a camera control unit 54, and a touchscreen display control unit 60.

[0050] CPU 30 reads the OS (Operating System), applications running on the OS, and standard data stored in non-volatile memory 36, and unfolds them in main memory 34 to execute action programs, thereby acting as a control unit that controls the overall operation of the computer.

[0051] The main memory 34 is, for example, composed of RAM (Random Access Memory) and functions as the working memory of the CPU 30.

[0052] The non-volatile memory 36 is, for example, composed of flash memory EEPROM (Electrically Erasable Programmable Read Only Memory) and stores the OS, applications, and various standard data. Furthermore, the non-volatile memory 36 functions as the storage unit of the tablet computer 10 and stores various types of data.

[0053] The mobile communication unit 38 transmits and receives data with the nearest radio base station (not shown) via antenna 38A through a 4th generation mobile communication system (4G) or a 5th generation mobile communication system (5G).

[0054] The wireless LAN communication unit 40, for example, communicates wirelessly with a wireless LAN access point or an external device capable of wireless LAN communication via antenna 40A, according to wireless LAN communication standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.11a / b / g / n.

[0055] The short-range wireless communication unit 42 communicates with other Bluetooth (registered trademark) standard devices, for example, within a range of level 2 (approximately 10m radius), via antenna 42A.

[0056] The wired communication unit 44 communicates with an external device connected by a cable (not shown) via an external connection terminal 24 in accordance with the prescribed communication standards.

[0057] The key input section 50 is composed of a drive circuit that drives multiple operation buttons 16.

[0058] The voice processing unit 52 converts the digital voice data transmitted via the system bus 32 into analog data. The analog voice signal is then played out from the speaker 18.

[0059] The camera control unit 54 digitizes the analog image signal output from the built-in camera 20, which is equipped with an imaging lens 20A and an imaging element 20B, and performs the required signal processing before outputting the signal.

[0060] The touch screen display 14 includes: a display section 14A, a color LCD (Liquid Crystal Display) panel for displaying images, etc.; a touch panel section 14B, a transparent electrode disposed on the front surface of the color LCD panel and receiving touch input, etc.; and a pressure sensor 14C for detecting the pressure of the touch panel section 14B at the contact position.

[0061] The touch panel 14B is, for example, a capacitive touch panel, which is made of glass or plastic and has: a substrate body that is light-transmitting; position detection electrodes that are formed in a planar shape on the substrate body and are light-transmitting; and an insulating layer disposed on the position detection electrodes. The touch panel 14B is disposed on the front surface of the display 14A and generates and outputs two-dimensional position coordinate information corresponding to the user's touch operation.

[0062] The touch screen display control unit 60 controls the touch screen display 14. The touch screen display control unit 60 establishes a corresponding association between the coordinates of the image displayed on the display unit 14A and the coordinates of the touch operation on the touch panel unit 14B.

[0063] Figure 4This is a block diagram showing the system structure of the touchscreen display control unit 60. For example... Figure 4 As shown, the touch screen display control unit 60 includes a display control unit 62, an input recognition unit 68, and an image processing unit 76.

[0064] The display control unit 62 displays various information on the display unit 14A according to instructions from the CPU 30. The display control unit 62 includes an image display unit 64.

[0065] The image display unit 64 displays various images on the display unit 14A according to instructions from the CPU 30. For example, the image display unit 64 displays video images captured by the built-in camera 20 on the display unit 14A. Furthermore, the image display unit 64 displays images read from the non-volatile memory 36 on the display unit 14A. In addition, the image display unit 64 displays images acquired via the mobile communication unit 38, the wireless LAN communication unit 40, the short-range wireless communication unit 42, and the wired communication unit 44 on the display unit 14A.

[0066] The input recognition unit 68 acquires two-dimensional position coordinate information corresponding to the user's touch operation from the touch panel unit 14B and recognizes the user's input action. The input recognition unit 68 includes a pinch action recognition unit 70, a contact area recognition unit 72, and an action transition recognition unit 74.

[0067] The pinch action recognition unit 70 recognizes the pinch action of the user on the touch panel 14B. The pinch action is, for example, when a finger is in contact with two parts of the touch panel 14B, the contact points are brought closer together by reducing the distance between the two points.

[0068] The contact area recognition unit 72 recognizes the contact area that comes into contact with the touch panel unit 14B during a pinching action, for example, when a finger contacts the touch panel unit 14B.

[0069] Figure 5 This diagram illustrates the contact area. Starting from a position where the right index finger touches the touchscreen display 14 (starting point PS1) and the right thumb touches the touchscreen display 14 (starting point PS2), the right index finger moves to an endpoint PE1 and the right thumb moves to an endpoint PE2, thus performing a pinching action. In this case, the contact area refers to area A1 where the right index finger touches the touchscreen display 14 from starting point PS1 to endpoint PE1, and area A2 where the right thumb touches the touchscreen display 14 from starting point PS2 to endpoint PE2. Area A1 includes starting point PS1 and endpoint PE1, and area A2 includes starting point PS2 and endpoint PE2. Thus, the contact area refers to the area that comes into contact with the touchscreen display 14 during the pinching action.

[0070] Return to Figure 4The motion transition recognition unit 74 recognizes transitions from a pinching motion to a motion other than a pinching motion. Motions other than a pinching motion include, for example, pinching open, dragging, rotating, and non-contact motions.

[0071] A pinching action is, for example, an action that moves the contact points away by increasing the distance between the two points while the finger is in contact with two parts of the touch panel 14B. A dragging action is an action that moves the contact point of the touch panel 14B, including an action that moves the contact point while maintaining a constant distance between the two points from a pinched state (pinched state).

[0072] A rotational action is an action that rotates (twists) the finger from a pinched state to change the relative position of the two contact points. A non-contact action is an action that sets the contact with the touch panel 14B to non-contact.

[0073] Furthermore, in addition to the pinching action, the action can be a pushing action, which makes the pressing pressure on the touch panel 14B relatively greater than the pressing pressure in the pinching action.

[0074] The image processing unit 76 processes the input image. The image processing unit 76 includes an object region determination unit 78 and an output behavior unit 80.

[0075] The object area determination unit 78 determines the object area occupied by an object that includes at least a portion of the contact area relative to the image displayed on the display unit 14A based on the contact area identified by the contact area recognition unit 72.

[0076] When transitioning from a pinching action to an action other than a pinching action, the output behavior unit 80 performs actions related to the object area on the image displayed on the display unit 14A.

[0077] In the following description, the display of the display unit 14A and the touch operation of the touch panel unit 14B are simply described as the display of the touch screen display 14 and the operation of the touch screen display 14.

[0078] Control methods for devices with touch screen displays

[0079] The tablet computer 10 can perform desired processing on objects within an image through user operation. Figure 6 This is a flowchart illustrating the control method of the tablet computer 10. For example... Figure 6 As shown, the control method of the tablet computer 10 includes an image display process (step S1), a pinch action recognition process (step S2), a contact area recognition process (step S3), an object area determination process (step S4), an action transition recognition process (step S5), and an output behavior process (step S6).

[0080] In step S1, the image display unit 64 displays an image on the touchscreen display 14. The user can display a desired image on the touchscreen display 14 by performing a touch operation. In this embodiment, the image display unit 64 displays, for example, an image read from the non-volatile memory 36. The image displayed here is any one of the following: a photographic image captured using an imaging element, an image not created by a computer, or an unsegmented image.

[0081] In step S2, the pinch action recognition unit 70 recognizes a pinch action performed by the user on the touchscreen display 14. The user performs a pinch action to make at least a portion of the object to be processed in the image displayed on the touchscreen display 14 become the contact area. If the pinch action recognition unit 70 recognizes a pinch action, the process proceeds to step S3.

[0082] In step S3, the contact area recognition unit 72 recognizes the contact area that comes into contact with the touch screen display 14 through the pinching action.

[0083] In step S4, the object area determination unit 78 determines the object area selected by the user. Specifically, based on the contact area identified in step S3, it determines the object area occupied by the object, which includes at least a portion of the contact area, relative to the image displayed on the touch screen display 14.

[0084] Therefore, the object region determination unit 78 first performs segmentation processing on the image displayed on the touch screen display 14 and extracts multiple object regions from the image. As region extraction processing, methods such as binarization (multi-valuedness) based on pixel value distribution, image segmentation based on the connection between adjacent pixels, region growing that extracts while expanding pixel regions that meet the conditions, and region extraction using convolutional neural networks can be considered, but it is not limited to these methods, and other known methods can also be used.

[0085] Next, the object region determination unit 78 compares the position coordinate information of the contact area with the position coordinate information of multiple object regions, and searches for an object region that includes at least a portion of the contact area.

[0086] The object region determination unit 78 uses the contact region as a seed point (foreground seed, foreground label) to perform region extraction processing based on graphic cutting or region expansion methods, thereby searching for object regions that include at least a portion of the contact region.

[0087] When there is only one object area including at least a portion of the contact area, the object area determining unit 78 determines that object area as the user-selected object area. Furthermore, when there are multiple object areas including at least a portion of the contact area, the object area including the most contact area can be determined as the user-selected object area, or the object area including two contact areas (…) can be determined as… Figure 5 The object areas of regions A1 and A2 shown are determined as the object areas selected by the user.

[0088] In step S5, the motion transition recognition unit 74 recognizes a transition from a pinch action to an action other than a pinch action. The user transitions their touch operation on the touchscreen display 14 from a pinch action to an action other than a pinch action, thereby enabling the user to perform desired processing on the selected object. If the motion transition recognition unit 74 recognizes a transition to an action other than a pinch action, the process proceeds to step S6.

[0089] In step S6, the output behavior unit 80 performs behavior on the image displayed on the touch screen display 14 related to the object area determined in step S4, and performs behavior assigned to actions other than the pinching action based on step S5.

[0090] Thus, according to this embodiment, desired actions related to the object region can be performed on objects within an image with simple user operations. Hereinafter, "actions other than pinching" and "actions related to the object region" will be described in detail.

[0091] [Copy / move to the same image]

[0092] Figure 7 This diagram illustrates an example of image G1 displayed on the touchscreen display 14. Image G1 is a photographic image of insects and trees. Image G1 is an image without intra-image segmentation.

[0093] Figure 8 This diagram illustrates the pinching action on the touchscreen display 14. (Example) Figure 8 As shown, the pinching motion is performed, for example, using two fingers. Figure 8 In the example, the user makes a pinch gesture near the area in image G1 where the insect is captured. More specifically, the user makes the pinch gesture such that at least a portion of the insect becomes the contact area.

[0094] For this pinching action, the object region determination unit 78 performs segmentation processing on the image G1, extracting the object region of the insect and the object region of the tree from the image G1. Furthermore, the object region determination unit 78 determines the object region occupied by the object that includes at least a portion of the contact area from the object regions of the insect and the tree. In this example, the object region determination unit 78 determines the object region OB1A of the insect.

[0095] Figure 9 This is a diagram used to illustrate the transition from pinching actions to actions other than pinching, and the behavior related to the object area. Figure 9 In the example, the user transitions from a pinching motion, keeping their fingers in a pinching state, to a dragging motion.

[0096] As an output behavior corresponding to the transition to the dragging action, the output behavior unit 80 displays a copy area of ​​the insect object area OB1A and an object area OB1B as a transparent copy area at the finger contact position of the touch screen display 14.

[0097] In addition, Figure 9 In the example, the user transitions from a dragging motion to a pinching motion at the location where they want to copy the insect.

[0098] As an output behavior corresponding to the transition to the pinching action, the output behavior unit 80 copies the copied area of ​​the insect's object area OB1A, i.e., the object area OB1C, to the position where the pinching action has been performed. The position where the pinching action has been performed is, for example, the center position of the contact position between the two points at the starting point of the pinching action. Furthermore, the position where the pinching action has been performed can be the center position of the contact position between the two points at the ending point of the pinching action.

[0099] Figure 10 This is a diagram showing image G2 displayed on the touchscreen display 14 at the point in time when the series of actions ends. Image G2 becomes... Figure 7 The image G1 shown was copied and an image of an insect was added.

[0100] As described above, according to this embodiment, objects within an image can be copied to different locations within the same image with simple user operations. Specifically, the user performs a pinch action by touching at least a portion of the object to be copied, thereby extracting the desired object region. Furthermore, by simply performing a dragging action and a pinch-open action consecutively from the pinch action, the object can be copied to the desired location within the same image.

[0101] Previously, extracting the object region required many steps, which was complex for the user. For example, in software not covered in Patent Document 1, at least three separate actions were required: the user setting a seed point, the user initiating the object region determination step, and the user selecting the determined object region. However, in this embodiment, the pinching action can be replaced by a single action instead of the three actions mentioned above in the prior art, which helps to visualize the required user action.

[0102] Thus, in this embodiment, users are not required to remove their fingers from the touchscreen display to perform unnecessary actions such as button operations. Even for images where the object area has not yet been extracted, users can copy the object through intuitive operations.

[0103] Furthermore, while this embodiment describes an example of copying an object within an image to different locations within the same image, it is also possible to move an object within an image to different locations within the same image. In this case, the original position of the moved object region can be interpolated using the surrounding pixel values.

[0104] [Copying / moving between different images]

[0105] Objects within an image can also be copied or moved to different images.

[0106] Figure 11 This is a diagram used to illustrate copying objects between different images. Figure 11 In the example, the image display unit 64 displays images G1 and G3 side by side on the touch screen display 14. Image G1 is a camera image of an insect and a tree, and image G3 is an image of a tree.

[0107] like Figure 11 As shown, if the user performs a pinching action, similar to the case of copying within the same image, the object region determination unit 78 determines the object region OB1A of the insect as the object region occupied by the object including at least a portion of the contact area.

[0108] And, as Figure 11 As shown, if the user transitions from a pinching action to a dragging action, as an output behavior corresponding to the transition to the dragging action, the output behavior unit 80 displays a copy area of ​​the insect object area OB1A and a transparent copy area, namely the object area OB1D, at the touch position of the finger on the touch screen display 14.

[0109] The dragging action here is performed from the position where image G1 is displayed to the position where image G3 is displayed. Furthermore, as... Figure 11 As shown, the user transitions to a pinching motion at position G3 of the image of the object area where the insect is to be copied.

[0110] As an output behavior corresponding to the transition to the pinching action, the output behavior unit 80 copies the copy area of ​​the insect's object area OB1A, namely the object area OB1E, to the position where the pinching action has been performed.

[0111] Figure 12 This is a diagram showing the touchscreen display 14 at the point in time when the series of actions ends. (Example) Figure 12 As shown, images G1 and G4 are displayed side-by-side on the touchscreen display 14. Image G4 becomes... Figure 11 The image G3 shown was copied and the image of the insect was added.

[0112] As described above, it is possible to copy objects within an image to desired locations in different images with simple user operations. Objects can be moved rather than copied.

[0113] This concludes the example of copying the insect object region in image G1. However, it is also possible to select the tree object region by pinching and then copy or move the tree object region.

[0114] Figure 13 It is used to explain and Figure 11 A diagram illustrating the replication of different objects. Figure 13 In the example, the image display unit 64 displays image G1 and image G5 side by side on the touch screen display 14. Image G1 is a camera image of an insect and a tree, and image G5 is an image of an animal.

[0115] like Figure 13 As shown, the user first performs a pinching motion. Figure 13 In the example, the user pinches the image near the area where the tree is captured in image G1. More specifically, the user pinches the image so that at least a portion of the tree becomes the contact area.

[0116] The object region determination unit 78 determines the object region OB2A of the tree as the object region occupied by the object that includes at least a part of the contact area.

[0117] And, as Figure 13 As shown, the user transitions from a pinch action to a drag action. As an output behavior corresponding to the transition to this drag action, the output behavior unit 80 can also display a copy of the tree object area OB2A at the touch position of the finger on the touch screen display 14, and the copy area is transparent in color.

[0118] Furthermore, the user transitions to a pinch action at the location of image G5, the object area of ​​the tree to be copied. As an output behavior corresponding to the transition to this drag and pinch action, the output behavior unit 80 copies the copy area of ​​the tree object area OB2A, i.e., object area OB2B, to the location where the pinch action has been performed.

[0119] Figure 14 This is a diagram showing the touchscreen display 14 at the point in time when the series of actions has ended. (Example) Figure 14 As shown, images G1 and G6 are displayed side-by-side on the touchscreen display 14. Image G6 becomes... Figure 13 The image shown, G5, was copied and the image of the trees was added.

[0120] Here, trees and insects are copied together, but it is also possible to distinguish between insect objects and tree objects and copy only the tree objects.

[0121] In this way, by varying the contact area of ​​the pinching action, users can select the desired object region from among multiple objects included in the image.

[0122] It is not limited to different images; of course, the object area of ​​the tree can also be copied or moved within the same image.

[0123] [Rotation]

[0124] The tablet computer 10 can change the orientation of the desired object. Figure 15 It is a diagram used to illustrate the rotation of an object.

[0125] If the user Figure 7 The image G1 shown transitions from a pinching motion to a dragging motion, as follows: Figure 15 As shown, as an output behavior corresponding to the transition to the dragging action, the output behavior unit 80 displays a copy area of ​​the extracted object area OB1A and a transparent copy area, namely the object area OB1B, at the touch position of the finger on the touch screen display 14.

[0126] Then, the user transitions from the pinch state to a rotation action. As an output behavior corresponding to this transition to the rotation action, the output behavior unit 80 changes the orientation of the display object area OB1B based on the rotation amount of the relative positional relationship between the contact positions of the two points. Thus, by performing a rotation action, the user can rotate (change the orientation) the extracted object.

[0127] Then, if the user performs a pinching action, the output behavior unit 80 will copy the object area OB1B, which changes orientation according to the transition to the pinching action, to the position where the pinching action has been performed.

[0128] Figure 16This is a diagram showing image G7 displayed on the touchscreen display 14 at the point in time when the series of actions ends. Image G7 becomes... Figure 7 The image G1 shown was copied and an image of the insect with its orientation changed was added.

[0129] Here, after the pinching action, the sequence transitions to dragging, rotating, and pinching-opening actions. However, even if the sequence is rotated, dragging, and pinching-opening, the same result can be obtained. Furthermore, the dragging and pinching actions are not necessary during the rotation action. For example, performing a rotation action after the pinching action, and then transitioning to a non-contact action, can change the orientation of the defined object area from its original position, or it can place the object with the changed orientation back into its original position.

[0130] [Zoom in / out]

[0131] The tablet 10 can also zoom in or out on desired objects. Figure 17 It is a magnified diagram used to illustrate an object.

[0132] User Figure 7 The image G1 shown is a pinching motion. If the finger remains in contact with the touchscreen display 14 and transitions to a pinching motion, then as shown... Figure 17 As shown, as an output behavior corresponding to the transition to the pinching action, the output behavior unit 80 configures the object region OB1F of the extracted object region OB1A on the touch screen display 14, which is magnified.

[0133] In this case, since the pinching action is assigned to the amplified output behavior, the configured output behavior is performed as a non-contact action. However, the assignment of actions and behaviors is not limited to this.

[0134] The magnification can be adjusted based on the size of the contact area of ​​the pinching action (the distance between the start and end points of the pinching action).

[0135] Figure 18 This is a diagram representing the image G8 displayed on the touchscreen display 14 at the point in time when the series of actions ends. Image G8 becomes... Figure 7 The image G1 shown is a magnified image of an insect. This allows the user to change the size of the desired object.

[0136] Furthermore, when the object size is reduced, it is performed through a presupposition action. That is, the user pinches the image displayed on the touch screen display 14, and then, if the finger remains in contact with the touch screen display 14 while pinching and transitions to a presupposition action, as an output behavior corresponding to the transition to the presupposition action, the output behavior unit 8() displays the object area with the reduced size of the extracted object area on the touch screen display 14.

[0137] In this case, the shrinkage rate can be changed by an estimated number of times or an estimated time.

[0138] 〔save〕

[0139] The tablet computer 10 can also save desired objects in non-volatile memory 36.

[0140] For example, after a pinching motion, the image of the extracted object region is saved by transitioning to a non-contact motion. The motion used for saving is not limited to a non-contact motion; it can also be a pinching motion, a deduction motion, etc.

[0141] Furthermore, by combining this with the examples above, such as "Extract → Rotate → Save" or "Extract → Zoom In → Save", images of the extracted object regions that have been edited can be saved.

[0142] It can properly read the image of the object area stored in the non-volatile memory 36 and display it on the touch screen display 14.

[0143] <Other>

[0144] Assigning which "action other than pinch" to which "object-related behavior" is not limited to the examples above; it can be determined appropriately. The important thing is to be able to specify the object using a simple user action such as a pinch.

[0145] Here, a tablet computer terminal is described as an example of a device with a touch screen display, but it can also be applied to smartphones, mobile phones or mobile communication terminals such as PHS (Personal Handyphone System), and laptop computers.

[0146] The control method for the aforementioned device with a touch screen display is configured as a program for enabling the computer to perform various processes, or it can be configured as a non-temporary recording medium such as a CD-ROM (Compact Disk-Read Only Memory) storing the program.

[0147] In the embodiments described herein, the hardware structure of the processing unit that performs various processes of the touch screen display control unit 60 is as follows: various processors. These processors include: general-purpose processors that execute software (programs) and function as various processing units, such as CPUs (Central Processing Units); processors specifically designed for image processing, such as GPUs (Graphics Processing Units); processors whose circuit structure can be changed after manufacturing, such as FPGAs (Field Programmable Gate Arrays); processors with circuit structures specifically designed for performing specific processes, such as Programmable Logic Devices (PLDs); and ASICs (Application Specific Integrated Circuits), i.e., dedicated circuits.

[0148] A processing unit can be constructed from one of these various processors, or from two or more processors of the same or different types (e.g., multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Furthermore, a single processor can constitute multiple processing units. As examples of a single processor constituting multiple processing units, firstly, in computers such as servers and clients, a single processor is constructed from a combination of one or more CPUs and software, functioning as multiple processing units. Secondly, in systems-on-chips (SoCs), a processor is used to implement the overall system functionality, including multiple processing units, using a single integrated circuit (IC) chip. Thus, various processing units are constructed using one or more different processors in the hardware structure.

[0149] Moreover, more specifically, the hardware architecture of these various processors is a circuit that combines circuit elements such as semiconductor components.

[0150] The scope of the present invention is not limited to the scope described in the above embodiments. The structures and other features of each embodiment can be appropriately combined among the embodiments without departing from the spirit of the present invention.

[0151] Symbol Explanation

[0152] 10-Tablet PC terminal (Tablet PC), 12-Frame, 14-Touchscreen display, 14A-Display unit, 14B-Touch panel, 14C-Pressure sensor, 16-Operation buttons, 18-Speaker, 20-Built-in camera, 20A-Imaging lens, 20B-Imaging element, 24-External connection terminal, 32-System bus, 34-Main memory, 36-Non-volatile memory, 38-Mobile communication unit, 38A-Antenna, 40-Wireless LAN communication unit, 40A-Antenna, 4 2-Near-field wireless communication unit, 42A-Antenna, 44-Wired communication unit, 50-Key input unit, 52-Voice processing unit, 54-Camera control unit, 60-Touchscreen display control unit, 62-Display control unit, 64-Image display unit, 68-Input recognition unit, 70-Pinch action recognition unit, 72-Contact area recognition unit, 74-Action transition recognition unit, 76-Image processing unit, 78-Object area determination unit, 80-Output behavior unit, S1 to S6-Steps of the control method for the tablet computer.

Claims

1. A device with a touch screen display, comprising: Touchscreen display; An image display unit displays an image on the touchscreen display. The pinch action recognition unit recognizes the user's pinch action on the touch screen display; A contact area recognition unit identifies the contact area that comes into contact with the touch screen display through the pinching action; The object region determination unit extracts multiple object regions from the image by segmenting the region in the image that includes the contact region identified by the contact region recognition unit in response to the pinching action, and determines an object region relative to the image from the multiple object regions that includes at least a portion of the contact region. The action transition recognition unit recognizes the transition from the pinching action to an action other than the pinching action; and The output behavior unit, upon recognizing a transition to an action other than the pinching action, performs actions related to the determined object area on the image displayed on the touchscreen screen. The image is an image in which at least the region of the first object is included within the region of the second object. When at least a portion of the contact area is included in the area of ​​the first object within the area of ​​the second object, the object area determination unit determines the area of ​​the first object as the determined object area.

2. The device with a touch screen display according to claim 1, wherein, Actions other than the pinching action are pinching open actions.

3. The device with a touchscreen display according to claim 1 or 2, wherein, Actions other than the pinching action are actions that set the contact with the touch screen display to non-contact.

4. The device with a touchscreen display according to claim 1 or 2, wherein, In addition to the pinching motion, the other motion is the dragging motion.

5. The device with a touchscreen display according to claim 1 or 2, wherein, Actions other than the pinching action are actions that apply greater pressure to the touchscreen display than the pinching action.

6. The device with a touchscreen display according to claim 1 or 2, wherein, The behavior related to the object region is the behavior of copying or moving the object region.

7. The device with a touchscreen display according to claim 1 or 2, wherein, The action related to the object region is the action of recording the object region in memory.

8. The device with a touchscreen display according to claim 1 or 2, wherein, The behavior related to the object region is the behavior of changing the orientation of the object region.

9. The device with a touchscreen display according to claim 1 or 2, wherein, The behavior related to the object region is the behavior of enlarging or shrinking the object region.

10. The device with a touchscreen display according to claim 1 or 2, wherein, The image is a photographic image obtained by an imaging element.

11. The device with a touchscreen display according to claim 1 or 2, wherein, The first object is the object configured on the second object.

12. The device with a touchscreen display according to claim 1 or 2, wherein, When the contact area is included in the area of ​​the second object, the object area determination unit determines the area of ​​the first object and the second object as the object area.

13. A control method for a device with a touch screen display, comprising: The image display process displays the image on the touchscreen display. The pinching action recognition process identifies the user's pinching action on the touch screen display; The contact area recognition process identifies the contact area that comes into contact with the touch screen display through the pinching action; The object region determination process involves segmenting the region in the image that includes the contact region identified by the contact region recognition process in response to the pinching action, extracting multiple object regions from the image, and determining an object region relative to the image from the multiple object regions that includes at least a portion of the contact region. The action transition recognition process identifies the transition from the pinching action to an action other than the pinching action; and The output behavior process, upon recognizing a transition to an action other than the pinching action, performs an action related to the determined object area on the image displayed on the touchscreen display. The image is an image in which at least the region of the first object is included within the region of the second object. In the object region determination process, if at least a portion of the contact area is included in the area of ​​the first object within the area of ​​the second object, the area of ​​the first object is determined as the determined object region.

14. A recording medium, being non-transitory and computer-readable, which, when an instruction stored in the recording medium is read by a computer, causes the computer to execute the control method of the device with a touch screen display as described in claim 13.

Images