Control device and control method

Abstract

To provide a technology which hardly causes a user to have a sense of incongruity, in the technology of processing an image depending on the movement of a part of the user's body.SOLUTION: In a controller 12, a reference point setting unit 30: acquires space coordinates of each of a first part, a second part, and a third part of a user's body located in front of a display 14; identifies a midpoint between the space coordinates of the first part and the space coordinates of the second part; and sets a point on the display 14 in which virtual straight lines passing through the identified midpoint and the space coordinates of the third part cross as a reference point. A function execution unit 32 executes processing according to the movement of the first part and the second part with respect to an image displayed on the display 14, based on the set reference point.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a control device and a control method. [Background technology]

[0002] BACKGROUND ART A virtual touch device for remotely operating an electronic device having a display surface is known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2013 / 0321347 Summary of the Invention [Problem to be solved by the invention]

[0004] The inventor recognized that in a technology that detects when a user in front of a display moves two fingertips in the air and enlarges, reduces, rotates, etc. an image in accordance with the detected movement, if the enlargement, etc. is centered around a position that the user does not intend, this may cause discomfort to the user.

[0005] An object of the present disclosure is to provide a technology for processing images in accordance with the movement of a part of the user's body that is less likely to cause discomfort to the user. [Means for solving the problem]

[0006] In order to solve the above problem, a control device according to one embodiment of the present disclosure includes a reference point setting unit that acquires the spatial coordinates of a first part, a second part, and a third part of a user's body positioned in front of a display, identifies a midpoint between the spatial coordinates of the first part and the spatial coordinates of the second part, and sets a point on the display where a virtual line passing through the identified midpoint and the spatial coordinate of the third part intersects as a reference point, and a function execution unit that performs processing on an image displayed on the display according to the movement of the first part and the second part, using the set reference point as a reference.

[0007] Another aspect of the present disclosure is a control method comprising the steps of: acquiring spatial coordinates of a first part, a second part, and a third part of a body of a user positioned in front of a display; identifying an intermediate point between the spatial coordinates of the first part and the spatial coordinates of the second part, and setting a point on the display where an imaginary line passing through the identified intermediate point and the spatial coordinate of the third part intersects as a reference point; and performing processing on an image displayed on the display according to movements of the first part and the second part, using the set reference point as a reference. [Effects of the Invention]

[0008] According to the present disclosure, a technique for processing an image in accordance with the movement of a part of a user's body can be made to reduce the sense of discomfort felt by the user. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of a display system according to an embodiment. [Figure 2] 2 is a perspective view showing a situation in which a user inputs an operation to the display system of FIG. 1. FIG. [Figure 3] FIG. 10 is a diagram showing an example in which a user performs a rotation operation. [Figure 4] FIG. 2 is a diagram showing a situation in which the display system of FIG. 1 recognizes two fingertips of a user. [Figure 5] 5 is a diagram showing a situation following FIG. 4 in which the user keeps two fingertips still for a predetermined period of time. [Figure 6]FIG. 10 is a diagram showing an example in which two users each perform multiple operations. [Figure 7] FIG. 10 is a diagram showing an example in which a user performs a multi-operation and another user performs a single operation. [Figure 8] 2 is a flowchart showing the processing of the display system of FIG. 1. [Figure 9] 9 is a flowchart showing the processing for one user in S14 of FIG. 8. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, the same or equivalent components, parts, and steps shown in each drawing will be denoted by the same reference numerals, and redundant explanations will be omitted where appropriate. In addition, the dimensions of the parts in each drawing are enlarged or reduced as appropriate for ease of understanding.

[0011] Fig. 1 is a block diagram of a display system 1 according to an embodiment. Fig. 2 is a perspective view showing a situation in which a user U1 inputs an operation to the display system 1 of Fig. 1. The display system 1 includes a sensor 10, a control device 12, and a display 14. The control device 12 is not shown in Fig. 2.

[0012] The display system 1 may be installed indoors or in a vehicle such as an automobile, and its use is not particularly limited. By moving a user U1's fingertip in the air without touching the display 14, an operation corresponding to the movement can be input to the control device 12, and the control device 12 executes processing corresponding to the input operation. The operation of the user U1 moving a fingertip in the air can also be called an air touch operation or a gesture operation.

[0013] The sensor 10 is, for example, a depth camera, and is installed in a position where it can capture an image of a user U1 positioned in front of the display surface of the display 14. The sensor 10 periodically captures an image of the user U1 and outputs the captured time-series distance images to the control device 12. The frequency of capturing images can be determined appropriately through experiments or simulations, and may be, for example, multiple times per second. The distance image includes information on the spatial coordinates of each position within the image. These spatial coordinates are coordinates in a three-dimensional Cartesian coordinate system specific to the sensor 10.

[0014] The control device 12 recognizes the movement of the fingertip of the user U1 based on the time-series distance images supplied from the sensor 10, accepts operation input according to the movement of the fingertip, and controls the image displayed on the display 14 according to that operation.

[0015] The display 14 displays an image under the control of the control device 12. The display 14 may have a screen size of several tens of inches or more for indoor installation, or may have a smaller screen size for in-vehicle installation.

[0016] There may be one or more users. An example of operation by two users will be described later. The number of fingertips that can be used for operation input per user is, for example, one or two. An example of operation by one fingertip will also be described later.

[0017] 2, for example, when user U1 moves fingertip 50 of his right hand and fingertip 52 of his left hand in the direction of arrow A1 to perform a pinch-in motion, bringing the two fingertips closer together, control device 12 reduces the image on the screen of display 14 around reference point P5. The midpoint between position P1 of fingertip 50 and position P2 of fingertip 52 is defined as P4, and reference point P5 is the point on the screen of display 14 where an imaginary line passing through position P3 of one of user U1's eyes 54 and midpoint P4 intersects. Control device 12 may or may not display an image indicating reference point P5 on display 14.

[0018] Furthermore, when the user moves two fingertips in the opposite direction of the arrow A1 and performs a pinch-out motion by moving the two fingertips apart, the control device 12 enlarges the image around the reference point P5 on the screen.

[0019] In this way, the image can be reduced or enlarged around the viewpoint of user U1 on the image when the user U1's line of sight passes through midpoint P4 between the two fingertips, which makes it less likely that the user U1 will feel uncomfortable. In contrast, in a comparative example in which the image is reduced or enlarged around a position on the image that is far away from the viewpoint of user U1 when the user U1's line of sight passes through midpoint P4, a reduced or enlarged image that is different from the user U1's intention may be obtained, which may cause the user U1 to feel uncomfortable.

[0020] Fig. 3 shows an example of a rotation action performed by user U1. Fig. 3 shows the X-axis, Y-axis, and Z-axis of a three-dimensional Cartesian coordinate system of display system 1. When user U1 performs a rotation action of rotating fingertip 50 of his right hand and fingertip 52 of his left hand around midpoint P4 in the direction of arrow A2, that is, rotating the two fingertips around an axis parallel to the Y-axis, control device 12 rotates the image around an axis parallel to the Y-axis on the screen, with reference point P5 on the screen as the center.

[0021] Furthermore, when user U1 rotates fingertip 50 of his right hand and fingertip 52 of his left hand around midpoint P4 in the direction of arrow A3, i.e., rotates the two fingertips around an axis parallel to the X-axis, control device 12 rotates the image around an axis parallel to the X-axis on the screen, with reference point P5 on the screen as the center.

[0022] Although not shown, when user U1 performs a rotational action of rotating two fingertips around any virtual axis, control device 12 rotates the image around an axis on the screen parallel to the virtual axis, with reference point P5 on the screen as the center.

[0023] Therefore, the image can be rotated around the viewpoint of the user U1 on the image when the line of sight of the user U1 passes through the midpoint P4 between the two fingertips.

[0024] An operation using a pinch-in operation, a pinch-out operation, and a rotation operation using two fingertips of one user is called a multi-operation. The multi-operation may include other operations. The two fingertips used for the multi-operation may be any two fingertips of one hand, for example, the tips of the thumb and index finger of one hand.

[0025] The two fingertips 50 and 52 used for operation are merely examples and may be the first and second parts of the user's body. The first and second parts may be any parts that can be moved independently to perform multiple operations, such as two wrists, two fists, or two palms. The eye 54 is also an example and may be a third part of the user's body that is different from the first and second parts. The third part may be, for example, a part of the face, such as the nose, mouth, the midpoint between two eyes, or the midpoint between two ears. The closer the third part is to the eye, the closer the reference point P5 can be set to the viewpoint on the image when the user's line of sight passes through the midpoint P4. In the embodiment, an example will be described in which the first and second parts are two fingertips and the third part is an eye.

[0026] 1, the control device 12 includes a control unit 20 and a storage unit 22. The control unit 20 has an image acquisition unit 24, a coordinate calculation unit 26, a determination unit 28, a reference point setting unit 30, a function execution unit 32, and a display output unit 34.

[0027] The configuration of the control unit 20 can be realized in hardware terms by the CPU, memory, and other LSI of any computer, and in software terms by programs loaded into memory, but here we depict functional blocks realized by the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms using only hardware, only software, or a combination of both.

[0028] The image acquisition unit 24 acquires, from the sensor 10, time-series distance images of one or more users positioned in front of the display 14, and performs image recognition on each of the acquired distance images to detect the first fingertip, the second fingertip, and the eyes of the one or more users. Known image recognition techniques can be used to detect the fingertips and eyes.

[0029] The image acquisition unit 24 detects one of two eyes located within a predetermined distance in the distance image. The predetermined distance is, for example, several centimeters, and two eyes located within the predetermined distance can be considered to be the eyes of the same user. This process corresponds to the image acquisition unit 24 detecting the eyes of each user. If there are multiple users in the distance image, multiple eyes are detected. The number of detected eyes represents the number of users in front of the display 14.

[0030] The image acquisition unit 24 acquires the spatial coordinates of the detected first fingertip, second fingertip, and eye in time series, and outputs them to the coordinate calculation unit 26.

[0031] The coordinate calculation unit 26 calculates, in time series, the spatial coordinates of the user's first fingertip, second fingertip, and eye in the three-dimensional Cartesian coordinate system of the display system 1, based on the spatial coordinates of the user's first fingertip, second fingertip, and eye in the coordinate system of the sensor 10 supplied from the image acquisition unit 24, and the spatial coordinates of the installation position of the sensor 10 and the attachment angle of the sensor 10 included in the coordinate information stored in the storage unit 22. The coordinate calculation unit 26 outputs the calculated spatial coordinates to the determination unit 28.

[0032] The determination unit 28 determines the type of operation to be accepted for each of the one or more detected eyes, i.e., for each user, based on the spatial coordinates supplied from the coordinate calculation unit 26 and the pairing information stored in the storage unit 22, and outputs the determined information to the function execution unit 32. The pairing information includes a first distance, a second distance, and a predetermined time, which will be described below.

[0033] When the first fingertip and the second fingertip are located within a first distance range from each of the detected one or more eyes, the determination unit 28 instructs the reference point setting unit 30 to set a reference point. The reference point setting process will be described later.

[0034] For each of the one or more detected eyes, when the first fingertip and the second fingertip are located within a range of a first distance from the eye and a predetermined multi-operation condition is satisfied, the determination unit 28 determines to accept the multi-operation. The determination to accept the multi-operation corresponds to pairing the first fingertip and the second fingertip located within the range of the first distance from the eye.

[0035] The multi-operation condition is that the distance between the first fingertip and the second fingertip is equal to or less than a second distance, and the first fingertip and the second fingertip do not remain stationary for a predetermined period of time or longer.

[0036] The first distance may be, for example, several tens of centimeters, may be a value in the range of 40 cm to 60 cm, or may be 50 cm. The second distance may be, for example, several tens of centimeters, may be a value in the range of 40 cm to 60 cm, or may be 50 cm. The first distance and the second distance may be equal to or different from each other. The predetermined time may be, for example, several seconds, or may be 2 seconds. The first distance, the second distance, and the predetermined time may be determined appropriately through experiments or simulations. The first distance, the second distance, and the predetermined time may be set by the user.

[0037] For each of the one or more detected eyes, if the first fingertip and the second fingertip are located within a range of a first distance from the eye but the multi-operation condition is not satisfied, or if only one of the first fingertip and the second fingertip is located within a range of the first distance from the eye, the determination unit 28 determines to accept a single operation. A single operation is an operation using an operation point on the screen corresponding to one or two fingertips, and specific examples will be described later. If only one of the first fingertip and the second fingertip is located within a range of the first distance from the eye, the determination unit 28 accepts a single operation by the movement of the fingertip located within the range of the first distance from the eye.

[0038] When determining whether to accept a multi-operation or a single operation, determination unit 28 determines a process according to the movements of the first and second fingertips based on the spatial coordinates of the first and second fingertips and the command information stored in storage unit 22. The command information is information that associates the movements of the first and second fingertips with predetermined processes.

[0039] The determination unit 28 determines not to execute processing if the first fingertip and the second fingertip are not located within a first distance range from each of the detected one or more eyes. In this case, it is assumed that the user's eyes have been detected but the user's fingertips have not been detected, that is, the user has not extended his / her fingertips and is not attempting to perform a gesture operation.

[0040] In accordance with instructions from the determination unit 28, the reference point setting unit 30 acquires the spatial coordinates of the first fingertip, the second fingertip, and the eye for each of the detected one or more eyes from the determination unit 28, identifies the midpoint between the spatial coordinates of the first fingertip and the spatial coordinates of the second fingertip, and sets the point on the display 14 where a virtual line passing through the identified midpoint and the spatial coordinates of the eye intersects as a reference point.

[0041] The reference point setting unit 30 sets a reference point based on the spatial coordinates of the installation position of the display 14 included in the coordinate information stored in the storage unit 22. The reference point is a coordinate in a two-dimensional Cartesian coordinate system on the display surface of the display 14.

[0042] The reference point setting unit 30 stores the coordinate information of the set reference point as reference point information in the storage unit 22. The set reference point is maintained until, for example, a switch is made to accepting a single operation.

[0043] When the determination unit 28 determines that the multi-operation has been accepted, the function execution unit 32 executes processing according to the movements of the first fingertip and the second fingertip on the image displayed on the display 14, with the set reference point as a reference, based on the content of the processing determined by the determination unit 28 for that user and the reference point information stored in the storage unit 22. The function execution unit 32 generates image data on which the processing according to the operations has been executed, and supplies the image data to the display output unit 34.

[0044] As a process according to the movement of the first fingertip and the second fingertip, the function executing unit 32 executes a process of reducing the image around the reference point in response to a pinch-in action of bringing the first fingertip and the second fingertip closer together. The function executing unit 32 reduces the image by a degree according to the movement distance of the first fingertip and the second fingertip.

[0045] As a process in response to the movement of the first fingertip and the second fingertip, the function executing unit 32 executes a process of enlarging the image around the reference point in response to a pinch-out action of moving the first fingertip and the second fingertip away from each other. The function executing unit 32 enlarges the image by a degree in response to the movement distance of the first fingertip and the second fingertip.

[0046] The function execution unit 32 executes a process of rotating an image around a reference point in response to a rotation action of rotating the first fingertip and the second fingertip around a midpoint as a process corresponding to the movement of the first fingertip and the second fingertip. The image to be rotated may be a target image such as a partial three-dimensional image on the screen. The function execution unit 32 rotates the image by an angle corresponding to the rotation angle of a virtual line connecting the first fingertip and the second fingertip.

[0047] When the determination unit 28 determines that a single operation is accepted, the function execution unit 32 executes a process for the user according to at least one of a first single operation performed by a first fingertip movement and a second single operation performed by a second fingertip movement, based on the content of the process determined by the determination unit 28.

[0048] The function executing unit 32 does not execute processing when the first fingertip and the second fingertip are not located within a first distance range from each detected eye.

[0049] The display output unit 34 outputs the image data supplied from the function execution unit 32 to the display 14, causing the display 14 to display the image.

[0050] Next, examples of various operations will be given with reference to FIGS. 4 shows a situation in which the display system 1 of FIG. 1 recognizes two fingertips of the user U1. The distance between the two fingertips is assumed to be equal to or less than a second distance. The control device 12 sets a reference point P5. In this state, the control device 12 can accept multiple operations.

[0051] 5, which follows on from FIG. 4, shows a situation in which the user U1 keeps his or her two fingertips still for a predetermined time. With the two fingertips still for the predetermined time in the state of FIG. 4, the control device 12 switches to accepting a single operation. Therefore, the user U1 can perform a single operation using an operation point P5a on the screen corresponding to the spatial coordinates of the position P1 of the fingertip 50 and an operation point P5b on the screen corresponding to the spatial coordinates of the position P2 of the fingertip 52.

[0052] For example, the control device 12 may set the point on the display 14 where an imaginary line passing through the eye position P3 and the position P1 of the fingertip 50 intersects as the operation point P5a, and may set the point on the display 14 where an imaginary line passing through the eye position P3 and the position P2 of the fingertip 52 intersects as the operation point P5b. The control device 12 may or may not display images such as cursors indicating the operation points P5a and P5b on the display 14.

[0053] User U1 can move operation point P5a within the screen by moving fingertip 50 in the air, and can move operation point P5b within the screen by moving fingertip 52 in the air, and can use operation point P5a and operation point P5b to perform various well-known operations that are different from multi-operation. For example, the image of an object displayed on display 14 can be sandwiched between operation point P5a and operation point P5b from both sides, and the image of the object can be moved by moving operation point P5a and operation point P5b in that state.

[0054] 5, in which single operation is accepted, if fingertips 50 and 52 are again kept stationary for a predetermined time, the state returns to that of FIG. 4, and control device 12 accepts multi-operation. This makes it possible to easily switch between multi-operation and single operation in a short time using two fingertips, which is convenient.

[0055] Furthermore, when the distance between the two fingertips is increased from the state in which the distance between the two fingertips is equal to or less than the second distance shown in Fig. 4 to a distance greater than the second distance, the control device 12 may also determine to accept a single operation. When the distance between the two fingertips is reduced to the second distance or less, the control device 12 again determines to accept a multi-operation. This allows easy switching between a multi-operation and a single operation in a short time.

[0056] Next, a specific example of processing in the case of two users will be described. 6 shows an example in which two users, U1 and U2, perform multiple operations. The distance d1 between the eye position P3 of user U1 and the fingertip position P1, and the distance d2 between the eye position P3 and the fingertip position P2, are each equal to or less than a first distance. The control device 12 pairs two fingertips of user U1 that are within the first distance from the eye position P3 of user U1, and accepts multiple operations using these fingertips.

[0057] The distance d11 between the eye position P13 of the user U2 and the fingertip position P11, and the distance d12 between the eye position P13 and the fingertip position P12 are also each equal to or less than the first distance. The control device 12 pairs two fingertips of the user U2 that are within the first distance from the eye position P13 of the user U2 and accepts multi-operations using these fingertips. The control device 12 identifies a midpoint P14 between the fingertip positions P11 and P12 for the user U2 and sets the point on the display 14 where a virtual line passing through the midpoint P14 and the eye position P13 intersects as a reference point P15. The control device 12 performs processing on the image of the user U2 according to the movement of the two fingertips, with the reference point P15 as the center.

[0058] For example, when the display 14 is displaying two images, user U1 can perform multiple operations on one image, and user U2 can perform multiple operations on the other image, which is very convenient. At least one of the users can also perform multiple operations using two fingertips of one hand.

[0059] 7 shows an example in which user U1 performs a multi-operation and another user U2 performs a single-operation. As in FIG. 6, the control device 12 pairs two fingertips of user U1 that are within a first distance from the eye position P3 of user U1 and accepts the multi-operation by these fingertips. Because the distance d3 between position P3 and position P11 of one fingertip on user U2's right hand is greater than the first distance, the control device 12 does not pair the fingertip at position P11 with the fingertip at position P2.

[0060] Since only one fingertip of the user U2's right hand is present within the first distance from the eye position P13 of the user U2, the control device 12 accepts a single operation using this fingertip. The user U2 can perform a single operation using an operation point P15a on the screen that corresponds to the spatial coordinates of the fingertip position P11. The user U2 may perform a single operation using the fingertip of his left hand instead of the fingertip of his right hand.

[0061] Although not shown, when user U1 holds up one finger and user U2 holds up one finger, the control device 12 does not pair these two fingertips and accepts single operations from each user.

[0062] Although not shown, for example, if one user U2 does not extend his / her finger, the control device 12 detects the eyes of the user U2 but does not execute any processing for the user U2.

[0063] Next, the overall operation of the display system 1 configured as described above will be described. FIG. 8 is a flowchart showing the processing of the display system 1 in FIG. 1. The image acquisition unit 24 determines whether an eye and a fingertip have been detected (S10), and if not (N in S10), the process returns to S10. If an eye and a fingertip have been detected (Y in S10), the coordinate calculation unit 26 calculates the three-dimensional coordinates of the eye and the fingertip (S12). The determination unit 28 determines the type of operation to be accepted for each user (S14), and the function execution unit 32 executes processing corresponding to the operation for each user (S16), and the process ends.

[0064] If there are a plurality of users, the function executing unit 32 may execute the process according to the operation in order from the user whose type of operation was determined in S14.

[0065] Fig. 9 is a flowchart showing the process for one user in S14 of Fig. 8. If there are multiple users, the process of Fig. 9 is executed for each user in S14 of Fig. 8.

[0066] If the number of fingertips within the range of the first distance from the eye is "0" ("0" in S20), the determination unit 28 determines not to execute the process (S34) and ends the process. If the number of fingertips within the range of the first distance from the eye is "1" ("1" in S20), the determination unit 28 determines to accept the single operation (S32) and ends the process.

[0067] If the number of fingertips within the first distance from the eye is "2" ("2" in S20) and the distance between the two fingertips is equal to or less than the second distance (Y in S22), the reference point setting unit 30 derives the three-dimensional coordinates of the midpoint between the two fingertips (S24) and sets a reference point on the display 14 from the three-dimensional coordinates of the midpoint and the three-dimensional coordinates of the eye (S26). If the fingertips do not remain stationary for a predetermined time or longer (N in S28), the determination unit 28 decides to accept the multi-operation (S30) and ends the process.

[0068] If the distance between the two fingertips is not equal to or less than the second distance in S22 (N in S22), the process proceeds to S32. If the fingertips remain stationary for a predetermined time or longer in S28 (Y in S28), the process proceeds to S32.

[0069] According to this embodiment, the user's viewpoint on the image when the user's line of sight passes through the midpoint between the two fingertips is used as the reference point, and therefore, the image can be reduced, enlarged, rotated, etc., making it less likely that the user will feel uncomfortable.

[0070] The present disclosure has been described above based on the embodiments. These embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component or each treatment process, and that such modifications are also within the scope of the present disclosure.

[0071] A control device according to one aspect of the present disclosure includes: a reference point setting unit that acquires spatial coordinates of a first part, a second part, and a third part of the body of a user positioned in front of a display, identifies an intermediate point between the spatial coordinates of the first part and the spatial coordinates of the second part, and sets a point on the display where a virtual line passing through the identified intermediate point and the spatial coordinate of the third part intersects as a reference point; a function execution unit that executes processing according to the movements of the first portion and the second portion on the image displayed on the display, using the set reference point as a reference; Equipped with.

[0072] According to this aspect, processing can be performed on the image using a position on the image that is close to the viewpoint when the user's line of sight passes through the midpoint between the first and second portions as the reference point, thereby making it possible to reduce the sense of discomfort felt by the user.

[0073] the first portion is a first fingertip of the user; the second portion is a second fingertip of the user; The third portion may be a portion of the user's face.

[0074] In this case, the image can be easily manipulated based on the movement of the two fingertips. Also, because the third part is part of the user's face, the reference point can be set to a position closer to the user's viewpoint on the image when the user's line of sight passes through the midpoint between the first and second parts. This makes it less likely that the user will feel uncomfortable.

[0075] The third portion may be the user's eye.

[0076] In this case, the reference point can be the viewpoint on the image when the user's line of sight passes through the midpoint between the first fingertip and the second fingertip, which further reduces the sense of discomfort felt by the user.

[0077] the control device further includes an image acquisition unit that acquires images of one or more of the users and detects the third portion of each of the users based on the acquired images; The reference point setting unit may set the reference point for each of the detected one or more third portions when the first portion and the second portion are located within a range of a first distance from the third portion.

[0078] In this case, a reference point can be set for each of the one or more users who satisfy the conditions.

[0079] The function executing unit may perform processing on the image according to the movement of the first part and the second part, using the reference point as a reference, when a predetermined multi-operation condition is satisfied, for each of the detected one or more third parts, if the first part and the second part are located within the range of the first distance from the third part.

[0080] In this case, for each of the one or more users who satisfy the conditions, processing can be performed on the image using the reference point as a reference, thereby enabling operations by multiple users to be handled.

[0081] The function executing unit may execute, for each of the one or more detected third parts, at least one of a process according to the movement of the first part and a process according to the movement of the second part when the first part and the second part are located within the first distance range from the third part but the multi-operation condition is not satisfied, or when only one of the first part and the second part is located within the first distance range from the third part.

[0082] In this case, for each of the one or more users who satisfy the conditions, at least one of the process according to the first portion of the movement and the process according to the second portion of the movement can be executed, thereby enabling various operations by multiple users to be handled.

[0083] The multi-operation condition may be that the distance between the first portion and the second portion is equal to or less than a second distance, and the first portion and the second portion are not stationary for a predetermined period of time or longer.

[0084] In this case, the user can easily satisfy the multi-operation condition.

[0085] The function execution unit performs the process according to the movement of the first part and the second part by: a process of reducing the image around the reference point in response to a pinch-in action of bringing the first portion and the second portion closer together; A process of enlarging the image around the reference point may be executed in response to a pinch-out motion that moves the first portion and the second portion away from each other.

[0086] In this case, the image can be enlarged or reduced around the reference point in accordance with the operations of the first and second parts.

[0087] The function executing unit may perform a process of rotating the image around the reference point in accordance with a rotation operation that rotates the first part and the second part around the midpoint, as a process in accordance with the movement of the first part and the second part.

[0088] In this case, the image can be rotated around the reference point in response to the actions of the first and second parts.

[0089] A control method according to one aspect of the present disclosure includes: obtaining spatial coordinates of a first portion, a second portion, and a third portion of a user's body positioned in front of a display; specifying an intermediate point between the spatial coordinates of the first portion and the spatial coordinates of the second portion, and setting a point on the display where a virtual line passing through the specified intermediate point and the spatial coordinates of the third portion intersects as a reference point; a step of executing a process according to the movement of the first portion and the second portion on the image displayed on the display, using the set reference point as a reference; Equipped with.

[0090] According to this embodiment, it is possible to make it less likely that the user will feel uncomfortable. [Explanation of symbols]

[0091] 1...display system, 10...sensor, 12...control device, 14...display, 20...control unit, 22...memory unit, 24...image acquisition unit, 26...coordinate calculation unit, 28...judgment unit, 30...reference point setting unit, 32...function execution unit, 34...display output unit, 50...fingertip, 52...fingertip, 54...eye, P4...intermediate point, P5...reference point.

Claims

1. a reference point setting unit that acquires spatial coordinates of a first part, a second part, and a third part of the body of a user positioned in front of a display, identifies an intermediate point between the spatial coordinates of the first part and the spatial coordinates of the second part, and sets a point on the display where a virtual line passing through the identified intermediate point and the spatial coordinate of the third part intersects as a reference point; a function execution unit that executes processing according to the movements of the first and second parts on the image displayed on the display, using the set reference point as a reference; an image acquisition unit that acquires images of one or more of the users and detects the third portion of each of the users based on the acquired images; Equipped with the third portion is a part of the user's face, the reference point setting unit sets the reference point when the first portion and the second portion are located within a first distance range from each of the detected one or more third portions. Control device.

2. the first portion is a first fingertip of the user; the second portion being a second fingertip of the user; The control device according to claim 1 .

3. the third part is the user's eye; The control device according to claim 2 .

4. and when a predetermined multi-operation condition is satisfied, the function execution unit executes a process on the image according to the movement of the first part and the second part, using the reference point as a reference, for each of the one or more detected third parts, when the first part and the second part are located within the first distance range from the third part. The control device according to claim 1 .

5. the function execution unit executes, for each of the one or more detected third portions, at least one of a process corresponding to a movement of the first portion and a process corresponding to a movement of the second portion when the first portion and the second portion are located within the first distance range from the third portion but the multi-operation condition is not satisfied, or when only one of the first portion and the second portion is located within the first distance range from the third portion. The control device according to claim 4.

6. the multi-operation condition is that the distance between the first portion and the second portion is equal to or less than a second distance, and the first portion and the second portion are not stationary for a predetermined time or longer; The control device according to claim 4 or 5.

7. The function execution unit performs the process according to the movement of the first part and the second part by: a process of reducing the image around the reference point in response to a pinch-in action of bringing the first portion and the second portion closer together; and enlarging the image around the reference point in response to a pinch-out motion that moves the first portion and the second portion away from each other. The control device according to any one of claims 1 to 6.

8. the function executing unit executes, as the processing in accordance with the movement of the first part and the second part, a process of rotating the image around the reference point in accordance with a rotation operation of rotating the first part and the second part around the midpoint. The control device according to any one of claims 1 to 7.

9. obtaining spatial coordinates of a first portion, a second portion, and a third portion of a user's body located in front of a display; specifying an intermediate point between the spatial coordinates of the first portion and the spatial coordinates of the second portion, and setting a point on the display where a virtual line passing through the specified intermediate point and the spatial coordinates of the third portion intersects as a reference point; a step of executing a process according to the movement of the first portion and the second portion on the image displayed on the display, using the set reference point as a reference; Equipped with images of one or more of the users are acquired, and the third portion of each of the users is detected based on the acquired images; the third portion is a part of the user's face, For each of the one or more detected third portions, the reference point is set when the first portion and the second portion are located within a first distance range from the third portion. Control method.

Images