Control device and control method

Abstract

To provide a control device capable of achieving further improvements.SOLUTION: A coordinate acquisition unit 26 acquires time-series spatial coordinates of a detection target. The spatial coordinates include a coordinate in a first direction, a coordinate in a second direction intersecting the first direction, and a coordinate in a third direction intersecting the first direction and the second direction. When a given second condition regarding the change in the coordinate of the first direction is satisfied at second time after a given first condition regarding the change in the coordinate of the first direction is satisfied at first time, a function execution unit 34 executes a series of processing for a piece of electronic equipment that is the target of operation based on the coordinate in the second direction and the coordinate in the third direction at one or more times between the first time and the second time.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a control device and a control method.

Background Art

[0002] An information processing device in which a user inputs an instruction by gesture is known (see, for example, Patent Document 1). In this information processing device, a coordinate calculation unit calculates coordinates of at least one location of an object to be detected. A control unit causes a display unit to display that at least one item out of a plurality of items with a predetermined order is a selection target, and changes the selection target item according to the predetermined order based on a change in coordinates. When the change in coordinates satisfies a predetermined condition, the control unit executes a process according to the selected target item.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a control device that receives an operation input by a user, further improvement is required.

Means for Solving the Problems

[0005] To solve the above problems, a control device according to one aspect of the present disclosure includes: a coordinate acquisition unit that acquires the time-series spatial coordinates of an object to be detected, wherein the spatial coordinates include coordinates in a first direction, coordinates in a second direction intersecting the first direction, and coordinates in a third direction intersecting the first and second directions; and a function execution unit that, after a predetermined first condition relating to a change in the coordinates in the first direction is met at a first time, and a predetermined second condition relating to a change in the coordinates in the first direction is met at a second time, executes processing on an electronic device that is the object to be operated based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time point and the second time point.

[0006] Another aspect of the present disclosure is a control method. This method comprises the steps of: obtaining the time-series spatial coordinates of an object to be detected, the spatial coordinates including a coordinate in a first direction, a coordinate in a second direction intersecting the first direction, and a coordinate in a third direction intersecting the first and second directions; and, if a predetermined first condition relating to a change in the coordinate in the first direction is met at a first time, and a predetermined second condition relating to a change in the coordinate in the first direction is met at a second time, then performing processing on an electronic device to be operated on based on the coordinate in the second direction and the coordinate in the third direction at one or more time points between the first time and the second time. [Effects of the Invention]

[0007] Further improvements can be achieved through the above-described method. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram of the information processing system according to the first embodiment. [Figure 2] This is a perspective view illustrating the aerial pushing motion. [Figure 3] This figure shows an example of the relationship between fingertip position and time during an air-pressing motion. [Figure 4] This diagram illustrates the processing of the control device shown in Figure 1 during an aerial pushing operation. [Figure 5]Figure 1 is a flowchart showing the processing steps of the information processing system. [Figure 6] This diagram illustrates the processing of the control device during an aerial pushing operation, using a different second condition. [Figure 7] This diagram illustrates the processing of the control device during an aerial pushing operation, using yet another second condition. [Figure 8] This is a diagram illustrating the processing of the information processing system in the second embodiment. [Figure 9] This is a flowchart showing the processing of the information processing system in the second embodiment. [Figure 10] This is a block diagram of the information processing system according to the third embodiment. [Figure 11] This is a flowchart showing the processing of the information processing system in the third embodiment. [Figure 12] This is a diagram illustrating the processing of the control device during the air-pushing operation of the fourth embodiment. [Figure 13] This is a flowchart showing the processing of the information processing system according to the fourth embodiment. [Figure 14] This is a flowchart illustrating the processing of an information processing system that combines the second, third, and fourth embodiments. [Figure 15] This is a diagram illustrating the aerial pushing operation for the information processing system of the fifth embodiment. [Modes for carrying out the invention]

[0009] In the following, identical or equivalent components, members, and processes shown in each drawing will be denoted by the same reference numerals, and redundant explanations will be omitted where appropriate. Furthermore, the dimensions of the members in each drawing will be enlarged or reduced as appropriate for ease of understanding.

[0010] (First Embodiment) FIG. 1 is a block diagram of an information processing system 1 according to the first embodiment. The information processing system 1 includes a sensor 10, a control device 12, and a display 14.

[0011] The information processing system 1 may be installed indoors or the like, or may be mounted on a vehicle such as an automobile, and its application is not particularly limited. The user can input an operation corresponding to the movement to the control device 12 by moving the fingertips in the air without touching the display 14, and the control device 12 executes processing corresponding to the input operation. The operation of the user moving the fingertips in the air can also be called an air touch operation or a gesture operation. In the present embodiment, the process of detecting the operation of the user pushing the fingertips into the air toward the display surface of the display 14 will be mainly described. This operation can also be called an air push operation or an air click operation.

[0012] The sensor 10 is, for example, a depth camera, and is installed at a position where it can image a user located in front of the display surface 15 of the display 14. The sensor 10 periodically images the user and outputs a time-series distance image obtained by the imaging to the control device 12. The imaging frequency can be appropriately determined by experiments or simulations, and may be, for example, multiple times per second. The distance image includes information on the spatial coordinates of each position in the image. These spatial coordinates are coordinates in a three-dimensional orthogonal coordinate system unique to the sensor 10.

[0013] Based on the time-series distance images supplied from the sensor 10, the control device 12 recognizes the movement of the user's fingertips, accepts an operation input corresponding to the movement of the fingertips, and executes processing corresponding to the operation. The processing corresponding to the operation includes processing for the display 14 which is the operation target. The operation target is not limited to the display 14, and may be other electronic devices such as a lighting device or a speaker, for example.

[0014] The fingertip is an example of a detection target used for operation input, and the detection target is not limited to the fingertip. As another example of the detection target, a clenched fist or the palm, which is a part of the user's body, can be mentioned. The detection target is not limited to a part of the user's body, and may be an object held by the user's hand or an object worn on the user's body.

[0015] The display 14 displays an image according to the control of the control device 12. The display 14 may be, for example, one with a screen size of several tens of inches or more for indoor installation, or one with a smaller screen size for in-vehicle use.

[0016] Figure 2 is a perspective view for explaining the air pushing operation. In Figure 2, the illustration of the sensor 10 and the control device 12 is omitted. As shown in Figure 2, the information processing system 1 defines the X-axis, Y-axis, and Z-axis of a three-dimensional orthogonal coordinate system. The Z-axis extends in the first direction. The first direction can also be called the Z direction. In the illustrated example, the first direction is the normal direction of the display surface 15 of the display 14. The coordinate in the first direction is the Z coordinate. For example, the Z coordinate on the display surface 15 is 0, and the Z coordinate increases as it moves away from the display surface 15. The X-axis extends in a second direction orthogonal to the first direction. The second direction can also be called the X direction. The coordinate in the second direction is the X coordinate. The Y-axis extends in a third direction orthogonal to the first direction and the second direction. The third direction can also be called the Y direction. The coordinate in the third direction is the Y coordinate. The X-axis and the Y-axis extend, for example, within the display surface 15.

[0017] As shown in Figure 2, when the user performs an air pushing operation of approaching and then moving away the fingertip 50 from the display surface 15 along the arrow A1, the control device 12 detects this operation.

[0018] Figure 3 shows an example of the relationship between the position and time of the fingertip 50 during an air-pressing operation. Figure 3 shows the position of the fingertip 50 at equal time intervals. Assume that the velocity of the fingertip 50 in the Z direction is negative when the Z coordinate of the fingertip 50 decreases. When the velocity of the fingertip 50 in the Z direction is negative, assume that the velocity of the fingertip 50 in the Z direction is negative. The inventors analyzed the change in the coordinate of the fingertip 50 during an air-pressing operation and found that, as shown in Figure 3, the absolute value of the velocity of the fingertip 50 in the Z direction increases and then decreases as the fingertip 50 approaches the display 14, then the fingertip 50 stops, and then the fingertip 50 returns in the opposite direction. Such movement of the fingertip 50 can also be called a reaction movement of the pressing operation.

[0019] Therefore, if the velocity of the fingertip 50 in the Z direction is negative and the absolute value of the velocity in the Z direction becomes greater than or equal to a first velocity threshold which is a predetermined positive value, and then the velocity in the Z direction reverses from negative to positive, the control device 12 identifies the X and Y coordinates of the fingertip 50 at the time the velocity in the Z direction reverses from negative to positive, and performs processing corresponding to the pressing operation based on the identified X and Y coordinates. The processing corresponding to the pressing operation may include, for example, the process of displaying an image on the screen of the display 14 at the identified X and Y coordinate positions to indicate that the position has been clicked, that is, that the position has been determined.

[0020] For example, if multiple items such as operation buttons are displayed on the display 14, the control device 12 will not highlight the item corresponding to the direction the fingertip 50 is pointing before detecting the pressing motion. In other words, the control device 12 will not display an image on the display 14 that would allow the user to identify the item to be selected corresponding to the position of the fingertip 50 before detecting the pressing motion.

[0021] Therefore, the user can intuitively perform the action to determine the X and Y coordinates of the fingertip 50 in a relatively short time, without having to visually confirm that the desired position is selected.

[0022] In contrast, as a comparative example, we consider a detection device that highlights the item to be selected corresponding to the fingertip on the display before detecting a predetermined decision action. In this comparative example, the item to be selected can be changed from among multiple items depending on the position of the fingertip. The user moves their fingertip up, down, left, and right in front of the display, and when the desired item is highlighted as the item to be selected, they perform a predetermined decision action with their fingertip. When the detection device detects that the decision action has been completed, it determines that the item has been selected and executes processing according to the selected item. In the comparative example, the user must visually confirm that the desired item has been selected before performing the decision action, which may impair intuitiveness. Also, the decision actions in the comparative example include, for example, stopping the fingertip for a certain period of time, performing specific gestures such as drawing a circle with the fingertip, or pulling the fingertip back a certain distance. The time it takes for the detection device in the comparative example to detect that these decision actions have been completed is relatively long.

[0023] As described above, this embodiment can solve the problems of these comparative examples. For example, when a user repeatedly performs an air-pushing motion at different positions in succession, the control device 12 can accept multiple operations in a shorter time than the comparative examples.

[0024] The operation of the control device 12 will be described in more detail below. As shown in Figure 1, the control device 12 comprises a control unit 20 and a storage unit 22. The control unit 20 includes an image acquisition unit 24, a coordinate acquisition unit 26, a calculation unit 28, a determination unit 30, a function execution unit 34, and a display output unit 36.

[0025] The configuration of the control unit 20 can be realized in hardware terms using the CPU, memory, and other LSIs of any computer, and in software terms by programs loaded into memory, etc., but here we are describing the functional blocks realized by the cooperation of these.Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by hardware alone, software alone, or a combination thereof.The control device 12 comprises, for example, a processor and memory, and the processor executes programs loaded into memory to realize the functions of the control unit 20 and each of the functional blocks of the control unit 20.

[0026] The storage unit 22 stores various types of information. The storage unit 22 is implemented by hardware for storing information (in other words, data), such as memory or storage. The storage unit 22 holds coordinate information, velocity threshold information, and mode information. The coordinate information includes the spatial coordinates of the installation position of the sensor 10, the mounting angle of the sensor 10, and the spatial coordinates of the installation position of the display 14. The velocity threshold information includes information on the first velocity threshold and the second velocity threshold. Details of the coordinate information, velocity threshold information, and mode information will be described later.

[0027] The image acquisition unit 24 acquires time-series distance images of the user positioned in front of the display 14 from the sensor 10, and detects the user's fingertip 50 by performing image recognition on each acquired distance image. Known image recognition techniques can be used to detect the fingertip 50. The image acquisition unit 24 acquires the spatial coordinates of the detected fingertip 50 in time series and outputs them to the coordinate acquisition unit 26.

[0028] The coordinate acquisition unit 26 calculates and acquires the time-series spatial coordinates of the user's fingertip 50 in the three-dimensional Cartesian coordinate system of the information processing system 1, based on the spatial coordinates of the user's fingertip 50 in the coordinate system of the sensor 10 supplied by the image acquisition unit 24, and the spatial coordinates of the installation position of the sensor 10 and the mounting angle of the sensor 10 included in the coordinate information stored in the storage unit 22. The coordinate acquisition unit 26 outputs the acquired spatial coordinates to the calculation unit 28.

[0029] The calculation unit 28 calculates the velocity of the fingertip 50 in the Z direction over time based on the spatial coordinates supplied from the coordinate acquisition unit 26, and supplies the calculated velocity in the Z direction to the determination unit 30.

[0030] The determination unit 30 determines the relative magnitudes of the Z-direction velocity supplied from the calculation unit 28, the first velocity threshold, and the second velocity threshold, based on the velocity threshold information stored in the memory unit 22. The determination unit 30 determines whether a predetermined first condition is met, which is that the Z-direction velocity of the fingertip 50 is a negative velocity and the absolute value of the Z-direction velocity is greater than or equal to the first velocity threshold. If the first condition is met, the determination unit 30 transitions to click mode. Click mode is an operation acceptance mode for accepting air-pressing operations.

[0031] The determination unit 30 determines whether a predetermined second condition is met in click mode, which is that the velocity in the Z direction is a positive velocity and is greater than or equal to a second velocity threshold. Here, the second velocity threshold is a positive value slightly greater than zero. The second velocity threshold may be set so as to accurately determine the time when the velocity in the Z direction reverses from a negative velocity to a positive velocity. In other words, the second condition may be that the velocity in the Z direction is a positive velocity.

[0032] The first and second conditions are conditions relating to the change in the Z-axis coordinate of the fingertip 50, respectively. The first and second velocity thresholds can be determined as appropriate through experimentation or simulation. The first and second velocity thresholds may also be set by the user. The determination unit 30 supplies the determination result to the function execution unit 34.

[0033] If the determination unit 30 determines that the first condition is met at the first time step and the second condition is met at the second time step which is after the first time step, the function execution unit 34 executes processing on the display 14 based on the X and Y coordinates of the fingertip 50 at the second time step. The mode information held in the storage unit 22 includes information about the processing to be executed when the second condition is met in click mode. For example, based on the mode information held in the storage unit 22, the function execution unit 34 generates image data in which the determined X and Y coordinates and processing corresponding to the pressing operation have been executed, and supplies it to the display output unit 36.

[0034] Furthermore, the function execution unit 34 may perform processing based not only on the X and Y coordinates of the fingertip 50 at the second time step, but also on the X and Y coordinates of the fingertip 50 at any time step between the first and second time steps.

[0035] The display output unit 36 ​​outputs image data supplied from the function execution unit 34 to the display 14, causing the image to be displayed on the display 14.

[0036] Figure 4 is a diagram illustrating the processing of the control device 12 in Figure 1 during an air-pushing operation. In Figure 4, the horizontal axis represents the elapsed time from time 0, and the vertical axis represents the Z coordinate and velocity in the Z direction. In Figure 4, the change in the Z coordinate of the fingertip 50 according to the elapsed time is shown by a solid line, and the change in velocity in the Z direction at that time is shown by a dashed line.

[0037] The fingertip 50 begins to move toward the display surface of the display 14, and its velocity in the Z direction becomes negative, so that at the first time t1, the absolute value of the velocity in the Z direction becomes equal to the first velocity threshold Vth1. Subsequently, the fingertip 50 stops and moves away from the display surface, and its velocity in the Z direction becomes positive, so that at the second time t2, the velocity in the Z direction becomes equal to the second velocity threshold Vth2. The function execution unit 34 identifies the X and Y coordinates of the fingertip 50 at the second time t2 and performs processing on the display 14.

[0038] Next, the overall operation of the information processing system 1 with the above configuration will be explained. Figure 5 is a flowchart of the processing of the information processing system 1 in Figure 1. The determination unit 30 determines whether the velocity in the Z direction is a negative velocity and whether the absolute value of the velocity in the Z direction is greater than or equal to the first velocity threshold Vth1 (S10). If this condition is not met (N in S10), the determination unit 30 returns to the processing of S10. If the velocity in the Z direction is a negative velocity and whether the absolute value of the velocity in the Z direction is greater than or equal to the first velocity threshold Vth1 (Y in S10), the determination unit 30 transitions to click mode (S12). The determination unit 30 determines whether the velocity in the Z direction is a positive velocity and whether it is greater than or equal to the second velocity threshold Vth2 (S14). If this condition is not met (N in S14), the determination unit 30 returns to the processing of S14. If the velocity in the Z direction is positive and greater than or equal to the second velocity threshold Vth2 (Y in S14), the function execution unit 34 obtains the X and Y coordinates of the fingertip 50 at the time it was determined that the velocity in the Z direction is positive and greater than or equal to the second velocity threshold Vth2, and executes the process (S16). The determination unit 30 cancels the click mode (S18) and returns to the process of S10. Note that if the conditions that the velocity in the Z direction is positive and greater than or equal to the second velocity threshold Vth2 are not met even after a predetermined time has elapsed in the process of S14, the determination unit 30 may proceed to the process of S18 and cancel the click mode.

[0039] Here, another example of the second condition is described. The determination unit 30 may determine whether the second condition is met, which is that the velocity of the fingertip 50 in the Z direction is a negative velocity and the absolute value of the velocity in the Z direction is less than or equal to the second velocity threshold Vth2. The second velocity threshold Vth2 is smaller than the first velocity threshold Vth1.

[0040] Figure 6 illustrates the processing of the control device 12 during an air-pushing operation using a different second condition. The velocity in the Z direction becomes negative, and at the first time t1, the absolute value of the velocity in the Z direction equals the first velocity threshold Vth1. Then, before the fingertip 50 stops, at the second time t2a, the absolute value of the velocity in the Z direction equals the second velocity threshold Vth2. In other words, when the absolute value of the velocity moving the fingertip 50 in the Z direction decreases, the control device 12 can determine the X and Y coordinates of the fingertip 50. Compared to the example in Figure 4, the time from the first time t1 to the second time t2a can be shortened, and the control device 12 can determine that an air-pushing operation has occurred at an earlier timing.

[0041] Furthermore, another second condition may be used. The determination unit 30 may determine whether the second condition is met, that the velocity of the fingertip 50 in the Z direction is equal to the second velocity threshold Vth2, or that it is a positive velocity. The second velocity threshold Vth2 is 0. Figure 7 is a diagram illustrating the processing of the control device 12 during an air-pushing operation using yet another second condition. At the first time t1, the absolute value of the velocity in the Z direction becomes equal to the first velocity threshold Vth1, and then at the second time t2b, when the fingertip 50 stops, the velocity in the Z direction becomes equal to the second velocity threshold Vth2. In other words, when the fingertip 50 stops with respect to the Z direction, the control device 12 can determine the X and Y coordinates of the fingertip 50. Compared to the example in Figure 4, since the second velocity threshold Vth2 is small, the time from the first time t1 to the second time t2a can be shortened, and the control device 12 can determine that an air-pushing operation has been performed at an earlier timing.

[0042] (Second Embodiment) In the second embodiment, the conditions for the determination unit 30 to transition to click mode also include conditions related to the amount of movement in the X direction and the amount of movement in the Y direction, which is different from the first embodiment. The differences from the first embodiment will be explained below.

[0043] Figure 8 is a diagram illustrating the processing of the information processing system 1 in the second embodiment. If the information processing system 1 can also accept sliding or flicking movements of the fingertip 50 in the air, the fingertip 50 may slide or flick while approaching the display surface 15. In other words, the fingertip 50 may move in the direction of arrow A3 in Figure 8. In this case, in the processing of the first embodiment, even if the user moves the fingertip 50 in the direction of arrow A3 without intending to perform a pressing motion in the air, the determination unit 30 may transition to click mode.

[0044] Therefore, in this embodiment, the following processing is performed. The calculation unit 28 periodically calculates a first movement amount, which is the amount of movement of the fingertip 50 in the X direction per predetermined time, and a second movement amount, which is the amount of movement of the fingertip 50 in the Y direction per predetermined time, based on the calculated spatial coordinates. The calculation unit 28 supplies the calculated first movement amount and second movement amount to the determination unit 30. The predetermined time is a natural number multiple of the imaging interval of the distance image by the sensor 10, and can be appropriately determined by experiment or simulation.

[0045] The memory unit 22 further stores displacement threshold information (not shown). The displacement threshold information includes information on a first displacement threshold, which is a predetermined value, and a second displacement threshold, which is also a predetermined value. The first and second displacement thresholds can be determined as appropriate by experiment or simulation. The first and second displacement thresholds may also be set by the user.

[0046] The determination unit 30 determines whether the third condition is met, that the first movement amount calculated by the calculation unit 28 is less than or equal to the first movement amount threshold, based on the movement amount threshold information held in the storage unit 22, and also determines whether the fourth condition is met, that the second movement amount calculated by the calculation unit 28 is less than or equal to the second movement amount threshold. If the first, third, and fourth conditions are met, the determination unit 30 transitions to click mode.

[0047] The region R1 shown in Figure 8 is an example of a region defined by the first and second movement thresholds. When the fingertip 50 moves in the direction of arrow A3, the position of the fingertip 50 moves outside of region R1, and therefore the third and fourth conditions are not met. For this reason, in the second embodiment, the determination unit 30 does not transition to click mode. Thus, it is possible to suppress the execution of actions unintended by the user in the information processing system 1.

[0048] On the other hand, if the fingertip 50 moves in the direction of arrow A2, the position of the fingertip 50 is within region R1, so the third and fourth conditions are met. Therefore, the determination unit 30 transitions to click mode.

[0049] If the determination unit 30 determines that the first, third, and fourth conditions are met at the first time step, and the second condition is met at the second time step, the function execution unit 34 executes processing on the display 14 based on the X-coordinate and Y-coordinate at the time interval from the first time step to the second time step.

[0050] The determination unit 30 may use either the third or fourth condition and omit the other. For example, if the information processing system 1 can only accept sliding or flicking movements in the X direction, the user is unlikely to perform a sliding or flicking movement in the Y direction. In such cases, the determination unit 30 may use the third condition and omit the fourth condition. This simplifies the processing of the control unit 20.

[0051] Figure 9 is a flowchart showing the processing of the information processing system 1 in the second embodiment. In the processing shown in Figure 9, the processing of S10a is performed instead of the processing of S10 in Figure 5. The determination unit 30 determines whether the first movement amount in the X direction is less than or equal to the first movement amount threshold, the second movement amount in the Y direction is less than or equal to the second movement amount threshold, the velocity in the Z direction is a negative velocity, and the absolute value of the velocity in the Z direction is greater than or equal to the first velocity threshold Vth1 (S10a). If these conditions are not met (N in S10a), the determination unit 30 returns to the processing of S10a. If the first movement amount is less than or equal to the first movement amount threshold, the second movement amount is less than or equal to the second movement amount threshold, the velocity in the Z direction is a negative velocity, and the absolute value of the velocity in the Z direction is greater than or equal to the first velocity threshold Vth1 (Y in S10a), the determination unit 30 transitions to click mode (S12). The subsequent processing is the same as the processing shown in Figure 5.

[0052] (Third embodiment) In the third embodiment, the function execution unit 34 performs processing on the display 14 based on the average values ​​of the X and Y coordinates of the fingertip 50, which is different from the first embodiment. The differences from the first embodiment will be explained below.

[0053] Figure 10 is a block diagram of the information processing system 1 according to the third embodiment. The control unit 20 further comprises an averaging unit 32. The determination unit 30 also supplies the determination result to the averaging unit 32.

[0054] The averaging unit 32 calculates a first average coordinate, which is the average of the X-coordinate of the fingertip 50 during a predetermined period between the first and second time points, and a second average coordinate, which is the average of the Y-coordinate of the fingertip 50, based on the determination result by the determination unit 30 and the spatial coordinate calculated by the calculation unit 28. The averaging unit 32 supplies the calculated first average coordinate and second average coordinate to the function execution unit 34.

[0055] The predetermined period to be averaged may be the entire period from the first time point to the second time point, or it may be a portion of that period. The start and end points of the predetermined period can be appropriately determined through experimentation or simulation so that the first and second average coordinates approach the coordinates intended by the user. The start and end points of the predetermined period may also be set by the user.

[0056] If the determination unit 30 determines that the first condition is met at the first time step and the second condition is met at the second time step, the function execution unit 34 performs processing on the display 14 based on the first average coordinates and the second average coordinates supplied by the averaging unit 32.

[0057] This process corresponds to the function execution unit 34 executing processing on the display 14 based on the X-coordinate and Y-coordinate of the fingertip 50 at multiple time points from the first time point to the second time point, after the first condition regarding the change in the Z-coordinate is met at the first time point and the second condition regarding the change in the Z-coordinate is met at the second time point.

[0058] When a finger is pressed in mid-air and brought close to the display surface, its position tends to shift in the X and Y directions, which may result in unstable X and Y coordinates for the fingertip (50). By using the first and second mean coordinates, the accuracy of calculating the X and Y coordinates corresponding to the position of the fingertip (50) can be improved.

[0059] Figure 11 is a flowchart showing the processing of the information processing system 1 in the third embodiment. In the processing shown in Figure 11, the processing of S20 is added between the processing of S12 and S14 in Figure 5, and the processing of S16a is performed instead of the processing of S16 in Figure 5. After the processing of S12, the averaging unit 32 averages the X coordinate and Y coordinate of the fingertip 50, respectively (S20). Next, if the conditions that the velocity in the Z direction is a positive velocity and is greater than or equal to the second velocity threshold Vth2 are not met (N in S14), the processing of the information processing system 1 returns to the processing of S20 by the averaging unit 32. On the other hand, if the velocity in the Z direction is a positive velocity and is greater than or equal to the second velocity threshold Vth2 (Y in S14), the function execution unit 34 obtains the first average coordinate of the X coordinate of the fingertip 50 and the second average coordinate of the Y coordinate of the fingertip 50 and executes the processing (S16a).

[0060] The averaging unit 32 may calculate weighted averages of the X and Y coordinates of the fingertip 50. The weights may increase or decrease as the time approaches the second time. The weights can be appropriately determined by experiment or simulation so that the first average coordinate and the second average coordinate tend to approach the coordinate intended by the user. The weights may be predetermined values ​​regardless of the user, or they may be set for each user. By setting the weights for each user, the accuracy of calculating the X and Y coordinates corresponding to the position of the fingertip 50 can be further improved in accordance with the user's pressing habits.

[0061] (Fourth embodiment) In the fourth embodiment, the conditions for accepting an air-pressing motion also include conditions related to the amount of movement in the Z direction in click mode, which is different from the first embodiment. The differences from the first embodiment will be explained below.

[0062] The determination unit 30 also supplies the determination result to the calculation unit 28. Based on the calculated spatial coordinates, the calculation unit 28 calculates the total movement amount, which is the sum of the movement amounts of the fingertip 50 in the Z direction from the first time step to the second time step, and supplies it to the determination unit 30.

[0063] The memory unit 22 further stores displacement threshold information (not shown). This displacement threshold information includes information on a predetermined total displacement threshold. The total displacement threshold can be determined as appropriate through experimentation or simulation. The total displacement threshold may also be set by the user.

[0064] The determination unit 30 determines, based on the movement amount threshold information held in the storage unit 22, whether the movement amount condition is met, which is that the total movement amount calculated by the calculation unit 28 is equal to or greater than the total movement amount threshold, and supplies the determination result to the function execution unit 34.

[0065] If the determination unit 30 determines that the first condition is met at the first time step, the second condition is met at the second time step, and the movement amount condition is met, the function execution unit 34 performs processing on the display 14 based on the X-coordinate and Y-coordinate of the fingertip 50 at one time step from the first time step to the second time step.

[0066] Figure 12 is a diagram illustrating the processing of the control device 12 during an air-pushing operation in the fourth embodiment. In this example, since the total amount of movement in the Z direction from the first time t1 to the second time t2 is greater than or equal to the total amount of movement threshold, the function execution unit 34 performs processing based on the X and Y coordinates of the fingertip 50 at the second time t2.

[0067] For example, if the movement amount condition is not considered, even if the user moves their fingertip 50 in the Z direction without intending to perform an air-pressing action, it may be determined that the speed of the fingertip 50 in the Z direction has fallen below the first velocity threshold Vth1, and processing on the display 14 may be executed. As a result, there is a possibility that actions on the display 14 that the user did not intend will be executed. In this embodiment, by considering the movement amount condition, the likelihood of processing on the display 14 being executed only when the user moves their fingertip 50 with the intention of performing an air-pressing action is increased. Therefore, it is possible to suppress the execution of actions on the display 14 that the user did not intend.

[0068] Figure 13 is a flowchart showing the processing of the information processing system 1 according to the fourth embodiment. In the processing shown in Figure 13, the processing of S22 is added between the processing of S12 and S14 in Figure 5, and the processing of S14a is performed instead of the processing of S14 in Figure 5. After the processing of S12, the calculation unit 28 sums up the amount of movement of the fingertip 50 in the Z direction (S22). Next, the determination unit 30 determines whether the velocity in the Z direction is the velocity in the positive direction, whether the velocity in the Z direction is equal to or greater than the second velocity threshold Vth2, and whether the total amount of movement in the Z direction is equal to or greater than the total amount of movement threshold (S14a). If these conditions are not met (N in S14a), the processing of the information processing system 1 returns to the processing of S22 by the calculation unit 28. If the velocity in the Z direction is the velocity in the positive direction, whether the velocity in the Z direction is equal to or greater than the second velocity threshold Vth2, and whether the total amount of movement in the Z direction is equal to or greater than the total amount of movement threshold (Y in S14a), the processing of the information processing system 1 moves to the processing of S16 by the function execution unit 34. Furthermore, if, after a predetermined time has elapsed in the process of S14a, the conditions that the velocity in the Z direction is the velocity in the positive direction, the velocity in the Z direction is equal to or greater than the second velocity threshold Vth2, and the total amount of movement in the Z direction is equal to or greater than the total amount of movement threshold are not met, the determination unit 30 may proceed to the process of S18 and cancel the click mode.

[0069] Furthermore, any two of the second, third, and fourth embodiments described above may be combined, or any three may be combined. The new embodiments resulting from these combinations will possess the combined effects of each of the embodiments that are combined.

[0070] Figure 14 is a flowchart showing the processing of the information processing system 1, combining the second, third, and fourth embodiments. In the processing shown in Figure 14, the processes of S22 in Figure 13 and S20 in Figure 11 are added between the processes of S12 and S14 in Figure 9. The process of S14a in Figure 13 is performed instead of the process of S14 in Figure 9, and the process of S16a in Figure 11 is performed instead of the process of S16 in Figure 9.

[0071] Following the processing in S12, the calculation unit 28 sums the amount of movement of the fingertip 50 in the Z direction (S22). The averaging unit 32 averages the X and Y coordinates of the fingertip 50 (S20). The order of processing in S22 and S20 may be reversed. The determination unit 30 determines whether the velocity in the Z direction is a positive velocity, whether the velocity in the Z direction is greater than or equal to the second velocity threshold Vth2, and whether the total amount of movement in the Z direction is greater than or equal to the total amount of movement threshold (S14a). If these conditions are not met (N in S14a), the processing of the information processing system 1 returns to the processing in S22 by the calculation unit 28. If the velocity in the Z direction is positive velocity, the velocity in the Z direction is greater than or equal to the second velocity threshold Vth2, and the total amount of movement in the Z direction is greater than or equal to the total amount of movement threshold (Y in S14a), the function execution unit 34 obtains the first average X coordinate and the second average Y coordinate of the fingertip 50 and executes processing (S16a).

[0072] (Fifth embodiment) In the fifth embodiment, the information processing system 1 detects an air-pushing motion in a different direction than in the first embodiment. The differences from the first embodiment will be explained below.

[0073] Figure 15 is a diagram illustrating an air-pushing motion for the information processing system 1 in the fifth embodiment. Figure 15(a) shows an example in which the information processing system 1 detects an air-pushing motion in a direction parallel to the X direction. In this example, the information processing system 1 detects an air-pushing motion parallel to either the X direction, the Y direction, or the Z direction.

[0074] The information processing system 1 further comprises a first speaker 16a, a second speaker 16b, a third speaker 16c, and a fourth speaker 16d. Hereafter, these speakers will be referred to as "speaker 16" unless otherwise distinguished. The first speaker 16a and the second speaker 16b are arranged side by side in the Y direction on the left side of the display 14. The third speaker 16c and the fourth speaker 16d are arranged side by side in the Y direction on the right side of the display 14. Speaker 16 is an example of an electronic device to be operated. The positional relationship between the display 14 and the three-dimensional Cartesian coordinate system is the same as, for example, the example shown in Figure 2.

[0075] In the illustrated example, the user performs a pressing motion by moving their fingertip 50 from right to left toward the first speaker 16a. Based on the change in the spatial coordinates of the fingertip 50, the calculation unit 28 identifies whether the direction of movement of the fingertip 50 is in the positive or negative direction of the X, Y, or Z axis. In this example, the calculation unit 28 identifies that the direction of movement of the fingertip 50 is in the negative direction of the X axis. Based on the spatial coordinates of the fingertip 50, the calculation unit 28 calculates the velocity of the fingertip 50 in the X direction in a time series. After the absolute value of the velocity of the fingertip 50 in the X direction becomes greater than or equal to the first velocity threshold Vth1, the function execution unit 34 executes processing corresponding to the pressing motion based on the Y and Z coordinates of the fingertip 50 at the time the velocity in the X direction reversed between the negative and positive directions. The processing in response to the pressing motion may include, for example, the process of turning on the power to the first speaker 16a located in the negative direction of the X axis and corresponding to the specified Y and Z coordinates.

[0076] When the user moves their fingertip 50 from right to left towards the second speaker 16b, the function execution unit 34 turns on the power to the second speaker 16b if the velocity of the fingertip 50 in the X direction satisfies the previously described conditions. In this example, the second speaker 16b is located in the negative direction of the X axis and is at a position corresponding to the Y and Z coordinates of the fingertip 50 when the velocity of the fingertip 50 in the X direction reverses between the negative and positive directions.

[0077] On the other hand, when the user moves their fingertip 50 from left to right towards the third speaker 16c, the calculation unit 28 determines, based on the change in the spatial coordinates of the fingertip 50, that the direction of movement of the fingertip 50 is the positive direction of the X axis. In this case, the calculated velocity of the fingertip 50 in the X direction is a positive value. The function execution unit 34 turns on the power to the third speaker 16c when the absolute value of the velocity of the fingertip 50 in the X direction becomes equal to or greater than the first velocity threshold Vth1, and the velocity in the X direction reverses between the negative and positive directions. In this example, the third speaker 16c is located in the positive direction of the X axis and is at a position corresponding to the Y and Z coordinates of the fingertip 50 when the velocity in the X direction reverses between the negative and positive directions.

[0078] Furthermore, if the user moves their fingertip 50 toward the display surface of the display 14, the calculation unit 28 determines that the direction of movement of the fingertip 50 is in the negative direction of the Z axis. In this case, the processing of the first embodiment is executed.

[0079] Thus, the information processing system 1 can detect an air-pushing motion parallel to any of the X, Y, or Z directions. Therefore, by performing an air-pushing motion, the user can operate not only electronic devices located directly in front of the user, but also electronic devices located to the left, right, up, or down.

[0080] Figure 15(b) shows an example in which the information processing system 1 detects air-pushing motions in directions that intersect the X, Y, and Z directions. In this example, the information processing system 1 detects air-pushing motions in any direction. The information processing system 1 further includes a display 14a. The display surface 15a of the display 14a intersects the X, Y, and Z directions.

[0081] It is assumed that the user moves their fingertip 50 parallel to the normal direction of the display surface 15a of the display 14a. The calculation unit 28 identifies the direction of movement of the fingertip 50 based on the change in the spatial coordinates of the fingertip 50 in a three-dimensional Cartesian coordinate system that is set in advance with respect to the display 14, and sets the direction parallel to the identified direction of movement as the first direction. The calculation unit 28 newly sets a second direction orthogonal to the first direction, and a third direction orthogonal to the first and second directions. In other words, the calculation unit 28 sets a new three-dimensional Cartesian coordinate system based on the direction of movement of the fingertip 50, which is different from the pre-set three-dimensional Cartesian coordinate system. In the illustrated example, the first direction is the normal direction of the display surface 15a. The second and third directions are, for example, directions parallel to the display surface 15a. The coordinate acquisition unit 26 calculates and acquires the time-series spatial coordinates of the user's fingertip 50 in the new three-dimensional Cartesian coordinate system. The calculation unit 28 calculates the velocity of the fingertip 50 in a first direction over time, based on the spatial coordinates of the fingertip 50 in a new three-dimensional Cartesian coordinate system. In this case, the calculated velocity of the fingertip 50 in the first direction is assumed to be a negative value. Here, for the first direction, the direction approaching the display surface 15a is considered negative, and the direction moving away from the display surface 15a is considered positive.

[0082] The function execution unit 34 executes processing corresponding to a pressing operation based on the coordinates of the fingertip 50 in the second and third directions at the time the velocity in the first direction reverses from negative to positive, after the velocity in the first direction is negative and the absolute value of the velocity in the first direction becomes greater than or equal to the first velocity threshold Vth1. The coordinates in the second and third directions are coordinates in a new three-dimensional Cartesian coordinate system. The processing corresponding to the pressing operation may include, for example, processing to display an image indicating that a position has been clicked at the specified coordinates in the second and third directions on the display surface 15a of the display 14a, which is parallel to the second and third directions.

[0083] Thus, the information processing system 1 can detect air-pressing motions in directions that intersect the X, Y, and Z directions. Therefore, the user can also operate electronic devices located to the lower left, lower right, upper left, or upper right of the user by performing air-pressing motions.

[0084] Furthermore, the fifth embodiment may be combined with at least one of the second, third, and fourth embodiments.

[0085] The present disclosure has been described above based on embodiments. These embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible for each component or combination of processing processes, and that such modifications are also within the scope of the present disclosure.

[0086] One aspect of this disclosure is as follows:

[0087] [Item 1] A coordinate acquisition unit that acquires the time-series spatial coordinates of an object to be detected, wherein the spatial coordinates include coordinates in a first direction, coordinates in a second direction intersecting the first direction, and coordinates in a third direction intersecting the first and second directions, After a predetermined first condition relating to the change in the coordinates in the first direction is met at the first time step, and a predetermined second condition relating to the change in the coordinates in the first direction is met at the second time step, a function execution unit executes processing on the electronic device being operated on based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time step and the second time step. A control device equipped with the following features. According to this embodiment, the user can perform the actions to determine the coordinates of the second and third directions in a short amount of time and intuitively.

[0088] [Item 2] A calculation unit that calculates the velocity of the object to be detected in the first direction based on the spatial coordinates, wherein the velocity is a negative velocity when the coordinates of the object to be detected in the first direction become smaller. A determination unit that determines whether the first condition is met, that the speed is a negative speed and the absolute value of the speed is greater than or equal to a first speed threshold which is a predetermined value, and determines whether the second condition is met, that the speed is a positive speed. Furthermore, If the determination unit determines that the first condition is met at the first time and the second condition is met at the second time, the function execution unit executes processing on the electronic device based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. The control device described in item 1. In this case, the coordinates can be intuitively determined by pushing the object to be detected in the air in a first direction and then returning it.

[0089] [Item 3] The determination unit determines whether the second condition, that the velocity is 0 or a velocity in the positive direction, is met. The control device described in item 2. In this case, the object to be detected can be pushed in the first direction and stopped, after which its coordinates can be determined.

[0090] [Item 4] The determination unit determines whether the second condition is met, which is that the speed is a speed in the positive direction and is equal to or greater than a predetermined second speed threshold. The control device described in item 2. In this case, the coordinates can be determined when the object to be detected is pushed in the first direction and then released.

[0091] [Item 5] A calculation unit that calculates the velocity of the object to be detected in the first direction based on the spatial coordinates, wherein the velocity is a negative velocity when the coordinates of the object to be detected in the first direction become smaller. A determination unit that determines whether the first condition is met, that the speed is a negative speed and the absolute value of the speed is greater than or equal to a first speed threshold which is a predetermined value, and determines whether the second condition is met, that the speed is a negative speed and the absolute value of the speed is less than or equal to a second speed threshold which is a predetermined value that is less than the first speed threshold, Furthermore, If the determination unit determines that the first condition is met at the first time and the second condition is met at the second time, the function execution unit executes processing on the electronic device based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. The control device described in item 1. In this case, the coordinates can be determined when the speed at which the object to be detected moves in the first direction is reduced.

[0092] [Item 6] The aforementioned electronic device is a display, The first direction is the normal direction to the display surface of the display. A control device as described in any one of items 2 through 5. In this case, the coordinates can be intuitively determined by pushing the object to be detected towards the display surface in mid-air.

[0093] [Item 7] The calculation unit periodically calculates a first movement amount, which is the amount of movement of the object to be detected per predetermined time in the second direction, based on the spatial coordinates. The determination unit determines whether the third condition is met, that the first displacement is less than or equal to a predetermined value, which is a first displacement threshold. If the determination unit determines that the first condition and the third condition are met at the first time and the second condition is met at the second time, the function execution unit executes processing on the electronic device based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. A control device as described in any one of items 2 through 6. In this case, it is possible to prevent actions that the user did not intend from being performed.

[0094] [Item 8] The calculation unit periodically calculates a second movement amount, which is the amount of movement of the object to be detected per predetermined time in the third direction, based on the spatial coordinates. The determination unit determines whether the fourth condition is met, that the second displacement is less than or equal to a predetermined value, which is the second displacement threshold. If the determination unit determines that the first condition, the third condition, and the fourth condition are met at the first time, and the second condition is met at the second time, the function execution unit executes processing on the electronic device based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. The control device described in item 7. In this case, it is possible to prevent actions that the user did not intend from being performed.

[0095] [Item 9] The system includes an averaging unit that calculates a first average coordinate obtained by averaging the coordinates in the second direction and a second average coordinate obtained by averaging the coordinates in the third direction over a predetermined period between the first time and the second time, based on the spatial coordinates. The function execution unit, when the determination unit determines that the first condition is met at the first time and the second condition is met at the second time, executes processing on the electronic device based on the first average coordinates and the second average coordinates. A control device as described in any one of items 2 through 6. In this case, the accuracy of coordinate calculation can be improved.

[0096] [Item 10] The calculation unit calculates the total movement amount, which is the sum of the movement amounts of the object to be detected in the first direction from the first time to the second time, based on the spatial coordinates. The determination unit determines whether the movement amount condition is met, which is that the total movement amount is equal to or greater than a predetermined total movement amount threshold. The function execution unit, when the determination unit determines that the first condition is met at the first time, the second condition is met at the second time, and the movement amount condition is met, executes processing on the electronic device based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. A control device as described in any one of items 2 through 6. In this case, it is possible to prevent unintended actions from being performed on electronic devices.

[0097] [Item 11] A step of obtaining the time-series spatial coordinates of an object to be detected, wherein the spatial coordinates include coordinates in a first direction, coordinates in a second direction intersecting the first direction, and coordinates in a third direction intersecting the first and second directions. If a predetermined first condition relating to the change in the coordinates in the first direction is met at the first time, and a predetermined second condition relating to the change in the coordinates in the first direction is met at the second time, the process of performing an operation on the electronic device to be operated on is performed based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. A control method comprising the following features. According to this embodiment, the user can perform the actions to determine the coordinates of the second and third directions in a short amount of time and intuitively. [Explanation of Symbols]

[0098] 1... Information processing system, 10... Sensor, 12... Control device, 14, 15, 18... Judgment unit, 20... Control unit, 22... Storage unit, 24... Image acquisition unit, 26... Coordinate acquisition unit, 28... Calculation unit, 30... Judgment unit, 32... Averaging unit, 34... Function execution unit, 36... Display output unit.

Claims

1. A coordinate acquisition unit that acquires the time-series spatial coordinates of an object to be detected, wherein the spatial coordinates include coordinates in a first direction, coordinates in a second direction intersecting the first direction, and coordinates in a third direction intersecting the first and second directions, After a predetermined first condition relating to the change in the coordinates in the first direction is met at a first time step, and a predetermined second condition relating to the change in the coordinates in the first direction is met at a second time step, a function execution unit executes processing on the display being operated on based on the coordinates in the second direction and the coordinates in the third direction at one or more time steps between the first and second time steps. A calculation unit that calculates the velocity of the object to be detected in the first direction based on the spatial coordinates, wherein the velocity is a negative velocity when the coordinates of the object to be detected in the first direction become smaller. A determination unit that determines whether the first condition is met, that the speed is a negative speed and the absolute value of the speed is greater than or equal to a predetermined first speed threshold, and determines whether the second condition is met, that the speed is a positive speed. Equipped with, The first direction is the normal direction to the display surface of the display, If the determination unit determines that the first condition is met at the first time and the second condition is met at the second time, the function execution unit executes processing on the display based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. Control device.

2. The determination unit determines whether the second condition, that the velocity is 0 or a velocity in the positive direction, is met. The control device according to claim 1.

3. The determination unit determines whether the second condition is met, which is that the speed is a speed in the positive direction and is equal to or greater than a predetermined second speed threshold. The control device according to claim 1.

4. A coordinate acquisition unit for acquiring the spatial coordinates of a time-series object to be detected, wherein the spatial coordinates include coordinates in a first direction, coordinates in a second direction intersecting the first direction, and coordinates in a third direction intersecting the first and second directions, After a predetermined first condition relating to the change in the coordinates in the first direction is met at a first time step, and a predetermined second condition relating to the change in the coordinates in the first direction is met at a second time step, a function execution unit executes processing on the display being operated on based on the coordinates in the second direction and the coordinates in the third direction at one or more time steps between the first and second time steps. A calculation unit that calculates the velocity of the object to be detected in the first direction based on the spatial coordinates, wherein the velocity is a negative velocity when the coordinates of the object to be detected in the first direction become smaller. A determination unit that determines whether the first condition is met, that the speed is a negative speed and the absolute value of the speed is greater than or equal to a first speed threshold which is a predetermined value, and determines whether the second condition is met, that the speed is a negative speed and the absolute value of the speed is less than or equal to a second speed threshold which is a predetermined value that is less than the first speed threshold, Equipped with, The first direction is the normal direction to the display surface of the display, If the determination unit determines that the first condition is met at the first time and the second condition is met at the second time, the function execution unit executes processing on the display based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. Control device.

5. The calculation unit periodically calculates a first movement amount, which is the amount of movement of the object to be detected per predetermined time in the second direction, based on the spatial coordinates. The determination unit determines whether the third condition is met, that the first displacement is less than or equal to a predetermined value, which is a first displacement threshold. The function execution unit, when the determination unit determines that the first condition and the third condition are met at the first time and the second condition is met at the second time, executes processing on the display based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. The control device according to any one of claims 1 to 4.

6. The calculation unit periodically calculates a second movement amount, which is the amount of movement of the object to be detected per predetermined time in the third direction, based on the spatial coordinates. The determination unit determines whether the fourth condition is met, that the second displacement is less than or equal to a predetermined value, which is the second displacement threshold. The function execution unit, when the determination unit determines that the first condition, the third condition, and the fourth condition are met at the first time, and that the second condition is met at the second time, executes processing on the display based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. The control device according to claim 5.

7. The system includes an averaging unit that calculates a first average coordinate obtained by averaging the coordinates in the second direction and a second average coordinate obtained by averaging the coordinates in the third direction over a predetermined period between the first time and the second time, based on the spatial coordinates. The function execution unit, when the determination unit determines that the first condition is met at the first time and the second condition is met at the second time, executes processing on the display based on the first average coordinates and the second average coordinates. The control device according to any one of claims 1 to 4.

8. The calculation unit calculates the total movement amount, which is the sum of the movement amounts of the object to be detected in the first direction from the first time to the second time, based on the spatial coordinates. The determination unit determines whether the movement amount condition is met, which is that the total movement amount is equal to or greater than a predetermined total movement amount threshold. The function execution unit, when the determination unit determines that the first condition is met at the first time, the second condition is met at the second time, and the movement amount condition is met, executes processing on the display based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. The control device according to any one of claims 1 to 4.

9. A step of obtaining the time-series spatial coordinates of an object to be detected, wherein the spatial coordinates include coordinates in a first direction, coordinates in a second direction intersecting the first direction, and coordinates in a third direction intersecting the first and second directions. After a predetermined first condition relating to the change in the coordinates in the first direction is met at a first time step, and a predetermined second condition relating to the change in the coordinates in the first direction is met at a second time step, an execution step is performed to perform processing on the display to be operated on, based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time step and the second time step. A step of calculating the velocity of the object to be detected in the first direction based on the spatial coordinates, wherein the velocity is in the negative direction when the coordinates of the object to be detected in the first direction become smaller. A determination step to determine whether the first condition is met, that the speed is a negative speed and the absolute value of the speed is greater than or equal to a predetermined first speed threshold, and whether the second condition is met, that the speed is a positive speed. Equipped with, The first direction is the normal direction to the display surface of the display, In the determination step, if it is determined that the first condition is met at the first time and the second condition is met at the second time, then in the execution step, processing is performed on the display based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. Control method.

10. A step of obtaining the spatial coordinates of a time series of an object to be detected, wherein the spatial coordinates include coordinates in a first direction, coordinates in a second direction intersecting the first direction, and coordinates in a third direction intersecting the first and second directions. After a predetermined first condition relating to the change in the coordinates in the first direction is met at a first time step, and a predetermined second condition relating to the change in the coordinates in the first direction is met at a second time step, an execution step is performed to perform processing on the display to be operated on, based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time step and the second time step. A step of calculating the velocity of the object to be detected in the first direction based on the spatial coordinates, wherein the velocity is in the negative direction when the coordinates of the object to be detected in the first direction become smaller. A determination step to determine whether the first condition is met, that the speed is a negative speed and the absolute value of the speed is greater than or equal to a first speed threshold which is a predetermined value, and whether the second condition is met, that the speed is a negative speed and the absolute value of the speed is less than or equal to a second speed threshold which is a predetermined value that is less than the first speed threshold, Equipped with, The first direction is the normal direction to the display surface of the display, In the determination step, if it is determined that the first condition is met at the first time and the second condition is met at the second time, then in the execution step, processing is performed on the display based on the coordinates in the second direction and the coordinates in the third direction at one or more time points between the first time and the second time. Control method.

Images