Program for operating the target device

Abstract

To provide a program for apparatus operation which has versatility and does not select the size of a device such as a touch panel to be used for operation.SOLUTION: A program for operating an operation object apparatus having a driving device causes an information processing device connected to a display device for receiving operation by an indication body or provided with the display device to function as operation information generation and holding means for generating operation information to operate the operation object apparatus on the basis of the operation by the indication body to the display device, and holding the operation information even in the case that the indication body does not operate the display device, and operation means for operating the operation object apparatus by transmitting the held operation information to the operation object apparatus. The operation information generation and holding means determines whether or not operation using the display device is performed during swipe, and holds current operation information to the operation object apparatus in the case of determining that the operation using the display device is not performed during the swipe.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a program for operating an operation target device, and more particularly, Operation using a finger or other indicator The present invention relates to a program for operating a target device. [Background technology]

[0002] Conventionally, when operating a target device such as a drone, there are cases where dedicated operating hardware (such as a controller) is used, and cases where the target device is operated using a touch panel (also called a touch screen; the same applies below), as described in Patent Documents 1 to 5.

[0003] Patent Document 1 discloses an operating device that externally controls a mobile device that holds an imaging means. The operating device described in Patent Document 1 has display means that displays an image captured by the imaging means, touch operation means that detects a touch operation, and instruction means that, in response to the touch operation detected by the touch operation means, moves the mobile device based on the operation direction of the move operation if the touch operation is a move operation. If the touch operation means detects a continuous touch-on state after the move operation, the instruction means continues the movement based on the move operation.

[0004] Patent Document 2 discloses an operating device that externally controls a mobile device that holds an imaging means. The operating device described in Patent Document 2 has display means that displays an image captured by the imaging means, touch operating means that detects touch operations, and instruction means that, in response to the touch operation detected by the touch operating means, moves the mobile device forward if the touch operation is a pinch-out operation, and moves the mobile device backward if the touch operation is a pinch-in operation. If the touch operating means detects that a touch-on state continues after a pinch-out operation or a pinch-in operation, the instruction means continues the movement based on the pinch-out operation or the pinch-in operation.

[0005] The mobile device operating terminal described in claim 4 of Patent Document 3 includes a display means for displaying an image captured by an imaging means of the mobile device on a screen, a receiving means for receiving an operation specifying a movement direction of the captured image, and a control means for moving the mobile device so that the movement direction is captured. The display means and the receiving means are touch panels. Based on a drag operation in which a finger touches and slides on the touch panel, the control means determines that the drag operation is a horizontal operation and rotates the mobile device if the angle of the sliding direction of the finger relative to the horizontal direction of the touch panel is within a certain angle, and determines that the drag operation is a vertical operation and raises or lowers the mobile device if the angle is not within the certain angle.

[0006] Paragraphs 0070 and 0071 of Patent Document 4 state the following: "As shown in Fig. 3b, to control the drone, the user moves their finger within the control icon 30, for example, to the upper left as indicated by the gray circle. According to this new position of the finger, the controller 15 generates commands that make the drone rise and simultaneously rotate the drone to the left. Therefore, the drone performs a slight rotation to the left while gaining altitude. To hold these control commands, the user applies a higher pressure with their finger to the touch-sensitive area of ​​the controller screen at the location of the control command they want to lock, so that these control commands continue to be applied to control the drone."

[0007] Patent Document 5 discloses an information processing device communicably connected to a display device and a mobile object. The information processing device includes a receiving means for receiving a first operation for determining at least two points on the screen of the display device, and a transmitting means for transmitting to the mobile object information for causing the mobile object to move along a predetermined pattern of route, excluding routes connecting the points with straight lines, based on the positional relationship of the points determined by the first operation received by the receiving means. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent No. 6855616 [Patent Document 2] Patent No. 6685742 [Patent Document 3] Patent No. 6470112 [Patent Document 4] Japanese Patent Application Laid-Open No. 2017-123148 [Patent Document 5] Patent Publication No. 2021-12612 Summary of the Invention [Problem to be solved by the invention]

[0009] Conventional operation controllers are dedicated to the device to be operated and lack versatility.

[0010] As described in Patent Documents 1 to 5, even when a touch panel is used to operate a device to be operated, the size of the touch panel that can be practically used for operation is limited.

[0011] In the operating device of Patent Document 1, if a touch-on state is continuously detected after a move operation, the instruction means continues the movement based on the move operation. In the operating device of Patent Document 2, if a touch-on state is continuously detected after a pinch-out operation or a pinch-in operation, the instruction means continues the movement based on the pinch-out operation or the pinch-in operation. Therefore, in the case of the operation devices of Patent Documents 1 and 2, a touch panel of a size that makes it easy to maintain a touch-on state must be used.

[0012] The mobile device operating terminal of Patent Document 3 determines whether the movement is horizontal or vertical depending on the angle of the drag operation. However, such an operation does not allow for detailed operation of the device to be operated.

[0013] In the device of US Pat. No. 6,299,523, the user must use his or her finger to exert more pressure on the touch-sensitive area of ​​the control screen at the location of the control command that the user wishes to lock. Therefore, a touch panel of a size that allows a finger to be pressed continuously against the touch sensitive area must be used.

[0014] The information processing device of Patent Document 5 is operated by determining at least two points on the screen. However, a touch panel of a size that allows for such two-point determination operation must be used.

[0015] As described above, in the devices described in any of Patent Documents 1 to 5, a touch panel having a certain size is required for operation. In addition, the operation patterns are limited. Furthermore, these devices require continuous finger contact to operate, and when a small touch panel is used, continuous finger contact becomes more difficult.

[0016] Therefore, the present invention is versatile and easy to operate. Rude To provide a program for operating equipment that does not require selection of the size of a device. [Means for solving the problem]

[0017] In order to solve the above problems, the present invention has the following features: The present invention is a program for operating an operation target device having a drive device, the program including: operation A computer that is connected to a display device that receives the operation and generating operation information for operating the device to be operated based on the Use Even if the device is not operated, the device functions as an operation means for operating the device to be operated by transmitting the operation information generated and stored in the operation information generating and storing means to the device to be operated. The operation information generating and storing means is characterized in that, when an operation using the display device is performed again, the generation of the operation information is resumed based on the latest operation information stored. It is a program.

[0018] Preferably, the operation information generating and storing means is configured to From operations using Determine whether or not the user has left the screen and whether the display is From operations using If it is determined that the user has left the device, the current operation information for the device to be operated is stored. The operation means is a display device. Operations using Even when the operation is not being performed, the currently held operation information is transmitted to the operation target device to operate the operation target device.

[0019] In one embodiment, the program further causes the computer to region In the above, an area for performing operations on the operation target device is defined as an operation area, and a predetermined range on the display device is defined as an operation area. Display area As a judgment frame, a virtual controller image is displayed on the display device. Therefore Displayed R The controller functions as a virtual controller image display means. The operation information generating and storing means stores the operation information even if the operation by the indicator starts from the operation area and goes beyond the operation area. Display area Operation information is generated until the judgment frame is reached.

[0020] In one embodiment, the program further divides the computer into at least a first area for performing a first operation on the operation target device and a second area for performing a second operation on the operation target device, and displays a virtual controller image on the display device. Therefore Displayed R The controller functions as a virtual controller image display means. The operation information generating and storing means generates operation information for a first operation based on an operation by the indicator in the first area, and generates operation information for a second operation based on an operation by the indicator in the second area.

[0021] In one embodiment, the operation information generating and storing means generates operation information as an operation in the first area, even if the operation by the indicator starts in the first area, goes beyond the first area, and reaches the second area.

[0022] In one embodiment, a predetermined Display area Indicator outside the judgment frame Operation by If it is determined that the user has moved away, the operation information generating and holding means updates the current operation information to be operation information of a higher strength. To accept operations To the operation input section input When the above operation is not being performed, the operation means operates the operation target device based on the updated current operation information, and displays the display device. Operations using If this is done again, Operation resumed The operation of the device to be operated is resumed based on the current operation information stored as operation information in the area to which the position belongs.

[0023] In one embodiment, a predetermined Display area Outside the judgement box The operation by the indicator If it is determined that the person has left, and , away If the moving direction when the operation information is input is a predetermined downward direction, the operation information generating and storing means , away The current operation information generated based on the selected indicator can be updated as operation information of a next lower strength. display device Operations using When the above operation is not being performed, the operation means operates the operation target device based on the updated current operation information. display device Operations using When the operation is performed again, the operation means Operation resumed The operation of the device to be operated is resumed based on the current operation information stored as operation information in the area to which the position belongs.

[0024] In one embodiment, the program causes the computer to function as upward flick determination means that determines whether or not an operation on the display device is a predetermined upward flick. When it is determined that the flick is in the predetermined upward direction, the operation information generating and holding means updates the current operation information as operation information of a higher strength. To accept operations To the operation input section input When the above operation is not being performed, the operation means operates the operation target device based on the updated current operation information. To the operation input section input When the operation is performed again, the operation means The input was The operation of the device to be operated is resumed based on the current operation information stored as operation information in the area to which the position belongs.

[0025] In one embodiment, the program causes a computer to function as tap / downward flick determination means that determines whether an operation on the display device is a tap or a predetermined downward flick. When it is determined that the input is a tap or a flick in a predetermined downward direction, the operation information generating and storing means , away The current operation information generated based on the pointer that has been moved is updated as operation information of a next lower strength. To accept operations To the operation input section input When the above operation is not being performed, the operation means operates the operation target device based on the updated current operation information. To the operation input section input When the operation is performed again, the operation means The input was The operation of the device to be operated is resumed based on the current operation information stored as operation information in the area to which the position belongs.

[0026] In one embodiment, the program causes the computer to function as double tap determination means for determining whether an operation on the display device is a double tap. When it is determined that the touch is a double tap, the operation information generating and storing means The input was The information held as operation information for the area to which the position belongs is discarded. The operation means is for the operation target device, The input was Perform an operation to stop movement within the area.

[0027] In one embodiment, the program causes the computer to function as stop button determination means for determining whether the operation on the display device is pressing the stop button. If it is determined that the stop button has been pressed, the operation information generating and holding means executes a stop command in a pre-specified area and discards the information held in each area. The operation means performs an operation to stop movement of the device within a pre-specified area, or, if the pre-specified area is the entire area, performs an operation to stop the device to be operated or make it wait in place.

[0028] In one embodiment, the operation information generating and storing means generates the operation information based on a two-dimensional vector of a swipe made by the indicator on the display device.

[0029] In one embodiment, the operation information generating and storing means generates the operation information based on the x component and the y component of the two-dimensional vector.

[0030] In one embodiment, when the device to be operated is a drone, the operation information generating and storing means generates operation information that indicates the direction and strength of the ladder and elevator using the x component and y component.

[0031] In one embodiment, when the device to be operated is a drone, the operation information generating and storing means generates operation information that indicates the direction and strength of the aileron and throttle using the x component and y component.

[0032] In one embodiment, when the device to be operated is a drone, the operation information generating and storing means generates operation information that indicates the direction and strength of any two of the rudder, elevator, aileron, and throttle using the x component and y component.

[0033] In one embodiment, the program further divides the computer into a first area for performing two operations among the rudder, elevator, aileron, and throttle when the device to be operated is a drone, and a second area for performing two operations other than the operations in the first area, and displays a virtual controller image on the display device. Therefore, it is displayed It functions as a virtual controller image display means. The operation information generating and storing means generates operation information for the first area based on a swipe operation by the indicator in the first area, and generates operation information for the second area based on a swipe operation by the indicator in the second area. [Effects of the Invention]

[0034] According to the present invention, the generated operation information is stored, and the indicator is displayed on the display device. No operations using Even in this case, the device to be operated is operated based on the stored operation information. This allows operation using software, eliminating the need for dedicated operating hardware, and providing a versatile operating program.

[0035] The indicator is a display device No operations using Even in the case of a display device, operation is possible, so for operation, a pointer is used. Perform an operation using Therefore, even if the display device is small, it is easy to operate the target device. A program that can be operated regardless of the size of the display device is provided.

[0036] One display region It is also possible to arrange multiple virtual controller images on the display, making it possible to set multiple operability and operation patterns on a single display device. It is also possible to select from the multiple operation patterns set or use them together, allowing for a variety of operations to be realized.

[0037] Operation area and Display area By setting a judgment frame, even if you go outside the operation area, Display areaIf you generate operation information until the judgment frame is reached, even if the operation area where you start the operation is small, Display area Therefore, it is possible to use the entire judgment frame for operation. Therefore, not only one virtual controller image but also multiple virtual controller images can be appropriately arranged, Display area This allows for operations such as using the entire screen, making it possible to provide a variety of operation patterns.

[0038] display region is divided into a first area and a second area, a virtual controller image is arranged in each area, and operation information is generated by separating the first operation in the first area from the second operation in the second area, thereby making it possible to provide a variety of operation patterns.

[0039] Even if the operation of the pointer starts from the first region, goes beyond the first region, and reaches the second region, if operation information is generated as an operation in the first region, Display area The first operation is possible by making extensive use of

[0040] While swiping, Display area If the finger is removed from the detection frame, the strength will be increased to the next level, and the operation of the target device will continue. operation In this case, it is possible to continue the operation at a higher strength.

[0041] While swiping, move downwards to Display area When the finger leaves the detection frame, the strength is reduced to the next level and the operation of the target device continues. operation In this case, operation can be continued at a lower strength.

[0042] When a specified upward flick is performed, the strength increases and the operation of the target device continues. operation In this case, it is possible to continue the operation at a higher strength.

[0043] When a tap or a specific downward flick is performed, the strength is reduced to one level, and the movement display of the device to be operated continues. operationIn this case, operation can be continued at a lower strength.

[0044] When a double tap or a stop button is pressed, the device to be operated can be stopped or put on standby at the same time.

[0045] By generating operation information based on the two-dimensional vector of the swipe, accurate operation information can be generated.

[0046] For example, if control information is generated based on the x and y components of a two-dimensional vector, it becomes possible to control a drone: the direction and strength of the rudder, elevator, aileron, and throttle can be determined by the x and y components.

[0047] By dividing the virtual controller images corresponding to the rudder, elevator, aileron, and throttle into a first area and a second area, it becomes possible to arrange the virtual controller images for operating the drone on the display device. Display area The effect of providing a judgment frame is the same as that described above.

[0048] These and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0049] [Figure 1] FIG. 1 is a block diagram showing the functional configuration of an information processing device 1 according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the operation target device 10 according to an embodiment of the present invention. [Figure 3] FIG. 3 is a flowchart showing an outline of main processes of the information processing device 1 and the operation target device 10 when swiping is taken as an example. [Figure 4] FIG. 4 is a diagram showing an outline of the processing of the information processing device 1 and the operation target device 10. As shown in FIG. [Figure 5] FIG. 5 is a flowchart showing a process for establishing communication between the information processing device 1 and the operation target device 10. In FIG. [Figure 6] FIG. 6 is a flowchart showing the process of checking the communication status in the information processing device 1. [Figure 7] FIG. 7 is a flowchart showing the behavioral processing of the operation target device 10. [Figure 8] FIG. 8 is a flowchart showing the process of transmitting and receiving various types of information in the information processing device 1. [Figure 9] FIG. 9 is a flowchart showing the operation process of the controller in the information processing device 1. [Figure 10] FIG. 10 is a diagram for explaining a display example of the display area and the operation when connecting to the operation target device 10. In FIG. [Figure 11A] FIG. 11A is a diagram for explaining a display example and operation of the display area when the lower part of the boundary line is swiped. [Figure 11B] FIG. 11B is a diagram for explaining a display example and operation of the display area when a swipe crosses the boundary line and when a swipe reaches the display area determination frame. [Figure 11C] FIG. 11C is a diagram for explaining a display example and operation of the display area when the swipe is made to go beyond the display area determination frame and when the operation is made again to an arbitrary position below the boundary line. [Figure 11D] FIG. 11D is a diagram for explaining a display example of the display area and an operation when the finger is released after swiping. [Figure 11E] FIG. 11E is a diagram for explaining a display example and operation of the display area when the user performs a swipe again on the bottom of the boundary line. [Figure 11F] FIG. 11F is a diagram for explaining a display example and operation of the display area when the finger is released after swiping and when the operation is performed again above the boundary line. [Figure 11G] FIG. 11G is a diagram for explaining a display example and operation of the display area when the upper part of the boundary line is swiped and when the display area determination frame is reached. [Figure 11H] FIG. 11H is a diagram for explaining a display example and operation of the display area when the display area goes outside the display area determination frame and when an operation is performed again above the boundary line. [Figure 11I] FIG. 11I is a diagram for explaining a display example and operation of the display area when the upper part of the boundary line is swiped and when the display area determination frame is reached. [Figure 11J] FIG. 11J is a diagram for explaining a display example and operation of the display area when the display area goes outside the display area determination frame and when an operation is performed again above the boundary line. [Figure 12] FIG. 12 is a diagram for explaining a display example and operation of the display area when tapped. [Figure 13A] FIG. 13A is a diagram for explaining a display example and operation of the display area when double-tapping is performed. [Figure 13B] FIG. 13B is a diagram for explaining a display example of the display area and an operation when the stop button is tapped. [Figure 14] FIG. 14 is a diagram for explaining a display example and operation of the display area when flicked upward. [Figure 15] FIG. 15 is a diagram for explaining a display example and operation of the display area when flicked downward. [Figure 16] FIG. 16 is a diagram for explaining the assignment of the operation area. [Figure 17A] FIG. 17A is a diagram for explaining an application example of the present invention. [Figure 17B] FIG. 17B is a diagram for explaining an application example of the present invention. [Figure 17C] FIG. 17C is a diagram for explaining an application example of the present invention. [Figure 17D] FIG. 17D is a diagram for explaining an application example of the present invention. [Figure 17E] FIG. 17E is a diagram for explaining an application example of the present invention. [Figure 18] FIG. 18 is a diagram for explaining the versatility of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0050] 1, information processing device 1 is a computer including a control unit 2, a storage unit 3, a display device 4, a communication unit 5, and an input unit 6. Note that display device 4 may be included in information processing device 1, or may be a separate entity from information processing device 1 and connected to information processing device 1 by wire and / or wirelessly. Note that storage unit 3 may be included in information processing device 1, or may be a separate entity from information processing device 1 and connected to information processing device 1 by wire and / or wirelessly or via a network. Note that storage unit 3 also includes external storage connected to a network.

[0051] The storage unit 3 stores a program (referred to as an "operation program") for operating the operation target device 10 and operation information for each controller. Here, the controller refers to at least one controller displayed on the display device 4, and is also called a virtual controller.

[0052] The control unit 2 reads and executes an operation program stored in the storage unit 3, and executes processes related to the operation of the operation target device 10 and processes related to display.

[0053] The display device 4 includes a display unit 4a and an operation input unit 4b. The display device 4 may be, for example, what is generally called a touch panel, but is not limited to this. The display unit 4a is a liquid crystal panel, an organic EL panel, or the like. The operation input unit 4b is a device that accepts input from the user using a finger, a touch pen, or the like. operation Accept.

[0054] The input unit 6 is a device for receiving inputs other than those from the operation input unit 4b.

[0055] The communication unit 5 is a device for communicating with communication devices such as Wi-Fi (registered trademark) devices and Bluetooth (registered trademark) devices. The communication unit 5 can communicate with the communication unit 15 of the operation target device 10 directly or via a network.

[0056] As shown in FIG. 2, the operation target device 10 includes a drive unit 11, a control unit 12, a storage unit 13, and a communication unit 15.

[0057] The driving unit 11 is a driving device for operating the operation target device 10. The storage unit 13 stores programs and data required for the control unit 12 to control the drive unit 11 . The communication unit 15 can communicate with the communication unit 5 of the communication device 1 directly or via a network.

[0058] For example, if the operation target device 10 is a drone, the drive unit 11 is made up of a plurality of motors and propellers connected thereto. The control unit 12 controls the rotational speeds of multiple motors based on operation information from the information processing device 1, thereby enabling the target device 10 to ascend / descend (throttle), turn left / right (rudder), move forward / backward (elevator), and move left / right in parallel (aileron).

[0059] Here, an example of operation information when operating a drone will be described. The following components are used to operate the drone: 1. Ascent / Descent (Throttle) 2.Rudder 3. Forward / backward (elevator) 4. Left and right parallel movement (aileron)

[0060] display region By swiping the controller displayed in the figure, a two-dimensional vector is obtained depending on the direction and length of the swipe. The direction of the swipe corresponds to the direction of the two-dimensional vector, and the length of the swipe corresponds to the magnitude of the two-dimensional vector.

[0061] When operating a drone, the four components 1 to 4 above are divided into two two-dimensional vectors, and the drone is controlled by manipulating each of the two-dimensional vectors. For example, display regionLet's say we set a boundary line and divide it into two controllers, "Top of boundary line" and "Bottom of boundary line." In this case, we assign the strengths corresponding to 1 to 4 above to the two controllers, "Top of boundary line" and "Bottom of boundary line," as two-dimensional vector information, as shown below.

[0062] A. Up, down, left and right movement (operated with the controller above the border) X-axis direction: Strength of left and right translation (aileron) Y-axis direction: Ascent / Descent (throttle) strength B. Forward and backward turning movement (operated with the controller below the boundary line) X-axis direction: Rudder strength Y-axis direction: Forward / backward (elevator) strength In this case, the intensity in the X-axis direction is the magnitude of the x-component of the two-dimensional vector, and the intensity in the Y-axis direction is the magnitude of the y-component of the two-dimensional vector.

[0063] Although the above allocations are given as an example, the above 1 to 4 may be allocated anywhere in the X-axis direction and Y-axis direction of A, and the above X-axis direction and Y-axis direction of B.

[0064] For example, a case where a swipe is performed using the controller at the bottom of the border will be described. At this time, the turning operation is performed in accordance with the strength of the X-axis component, and if the X-axis component is positive, it is a right turn, and if the X-axis component is negative, it is a left turn. At the same time, the operation is to move forward or backward according to the strength of the Y-axis component; if the Y-axis component is positive, it is to move forward, and if the Y-axis component is negative, it is to move backward. In addition, Display area The right side of the graph is the positive X axis, the left side is the negative X axis, the top is the positive Y axis, and the bottom is the negative Y axis, but this is not limited to this and the reverse relationship is also possible. Display area The orientation of the image may be changed as appropriate.

[0065] For example, if the controller at the bottom of the boundary line is used and swiped to the upper right, the "X-axis component is positive and the Y-axis component is positive," which results in an operation of "moving forward while turning right," and the target device 10 (such as a drone) moves in a clockwise circular motion when viewed from above. Similarly, if the controller is swiped to the upper left, the "X-axis component is negative and the Y-axis component is positive," which results in an operation of "moving forward while turning left." If the controller is swiped to the lower right, the "X-axis component is positive and the Y-axis component is negative," which results in an operation of "moving backward while turning right." If the controller is swiped to the lower left, the "X-axis component is negative and the Y-axis component is negative," which results in an operation of "moving backward while turning left." The radius of the turning circle varies depending on the strength of each of the X-axis and Y-axis components.

[0066] If you swipe completely upward, the X-axis component will be 0 and the Y-axis component will be positive, which means you will move forward. If you swipe completely downward, the X-axis component will be 0 and the Y-axis component will be negative, which means you will move backward. If you swipe completely right, the X-axis component will be positive and the Y-axis component will be 0, which means you will turn right. If you swipe completely left, the X-axis component will be negative and the Y-axis component will be 0, which means you will turn left.

[0067] Based on this example, the operation information will be described. The operation information is information for issuing an operation command to the operation target device 10. The operation information includes information stored in a first memory frame and information stored in a second memory frame for each controller.

[0068] In the first memory frame, "operation strength", "operation strength level", "basic operation strength", and "movement direction" are stored as "current operation information". The "Movement direction" in "Current operation information" represents the direction of the two-dimensional vector when swiping. The "operation strength" of the "current operation information" represents the magnitude of a two-dimensional vector when swiping. The "operation strength" is stored in a format suitable for the operation content of the operation target device 10, and may be stored as the magnitude of the two-dimensional vector itself, or may be decomposed into the magnitude of the X-axis component and the magnitude of the Y-axis component and represented, or may be stored in another format. The "operation strength level" in the "current operation information" indicates the level of operation strength. The "basic operation strength" in the "current operation information" indicates the basic operation strength in the "operation strength level." The second memory frame stores a "basic operation strength" for each operation strength level.

[0069] The operations of the information processing device 1 and the operation target device 10 will be described below. First, an outline of main processes of the information processing device 1 and the operation target device 10 when swiping is taken as an example will be described with reference to FIG.

[0070] First, the information processing device 1 requests a communication connection from the operation target device 10 (S100). The operation target device 10, which has been waiting in a communication-enabled state (S120), receives the communication connection request from the information processing device 1 and accepts the connection (S121). The information processing device 1 confirms the connection acceptance from the operation target device 10 and establishes the connection (S101).

[0071] 10, when connecting to the operation target device 10, Display area As shown in Figure 10, same area When you press the "Connect" button at the bottom right of the lower controller in the "Operation Terminal" box in the left frame, you will be connected to the device 10 to be operated, such as a drone or radio-controlled car, via wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). same areaThe "field" range shown in the "Operation Target" box on the right side of the figure is the range where wireless connection is possible. The "field" range can also be expanded using a repeater. If a range is specified within the connectable range, the range specification takes priority. However, the connectable range does not limit the present invention.

[0072] It is assumed that the information processing device 1 and the operation target device 10 are constantly checking the connection state (S102) and sending a connection response (S122).

[0073] Once the connection has been established in this way, the user presses the Display area Any position of Use your fingers to operate (S103). The information processing device 1 the The controller to be used is determined according to the position (S104). Then, when a swipe is performed (S105), the information processing device 1 transmits operation information corresponding to the swipe to the operation target device 10 (S106).

[0074] The operation target device 10 receives the operation information (S123) and classifies the operation information, such as the recognition of the device to be operated, such as the motor, and the processing speed (S124). Each device corresponding to the operation executes processing (S125).

[0075] For example, as shown in Figure 11A, Display area Below the left border of same area bottom area Operate with Then, the information processing device 1 same area The target device 10 is operated using the controller assigned to the lower area.

[0076] Here, we will use drones as an example. Display area The lower area is a virtual controller for turning the operation target device 10 left and right (rudder) and moving forward and backward (elevator). same areaIn the lower area, a virtual controller, for example, as shown in the figure, is used as a virtual controller image, which represents a movement direction. The virtual controller image shown here is merely an example and does not limit the present invention. As described above, the target device 10 turns left and right, moves forward, and moves backward based on the direction and magnitude of the two-dimensional vector of the swipe.

[0077] Display area In the "top view" on the right side of the screen, the position of the device to be operated 10 may be displayed superimposed on an image captured by the device to be operated 10.

[0078] After swiping, your finger Display area It is assumed that the judgment frame has been reached (S107). Display area The judgment frame is a predetermined frame. In the example of FIG. 11A, operation If it starts, as shown, Display area The top, bottom, left and right frames of Display area This becomes the judgment frame. From the top of the boundary line operation Even if it starts Display area The top, bottom, left and right frames of Display area Here, the boundary line is Display area Although it is not a judgment frame, the boundary line Display area It may also be used as a judgment frame. The expression "display area judgment frame" may be replaced with "screen edge judgment frame." The term "display area" may be replaced with "screen." The expression "operation" may also be expressed as "contact." In the drawings, the expressions "screen edge determination frame" and "contact" are used in some places, but what they actually mean are "display area determination frame" and "operation."

[0079] And the fingers, Display area It is assumed that the operation has gone outside the judgment frame (S108). The information processing device 1 updates the "basic operation strength" to the "operation strength before going outside the judgment frame" (S109), adds the "operation strength level" (S110), and maintains the "operation strength" as it is (S111). Then, the information processing device 1 returns to the operation of S103. This way, if a swipe is made, operationA controller according to the position is used, operation information according to the operation is transmitted to the operation target device 10, and the operation target device 10 is operated.

[0080] If the communication is to be disconnected (S112), the information processing device 1 transmits a communication disconnection execution to the operation target device 10 (S113). The operation target device 10 receives the disconnection instruction (S126), performs pre-disconnection processing (S128), and then disconnects the communication (S129). Also, if a cause for communication disconnection occurs on the operation target device 10 side (S127), the operation target device 10 also performs pre-disconnection processing (S128), and then disconnects the communication (S129).

[0081] Furthermore, if it becomes impossible to check the communication status due to factors such as a lack of battery or distance (if a communication timeout occurs) (S114), the information processing device 1 returns to the operation of S100.

[0082] Next, the outline of the processing executed in the information processing device 1 and the operation target device 10 will be described with reference to FIG. The information processing device 1 and the operation target device 10 execute a communication establishment process so as to enable two-way communication between them (S301). After the communication is established, the information processing device 1 executes a communication status check process as needed (S302). The operation target device 10 receives the operation information from the information processing device 1 and executes a process (action process) according to the operation information (S303). The information processing device 1 executes a process of transmitting and receiving various types of information (S304). The information processing device 1 executes the operation process of the controller and generates operation information obtained by the operation of the user (S305).

[0083] The process of establishing communication between the information processing device 1 and the operation target device 10 will be described with reference to FIG. The information processing device 1 requests a communication connection to the operation target device 10 (S401). The communication method at this time complies with each communication standard being used (the same applies below). Then, the information processing device 1 waits for reception (S402).

[0084] When the operation target device 10 receives a request from the information processing device 1 while waiting in a communication-enabled state (S501), the operation target device 10 accepts the connection (S502) and starts transmission and reception (S503). The operation target device 10 transmits the communication status to the information processing device 1 (S504).

[0085] The information processing device 1 determines whether or not the communication status from the operation target device 10 has been received (S403). If the communication status has not been received, the information processing device 1 waits (S402). If the communication status has been received, the information processing device 1 establishes a communication connection with the operation target device 10 (S404).

[0086] The communication status confirmation process will be described with reference to FIG. The information processing device 1 checks the communication status with the operation target device 10 (S601). If the check is possible (Yes in S602), the information processing device 1 determines that the device is connected (S603) and returns to S601. If the check is not possible (No in S602), the information processing device 1 disconnects the communication (S604) and returns to the communication establishment process (S301). In this way, the information processing device 1 constantly checks whether communication with the operation target device 10 is maintained.

[0087] The behavioral processing of the operation target device 10 will be described with reference to FIG. The operation target device 10 receives operation information transmitted from the information processing device 1 (S701) and responds to the reception (S702). The operation target device 10 determines whether the received operation information is appropriate (S703). If the received operation information is not appropriate, the operation target device 10 skips the process and returns to S701 (S704). If the received operation information is appropriate, the operation target device 10 executes a process such as controlling the driving device 11 in accordance with the operation information (S705).

[0088] The transmission and reception process of various types of information will be described with reference to FIG. First, the information processing device 1 displays an image of a controller (a virtual controller image) on the display device (S800). The user operates the controller with a pointer such as a finger. In response to the operation, operation information is generated and stored according to the process shown in FIG. 9, which will be described later. The information processing device 1 sets a predetermined range Display area A virtual controller image is generated as a judgment frame. Furthermore, the information processing device 1 may cause the display device 4 to display an image that indicates the position of the device 10 to be operated.

[0089] In addition, when operating a drone, as an example, for example, as shown in FIG. 11A, the upper part of the boundary line may be a virtual controller image for operating the ailerons and throttle, and the lower part of the boundary line may be a virtual controller image for operating the rudder and elevator. Needless to say, it is also possible to provide respective virtual controller images on the left and right of the boundary line. Also, as shown in the "top view" of FIG. 11A, the information processing device 1 displays an image on the display device 4 that shows the position of the device 10 to be operated. In this way, when the device 10 to be operated is a drone, the display on the display device 4 region It is advisable to separate the first area for operating the rudder and elevator from the second area for operating the ailerons and throttle. Note that the first area and the second area are relative terms, so the first area may be called the second area and the second area may be called the first area.

[0090] The information processing device 1 transmits the operation information input to the controller to the operation target device 10 (S801), and determines whether the transmission was successful (S802). If the transmission was successful (Yes in S802), the information processing device 1 waits for a specified time (S803). If the transmission failed (No in S802), the information processing device 1 determines whether the transmission failed a specified number of times (S804). If the transmission failed the specified number of times (Yes in S804), the information processing device 1 disconnects communication with the operation target device 10 (S805) and returns to the communication establishment process (S301). If the transmission has not failed the specified number of times (No in S804), the information processing device 1 returns to the process of S801.

[0091] In this way, the information processing device 1 functions as an operating unit that operates the operation target device 10 by transmitting transmission information to the operation target device 10. The information processing device 1 transmits the operation information stored therein to the operation target device 10. Therefore, the indicator for the display device 4 Operation with Even when this is not being done, the information processing device 1 can operate the operation target device 10.

[0092] In addition to transmitting the above operation information, the information processing device 1 receives the status of the operation target device 10 (S806) and determines whether the reception was successful (S807). If the reception was successful (Yes in S807), the information processing device 1 waits for a specified time (S808). If the reception failed (No in S807), the information processing device 1 determines whether the reception failed a specified number of times (S809). If the reception failed a specified number of times (Yes in S809), the information processing device 1 disconnects communication with the operation target device 10 (S810) and returns to the communication establishment process (S301). If the reception has not failed a specified number of times (No in S809), the information processing device 1 returns to the process of S806.

[0093] Using FIG. 9, the flow of generating and storing operation information by the operation processing of the controller will be explained. Operate with However, other pointers such as a touch pen can also be used. Operate with The same processing flow will be followed even if Display areaThe explanation will be given assuming that each controller has a designated area of ​​responsibility. region All of these may be assigned to one controller as their respective areas of responsibility.

[0094] Point to any position within the area of ​​responsibility Operate with If so (S201), the control unit 2 updates the "operation strength" of the "current operation information" to the "basic operation strength" (S202).

[0095] There are two timings for starting the operation of S201. The first is when it is the first operation or when the "current operation information" has been reset. The second is when the information before the finger was released is retained in the "current operation information."

[0096] In the first case, in the operation of S202, the "operation strength" of the "current operation information" is set to a default value (for example, "0").

[0097] In the second case, the "operation strength" before the finger is released is stored as the "basic operation strength" by the operation of S212, S217, or S223 described later. In the second case, the "operation strength" of the "current operation information" is updated to the stored "basic operation strength" by the operation of S202. Display area Finger Operate with If you do, the operation you performed just before your finger was released will continue.

[0098] In the embodiment of the present invention, one or more areas of the virtual controller image are provided. Operation Determine the position and place The current operation information stored as the operation information in the area to which the device belongs is operation This will be used for subsequent operations. operation If the operation is completed, the operation of the operation target device 10 is resumed based on the current operation information stored in the first area, and the operation is resumed in the second area described later. operationIf so, the operation of the operation target device 10 is resumed based on the current operation information stored for the second area. operation The current operation information used in this case is the same for all of the cases of swiping, flicking up, flicking down, and double-touching. operation In this case, Re-operation It is assumed that the operation is resumed based on the operation information stored for the area corresponding to the position, and a description of each step will be omitted.

[0099] Then, it is determined whether the operation performed was a swipe, an upward flick (indicated as "upward flick" in the top of Figure 9), a tap or downward flick (indicated as "downward flick" in the top of Figure 9), a double tap, or a press of the stop button (S203).

[0100] The upward direction that serves as the criterion for determining an upward flick may be set to a predetermined angle as appropriate. Display area The upward direction does not necessarily have to be completely parallel to the side surface of the device, but refers to an upward direction within a predetermined angle range.

[0101] The downward direction, which is the criterion for determining a downward flick, may also be set to an angle determined in advance. Display area The downward direction does not necessarily have to be completely parallel to the side surface of the device, but refers to a downward direction within a predetermined angle range.

[0102] Furthermore, since up and down are determined based on the positional relationship with the display device, the orientation and tilt direction of the display device, etc., the up direction and down direction can be conceptually expressed as the first direction and the second direction.

[0103] The difference between a swipe and a flick may be appropriately defined based on known techniques, as long as the two are distinguishable from each other.

[0104] Furthermore, even if swiping is expressed as sliding or scrolling, it is considered to be included in the term swiping.

[0105] What is a tap? Display area A double tap is an operation to lightly touch the screen. Display area This is an operation to lightly touch twice in succession.

[0106] <Swipe action> In S203, when the control unit 2 determines that the operation is a swipe (S204), the control unit 2 increases the "operation strength" of the "current operation information" according to the length of the swipe (S208), and changes the "movement direction" of the "current operation information" according to the direction of the swipe (S209). operation In this case, the operation strength may be temporarily returned to the basic operation strength and then the increase in the strength by the swipe may be resumed, or the increase in the strength by the swipe may be resumed from the current operation strength that is held. operation In this case, the information processing device 1 can generate new operation information based on the operation information that has been stored, and the strength at which operation is restarted is arbitrary.

[0107] In addition, when I actually tried to test the behavior, operation The advantage of returning to the basic operation strength and then restarting the increase in strength by swiping when the touch screen is touched has been confirmed, and will be explained below. Of course, this is not intended to limit the scope of the present invention.

[0108] For example, suppose the basic operation strength of the n-1 level (referred to as the "default of the n-1 level") is 10, and the basic operation strength of the n level (referred to as the "default of the n level") is 300.

[0109] Suppose, again operation Sometimes, again operationConsider the case where you resume increasing the strength by swiping from the current operation strength that was saved before the swipe. In this case, while in the n-th stage, you can return to the default n-1 stage by tapping as described below, but you cannot return to the default n-th stage. Also, to return to the default n-th stage, you must advance to the n+1th stage and then tap. In addition, if you tap while in the n-th stage, you will return to the default n-1 stage, which will likely result in extreme slowdown and make operation more difficult. Furthermore, if you tap again while in the n-1th stage, operation By repeating this, the strength can be increased. Display area The size of the judgment frame cannot limit the amount of strength addition, and the difference in strength between each operation strength level becomes large, resulting in a decrease in operation accuracy. Also, it serves as the default speed of the basic operation strength for operations of the same strength level, and as a limit to the strength addition range at the same strength level. Display area The role of the judgment frame disappears.

[0110] On the other hand, operation Sometimes, you can go back to the basic operation strength and increase the strength by swiping. If you want to resume Since it starts from the n-level default, it is possible to realize operations by selectively using the n-1 level default, the n level default, and the n+1 level default, which has the advantage of being easy to operate.

[0111] As mentioned above, in the example of operating a drone, the control unit 2 recognizes the swipe as a two-dimensional vector, the direction of the two-dimensional vector is the "movement direction," the magnitude of the x-component of the two-dimensional vector is the "operation strength" of the ladder, and the magnitude of the y-component is the "operation strength" of the elevator.

[0112] Next, the control unit 2 Display area It is determined whether or not the finger has been removed (S210). same area Needless to say, an interrupt process when the finger is removed from the same area While the finger is not removed from the touch panel, that is, while the swipe continues, the control unit 2 executes the operations of S208 and S209. same area When you release your finger, the control unit 2 Display areaIt is determined whether or not the object is outside the determination frame (S211).

[0113] Here, " Display area The "judgment frame" may be the area of ​​responsibility for the operation, or it may be the range beyond the area of ​​responsibility. Display area It may also be used as a judgment frame, Display area The whole Display area It may also be used as a judgment frame.

[0114] In any of the above cases, regardless of whether or not there is a virtual controller image, object, or other Display area Alternatively, some display portions may be configured not to accept any operation. same area The present invention does not limit in any way which area of ​​the screen accepts an operation.

[0115] In S210, the finger Display area The determination of whether or not the vehicle has left the area is based on S211. Display area This is before the judgment of whether the finger is released outside the judgment frame. same area The phenomenon of being separated from Display area If the finger is released inside the judgment frame, Display area This includes both cases where the finger is released outside the judgment frame.

[0116] Depending on the hardware or program implemented, the decision may be made in one step rather than in two steps like S210 and S211. Display area Did you release your finger inside the judgment frame? Display area It may be possible to determine whether the finger is released outside the detection frame. Display area In other words, it is determined whether or not the device has moved away from the target position, and such hardware or program is also included in the present invention.

[0117] In the S211, the finger was released Display areaIf the operation is inside the determination frame (No in S211), the control unit 2 stores the "basic operation strength" of the "current operation information" stored in the first storage frame (S212), stores the "operation strength level" of the "current operation information" (S213), stores the "operation strength" of the "current operation information" (S214), stores the "movement direction" of the "current operation information" (S215), and stores the "operation strength" of the "current operation information" at any position within the area of ​​responsibility. It will be re-operated with The data is stored for each controller corresponding to the assigned area.

[0118] This way, after swiping, your finger Display area If the finger is released within the judgment frame, the "current operation information" remains stored. Therefore, even if the finger is released, the stored "current operation information" is transmitted to the operation target device 10 (S801).

[0119] In addition, since the "current operation information" is retained in this way, operation In this case, the operation before the finger was released will be continued in steps S202 and onwards based on the stored "current operation information". operation In the operation determination (S203) at this time, not only if a similar swipe operation is performed, but also if an operation other than a swipe is performed, the operation before the finger was released will be continued, thereby achieving intuitive and excellent operability.

[0120] In the S211, the finger was released Display area If the finger is outside the judgment frame (Yes in S211), the control unit 2 Display area The "operation strength" in the first memory frame immediately before leaving the judgment frame is stored as the "basic operation strength" in the second memory frame (S216). At this time, the "operation strength level" in the second memory frame for storing the "basic operation strength" becomes the "operation strength level" in the "current operation information."

[0121] Next, the control unit 2 sets the value of the "operation strength" of the first memory frame stored in the second memory frame to the "basic operation strength" of the first memory frame, and updates the "current operation information" (S217).

[0122] Next, since the "basic operation strength" has been newly set, the control unit 2 adds the "operation strength level" to the "basic operation strength" to set it to the next higher operation strength and updates the "current operation information" (S218). The control unit 2 keeps the "operation strength" of the "current operation information" as it is (S219), keeps the "movement direction" as it is (S220), and moves the cursor to any position within the area of ​​responsibility. It will be re-operated with Wait until

[0123] Display area Even if the finger is removed outside the judgment frame, the "operation strength" and "movement direction" of the "current operation information" are maintained by the operations of S219 and S220. Therefore, similar to the cases of S212 to S215, when the frame timing arrives, the target device 10 can be operated based on the maintained "current operation information" even if the finger is removed.

[0124] In addition, the "current operation information" is retained, so you can manipulated In this case, the operation before the finger was released will be continued in steps S202 and onwards based on the stored "current operation information". operation In the operation determination (S203) at this time, not only if a similar swipe operation is performed, but also if an operation other than a swipe is performed, the operation before the finger was released will be continued, thereby achieving intuitive and excellent operability.

[0125] Then, in S216, the "basic operation strength" of the operation strength stage of the second memory frame corresponding to the "operation strength stage" of the "current operation information" of the first memory frame is set, and in S217 and S218, the "operation strength stage" of the "current operation information" is added, and the "operation strength" of the new operation strength stage one level higher is updated at the time of the next swipe operation. Display area If the swipe reaches the outside of the judgment frame, the next operation will be performed at a higher "operation strength level," making it possible to perform operations in stages.

[0126] The process of increasing the strength of the current operation information by one level is as follows: Separated fingers The current operation information is generated based on the operation The operation information corresponds to the area between the two points.

[0127] <Swipe display example: display An example is drone operation. display For example, display The same applies to examples. Example of Figure 11A: Area below the border Operate with your fingers If swiping starts > First, the bottom left controller Operate with This section explains the case where the area below the boundary line is Operated with Then, when the finger moves, the virtual controller image changes to an image that represents the direction of the finger movement. The "operation strength" value is updated according to the distance the finger moves. Operation information is transmitted from the information processing device 1 to the operation target device 10 so that the operation target device 10 moves forward and turns right in accordance with the direction of the two-dimensional vector of the swipe. The view from above is an image that represents the moving direction of the operation target device 10 according to the operation information.

[0128] <Example of Figure 11B: When the finger crosses the boundary line> Point to the area below the border (first area) Operate with However, even if a swipe operation extends beyond the boundary line and reaches a controller above the boundary line (second area), the controller above the boundary line (second area) will not react. An operation that begins below the boundary line (first area) is treated as an operation using the controller below the boundary line (first area). The operation strength is updated according to the distance the finger moves. The virtual controller image also represents the direction of finger movement. The target device 10 moves forward according to the magnitude of the y component of the strength corresponding to the distance the finger moves, and turns according to the magnitude of the x component of the strength.

[0129] Example of Figure 11C: Fingers Display area If it goes beyond the judgement frame> My fingers Display area If the object moves outside the judgment frame, the operation strength is maintained, the operation strength level is incremented by one, and the basic operation strength is updated. The operation target device 10 then continues moving according to the operation strength and movement direction, which are the current operation information that has been maintained. Then, place your finger again at any position below the border. to operate again The operation strength, operation strength stage, and basic operation strength are maintained. operation The current operation information will be updated according to subsequent finger movements.

[0130] Example of Figure 11D: operation For later swipes > Reattach to bottom of border operation After that, the operation strength is updated according to the distance the finger is moved. The operation strength is added based on the basic operation strength (called continuous addition). The direction of the arrow on the virtual controller image changes according to the direction of the finger movement. Even if the finger is released, the operation strength and rotation angle (movement direction) are maintained, so the operation target device 10 also continues to move according to the same operation strength and movement direction.

[0131] Example of Figure 11E: After Figure 11D operation For later swipes > In Figure 11D, Display area The finger is released before reaching the judgment frame. operation If you release your finger once and then press it again, operation If the user presses the button, the operation strength returns to the basic operation strength. Here, operation If so, operation When the touch screen is displayed, the intensity returns to the basic operation intensity and then the intensity can be increased by swiping again. operation If so, the strength of the swipe may be increased again from the current strength of the operation that is held. Then, the operation strength is added (continuous addition) in accordance with the distance the finger is moved, and the direction of the arrow on the virtual controller image changes in accordance with the direction of the finger movement.

[0132] Example of Figure 11F: Controller at the top of the border Operation with In the case of> 11E, even if the finger is released, the operation strength and the rotation angle (turning direction) are maintained as they are, and therefore the operation target device 10 also moves as it is. Operate with your fingers In this case, since the operation is performed with the upper boundary line area as the assigned area, the operation strength and rotation angle (turning direction) of the lower boundary line area are not updated. Here, the operation at the upper boundary line area is assumed to be the raising, lowering, and left / right movement of the operation target device 10. As shown in the lower right diagram of FIG. 11F, a diagram viewed from the side is preferably displayed to represent the up / down and left / right movement of the operation target device 10, but this is not limited thereto.

[0133] <Example of Figure 11G: Swipe with the controller at the top of the border> The strength of the operation is updated according to the distance the finger is moved. The virtual controller image also rotates according to the direction of the finger movement. Ascent / descent (throttle) and left / right translation (aileron) occur according to the direction and magnitude of the two-dimensional vector of the swipe at the top of the boundary line. As shown in the side view, the magnitude of the x component of the two-dimensional vector is the strength of the left / right translation (aileron), and the magnitude of the y component is the strength of the ascent / descent (throttle). While the controller at the top of the boundary line is being operated, operation by the controller at the bottom of the boundary line is not released, so it is possible to ascend, descend, move right, or move left while turning left, turning right, moving forward, or moving backward.

[0134] Example of Figure 11H: When using the controller at the top of the border, your finger Display area If it goes outside the judgment frame or if it is re-selected at any position above the boundary line operation If From the operation in Figure 11G, the finger Display area If the finger goes outside the judgment frame, the operation strength of the controller at the top of the boundary line is maintained, the operation strength level is added, and the basic operation strength is updated. Then, even if the finger is released, forward movement and turning continue based on the operation information at the bottom of the boundary line, and rising and movement continue based on the operation information at the top of the boundary line.

[0135] In this state, if your finger is placed again at any position above the border, operation Let us assume that a swipe is performed as shown in Figure 11I.

[0136] Example of Figure 11I: Re-positioning at any position above the boundary line operation For later swipes > To the top of the border the As long as you operate it as is, it will be controlled by the controller at the top of the border. From that If you move your finger to the bottom of the border, the controller below the border will not respond. The operation strength of the controller at the top of the boundary line is updated according to the distance the finger is moved, and the virtual controller image rotates according to the direction the finger is moved. As shown in the side view, the target device 10 descends with a strength corresponding to the magnitude of the y component of the two-dimensional vector of the swipe, and moves to the left with a strength corresponding to the magnitude of the x component. And the fingers Display area Assume that the judgment frame has been reached.

[0137] Example of Figure 11J: Fingers Display area If it goes outside the judgment frame or is repositioned at any position above the boundary line operation If you do After the right figure in Figure 11I, the fingers Display area If the operation goes outside the determination frame, the operation strength is maintained, the operation strength level is incremented, and the basic operation strength is updated. Then, the operation target device 10 continues to move based on the current operation information.

[0138] As shown in the right image of Figure 11J, the finger is repositioned at the top of the boundary line. operation In this case, the operation by the controllers above and below the boundary line is not canceled, and the operation target device 10 continues to move.

[0139] <Upward flick action> It is assumed that the control unit 2 recognizes an upward flick based on the operation determination in S203 (S205: upward flick determination means). In the case of an upward flick, the control unit 2 proceeds to the processing of S216 and thereafter. That is, in the case of an upward flick, unlike in the case of a swipe, the "operation strength" is not increased and the "movement direction" is not changed (S208, S209) according to the length and direction of the swipe, but the "operation strength level" is increased.

[0140] <Example of Figure 14: Up flick> As shown in the left diagram of FIG. 14, suppose an upward flick (upward flick) is performed at the bottom of the boundary line. At this time, as shown in FIG. 9, the "operation strength" is not increased and the "movement direction" is not changed (S208, S209), but the operation proceeds to S216 and subsequent steps. Display area The same behavior as when the object moves out of the judgment frame is performed. That is, the operation strength is maintained, the operation strength level is incremented by one, and the basic operation strength is updated. The operation target device 10 then continues moving according to the operation strength and movement direction that are the maintained current operation information.

[0141] <Operation when tapping or flicking down> It is assumed that the control unit 2 has recognized a tap or a downward flick based on the operation determination in S203 (S206: tap / downward flick determination means). The control unit 2 discards the stored information on the current operation strength stage (S221). Next, the control unit 2 subtracts the "operation strength stage" of the "current operation information" so that it becomes one stage lower (S222), and updates the "current operation information" so that the "basic operation strength" stored in the second memory frame corresponding to the "operation strength stage" after the subtraction becomes the "basic operation strength" of the "current operation information" (S223). Then, the control unit 2 updates the "current operation information" so that the "operation strength" of the "current operation information" becomes the "basic operation strength" (S224). The control unit 2 keeps the "movement direction" of the "current operation information" unchanged (S225), and updates the "current operation information" so that the "operation strength" of the "current operation information" becomes the "basic operation strength" at any position within the area of ​​responsibility. Operate with Wait until

[0142] The process of lowering the strength of the current operation information by one level is as follows: Separated fingers The current operation information is generated based on the operation The operation information corresponds to the area between the two points.

[0143] And, since the "current operation information" is saved, Operation in the display area In this case, the operation before the finger was released will be continued in steps S202 and onwards based on the stored "current operation information". operation In the operation determination (S203) at this time, not only if the same tap or down click operation is performed, but also if an operation other than a tap or down click is performed, the operation before the finger was released is continued, thereby achieving intuitive and excellent operability.

[0144] <Example of Figure 12: Tap> <When tapping or flicking down display Example> As shown in the left diagram of Figure 12, suppose you tap anywhere on the bottom of the boundary line. The operation strength is updated to the previous basic operation strength, the operation strength step is subtracted, and the basic operation strength is updated to the previous basic operation strength. The movement direction is maintained. In this way, the operation strength is reset to the basic operation strength. Various information and directions of the controller above the boundary line are maintained.

[0145] As shown in the diagram on the right side of Figure 12, suppose you tap anywhere on the upper boundary line. The operation strength is updated to the previous basic operation strength, the operation strength level is subtracted, and the basic operation strength is updated to the previous basic operation strength. The movement direction is maintained. In this way, the operation strength is reset to the basic operation strength. Various information and directions of the controller below the boundary line are maintained.

[0146] <Example of Figure 15: Downward flick> Basically, a "downward flick" and a "tap" behave the same. However, in the case of throttle operation for ascending and descending as shown in Figure 15, a downward swipe is considered a "downward movement," so the operator may sense a concept of strength addition in a "downward flick," which may contradict the basic behavior. Therefore, when operating the elevator, "downward flick" and "tap" have the same behavior, and when operating the throttle, "downward flick" and "downward swipe" have the same behavior. Display area Since it is more intuitive for the operator to have the same behavior when the finger goes outside the judgment frame, FIG. 15 shows an example of such a case.

[0147] <Action when double tapping or pressing the stop button> Based on the operation determination in S203, the control unit 2 recognizes a double tap (S207: double tap determination means) or the stop button has been pressed (S240: stop button determination means). The control unit 2 resets the display of the "operation strength level" displayed on the virtual controller image (S226), resets the display of the "operation strength level" (S227), resets the display of the "operation strength" (S228), and resets the display of the "movement direction" (S229).

[0148] The information that is discarded by double tapping is operation When the information is discarded, the information processing device 1 sends the following to the operation target device 10: operation An operation is performed to stop movement within the specified area.

[0149] The information that is discarded when the stop button is pressed is the retained information that belongs to a pre-specified area. The information of all areas may be discarded, or the information of some areas may be discarded. Then, when the stop button is pressed, the information processing device 1 performs an operation to stop the movement of the device that belongs to the pre-specified area. If the pre-specified area is the entire area, the information processing device 1 performs an operation to stop the operation target device 10 or make it wait in place.

[0150] Then, the control unit 2 discards the information stored in the first memory frame and the second memory frame for all operation strength levels (S230), and then stores the information in an arbitrary position within the assigned area. Operate with Wait until

[0151] <Example of Figure 13A: Double tap> Only the operation information corresponding to the controller in the double-tapped area is reset, and the operation information corresponding to the controller in the other areas is not reset. As shown in the left diagram of FIG. 13A, when a double tap is performed at any position below the border, the operation information of the lower controller is reset, but the operation information of the upper controller remains maintained. As shown in the right diagram of FIG. 13A, when a double tap is performed at any position above the boundary line, the operation information of the upper controller is reset, but the operation information of the lower controller remains maintained.

[0152] <Example of Figure 13B: When the stop button is tapped> For example, a stop button may be placed on the boundary line as shown in Fig. 13B. The stop button may be placed at any position. When the stop button is tapped, all controller operation information is reset. The controller that is reset by the stop button can be preset arbitrarily. In the example shown in FIG. 13B, the operation information corresponding to the controllers above and below the boundary line is reset simultaneously.

[0153] As described above, by the operation of FIG. 9, the information processing device 1 performs the operation of operation Based on the above, operation information for operating the operation target device 10 is generated, and the indicator is displayed on the display device 4. Operate It can be seen that even when the operation information generating and storing unit 100 does not store the operation information, the operation information generating and storing unit 100 functions as an operation information generating and storing unit.

[0154] <Operation area and Display areaAbout the Judgment Frame> In the above, the area for operation and Display area The judgment frame has been explained. Now, the operation area and Display area The evaluation framework will be summarized below. The information processing device 1 is a display device 4. region In the above, the area for performing operations on the operation target device 10 is defined as the operation area. A predetermined range that is the same as the operation area or larger than the operation area is defined as the operation area. Display area The information processing device 1 then displays the virtual controller image in the operation area. region In the above, an area for performing an operation on the operation target device 10 is defined as an operation area, and a predetermined range on the display device 4 is defined as an operation area. Display area As a judgment frame, it functions as a virtual controller image display means for displaying a virtual controller image on the display device 4.

[0155] Then, even if the operation starts in the operation area and goes beyond the operation area, the information processing device 1 Display area Operation information is generated until the judgment frame is reached.

[0156] Furthermore, when the information processing device 1 functions as a virtual controller image display means, it may be divided into a first area for performing a first operation on the device 10 to be operated (rudder and elevator in the case of the drone exemplified), and a second area for performing a second operation on the device 10 to be operated (aileron and throttle in the case of the drone exemplified), as exemplified in the case of the drone operation.

[0157] When the information processing device 1 functions as an operation information generating and storing means, it generates operation information for a first operation based on an operation by an indicator in the first area, and generates operation information for a second operation based on an operation by an indicator in the second area, and stores each of them.

[0158] As shown in Figure 16, unless the finger is released, operation The controller in the area containing the selected position will be responsible for the operation. In the left figure of Figure 16, the top of the boundary line Operate with So the controller at the top of the boundary is responsible for the operation. In this case, the virtual operation region (Finger movement range) extends to the area below the border.

[0159] Also, in the right figure of Figure 16, the bottom of the boundary line Operate with So the controller at the bottom of the border is in charge of the operation. In this case, the virtual operation region (Finger movement range) extends to the area above the boundary line.

[0160] Display area The judgment frame is the virtual operation shown in Figure 16. region This means that it is the edge of the

[0161] <Application example> 17A to 17E show application examples. As shown in FIG. 17A, Display area Controllers with different "operation characteristics" can be placed on each device. For example, a wide variety of operations, from ultra-fine operations to ultra-high speed operations, can be realized with a single device. For example, the controller in Figure 17A can rapidly bring the target object close to the destination and then adjust its position with high precision.

[0162] As shown in Figure 17B, Display area It is possible to place controllers with different "operable parts" on each device. For example, in deep-sea exploration equipment or remote surgery equipment, multiple parts such as Arm A, Arm B, Arm C, etc. can be operated with a single device. Also, in humanoid robots and disaster relief robots, for example, multiple actions such as viewpoint change, movement, rotation, and balance can be widely controlled with a single device.

[0163] As shown in FIG. 17C, by making the increase in the operating strength minute, it becomes possible to perform an operation with ultra-high precision.

[0164] As shown in Figure 17D, by adjusting the operation unit with the left controller and performing the operation with the right controller, it becomes possible to perform operations while adjusting the unit of operation strength. Also, by increasing the number of touch detection points, multiple simultaneous operations become possible.

[0165] As shown in Figure 17E, a small Display area Furthermore, if connection is possible, it is possible to operate the device with multiple operating terminals with different characteristics.

[0166] Although not shown here, it is also possible to operate a plurality of devices as the operation target device 10 from one terminal by assigning a controller to each device.

[0167] <Diversity> As shown in Figure 18, various devices can be used, including smartphones, smartwatches, tablets, touch screens, and 3D images. Operations can be performed on a single device, making it appear as if multiple devices are being used. The type of device is not important as long as it can be operated by touching the controller displayed in the program.

[0168] <Number of devices to be operated> In the above embodiment, the number of the operation target device 10 is one, but it is also possible to use a plurality of controllers to control a plurality of the operation target devices 10. In this case, the information processing device 1 only needs to transmit operation information obtained by the controller to the device to be operated, and the techniques of the above embodiment can be used for other operations.

[0169] <About the types of devices that can be operated> In the above embodiment, the operation target device 10 has been mainly described as a drone, but the operation target device 10 is not limited to a drone. As long as operation information input to the controller is transmitted to the operation target device in accordance with the operation target device, the operation target device is not limited in the present invention.

[0170] (Variation) In the embodiment shown in FIG. Display area When determining whether or not the finger is released from the determination frame, it is not determined whether the finger is in the upward direction, the downward direction, or any other direction. However, depending on the operation target device 10, it may be determined that the finger is released from the determination frame even if the finger is swiped downward. Display area When the finger is removed from outside the judgment frame, it may be more intuitive to decelerate the movement.

[0171] Also, by flicking downwards, the user tries to subtract "operation strength level" but fails, and the finger Display area In the embodiment shown in FIG. 9, if the operation goes outside the judgment frame, the "operation strength level" may be increased, resulting in an operation that is the opposite of the intuitive operation.

[0172] Furthermore, flick operations vary in sensory perception from person to person, which has a large impact on the quality of operability.

[0173] Also, Display area Consider the case where an operator taps to decrease the "operation strength level" (for example, to slow down) in an environment where a button (for example, a pause button) is located above the screen. If the operator is engrossed in an operation and taps to slow down, and the pause button is located at that position, the pause may be executed, contrary to the operator's intention to slow down. In this way, even when the "operation strength level" is decreased by tapping, there is a possibility that an operation different from the operator's intention may occur.

[0174] In the embodiment of FIG. 9, if you want to increase the "operation level strength," you swipe Display areaYou can either release your finger outside the detection frame or flick upwards. If you want to lower it, you tap or flick downwards.

[0175] However, as mentioned above, from an intuitive perspective, swiping downwards Display area If the "operation strength level" can be lowered by releasing the finger outside the judgment frame, operability may be further improved. Also, considering the possibility of tapping or flicking incorrectly, such an operation is considered to be effective.

[0176] Therefore, the information processing device 1 performs the following when swiping: Display area When the finger is released outside the judgment frame, it is preferable to determine whether the movement direction when the finger is released is a predetermined downward direction. Here, the predetermined downward direction is Display area The downward direction does not necessarily have to be completely parallel to the side surface of the device, but refers to a downward direction within a predetermined angle range.

[0177] And in any direction other than downwards, Display area If the finger is released outside the judgment frame, the information processing device 1 proceeds to the processing of S216 and thereafter, and performs processing to increase the "operation strength level" by one level. On the other hand, if the finger is released in a predetermined downward direction, the information processing device 1 proceeds to the processing of S221 and thereafter, and performs processing to decrease the "operation strength level" by one level, as in the case of a tap or downward flick.

[0178] In addition, the above process of moving up or down by one step is as follows: From operation The separated fingers, operation The strength of the current operation information being generated during

[0179] This allows you to swipe down and still Display area If the finger is released outside the judgment frame, the process proceeds to S221 and subsequent steps, the "operation strength stage" is decremented by one step, and the "current operation information" is updated to the "operation strength" corresponding to the decremented "operation strength stage."

[0180] In addition, the addition and subtraction processing of the operation strength may be changed as appropriate in accordance with the characteristics of the operation target, the operation environment, and the like.

[0181] For example, the finger Display area When the user moves outside the judgment frame, it may be possible to determine whether the swipe was an upward swipe, a rightward swipe, a leftward swipe, or a downward swipe, all or some of these, and perform processing to adjust the strength of the operation according to the direction of the swipe.

[0182] According to the above modification, a downward swipe Display area When the finger is released outside the judgment frame, the strength is reduced by one level, allowing for intuitively appropriate operation.

[0183] Although the present invention has been described in detail above, the above description is merely illustrative of the present invention in all respects and is not intended to limit its scope. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. Each of the constituent elements of the invention disclosed in this specification is considered to be an independent, stand-alone invention. Inventions in which the constituent elements are combined in any manner are also included within the scope of the present invention. [Industrial Applicability]

[0184] The present invention is a program for operating a device to be operated, and is industrially applicable. [Explanation of symbols]

[0185] 1. Information processing equipment 2. Control section 3 Storage section 4 Display device 4a Display section 4b Operation input section 5. Communications Department 6 Input section 10. Devices to be operated 11 Drive unit 12 Control Unit 13 Storage section 15 Communications Department

Claims

1. A program for operating an operation target device having a drive device, the program comprising: a computer connected to a display device that accepts operations by a pointer or that includes the display device; an operation information generating and storing means for generating operation information for operating the operation target device based on an operation by the indicator on the display device, and storing the operation information even when the indicator does not operate the device using the display device; and transmitting the stored operation information to the operation target device, thereby causing the device to function as an operation unit for operating the operation target device; The operation information generating and storing means 11. A program for resuming generation of operation information based on latest operation information stored when an operation using the display device is performed again.

2. The operation information generating and storing means determining whether an operation using the display device has been terminated during the swipe; When it is determined that the user has left the operation using the display device during the swipe, current operation information for the operation target device is stored; The operating means is The program according to claim 1 , wherein the program transmits the stored current operation information to the operation target device and operates the operation target device even when no operation is being performed using the display device.

3. Furthermore, the computer an area in a display area of ​​the display device for performing operations on the operation target device is defined as an operation area, and a predetermined range on the display device is defined as a display area determination frame, and a virtual controller image is displayed by the display device as virtual controller image display means; The program according to claim 2, wherein the operation information generating and storing means generates the operation information until the operation by the indicator starts from the operation area and reaches the display area judgment frame, even if the operation by the indicator goes beyond the operation area.

4. Furthermore, the computer dividing the display device into at least a first area for performing a first operation on the operation target device and a second area for performing a second operation on the operation target device, and causing the display device to function as a virtual controller image display means for displaying a virtual controller image; 3. The program according to claim 2, wherein the operation information generating and storing means generates operation information for the first operation based on an operation by the indicator in the first area, and generates operation information for the second operation based on an operation by the indicator in the second area.

5. The program according to claim 4, characterized in that the operation information generating and storing means generates the operation information as an operation in the first area, even if the operation by the indicator starts from the first area, extends beyond the first area, and reaches the second area.

6. When it is determined that the operation by the indicator has been released outside a predetermined display area determination frame, the operation information generating and holding means updates the current operation information as operation information of a higher strength; When no input is being made to an operation input unit for receiving the operation, the operation means operates the operation target device based on the updated current operation information; The program according to claim 2, characterized in that when an operation using the display device is performed again, operation of the device to be operated is resumed based on the current operation information stored as operation information in the area to which the position where the operation was resumed belongs.

7. When it is determined that the operation by the indicator has been released outside a predetermined display area determination frame, and when the moving direction at the time of release is a predetermined downward direction, the operation information generating and holding means may update the current operation information generated based on the released indicator as operation information of one level lower intensity, When no operation is being performed using the display device, the operation means operates the operation target device based on the updated current operation information; The program according to claim 2, characterized in that when an operation using the display device is performed again, the operation means resumes operation of the device to be operated based on the current operation information stored as operation information in the area to which the position where the operation is resumed belongs.

8. The computer functioning as an upward flick determination means for determining whether an operation on the display device is a predetermined upward flick; When it is determined that the flick is the predetermined upward flick, the operation information generating and holding means updates the current operation information as operation information of a higher strength; When no input is being made to an operation input unit for receiving the operation, the operation means operates the operation target device based on the updated current operation information; The program according to claim 2, characterized in that when input is made to the operation input unit again, the operation means resumes operation of the device to be operated based on the current operation information stored as operation information in the area to which the position where the input was made belongs.

9. The computer functioning as a tap / downward flick determination means for determining whether an operation on the display device is a tap or a flick in a predetermined downward direction; When it is determined that the operation is a tap or a flick in the predetermined downward direction, the operation information generating and holding means updates the current operation information generated based on the separated pointer to operation information of a next lower strength; When no input is being made to an operation input unit for receiving the operation, the operation means operates the operation target device based on the updated current operation information; The program according to claim 2, characterized in that when input is made to the operation input unit again, the operation means resumes operation of the device to be operated based on the current operation information stored as operation information in the area to which the position where the input was made belongs.

10. The computer a double tap determining unit that determines whether an operation on the display device is a double tap; When it is determined that the input is a double tap, the operation information generating and storing means discards the information stored as operation information for the area to which the input position belongs, the operation means performs an operation on the operation target device to stop movement within the area where the input has been made; 3. The program according to claim 2.

11. The computer functioning as a stop button determination means for determining whether the operation on the display device is a press of a stop button; When it is determined that the stop button has been pressed, the operation information generating and storing means executes a stop command in a pre-specified area, discards the stored information belonging to each area, and The operation means performs an operation to stop movement of the device in a pre-specified area, or, if the pre-specified area is the entire area, performs an operation to stop the device to be operated or make it wait at the same place.

3. The program according to claim 2.

12. The program according to claim 2 , wherein the operation information generating and storing means generates the operation information based on a two-dimensional vector of a swipe on the display device by the indicator.

13. 13. The program according to claim 12, wherein the operation information generating and holding means generates the operation information based on an x ​​component and a y component of the two-dimensional vector.

14. 14. The program according to claim 13, wherein, when the device to be operated is a drone, the operation information generating and storing means generates the operation information so that the x component and the y component indicate the direction and strength of two of rudder, elevator, aileron, and throttle.

15. 15. The program according to claim 14, wherein, when the device to be operated is a drone, the operation information generating and storing means generates the operation information so as to indicate directions and intensities of rudder, elevator, aileron, and throttle other than the directions and intensities of the two rudder, elevator, aileron, and throttle, using the x component and y component.

16. Furthermore, the computer When the device to be operated is a drone, the device is divided into a first area for operating two of the rudder, elevator, aileron, and throttle, and a second area for operating two other operations than those in the first area, and a virtual controller image is displayed by the display device as a virtual controller image display means; The program according to claim 2, characterized in that the operation information generating and storing means generates operation information in the first area based on a swipe operation by the indicator in the first area, and generates operation information in the second area based on a swipe operation by the indicator in the second area.

Images