Control method, control system and vehicle comprising control system

Abstract

The invention discloses a control method which comprises the steps that a first three-dimensional coordinate of a first position and a second three-dimensional coordinate of a second position of a limb in space are obtained, and the limb moves from the first position to the second position on the operation side of a controllable surface; based on the first three-dimensional coordinate and the second three-dimensional coordinate, judging whether the movement of the limb from the first position to the second position is one of at least one preset control action or not; if it is determined that the movement of the limb from the first position to the second position is one of the preset at least one control action, the controllable surface is controlled to execute an operation corresponding to the determined control action. According to the control method, the interaction function of the controllable surface is achieved by recognizing different types of control actions based on the three-dimensional coordinates of the limbs of the user at different positions, the control actions of the user can be accurately recognized and responded, and a touch detection mechanism does not need to be customized, so that the production and manufacturing cost and difficulty are reduced, and the production efficiency is improved. And the method is relatively high in universality.

Description

Technical Field

[0001] The present invention relates to the field of human - machine interaction, and more specifically, to a control method, a computer - readable storage medium, a control system, and a vehicle. Background Art

[0002] With the improvement of people's living standards and the continuous development of technology, people's requirements for the refinement and intelligence of automobile interiors are also increasing. At present, in order to improve the user's operation experience and the internal design of the vehicle cockpit, various intelligent interaction devices are increasingly used in the cockpit to execute certain functions by recognizing the user's control actions.

[0003] However, some existing solutions detect the position of the user's hand through the infrared images obtained by an infrared emitter and an infrared camera, which not only has low accuracy, but also has relatively fixed device layout and poor versatility. Other solutions determine the detection area through a series of infrared sensors, which can only detect whether there is an object in the detection area, but cannot determine the exact position. At the same time, the detection area is relatively fixed, and there is also the problem of poor versatility. For a curved screen, the process of arranging a touch film or other touch - detection sensors on it is relatively complex, with high difficulty and cost. Summary of the Invention

[0004] In view of the above - mentioned technical problems, the first aspect of the present invention provides a control method, including: obtaining a first three - dimensional coordinate of a first position of a limb in space and a second three - dimensional coordinate of a second position of the limb, wherein the limb moves from the first position to the second position on the operation side of the controllable surface; based on the first three - dimensional coordinate and the second three - dimensional coordinate, determining whether the movement of the limb from the first position to the second position is one of at least one preset control action; if it is determined that the movement of the limb from the first position to the second position is one of the at least one preset control action, controlling the controllable surface to execute an operation corresponding to the determined control action.

[0005] In some embodiments, the step of determining whether the movement of the limb from the first position to the second position is one of at least one preset control action based on the first three-dimensional coordinates and the second three-dimensional coordinates includes: determining whether the first position is within a first valid region based on the first three-dimensional coordinates; if it is determined that the first position is within the first valid region, then determining whether the second position is also within the first valid region based on the second three-dimensional coordinates; if it is determined that the second position is also within the first valid region, then determining whether the movement of the limb from the first position to the second position satisfies a first movement condition based on the first three-dimensional coordinates and the second three-dimensional coordinates; if it is determined that the first movement condition is satisfied, then determining that the movement of the limb from the first position to the second position is a first control action.

[0006] In some embodiments, the first movement condition includes: the movement duration of the limb from the first position to the second position is within a first preset duration, and the movement direction and / or distance from the first position to the second position are within a first preset range.

[0007] In some embodiments, the first control action is a swipe gesture.

[0008] In some embodiments, the step of determining whether the movement of the limb from the first position to the second position is one of at least one preset control action based on the first three-dimensional coordinates and the second three-dimensional coordinates further includes: if it is determined that the first position is not within the first valid region, or the second position is not within the first valid region, or the movement of the limb from the first position to the second position does not satisfy the first movement condition, then determining whether the second position is within a second valid region based on the second three-dimensional coordinates; if it is determined that the second position is within the second valid region, then determining whether the movement of the limb from the first position to the second position satisfies a second movement condition based on the first three-dimensional coordinates and the second three-dimensional coordinates; if it is determined that the second movement condition is satisfied, then determining that the movement of the limb from the first position to the second position is a second control action.

[0009] In some embodiments, the second movement condition includes: the movement duration of the limb from the first position to the second position is within a second preset duration, and the movement direction and / or distance from the first position to the second position are within a second preset range.

[0010] In some embodiments, the second control action is a tap gesture.

[0011] In some embodiments, the control method further includes: obtaining third three-dimensional coordinates of the limb at a third position in the space, where the limb moves from the second position to the third position on the operating side of the controllable surface; based on the second three-dimensional coordinates and the third three-dimensional coordinates, determining whether the movement of the limb from the second position to the third position is one of the preset at least one control action; if it is determined that the movement of the limb from the second position to the third position is one of the preset at least one control action, controlling the controllable surface to perform an operation corresponding to the determined control action.

[0012] In some embodiments, the step of controlling the controllable surface to perform an operation corresponding to the determined control action includes: generating an interactive control instruction based on the determined control action; controlling the controllable surface to perform the operation via the interactive control instruction.

[0013] In some embodiments, the step of controlling the controllable surface to perform an operation corresponding to the determined control action includes: controlling the controllable surface to present a visual effect corresponding to a trajectory between the first position and the second position and / or triggering a function corresponding to the determined control action.

[0014] A second aspect of the present invention provides a computer-readable storage medium having computer-executable instructions stored thereon for executing the control method according to any one of the embodiments of the first aspect.

[0015] A third aspect of the present invention provides a control system, including: a display / function unit having a controllable surface; a position detection unit configured to detect first three-dimensional coordinates of a limb at a first position in the space and second three-dimensional coordinates of a second position, where the limb moves from the first position to the second position on the operating side of the controllable surface; a control unit communicatively coupled to the position detection unit and including: a processor; and a memory for storing computer-executable instructions, which when executed cause the processor to perform the following method: obtaining the first three-dimensional coordinates and the second three-dimensional coordinates; based on the first three-dimensional coordinates and the second three-dimensional coordinates, determining whether the movement of the limb from the first position to the second position is one of the preset at least one control action; if it is determined that the movement of the limb from the first position to the second position is one of the preset at least one control action, controlling the controllable surface to perform an operation corresponding to the determined control action.

[0016] In some embodiments, when the executable instructions of the computing unit are executed, the processor determines whether the movement of the limb from the first position to the second position is one of at least one preset control action based on the first three-dimensional coordinates and the second three-dimensional coordinates by performing the following method: determining whether the first position is within a first valid region based on the first three-dimensional coordinates; if it is determined that the first position is within the first valid region, then determining whether the second position is also within the first valid region based on the second three-dimensional coordinates; if it is determined that the second position is also within the first valid region, then determining whether the movement of the finger limb from the first position to the second position satisfies a first movement condition based on the first three-dimensional coordinates and the second three-dimensional coordinates; if it is determined that the first movement condition is satisfied, then determining that the movement of the finger limb from the first position to the second position is a first control action.

[0017] In some embodiments, the first movement condition includes: the movement duration of the limb from the first position to the second position is within a first preset duration, and the movement direction and / or distance from the first position to the second position are within a first preset range.

[0018] In some embodiments, the first control action is a swipe gesture.

[0019] In some embodiments, when the executable instructions of the computing unit are executed, the processor determines whether the movement of the limb from the first position to the second position is one of at least one preset control action based on the first three-dimensional coordinates and the second three-dimensional coordinates by performing the following method: if it is determined that the first position is not within the first valid region, or the second position is not within the first valid region, or the movement of the limb from the first position to the second position does not satisfy the first movement condition, then determining whether the second position is within a second valid region based on the second three-dimensional coordinates; if it is determined that the second position is within the second valid region, then determining whether the movement of the limb from the first position to the second position satisfies a second movement condition based on the first three-dimensional coordinates and the second three-dimensional coordinates; if it is determined that the second movement condition is satisfied, then determining that the movement of the limb from the first position to the second position is a second control action.

[0020] In some embodiments, the second movement condition includes: the movement duration of the limb from the first position to the second position is within a second preset duration, and the movement direction and / or distance from the first position to the second position are within a second preset range.

[0021] In some embodiments, the second control action is a tap gesture.

[0022] In some embodiments, the position detection unit is further configured to detect the third three-dimensional coordinates of the limb at a third position in the space, wherein the limb moves from the second position to the third position on the operating side of the controllable surface, and when the executable instruction of the computing unit is executed, the processor is caused to execute the following method: obtain the third three-dimensional coordinates; based on the second three-dimensional coordinates and the third three-dimensional coordinates, determine whether the movement of the limb from the second position to the third position is one of the preset at least one control action; if it is determined that the movement of the limb from the second position to the third position is one of the preset at least one control action, control the controllable surface to perform an operation corresponding to the determined control action.

[0023] In some embodiments, when the executable instruction of the computing unit is executed, the processor is caused to execute the following method to control the controllable surface to perform an operation corresponding to the determined control action: generate an interactive control instruction based on the determined control action; control the controllable surface to perform the corresponding operation via the interactive control instruction.

[0024] In some embodiments, when the executable instruction of the computing unit is executed, the processor is caused to execute the following method to control the controllable surface to perform an operation corresponding to the determined control action: control the controllable surface to present a visual effect corresponding to the trajectory between the first position and the second position and / or trigger a function corresponding to the determined control action.

[0025] In some embodiments, the position detection unit is a vision sensor or an image capture device.

[0026] A fourth aspect of the present invention provides a vehicle, including the control system of any one of the embodiments of the third aspect described above.

[0027] In some embodiments, the display / function unit includes at least one of the following: the cockpit ceiling, door panel, armrest, instrument panel, glass, steering wheel, decorative strip, and seat of the vehicle.

[0028] In various embodiments of the present invention, the interactive function of the controllable surface is realized by identifying different types of control actions based on the three-dimensional coordinates of the user's limb at different positions, which can accurately identify the user's control actions and respond, and there is no need to customize a touch detection mechanism, so the production and manufacturing cost and difficulty are reduced, and it has strong versatility. Description of the Drawings

[0029] Features, advantages, and other aspects of the embodiments of the present invention will become more apparent by referring to the accompanying drawings and the following detailed description. A number of embodiments of the present invention are shown herein in an illustrative, but not restrictive, manner. In the drawings:

[0030] Figure 1 A schematic structural block diagram of a control system according to an embodiment of the present invention is shown;

[0031] Figure 2 A schematic diagram of the spatial relationship between a position detection unit and a controllable surface according to an embodiment of the present invention is shown;

[0032] Figure 3 A schematic diagram of a three-dimensional coordinate system according to an embodiment of the present invention is shown;

[0033] Figure 4 A schematic diagram of the positions of a position detection unit, a limb position, and a controllable surface in the depth direction according to an embodiment of the present invention is shown;

[0034] Figure 5 A schematic flowchart of a control method according to an embodiment of the present invention is shown;

[0035] Figure 6 A schematic diagram of a swipe gesture and an interaction effect according to an embodiment of the present invention is shown;

[0036] Figure 7 A schematic diagram of a tap gesture and an interaction effect according to an embodiment of the present invention is shown;

[0037] Figure 8 Another schematic flowchart of a control method according to an embodiment of the present invention is shown;

[0038] Figure 9 Shows Figure 8 A schematic diagram of the movement trajectory of a user's fingertip in an embodiment of. Detailed implementation manners

[0039] The following describes in detail various exemplary embodiments of the present invention with reference to the accompanying drawings. Although the exemplary methods and apparatuses described below include software and / or firmware executed on hardware among other components, it should be noted that these examples are merely illustrative and should not be considered restrictive. For example, it is contemplated that any or all of the hardware, software, and firmware components may be implemented exclusively in hardware, exclusively in software, or in any combination of hardware and software. Thus, although the exemplary methods and apparatuses have been described below, those skilled in the art will readily appreciate that the examples provided are not intended to limit the manner in which these methods and apparatuses are implemented.

[0040] In addition, the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the methods and systems according to various embodiments of the present invention. It should be noted that the functions marked in the blocks may also occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the flowchart and / or block diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or can be implemented using a combination of dedicated hardware and computing unit instructions.

[0041] The terms "comprising", "including" and similar terms used in the present invention are open-ended terms, i.e., "including / including but not limited to", indicating that other contents may also be included. The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment", and so on.

[0042] For ease of description, spatially relative terms such as "below", "under", "beneath", "above" and "on" are used herein to describe the relationship of one element or feature to another element or feature as shown in the accompanying drawings. It will be understood that, in addition to the orientation depicted in the accompanying drawings, the spatially relative terms are also intended to cover different orientations of the device or component during use or operation. For example, when the device or component in the accompanying drawings is flipped, an element described as "below" or "beneath" another element or feature will then be oriented "above" the other element or feature. Thus, the example terms "below" and "beneath" can cover both orientations of above and below. The device or component may be otherwise oriented (e.g., rotated 90 degrees or at other orientations), and the spatially relative terms used herein should be interpreted accordingly.

[0043] In the prior art, some solutions detect the position of a user's limb through an infrared emitter and an infrared camera. Not only is the accuracy not high, but the device layout is relatively fixed, resulting in poor versatility. Other solutions determine a detection area through a series of infrared sensors, which can only detect whether there is an object in the detection area but cannot determine the exact position. At the same time, the detection area is relatively fixed, and there is also a problem of poor versatility. For a curved screen, the process of arranging a touch film or other touch detection sensors on it is relatively complex, difficult, and costly. The solution proposed by the present invention can be applied to any place or space, including automotive cockpits, aircraft cabins, high-speed train carriages, shopping malls, hotels, stations, offices, and schools, etc. In the solution proposed by the present invention, different types of control actions are recognized based on the three-dimensional coordinates of the limb in different positions to achieve the interactive function of the controllable surface. It can accurately recognize the user's control actions and respond, and there is no need to customize a touch detection mechanism. Therefore, the cost and difficulty of production and manufacturing are reduced, and it has strong versatility. The content of the present invention will be described below according to several embodiments.

[0044] First, refer to Figures 1-4 to describe the control system according to an embodiment of the present invention. Figure 1 FIG. shows a schematic structural block diagram of a control system according to an embodiment of the present invention. Figure 2 FIG. shows a schematic diagram of the spatial relationship between a position detection unit and a display / function unit according to an embodiment of the present invention. Figure 3 FIG. shows a schematic diagram of a three-dimensional coordinate system according to an embodiment of the present invention. Figure 4 FIG. shows a schematic diagram of the positions of a position detection unit, a limb position, and a controllable surface in the depth direction according to an embodiment of the present invention.

[0045] As Figure 1As shown in the figure, the control system 100 includes a position detection unit 101, a control unit 102, and a display / function unit having a controllable surface 103. The display / function unit can be any object having a controllable surface 103. For example, when the display / function unit is located inside a vehicle, it can be the cockpit ceiling, door panel, armrest, instrument panel, window, steering wheel, trim strip, seat, curved screen, and other irregular surfaces of the vehicle. For another example, when the display / function unit 103 is located inside a mall, it can be the mall's window, wall, door panel, van elevator, guide sign, etc. The controllable surface 103 is located on the surface of these objects and can be controlled to display a specific picture and / or trigger the execution of a specific function. The user's limb performs a control action on the operation side of the controllable surface 103. The operation side of the controllable surface 103 refers to the space on one side where the user can perform a control action with respect to the controllable surface 103. The user's limb can be any part of the user's body, such as a finger, arm, torso, leg, etc. The user's limb can contact or not contact the controllable surface 103. The position detection unit 101 is used to detect the three-dimensional coordinates of the user's limb in space. The three-dimensional coordinates of a specific position of the limb can be used as the three-dimensional coordinates of the limb. For example, the three-dimensional coordinates of the fingertip are used as the three-dimensional coordinates of the finger, and the three-dimensional coordinates of the elbow joint are used as the three-dimensional coordinates of the arm, and so on. The control unit 102 is communicatively coupled to the position detection unit 101 and includes a processor 1021 and a memory 1022. The memory 1022 is used to store executable instructions for the computing unit. When the executable instructions for the computing unit are executed, the processor 1021 obtains the three-dimensional coordinates of the user's limb in space and controls the controllable surface 103 based on these three-dimensional coordinates. The control unit 102 can be any independent computing device, such as the main controller of a vehicle or an electronic control unit (ECU), or can be integrated with the position detection unit 101 and / or the display / function unit.

[0046] It should be noted that the limb can be defined as the fingertip of the occupant or user, any joint of the finger, or any point on the palm. In the following description, the fingertip, that is, the fingertip meaning, is used to explain the limb.

[0047] In some embodiments, the controllable surface 103 is an intelligent surface, and its surface material can be PMMA plastic, glass, or PC plastic, and is coated with an AR coating, so that the reflectivity of the surface is less than 5%, which helps to improve the detection accuracy of the position detection unit 101. In other embodiments, the controllable surface 103 can also be other types of surfaces, such as a projection surface.

[0048] In some embodiments, the operation side of the controllable surface 103 is the upper space of the controllable surface 103. As Figure 1As shown in the figure, the position detection unit 101 is arranged directly above the controllable surface 103, and the area within the dashed line is the field of view area of the position detection unit 101. In some other embodiments, the position detection unit 101 may also be arranged in other orientations of the controllable surface 103, such as the upper left or upper right, as long as it can detect the movement of the user's limb on the operation side of the controllable surface 103.

[0049] In some embodiments, the position detection unit 101 is a time-of-flight (TOF) module. In some other embodiments, the position detection unit 101 may be other types of visual sensors or image capture devices (such as a camera device). In some other embodiments, the position detection unit 101 may also be a combination of two or more devices, such as a combination of two devices respectively for detecting the planar coordinates and depth information of the user's limb.

[0050] In some embodiments, the position detection unit 101 detects the three-dimensional coordinates of a single limb (such as a finger) of the user at different positions in space. In some other embodiments, the three-dimensional coordinates of multiple limbs (such as multiple fingers) of the user at different positions in space can be detected, so that the control actions of the user can be recognized by combining the three-dimensional coordinates of multiple limbs.

[0051] In some embodiments, the vertical projection plane of the shooting field of view of the position detection unit 101 is used as the reference plane 1011, the center of the field of view of the position detection unit 101 is defined as the origin of the spatial coordinate system, and the reference plane 1011 is used as the reference for depth information. Refer Figure 2 and Figure 3 , the three-dimensional coordinates of the position A where the limb is located in space are (x, y, d), and its depth information is d. The position detection unit 101 also collects the depth information of the controllable surface 103 within the field of view. Each point on the surface 103 corresponds to a three-dimensional coordinate (x0, y0, d0), and its depth information is d0. Refer Figure 4 , the planar coordinates of point A' on the surface 103 are the same as the planar coordinates of the position A where the limb is located, and the distance h between the position A and the point A' is equal to d0 - d.

[0052] When the user's limb moves from the first position to the second position on the operation side of the controllable surface 103, the position detection unit 101 detects the first three-dimensional coordinates of the user's limb at the first position and the second three-dimensional coordinates at the second position, and sends this three-dimensional coordinate data to the control unit 102. The processor 1021 of the control unit 102 determines, based on the first three-dimensional coordinates and the second three-dimensional coordinates, whether the movement of the limb from the first position to the second position is one of at least one preset control action. If it is determined that the movement of the limb from the first position to the second position is one of at least one preset control action, then the controllable surface 103 is controlled to perform an operation corresponding to the determined control action. The at least one preset control action can be any limb movement, such as a gesture, a head movement, a leg movement, etc. The gesture can include, for example, a swipe gesture, a tap gesture, a drag control action, a zoom control action, etc.

[0053] In some embodiments, the processor 1021 of the control unit 102 first determines, based on the first three-dimensional coordinates, whether the first position is within the first valid region. If it is determined that the first position is within the first valid region, then based on the second three-dimensional coordinates, it is determined whether the second position is also within the first valid region. If it is determined that the second position is also within the first valid region, then based on the first three-dimensional coordinates and the second three-dimensional coordinates, it is determined whether the movement of the limb from the first position to the second position satisfies the first movement condition. If it is determined that the first movement condition is satisfied, then it is determined that the movement of the limb from the first position to the second position is the first control action.

[0054] In some embodiments, the first movement condition includes: the movement duration of the limb from the first position to the second position is within the first preset duration, and the movement direction and / or distance from the first position to the second position are within the first preset range. The first preset duration and the first preset range can be defined according to the type of the first control action, the field of view of the position detection unit 101, and the arrangement of the position detection unit 101 relative to the controllable surface 103. For example, if within a certain time, the limb has a certain displacement in the x or y direction and the d direction from the first position to the second position, that is, compared with the first three-dimensional coordinates, the x value or y value and the d value of the second three-dimensional coordinates have changed by a certain amount, then it is determined that the first control action is a swipe gesture.

[0055] In some embodiments, if the processor 1021 of the control unit 102 determines that the first position is not within the first effective area, or the second position is not within the first effective area, or the movement of the limb from the first position to the second position does not meet the first movement condition, then it is determined whether the second position is within the second effective area based on the second three-dimensional coordinates. If it is determined that the second position is within the second effective area, then based on the first three-dimensional coordinates and the second three-dimensional coordinates, it is determined whether the movement of the limb from the first position to the second position meets the second movement condition. If it is determined that the second movement condition is met, then the movement of the limb from the first position to the second position is determined to be a second control action.

[0056] In some embodiments, the second movement condition includes: the movement duration of the limb from the first position to the second position is within a second preset duration, and the movement direction and / or distance from the first position to the second position are within a second preset range. The second preset duration and the second preset range can be predefined according to the type of the second control action, the field of view of the position detection unit 101, and the arrangement of the position detection unit 101 relative to the display / function unit 103. For example, if within a certain period of time, the limb has a significant displacement from the first position to the second position in the d direction, that is, the d value of the second three-dimensional coordinate changes compared with the first three-dimensional coordinate, then the second control action is determined to be a point-touch gesture.

[0057] In some embodiments, if the processor 1021 of the control unit 102 determines that the movement of the limb from the first position to the second position is one of at least one preset control action, then an interactive control instruction is generated based on the determined control action and sent to the controllable surface 103 (such as via the CAN bus). The interactive control instruction may include the type of the determined control action and the three-dimensional coordinates of the limb at the two positions. The controllable surface 103 performs corresponding operations in response to the interactive control instruction. Performing the corresponding operations may include presenting a visual effect corresponding to the trajectory between the first position and the second position of the limb, such as displaying lights, backlights, or pictures along the trajectory between the first position and the second position. The position where the visual effect is presented on the controllable surface 103 can be determined according to the limb position and / or the user's perspective. Performing the corresponding operations may also include triggering a function corresponding to the determined control action, such as triggering a certain function of the vehicle (such as opening the sunroof, playing music, opening / closing an application, etc.).

[0058] In some embodiments, the limb continues to move from the second position to the third position. The position detection unit 102 continues to detect the third three-dimensional coordinates of the limb at the third position and sends the third three-dimensional coordinates to the control unit 102. After the processor 1021 of the control unit 102 obtains the third three-dimensional coordinates, based on the second three-dimensional coordinates and the third three-dimensional coordinates, it determines whether the movement of the limb from the second position to the third position is one of at least one preset control action. If it is determined that the movement of the limb from the second position to the third position is one of the above-mentioned at least one preset control action, the controllable surface 103 can be controlled to perform an operation corresponding to the determined control action. By continuously recognizing the user's control actions and controlling the controllable surface 103 to perform the corresponding operations, the "hand-following" operation can be realized, further improving the user experience.

[0059] The following refers to Figure 5 , Figure 5 which shows a schematic flowchart of a control method according to an embodiment of the present invention. The control method 200 can be executed by Figure 1 the control unit 102 therein.

[0060] In step 201, the first three-dimensional coordinates of the limb at the first position in space and the second three-dimensional coordinates of the second position are obtained. The limb moves from the first position to the second position on the operation side of the controllable surface.

[0061] In some embodiments, the position detection unit can detect the limb position on the operation side of the display / function unit and then send it to the control unit 102. The position detection unit can be a single device, such as a depth camera, or a combination of two or more devices, as long as it can detect the three-dimensional coordinates of the user's limb.

[0062] In step 202, based on the first three-dimensional coordinates and the second three-dimensional coordinates, it is determined whether the movement of the limb from the first position to the second position is one of at least one preset control action. After obtaining the three-dimensional coordinates of the limb moving to different positions in space, the control actions can be recognized according to these three-dimensional coordinates.

[0063] In some embodiments, step 202 further includes: first, based on the first three-dimensional coordinates, it is determined whether the first position is within the first effective area. If it is determined that the first position is within the first effective area, then based on the second three-dimensional coordinates, it is determined whether the second position is also within the first effective area. If it is determined that the second position is also within the first effective area, then based on the first three-dimensional coordinates and the second three-dimensional coordinates, it is determined whether the movement of the limb from the first position to the second position satisfies the first movement condition. If it is determined that the first movement condition is satisfied, it is determined that the movement of the limb from the first position to the second position is the first control action.

[0064] In some embodiments, the first motion condition includes: the motion duration of the limb from the first position to the second position is within a first preset duration, and the motion direction and / or distance from the first position to the second position are within a first preset range.

[0065] In some embodiments, step 202 further includes: if it is determined that the first position is not within the first valid area, or the second position is not within the first valid area, or the motion of the limb from the first position to the second position does not meet the first motion condition, then based on the second three-dimensional coordinates, determine whether the second position is within the second valid area; if it is determined that the second position is within the second valid area, then based on the first three-dimensional coordinates and the second three-dimensional coordinates, determine whether the motion of the limb from the first position to the second position meets the second motion condition; if it is determined that the second motion condition is met, then determine that the motion of the limb from the first position to the second position is a second control action.

[0066] In some embodiments, the second motion condition includes: the motion duration of the limb from the first position to the second position is within a second preset duration, and the motion direction and / or distance from the first position to the second position are within a second preset range.

[0067] In step 203, if it is determined that the motion of the limb from the first position to the second position is one of at least one preset control action, then control the display / function unit to execute an operation corresponding to the determined control action.

[0068] In some embodiments, step 203 further includes: generating an interactive control instruction based on the determined control action; and sending the interactive control instruction to the display / function unit, thereby controlling the display / function unit to execute an operation.

[0069] In some embodiments, step 203 further includes: controlling the display / function unit to present a visual effect corresponding to the trajectory between the first position and the second position and / or execute an application operation instruction corresponding to the determined control action.

[0070] In some embodiments, the control method 200 further includes: obtaining third three-dimensional coordinates of a third position of the limb in space, where the limb moves from the second position to the third position on the operation side of the controllable surface. Based on the second three-dimensional coordinates and the third three-dimensional coordinates, determine whether the motion of the limb from the second position to the third position is one of at least one preset control action. If it is determined that the motion of the limb from the second position to the third position is one of at least one preset control action, then control the controllable surface to execute an operation corresponding to the determined control action.

[0071] Next, refer to Figures 6-9 to illustrate the control method according to a specific embodiment of the present invention. Figure 6Shows a schematic diagram of a sliding gesture and an interaction effect according to an embodiment of the present invention. Figure 7 Shows a schematic diagram of a touch gesture and an interaction effect according to an embodiment of the present invention. Figure 8 Shows another schematic flowchart of a control method according to an embodiment of the present invention. Figure 9 Shows Figure 8 A schematic diagram of the movement trajectory of the user's fingertip in the embodiment of

[0072] In this embodiment, the movement trajectory of the user's finger in space is continuously recognized. This embodiment uses Figure 3 The spatial coordinate system shown, that is, the vertical projection plane of the shooting field of view of the position detection unit 101 is used as the reference plane 1011, the center of the field of view of the position detection unit 101 is defined as the origin of the spatial coordinate system, and the reference plane 1011 is used as the benchmark for depth information. In addition, this embodiment recognizes two user control actions: sliding gestures and touch gestures.

[0073] Figure 6 Shows a sliding gesture where the user's gesture moves from the first position A to the second position B. The first three-dimensional coordinates of the finger at the first position A are (x’, y’, d’), and the second three-dimensional coordinates at the second position B are (x”, y”, d”). There is a point A’ on the controllable surface 103 with the same planar coordinates (x’, y’) as the first position A, and its three-dimensional coordinates are (x’, y’, d0’), and a point B’ with the same planar coordinates (x”, y”) as the second position B, and its three-dimensional coordinates are (x”, y”, d0”). The distance h ′ = d0 ′ - d′, and the distance h″ between the second position B and the point B’ is d0″ - d″. Set the difference between the second three-dimensional coordinates and the first three-dimensional coordinates as: Δx = x″ - x′, Δy = y″ - y′, Δd = d″ - d′, which describe the movement direction and distance of the finger. In this embodiment, the first effective area (i.e., the sliding effective area) is defined as x ∈ [x1, x2], y ∈ [y1, y2], h ∈ [h1, h2], that is, if the x, y values of the finger's position and the distance from the controllable surface 103 are within the above ranges, it means that the position is in the first effective area. The first movement condition (i.e., the sliding condition) is defined as (Δx ∈ (0, a] / Δy ∈ (0, b]) & Δd ∈ [0, c), and the movement time T ∈ [t1, t2], that is, if the finger makes a displacement within the above ranges in the x direction and / or y direction, makes a displacement within the above range or no displacement in the d direction, and the movement time T is within the above range, then the first movement condition is satisfied. The above numerical ranges can be set according to the field of view range of the position detection unit and its arrangement direction with respect to the controllable surface 103.

[0074] Figure 7 shows a tap gesture where the user's finger moves from a first position A to a second position B. The first three-dimensional coordinates of the finger at the first position A are (x', y', d'), and the second three-dimensional coordinates at the second position B are (x", y", d"). There is a point A' on the controllable surface 103 with the same planar coordinates (x', y') as the first position A, and its three-dimensional coordinates are (x', y', d0'), and a point B' with the same planar coordinates (x", y") as the second position B, and its three-dimensional coordinates are (x", y", d0"). The distance h between the first position A and the point A' ′ = d0 ′ - d', and the distance h" between the second position B and the point B' = d0" - d". Set the difference between the second three-dimensional coordinates and the first three-dimensional coordinates as: Δx = x" - x', Δy = y" - y', Δd = d" - d', which describe the movement direction and distance of the finger. In this embodiment, the second effective region (i.e., the tap effective region) is defined as x ∈ [x3, x4], y ∈ [y3, y4], h ∈ [h3, h4], that is, if the x, y values of the finger's position and the distance from the controllable surface 103 are within the above ranges, it means the position is in the second effective region. The second movement condition (i.e., the tap condition) is defined as (Δx ∈ [0, d) / Δy ∈ [0, e)) & Δd ∈ (f, g], and the movement time T ∈ [t3, t4], that is, if the finger makes a displacement within the above ranges in the d direction, makes a displacement within the above ranges or no displacement in the x direction and / or y direction, and the movement time T is within the above ranges, then the second movement condition is satisfied. The above numerical ranges can be set according to the viewing range of the position detection unit and its layout direction with respect to the controllable surface 103.

[0075] Refer also to Figure 8 and Figure 9 . When the user's finger starts to move in space, the control method of Figure 8 is started. As shown in Figure 9 , the movement trajectory of the user's finger in space is composed of a series of continuous points, and the control method 300 shown in Figure 8 is executed for each two adjacent points. In the control method 300, the first position and the second position are respectively Figure 8 two adjacent points. For example, when the first position is A0, the second position is B0; when the first position is B0 (A1), the second position is B1; when the first position is B1 (A2), the second position is B2... and so on.

[0076] The control method 300 first includes step 301 of obtaining the first three-dimensional coordinates of the finger at the first position and the second three-dimensional coordinates of the finger at the second position in space. As described above, the finger moves from the first position to the second position on the operation side of the controllable surface. The position of the finger can be detected by the position detection unit on the operation side of the controllable surface. The position detection unit can be a single device or a combination of multiple devices.

[0077] Next, in step 302, based on the first three-dimensional coordinates, it is determined whether the first position is within the first valid region, that is, the distance h between the first position and the point on the controllable surface 103 with the same planar coordinates (x, y) is calculated, and it is determined whether (x, y, h) is within the following range: x ∈ [x1, x2], y ∈ [y1, y2], h ∈ [h1, h2]. If the first position is within the first valid region, the method 300 proceeds to step 303 to determine whether the second position is also within the first valid region based on the second three-dimensional coordinates. If the second position is also within the first valid region, then in step 304, based on the first three-dimensional coordinates and the second three-dimensional coordinates, it is determined whether the movement of the finger from the first position to the second position satisfies the first movement condition, that is, it is determined whether the displacements of the finger in the x direction and / or y direction are within the following ranges: Δx ∈ (0, a] and / or Δy ∈ (0, b], whether the displacement in the d direction is within the following range: Δd ∈ [0, c), and whether the movement time T is within the following range: T ∈ [t1, t2]. If the first movement condition is satisfied, then in step 305, the movement of the finger from the first position to the second position is determined as a swipe gesture. Subsequently, in step 306, an interaction control instruction corresponding to the swipe gesture is generated and sent to the controllable surface 103. The controllable surface 103 responds to the interaction control instruction and presents an interaction effect at any angle, such as Figure 6 the swipe effect from A” to B” shown in

[0078] If it is determined in step 302 that the first position is not within the first valid area, or in step 303 that the second position is not within the first valid area, or in step 304 that the movement of the finger from the first position to the second position does not meet the first movement condition, then method 300 proceeds to step 307. In step 307, it is determined whether the second position is within the second valid area based on the second three-dimensional coordinates, that is, the distance h between the second position and the point on the controllable surface 103 having the same planar coordinates (x, y) is calculated, and it is determined whether (x, y, h) is within the following range: x ∈ [x3, x4], y ∈ [y3, y4], h ∈ [h3, h4]. If the second position is within the second valid area, then in step 308, based on the first three-dimensional coordinates and the second three-dimensional coordinates, it is determined whether the movement of the finger from the first position to the second position meets the second movement condition, that is, it is determined whether the displacement of the finger movement in the x direction and / or the y direction is within the following range: Δx ∈ [0, d) and / or Δy ∈ [0, e), whether the displacement in the d direction is within the following range: Δd ∈ (f, g], and whether the movement time T is within the following range: T ∈ [t3, t4]. If the second movement condition is met, then in step 309, the movement of the finger from the first position to the second position is determined as a point touch gesture. Subsequently, in step 310, an interaction control instruction corresponding to the point touch gesture is generated and sent to the controllable surface 103. The controllable surface 103 responds to the interaction control instruction and presents an interaction effect at any angle, such as Figure 7 the point touch effect at point B” shown in

[0079] If it is determined in step 307 that the second position is not within the second valid area, or in step 308 that the movement of the finger from the first position to the second position does not meet the second movement condition, then it is determined as an invalid gesture in step 311. Accordingly, the controllable surface 103 does not make any response.

[0080] As described above, for Figure 9 two adjacent points, after performing steps 306, 310 or 311, the three-dimensional coordinates of the finger at the next point are continuously obtained, and the steps in control method 300 are repeatedly executed. It can be understood that the gestures between every two adjacent points may be different. For example, from A0 to B0, B0 to B1, and B1 to B2 are all invalid gestures, from B2 to B3 and B3 to B4 are both sliding gestures, from B4 to B5 is an invalid gesture, and from B5 to B6 is a point touch gesture. Accordingly, the controllable surface 103 presents a sliding effect, a point touch effect, or no effect at the corresponding positions respectively.

[0081] The present invention also provides a vehicle, which includes a control system according to an embodiment of the present invention, such as the one described above according to Figures 1-4 the described control system.

[0082] In addition, alternatively, the above method can be implemented by a computer-readable storage medium. A computer-readable program instruction for executing each embodiment of the present invention is uploaded on the computer-readable storage medium. The computer-readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punched card or raised structures in a groove storing instructions thereon, and any suitable combination of the above. The computer-readable storage medium used herein is not construed as an instantaneous signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0083] Therefore, in some other embodiments, the present invention proposes a computer-readable storage medium having computer-executable instructions stored thereon for executing the control methods in the various embodiments regarding Figure 5 and Figure 8 described.

[0084] Generally speaking, the various exemplary embodiments of the present invention can be implemented in hardware or a dedicated circuit, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, a microprocessor, or other computing devices. When aspects of the embodiments of the present invention are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, devices, systems, technologies, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, a dedicated circuit or logic, general hardware or a controller or other computing devices, or some combination thereof.

[0085] The computer-readable program instructions or computer program products for executing each embodiment of the present invention can also be stored in the cloud. When needed, a user can access the computer-readable program instructions stored in the cloud for executing an embodiment of the present invention through a mobile Internet, a fixed network, or other networks, so as to implement the technical solutions disclosed in each embodiment of the present invention.

[0086] Although embodiments of the present invention have been described with reference to several specific embodiments, it should be understood that the embodiments of the present invention are not limited to the specific embodiments disclosed. Embodiments of the present invention are intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A control method, comprising: Obtaining a first three-dimensional coordinate of a first position of a limb in space and a second three-dimensional coordinate of a second position, wherein the limb moves from the first position to the second position on an operating side of a controllable surface; Based on the first three-dimensional coordinate and the second three-dimensional coordinate, determining whether the movement of the limb from the first position to the second position is one of at least one preset control action; If it is determined that the movement of the limb from the first position to the second position is one of the at least one preset control action, controlling the controllable surface to perform an operation corresponding to the determined control action.

2. The control method according to claim 1, wherein, The step of determining whether the movement of the limb from the first position to the second position is one of at least one preset control action based on the first three-dimensional coordinate and the second three-dimensional coordinate includes: Based on the first three-dimensional coordinate, determining whether the first position is within a first valid area; If it is determined that the first position is within the first valid area, then based on the second three-dimensional coordinate, determining whether the second position is also within the first valid area; If it is determined that the second position is also within the first valid area, then based on the first three-dimensional coordinate and the second three-dimensional coordinate, determining whether the movement of the limb from the first position to the second position satisfies a first movement condition; If it is determined that the first movement condition is satisfied, determining that the movement of the limb from the first position to the second position is a first control action.

3. The control method according to claim 2, wherein The first movement condition includes: the movement duration of the limb from the first position to the second position is within a first preset duration, and the movement direction and / or distance from the first position to the second position are within a first preset range.

4. The control method according to claim 2 or 3, wherein The first control action is a swipe gesture.

5. The control method according to claim 2, wherein, The step of determining whether the movement of the limb from the first position to the second position is one of at least one preset control action based on the first three-dimensional coordinate and the second three-dimensional coordinate further includes: If it is determined that the first position is not within the first valid area, or the second position is not within the first valid area, or the movement of the limb from the first position to the second position does not satisfy the first movement condition, then based on the second three-dimensional coordinate, determining whether the second position is within a second valid area; If it is determined that the second position is within the second valid area, then based on the first three-dimensional coordinate and the second three-dimensional coordinate, determining whether the movement of the limb from the first position to the second position satisfies a second movement condition; If it is determined that the second movement condition is satisfied, determining that the movement of the limb from the first position to the second position is a second control action.

6. The control method according to claim 5, wherein, The second movement condition includes: the movement duration of the limb from the first position to the second position is within a second preset duration, and the movement direction and / or distance from the first position to the second position are within a second preset range.

7. The control method according to claim 5 or 6, wherein The second control action is a tap gesture.

8. The control method according to claim 1, further comprising: Obtain the third three-dimensional coordinates of the limb at the third position in the space, where the limb moves from the second position to the third position on the operating side of the controllable surface; Based on the second three-dimensional coordinates and the third three-dimensional coordinates, determine whether the movement of the limb from the second position to the third position is one of the preset at least one control action; If it is determined that the movement of the limb from the second position to the third position is one of the preset at least one control action, then control the controllable surface to perform an operation corresponding to the determined control action.

9. The control method according to claim 1, wherein The step of controlling the controllable surface to perform an operation corresponding to the determined control action includes: Generate an interactive control instruction based on the determined control action; Control the controllable surface to perform the operation via the interactive control instruction.

10. The control method according to claim 1, wherein, The step of controlling the controllable surface to perform an operation corresponding to the determined control action includes: Control the controllable surface to present a visual effect corresponding to the trajectory between the first position and the second position and / or trigger a function corresponding to the determined control action.

11. A computer-readable storage medium having computer-executable instructions stored thereon for performing the control method according to any one of claims 1-10.

12. A control system, comprising: A display / function unit having a controllable surface; A position detection unit configured to detect the first three-dimensional coordinates of the limb at the first position and the second three-dimensional coordinates of the second position in the space, where the limb moves from the first position to the second position on the operating side of the controllable surface; A control unit communicatively coupled to the position detection unit and comprising: A processor; and A memory for storing computer-executable instructions that, when executed, cause the processor to perform the following method: Obtain the first three-dimensional coordinates and the second three-dimensional coordinates; Based on the first three-dimensional coordinates and the second three-dimensional coordinates, determine whether the movement of the limb from the first position to the second position is one of the preset at least one control action; If it is determined that the movement of the limb from the first position to the second position is one of the preset at least one control action, then control the controllable surface to perform an operation corresponding to the determined control action.

13. The control system according to claim 12, wherein When the computer-executable instructions are executed, cause the processor to determine whether the movement of the limb from the first position to the second position is one of the preset at least one control action based on the first three-dimensional coordinates and the second three-dimensional coordinates by performing the following method: Based on the first three-dimensional coordinates, determine whether the first position is within the first valid region; If it is determined that the first position is within the first valid region, then based on the second three-dimensional coordinates, determine whether the second position is also within the first valid region; If it is determined that the second position is also within the first effective area, then based on the first three-dimensional coordinates and the second three-dimensional coordinates, determine whether the movement of the limb from the first position to the second position satisfies the first movement condition; If it is determined that the first movement condition is satisfied, then determine that the movement of the limb from the first position to the second position is a first control action.

14. The control system according to claim 13, wherein, The first movement condition includes: the movement duration of the limb from the first position to the second position is within a first preset duration, and the movement direction and / or distance from the first position to the second position are within a first preset range.

15. The control system according to claim 13 or 14, wherein, The first control action is a swipe gesture.

16. The control system according to claim 13, wherein, When the executable instruction of the calculation unit is executed, the processor is caused to determine whether the movement of the limb from the first position to the second position is one of at least one preset control action based on the first three-dimensional coordinates and the second three-dimensional coordinates by executing the following method: If it is determined that the first position is not within the first effective area, or the second position is not within the first effective area, or the movement of the limb from the first position to the second position does not satisfy the first movement condition, then determine whether the second position is within a second effective area based on the second three-dimensional coordinates; If it is determined that the second position is within the second effective area, then based on the first three-dimensional coordinates and the second three-dimensional coordinates, determine whether the movement of the limb from the first position to the second position satisfies the second movement condition; If it is determined that the second movement condition is satisfied, then determine that the movement of the limb from the first position to the second position is a second control action.

17. The control system according to claim 16, wherein, The second movement condition includes: the movement duration of the limb from the first position to the second position is within a second preset duration, and the movement direction and / or distance from the first position to the second position are within a second preset range.

18. The control system according to claim 16 or 17, wherein, The second control action is a tap gesture.

19. The control system according to claim 12, wherein, The position detection unit is further configured to detect the third three-dimensional coordinates of the third position of the limb in the space, wherein the limb moves from the second position to the third position on the operation side of the controllable surface, and when the executable instruction of the calculation unit is executed, the processor is caused to execute the following method: Obtain the third three-dimensional coordinates; Based on the second three-dimensional coordinates and the third three-dimensional coordinates, determine whether the movement of the limb from the second position to the third position is one of at least one preset control action; If it is determined that the movement of the limb from the second position to the third position is one of at least one preset control action, then control the controllable surface to execute an operation corresponding to the determined control action.

20. The control system according to claim 12, wherein, When the executable instruction of the calculation unit is executed, the processor is caused to execute the following method to control the controllable surface to execute an operation corresponding to the determined control action: Generate an interactive control instruction based on the determined control action; Control the controllable surface to execute the corresponding operation via the interactive control instruction.

21. The control system according to claim 12, wherein, When the executable instructions of the computing unit are executed, the processor is caused to execute the following method to control the controllable surface to perform an operation corresponding to the determined control action: Control the controllable surface to present a visual effect corresponding to the trajectory between the first position and the second position and / or perform a function corresponding to the determined control action.

22. The control system according to claim 12, wherein, The position detection unit is a vision sensor or an image capture device.

23. A vehicle, comprising the control system according to any one of claims 12-22.

24. The vehicle according to claim 23, wherein, The display / function unit includes at least one of the following: the cockpit ceiling, door panel, armrest, instrument panel, window, steering wheel, decorative strip, and seat of the vehicle.

Images