Authority verification and equipment control method based on gesture recognition in intelligent cabin scene

Abstract

The invention discloses an authority verification and equipment control method based on gesture recognition in an intelligent cabin scene, and the method comprises the steps: constructing an incidence relation between a thumb stroke and a gesture curvature in each thumb stroke segment; the association relationship accurately represents the subtle feature difference between different gesture states corresponding to each captured gesture instruction, and based on the subtle feature difference, the suspected gesture instruction corresponding to the gesture state data collected in real time can be accurately identified. And the similarity between the first change state and the second change state is verified through the step S5, so that the accuracy of gesture state recognition based on the subtle feature differences is improved, it is possible to avoid using complex gesture recognition algorithms such as neural network model learning, and meanwhile, the gesture recognition efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of smart cockpits, and in particular to a method for authority verification and device control based on gesture recognition in a smart cockpit scenario. Background Art

[0002] Controlling devices to execute corresponding instructions through different gesture states has been widely used in various fields. For example, in the cockpit of a car, different gesture states are identified through image recognition, and then the device is controlled to execute corresponding instructions. However, the image recognition method must ensure the accuracy of gesture recognition, and the distinction between different gesture states corresponding to different instructions must be large, such as controlling the device to execute the first instruction when the gesture state is "clenched fist", and controlling the device to execute the second instruction when the gesture state is "stretched palm". However, when the distinction between different gesture states is small, such as controlling the device to execute the first instruction by "clenching the thumb and index finger" in the stretched palm state, and controlling the device to execute the second instruction by "clenching the thumb and middle finger" in the stretched palm state, the recognition accuracy of the image recognition method will be greatly reduced when the distinction between gesture states is so subtle. In order to improve the recognition accuracy of image recognition methods for these subtle gesture states, gesture state model learning methods such as neural networks have been studied in this field. Although the recognition accuracy has been improved, the effect is not good, and the more complex the recognition algorithm for the gesture state, the more time-consuming the recognition, and the higher the performance requirements for the vehicle system, which is difficult to accept in this field. Therefore, in the gesture command control scenario where the initial state of the gesture is controlled by stretching the palm, and the gesture change state is controlled by the clenched state of one or more fingers, ensuring the recognition accuracy and efficiency of the gesture state while reducing the use of complex algorithms has become a technical problem that needs to be urgently solved in this field. Summary of the invention

[0003] The present invention reduces the use of complex gesture recognition algorithms in a gesture command control scenario in which the initial state of the gesture is controlled by stretching the palm, and the gesture change state is controlled by the clenched state of one or more fingers. Under the premise of ensuring the accuracy of gesture state recognition, the present invention aims to improve the gesture state recognition efficiency, and to improve the control accuracy and response speed of the device. A method for permission verification and device control based on gesture recognition in a smart cockpit scenario is provided.

[0004] To achieve this object, the present invention adopts the following technical solutions:

[0005] A method for authority verification and device control based on gesture recognition in a smart cockpit scenario is provided, the steps comprising:

[0006] S1, setting the initial state of the slide to a stretched state that caters to the user's palm, and then learning the gesture state characteristics of the user under different gesture instructions, the learning content includes: gesture stroke data and gesture curvature data under each gesture instruction, and the correlation relationship between the gesture stroke data and the gesture curvature data generated in the same stroke segment;

[0007] S2, identifying suspected gesture instructions based on the gesture state data collected and processed in real time from the user, and then obtaining gesture learning data learned by the user for the suspected gesture instructions, including a stroke learning fitting curve drawn in advance based on the gesture stroke data learned for the suspected gesture instructions;

[0008] S3, drawing a stroke fitting curve of the user's thumb fingertip point according to the gesture state data collected in real time;

[0009] S4, determining whether the travel fitting curve and the travel learning fitting curve have similarity,

[0010] If yes, go to step S5;

[0011] If not, an alarm is given and the user is prompted that he does not have the authority to control the corresponding device with the suspected gesture instruction, and then the process returns to step S2;

[0012] S5, determining whether a first change state of the user's gesture recorded in the gesture state data in each first stroke segment of the thumb fingertip point is similar to a second change state of the gesture in the corresponding second stroke segment recorded in the gesture learning data,

[0013] If yes, the corresponding device is controlled to work by using the suspected gesture command;

[0014] If not, the user is prompted that the control gesture is invalid, and then the process returns to step S2.

[0015] Preferably, in step S1, the method of setting the initial state of the slideway to the extended state comprises the steps of:

[0016] A1, the user presses the thumb button at the initial position on the gesture learning device with the tip of the thumb, so that the thumb button is fixed to the slide;

[0017] A2, the thumb tip pushes the thumb button to drive the slideway to slide along the slide rail provided in the slideway until the user's middle finger tip moves to the press button provided on the gesture learning device when the palm is stretched;

[0018] A3, press the push button to fix the slideway relative to the slide rail, and then press the thumb button a second time to allow the thumb button to slide freely in the slideway.

[0019] Preferably, the slide covers the thumb button with a soft material, and the thumb button is provided with a protrusion for pressing against the thumb.

[0020] Preferably, in step S1, the method for learning the gesture state of the user under different gesture instructions comprises the steps of:

[0021] B1, after selecting the minimum rectangle of the key points of the hand in the palm image in the stretched state, the selected rectangle is discretized into grids and the discrete grids are numbered to obtain the numbered grid where each key point of the hand is located;

[0022] B2, collecting each frame of gesture state image under each gesture instruction at a specified frequency, and calculating the position change data of the gesture state in each frame image under the corresponding stroke segment, wherein the first position change data of the thumb fingertip in each stroke segment constitutes the gesture stroke data learned under the gesture instruction, and the second position change data of each hand key point in each stroke segment constitutes the gesture curvature data learned under the gesture instruction.

[0023] As a preferred embodiment, let the sequence s consisting of a plurality of frames of gesture state images collected for the specified gesture instruction i in step B2 be i ={p1, p2, ..., p j ,…,p m}, p j represents the j-th frame gesture state image, j=1, 2, ..., m, m represents the image frame number, the first position change data includes: the first relative coordinate position of the thumb fingertip point and the lower right vertex of the initial numbering grid where it is located in the stretched state at the end j+1 time point of the stroke segment forming the stroke segment, and the first numbering jump number of the first fingertip point numbering grid where the thumb fingertip point is located at the end j+1 time point of the stroke segment relative to the second fingertip point numbering grid where the thumb fingertip point is located at the start j time point of the stroke segment forming the stroke segment;

[0024] The second position change data includes: the second relative coordinate position of the hand key point and the upper left vertex of the starting numbering grid where the hand key point is located in the stretched state at the j+1 time point when the stroke segment ends, and the second number jump number of the first hand key point numbering grid where the hand key point is located at the j+1 time point when the stroke segment ends relative to the second hand key point numbering grid where the hand key point is located at the j time point when the stroke segment starts.

[0025] Preferably, in step S2, the method for collecting the gesture state data of the user in real time comprises the steps of:

[0026] C1, using the frame selection rectangle formed after the grid discretization and numbering in step B1 as a virtual projection tracking to obtain the first frame gesture image collected in quasi-real time;

[0027] C2, starting from tracking the first frame gesture image, collecting a continuous frame image sequence in real time at a preset frequency until a continuous collection termination condition is reached, and using the position change data of the gesture state in each frame image in the continuous frame image sequence obtained after the collection in the corresponding travel segment as the gesture state data collected in real time from the user;

[0028] The continuous acquisition termination condition is that the relative distance between the thumb tip point in each frame in the continuous frame image sequence and the lower right vertex of the initial numbered grid in the frame selection rectangle decreases as the number of acquisition frames increases and reaches a minimum relative distance.

[0029] Preferably, in step C1, the method for tracking the first frame gesture image collected in real time comprises the steps of:

[0030] C11, adjusting the projection posture of the selection rectangle so that the thumb tip point recognized in real time falls into the initial numbered grid in the minimum rectangular frame projected parallel to the gesture;

[0031] C12, judging whether the middle finger tip point recognized in real time falls into the middle finger tip landing area grid in the projected frame selection rectangle,

[0032] If yes, it is determined that the first frame gesture image is tracked successfully;

[0033] If not, return to step C11.

[0034] Preferably, in step S2, the method for identifying the suspected gesture instruction comprises the steps of:

[0035] D1, calculating the position change of the thumb fingertip point in the last frame relative to the first frame in the continuous frame image sequence, and calculating the number of grid number jumps of the last frame number grid where each hand key point of the user is located in the last frame relative to the first frame number grid where each hand key point is located in the first frame, and then solving the average number of jumps of the grid number jumps corresponding to all the hand key points respectively;

[0036] D2, matching whether there is a gesture instruction whose first deviation between the position change scalar and the position change is less than a preset first deviation threshold, and whose second deviation between the standard mean of the number of beats and the mean of the number of beats is less than a preset second deviation threshold from the gesture instruction library that has completed gesture state learning,

[0037] If yes, the matched gesture instruction is identified as the suspected gesture instruction that the continuous frame image sequence intends to represent;

[0038] If not, it is determined that the recognition of the suspected gesture instruction of the continuous frame image sequence has failed, and the user is prompted to learn the instruction.

[0039] Preferably, the continuous frame image sequence is divided into a plurality of the first run segments, and a continuous frame image subsequence corresponding to each of the first run segments is obtained. In step S5, the first change state corresponding to the first run segment includes: the number of first grid number jumps and the first position change of the position of each hand key point recognized by the user in real time in the last frame relative to the position of the first frame in the continuous frame image subsequence; the second change state corresponding to the second run segment having a corresponding relationship with the first run segment includes: the number of second grid number jumps and the second position change of the position of each hand key point learned by the user for the second run segment in the suspected finger instruction in the last frame of the continuous frame learning subsequence relative to the position of the first frame;

[0040] The corresponding relationship between the first trip segment and the second trip segment is that the first trip segment and the second trip segment have the same data collection start frequency and the same data collection end frequency.

[0041] Preferably, the method for determining similarity in step S5 comprises the steps of:

[0042] D1, calculating a first mean of the number of jumps of the first grid numbers of each of the hand key points in each of the first travel segments, and then calculating a second mean of the first mean values ​​associated with each of the first travel segments; and calculating a third mean of the number of jumps of each of the second grid numbers in each of the second travel segments, and then calculating a fourth mean of the third mean values ​​associated with each of the second travel segments;

[0043] D2, determining whether the deviation between the second mean and the fourth mean is less than a preset third deviation threshold,

[0044] If yes, go to step D3;

[0045] If not, determining that the first change state and the second change state have no similarity;

[0046] D3, calculating the fifth mean of the first position change of each of the hand key points in each of the first travel segments, and then calculating the sixth mean of the fifth mean associated with each of the first travel segments; and calculating the seventh mean of each of the second change in each of the second travel segments, and then calculating the eighth mean of the seventh mean associated with each of the second travel segments;

[0047] D4, determining whether the deviation between the sixth mean and the eighth mean is less than a preset fourth deviation threshold,

[0048] If so, determining that the first change state and the second change state have similarity;

[0049] If not, it is determined that the first change state and the second change state have no similarity.

[0050] The present invention has the following beneficial effects:

[0051] 1. By constructing an association relationship between the thumb stroke and the gesture curvature in each thumb stroke segment, the association relationship is used to accurately characterize the subtle feature differences between different gesture states corresponding to each captured gesture instruction. Based on these subtle feature differences, the suspected gesture instructions corresponding to the gesture state data collected in real time can be accurately identified, and the similarity between the first change state and the second change state is verified through step S5, thereby improving the accuracy of gesture recognition based on these subtle feature differences, making it possible to avoid using complex gesture recognition algorithms such as neural network model learning, and at the same time improving the efficiency of gesture recognition.

[0052] 2. By utilizing the association between the thumb travel and the gesture curvature in each thumb travel segment, and through steps S2-S4, whether the user has the authority to control the corresponding device with the suspected gesture command is accurately verified, thereby reducing the possibility of other users operating the device incorrectly.

[0053] 3. Verification of the similarity between the first change state and the second change state is achieved by judging the number of grid number jumps and the change in curvature stroke. Specifically, based on the number of grid number jumps and the change in curvature stroke, in each grid of the selection rectangle learned in steps B1-B2, find the data feature differences between the gesture state data collected in real time and the gesture learning data, so that the accuracy of verifying that the suspected gesture command identified in step S2 is a real gesture command is greatly improved, ensuring the accuracy of the gesture command generated based on the identified gesture state without using a complex gesture recognition algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 is a diagram of implementation steps of a method for authority verification and device control based on gesture recognition in a smart cockpit scenario provided by an embodiment of the present invention;

[0056] Figure 2 It is a schematic diagram of the key points of the hand in the palm;

[0057] Figure 3 is a schematic diagram of the structure of the gesture learning device provided in this embodiment;

[0058] Figure 4 It is a schematic diagram of the thumb button set on the slide;

[0059] Figure 5 is an example diagram of a selection rectangle generated by a gesture learning device in this embodiment;

[0060] Figure 6 This is an example diagram in which the stroke learning fitting curve and the stroke fitting curve are plotted in the same xy axis coordinate system. DETAILED DESCRIPTION

[0061] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.

[0062] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0063] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as a limitation on this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0064] In the description of the present invention, unless otherwise clearly specified and limited, if the term "connection" or the like appears to indicate the connection relationship between components, the term should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0065] The embodiment of the present invention provides a method for authority verification and device control based on gesture recognition in a smart cockpit scenario, such as Figure 1 As shown, the steps include:

[0066] S1, setting the initial state of the slide to a stretched state that caters to the user's palm, and then learning the user's gesture state characteristics under different gesture commands, the learning content includes: gesture stroke data and gesture curvature data under each gesture command, and the correlation between the gesture stroke data and gesture curvature data generated in the same stroke segment;

[0067] The following combination Figure 2 , Figure 3 , Figure 4 The method of setting the initial state of the slideway to the stretched state of the user's palm in step S1 is described, and the steps specifically include:

[0068] A1, the user presses with the tip of the thumb Figure 3 The thumb button 10 on the gesture learning device shown is in the initial position, so that the thumb button 10 is fixed to the slide 20; the initial position of the thumb button 10 is explained as follows: when the thumb button 10 and the slide 20 are in a state where they can slide relative to each other, the thumb button 10 has a maximum sliding stroke, and the distance Figure 5 The middle fingertip landing area 100 shown in FIG. is the farthest, and the thumb button 10 is at the initial position at this time;

[0069] A2, the thumb pushes the thumb button 10 to drive the slide 20 along the slide rail 30 ( Figure 3 ) until your palm is stretched (five fingers naturally spread apart to Figure 5 In the stretched state shown in FIG, the user's middle finger tip moves to the position set at Figure 3 On the gesture learning device shown, a button 40 is pressed;

[0070] A3, press the push button 40 to fix the slideway 20 relative to the slide rail 30, and then press the thumb button 10 for the second time to allow the thumb button 10 to slide freely in the slideway 20.

[0071] It should be noted that the fixing and separation of the thumb button 10 and the slide 20 can be achieved using a conventional structure, such as using a spring device to press the thumb button 10. Figure 3 The spring 60 shown in the figure is compressed, so that the thumb button 10 is tightly fixed to the slide 20. At this time, the thumb button 10 is pushed and the slide 20 is driven together. After pressing the spring again, the thumb button 10 is separated from the slide 20, and the thumb button 10 can slide relative to the slide 20.

[0072] In order to facilitate the thumb button to slide freely in the slideway and not to be easily disengaged, preferably, the slideway 20 covers the thumb button 10 with a soft material (such as silica gel), and the thumb button 10 is provided with a member such as a thumb member for pressing against the thumb. Figure 4 The protrusion 50 is shown in FIG.

[0073] In step S1, the method for learning the gesture state of the user under different gesture instructions includes the steps of:

[0074] B1, the gesture learning device selects the smallest rectangle of the hand key points in the palm image in the stretched state, discretizes the selected rectangle into grids and numbers the discrete grids to obtain the numbered grid where each hand key point is located;

[0075] Specifically, through the above steps A1-A3, the user's middle finger tip falls on Figure 5 After the tip of the middle finger on the gesture learning device shown in the figure falls on area 100, the gesture learning device starts to collect the palm image of the user in the stretched palm state, and then detects each hand key point in the palm image through the existing algorithm, and then selects the palm area image in the palm image by the existing minimum rectangular frame selection method. In this embodiment, the rectangular frame that selects the palm area image is defined as the frame selection rectangle.

[0076] Then Figure 5 The selection rectangle 200 is discretized into grids and the discretized grids are numbered, for example, 1, 2, 3, ... from left to right and from top to bottom, thereby obtaining the numbered grid where each hand key point is located. Figure 5 In FIG. 6 , the hand key point k12 (i.e., the tip of the middle finger) is located in the numbered grid 6. It should be noted that each numbered grid is preferably a square with the same size, and the side length is preferably the longitudinal axis between two hand key points in the same palm (in Figure 5 The horizontal side of the selection rectangle 200 shown in FIG. 2 is the horizontal axis of the xy axis coordinate system, and the vertical side is the vertical axis) of the selection rectangle 200 is the minimum value of the distance.

[0077] B2, collect each frame of gesture state image under each gesture instruction at a specified frequency, and calculate the position change data of the gesture state in each frame image in the corresponding travel segment, wherein the first position change data of the thumb fingertip in each travel segment constitutes the gesture travel data learned under the gesture instruction, and the second position change data of each hand key point in each travel segment constitutes the gesture curvature data learned under the gesture instruction.

[0078] For example, assuming that the gesture instruction i1 is to control a device to perform action a, the gesture state corresponding to the activation of the gesture instruction i1 is: the palm is stretched out, the thumb and index finger gradually move closer and finally touch, and the other fingers are naturally bent in the process. Then in step B2, each frame of the gesture state image of the user making the above gesture state under the gesture instruction i1 is collected at a specified frequency, and the position change data of the gesture state in each frame of the image in the corresponding travel segment is calculated.

[0079] The first position change data and the second position change data are specifically described below:

[0080] Let the sequence of several frames of gesture state images collected for the specified gesture instruction, such as gesture instruction i1, in step B2 be s i1 ={p1, p2, ..., p j ,…,p m}, p j represents the j-th frame gesture state image, j=1,2,…,m, m represents the image frame number. The first position change data includes: the first relative coordinate position of the thumb fingertip point and the lower right vertex of the initial numbering grid where it is located in the stretched state at the end of the stroke segment j+1 time point of the stroke segment, and the first number jump number of the first fingertip point numbering grid where the thumb fingertip point is located at the end of the stroke segment j+1 time point relative to the second fingertip point numbering grid where the thumb fingertip point is located at the start of the stroke segment forming the stroke segment j time point.

[0081] For example, assuming that each frame is a travel segment, when j=1, it is the start time point of the travel segment when the thumb tip slides in a travel segment, and when j+1 is the end time point of the travel segment when the thumb tip slides in the travel segment. At the start time point of the travel segment, assuming that the thumb tip is located at Figure 5 In the numbered grid numbered "108" shown, the numbered grid 108 is assumed to be the initial numbered grid where the thumb tip is in the stretched palm state. The lower right vertex of the numbered grid 108 is Figure 5 Assume that at time j+1, the thumb tip point k4 falls into Figure 5For the grid No. 95 shown in the figure, the first relative coordinate position of the thumb tip point in this travel segment is: the relative coordinate position of the thumb tip point k4 in the grid No. 95 and the lower right vertex p1 in the grid No. 108 (defined as the first relative coordinate position). The "relativity" of the coordinate position here is reflected in: the relative distance and relative direction between the thumb tip point k4 and the lower right vertex p1. The first number jump number generated in this travel segment is: the number jump number of the grid No. 95 and the grid No. 108 (defined as the first number jump number), that is, 108-95=13.

[0082] The second position change data includes: the second relative coordinate position of the hand key point and the upper left vertex of the starting number grid where the hand key point is located in the stretched state at the j+1 time point when the stroke segment ends, and the second number jump number of the first hand key point number grid where the hand key point is located at the j+1 time point when the stroke segment ends relative to the second hand key point number grid where the hand key point is located at the j time point when the stroke segment starts.

[0083] For example, Figure 5 In the segment where the thumb tip point k4 moves from the 108th grid to the 95th grid, assume that Figure 5 The index fingertip key point k8 in the stroke segment moves from the 21st grid to the 34th grid. The second relative coordinate position of the index fingertip key point k8 in the stroke segment is: the relative coordinate position (defined as the second relative coordinate position) of the top left vertex p2 of the starting grid (21st grid) where the index fingertip key point k8 is located when the palm is stretched at the end of the stroke segment j+1. In this stroke segment, the second number jump number corresponding to the index fingertip key point k8 is 34-21=13.

[0084] In the above example, it should be noted that the first relative coordinate position and the second relative coordinate position can be calculated by the existing algorithm, because Figure 5 The coordinates of each vertex of each numbered grid in the shown selection rectangle are known, and the distance between the hand key point and each vertex of the numbered grid can be calculated using an existing algorithm. Therefore, the coordinates of the landing point of the hand key point in the numbered grid can be derived and calculated, and finally the above-mentioned relative coordinate position can be calculated.

[0085] In addition, in step S1, the method for associating the gesture stroke data and the gesture curvature data generated in the same stroke segment is, for example:

[0086] A unique code is assigned to each stroke segment under each gesture instruction, and the gesture stroke data and gesture curvature data generated under the same stroke segment are assigned a unique code of the corresponding stroke segment. In this way, an association relationship marked by the unique stroke segment code is established between the gesture stroke data and the gesture curvature data.

[0087] After completing the learning of the gesture state characteristics of each type of gesture command in step S1, the permission verification and device control method based on gesture recognition in the smart cockpit scenario provided by this embodiment proceeds to step:

[0088] S2, identifying a suspected gesture instruction based on the gesture state data collected from the user in real time, and then obtaining gesture learning data learned by the user for the suspected gesture instruction, including a stroke learning fitting curve drawn in advance based on the gesture stroke data learned for the suspected gesture instruction;

[0089] Specifically, in step S2, the method for collecting the user's gesture state data in real time includes the following steps:

[0090] C1 is formed by discretizing and numbering the grid in step B1. Figure 5 The selection rectangle shown is the first frame gesture image obtained by virtual projection tracking in quasi-real-time acquisition, which specifically includes the following steps:

[0091] C11, Adjustment Figure 5 The projection posture of the selection rectangle shown in FIG. 1 makes the recognized thumb tip point fall into the initial numbered grid in the selection rectangle projected parallel to the gesture; for example, after the camera recognizes the key points of the hand, the recognized thumb tip point is used as the alignment point for virtually projecting the selection rectangle onto the recognized gesture, and the corresponding size generated in step B1 is as follows Figure 5 The initial numbered grid in the selection rectangle shown (i.e. Figure 5 108 grid in the figure) is projected onto the recognized thumb tip point; it should be noted here that the projection of the selection rectangle onto the gesture is a parallel projection, which means that the recognized hand key points fall into the projected selection rectangle.

[0092] C12, judging whether the end fingertip point recognized in real time falls into the middle fingertip landing area in the projected frame selection rectangle (i.e. Figure 5 The middle fingertip landing area 100),

[0093] If yes, then determine the first frame of gesture image that is successfully tracked;

[0094] If not, the user is prompted to adjust the gesture posture to make the palm stretch out, and then the process returns to step C11.

[0095] Steps C11-C12 will ensure the accuracy of gesture recognition by partially pre-aligning the thumb and middle finger tips through the subsequent recognition algorithm, and calculate the gesture travel data and gesture curvature data in each travel segment for each frame of the real-time collected image in a projection manner through a gridded and numbered minimum rectangular frame, which greatly simplifies the subsequent gesture recognition process and is conducive to greatly improving the accuracy of gesture recognition.

[0096] After the first frame of gesture image to be collected in real time is tracked through step C1, in step S2, the method for collecting the user's gesture state data in real time proceeds to step:

[0097] C2, starting from the first frame of gesture image tracking, real-time acquisition of continuous frame image sequence at a preset frequency until the continuous acquisition termination condition is reached, and the position change data of the gesture state in each frame image in the continuous frame image sequence obtained after the acquisition in the corresponding travel segment is used as the gesture state data collected in real time from the user;

[0098] The termination condition of continuous acquisition is: the relative distance between the thumb tip point in each frame of the continuous frame image sequence and the lower right vertex of the initial numbered grid in the selection rectangle decreases with the increase of the number of acquisition frames and reaches the minimum relative distance. Figure 5 The relative distance of the lower right vertex p1 of the initial numbered grid (grid numbered 108) in the selected rectangle shown decreases with the increase of the number of acquired frames and reaches the minimum relative distance.

[0099] It should also be noted that in step C1, tracking the first frame of gesture image collected in real time is not only to obtain a continuous frame image sequence as a basis for confirming whether the suspected gesture instruction is the real corresponding gesture instruction, but also to make a preliminary verification of whether the user has the authority to control the corresponding device with gesture instructions. When the first frame of gesture image cannot be tracked continuously, the system will determine that the user does not have the authority to control the device with gesture instructions, and issue a prompt alarm, such as prompting the user to learn gesture instructions.

[0100] In step S2, according to the gesture state data collected in real time, the method for identifying suspected gesture instructions specifically includes the following steps:

[0101] D1, calculate the position change of the thumb tip point in the last frame relative to the first frame in the continuous frame image sequence, and calculate the number of grid number jumps of the last frame number grid where each hand key point of the user is located in the last frame relative to the first frame number grid where it is located in the first frame, and then solve the average number of jumps of the grid number corresponding to all hand key points;

[0102] D2, match whether there is a gesture instruction whose first deviation between the position change scalar and the position change (the absolute value of the difference between the position change scalar and the position change) is less than a preset first deviation threshold, and whose second deviation between the standard mean of the number of beats and the mean of the number of beats (the absolute value of the difference between the standard mean of the number of beats and the mean of the number of beats) is less than a preset second deviation threshold from the gesture instruction library that has completed gesture state learning.

[0103] If yes, the matched gesture command is identified as a suspected gesture command to be represented by the continuous frame image sequence;

[0104] If not, it is determined that the recognition of the suspected gesture instruction of the continuous frame image sequence has failed, and the user is prompted to learn the suspected gesture instruction.

[0105] The following is an example of implementing steps D1-D2:

[0106] In step D1, it is assumed that in the first frame of the continuous frame image sequence, the thumb tip point falls within Figure 5 The distance between the thumb tip and the lower right vertex p1 of the 108th grid is assumed to be L1. In the last frame, the thumb tip is assumed to fall into the Figure 5 Grid No. 57 in , at this time, its distance from the lower right vertex p1 is assumed to be L2, then the position change of the thumb tip point in the last frame relative to the first frame in the continuous frame image sequence is L2-L1.

[0107] Taking the number of grid number jumps of the thumb fingertip as an example, the number of grid number jumps of each hand key point of the user in the last frame number grid relative to the first frame number grid in the first frame is explained. For example, in the above example, the thumb fingertip jumps from the number grid 108 to the number grid 57, then the number of grid number jumps of the thumb fingertip in the first frame relative to the last frame is 108-57=51. The mean of the number of jumps is the mean of the number of grid number jumps corresponding to all hand key points. The number of grid number jumps is the absolute value of the difference between the number of grid number jumps of the first frame number grid and the number of grid number jumps of the last frame number grid.

[0108] In step D2, the calculation method of the position change scalar is the same as the calculation method of the position change amount, and the calculation method of the standard mean of the number of jumps is also the same as the mean of the number of jumps. The difference is that the calculation basis data of the position change scalar and the standard mean of the number of jumps is the gesture state feature learning data of the suspected gesture command.

[0109] In step S2, the method for drawing the travel learning fitting curve is briefly described as follows:

[0110] like Figure 6In the stroke fitting curve 300 and the stroke learning fitting curve 400, the fitting points have the corresponding acquisition frequency in the continuous frame image sequence as the coordinate value of the x-axis, and the distance of the thumb tip from the lower right vertex p1 of the initial numbered grid in the acquisition frequency as the coordinate value of the y-axis. The fitting points corresponding to each frame in the continuous frame image sequence constitute the data points used to fit the fitting curve 300. Then, a quadratic function or other high-order equations are used to fit each data point to obtain the stroke fitting curve. The fitting method of the stroke learning fitting curve is the same as the fitting method of the stroke fitting curve, except that the fitting data is the learning data of the gesture state feature.

[0111] After obtaining the gesture learning data learned by the user for the suspected gesture command through step S2, the permission verification and device control method based on gesture recognition in the smart cockpit scenario provided by this embodiment proceeds to step:

[0112] S3, based on the real-time collected gesture state data, draw a travel fitting curve of the user's thumb fingertip point (such as Figure 6 Figure 300 (example);

[0113] S4, judging whether the itinerary fitting curve and the itinerary learning fitting curve are similar.

[0114] If yes, go to step S5;

[0115] If not, an alarm is given to the user indicating that the user does not have the authority to control the corresponding device with the suspected gesture instruction, and then the process returns to step S2;

[0116] If it is necessary to explain here, Figure 6 There are many existing methods for similarity matching of the two curves in the example, such as determining whether the overlap of the two curves exceeds an overlap threshold. If so, it is determined that the two curves are similar, so no specific expansion is made.

[0117] The purpose of step S4 is to identify irregular control gestures to reduce the possibility of false triggering of commands, on the premise that it is determined by step S2 that the user has the authority to control the corresponding device with suspected gesture commands, especially when there are a large number of commands controlled by the same palm and the distinction between the gesture state characteristics of each command is low, so that irregular control fingers can be effectively identified to reduce the possibility of false triggering of commands.

[0118] After step S4, Figure 1 As shown, the permission verification and device control method based on gesture recognition in the smart cockpit scenario provided by this embodiment enters the steps:

[0119] S5, determining whether a first change state of the user's gesture recorded in the real-time collected and processed gesture state data at the thumb tip point in each first travel segment is similar to a second change state of the gesture in the corresponding second travel segment recorded in the gesture learning data of the suspected gesture instruction,

[0120] If yes, the corresponding device is controlled by the suspected gesture command;

[0121] If not, the user is prompted that the control gesture is invalid, and then the process returns to step S2.

[0122] The data contents of the first change state and the second change state are explained below:

[0123] The continuous frame image sequence is divided into a number of first run segments, and a continuous frame image subsequence corresponding to each first run segment is obtained. i ={p1, p2, ..., p j ,…,p m}, assuming that three frames of images constitute a first run segment, the continuous frame image subsequence corresponding to the first run segment is expressed as {p j , p j+1 , p j+2}.

[0124] The first change state corresponding to the first stroke segment includes: the first grid number jump number and the first change position change amount of the position of each hand key point recognized by the user in real time in the last frame relative to the first frame of the continuous frame image subsequence. The first grid number jump number is, for example: assuming that the last frame of the thumb tip in a first stroke segment falls into Figure 5 The first frame of the grid No. 82 shown in the figure falls on Figure 5 The first change is: the number of grids with the number 108 in the last frame after the thumb tip point falls into the number 82 in the last frame. Figure 5 The absolute value of the difference between the distance L3 from the lower right vertex p1 in the first frame and the distance L4 from the lower right vertex p1 when it falls into the numbered grid No. 108 in the first frame. The calculation method of the first grid number jump number and the first change amount of other hand key points in each first stroke segment is the same as the calculation method of the first grid number jump number and the first change amount of the thumb fingertip point in the above example, except that when calculating the first change amount of other hand key points, the relative distance between the hand key point and the upper left vertex of the numbered grid where it falls in the first frame of the stroke segment is calculated.

[0125] The calculation principle of the second grid number jump number and the second position change is the same as the calculation principle of the first grid number jump number and the first position variable, the difference is that the gesture learning data for the suspected gesture instruction is used.

[0126] The corresponding relationship between the first stroke segment and the second stroke segment is described as follows:

[0127] For example, for suspected gesture commands, by expressing s i-学习 ={p 1-学习 , p 2-学习 ,…,p j-学习 ,…,p m-学习 The gesture state learning is performed on the continuous frame image sequence of}, and the continuous frame image sequence is divided into a number of second trip segments arranged in sequence, with every 3 frames as a second trip segment. The gesture state data collected in real time is expressed as s i ={p1, p2, ..., p j ,…,p m}, and each 3 frames is a first run segment. i The first trip segments and the second trip segments with the same order have a corresponding relationship.

[0128] Step S5 specifically includes the following steps:

[0129] D1, calculate the mean of the number of jumps of the first grid number of each hand key point in each first stroke segment (defined as the first mean), and then calculate the mean of the first mean associated with each first stroke segment (defined as the second mean); and calculate the mean of the number of jumps of each second grid number in each second stroke segment (defined as the third mean), and then calculate the mean of the third mean associated with each second stroke segment (defined as the fourth mean);

[0130] D2, judging whether the deviation (absolute value of the difference) between the second mean and the fourth mean is less than a preset third deviation threshold,

[0131] If yes, go to step D3;

[0132] If not, it is determined that the first change state and the second change state are not similar;

[0133] D3, calculate the mean of the first position change of each hand key point in each first stroke segment (defined as the fifth mean), and then calculate the mean of the fifth mean associated with each first stroke segment (defined as the sixth mean); and calculate the mean of each second change in each second stroke segment (defined as the seventh mean), and then calculate the mean of the seventh mean associated with each second stroke segment (defined as the eighth mean);

[0134] D4, judging whether the deviation (absolute value of the difference) between the sixth mean and the eighth mean is less than a preset fourth deviation threshold,

[0135] If so, it is determined that the first change state and the second change state are similar;

[0136] If not, it is determined that the first change state and the second change state have no similarity.

[0137] It should be noted that when the same palm can complete more gesture commands, the feature differences between different gesture states will generally be smaller, and the same hand key point may have the same number of grid number jumps but different position changes in the first and second travel segments with a corresponding relationship, or the same position change but different number of grid number jumps. In order to reduce the occurrence of misidentification of gesture commands, in this embodiment, the accuracy of gesture command recognition is improved through the dual judgment of step D2 and step D4.

[0138] In summary, the present invention constructs a correlation relationship between the thumb stroke and the gesture curvature in each thumb stroke segment, and uses the correlation relationship to accurately characterize the subtle feature differences between different gesture states corresponding to each captured gesture instruction. Based on these subtle feature differences, the suspected gesture instructions corresponding to the gesture state data collected in real time can be accurately identified, and the similarity between the first change state and the second change state is verified through step S5, thereby improving the accuracy of gesture state recognition based on these subtle feature differences, making it possible to avoid the use of complex gesture recognition algorithms such as neural network model learning, and at the same time improving the efficiency of gesture recognition.

[0139] It should be noted that the above specific implementations are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art should understand that various modifications, equivalent substitutions, changes, etc. can be made to the present invention. However, as long as these changes do not deviate from the spirit of the present invention, they should be within the scope of protection of the present invention. In addition, some terms used in the specification and claims of this application are not restrictive, but are only for the convenience of description.

Claims

1. A method for authority verification and device control based on gesture recognition in a smart cockpit scenario, characterized in that the steps include: S1, setting the initial state of the slide to a stretched state that caters to the user's palm, and then learning the gesture state characteristics of the user under different gesture instructions, the learning content includes: gesture stroke data and gesture curvature data under each gesture instruction, and the correlation relationship between the gesture stroke data and the gesture curvature data generated in the same stroke segment; S2, identifying suspected gesture instructions based on the gesture state data collected and processed in real time from the user, and then obtaining gesture learning data learned by the user for the suspected gesture instructions, including a stroke learning fitting curve drawn in advance based on the gesture stroke data learned for the suspected gesture instructions; S3, drawing a stroke fitting curve of the user's thumb fingertip point according to the gesture state data collected in real time; S4, determining whether the travel fitting curve and the travel learning fitting curve have similarity, If yes, go to step S5; If not, an alarm is given and the user is prompted that he does not have the authority to control the corresponding device with the suspected gesture instruction, and then the process returns to step S2; S5, determining whether a first change state of the user's gesture recorded in the gesture state data in each first stroke segment of the thumb fingertip point is similar to a second change state of the gesture in the corresponding second stroke segment recorded in the gesture learning data, If yes, the corresponding device is controlled to work by using the suspected gesture command; If not, the user is prompted that the control gesture is invalid, and then the process returns to step S2.

2. The method for authority verification and device control based on gesture recognition in a smart cockpit scenario according to claim 1 is characterized in that: In step S1, the method of setting the initial state of the slideway to the extended state comprises the steps of: A1, the user presses the thumb button at the initial position on the gesture learning device with the tip of the thumb, so that the thumb button is fixed to the slide; A2, the thumb tip pushes the thumb button to drive the slideway to slide along the slide rail provided in the slideway until the user's middle finger tip moves to the press button provided on the gesture learning device when the palm is stretched; A3, press the push button to fix the slideway relative to the slide rail, and then press the thumb button a second time to allow the thumb button to slide freely in the slideway.

3. The method for authority verification and device control based on gesture recognition in a smart cockpit scenario according to claim 2 is characterized in that: The slideway covers the thumb button with a soft material, and the thumb button is provided with a protrusion for pressing against the thumb.

4. The method for authority verification and device control based on gesture recognition in a smart cockpit scenario according to claim 1 is characterized in that: In step S1, the method for learning the gesture state of the user under different gesture instructions includes the steps of: B1, after selecting the minimum rectangle of the key points of the hand in the palm image in the stretched state, the selected rectangle is discretized into grids and the discrete grids are numbered to obtain the numbered grid where each key point of the hand is located; B2, collecting each frame of gesture state image under each gesture instruction at a specified frequency, and calculating the position change data of the gesture state in each frame image under the corresponding stroke segment, wherein the first position change data of the thumb fingertip in each stroke segment constitutes the gesture stroke data learned under the gesture instruction, and the second position change data of each hand key point in each stroke segment constitutes the gesture curvature data learned under the gesture instruction.

5. The method for authority verification and device control based on gesture recognition in a smart cockpit scenario according to claim 4 is characterized in that: Let the sequence s consisting of several frames of gesture state images collected for the specified gesture instruction i in step B2 be i ={p1,p2,…,p j ,…,p m },p j represents the j-th frame gesture state image, j=1,2,…,m, m represents the image frame number, the first position change data includes: the first relative coordinate position of the thumb fingertip point and the lower right vertex of the initial numbering grid where it is located in the stretched state at the end j+1 time point of the stroke segment forming the stroke segment, and the first numbering jump number of the first fingertip point numbering grid where the thumb fingertip point is located at the end j+1 time point of the stroke segment relative to the second fingertip point numbering grid where the thumb fingertip point is located at the start j time point of the stroke segment forming the stroke segment; The second position change data includes: the second relative coordinate position of the hand key point and the upper left vertex of the starting numbering grid where the hand key point is located in the stretched state at the j+1 time point when the stroke segment ends, and the second number jump number of the first hand key point numbering grid where the hand key point is located at the j+1 time point when the stroke segment ends relative to the second hand key point numbering grid where the hand key point is located at the j time point when the stroke segment starts.

6. The method for authority verification and device control based on gesture recognition in a smart cockpit scenario according to claim 4 is characterized in that: In step S2, the method for collecting the gesture state data of the user in real time comprises the steps of: C1, using the frame selection rectangle formed after the grid discretization and numbering in step B1 as a virtual projection tracking to obtain the first frame gesture image collected in quasi-real time; C2, starting from tracking the first frame gesture image, collecting a continuous frame image sequence in real time at a preset frequency until a continuous collection termination condition is reached, and using the position change data of the gesture state in each frame image in the continuous frame image sequence obtained after the collection in the corresponding travel segment as the gesture state data collected in real time from the user; The continuous acquisition termination condition is that the relative distance between the thumb tip point in each frame in the continuous frame image sequence and the lower right vertex of the initial numbered grid in the frame selection rectangle decreases as the number of acquisition frames increases and reaches a minimum relative distance.

7. The method for authority verification and device control based on gesture recognition in a smart cockpit scenario according to claim 6 is characterized in that: In step C1, the method for tracking the first frame gesture image collected in real time includes the following steps: C11, adjusting the projection posture of the selection rectangle so that the thumb tip point recognized in real time falls into the initial numbered grid in the minimum rectangular frame projected parallel to the gesture; C12, judging whether the middle finger tip point recognized in real time falls into the middle finger tip landing area grid in the projected frame selection rectangle, If yes, it is determined that the first frame gesture image is tracked successfully; If not, return to step C11.

8. The method for authority verification and device control based on gesture recognition in a smart cockpit scenario according to claim 6 is characterized in that: In step S2, the method for identifying the suspected gesture instruction includes the steps of: D1, calculating the position change of the thumb fingertip point in the last frame relative to the first frame in the continuous frame image sequence, and calculating the number of grid number jumps of the last frame number grid where each hand key point of the user is located in the last frame relative to the first frame number grid where each hand key point is located in the first frame, and then solving the average number of jumps of the grid number jumps corresponding to all the hand key points respectively; D2, matching whether there is a gesture instruction whose first deviation between the position change scalar and the position change is less than a preset first deviation threshold, and whose second deviation between the standard mean of the number of beats and the mean of the number of beats is less than a preset second deviation threshold from the gesture instruction library that has completed gesture state learning, If yes, the matched gesture instruction is identified as the suspected gesture instruction that the continuous frame image sequence intends to represent; If not, it is determined that the recognition of the suspected gesture instruction of the continuous frame image sequence has failed, and the user is prompted to learn the instruction.

9. The method for authority verification and device control based on gesture recognition in a smart cockpit scenario according to claim 6 is characterized in that: The continuous frame image sequence is divided into a plurality of the first run segments, and a continuous frame image subsequence corresponding to each of the first run segments is obtained. In step S5, the first change state corresponding to the first run segment includes: the number of first grid number jumps and the first position change amount of the position of the last frame relative to the first frame in the continuous frame image subsequence of each hand key point recognized by the user in real time; the second change state corresponding to the second run segment having a corresponding relationship with the first run segment includes: the number of second grid number jumps and the second position change amount of the position of the last frame relative to the first frame in the continuous frame learning subsequence of each hand key point learned by the user for the second run segment in the suspected finger instruction; The corresponding relationship between the first trip segment and the second trip segment is that the first trip segment and the second trip segment have the same data collection start frequency and the same data collection end frequency.

10. The method for authority verification and device control based on gesture recognition in a smart cockpit scenario according to claim 9, characterized in that: The method for determining similarity in step S5 comprises the following steps: D1, calculating a first mean of the number of jumps of the first grid numbers of each of the hand key points in each of the first travel segments, and then calculating a second mean of the first mean values ​​associated with each of the first travel segments; and calculating a third mean of the number of jumps of each of the second grid numbers in each of the second travel segments, and then calculating a fourth mean of the third mean values ​​associated with each of the second travel segments; D2, determining whether the deviation between the second mean and the fourth mean is less than a preset third deviation threshold, If yes, go to step D3; If not, determining that the first change state and the second change state have no similarity; D3, calculating the fifth mean of the first position change of each of the hand key points in each of the first travel segments, and then calculating the sixth mean of the fifth mean associated with each of the first travel segments; and calculating the seventh mean of each of the second change in each of the second travel segments, and then calculating the eighth mean of the seventh mean associated with each of the second travel segments; D4, determining whether the deviation between the sixth mean and the eighth mean is less than a preset fourth deviation threshold, If so, determining that the first change state and the second change state have similarity; If not, it is determined that the first change state and the second change state have no similarity.

Images