An intelligent cockpit gesture control method, device and storage medium

Abstract

The application relates to an intelligent cockpit gesture control method and device and a storage medium. The method comprises the following steps: acquiring a to-be-recognized image sequence; the to-be-recognized image sequence is obtained by a gesture acquisition unit collecting a preset area within a preset time; the preset area comprises a plurality of sub-areas, and the plurality of sub-areas correspond to a plurality of controllable units in the intelligent cockpit in a one-to-one manner; a first gesture and a second gesture are sequentially recognized from a preset frame number of the to-be-recognized image sequence; a to-be-controlled unit is determined from the plurality of controllable units according to the first gesture, the second gesture and a sub-area corresponding to the first gesture; the second gesture is tracked from a first frame image in which the second gesture is recognized, and an activity track corresponding to the second gesture is recognized; a control instruction is determined based on the activity track, and the to-be-controlled unit is controlled according to the control instruction. In this way, the control range of gesture operation can be greatly expanded in the intelligent cockpit, thereby improving the interactive experience.

Description

Technical Field

[0001] This application relates to the field of automotive technology, and in particular to a smart cockpit gesture control method, device, and storage medium. Background Technology

[0002] In recent years, intelligent cockpits have become a key research area for major OEMs both domestically and internationally. Intelligent cockpits manifest in many ways, one important aspect being the innovation of human-machine interaction, moving beyond simply tapping screens as the core of operation.

[0003] Currently, smart cockpits have made rapid progress in voice interaction, achieving significant milestones. Like voice interaction, gesture interaction, as a natural form of human interaction, is the second largest channel for humans to output information naturally, and also the most important supplementary communication method when language barriers exist (such as sign language for the deaf and mute, and gestures when language is not used).

[0004] Currently, gesture control in car cabins mainly focuses on recognizing static gestures and single dynamic gestures. The gesture states are relatively simple, and the corresponding operation commands in the car are also relatively simple. Some gestures are not commonly used and do not conform to human operating intuition. Summary of the Invention

[0005] This application provides a smart cockpit gesture control method, device, and storage medium, which can greatly expand the control range of gesture operations in a smart cockpit, thereby improving the interactive experience.

[0006] On one hand, embodiments of this application provide a smart cockpit gesture control method, including:

[0007] Acquire the image sequence to be recognized; the image sequence to be recognized is acquired by the gesture acquisition unit from the preset area within a preset time; the preset area includes multiple sub-areas, and each sub-area corresponds to a different controllable unit in the smart cockpit.

[0008] The first gesture and the second gesture are sequentially identified from a preset number of frames in the image sequence to be identified; the first gesture corresponds to any one of multiple sub-regions; the second gesture corresponds to the same sub-region as the first gesture; the similarity value between the first gesture and the second gesture is less than or equal to a first preset value.

[0009] Based on the first gesture, the second gesture, and the sub-region corresponding to the first gesture, the unit to be controlled is determined from multiple controllable units;

[0010] Starting from the first frame image of the second gesture, the second gesture is tracked to identify the activity trajectory corresponding to the second gesture;

[0011] The control commands are determined based on the activity trajectory, and the control unit to be controlled is controlled according to the control commands.

[0012] Optionally, after controlling the control unit according to the control command, the following steps are also included:

[0013] The third gesture is identified from the remaining images in the image sequence to be identified; the third gesture corresponds to any one or more sub-regions in multiple sub-regions; the similarity value between the third gesture and the second gesture is less than a second preset value; the remaining images are images in the image sequence to be identified excluding the images corresponding to the preset frame number and the images used to track the second gesture;

[0014] The release command is determined based on the third gesture, and the state of controlling the unit to be controlled is exited based on the release command.

[0015] Optionally, the first gesture and the second gesture are sequentially identified from a preset number of frames in the image sequence to be identified, including:

[0016] Obtain the trained first gesture recognition model;

[0017] The preset number of frames of the image sequence to be recognized are sequentially input into the first gesture recognition model for the first gesture recognition;

[0018] When the first gesture is recognized, the next frame image after the first gesture and the images after the next frame image are sequentially input into the acquired trained second gesture recognition model for second gesture recognition until the second gesture is recognized.

[0019] Optionally, starting from the first frame image where the second gesture is recognized, the second gesture is tracked to identify the activity trajectory corresponding to the second gesture, including:

[0020] Starting from the first frame image in which the second gesture is recognized, determine the position of the second gesture in each frame image;

[0021] The trajectory of the activity to be identified is determined based on the position of the second gesture in each frame of the image;

[0022] Obtain a set of preset activity trajectories;

[0023] From the preset set of activity trajectories, determine the activity trajectory that matches the activity trajectory to be identified, and define the activity trajectory that matches the activity trajectory to be identified as the activity trajectory corresponding to the second gesture.

[0024] Optionally, the gesture acquisition unit includes a camera; the preset area is located above the secondary instrument panel in the smart cockpit;

[0025] Obtain the image sequence to be recognized, including:

[0026] When entering the smart cockpit gesture control state, the camera continuously or at a set frequency takes pictures of the area above the passenger instrument panel in the smart cockpit to obtain a sequence of images to be recognized.

[0027] Optionally, multiple controllable units include the front passenger seat; the sub-area corresponding to the first gesture corresponds to the front passenger seat;

[0028] The activity trajectory includes any one or more combinations of moving in the first direction, moving in the second direction, moving in the third direction, moving in the fourth direction, moving in the fifth direction, moving in the sixth direction, flipping forward, and flipping backward;

[0029] Control commands are determined based on the activity trajectory, including:

[0030] When the activity trajectory is moving in the first direction, the control command is determined to be to move the passenger seat forward.

[0031] Alternatively; when the movement trajectory is forward flip, the control command is determined to tilt the back of the passenger seat forward.

[0032] Optionally, the first and third gestures are both with the palm open, and the second gesture is with the hand clenched into a fist.

[0033] Optionally, the preset area is located above the rear console of the smart cockpit;

[0034] Obtain the image sequence to be recognized, including:

[0035] When the smart cockpit gesture control mode is entered, the gesture acquisition unit continuously or at a set frequency takes pictures of the area above the rear console of the smart cockpit to obtain the image sequence to be recognized.

[0036] On the other hand, embodiments of this application provide a smart cockpit gesture control device, including:

[0037] The acquisition module is used to acquire the image sequence to be recognized. The image sequence to be recognized is acquired by the gesture acquisition unit from the preset area within a preset time. The preset area includes multiple sub-areas, and each sub-area corresponds to a different controllable unit in the smart cockpit.

[0038] The first recognition module is used to sequentially recognize a first gesture and a second gesture from a preset number of frames in the image sequence to be recognized; the first gesture corresponds to any one of multiple sub-regions; the second gesture corresponds to the same sub-region as the first gesture; the similarity value between the first gesture and the second gesture is less than or equal to a first preset value.

[0039] The first determining module is used to determine the unit to be controlled from multiple controllable units based on the first gesture, the second gesture, and the sub-region corresponding to the first gesture.

[0040] The second recognition module is used to track the second gesture from the first frame image after the second gesture is recognized, and to recognize the activity trajectory corresponding to the second gesture.

[0041] The second determining module is used to determine the control command based on the activity trajectory and to control the unit to be controlled according to the control command.

[0042] On the other hand, embodiments of this application provide a computer storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described smart cockpit gesture control method.

[0043] The smart cockpit gesture control method, device, and storage medium provided in this application have the following beneficial effects:

[0044] The process involves acquiring a sequence of images to be recognized. This sequence is obtained by a gesture acquisition unit capturing images of a preset area within a preset time period. The preset area includes multiple sub-regions, each corresponding to a different controllable unit within the smart cockpit. A first gesture and a second gesture are sequentially identified from a preset number of frames in the image sequence. The first gesture corresponds to any one of the sub-regions. The second gesture corresponds to the same sub-region as the first gesture. The similarity between the first and second gestures is less than or equal to a first preset value. Based on the first gesture, the second gesture, and the sub-region corresponding to the first gesture, a controllable unit is determined from the multiple controllable units. Starting from the first frame image after the second gesture is identified, the second gesture is tracked to determine its corresponding movement trajectory. A control command is determined based on the movement trajectory, and the controllable unit is then controlled according to the command. This significantly expands the control range of gesture operations within the smart cockpit, thereby enhancing the interactive experience. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of an intelligent cockpit application scenario provided in an embodiment of this application;

[0047] Figure 2 This is a flowchart illustrating a smart cockpit gesture control method provided in an embodiment of this application;

[0048] Figure 3This is a schematic diagram of a preset region and sub-regions in an image to be identified, provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of a specific gesture provided in an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of a gesture recognition result provided in an embodiment of this application;

[0051] Figure 6 This is a schematic diagram of the structure of an intelligent cockpit gesture control device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0053] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0054] Please see Figure 1 , Figure 1 This is a schematic diagram of an intelligent cockpit application scenario provided in an embodiment of this application. The intelligent cockpit is equipped with a gesture acquisition unit 101, which is used to acquire data from a preset area 102 and input the acquired information into a computing unit for calculation and recognition.

[0055] It should be noted that, Figure 1 The gesture acquisition unit 101 is located above the central control display screen, and the preset area 102 is located above the central control panel, in the area where the driver's right hand can move. In other embodiments, depending on actual needs, the gesture acquisition unit 101 can also be located in other parts of the smart cockpit, such as behind the front center armrest, so as to detect and acquire the preset area above the rear center armrest.

[0056] The gesture acquisition unit 101 acquires a sequence of images to be recognized from a preset area 102 within a preset time. The preset area includes multiple sub-areas, each corresponding to a different controllable unit within the smart cockpit. Then, a first gesture and a second gesture are sequentially identified from a preset number of frames in the sequence of images to be recognized. The first gesture corresponds to any one of the multiple sub-areas. The second gesture corresponds to the same sub-area as the first gesture. The similarity between the first and second gestures is less than or equal to a first preset value. Then, based on the first gesture, the second gesture, and the sub-area corresponding to the first gesture, the controllable unit is determined from the multiple controllable units. Starting from the first frame image of the identified second gesture, the second gesture is tracked to identify the activity trajectory corresponding to the second gesture. Finally, the control command is determined based on the activity trajectory, and the controllable unit is controlled according to the control command.

[0057] Optionally, the smart cockpit may include multiple controllable units such as seat controllers, window controllers, power door controllers, air conditioning vent controllers, streaming rearview mirrors, and instrument panel controllers.

[0058] The following describes a specific embodiment of a smart cockpit gesture control method according to this application. Figure 2 This is a flowchart illustrating a smart cockpit gesture control method provided in an embodiment of this application. This specification provides the method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server product execution, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include:

[0059] S201: Acquire the image sequence to be recognized; the image sequence to be recognized is acquired by the gesture acquisition unit from the preset area within a preset time; the preset area includes multiple sub-areas, and each sub-area corresponds to a different controllable unit in the smart cockpit.

[0060] In this embodiment, a gesture acquisition unit is provided in the smart cockpit. This unit acquires a sequence of images to be recognized from a preset area within a preset time period. The main purpose of limiting the preset time is to analyze only the gesture information acquired within that preset time; gesture information acquired after the preset time is considered invalid. The preset area can be represented as a three-dimensional detection space. The gesture acquisition unit only recognizes gestures falling within this three-dimensional detection space. Multiple sub-regions are derived from this three-dimensional detection space, and each sub-region corresponds one-to-one with multiple controllable units within the smart cockpit. Controllable units refer to components or devices in the smart cockpit that can be controlled by a controller; each sub-region controls one component or device. The size of each sub-region can be the same or different depending on actual needs. The preset area represents the image size of the image sequence to be recognized in the two-dimensional image dimension. Each sub-region corresponds to the same location in each frame of the image sequence to be recognized; for example… Figure 3 As shown, Figure 3 This is a schematic diagram of a preset region and sub-regions in an image to be identified, provided in an embodiment of this application. Sub-region a corresponds to the passenger seat in the smart cockpit, sub-region b corresponds to the air conditioning unit in the smart cockpit, sub-region c corresponds to the passenger window in the smart cockpit, and sub-regions d, etc., can correspond to other components or devices in the smart cockpit according to actual needs. Sub-regions a, b, c, and d correspond to different monitoring spaces in three-dimensional space. It should be noted that... Figure 3 Only one method for dividing sub-regions in an elliptic image is given. Depending on the actual needs, it can also be set to a 9-grid or other irregular division method.

[0061] In one optional implementation, the gesture acquisition unit includes a camera; the preset area is located above the passenger-side dashboard in the smart cockpit, which can also be understood as the area where the driver's right hand can move, facilitating driver control; step S201 specifically includes: when entering the smart cockpit gesture control state, continuously or at a set frequency taking pictures of the area above the passenger-side dashboard in the smart cockpit using the camera to obtain a sequence of images to be recognized. The determination of entering the smart cockpit gesture control state can be based on physical buttons or other contactless methods such as voice.

[0062] In another optional implementation, the preset area is located above the rear console of the smart cockpit, that is, above the center armrest of the rear seat, so as to facilitate the rear passengers to operate the smart cockpit; step S201 may specifically include: when entering the smart cockpit gesture control state, the gesture acquisition unit continuously or at a set frequency takes pictures of the area above the rear console of the smart cockpit to obtain the image sequence to be recognized.

[0063] S203: Sequentially identify the first gesture and the second gesture from a preset number of frames in the image sequence to be identified; the first gesture corresponds to any one of the multiple sub-regions; the second gesture corresponds to the same sub-region as the first gesture; the similarity value between the first gesture and the second gesture is less than or equal to a first preset value.

[0064] In this embodiment, the first gesture represents the selection of a controllable unit, and switching from the first gesture to the second gesture represents the selection of the current controllable unit. The first and second gestures must be two gestures that the computer can recognize; therefore, the similarity value between the first and second gestures must be less than or equal to a first preset value. In extreme cases, the first preset value is 0, meaning the first and second gestures are completely different gestures. By setting the first preset value, the sensitivity and accuracy of recognition can be adjusted. The larger the first preset value, the more sensitive the recognition but the lower the accuracy; the smaller the first preset value, the lower the recognition sensitivity but the higher the accuracy. The preset frame number limits the duration of the user's operation. The first and second gestures are sequentially recognized from the preset frame number of the image sequence to be recognized, indicating that the user has used the first and second gestures consecutively. At the same time, the sub-regions corresponding to the second gesture and the first gesture are the same.

[0065] In one optional implementation, step S203 may specifically include: acquiring a trained first gesture recognition model; sequentially inputting a preset number of frames of the image sequence to be recognized into the first gesture recognition model for first gesture recognition; when the first gesture is recognized, sequentially inputting the next frame image of the currently recognized first gesture and the images after the next frame image into the acquired trained second gesture recognition model for second gesture recognition, until the second gesture is recognized.

[0066] Specifically, training images of the first gesture are collected, and the first gesture recognition model is trained to obtain a trained first gesture recognition model; training images of the second gesture are collected, and the second gesture recognition model is trained to obtain a trained second gesture recognition model; the first gesture and the second gesture are identified from a preset number of frames in the image sequence to be recognized based on an image detection algorithm.

[0067] S205: Determine the controllable unit from multiple controllable units based on the first gesture, the second gesture, and the sub-region corresponding to the first gesture.

[0068] In this embodiment, the unit to be manipulated is determined from multiple manipulateable units based on a first gesture, a second gesture, and a sub-region corresponding to the first gesture. For example, the sub-region corresponding to the first gesture and the second gesture is... Figure 3 In sub-region a, the passenger seat is identified as the unit to be controlled.

[0069] S207: Starting from the first frame image of the second gesture, track the second gesture and identify the activity trajectory corresponding to the second gesture.

[0070] S209: Determine the control command based on the activity trajectory, and control the control unit to be controlled according to the control command.

[0071] In this embodiment of the application, after identifying an image containing a second gesture from the image to be identified, the second gesture is tracked starting from the first frame image of the identified second gesture, and the activity trajectory corresponding to the second gesture is identified. The activity trajectory corresponding to the second gesture represents the user's intention to manipulate, that is, the user wants to control the function of a certain component or device. Thus, the control command is determined according to the activity trajectory, and the control unit to be controlled is controlled according to the control command.

[0072] In one optional implementation, step S207 may specifically include: determining the position of the second gesture in each frame image starting from the first frame image where the second gesture is recognized; determining the activity trajectory to be recognized based on the position of the second gesture in each frame image; obtaining a preset set of activity trajectories; determining the activity trajectory that matches the activity trajectory to be recognized from the preset set of activity trajectories, and determining the activity trajectory that matches the activity trajectory to be recognized as the activity trajectory corresponding to the second gesture.

[0073] Specifically, the preset activity trajectory set may include any one or more combinations of moving in the first direction, moving in the second direction, moving in the third direction, moving in the fourth direction, moving in the fifth direction, moving in the sixth direction, flipping forward, and flipping backward.

[0074] In one optional implementation, the plurality of controllable units include a passenger seat; the sub-area corresponding to the first gesture corresponds to the passenger seat, i.e., the unit to be controlled is the passenger seat; then when the movement trajectory is moving in the first direction, the control command is determined to move the passenger seat forward; or; when the movement trajectory is flipping forward, the control command is determined to tilt the back of the passenger seat forward.

[0075] In an optional implementation, after step S209, the method may further include: identifying a third gesture from the remaining images of the image sequence to be identified; the third gesture corresponds to any one or more sub-regions within a plurality of sub-regions; the similarity value between the third gesture and the second gesture is less than a second preset value; the remaining images are images in the image sequence to be identified excluding images corresponding to a preset frame number and images used to track the second gesture; determining a release command based on the third gesture, and exiting the state of controlling the unit to be controlled based on the release command. The third gesture may be the same as the first gesture.

[0076] Specifically, such as Figure 4As shown, the first and third gestures both involve an open palm, while the second gesture involves a clenched fist. The following specific example illustrates steps S201-S209 and some optional implementation methods. Figure 1 In the application scenario shown, the gesture acquisition unit 101 acquires a sequence of images to be recognized from a preset area 102 within a preset time. The first gesture (open palm) and the second gesture (clenched fist) are then sequentially recognized from a preset number of frames in the sequence of images to be recognized. Figure 5 As shown, in frames t0 and t1, the open hand is located in sub-region a, indicating that it is reaching towards the passenger seat. Similarly, a second gesture (hand clenched into a fist) is detected in sub-region a, indicating selection. The determined control unit is the passenger seat corresponding to sub-region a. According to the movement trajectory of the second gesture (hand clenched into a fist) recognized in frames t3 to tn, which is moving in the first direction (forward), the control command is determined to move the passenger seat forward. Similarly, when the movement trajectory of the second gesture (hand clenched into a fist) recognized in frames t3 to tn is moving in the second direction (backward), the control command is determined to move the passenger seat backward. When the movement trajectory of the second gesture (hand clenched into a fist) recognized in frames t3 to tn is flipping backward, the control command is determined to tilt the passenger seat back backward. When a third gesture (hand open) is recognized in frame tn+1, a release command is generated, and the state of controlling the passenger seat is exited according to the release command.

[0077] It should be noted that, Figure 5 The front, back, left, and right directions represent movement in the first, second, third, and fourth directions, respectively. Movement in the fifth and sixth directions can be determined based on the size changes of the second gesture in the images from frame t3 to frame tn. For example, when the proportion of a clenched fist in frame tn is greater than that in frame t3, it indicates movement in the fifth direction; when the proportion of a clenched fist in frame tn is less than that in frame t3, it indicates movement in the sixth direction.

[0078] The same principle applies to controlling the passenger seat as to controlling the air conditioning, passenger window, and passenger door within the smart cockpit. For example, if the movement trajectory of the second gesture (fist) recognized in frames t3-tn is a third direction (left), the control command is determined to adjust the air conditioning vent direction to the left; if the movement trajectory is a fourth direction (right), the control command is determined to adjust the air conditioning vent direction to the right; if the movement trajectory is a fifth direction, the control command is determined to reduce the air conditioning fan speed. For another example… When the trajectory of the second gesture (fist clenched) recognized in frames t3-tn is a movement in the first direction (forward), the control command is determined to be to move the passenger-side window upward (close the window); when the trajectory of the second gesture (fist clenched) recognized in frames t3-tn is a movement in the second direction (backward), the control command is determined to be to move the passenger-side window downward (open the window); for example, when the trajectory of the second gesture (fist clenched) recognized in frames t3-tn is a movement in the fourth direction (right), the control command is determined to be to open the passenger-side electric door; other electronic control components or devices in the vehicle can be operated in this way, and will not be described in detail. It can be seen that the intelligent cockpit gesture control method provided in this application embodiment can greatly expand the control range of gesture operation and improve the interactive experience in the car cockpit.

[0079] The methods and embodiments provided in this application can be executed on a computer terminal, server, or similar computing device.

[0080] This application also provides an intelligent cockpit gesture control device. Figure 6 This is a schematic diagram of the structure of a smart cockpit gesture control device provided in an embodiment of this application, as shown below. Figure 6 As shown, the device includes:

[0081] The acquisition module 601 is used to acquire the image sequence to be recognized; the image sequence to be recognized is acquired by the gesture acquisition unit from the preset area within a preset time; the preset area includes multiple sub-areas, and the multiple sub-areas correspond one-to-one with multiple controllable units in the smart cockpit;

[0082] The first recognition module 602 is used to sequentially recognize a first gesture and a second gesture from a preset number of frames in the image sequence to be recognized; the first gesture corresponds to any one of multiple sub-regions; the second gesture corresponds to the same sub-region as the first gesture; the similarity value between the first gesture and the second gesture is less than or equal to a first preset value.

[0083] The first determining module 603 is used to determine the unit to be controlled from multiple controllable units based on the first gesture, the second gesture, and the sub-region corresponding to the first gesture.

[0084] The second recognition module 604 is used to track the second gesture from the first frame image of the recognized second gesture and recognize the activity trajectory corresponding to the second gesture.

[0085] The second determining module 605 is used to determine the control command based on the activity trajectory and to control the control unit to be controlled according to the control command.

[0086] The apparatus and method embodiments in this application are based on the same application concept.

[0087] In one optional embodiment, the device further includes a third determining module, used to identify a third gesture from the remaining images of the image sequence to be identified; the third gesture corresponds to any one or more sub-regions in a plurality of sub-regions; the similarity value between the third gesture and the second gesture is less than a second preset value; the remaining images are images in the image sequence to be identified other than the images corresponding to a preset frame number and the images used to track the second gesture; a release command is determined based on the third gesture, and the state of controlling the unit to be controlled is exited based on the release command.

[0088] In one optional implementation, the first recognition module 602 is specifically used to: acquire a trained first gesture recognition model; input a preset number of frames of the image sequence to be recognized into the first gesture recognition model for first gesture recognition; when the first gesture is recognized, input the next frame image of the currently recognized first gesture and the images after the next frame image into the acquired trained second gesture recognition model for second gesture recognition, until the second gesture is recognized.

[0089] In one optional implementation, the second recognition module 604 is specifically used to: determine the position of the second gesture in each frame image starting from the first frame image where the second gesture is recognized; determine the activity trajectory to be recognized based on the position of the second gesture in each frame image; obtain a preset set of activity trajectories; determine the activity trajectory that matches the activity trajectory to be recognized from the preset set of activity trajectories, and determine the activity trajectory that matches the activity trajectory to be recognized as the activity trajectory corresponding to the second gesture.

[0090] In one optional implementation, the gesture acquisition unit includes a camera; the preset area is located above the passenger instrument panel in the smart cockpit; the acquisition module 601 is specifically used to: when entering the smart cockpit gesture control state, continuously or at a set frequency take pictures of the area above the passenger instrument panel in the smart cockpit through the camera to obtain a sequence of images to be recognized.

[0091] In one optional implementation, the plurality of controllable units include a passenger seat; the sub-area corresponding to the first gesture corresponds to the passenger seat; the activity trajectory includes any one or more combinations of moving in a first direction, moving in a second direction, moving in a third direction, moving in a fourth direction, moving in a fifth direction, moving in a sixth direction, flipping forward, and flipping backward; the second determining module 605 is specifically used to: when the activity trajectory is moving in the first direction, determine the control command as moving the passenger seat forward; or; when the activity trajectory is flipping forward, determine the control command as tilting the back of the passenger seat forward.

[0092] In one alternative implementation, the first gesture and the third gesture are both with the palm open, and the second gesture is with the hand clenched into a fist.

[0093] In one optional implementation, the preset area is located above the rear console of the smart cockpit; the acquisition module 601 is specifically used to: when entering the smart cockpit gesture control state, continuously or at a set frequency take pictures of the area above the rear console of the smart cockpit through the gesture acquisition unit to obtain the image sequence to be recognized.

[0094] The embodiments of this application also provide a storage medium, which can be located in a server to store at least one instruction, at least one program, code set, or instruction set related to implementing a smart cockpit gesture control method in the method embodiments. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-described smart cockpit gesture control method.

[0095] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0096] As can be seen from the embodiments of the smart cockpit gesture control method, device, or storage medium provided in this application, this application acquires a sequence of images to be recognized; the sequence of images to be recognized is obtained by a gesture acquisition unit acquiring images of a preset area within a preset time; the preset area includes multiple sub-areas, each sub-area corresponding to a multiple controllable units within the smart cockpit; a first gesture and a second gesture are sequentially identified from a preset number of frames in the sequence of images to be recognized; the first gesture corresponds to any one of the multiple sub-areas; the second gesture corresponds to the same sub-area as the first gesture; the similarity value between the first gesture and the second gesture is less than or equal to a first preset value; based on the first gesture, the second gesture, and the sub-area corresponding to the first gesture, a controllable unit is determined from the multiple controllable units; starting from the first frame image of the identified second gesture, the second gesture is tracked, and the activity trajectory corresponding to the second gesture is identified; a control command is determined based on the activity trajectory, and the controllable unit is controlled according to the control command. In this way, the control range of gesture operations within the smart cockpit can be greatly expanded, thereby improving the interactive experience.

[0097] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0098] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0099] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0100] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for gesture control of an intelligent cockpit, characterized in that, include: When the smart cockpit enters gesture control mode, the gesture acquisition unit continuously or at a set frequency captures images of a preset area within a preset time to obtain a sequence of images to be recognized. The preset area is located above the passenger instrument panel or the rear console of the smart cockpit. The preset area can be represented as a three-dimensional detection space, including multiple sub-areas divided by the corresponding three-dimensional detection space. Each sub-area corresponds to a multiple controllable unit in the smart cockpit. The controllable unit represents a component or device in the smart cockpit that can be controlled by a controller. The multiple controllable units in the smart cockpit include a seat controller, window controller, electric door controller, air conditioning vent controller, streaming rearview mirror, and instrument panel controller. Obtain the trained first gesture recognition model; The preset number of frames of the image sequence to be recognized are sequentially input into the first gesture recognition model for first gesture recognition; When the first gesture is recognized, the next frame image of the first gesture and the images after the next frame image are sequentially input into the acquired trained second gesture recognition model for second gesture recognition until the second gesture is recognized. The first gesture corresponds to any one of the plurality of sub-regions; The second gesture corresponds to the same sub-region as the first gesture; the similarity value between the first gesture and the second gesture is less than or equal to a first preset value; the first gesture indicates selection of the controllable unit; Based on the first gesture, the second gesture, and the sub-region corresponding to the first gesture, a controllable unit is determined from the plurality of controllable units; wherein, switching from the first gesture to the second gesture indicates the selection of the corresponding controllable unit; Starting from the first frame image of the second gesture, the second gesture is tracked to identify the activity trajectory corresponding to the second gesture; the activity trajectory includes any one or more combinations of moving in a first direction, moving in a second direction, moving in a third direction, moving in a fourth direction, moving in a fifth direction, moving in a sixth direction, flipping forward, and flipping backward. The control command is determined based on the activity trajectory, and the control unit to be controlled is controlled according to the control command; A third gesture is identified from the remaining images in the image sequence to be identified; the third gesture is a release gesture that is different from the second gesture and is used to exit the control state; the third gesture corresponds to any one or more sub-regions in the plurality of sub-regions; the similarity value between the third gesture and the second gesture is less than a second preset value; the remaining images are the images in the image sequence to be identified excluding the images corresponding to the preset frame number and the images used to track the second gesture; The release command is determined based on the third gesture, and the state of controlling the unit to be controlled is exited based on the release command.

2. The method of claim 1, wherein, The step of tracking the second gesture from the first frame image after the second gesture is identified, and identifying the activity trajectory corresponding to the second gesture, includes: Starting from the first frame image in which the second gesture is recognized, determine the position of the second gesture in each frame image; The trajectory of the activity to be identified is determined based on the position of the second gesture in each frame of the image; Obtain a set of preset activity trajectories; From the preset set of activity trajectories, an activity trajectory that matches the activity trajectory to be identified is determined, and the activity trajectory that matches the activity trajectory to be identified is determined as the activity trajectory corresponding to the second gesture.

3. The method of claim 1, wherein, The gesture acquisition unit includes a camera.

4. The method of claim 1, wherein, The plurality of controllable units include a passenger seat; The sub-area corresponding to the first gesture corresponds to the passenger seat. The determination of control commands based on the activity trajectory includes: When the activity trajectory is moving in the first direction, the control command is determined to be to move the passenger seat forward; Alternatively; when the activity trajectory is the forward flip, the control command is determined to be to tilt the back of the passenger seat forward.

5. The method of claim 1, wherein, Both the first and third gestures involve an open palm, while the second gesture involves a clenched fist.

6. A smart cockpit gesture control device, characterized in that, include: The acquisition module is used to continuously or at a set frequency capture images of a preset area within a preset time period through the gesture acquisition unit when entering the smart cockpit gesture control state, thereby obtaining a sequence of images to be recognized. The preset area is located above the passenger instrument panel or above the rear console of the smart cockpit. The preset area can be represented as a three-dimensional detection space, including multiple sub-areas divided by the corresponding three-dimensional detection space. Each sub-area corresponds to a multiple controllable unit in the smart cockpit. The controllable unit represents a component or device in the smart cockpit that can be controlled by a controller. The multiple controllable units in the smart cockpit include a seat controller, a window controller, an electric door controller, an air conditioning vent controller, a streaming media rearview mirror, and an instrument panel controller. The first recognition module is used to acquire a trained first gesture recognition model; and to input a preset number of frames of the image sequence to be recognized into the first gesture recognition model to perform first gesture recognition. When the first gesture is recognized, the next frame image of the first gesture and the images after the next frame image are sequentially input into the acquired trained second gesture recognition model for second gesture recognition until the second gesture is recognized. The first gesture corresponds to any one of the plurality of sub-regions; The second gesture corresponds to the same sub-region as the first gesture; the similarity value between the first gesture and the second gesture is less than or equal to a first preset value; The first gesture indicates selection of the controllable unit; The first determining module is used to determine the controllable unit from the plurality of controllable units based on the first gesture, the second gesture, and the sub-region corresponding to the first gesture; wherein switching from the first gesture to the second gesture indicates the selection of the corresponding controllable unit; The second recognition module is used to track the second gesture from the first frame image after the second gesture is recognized, and to recognize the activity trajectory corresponding to the second gesture; the activity trajectory includes any one or more combinations of moving in a first direction, moving in a second direction, moving in a third direction, moving in a fourth direction, moving in a fifth direction, moving in a sixth direction, flipping forward, and flipping backward. The second determining module is used to determine the control command based on the activity trajectory, and to control the unit to be controlled according to the control command. The third determining module is used to identify a third gesture from the remaining images of the image sequence to be identified; the third gesture is a release gesture different from the second gesture and used to exit the control state; the third gesture corresponds to any one or more sub-regions in the plurality of sub-regions; the similarity value between the third gesture and the second gesture is less than a second preset value; the remaining images are images in the image sequence to be identified other than the images corresponding to the preset frame number and the images used to track the second gesture; a release command is determined according to the third gesture, and the state of controlling the unit to be controlled is exited according to the release command.

7. A computer storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the smart cockpit gesture control method as described in any one of claims 1-5.

Images