Control method and device for vehicle

Abstract

The application discloses a control method and device of a vehicle, and belongs to the technical field of in-vehicle interaction. The method comprises the following steps: acquiring a hand video, wherein the hand video comprises continuous multiple frames of images, and each frame of image has coordinate information; performing combined tracking of deep features and shallow features on each frame of image in the hand video to determine hand information of each frame of image, wherein the hand information comprises a center position of a hand and a hand image; in response to a distance between the center positions of the hand in adjacent frame group images being less than a threshold value, determining that a hand state is a fixed state; in response to the hand state being the fixed state, identifying a target gesture in the adjacent frame group images, and outputting a target control instruction corresponding to the target gesture. The method can effectively and accurately identify a gesture when the hand is deformed, and improves the reliability of human-computer interaction of the vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of in-vehicle interaction, and particularly relates to a control method and device of a vehicle. BACKGROUND

[0002] At present, the man-machine interaction function of a vehicle can bring better use experience to a user. For example, the user can control the vehicle to perform different functions by making various gestures.

[0003] In the related art, in the control method of the vehicle, the position of the hand of the user is determined based on the color feature of the captured image. However, this determination method, especially when the hand is deformed, it is difficult to recognize the gesture only by the color feature, which affects the gesture recognition function and reduces the reliability of the vehicle man-machine interaction. SUMMARY

[0004] In view of this, the present application provides a control method and device of a vehicle, which can effectively and accurately recognize the gesture when the hand is deformed, and improves the reliability of the vehicle man-machine interaction.

[0005] Specifically, the technical solutions include the following:

[0006] In one aspect, the present application provides a control method of a vehicle, the method comprising:

[0007] obtaining a hand video, wherein the hand video comprises a plurality of continuous frames of images, and each frame of image has coordinate information;

[0008] tracking the combination of deep features and shallow features of each frame of image in the hand video to determine hand information of each frame of image, wherein the hand information comprises a center position of the hand and a hand image;

[0009] in response to a distance between the center positions of the hand in adjacent frame group images being less than a threshold value, determining that a hand state is a fixed state;

[0010] in response to the hand state being the fixed state, recognizing a target gesture in the adjacent frame group images, and outputting a target control instruction corresponding to the target gesture.

[0011] In some embodiments, the tracking the combination of deep features and shallow features of each frame of image in the hand video to determine hand information of each frame of image comprises:

[0012] inputting a first frame of image in the hand video into a backbone network to obtain initial hierarchical features;

[0013] For each frame image subsequent to the first frame, based on the image, a transition hand image is obtained, and the transition hand image is input into the backbone network to obtain a transition image hierarchical feature;

[0014] The initial image hierarchical feature and the transition image hierarchical feature are cross-correlated to obtain an image hierarchical score map;

[0015] Based on the image hierarchical score map, hand information is determined.

[0016] In some embodiments, the inputting the first frame image in the hand video into a backbone network to obtain an initial hierarchical feature includes:

[0017] The first frame image in the hand video is input into a backbone network to obtain an image hierarchical feature of a first preset convolutional layer and an image hierarchical feature of a second preset convolutional layer;

[0018] Based on the image hierarchical feature of the first preset convolutional layer and the image hierarchical feature of the second preset convolutional layer, the initial image hierarchical feature is obtained.

[0019] In some embodiments, the inputting the first frame image in the hand video into a backbone network to obtain an initial hierarchical feature includes:

[0020] For each frame image subsequent to the first frame image, a search region is determined based on a center position of a hand in a previous frame image;

[0021] Based on the search region, the transition hand image is obtained;

[0022] The transition hand image is input into the backbone network to obtain an image hierarchical feature of a third preset convolutional layer and an image hierarchical feature of a fourth preset convolutional layer;

[0023] Based on the image hierarchical feature of the third preset convolutional layer and the image hierarchical feature of the fourth preset convolutional layer, the transition image hierarchical feature is obtained.

[0024] In some embodiments, in a case where the hand information is a center position of a hand, the determining hand information based on the image hierarchical score map includes:

[0025] Based on a peak side lobe ratio function, a peak value of the image hierarchical score map is determined;

[0026] A position corresponding to a maximum value of the peak value is determined as the center position of the hand.

[0027] In some embodiments, the image layer score map includes a plurality of groups, and for each group of image layer score maps, is represented according to the following formula:

[0028]

[0029] wherein P t is the t-th group of image layer score maps, P t l is the t-th group of image shallow feature layer score maps, P t h is the t-th group of image deep feature layer score maps.

[0030] In some embodiments, the frame group image includes a plurality of continuous frame images, the adjacent frame group images include a first frame group and a second frame group, and the determining the hand state as the fixed state in response to a distance between center positions of the hand in the first frame image of the first frame group and the first frame image of the second frame group being less than a threshold value includes:

[0031] determining a target distance between the center positions of the hand in the first frame image of the first frame group and the first frame image of the second frame group based on coordinates of the first frame image of the first frame group and the first frame image of the second frame group;

[0032] determining the hand state as the fixed state in response to the target distance being less than the threshold value.

[0033] In some embodiments, the identifying a target gesture in the adjacent frame group images and outputting a target control instruction corresponding to the target gesture in response to the hand state being the fixed state includes:

[0034] identifying a gesture in the adjacent frame group images in response to the hand state being the fixed state, and determining the target gesture;

[0035] determining the target control instruction corresponding to the target gesture according to a correspondence relationship between vehicle control instructions and gestures;

[0036] outputting the target control instruction.

[0037] In some embodiments, the obtaining the hand video includes:

[0038] obtaining an in-vehicle video;

[0039] determining the in-vehicle video containing the hand feature as the hand video in response to the hand feature existing in the in-vehicle video.

[0040] In another aspect, the embodiments of the present application also provide a control device of a vehicle, which includes:

[0041] obtaining a hand video, wherein the hand video comprises a plurality of continuous frames, and each frame has coordinate information;

[0042] a first determining module configured to track each frame in the hand video by combining deep features and shallow features, and determine hand information of each frame, wherein the hand information comprises a center position of the hand and a hand image;

[0043] a second determining module configured to determine a hand state as a fixed state when a distance between the center positions of the hand in adjacent frame groups is less than a threshold value;

[0044] a recognizing module configured to recognize a target gesture in the adjacent frame groups when the hand state is the fixed state, and output a vehicle control instruction corresponding to the target gesture.

[0045] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0046] The control method of the vehicle provided by the embodiments of the present application determines the hand information of each frame in the hand video by tracking each frame in the hand video by combining deep features and shallow features, so as to improve the robustness of tracking by using deep features, and make up for the low tracking accuracy caused by the lack of detailed features of deep features by using shallow features, so that the hand information of each frame is determined accurately, and then when the distance between the center positions of the hand in adjacent frame groups is less than a threshold value, the hand state is determined as a fixed state, and then the target gesture in the adjacent frame groups is recognized, and a vehicle control instruction corresponding to the target gesture is output, so as to control the vehicle.

[0047] The method uses a tracking mode of deep features, which can determine the center position of the hand when the hand is deformed, and then combines the tracking mode of shallow features, i.e., combines the center position and the hand image, to realize accurate recognition of the target gesture, so that the method can effectively and accurately recognize the gesture when the hand is deformed, and improve the reliability of human-computer interaction of the vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0049] Figure 1 A flowchart of a control method of a vehicle provided by the embodiments of the present application;

[0050] Figure 2Another flowchart of a vehicle control method provided by an embodiment of the present application;

[0051] Figure 3 A flowchart of a method for obtaining a hand video in a vehicle control method provided by an embodiment of the present application;

[0052] Figure 4 A flowchart of a method for combining deep features and shallow features to track each frame of image in the hand video, and determining hand information of each frame of image in a vehicle control method provided by an embodiment of the present application;

[0053] Figure 5 A flowchart of a method for determining a hand state as a fixed state in response to a distance between center positions of a hand in adjacent frame group images being less than a threshold in a vehicle control method provided by an embodiment of the present application;

[0054] Figure 6 A flowchart of a method for recognizing a target gesture in adjacent frame group images and outputting a target control instruction corresponding to the target gesture in response to the hand state being the fixed state in a vehicle control method provided by an embodiment of the present application;

[0055] Figure 7 A structural schematic diagram of a vehicle control device provided by an embodiment of the present application.

[0056] The above-described drawings have shown the specific embodiments of the present application, which will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative work fall within the scope of protection of the present application.

[0058] Unless otherwise defined, all the technical terms used in the embodiments of the present application have the same meanings as commonly understood by those of ordinary skill in the art.

[0059] In order to make the technical solutions and advantages of the present application more clear, the embodiments of the present application will be described in further detail below in conjunction with the drawings.

[0060] Currently, the functional buttons of a vehicle are increasing, and it is inconvenient for a driver to operate the functional buttons when driving the vehicle. Therefore, the human-computer interaction function of the vehicle can bring better use experience to the user. For example, the driver can control the vehicle to perform different functions by making various gestures, thereby improving driving safety and reducing the probability of traffic accidents.

[0061] In the related art, in the control method of the vehicle, the position of the hand of the user is usually determined based on the color feature of the captured image. However, this determination method is difficult to recognize the gesture only by the color feature, especially when the hand is deformed, so that the gesture recognition function is affected, and the reliability of the vehicle human-computer interaction is reduced.

[0062] In order to solve the technical problems existing in the related art, the embodiment of the present application provides a control method and device of a vehicle, which can effectively and accurately recognize gestures when the hand is deformed, and improve the reliability of vehicle human-computer interaction.

[0063] Figure 1 A flowchart of a control method of a vehicle is provided for the embodiment of the present application. Referring to Figure 1 The method is applied to a cabin domain controller of a vehicle, and includes the following steps:

[0064] Step 101, acquiring a hand video, wherein the hand video includes a plurality of continuous frames of images, and each frame of image has coordinate information.

[0065] Step 102, combining the deep features and the shallow features of each frame of image in the hand video to track, and determining the hand information of each frame of image, wherein the hand information includes the center position of the hand and the hand image.

[0066] Step 103, in response to the distance between the center positions of the hand in the adjacent frame group images being less than a threshold value, determining that the hand state is a fixed state.

[0067] Step 104, in response to the hand state being a fixed state, recognizing a target gesture in the adjacent frame group images, and outputting a target control instruction corresponding to the target gesture.

[0068] The control method of the vehicle provided in the embodiments of the present application determines hand information of each frame of image in the hand video by tracking the deep features and the shallow features of each frame of image in the hand video, so as to improve the robustness of tracking by using the deep features, and to make up for the low tracking accuracy caused by the lack of detailed features of the deep features by using the shallow features, so that the hand information of each frame of image is determined accurately, and then when the distance between the center positions of the hand in adjacent frame group images is less than a threshold, the hand state is determined as a fixed state, the target gesture in the adjacent frame group images is determined at this time, and the vehicle control instruction corresponding to the target gesture is output, so as to realize the control of the vehicle.

[0069] Therefore, the method uses the tracking mode of the deep features, so that the center position of the hand can be determined when the hand is deformed, and the tracking mode of the shallow features is combined at this time, that is, the center position and the hand image are combined, so that the target gesture is accurately recognized, and therefore the method can effectively and accurately recognize the gesture when the hand is deformed, and the reliability of the human-computer interaction of the vehicle is improved.

[0070] In some embodiments, the determining the hand information of each frame of image in the hand video by tracking the deep features and the shallow features of each frame of image in the hand video comprises:

[0071] inputting the first frame of image in the hand video into the backbone network to obtain initial layered features;

[0072] for each subsequent frame of image except the first frame, obtaining a transition hand image based on the image, and inputting the transition hand image into the backbone network to obtain transition image layered features;

[0073] correlating the initial image layered features and the transition image layered features to obtain an image layered score map;

[0074] determining the hand information based on the image layered score map.

[0075] In some embodiments, the inputting the first frame of image in the hand video into the backbone network to obtain initial layered features comprises:

[0076] inputting the first frame of image in the hand video into the backbone network to obtain image layered features of a first preset convolution layer and image layered features of a second preset convolution layer;

[0077] obtaining the initial image layered features based on the image layered features of the first preset convolution layer and the image layered features of the second preset convolution layer.

[0078] In some embodiments, for each subsequent frame of image except the first frame, obtaining a transition hand image based on the image, and inputting the transition hand image into the backbone network to obtain transition image layered features comprises:

[0079] For each frame image subsequent to the first frame image, based on the center position of the hand in the previous frame image, a search region is determined;

[0080] Based on the search region, a transition hand image is obtained;

[0081] The transition hand image is input into the backbone network to obtain image hierarchical features of a third preset convolutional layer and image hierarchical features of a fourth preset convolutional layer;

[0082] Based on the image hierarchical features of the third preset convolutional layer and the image hierarchical features of the fourth preset convolutional layer, transition image hierarchical features are obtained.

[0083] In some embodiments, when the hand information is the center position of the hand, based on the image hierarchical score map, determining the hand information includes:

[0084] Based on the peak side lobe ratio function, a peak of the image hierarchical score map is determined;

[0085] A position corresponding to a maximum value of the peak is determined as the center position of the hand.

[0086] In some embodiments, the image hierarchical score map includes multiple groups, and for each group of image hierarchical score maps, is represented according to the following formula:

[0087]

[0088] wherein P t is the tthgroup of image hierarchical score maps, P t l is the tthgroup of image shallow feature hierarchical score maps, P t h is the tthgroup of image deep feature hierarchical score maps.

[0089] In some embodiments, the frame group images include multiple continuous frame images, adjacent frame group images include a first frame group and a second frame group, and in response to a distance between the center positions of the hand in the adjacent frame group images being less than a threshold value, determining that the hand state is a fixed state includes:

[0090] Based on coordinates of the first frame image of the first frame group and the first frame image of the second frame group, a target distance between the center positions of the hand in the first frame image of the first frame group and the first frame image of the second frame group is determined;

[0091] In response to the target distance being less than the threshold value, it is determined that the hand state is a fixed state.

[0092] In some embodiments, in response to the hand state being a fixed state, a target gesture in the adjacent frame group images is recognized, and a target control instruction corresponding to the target gesture is outputted, including:

[0093] In response to the hand state being the fixed state, a gesture in the adjacent frame group image is recognized to determine a target gesture;

[0094] According to a correspondence relationship between the vehicle control instruction and the gesture, the target gesture is determined to correspond to a target control instruction;

[0095] The target control instruction is output.

[0096] In some embodiments, the hand video is obtained by:

[0097] The in-vehicle video is obtained.

[0098] In response to the presence of the hand feature in the in-vehicle video, the in-vehicle video containing the hand feature is determined as the hand video.

[0099] Figure 2 Another flowchart of a vehicle control method provided by the embodiments of the present application is provided. Referring to Figure 2 The method is applied to a cockpit domain controller and includes the following steps:

[0100] In step 201, a hand video is obtained, wherein the hand video includes a plurality of continuous frames of images, and each frame of image has coordinate information.

[0101] By obtaining the hand video containing a plurality of continuous frames of images, the subsequent step of extracting image deep features is prepared, and the coordinate information in each frame of image can be prepared for the subsequent step of obtaining the hand position.

[0102] In some embodiments, referring to Figure 3 Step 201 includes the following sub-steps:

[0103] In step 2011, an in-vehicle video is obtained.

[0104] In step 2012, in response to the presence of the hand feature in the in-vehicle video, the in-vehicle video containing the hand feature is determined as the hand video.

[0105] That is, through this step, the video containing the hand feature in the in-vehicle video is screened out, and irrelevant video data is removed, thereby improving the efficiency.

[0106] In some embodiments, the in-vehicle video containing the hand feature is determined as the hand video by using the YOLOv5 algorithm.

[0107] In step 202, the deep features and the shallow features of each frame of image in the hand video are combined and tracked to determine the hand information of each frame of image, and the hand information includes the center position of the hand and the hand image.

[0108] It can be understood that if the robustness of tracking is improved by using image depth features, shallow features also need to be used to compensate for the low tracking accuracy caused by the lack of details of deep features, so by combining deep features and shallow features for tracking each frame of image in the hand video, the hand information of each frame of image can be accurately determined.

[0109] In some embodiments, referring to Figure 4 , step 202 includes the following sub-steps:

[0110] Step 2021, input the first frame of image in the hand video into the backbone network to obtain the initial layered feature.

[0111] That is, the first frame of image in the hand video is processed by the backbone network to obtain the initial layered feature.

[0112] In some embodiments, step 2021 includes the following sub-steps:

[0113] Step one, input the first frame of image in the hand video into the backbone network to obtain the image layered feature of the first preset convolutional layer and the image layered feature of the second preset convolutional layer;

[0114] The first frame of image is processed by the backbone network to extract the shallow feature of the image by the first preset convolutional layer and the deep feature of the image by the second preset convolutional layer.

[0115] In some embodiments, the first preset convolutional layer can be the second feature extraction layer in the backbone network, and the second preset convolutional layer can be the third feature extraction layer in the backbone network.

[0116] Step two, based on the image layered feature of the first preset convolutional layer and the image layered feature of the second preset convolutional layer, the initial image layered feature is obtained.

[0117] That is, after obtaining the shallow feature and the deep feature of the initial image, the layered feature of the initial image can be obtained.

[0118] In some embodiments, the initial image layered feature is T1, the image layered feature of the first preset convolutional layer is T1 l , and the image layered feature of the second preset convolutional layer is T1 h , wherein T1={T1 l ,T1 h}.

[0119] Step 2022, for each subsequent frame of image except the first frame, based on the image, the transition hand image is obtained, and the transition hand image is input into the backbone network to obtain the transition image layered feature.

[0120] It can be understood that the transition image hierarchical features obtained by processing each subsequent image by using the backbone network can reflect the changes of the position and shape of the hand, thereby providing data support for the subsequent hand tracking.

[0121] In some embodiments, step 2022 comprises the following sub-steps:

[0122] Step one, for each subsequent image except the first image, determining a search region based on the center position of the hand in the previous image.

[0123] Step two, obtaining a transition hand image based on the search region.

[0124] It can be understood that before the accurate hand tracking is implemented, the subsequent hand image needs to be obtained by using the method of determining the approximate range as the search region.

[0125] Step three, inputting the transition hand image into the backbone network to obtain the image hierarchical features of the third preset convolutional layer and the image hierarchical features of the fourth preset convolutional layer.

[0126] It can be understood that the transition hand image is processed by the backbone network to extract the shallow features of the transition hand image through the third preset convolutional layer and extract the deep features of the transition hand image through the fourth preset convolutional layer.

[0127] In some embodiments, the transition image hierarchical features are X t , the image hierarchical features of the third preset convolutional layer are X t l , and the image hierarchical features of the fourth preset convolutional layer are X t h , wherein X t ={X t l ,X t h}.

[0128] In some embodiments, the third preset convolutional layer can be the second C3 layer of the backbone network, and the fourth preset convolutional layer can be the third C3 layer of the backbone network.

[0129] Step four, obtaining the transition image hierarchical features based on the image hierarchical features of the third preset convolutional layer and the image hierarchical features of the fourth preset convolutional layer.

[0130] Step 2023, performing cross-correlation on the initial image hierarchical features and the transition image hierarchical features to obtain an image hierarchical score map.

[0131] It can be understood that the hand information is reflected in the form of scores by cross-correlating the initial image layered features and the transition image layered features, so as to determine the position of the hand.

[0132] In step 2024, the hand information is determined based on the image layered score map.

[0133] In some embodiments, when the hand information is the hand center position, step 2024 includes the following sub-steps:

[0134] Step one, determine the peak value of the image layered score map based on the peak side lobe ratio function.

[0135] It can be understood that the weights of the shallow features and the deep features are determined by the peak side lobe ratio function, so as to balance the robustness and accuracy of tracking, and improve the reliability of vehicle human-computer interaction.

[0136] Step two, determine the position corresponding to the maximum value of the peak value as the center position of the hand.

[0137] In some embodiments, the image layered score map has multiple groups, and for each group of image layered score maps, it is represented according to the following formula:

[0138] P t ={P t l ,P t h}

[0139] Wherein, P t is the tthgroup of image layered score maps, P t l is the shallow feature layered score map of the tthgroup of images, P t h is the deep feature layered score map of the tthgroup of images.

[0140] Further, the calculation formula of the three layered score maps is as follows:

[0141] P t l =α t ·corr(T1 l ,X t l );

[0142] P t l =(1-α t )·corr(T1 h ,X t h );

[0143] Pt = a t · corr(T1 l , X t ) + (1 - a l ) · corr(T1 t , X h ) ; t h ) ;

[0144] wherein corr() is a cross-correlation function, a t is a weight of the shallow feature and the deep feature.

[0145] In some embodiments, the formula for calculating the weight of the shallow feature and the deep feature is:

[0146]

[0147] wherein PSR() is a function for calculating a peak side lobe ratio, and the value of a1 is a preset weight.

[0148] In some embodiments, the backbone network can be a backbone network of YOLOv5.

[0149] Step 203, in response to the distance between the center positions of the hands in the adjacent frame group images being less than a threshold value, determining that the hand state is a fixed state.

[0150] That is, when the distance between the center positions of the hands in the adjacent frame group images is less than a threshold value, it is determined that the hand state is a fixed state.

[0151] In some embodiments, the frame group image includes a plurality of continuous frame images, the adjacent frame group images include a first frame group and a second frame group, see Figure 5 , step 203 includes the following sub-steps:

[0152] Step 2031, based on the coordinates of the first frame image of the first frame group and the first frame image of the second frame group, determining the target distance between the center positions of the hands of the first frame image of the first frame group and the first frame image of the second frame group.

[0153] Step 2032, in response to the target distance being less than a threshold value, determining that the hand state is a fixed state.

[0154] In some embodiments, for example, if the fixation of the hand in the adjacent 10 frame images is taken as the judgment standard, the number of images in the frame group is set to 9, and the formula for calculating the target distance between the center positions of the hands at this time is:

[0155]

[0156] wherein D tis a target distance between the center positions of the hands, x and y are used to represent coordinates of the center positions of the hands in the image, the coordinates of the center position of the hand in the first frame image of the first frame group are (x t , y t ), and the coordinates of the center position of the hand in the second frame image of the second frame group are (x t-9 , y t-9 ).

[0157] In step 204, in response to the hand state being the fixed state, a target gesture in the adjacent frame group image is recognized, and a target control instruction corresponding to the target gesture is output.

[0158] That is, after it is determined that the hand state is fixed, a target control instruction corresponding to a target gesture is output by recognizing the target gesture in the adjacent frame group image, so as to realize gesture recognition and a human-computer interaction process, and enable the driver to control the vehicle through the fixed gesture.

[0159] In some embodiments, referring to Figure 6 , step 204 includes the following sub-steps:

[0160] In step 2041, in response to the hand state being the fixed state, a gesture in the adjacent frame group image is recognized, and a target gesture is determined.

[0161] Here, the target gesture can include a sweeping hand, a patting hand, a pointing hand, a grabbing hand, a gathering hand, or a waving hand, and the like. The target gesture can be an action implemented by one hand or an action implemented by both hands.

[0162] In step 2042, according to a corresponding relationship between a vehicle control instruction and a gesture, a target control instruction corresponding to the target gesture is determined.

[0163] In the embodiments of the present application, there is a one-to-one corresponding relationship between the vehicle control instruction and the gesture, and according to the target gesture, the vehicle control instruction can be determined through the relationship, so as to facilitate the driver to control the vehicle through the fixed gesture.

[0164] It should be noted that the corresponding relationship between the vehicle control instruction and the gesture is stored in the cabin domain controller.

[0165] In step 2043, the target control instruction is output.

[0166] By outputting the target control instruction, the vehicle can perform a related action based on the target control instruction, so as to realize control of the vehicle.

[0167] The control method of the vehicle provided in the embodiments of the present application determines hand information of each frame of image by tracking the deep features and the shallow features of each frame of image in the hand video, so as to improve the robustness of tracking by using the deep features, and meanwhile, compensate for the low tracking accuracy caused by the lack of detailed features of the deep features by using the shallow features, so that the hand information of each frame of image is determined accurately, and then when the distance between the center positions of the hand in adjacent frame group images is less than a threshold, the hand state is determined as a fixed state, at this time, the target gesture in the adjacent frame group images is identified, and the vehicle control instruction corresponding to the target gesture is output, so as to realize the control of the vehicle. Therefore, the tracking mode using the deep features can determine the center position of the hand when the hand is deformed, at this time, the tracking mode combining the shallow features is used, that is, the center position and the hand image are combined, so as to realize the accurate identification of the target gesture, and therefore, the method can effectively and accurately identify the gesture when the hand is deformed, and the reliability of the human-computer interaction of the vehicle is improved.

[0168] Figure 7 A structural schematic diagram of a control device of a vehicle provided in the embodiments of the present application is shown in Figure 7 The device 700 comprises:

[0169] The acquisition module 701 is configured to acquire a hand video, wherein the hand video comprises continuous multiple frames of image, and each frame of image has coordinate information.

[0170] The first determination module 702 is configured to track the deep features and the shallow features of each frame of image in the hand video, and determine hand information of each frame of image, wherein the hand information comprises a center position of the hand and a hand image.

[0171] The second determination module 703 is configured to determine that the hand state is a fixed state in response to the distance between the center positions of the hand in adjacent frame group images being less than a threshold.

[0172] The identification module 704 is configured to identify a target gesture in adjacent frame group images and output a vehicle control instruction corresponding to the target gesture in response to the hand state being the fixed state.

[0173] In some embodiments, the first determination module comprises:

[0174] The first obtaining sub-module is configured to input a first frame of image in the hand video into a backbone network to obtain initial hierarchical features.

[0175] The second obtaining sub-module is configured to, for each subsequent frame of image except the first frame, obtain a transition hand image based on the image, and input the transition hand image into the backbone network to obtain transition image hierarchical features.

[0176] The third obtaining sub-module is configured to obtain an image layer score map by cross-correlating the initial image layer features and the transition image layer features.

[0177] The first determining sub-module is configured to determine the hand information based on the image layer score map.

[0178] In some embodiments, the first obtaining sub-module includes:

[0179] The fourth obtaining sub-module is configured to input the first frame image in the hand video into the backbone network to obtain image layer features of a first preset convolutional layer and image layer features of a second preset convolutional layer.

[0180] The fifth obtaining sub-module is configured to obtain initial image layer features based on the image layer features of the first preset convolutional layer and the image layer features of the second preset convolutional layer.

[0181] In some embodiments, the second obtaining sub-module includes:

[0182] The second determining sub-module is configured to determine a search region based on the center position of the hand in the previous frame image for each subsequent frame image except the first frame image.

[0183] The sixth obtaining sub-module is configured to obtain a transition hand image based on the search region.

[0184] The seventh obtaining sub-module is configured to input the transition hand image into the backbone network to obtain image layer features of a third preset convolutional layer and image layer features of a fourth preset convolutional layer.

[0185] The eighth obtaining sub-module is configured to obtain transition image layer features based on the image layer features of the third preset convolutional layer and the image layer features of the fourth preset convolutional layer.

[0186] In some embodiments, when the hand information is the center position of the hand, the first determining sub-module includes:

[0187] The third determining sub-module is configured to determine a peak value of the image layer score map based on a peak-to-sidelobe ratio function.

[0188] The fourth determining sub-module is configured to determine a position corresponding to a maximum value of the peak value as the center position of the hand.

[0189] In some embodiments, the image layer score map includes multiple groups, and for each group of image layer score map, is represented according to the following formula:

[0190] P t ={P t l ,P t h}

[0191] P t is the t-th group image layer score map, P t t l P t is the t-th group image shallow feature layer score map, P t h P t is the t-th group image deep feature layer score map.

[0192] In some embodiments, the frame group images include a plurality of continuous frame images, adjacent frame group images include a first frame group and a second frame group, and the second determining module includes:

[0193] A fifth determining submodule is configured to determine a target distance between the center positions of the hands of the first frame image of the first frame group and the first frame image of the second frame group based on the coordinates of the first frame image of the first frame group and the first frame image of the second frame group.

[0194] A sixth determining submodule is configured to determine that the hand state is a fixed state in response to the target distance being less than a threshold value.

[0195] In some embodiments, the recognition module includes:

[0196] A first recognition submodule is configured to recognize a hand gesture in the adjacent frame group images to determine a target hand gesture in response to the hand state being a fixed state.

[0197] A seventh determining submodule is configured to determine a target control instruction corresponding to the target hand gesture according to a corresponding relationship between vehicle control instructions and hand gestures.

[0198] An output submodule is configured to output the target control instruction.

[0199] In some embodiments, the acquisition module includes:

[0200] An acquisition submodule is configured to acquire in-vehicle video.

[0201] An eighth determining submodule is configured to determine in-vehicle video containing hand features as hand video in response to the presence of hand features in the in-vehicle video.

[0202] ​Therefore, the control device of the vehicle provided by the embodiment of the present application determines the hand information of each frame of image by tracking the deep features and the shallow features of each frame of image in the hand video, so as to improve the robustness of tracking by using the deep features, and to make up for the low tracking accuracy caused by the lack of detailed features of the deep features by using the shallow features, so that the hand information of each frame of image is determined accurately, and then when the distance between the center positions of the hand in the adjacent frame group images is less than the threshold value, the hand state is determined as a fixed state, at this time, the target gesture in the adjacent frame group images is determined, and the vehicle control instruction corresponding to the target gesture is output, so as to realize the control of the vehicle. The device can effectively and accurately identify the gesture when the hand is deformed, and improves the reliability of the human-computer interaction of the vehicle.

[0203] In the present application, the terms "first" and "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance. The term "a plurality of" refers to two or more, unless otherwise explicitly limited.

[0204] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the application following the general principles thereof and including such departures from the present disclosure as come within known use or custom in the art. The specification and examples are to be regarded as illustrative only.

[0205] It should be understood that the present application is not limited to the precise structures described and shown in the drawings, and that various modifications and changes can be made without departing from the scope of the present application. The scope of the present application is limited only by the claims appended hereto.

Claims

1. A control method of a vehicle, characterized by, The method comprises: acquiring a hand video, wherein the hand video comprises a plurality of continuous frames of images, each frame of image has coordinate information; performing combined tracking of deep features and shallow features on each frame of image in the hand video to determine hand information of each frame of image, the hand information comprising a center position of a hand and a hand image; in response to a distance between center positions of the hand in adjacent frame group images being less than a threshold value, determining that a hand state is a fixed state; in response to the hand state being the fixed state, identifying a target gesture in the adjacent frame group images and outputting a target control instruction corresponding to the target gesture; wherein the performing combined tracking of deep features and shallow features on each frame of image in the hand video to determine hand information of each frame of image comprises: inputting a first frame of image in the hand video into a backbone network to obtain initial layered features; for each subsequent frame of image other than the first frame, based on the image, obtaining a transition hand image and inputting the transition hand image into the backbone network to obtain transition image layered features; performing cross-correlation on the initial image layered features and the transition image layered features to obtain an image layered score map; based on the image layered score map, determining hand information; the inputting the first frame of image in the hand video into the backbone network to obtain initial layered features comprises: inputting the first frame of image in the hand video into the backbone network to obtain image layered features of a first preset convolution layer and image layered features of a second preset convolution layer; based on the image layered features of the first preset convolution layer and the image layered features of the second preset convolution layer, obtaining the initial image layered features; the for each subsequent frame of image other than the first frame, based on the image, obtaining a transition hand image and inputting the transition hand image into the backbone network to obtain transition image layered features comprises: for each subsequent frame of image other than the first frame of image, based on a center position of a hand of a previous frame of image, determining a search region; based on the search region, obtaining the transition hand image; inputting the transition hand image into the backbone network to obtain image layered features of a third preset convolution layer and image layered features of a fourth preset convolution layer; based on the image layered features of the third preset convolution layer and the image layered features of the fourth preset convolution layer, obtaining the transition image layered features.

2. The control method of a vehicle according to claim 1, characterized by in a case where the hand information is a center position of a hand, the based on the image layered score map, determining hand information comprises: based on a peak side lobe ratio function, determining a peak value of the image layered score map; determining a position corresponding to a maximum value of the peak value as the center position of the hand.

3. The control method of a vehicle according to claim 2, characterized by the image layered score map comprises a plurality of groups, for each group of image layered score map, represented according to the following formula: where P t is the image layer score map for the t-th group of images, P t l is the shallow feature layer score map for the t-th group of images, P t h is the deep feature layer score map for the t-th group of images.

4. The control method of a vehicle according to claim 1, characterized by the frame group images comprise a plurality of continuous frames of images, the adjacent frame group images comprise a first frame group and a second frame group, the in response to a distance between center positions of the hand in adjacent frame group images being less than a threshold value, determining that a hand state is a fixed state comprises: determine a target distance between the center positions of the hands in the first frame image of the first frame group and the first frame image of the second frame group based on coordinates of the first frame image of the first frame group and the first frame image of the second frame group; determine the hand state as the fixed state in response to the target distance being less than the threshold value.

5. The control method of a vehicle according to claim 1, characterized by The identifying a target gesture in the adjacent frame group image and outputting a target control instruction corresponding to the target gesture in response to the hand state being the fixed state includes: identifying a gesture in the adjacent frame group image in response to the hand state being the fixed state, and determining the target gesture; determining a target control instruction corresponding to the target gesture according to a correspondence relationship between vehicle control instructions and gestures; outputting the target control instruction.

6. The control method of a vehicle according to claim 1, characterized by The obtaining the hand video includes: obtaining an in-vehicle video; determining the in-vehicle video containing the hand feature as the hand video in response to the hand feature existing in the in-vehicle video.

7. A control device of a vehicle characterized by comprising: The apparatus includes: an obtaining module configured to obtain a hand video, wherein the hand video includes a plurality of continuous frames of images, and each frame of image has coordinate information; a first determining module configured to track a deep feature and a shallow feature of each frame of image in the hand video in combination, and determine hand information of each frame of image, the hand information including a center position of a hand and a hand image; a second determining module configured to determine a hand state as a fixed state in response to a distance between the center positions of the hands in adjacent frame group images being less than a threshold value; an identifying module configured to identify a target gesture in the adjacent frame group image and output a vehicle control instruction corresponding to the target gesture in response to the hand state being the fixed state; The first determining module includes: a first obtaining sub-module configured to input a first frame image in the hand video into a backbone network to obtain an initial layered feature; a second obtaining sub-module configured to, for each frame of image after the first frame, obtain a transition hand image based on the image, and input the transition hand image into the backbone network to obtain a transition image layered feature; a third obtaining sub-module configured to cross-correlate the initial image layered feature and the transition image layered feature to obtain an image layered score map; a first determining sub-module configured to determine hand information based on the image layered score map; The first obtaining sub-module includes: a fourth obtaining sub-module configured to input the first frame image in the hand video into the backbone network to obtain an image layered feature of a first preset convolution layer and an image layered feature of a second preset convolution layer; a fifth obtaining sub-module configured to obtain the initial image layered feature based on the image layered feature of the first preset convolution layer and the image layered feature of the second preset convolution layer; The second obtaining sub-module includes: a second determining sub-module configured to, for each frame of image after the first frame image, determine a search region based on a center position of a hand in a previous frame of image; a sixth obtaining sub-module configured to obtain the transition hand image based on the search region; A seventh obtaining sub-module is configured to input the transition hand image into the backbone network to obtain image hierarchical features of a third preset convolutional layer and image hierarchical features of a fourth preset convolutional layer; An eighth obtaining sub-module is configured to obtain the transition image hierarchical features based on the image hierarchical features of the third preset convolutional layer and the image hierarchical features of the fourth preset convolutional layer.

Images