Vehicle control method and device, electronic equipment, vehicle and storage medium

Abstract

The invention discloses a vehicle control method and device, electronic equipment, a vehicle and a storage medium, and the method comprises the steps that a vehicle cabin is shot, multiple sets of first data of user limbs are obtained, and each set of first data comprises a two-dimensional image and a depth image, shot at the same moment, of the user limbs; sign language recognition is conducted on the multiple sets of first data, and sign language recognition results are obtained; under the condition that the sign language recognition result comprises the target object, the target object of the vehicle is controlled according to the sign language recognition result, in the steps, the two-dimensional image and the depth image are recognized, the sign language recognition result is obtained, and therefore the vehicle is controlled, the method is suitable for users who communicate through sign languages, and the user experience is improved. And the control requirements of special users on the vehicle can be met.

Description

Technical Field

[0001] The present application relates to the field of vehicle control, and in particular to a vehicle control method, device, electronic equipment, vehicle, and storage medium. Background Art

[0002] With the rapid development of smart cockpit technology, more and more smart in-vehicle devices have integrated functions such as touch interaction, voice recognition, and gesture recognition.

[0003] Currently, gesture recognition usually involves users customizing some simple gestures, such as raising a thumb to turn on music. During use, the user's gesture is compared with a pre-stored personalized gesture database for similarity to determine that the user's intention is to turn on music.

[0004] For special groups of people, such as users who interact using sign language, simple custom gestures cannot meet vehicle control needs. Summary of the Invention

[0005] The embodiments of the present application provide a vehicle control method, device, electronic device, vehicle, and storage medium, which recognize two-dimensional images and depth images to obtain sign language recognition results, thereby controlling the vehicle. The method is suitable for users who communicate using sign language and can meet the vehicle control needs of special users.

[0006] In a first aspect, an embodiment of the present application provides a vehicle control method, the method comprising:

[0007] photographing a vehicle cabin to obtain multiple sets of first data of a user's limbs, each set of the first data including a two-dimensional image and a depth image of the user's limbs photographed at the same time;

[0008] performing sign language recognition on the plurality of groups of the first data to obtain sign language recognition results;

[0009] In a case where the sign language recognition result includes a target object, the target object of the vehicle is controlled according to the sign language recognition result.

[0010] In one embodiment of the present application, performing sign language recognition on the plurality of sets of the first data to obtain a sign language recognition result includes:

[0011] Each set of the first data is processed as follows: joint recognition is performed on the user's limb in the two-dimensional image to obtain two-dimensional coordinates of multiple joint points of the user's limb; depth values ​​of the multiple joint points are obtained from the depth image; and three-dimensional coordinates of each joint point are obtained based on the two-dimensional coordinates of the multiple joint points and the depth values ​​of the multiple joint points;

[0012] The three-dimensional coordinates of the joint points corresponding to multiple sets of the first data and the shooting time corresponding to the first data are input into a pre-trained sign language recognition model to obtain the sign language recognition results output by the sign language recognition model. The sign language recognition model is trained using multiple training samples, each of the training samples includes multiple sets of second data, and sample sign language recognition results determined based on the multiple sets of second data, each of the second data includes the three-dimensional coordinates of multiple sample joint points and the shooting time corresponding to the second data.

[0013] In one embodiment of the present application, the sample sign language recognition result is determined according to the following method:

[0014] For each of the training samples, inputting the three-dimensional coordinates of the sample joint points corresponding to multiple sets of the second data and the shooting time corresponding to the second data into a pre-trained teacher model to obtain a sign language recognition result output by the teacher model;

[0015] Using the sign language recognition result output by the teacher model as the sample sign language recognition result in the training sample;

[0016] The sign language recognition model is obtained according to the following method:

[0017] During the training process, for each training sample, the three-dimensional coordinates of the sample joint points corresponding to multiple sets of the second data and the shooting time corresponding to the second data are input into the student model to obtain the sign language recognition result output by the student model;

[0018] Calculating based on the sign language recognition result output by the student model and the sample sign language recognition results in the training samples to obtain a loss value of the student model;

[0019] When the loss value is less than the preset loss value, the student model is used as the sign language recognition model.

[0020] In one embodiment of the present application, controlling the target object of the vehicle according to the sign language recognition result includes:

[0021] In a case where the sign language recognition result includes a first identifier and a control instruction, controlling the target object identified by the first identifier in the vehicle according to the control instruction;

[0022] or,

[0023] In a case where the sign language recognition result includes a control instruction, determining a target object corresponding to the control instruction, and controlling the target object in the vehicle according to the control instruction;

[0024] or,

[0025] In a case where the sign language recognition result includes a first identifier, a target object identified by the first identifier in the vehicle is controlled according to a preset instruction.

[0026] In one embodiment of the present application, when the sign language recognition result includes a control instruction, determining a target object corresponding to the control instruction includes:

[0027] When the sign language recognition result includes a control instruction, performing contextual semantic analysis on historical sign language recognition results within a first preset time to obtain a target object identifier, and determining the target object according to the target object identifier;

[0028] or,

[0029] When the sign language recognition result includes a control instruction, outputting prompt information, wherein the prompt information is used to request the user to determine the target object;

[0030] photographing the vehicle cabin within a second preset time to obtain multiple sets of third data;

[0031] The multiple groups of the third data are used as the multiple groups of the first data, and the process jumps to the step of performing sign language recognition on the multiple groups of the first data to obtain sign language recognition results.

[0032] In one embodiment of the present application, when the sign language recognition result includes a target object, after controlling the target object of the vehicle according to the sign language recognition result, the method further includes:

[0033] In a case where the sign language recognition result includes a target object, determining a plurality of sign language images corresponding to a control result for controlling the target object from preset sign language images;

[0034] Determining the time sequence of each sign language image according to a preset sign language word order rule to obtain a sign language video;

[0035] Output the sign language video.

[0036] In a second aspect, an embodiment of the present application provides a vehicle control device, which is applied to a vehicle and includes:

[0037] an acquisition module, configured to photograph the vehicle cabin to obtain multiple sets of first data of the user's limbs, each set of the first data comprising a two-dimensional image and a depth image of the user's limbs photographed at the same moment;

[0038] a recognition module, configured to perform sign language recognition on the plurality of sets of the first data to obtain sign language recognition results;

[0039] A control module is used to control the target object of the vehicle according to the sign language recognition result when the sign language recognition result includes the target object.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions;

[0041] When the processor executes the computer program instructions, the vehicle control method as described in the first aspect is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a vehicle comprising the electronic device as described in the third aspect.

[0043] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the vehicle control method as described in the first aspect is implemented.

[0044] In a sixth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the vehicle control method as described in the first aspect.

[0045] The vehicle control method, device, electronic device, vehicle and storage medium of the embodiments of the present application obtain multiple groups of first data of the user's limbs by photographing the vehicle cabin, each group of first data includes a two-dimensional image and a depth image of the user's limbs photographed at the same time; sign language recognition is performed on the multiple groups of first data to obtain sign language recognition results; when the sign language recognition results include a target object, the target object of the vehicle is controlled according to the sign language recognition results. In the above steps, the two-dimensional image and the depth image are recognized to obtain the sign language recognition results, thereby controlling the vehicle. The method is suitable for users who communicate using sign language and can meet the vehicle control needs of special users. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 This is a flow chart of a vehicle control method provided by an embodiment of the present application;

[0048] Figure 2 is another flow chart of the vehicle control method provided in an embodiment of the present application;

[0049] Figure 3 is a structural diagram of a vehicle control device provided in an embodiment of the present application;

[0050] Figure 4 is another structural schematic diagram of the vehicle control device provided in an embodiment of the present application;

[0051] Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0053] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0054] In each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the disclosed embodiment will be obtained.

[0055] The vehicle can be a private car, such as a sedan, SUV, MPV, or pickup truck. It can also be a commercial vehicle, such as a van, bus, small truck, or large trailer. It can be a gasoline vehicle or a new energy vehicle. When the vehicle is a new energy vehicle, it can be a hybrid vehicle or a pure electric vehicle.

[0056] In order to solve the problems of the prior art, the embodiments of the present application provide a vehicle control method, device, electronic device, vehicle, and storage medium. The vehicle control method provided by the embodiments of the present application is first introduced below.

[0057] Figure 1 FIG. 1 shows a flow chart of a vehicle control method provided by an embodiment of the present application. Figure 1 As shown, the vehicle control method provided in the embodiment of the present application is applied to an electronic device, such as a server, and includes the following steps 101 to 103, wherein:

[0058] Step 101 : photographing a vehicle cabin to obtain multiple sets of first data of a user's limbs, each set of the first data including a two-dimensional image and a depth image of the user's limbs photographed at the same time.

[0059] In this embodiment, the vehicle cabin is photographed to obtain multiple sets of first data of the user's limbs, each set of first data includes a two-dimensional image and a depth image of the user's limbs taken at the same time, wherein the two-dimensional image is obtained by an RGB camera, and the depth image is obtained by a time of flight (ToF) camera.

[0060] The vehicle cabin is photographed to obtain multiple two-dimensional images and multiple depth images of the user's limbs, and the multiple two-dimensional images and multiple depth images are subjected to spatiotemporal synchronization processing to obtain two-dimensional images and depth images corresponding to the same timestamp, that is, two-dimensional images and depth images of the user's limbs taken at the same time.

[0061] Step 102: Perform sign language recognition on the multiple groups of the first data to obtain sign language recognition results.

[0062] In this embodiment, sign language recognition is performed on multiple sets of first data, and sign language movements are converted into natural language to obtain sign language recognition results.

[0063] Optionally, the two-dimensional image may be preprocessed before recognition, the two-dimensional image in the first data may be preprocessed, and the captured two-dimensional image may be subjected to background removal and illumination compensation processing to improve the accuracy of sign language recognition.

[0064] Step 103 : When the sign language recognition result includes a target object, control the target object of the vehicle according to the sign language recognition result.

[0065] In this embodiment, when the sign language recognition result includes a target object, where the object is a component in a vehicle, such as an air conditioner, a window, or other controllable object, the target object of the vehicle is controlled according to the sign language recognition result.

[0066] In this embodiment, the vehicle cabin is photographed to obtain multiple sets of first data of the user's limbs, each set of first data includes a two-dimensional image and a depth image of the user taken at the same time, and sign language recognition is performed on the multiple sets of first data to obtain sign language recognition results. When the sign language recognition results include a target object, the target object of the vehicle is controlled according to the sign language recognition results. In the above process, the two-dimensional image and the depth image are recognized to obtain the sign language recognition results, thereby controlling the vehicle. This process is suitable for users who communicate using sign language and can meet the vehicle control needs of special users.

[0067] In one embodiment of the present application, performing sign language recognition on the plurality of sets of the first data to obtain sign language recognition results includes:

[0068] Each set of the first data is processed as follows: joint recognition is performed on the user's limb in the two-dimensional image to obtain two-dimensional coordinates of multiple joint points of the user's limb; depth values ​​of the multiple joint points are obtained from the depth image; and three-dimensional coordinates of each joint point are obtained based on the two-dimensional coordinates of the multiple joint points and the depth values ​​of the multiple joint points;

[0069] The three-dimensional coordinates of the joint points corresponding to multiple sets of the first data and the shooting time corresponding to the first data are input into a pre-trained sign language recognition model to obtain the sign language recognition results output by the sign language recognition model. The sign language recognition model is trained using multiple training samples, each of the training samples includes multiple sets of second data, and sample sign language recognition results determined based on the multiple sets of second data, each of the second data includes the three-dimensional coordinates of multiple sample joint points and the shooting time corresponding to the second data.

[0070] In this embodiment, each group of first data is processed as follows: the user joints in the two-dimensional image are identified, multiple joints of the user's limbs are identified, and the two-dimensional coordinates of the multiple joints of the user's limbs are obtained; the depth values ​​of the multiple joint points are obtained from the depth image; and the three-dimensional coordinates of each joint point are obtained based on the two-dimensional coordinates of the multiple joint points and the depth values ​​of the multiple joint points. By analyzing the changes in the depth values, the degree of bending of the fingers, the direction of the palm, and the extension direction of the arm can be accurately judged, which are crucial for accurate recognition of sign language, because different gestures may represent different sign language meanings.

[0071] A sign language recognition model is pre-trained, and multiple training samples are used to train the sign language recognition model, wherein each training sample includes multiple groups of second data and sample sign language recognition results, and the sample sign language recognition results are determined based on the multiple groups of second data, and each group of second data includes the three-dimensional coordinates of multiple sample joint points and the shooting time corresponding to the second data.

[0072] Specifically, the second data is obtained according to the following method: the user is photographed to obtain multiple sets of sample data of the user's limbs, each set of sample data includes a sample two-dimensional image and a sample depth image of the user's limbs photographed at the same time, and each set of sample data is processed as follows: joint recognition is performed on the user's limbs in the sample two-dimensional image to identify multiple joints of the user's limbs, the two-dimensional coordinates of multiple sample joint points of the user's limbs are obtained, the depths of the multiple sample joint points are obtained from the sample depth image, and the three-dimensional coordinates of each sample joint point are obtained based on the two-dimensional coordinates of the multiple sample joint points and the depth values ​​of the multiple sample joint points. The three-dimensional coordinates of the multiple sample joint points corresponding to each set of sample data and the shooting time of the sample data are taken as a set of second data, and multiple second data are obtained according to the above method.

[0073] The three-dimensional data of the joint points corresponding to the multiple sets of first data and the shooting time corresponding to the first data are output to the pre-trained sign language recognition model to obtain the sign language recognition results output by the sign language recognition model. Sign language recognition is to convert sign language movements into understandable natural language, such as text.

[0074] By fusing two-dimensional images and depth images and using a pre-trained sign language recognition model, the meaning of sign language can be identified from the image, and sign language movements can be recognized more accurately.

[0075] In one embodiment of the present application, the sample sign language recognition result is determined according to the following method:

[0076] For each of the training samples, inputting the three-dimensional coordinates of the sample joint points corresponding to multiple sets of the second data and the shooting time corresponding to the second data into a pre-trained teacher model to obtain a sign language recognition result output by the teacher model;

[0077] Using the sign language recognition result output by the teacher model as the sample sign language recognition result in the training sample;

[0078] The sign language recognition model is obtained according to the following method:

[0079] During the training process, for each training sample, the three-dimensional coordinates of the sample joint points corresponding to multiple sets of the second data and the shooting time corresponding to the second data are input into the student model to obtain the sign language recognition result output by the student model;

[0080] Calculating based on the sign language recognition result output by the student model and the sample sign language recognition results in the training samples to obtain a loss value of the student model;

[0081] When the loss value is less than the preset loss value, the student model is used as the sign language recognition model.

[0082] In this embodiment, for each training sample in a plurality of training samples, the three-dimensional coordinates of the sample joint points corresponding to the multiple sets of second data and the shooting time corresponding to the second data are input into the pre-trained teacher model to obtain the sign language recognition result output by the teacher model, and the sign language recognition result output by the teacher model is used as the sample sign language recognition result in the training sample.

[0083] The teacher model can be a ResNet model, which is a three-dimensional convolutional neural network architecture of a residual network (ResNet) for processing time series image data.

[0084] The sign language recognition model is obtained as follows:

[0085] During the training process, for each training sample, the three-dimensional coordinates of the sample joint points corresponding to the multiple sets of second data and the shooting time corresponding to the second data are input into the student model to obtain the sign language recognition result output by the student model. The loss value of the student model is calculated based on the sign language recognition result output by the student model and the sample sign language recognition result in the training sample. Specifically, the loss value is calculated according to the loss function model as shown below:

[0086]

[0087] Among them, L distill is the loss value, a is the hyperparameter, KL is the divergence, KL(p s ||p t ) represents the loss term based on the output probability distribution, which is used to measure the difference between the two probability distributions. The smaller the value, the closer Ps is to Pt. ps represents the output of the student model, and pt represents the output of the teacher model. represents the loss term based on feature representation, fs represents the student model features, and ft represents the teacher model features.

[0088] Among them, the output of the teacher model is as follows:

[0089]

[0090] Among them, pt represents the output of the teacher model. The formula means that the original logarithmic probability (logits) of the output layer is generated into a probability distribution pt through the softmax function with temperature. T is the temperature parameter, and T=2 is used for knowledge distillation.

[0091] Optionally, the student model can be a convolutional neural network (CNN) or a long short-term memory (LSTM) network.

[0092] Specifically, when the loss function is less than a preset loss function, the student model is used as the sign language recognition model.

[0093] In the field of machine learning and deep learning, the collaborative training model of teacher models and student models is usually called knowledge distillation. It is a way to achieve model optimization by "leading the small with the big". The teacher model is usually a "large model" with high complexity and strong performance, while the student model is a "small model" that is lightweight and easy to deploy. The prior knowledge accumulated by the teacher model through large-scale data pre-training can be transferred to the student model through distillation. The teacher model-assisted training method is conducive to improving training efficiency and training effects. Although the teacher model has high accuracy, it has a large number of parameters and is difficult to deploy directly on the vehicle side. The trained student model is used as the sign language recognition model. The student model is easier to deploy on the vehicle side due to its lightweight and low power consumption.

[0094] In one embodiment of the present application, controlling the target object of the vehicle according to the sign language recognition result includes:

[0095] In a case where the sign language recognition result includes a first identifier and a control instruction, controlling the target object identified by the first identifier in the vehicle according to the control instruction;

[0096] or,

[0097] In a case where the sign language recognition result includes a control instruction, determining a target object corresponding to the control instruction, and controlling the target object in the vehicle according to the control instruction;

[0098] or,

[0099] In a case where the sign language recognition result includes a first identifier, a target object identified by the first identifier in the vehicle is controlled according to a preset instruction.

[0100] In this embodiment, when the sign language recognition result includes a first identifier and a control instruction, the target object identified by the first identifier in the vehicle is controlled according to the control instruction. For example, the target object identified by the first identifier is an air conditioner, and the control instruction is to adjust the temperature to 26 degrees. The air conditioner entity in the vehicle is controlled to adjust the temperature of the air conditioner to 26 degrees.

[0101] Alternatively, when the sign language recognition result includes a control instruction but does not include a control object, it is necessary to determine the target object corresponding to the control instruction and control the target object in the vehicle according to the control instruction.

[0102] Alternatively, when the sign language recognition result includes the first identifier but does not include a control instruction, it means that the user has not triggered the control instruction, and the target object identified by the first identifier in the vehicle is controlled according to the preset instruction. The preset instruction is a default instruction that is automatically executed when no control instruction from the user is received. If the target object has a default instruction, the default instruction is used as the preset instruction. If the target object has multiple default instructions, one default instruction is selected as the preset instruction. For example, the target object identified by the first identifier is an air conditioner. The air conditioner has multiple default instructions according to multiple seasons. The default instruction adapted to the current season is selected as the preset instruction according to the current season. The default adapted to summer is the cooling mode and the temperature is 26 degrees.

[0103] Controlling the vehicle through sign language recognition results can meet the vehicle control needs of special users.

[0104] In one embodiment of the present application, when the sign language recognition result includes a control instruction, determining the target object corresponding to the control instruction includes:

[0105] When the sign language recognition result includes a control instruction, performing contextual semantic analysis on historical sign language recognition results within a first preset time to obtain a target object identifier, and determining the target object according to the target object identifier;

[0106] or,

[0107] When the sign language recognition result includes a control instruction, outputting prompt information, wherein the prompt information is used to request the user to determine the target object;

[0108] photographing the vehicle cabin within a second preset time to obtain multiple sets of third data;

[0109] The multiple groups of the third data are used as the multiple groups of the first data, and the process jumps to the step of performing sign language recognition on the multiple groups of the first data to obtain sign language recognition results.

[0110] In this embodiment, when the sign language recognition result includes a control instruction, a contextual semantic analysis is performed on the sign language recognition result within a first preset time. The first preset time can be set according to actual needs. For example, a contextual semantic analysis is performed on the sign language recognition result within 3 minutes. A natural language processing (NLP) model is used to perform the contextual semantic analysis to obtain a target object identifier, and the object corresponding to the target object identifier is determined as the target object. The target object in the vehicle is controlled according to the control instruction.

[0111] Alternatively, when the sign language recognition result only includes the target object, a prompt message is output. The prompt message may be in the form of text or a sign language video. For example, the prompt message "Please confirm the object to be controlled this time" may be output in the form of text or a sign language video may be output in the form of a sign language video. The prompt message may be output through the vehicle display screen.

[0112] After outputting the prompt information, the vehicle cabin is photographed within a second preset time to obtain multiple sets of third data, each set of third data including a two-dimensional image and a depth image of the user's limbs taken at the same time. The multiple sets of third data are used as multiple sets of first data, and sign language recognition is performed on the multiple sets of first data to obtain sign language recognition results until the target object is determined, and the target object in the vehicle is controlled according to the control instruction.

[0113] The controlled object is determined by context semantic analysis or by outputting prompt information, with multiple implementation methods provided.

[0114] In one embodiment of the present application, when the sign language recognition result includes a target object, after controlling the target object of the vehicle according to the sign language recognition result, the method further includes:

[0115] In a case where the sign language recognition result includes a target object, determining a plurality of sign language images corresponding to a control result for controlling the target object from preset sign language images;

[0116] Determining the time sequence of each sign language image according to a preset sign language word order rule to obtain a sign language video;

[0117] Output the sign language video.

[0118] In this embodiment, when the sign language recognition result includes a target object, multiple sign language images are pre-set, and multiple sign language images corresponding to the control result of controlling the target object are determined from the preset sign language images. For example, if the control result is that the air conditioner is turned on, the sign language image corresponding to "air conditioner" is determined, the sign language image corresponding to "already" is determined, and the sign language image corresponding to "turn on" is determined. According to the preset sign language word order rules, the timing of each sign language image is determined. The preset sign language word order rules include multiple sign language word order rules, which are selected based on different regions. For example, the preset sign language word order rules for region A are usually expressed in the order of "subject (air conditioner) → action (turn on) → tense (already)". At this time, the timing of multiple sign language images generates a sign language video in the above order, outputs the sign language video, and outputs the sign language video through the vehicle display screen.

[0119] By outputting sign language videos, the needs of special users can be met. Compared with existing static images or simplified animation feedback, sign language expression can be simulated more realistically.

[0120] The following is an example of the vehicle control method provided in the embodiments of the present application.

[0121] Step 1: Collect sign language images.

[0122] See also Figure 2 , collect sign language images, and collect them through RGB cameras and TOF cameras. The sign language images include multiple groups of sign language images, each group of sign language images includes a two-dimensional image and a depth image of the user's limbs taken at the same time (that is, the vehicle cabin is photographed above to obtain multiple groups of first data of the user's limbs, and each group of the first data includes a two-dimensional image and a depth image of the user's limbs taken at the same time). Optionally, the sampling rate of the collected sign language images can be determined according to the detected hand movement speed of the user, such as dynamically adjusting the sampling rate according to the user's hand movement speed. For example, if the hand speed is detected to be less than the preset speed, it means that it is static sign language, and the sampling rate is 15fps. If the hand speed is detected to be greater than or equal to the preset speed, it means that it is fast sign language, and the sampling rate is 60fps. By real-time analysis of the hand movement speed, the video frame acquisition frequency is dynamically adjusted. Or a TOF sensor is used to collect depth data, and the depth data includes a depth value.

[0123] Step 2: Sign language image preprocessing.

[0124] In this embodiment, the sign language images are preprocessed, including background removal and illumination compensation. Preprocessing is also performed on the two-dimensional images within the multiple sets of sign language images to improve the accuracy of sign language recognition. (This refers to the aforementioned preprocessing of the two-dimensional images within the first data, and the background removal and illumination compensation performed on the captured two-dimensional images.)

[0125] Step 3: Sign language algorithm recognition.

[0126] In this embodiment, sign language recognition is performed on multiple groups of sign language images using a sign language algorithm to obtain sign language recognition results (that is, sign language recognition is performed on multiple groups of first data to obtain sign language recognition results as described above).

[0127] See also Figure 3 The vehicle control device includes a sign language recognition module 301, an intelligent cockpit control module 302 and a sign language synthesis module 303. The sign language recognition module 301 is used to recognize sign language and obtain a sign language recognition result.

[0128] Specifically, the sign language algorithm can be a pre-trained sign language recognition model, which processes each set of sign language images as follows: performs joint recognition on the user's limbs in the two-dimensional image to obtain the two-dimensional coordinates of multiple joint points of the user's limbs; obtains the depth values ​​of the multiple joint points from the depth image; obtains the three-dimensional coordinates of each joint point based on the two-dimensional coordinates of the multiple joint points and the depth values ​​of the multiple joint points; inputs the three-dimensional coordinates of the joint points corresponding to the multiple sets of sign language images and the shooting time corresponding to the first data into the pre-trained sign language recognition model to obtain the sign language recognition results output by the sign language recognition model.

[0129] Step 4: Output control instructions.

[0130] In this embodiment, when the sign language recognition result includes the target object, a control instruction is output (that is, when the sign language recognition result includes the target object, the target object of the vehicle is controlled according to the sign language recognition result). Figure 3 The intelligent cockpit control module 302 of the vehicle control device is used to output control instructions.

[0131] Step 5: Generate sign language actions.

[0132] In this embodiment, a sign language action is generated based on the control result. The sign language action is based on a sign language video (i.e., when the sign language recognition result includes a target object, multiple sign language images corresponding to the control result of controlling the target object are determined from the preset sign language images, and the time sequence of each sign language image is determined according to the preset sign language word order rules to obtain the sign language video). Figure 3 The sign language synthesis module 303 of the vehicle control device is used to generate sign language actions.

[0133] Step 6: Display feedback information, which includes the generated sign language movements.

[0134] In this embodiment, feedback information including the generated sign language movements is displayed (i.e., the output sign language video mentioned above), and the generated sign language movements are shown to the user through the vehicle-mounted display screen or projection system, and the operation results are fed back to the user.

[0135] Cameras and sensors integrated into the vehicle capture sign language movements and utilize deep learning algorithms to accurately classify and semantically analyze them. This approach addresses the issues of low recognition accuracy and poor real-time performance. While recognizing sign language, a semantic analysis engine converts gestures into operational commands, enabling interactive tasks to be completed in conjunction with the intelligent cockpit control system. Furthermore, the system uses synthesis technology to convert text, voice, and other information into feedback that is understandable to specialized groups, such as the hearing-impaired. Using motion capture and animation generation technology, dynamic sign language movements are generated in real time for user feedback, enhancing the naturalness and fluency of sign language. Seamlessly integrated with the intelligent cockpit system, the system interacts with users through the vehicle's onboard system, responding to in-cabin user sign language commands and providing feedback.

[0136] Figure 4 FIG. 1 shows a structural diagram of a vehicle control device provided in an embodiment of the present application. Figure 4 As shown, a vehicle control device 400 is applied to a vehicle and includes:

[0137] An acquisition module 401 is configured to photograph the vehicle cabin to obtain multiple sets of first data of the user's limbs, each set of the first data including a two-dimensional image and a depth image of the user's limbs photographed at the same time;

[0138] A recognition module 402 is configured to perform sign language recognition on the plurality of sets of the first data to obtain a sign language recognition result;

[0139] The control module 403 is configured to control the target object of the vehicle according to the sign language recognition result when the sign language recognition result includes the target object.

[0140] In one embodiment of the present application, the identification module 402 includes a processing submodule and an identification submodule;

[0141] a processing submodule, configured to process each set of the first data as follows: perform joint recognition on the user's limb in the two-dimensional image to obtain two-dimensional coordinates of multiple joint points of the user's limb; obtain depth values ​​of the multiple joint points from the depth image; and obtain three-dimensional coordinates of each joint point based on the two-dimensional coordinates of the multiple joint points and the multiple depth values ​​of the joint points;

[0142] The recognition submodule is used to input the three-dimensional coordinates of the joint points corresponding to multiple groups of the first data and the shooting time corresponding to the first data into a pre-trained sign language recognition model to obtain the sign language recognition results output by the sign language recognition model, wherein the sign language recognition model is trained using multiple training samples, each of the training samples includes multiple groups of second data, and sample sign language recognition results determined based on the multiple groups of second data, each of the second data includes the three-dimensional coordinates of multiple sample joint points and the shooting time corresponding to the second data.

[0143] In one embodiment of the present application, the control module 403 includes a first determination submodule and a control submodule;

[0144] a control submodule, configured to control a target object identified by the first identifier in the vehicle according to the control instruction when the sign language recognition result includes a first identifier and a control instruction;

[0145] A first determination submodule, which determines a target object corresponding to a control instruction when the sign language recognition result includes a control instruction;

[0146] a control submodule, further configured to control the target object in the vehicle according to the control instruction;

[0147] The control submodule is further configured to control the target object identified by the first identifier in the vehicle according to a preset instruction when the sign language recognition result includes the first identifier.

[0148] In one embodiment of the present application, the first determination submodule includes a semantic analysis subunit, a first determination subunit, and an output subunit;

[0149] a semantic analysis subunit configured to, when the sign language recognition result includes a control instruction, perform contextual semantic analysis on historical sign language recognition results within a first preset time period to obtain a target object identifier; and a first determination subunit configured to determine the target object based on the target object identifier;

[0150] An output subunit, configured to output prompt information when the sign language recognition result includes a control instruction, wherein the prompt information is used to request the user to determine the target object;

[0151] The acquisition module 401 is further configured to photograph the vehicle cabin within a second preset time to obtain multiple sets of third data;

[0152] The recognition module 402 is further configured to treat the multiple sets of the third data as the multiple sets of the first data, jump to the step of performing sign language recognition on the multiple sets of the first data, and obtain sign language recognition results.

[0153] In one embodiment of the present application, the vehicle control device further includes a sign language processing module and a video output module;

[0154] a sign language processing module configured to, when the sign language recognition result includes a target object, determine, from the preset sign language images, a plurality of sign language images corresponding to a control result for controlling the target object; determine a time sequence of each of the sign language images according to a preset sign language word sequence rule, and obtain a sign language video;

[0155] A video output module is used to output the sign language video.

[0156] The vehicle control device provided in the embodiment of the present application can implement the various processes implemented in the aforementioned vehicle control method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0157] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0158] The electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0159] Specifically, the processor 501 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0160] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 502 may include removable or non-removable (or fixed) media. Where appropriate, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory.

[0161] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present disclosure.

[0162] The processor 501 reads and executes computer program instructions stored in the memory 502 to implement any one of the above methods in the above embodiments.

[0163] In one example, the electronic device may further include a communication interface 503 and a bus 510. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 and communicate with each other.

[0164] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0165] Bus 510 comprises hardware, software or both, and the parts of method as above or electronic equipment are coupled to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus 510 can comprise one or more buses.Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0166] In addition, an embodiment of the present application provides a vehicle, which includes the above-mentioned electronic device.

[0167] In addition, the embodiments of the present application may be implemented by providing a computer storage medium having computer program instructions stored thereon; when the computer program instructions are executed by a processor, any one of the vehicle control methods in the above embodiments is implemented.

[0168] In addition, the embodiments of the present application may be implemented by providing a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device implements any one of the vehicle control methods in the above embodiments.

[0169] It should be understood that the present application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described as examples. However, the method process of the present application is not limited to the specific steps described. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0170] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0171] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0172] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0173] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A vehicle control method, characterized in that: The method is applied to a vehicle, and comprises: photographing a vehicle cabin to obtain multiple sets of first data of a user's limbs, each set of the first data including a two-dimensional image and a depth image of the user's limbs photographed at the same time; performing sign language recognition on the plurality of groups of the first data to obtain sign language recognition results; In a case where the sign language recognition result includes a target object, the target object of the vehicle is controlled according to the sign language recognition result.

2. The vehicle control method according to claim 1, characterized in that: The performing sign language recognition on the plurality of groups of the first data to obtain a sign language recognition result includes: Each set of the first data is processed as follows: joint recognition is performed on the user's limb in the two-dimensional image to obtain two-dimensional coordinates of multiple joint points of the user's limb; depth values ​​of the multiple joint points are obtained from the depth image; and three-dimensional coordinates of each joint point are obtained based on the two-dimensional coordinates of the multiple joint points and the depth values ​​of the multiple joint points; The three-dimensional coordinates of the joint points corresponding to multiple sets of the first data and the shooting time corresponding to the first data are input into a pre-trained sign language recognition model to obtain the sign language recognition results output by the sign language recognition model. The sign language recognition model is trained using multiple training samples, each of the training samples includes multiple sets of second data, and sample sign language recognition results determined based on the multiple sets of second data, each of the second data includes the three-dimensional coordinates of multiple sample joint points and the shooting time corresponding to the second data.

3. The vehicle control method according to claim 2, characterized in that: The sample sign language recognition result is determined according to the following method: For each of the training samples, inputting the three-dimensional coordinates of the sample joint points corresponding to multiple sets of the second data and the shooting time corresponding to the second data into a pre-trained teacher model to obtain a sign language recognition result output by the teacher model; Using the sign language recognition result output by the teacher model as the sample sign language recognition result in the training sample; The sign language recognition model is obtained according to the following method: During the training process, for each training sample, the three-dimensional coordinates of the sample joint points corresponding to multiple sets of the second data and the shooting time corresponding to the second data are input into the student model to obtain the sign language recognition result output by the student model; Calculating based on the sign language recognition result output by the student model and the sample sign language recognition results in the training samples to obtain a loss value of the student model; When the loss value is less than the preset loss value, the student model is used as the sign language recognition model.

4. The vehicle control method according to claim 1, wherein: The controlling the target object of the vehicle according to the sign language recognition result includes: In a case where the sign language recognition result includes a first identifier and a control instruction, controlling the target object identified by the first identifier in the vehicle according to the control instruction; or, In a case where the sign language recognition result includes a control instruction, determining a target object corresponding to the control instruction, and controlling the target object in the vehicle according to the control instruction; or, In a case where the sign language recognition result includes a first identifier, a target object identified by the first identifier in the vehicle is controlled according to a preset instruction.

5. The vehicle control method according to claim 4, characterized in that: When the sign language recognition result includes a control instruction, determining a target object corresponding to the control instruction includes: When the sign language recognition result includes a control instruction, performing contextual semantic analysis on historical sign language recognition results within a first preset time to obtain a target object identifier, and determining the target object according to the target object identifier; or, When the sign language recognition result includes a control instruction, outputting prompt information, wherein the prompt information is used to request the user to determine the target object; photographing the vehicle cabin within a second preset time to obtain multiple sets of third data; The multiple groups of the third data are used as the multiple groups of the first data, and the process jumps to the step of performing sign language recognition on the multiple groups of the first data to obtain sign language recognition results.

6. The vehicle control method according to any one of claims 1 to 5, characterized in that: In the case where the sign language recognition result includes a target object, after controlling the target object of the vehicle according to the sign language recognition result, the method further includes: In a case where the sign language recognition result includes a target object, determining a plurality of sign language images corresponding to control results for controlling the target object from preset sign language images; Determining the time sequence of each sign language image according to a preset sign language word order rule to obtain a sign language video; Output the sign language video.

7. A vehicle control device, characterized in that: The device is applied to a vehicle and includes: an acquisition module, configured to photograph the vehicle cabin to obtain multiple sets of first data of the user's limbs, each set of the first data comprising a two-dimensional image and a depth image of the user's limbs photographed at the same moment; a recognition module, configured to perform sign language recognition on the plurality of sets of the first data to obtain sign language recognition results; A control module is used to control the target object of the vehicle according to the sign language recognition result when the sign language recognition result includes the target object.

8. An electronic device, characterized in that: include: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the vehicle control method according to any one of claims 1 to 6 is implemented.

9. A vehicle, characterized in that: Comprising the electronic device as claimed in claim 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the vehicle control method according to any one of claims 1 to 6 is implemented.

Images