Vehicle-mounted sign language recognition model training method, intelligent vehicle control method and device

Abstract

The application provides a vehicle-mounted sign language recognition model training method and intelligent vehicle control method and device, and relates to the technical field of intelligent control, to alleviate the technical problem that the fine dynamic characteristics of fingers are difficult to capture through sensors in the prior art, resulting in low recognition accuracy of complex sign language. By inputting video data stream into the vehicle-mounted sign language recognition model to obtain sign language semantic text sequence data, generating a control instruction based on the sign language semantic text sequence data, and controlling the intelligent vehicle to execute the control instruction, the accuracy of complex sign language recognition is improved, and the accuracy of vehicle-machine instruction interaction is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and more specifically, to a method for training an in-vehicle sign language recognition model, an intelligent vehicle control method, and a device. Background Technology

[0002] With the advancement and development of intelligent control technology, the functions of intelligent vehicles are becoming increasingly rich. Thanks to the development of machine learning and deep learning, intelligent vehicle control technology based on voice recognition is becoming more and more mature and accurate. However, existing intelligent vehicle control systems are clearly not considerate enough for people with language barriers and do not take into account their usage. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a vehicle-mounted sign language recognition model training method, an intelligent vehicle control method and device, so as to improve the accuracy of complex sign language recognition.

[0004] In a first aspect, the present invention provides a method for training an in-vehicle sign language recognition model, comprising:

[0005] Obtain a training sample data set; the training sample data set includes multiple training sample data; each training sample data includes the original video stream data and the hand key point label data, shoulder and hand key point label data, and sign language semantic text sequence label data corresponding to the sign language actions in the original video stream data;

[0006] Based on the training sample dataset, the in-vehicle sign language recognition model is iteratively trained. This model includes a keypoint detection sub-model for skip connections and a sign language recognition sub-model. The training operations include:

[0007] Select target training sample data from the training sample data set;

[0008] The original video stream data in the target training sample data is input into the vehicle-mounted sign language recognition model. The vehicle-mounted sign language recognition model extracts features and detects key points from multiple original video frame images in the original video stream data through the key point detection sub-model to obtain hand key point feature data and shoulder and hand key point feature data. The sign language recognition sub-model captures gesture change trajectories and performs sign language action semantic parsing on the hand key point feature data and shoulder and hand key point feature data corresponding to the original video stream data and multiple original video frame images to obtain sign language semantic text sequence prediction data.

[0009] Based on the sign language semantic text sequence prediction data, hand key point feature data, shoulder and hand key point feature data, and sign language semantic text sequence label data, hand key point label data, and shoulder and hand key point label data in the target training sample data, the current loss value is calculated, and the first weight parameter of the key point detection sub-model and the second weight parameter of the sign language recognition sub-model are updated based on the current loss value.

[0010] Optionally, based on the sign language semantic text sequence prediction data, hand keypoint feature data, and shoulder-hand keypoint feature data, as well as the sign language semantic text sequence label data, hand keypoint label data, and shoulder-hand keypoint label data in the target training sample data, the current loss value is calculated, including:

[0011] Based on the hand key point feature data and the hand key point label data in the target training sample data, the hand loss value is calculated using the target loss function;

[0012] Based on the shoulder and hand key point feature data and the shoulder and hand key point label data in the target training sample data, the shoulder and hand loss value is calculated using the target loss function.

[0013] Based on the hand loss value and the shoulder-hand loss value, the feature loss value is obtained;

[0014] Based on sign language semantic text sequence prediction data and sign language semantic text sequence label data, the sequence loss value is calculated using the connection time-series classification loss function;

[0015] The current loss value is calculated based on the feature loss value and the sequence loss value.

[0016] Optionally, the keypoint detection sub-model includes a first local feature extraction module, a first multi-scale feature extraction module, a first bidirectional feature extraction module, a multi-scale feature fusion module, and a keypoint detection module connected in sequence; the keypoint detection sub-model extracts features and detects keypoints in the original video stream data to obtain hand keypoint feature data and shoulder-hand keypoint feature data, including:

[0017] The first local feature extraction module preprocesses and extracts features from the original video stream data to obtain the first key point feature image;

[0018] The first key point feature image is obtained by performing multi-scale feature extraction on the features of the first key point feature image through the first multi-scale feature extraction module.

[0019] The first bidirectional feature extraction module extracts temporal features from the features of the second key point feature image to obtain the third key point feature image;

[0020] The feature image of the third key point is extracted and fused by a multi-scale feature fusion module to obtain a feature fusion image;

[0021] The keypoint detection module performs keypoint detection on the feature fusion image to obtain hand keypoint feature data and shoulder and hand keypoint feature data.

[0022] Optionally, the first local feature extraction module includes a first 3D convolutional layer; the first multi-scale feature extraction module includes multiple sequentially connected first multi-scale convolutional units, each first multi-scale convolutional unit including a first dilated convolutional layer, a first convolutional layer, and a second convolutional layer connected in parallel, and a first concatenation layer connected to the first dilated convolutional layer, the first convolutional layer, and the second convolutional layer respectively, and a first compressed excitation attention layer skipped to the first concatenation layer; the first bidirectional feature extraction module includes two sequentially connected first bidirectional long short-term memory layers; the multi-scale feature fusion module includes a multi-branch convolutional layer; the keypoint detection module includes two first fully connected layers; and the keypoint detection sub-model performs feature extraction and keypoint detection on the original video stream data to obtain hand keypoint feature data and shoulder and hand keypoint feature data, including:

[0023] The original video stream data is preprocessed and features are extracted by the first 3D convolutional layer in the first local feature extraction module to obtain the first key point feature image.

[0024] The first keypoint feature image is obtained by performing multi-scale feature extraction on the features of the first keypoint feature image through the first multi-scale convolution unit in multiple first multi-scale feature extraction modules;

[0025] The second keypoint feature image is obtained by extracting temporal features from the features of the second keypoint feature image through two first bidirectional long short-term memory layers in the first bidirectional feature extraction module;

[0026] The feature image of the third key point is extracted and fused by the multi-branch convolutional layer in the multi-scale feature fusion module to obtain the feature fusion image;

[0027] The first fully connected layer in the keypoint detection module is used to perform keypoint detection on the feature fusion image to obtain hand keypoint feature data and shoulder and hand keypoint feature data.

[0028] Optionally, the key point detection sub-model also includes a random multi-frame feature extraction module; the random multi-frame feature extraction module is connected to the first local feature extraction module and is used to extract multiple original video frame images by grouping the original video stream data.

[0029] Optionally, the sign language recognition sub-model includes a second local feature extraction module, a second multi-scale feature extraction module, a temporal feature extraction module, a second bidirectional feature extraction module, and a sign language action semantic parsing module connected in sequence. The sign language recognition sub-model performs gesture change trajectory capture and sign language action semantic parsing on the original video stream data and the hand and shoulder keypoint feature data corresponding to multiple original video frame images to obtain sign language semantic text sequence prediction data, including:

[0030] The second local feature extraction module captures the hand key point feature data and shoulder and hand key point feature data corresponding to the original video stream data and multiple original video frame images to obtain the first gesture feature image.

[0031] The second gesture feature image is obtained by performing multi-scale feature extraction on the gesture feature image through the second multi-scale feature extraction module.

[0032] The temporal information of the second gesture feature image is extracted by the temporal feature extraction module to obtain the third gesture feature image;

[0033] The second bidirectional feature extraction module extracts temporal features from the third gesture feature image to obtain a fused image.

[0034] The sign language action semantic parsing module performs sign language action semantic parsing on the fused images to obtain sign language semantic text sequence prediction data.

[0035] Optionally, the second local feature extraction module includes a second 3D convolutional layer; the second multi-scale feature extraction module includes multiple sequentially connected second multi-scale convolutional units, each including a second dilated convolutional layer, a third convolutional layer, and a fourth convolutional layer connected in parallel, a second concatenation layer connected to the second dilated convolutional layer, the third convolutional layer, and the fourth convolutional layer respectively, and a second compressed excitation attention layer skipped to the second concatenation layer; the temporal feature extraction module includes a temporal-space convolutional layer; the second bidirectional feature extraction module includes a second bidirectional long short-term memory layer; the sign language action semantic parsing module includes a second fully connected layer; and the sign language recognition sub-model performs gesture change trajectory capture and sign language action semantic parsing on the original video stream data and the hand key point feature data and shoulder and hand key point feature data corresponding to multiple original video frame images to obtain sign language semantic text sequence prediction data, including:

[0036] The second 3D convolutional layer in the second local feature extraction module captures the hand key point feature data and shoulder and hand key point feature data corresponding to the original video stream data and multiple original video frame images to obtain the first gesture feature image.

[0037] The second gesture feature image is obtained by performing multi-scale feature extraction on the features of the gesture feature image through the second multi-scale convolution unit in the second multi-scale feature extraction module;

[0038] Temporal information is extracted from the second gesture feature image by the temporal-space convolutional layer in the temporal feature extraction module to obtain the third gesture feature image;

[0039] Temporal features of the third gesture feature image are extracted by the second bidirectional long short-term memory layer in the second bidirectional feature extraction module to obtain a fused image.

[0040] The second fully connected layer in the sign language action semantic parsing module is used to perform sign language action semantic parsing on the fused image to obtain sign language semantic text sequence prediction data.

[0041] Secondly, the present invention provides an intelligent vehicle control method, comprising:

[0042] Acquire video stream data from inside the cockpit of intelligent vehicles;

[0043] The video stream data is input into the vehicle-mounted sign language recognition model to obtain the sign language semantic text sequence data corresponding to the video stream data; the vehicle-mounted sign language recognition model is trained using the above-mentioned vehicle-mounted sign language recognition model training method.

[0044] Based on sign language semantic text sequence data, control commands are generated, and the intelligent vehicle is controlled to execute the control commands.

[0045] Thirdly, the present invention provides a vehicle-mounted sign language recognition model training device, comprising:

[0046] The data acquisition module is used to acquire a training sample data set; the training sample data set includes multiple training sample data; each training sample data includes the original video stream data and the hand key point label data, shoulder and hand key point label data, and sign language semantic text sequence label data corresponding to the sign language actions in the original video stream data;

[0047] The model training module is used to iteratively train the in-vehicle sign language recognition model based on a training sample dataset. The in-vehicle sign language recognition model includes a keypoint detection sub-model for skip connections and a sign language recognition sub-model. The training operations include: selecting target training sample data from the training sample dataset; inputting the raw video stream data from the target training sample data into the in-vehicle sign language recognition model, so that the in-vehicle sign language recognition model can extract features and detect keypoints from multiple frames of the raw video stream data through the keypoint detection sub-model to obtain hand keypoint feature data and shoulder-hand keypoint feature data, and then using the hand... The speech recognition sub-model captures gesture change trajectories and performs semantic parsing of sign language actions on the original video stream data and the hand and shoulder key point feature data corresponding to multiple original video frame images to obtain sign language semantic text sequence prediction data. Based on the sign language semantic text sequence prediction data, hand and shoulder key point feature data, and sign language semantic text sequence label data, hand and shoulder key point label data in the target training sample data, the current loss value is calculated, and the first weight parameter of the key point detection sub-model and the second weight parameter of the sign language recognition sub-model are updated based on the current loss value.

[0048] Fourthly, the present invention provides an intelligent vehicle control device, comprising:

[0049] The acquisition module is used to acquire video stream data from inside the cockpit of intelligent vehicles;

[0050] The recognition module is used to input video stream data into the vehicle-mounted sign language recognition model to obtain sign language semantic text sequence data corresponding to the video stream data; wherein, the vehicle-mounted sign language recognition model is trained using the above-mentioned vehicle-mounted sign language recognition model training method.

[0051] The control module is used to generate control commands based on sign language semantic text sequence data and to control the intelligent vehicle to execute the control commands.

[0052] Fifthly, the present invention also provides an electronic device, comprising: a processor and a memory; the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the above-described vehicle-mounted sign language recognition model training method or the above-described intelligent vehicle control method.

[0053] In a sixth aspect, the present invention also provides a computer-readable storage medium storing a computer program, which, when run by a processor, implements the above-described vehicle-mounted sign language recognition model training method or the above-described intelligent vehicle control method.

[0054] The present invention provides a vehicle-mounted sign language recognition model training method, an intelligent vehicle control method, and an apparatus. By inputting video data streams into the vehicle-mounted sign language recognition model to obtain sign language semantic text sequence data, generating control commands based on the sign language semantic text sequence data, and controlling the intelligent vehicle to execute the control commands, the accuracy of complex sign language recognition is improved, thereby enhancing the accuracy of vehicle-machine command interaction.

[0055] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 The flowchart illustrates a method for training an in-vehicle sign language recognition model according to an embodiment of the present invention.

[0058] Figure 2 A flowchart illustrating the method for obtaining a training sample dataset provided in an embodiment of the present invention is shown.

[0059] Figure 3 This diagram illustrates the hand key point feature data structure provided in an embodiment of the present invention.

[0060] Figure 4 This diagram illustrates the shoulder and hand key point feature data structure provided in an embodiment of the present invention.

[0061] Figure 5 A schematic diagram of the structure of the vehicle-mounted sign language recognition model provided in an embodiment of the present invention is shown;

[0062] Figure 6 A schematic diagram of the key point detection sub-model structure provided in an embodiment of the present invention is shown;

[0063] Figure 7 A schematic diagram of the feature dimension extension structure provided in an embodiment of the present invention is shown;

[0064] Figure 8 A flowchart illustrating the loss calculation method for the vehicle-mounted sign language recognition model provided in an embodiment of the present invention is shown.

[0065] Figure 9 A flowchart illustrating an intelligent vehicle control method provided in an embodiment of the present invention is shown.

[0066] Figure 10 This diagram illustrates the structure of a vehicle-mounted sign language recognition model training device provided in an embodiment of the present invention.

[0067] Figure 11 This diagram illustrates the structure of an intelligent vehicle control device provided in an embodiment of the present invention.

[0068] Figure 12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0070] Currently, there are two main approaches to sign language recognition technology: The first involves acquiring sign language motion data by wearing sensors (such as accelerometers and gyroscopes) on the hand and then using a deep learning model to recognize the gestures. This method can accurately capture the movement trajectory and deformation of gestures and is unaffected by ambient light or individual differences such as age and body shape. However, this method also has certain drawbacks, such as the impact of wearing the device on user experience, high equipment costs, and difficulty in capturing subtle dynamic features of the fingers, resulting in lower recognition accuracy for complex sign language. The second approach is based on deep learning... Xi's sign language recognition method captures video sequences of individual sign language movements using a camera and then uses a deep learning model to detect and recognize the sign language. This approach has strong feature extraction and pattern recognition capabilities and can automatically learn high-level semantic features of sign language from a large amount of data. However, this method also has certain drawbacks: it is easily affected by factors such as changes in ambient lighting, occlusion, and cluttered backgrounds; it has high requirements for hardware; and its recognition accuracy tends to decrease in complex scenes. Furthermore, the model training process relies on a large amount of labeled data, and the scarcity of labeled data resources limits its generalization performance.

[0071] The two methods mentioned above have not yet been applied to the scenario of sign language recognition for deaf and mute people in the non-driver's seat of the intelligent vehicle cabin. This technology has not yet been effectively explored and applied in the cabin scenario, and related development is almost non-existent.

[0072] To this end, this application trains an in-vehicle sign language recognition model using raw video stream data and hand key point label data, shoulder and hand key point label data, and sign language semantic text sequence label data corresponding to sign language actions in the raw video stream data. By inputting video frame images into the in-vehicle sign language recognition model to obtain text data, control commands are generated based on the text data, and the intelligent vehicle is controlled to execute the control command actions, thereby improving the accuracy of complex sign language recognition and improving the accuracy of vehicle-machine command interaction. This fills the gap in sign language recognition technology for deaf and mute people in non-driver's seats in the intelligent vehicle cabin.

[0073] After introducing the application scenarios and design concepts of the present invention, the technical solutions provided by the present invention will be described in detail below.

[0074] This invention provides a method for training an in-vehicle sign language recognition model, such as... Figure 1 As shown, the vehicle-mounted sign language recognition model training method provided by this embodiment of the invention includes at least the following steps:

[0075] Step 110: Obtain the training sample data set; the training sample data set includes multiple training sample data; each training sample data includes the original video stream data and the hand key point label data, shoulder and hand key point label data and sign language semantic text sequence label data corresponding to the sign language actions in the original video stream data.

[0076] In this application embodiment, the following methods may be used, but are not limited to, when obtaining the training sample data set:

[0077] The system collects raw RGB-IR (RGB-Infrared) video stream data from inside the vehicle cabin using an Occupancy Monitoring System (OMS), and then performs frame-slicing processing on the raw video stream data to obtain multiple raw video frame images.

[0078] By manually or automatically annotating the key point data and sign language data in the original video frame images, hand key point label data, shoulder and hand key point label data and sign language semantic text sequence label data are obtained;

[0079] A training sample dataset is obtained based on the original video stream data, hand key point label data, shoulder and hand key point label data, and sign language semantic text sequence label data.

[0080] Specifically, by performing keypoint and sign language tasks on the raw RGB-IR video stream data, a training sample dataset is obtained. The specific process is as follows: Figure 2 As shown:

[0081] First, for the keypoint task, the RGB-IR video data is segmented into frames, and each frame is labeled to obtain hand keypoint label data and shoulder / hand keypoint label data. The hand keypoint label data consists of 2×21 keypoints, and the shoulder / hand keypoint label data consists of 7 keypoints. Specifically, a pre-defined annotation model is first used to automatically label the human hand, arm, and shoulder in each frame of the video image. Then, a keypoint detection and annotation tool is used for manual verification to obtain the hand keypoint label data GT and the arm / shoulder keypoint label data GT, as shown below. Figure 3 As shown, each keypoint in the hand keypoint label data has a number, starting from 0 and ending at 41. The first 21 points represent the left hand, and the last 21 points represent the right hand. Combinations of multiple points represent different fingers; for example... Figure 4 As shown, there are 7 shoulder and hand keypoint labels from left to right: left wrist, left elbow, left shoulder, neck, right shoulder, right elbow, and right wrist. Each keypoint includes (x, y) coordinates.

[0082] For the sign language task, the video segments containing the same sign language content in the RGB-IR raw video stream data are segmented to obtain sign language video images. These sign language video images are then labeled to obtain sign language semantic text sequence label data GT. The sign language video images are labeled by professionals, and the specific labeling format is [segment video path, "Shanghai's weather today is cloudy turning to rain", "Shanghai / today / weather / overcast / change / light / rain"]. Here, "segment video path" represents the path of the video data and is in mp4 format, "Shanghai's weather today is cloudy turning to rain" represents the content of the sign language in the video, and "Shanghai / today / weather / overcast / change / light / rain" represents the sign language expression.

[0083] Furthermore, the tag data GT is processed. Specifically, for the hand key point tag data GT, the coordinate information [x, y, index] of each tag needs to be converted into the format [tx, ty, index], where tx = x / w, w is the width of the sign language video image, ty = y / h, h is the height of the sign language video image, and the index is [0-41]. The index of the hand key point tag data GT is obtained by combining and sorting, combining 42 tags together to form a one-dimensional data group [1, 42, 2], and denoted as hand key point tag data GT-KEY-Hand;

[0084] For the shoulder and hand keypoint label data GT, the coordinate information [x, y, index] of each label needs to be converted into the format [tx, ty, index], where tx = x / w, w is the width, ty = y / h, h is the height, and the index is [0-6]. The index of the shoulder and hand keypoint label data GT is obtained by combining and sorting, combining 7 labels together to form a one-dimensional data group [1, 7, 2], and denoted as shoulder and hand keypoint label data GT-KEY-JS;

[0085] For sign language semantic text sequence label data GT, for example, if the original text label is: "Shanghai / Today / Weather / Cloudy / Change / Light / Rain / ", firstly, a word dictionary table is formed based on all the original text labels, with each character or word corresponding to a numerical ID. The text of the original text labels separated by " / " is transformed into ["Shanghai", "Today", "Weather", "Cloudy", "Change", "Light", "Rain"], and the transformed sentences are matched with the word dictionary, that is, transformed into [23, 564, 12, 234, 54, 170, 102]. The sign language semantic text sequence label data is denoted as GT-CTC.

[0086] Since the raw RGB-IR video stream data used contains two image formats, one is a normal RGB color image and the other is a grayscale image at night, in this embodiment of the application, the RGB color image and the grayscale image are normalized to obtain an image with a single channel image value between [0-1].

[0087] Specifically, in this embodiment, the cv2.imread function is used to read image data from the original video stream. First, the image data in the original video stream is converted from BGR to RGB using the cv2.cvtColor function to obtain the original dimension [B,C,H,W], where B represents the batch dimension, C represents the channel dimension (C=3), H represents the height dimension, and W represents the width dimension. Then, the original dimension [B,1,H,W] of the grayscale image and the original dimension [B,C,H,W] of the RGB image are concatenated to obtain the original feature dimension [B,4,H,W]. Finally, the concatenated original features are normalized, for example, to RGB-1. The original R values ​​are [230, 100, 20, 100]. After normalization, first dividing by 255 yields [0.0784, 0.0824, 0.0980, 0.0824]. Then, subtracting the average value [0.485, 0.456, 0.406, 0.456] in RGB-IR order, and then dividing by the standard deviation [0.229, 0.224, 0.225, 0.224], we finally obtain [-1.7754, -1.6681, -1.3687, -1.6681]. This ensures that all channel values ​​have a mean of 0 and a unit variance. The processed feature data is then input into the vehicle sign language recognition model to accelerate model convergence and achieve more stable training.

[0088] Step 120: Select target training sample data from the training sample data set.

[0089] Step 130: Input the original video stream data from the target training sample data into the vehicle-mounted sign language recognition model, so that the vehicle-mounted sign language recognition model can extract features and detect key points from multiple original video frame images in the original video stream data through the key point detection sub-model to obtain hand key point feature data and shoulder and hand key point feature data. Then, through the sign language recognition sub-model, capture gesture change trajectory and perform sign language action semantic parsing on the hand key point feature data and shoulder and hand key point feature data corresponding to the original video stream data and multiple original video frame images to obtain sign language semantic text sequence prediction data.

[0090] Step 140: Based on the sign language semantic text sequence prediction data, hand key point feature data, shoulder and hand key point feature data, and sign language semantic text sequence label data, hand key point label data, and shoulder and hand key point label data in the target training sample data, calculate the current loss value, and update the first weight parameter of the key point detection sub-model and the second weight parameter of the sign language recognition sub-model based on the current loss value.

[0091] Step 150: Determine if the iterative training termination condition is met; if yes, proceed to step 160; if no, return to step 120; wherein, the iterative training termination condition is that the number of iterations is not less than the number threshold, or the first loss value is not higher than the first loss value threshold.

[0092] Step 160: Based on the parameters of the key point detection sub-model and the sign language recognition sub-model updated during the last iteration of training, the vehicle-mounted sign language recognition model is obtained.

[0093] In the embodiments of this application, such as Figure 5 As shown, the vehicle-mounted sign language recognition model includes a keypoint detection sub-model and a sign language recognition sub-model. The original video stream data is input into the vehicle-mounted sign language recognition model, which uses the keypoint detection sub-model to extract features and detect key points from multiple frames of the original video stream data to obtain hand and shoulder / hand keypoint feature data. The sign language recognition sub-model uses the hand and shoulder / hand keypoint feature data corresponding to the original video stream data and the multiple frames of the original video stream data to capture gesture change trajectories and perform semantic parsing of sign language actions to obtain sign language semantic text sequence prediction data. Based on the sign language semantic text sequence prediction data, hand and shoulder / hand keypoint feature data, and the sign language semantic text sequence label data GT-CTC, hand keypoint label data GT-KEY-Hand, and shoulder / hand keypoint label data GT-KEY-JS from the target training sample data, the current loss value is calculated. Based on the current loss value, the first weight parameter of the keypoint detection sub-model and the second weight parameter of the sign language recognition sub-model are updated.

[0094] Furthermore, the keypoint detection sub-model backbone is based on the traditional residual network ResNet. It introduces a multi-scale feature extraction module InceptionBlock to achieve multi-scale feature fusion through parallel convolutional kernels of different sizes. It also introduces a bidirectional feature extraction module BiLSTM to extract higher-level temporal features, enhance sensitivity to subtle changes, and accurately capture forward and backward temporal dependencies. A multi-scale feature fusion module Multi-CNN is introduced to further refine feature information, including the trend, direction, and amplitude of the action. Two fully connected layers (FC) are introduced to predict 42 keypoints of the hand keypoint feature data and 7 keypoints of the shoulder and hand keypoint feature data, respectively.

[0095] The sign language recognition sub-model backbone is based on the traditional residual network ResNet, and a temporal feature extraction module TCNN is introduced to enhance the extraction and compression of time series features; a bidirectional feature extraction module BiLSTM is introduced to keep the feature dimension of the extracted features unchanged; and a fully connected layer is introduced to obtain text data.

[0096] The loss calculation of the vehicle-mounted sign language recognition model is based on the following: L2-hand-loss is obtained from hand key point feature data and hand label data GT-KEY-Hand; L2-JS-loss is obtained from shoulder and hand key point feature data and shoulder and hand label data GT-KEY-JS; and CTC-loss is obtained from text data and sign language label data GT-CTC. The weights of the key point detection sub-model are updated based on the L2-hand-loss and L2-JS-loss, and the weights of the sign language recognition sub-model are updated based on the CTC-loss.

[0097] In one optional embodiment, the keypoint detection sub-model includes a first local feature extraction module, a first multi-scale feature extraction module, a first bidirectional feature extraction module, a multi-scale feature fusion module, and a keypoint detection module connected in sequence. The keypoint detection sub-model performs feature extraction and keypoint detection on the original video stream data to obtain hand keypoint feature data and shoulder / hand keypoint feature data, including: preprocessing and extracting features from the original video stream data using the first local feature extraction module to obtain a first keypoint feature image; performing multi-scale feature extraction on the features of the first keypoint feature image using the first multi-scale feature extraction module to obtain a second keypoint feature image; performing temporal feature extraction on the features of the second keypoint feature image using the first bidirectional feature extraction module to obtain a third keypoint feature image; extracting and fusing the third keypoint feature image using the multi-scale feature fusion module to obtain a feature fusion image; and performing keypoint detection on the feature fusion image using the keypoint detection module to obtain hand keypoint feature data and shoulder / hand keypoint feature data.

[0098] Furthermore, the first local feature extraction module includes a first 3D convolutional layer; the first multi-scale feature extraction module includes multiple sequentially connected first multi-scale convolutional units, each first multi-scale convolutional unit including a first dilated convolutional layer, a first convolutional layer, and a second convolutional layer connected in parallel, and a first splicing layer connected to the first dilated convolutional layer, the first convolutional layer, and the second convolutional layer respectively, and a first compressed excitation attention layer skipped to the first splicing layer; the first bidirectional feature extraction module includes two sequentially connected first bidirectional long short-term memory layers; the multi-scale feature fusion module includes a multi-branch convolutional layer; the keypoint detection module includes two first fully connected layers; and the keypoint detection sub-model performs feature extraction and keypoint detection on the original video stream data to obtain hand keypoint feature data and shoulder and hand keypoint feature data, including: through... The first 3D convolutional layer in the first local feature extraction module preprocesses and extracts features from the original video stream data to obtain the first keypoint feature image. The first multi-scale convolutional units in multiple first multi-scale feature extraction modules then perform multi-scale feature extraction on the features of the first keypoint feature image to obtain the second keypoint feature image. Two first bidirectional long short-term memory layers in the first bidirectional feature extraction module then perform temporal feature extraction on the features of the second keypoint feature image to obtain the third keypoint feature image. The multi-branch convolutional layer in the multi-scale feature fusion module extracts and fuses the features of the third keypoint feature image to obtain a feature fusion image. Finally, the first fully connected layer in the keypoint detection module performs keypoint detection on the feature fusion image to obtain hand keypoint feature data and shoulder / hand keypoint feature data.

[0099] In the embodiments of this application, such as Figure 6 As shown, the keypoint detection sub-model comprises, in sequence, a first 3D convolutional layer (Conv3d) in the first local feature extraction module, four first multi-scale convolutional units (InceptionBlock) in the first multi-scale feature extraction module, two first bidirectional long short-term memory (BiLSTM) layers in the first bidirectional feature extraction module, a multi-branch convolutional layer (Multi-CNN) in the multi-scale feature fusion module, and two first fully connected layers (FC) in the keypoint detection module. The 3D convolutional layer (Conv3d) has 64 convolutional kernels with a depth of 4. Four multi-scale feature extraction layers enhance the ability to extract fine-grained features. Two first bidirectional long short-term memory (BiLSTM) layers extract higher-level temporal features to enhance sensitivity to subtle changes and accurately capture forward and reverse temporal dependencies, thereby improving the ability to model dynamic changes. The multi-branch convolutional layer (Multi-CNN) further refines feature information, mainly including the trend, direction, and amplitude of the action. Two fully connected layers predict 42 keypoints of the hand and 7 keypoints of the shoulder and hand, respectively.

[0100] To highlight key details, an attention mechanism (SE) module is embedded in the multi-scale feature extraction module. Specifically, the multi-scale feature extraction module also includes a first compressed excitation attention layer, which adaptively adjusts the feature weights of different channels or spatial locations, especially the fine motor features of the fingers.

[0101] In order to perform subsequent feature fusion more efficiently, reduce the computational cost of the vehicle sign language recognition model and reduce feature redundancy, in this embodiment of the application, the key point detection sub-model further includes a random multi-frame feature extraction module; the random multi-frame feature extraction module is connected to the first local feature extraction module and is used to extract multiple frames of original video stream data in groups to obtain multiple original video frame images.

[0102] Specifically, a random multi-frame feature extraction module is introduced into the vehicle-mounted sign language recognition model. The original video stream data is input into the random multi-frame feature extraction module so that the random multi-frame feature extraction module extracts k frames as input to the key point detection sub-model. The original video stream data is divided into K groups on average, and one frame is extracted from each of the K groups to form k frames. The k frames are not frames at fixed times, where K and k are both integers.

[0103] In one optional embodiment, the sign language recognition sub-model includes a second local feature extraction module, a second multi-scale feature extraction module, a temporal feature extraction module, a second bidirectional feature extraction module, and a sign language action semantic parsing module connected in sequence. The sign language recognition sub-model performs gesture change trajectory capture and sign language action semantic parsing on the original video stream data and the hand and shoulder keypoint feature data corresponding to multiple original video frame images to obtain sign language semantic text sequence prediction data. This includes: capturing gesture change trajectories from the original video stream data and the hand and shoulder keypoint feature data corresponding to multiple original video frame images using the second local feature extraction module to obtain a first gesture feature image; extracting multi-scale features from the gesture feature image using the second multi-scale feature extraction module to obtain a second gesture feature image; extracting temporal information from the second gesture feature image using the temporal feature extraction module to obtain a third gesture feature image; extracting temporal features from the third gesture feature image using the second bidirectional feature extraction module to obtain a fused image; and performing sign language action semantic parsing on the fused image using the sign language action semantic parsing module to obtain sign language semantic text sequence prediction data.

[0104] Furthermore, the second local feature extraction module includes a second 3D convolutional layer; the second multi-scale feature extraction module includes multiple sequentially connected second multi-scale convolutional units, each including a second dilated convolutional layer, a third convolutional layer, and a fourth convolutional layer connected in parallel, a second splicing layer connected to the second dilated convolutional layer, the third convolutional layer, and the fourth convolutional layer respectively, and a second compressed excitation attention layer skipped to the second splicing layer; the temporal feature extraction module includes a temporal-spatial convolutional layer; the second bidirectional feature extraction module includes a second bidirectional long short-term memory layer; the sign language action semantic parsing module includes a second fully connected layer; and the sign language recognition sub-model performs gesture change trajectory capture and sign language action semantic parsing on the original video stream data and the hand key point feature data and shoulder and hand key point feature data corresponding to multiple original video frame images to obtain sign language semantic text sequence prediction data. The process includes: capturing hand and shoulder keypoint feature data corresponding to the original video stream data and multiple original video frame images through the second 3D convolutional layer in the second local feature extraction module to obtain a first gesture feature image; extracting multi-scale features from the gesture feature image through the second multi-scale convolutional unit in the second multi-scale feature extraction module to obtain a second gesture feature image; extracting temporal information from the second gesture feature image through the temporal-space convolutional layer in the temporal feature extraction module to obtain a third gesture feature image; extracting temporal features from the third gesture feature image through the second bidirectional long short-term memory layer in the second bidirectional feature extraction module to obtain a fused image; and performing sign language semantic analysis on the fused image through the second fully connected layer in the sign language action semantic analysis module to obtain sign language semantic text sequence prediction data.

[0105] To ensure the integrity of sign language actions, in the embodiments of this application, one input of the sign language recognition sub-model is the original video stream data, whose feature dimension is [B, maxLen, C, w, h], where B represents the batch size, maxLen represents the maximum value of the short video frame data in the current batch, which is a dynamic value, C represents the number of channels, generally 3, w represents the width, and h represents the height. After preprocessing, it is sampled to [B, maxLen, C, 224, 224]. The preprocessed original video is processed by the residual network of the sign language recognition sub-model to obtain a residual feature dimension of [xLen, 1, 1024], where xLen is the dynamic value of the feature length, 1 is the single position after spatial dimension compression, and 1024 is the number of feature channels. The residual feature dimension is denoted as fea1. Another input of the sign language recognition sub-model is the hand key point feature data and shoulder-hand key point feature data output by the key point detection sub-model, whose feature dimension is [xmLen, 1, 1024], and the feature dimension is denoted as fea0. Since xmLen < xLen and the two features cannot be directly added, fea0 is first extended in the first dimension to xLen. The specific operation is as Figure 7 shown. Assume xmLen = 3 and xLen = 6. The first dimension xmLen = 3 is extended to xLen = 6. The extension is a uniform extension to ensure the generality of the features. The extended fea0 feature is denoted as fea0n. Secondly, fea1 and fea0n are added. The dimension after addition remains unchanged, and the feature dimension after addition is [xLen, 1, 1024], denoted as the feature fea. Then, the feature dimension after addition is input into the temporal feature extraction module to extract and compress the time series features, and important temporal information is refined through layer-by-layer convolution and pooling operations. Then, the image output by the temporal feature extraction module is input into the second bidirectional feature extraction module BiLSTM to keep the number of feature dimensions unchanged. Finally, the image feature dimension output by the second bidirectional feature extraction module BiLSTM is input into the second fully connected layer FC to obtain the sign language semantic text sequence prediction data, where the number of feature dimensions output by the second fully connected layer FC is [xLen, dictNum], and dictNum is the size of the sign language corresponding Chinese character dictionary table.

[0106] In one optional embodiment, the current loss value is calculated based on sign language semantic text sequence prediction data, hand keypoint feature data, shoulder and hand keypoint feature data, and sign language semantic text sequence label data, hand keypoint label data, and shoulder and hand keypoint label data in the target training sample data. This includes: calculating a hand loss value using a target loss function based on the hand keypoint feature data and the hand keypoint label data in the target training sample data; calculating a shoulder and hand loss value using a target loss function based on the shoulder and hand keypoint feature data and the shoulder and hand keypoint label data in the target training sample data; obtaining a feature loss value based on the hand loss value and the shoulder and hand loss value; calculating a sequence loss value using a connection-time classification loss function based on the sign language semantic text sequence prediction data and the sign language semantic text sequence label data; and calculating the current loss value based on the feature loss value and the sequence loss value.

[0107] In one optional embodiment, the calculation of the current loss value based on sign language semantic text sequence prediction data, hand key point feature data, shoulder and hand key point feature data, and sign language semantic text sequence label data, hand key point label data, and shoulder and hand key point label data in the target training sample data further includes: calculating the sequence loss value using a connection-time classification loss function based on the sign language semantic text sequence prediction data and the sign language semantic text sequence label data; determining the gradient value corresponding to the sign language recognition sub-model based on the sequence loss value and the sign language semantic text sequence prediction data; updating the gradient value based on an optimization function, and inputting the updated gradient value into the sign language recognition sub-model to perform backpropagation to update the second weight parameter of the sign language recognition sub-model.

[0108] In this embodiment, the keypoint detection sub-model uses two losses, L1-hand-loss and L1-JS-loss, to calculate the loss value between the predicted keypoint coordinates and the standard coordinates. The sign language recognition sub-model uses CTC-loss as the loss value, which is used for sequence-to-sequence tasks to solve the problem of misalignment between input and output lengths. It efficiently calculates the alignment path through dynamic programming and automatically learns the optimal input-to-output mapping during training. The parameters of the vehicle sign language recognition model are jointly optimized by multiplying the L1-hand-loss, L1-JS-loss, and CTC-loss values ​​by adaptive weight parameters and then summing them together.

[0109] Specifically, such as Figure 8As shown, in this application, the optimization of the vehicle-mounted sign language recognition model through loss calculation mainly includes two branches. One branch is to input the original video stream data into the key point detection sub-model to obtain hand key point feature data and shoulder and hand key point feature data. Based on the hand key point feature data and shoulder and hand key point feature data, along with hand key label data and shoulder and hand key label data, a first loss value, i.e., result 1, is obtained. The other branch is to input the original video stream data, hand key point feature data, and shoulder and hand key point feature data into the sign language recognition sub-model to obtain sign language semantic text sequence prediction data. Based on the sign language semantic text sequence prediction data and sign language semantic text sequence label data, a second loss value, i.e., result 2, is obtained. Gradient optimization is performed after backpropagation based on the second loss value to update the first weight parameter and the second weight parameter of the sign language recognition sub-model until the loss value reaches a preset threshold, thus obtaining the weight parameters of the vehicle-mounted sign language recognition model.

[0110] In this embodiment, by analyzing the factors for sign language recognition, and combining the overall information of the user's upper body, key points of the hands, and key points of the arms, the dependent features of the entire vehicle sign language recognition model are formed to achieve comprehensive and accurate sign language translation. Furthermore, the vehicle sign language recognition model is obtained by modeling and learning based on time-frequency data and deep learning models. Compared with single-modal information modeled based on sensor data, the vehicle sign language recognition model can encompass more information, making it more generalizable and stable.

[0111] Figure 9 This is a flowchart illustrating an intelligent vehicle control method provided in an embodiment of the present invention, as shown below. Figure 9 As shown, the intelligent vehicle control method includes at least the following steps:

[0112] Step 210: Obtain video stream data from inside the smart vehicle's cockpit.

[0113] Step 220: Input the video stream data into the vehicle-mounted sign language recognition model to obtain the sign language semantic text sequence data corresponding to the video stream data; wherein, the vehicle-mounted sign language recognition model is trained using the above-mentioned vehicle-mounted sign language recognition model training method.

[0114] Step 230: Based on the sign language semantic text sequence data, generate control commands and control the intelligent vehicle to execute the control commands.

[0115] In this embodiment, a vehicle-mounted sign language recognition model is used to translate sign language information in video stream data. Based on the translated sign language information, intelligent vehicle control commands are generated to enable disabled persons (non-drivers) to control the air conditioning, audio, and other equipment of the intelligent vehicle through sign language. Additionally, video stream data is input into the vehicle-mounted sign language recognition model to obtain sign language semantic text sequence data, thereby eliminating the need for wearable hand sensors and using sensors already deployed in the intelligent vehicle to acquire video image information, thus reducing costs.

[0116] Figure 10 This is a schematic diagram of the structure of a vehicle-mounted sign language recognition model training device provided in an embodiment of the present invention, as shown below. Figure 10 As shown, the model training device includes at least:

[0117] The data acquisition module 310 is used to acquire a training sample data set; wherein the training sample data set includes multiple training sample data; each training sample data includes the original video stream data and the hand key point label data, shoulder and hand key point label data, and sign language semantic text sequence label data corresponding to the sign language actions in the original video stream data;

[0118] The model training module 320 is used to iteratively train the vehicle-mounted sign language recognition model based on a training sample dataset. The vehicle-mounted sign language recognition model includes a keypoint detection sub-model for skip connections and a sign language recognition sub-model. The training operation includes: selecting target training sample data from the training sample dataset; inputting the original video stream data from the target training sample data into the vehicle-mounted sign language recognition model, so that the vehicle-mounted sign language recognition model can extract features and detect keypoints from multiple frames of original video frames in the original video stream data through the keypoint detection sub-model to obtain hand keypoint feature data and shoulder-hand keypoint feature data. The sign language recognition sub-model captures gesture change trajectories and performs semantic parsing of sign language actions on the original video stream data and the hand and shoulder key point feature data corresponding to multiple original video frame images to obtain sign language semantic text sequence prediction data. Based on the sign language semantic text sequence prediction data, hand and shoulder key point feature data, and sign language semantic text sequence label data, hand and shoulder key point label data in the target training sample data, the current loss value is calculated, and the first weight parameter of the key point detection sub-model and the second weight parameter of the sign language recognition sub-model are updated based on the current loss value.

[0119] Figure 11 A schematic diagram of the structure of an intelligent vehicle control device is provided for an embodiment of the present invention, as shown below. Figure 11 As shown, the control device includes at least:

[0120] Acquisition module 410 is used to acquire video stream data inside the cockpit of an intelligent vehicle;

[0121] The recognition module 420 is used to input video stream data into the vehicle-mounted sign language recognition model to obtain sign language semantic text sequence data corresponding to the video stream data; wherein, the vehicle-mounted sign language recognition model is trained using the above-mentioned vehicle-mounted sign language recognition model training method.

[0122] The control module 430 is used to generate control commands based on sign language semantic text sequence data and control the intelligent vehicle to execute the control commands.

[0123] Based on the above embodiments, the present invention provides an electronic device, see below. Figure 12 As shown, the electronic device 600 provided in this embodiment of the invention includes at least a controller 601, a memory 602, and a computer program stored in the memory 602 and executable on the controller 601. When the controller 601 executes the computer program, it implements the above-mentioned vehicle sign language recognition model training method or intelligent vehicle control method provided in this embodiment of the invention.

[0124] The electronic device 600 provided in this embodiment of the invention may further include a bus 603 connecting different components (including a controller 601 and a memory 602). The bus 603 represents one or more types of bus structures, including a memory bus, a peripheral bus, a local area bus, etc.

[0125] Memory 602 may include a readable storage medium in the form of volatile memory, such as random access memory (RAM) 6021 and / or cache memory 6022, and may further include read-only memory (ROM) 6023. Memory 602 may also include a program tool 6025 having a set (at least one) of program modules 6024, including but not limited to an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0126] The controller 601 can be a single processing element or a collective term for multiple processing elements. For example, the controller 601 can be a central processing unit (CPU), or one or more integrated circuits configured to implement the vehicle-mounted sign language recognition model training method or intelligent vehicle control method provided in the embodiments of the present invention. Specifically, the controller 601 can be a general-purpose controller, including but not limited to a CPU, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0127] Electronic device 600 can communicate with one or more external devices 604 (e.g., keyboard, remote control, etc.), and also with one or more devices that enable a user to interact with electronic device 600 (e.g., mobile phone, computer, etc.), and / or with devices that enable electronic device 600 to communicate with one or more other electronic devices 600 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 605. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 606. Figure 6 As shown, network adapter 606 communicates with other modules of electronic device 600 via bus 603. It should be understood that, although... Figure 6 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to microcode, device drivers, redundant controllers, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0128] It should be noted that, Figure 12 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0129] The computer-readable storage medium provided in the embodiments of the present invention is described below. The computer-readable storage medium provided in the embodiments of the present invention stores computer instructions, which, when executed by a controller, implement the above-described vehicle-mounted sign language recognition model training method or intelligent vehicle control method provided in the embodiments of the present invention. In specific implementation, the computer instructions can be built into or installed in the controller, so that the controller can implement the above-described vehicle-mounted sign language recognition model training method or intelligent vehicle control method provided in the embodiments of the present invention by executing the built-in or installed computer instructions.

[0130] Furthermore, the vehicle-mounted sign language recognition model training method or intelligent vehicle control method provided in the embodiments of the present invention can also be implemented as a computer program product. This computer program product includes program code, which, when run on a controller, implements the vehicle-mounted sign language recognition model training method or intelligent vehicle control method provided in the embodiments of the present invention.

[0131] The computer program product provided in the embodiments of the present invention may employ one or more computer-readable storage media, which may be, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination thereof. Specifically, more specific examples (a non-exhaustive list) of computer-readable storage media include electrical connections having one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0132] The computer program product provided in this embodiment of the invention can be a CD-ROM and include program code, and can also run on electronic devices such as in-vehicle sign language recognition model training devices or intelligent vehicle control devices. However, the computer program product provided in this embodiment of the invention is not limited thereto. In this embodiment of the invention, the computer-readable storage medium can be any tangible medium that contains or stores program code, which can be used by or in conjunction with an instruction execution system, device, or apparatus.

[0133] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0134] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0135] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0136] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for training a vehicle-mounted sign language recognition model, characterized in that, include: Obtain the training sample data set; The training sample data set includes multiple training sample data sets; each training sample data set includes original video stream data and hand key point label data, shoulder and hand key point label data, and sign language semantic text sequence label data corresponding to sign language actions in the original video stream data; Based on the aforementioned training sample dataset, the in-vehicle sign language recognition model is iteratively trained; wherein, the in-vehicle sign language recognition model includes a keypoint detection sub-model for skip connections and a sign language recognition sub-model; the training operation includes: Select target training sample data from the training sample data set; The original video stream data in the target training sample data is input into the vehicle-mounted sign language recognition model. The vehicle-mounted sign language recognition model uses the keypoint detection sub-model to extract features and detect keypoints in multiple frames of the original video stream data to obtain hand and shoulder / hand keypoint feature data. The sign language recognition sub-model then uses the original video stream data and the corresponding hand and shoulder / hand keypoint feature data to capture gesture change trajectories and perform semantic parsing of sign language actions to obtain sign language semantic text sequence prediction data. The keypoint detection sub-model includes a first local feature extraction module, a first multi-scale feature extraction module, a first bidirectional feature extraction module, a multi-scale feature fusion module, and a keypoint detection module connected sequentially. The model performs feature extraction and keypoint detection on the original video stream data to obtain hand keypoint feature data and shoulder / hand keypoint feature data, including: preprocessing and extracting features from the original video stream data using the first local feature extraction module to obtain a first keypoint feature image; performing multi-scale feature extraction on the features of the first keypoint feature image using the first multi-scale feature extraction module to obtain a second keypoint feature image; performing temporal feature extraction on the features of the second keypoint feature image using the first bidirectional feature extraction module to obtain a third keypoint feature image; extracting and fusing the third keypoint feature image using the multi-scale feature fusion module to obtain a feature fusion image; and performing keypoint detection on the feature fusion image using the keypoint detection module to obtain hand keypoint feature data and shoulder / hand keypoint feature data. Based on the sign language semantic text sequence prediction data, the hand key point feature data, the shoulder and hand key point feature data, and the sign language semantic text sequence label data, the hand key point label data, and the shoulder and hand key point label data in the target training sample data, the current loss value is calculated, and the first weight parameter of the key point detection sub-model and the second weight parameter of the sign language recognition sub-model are updated based on the current loss value.

2. The vehicle-mounted sign language recognition model training method according to claim 1, characterized in that, Based on the sign language semantic text sequence prediction data, the hand key point feature data, and the shoulder and hand key point feature data, as well as the sign language semantic text sequence label data, the hand key point label data, and the shoulder and hand key point label data in the target training sample data, the current loss value is calculated, including: Based on the hand key point feature data and the hand key point label data in the target training sample data, the hand loss value is calculated using the target loss function; Based on the shoulder and hand key point feature data and the shoulder and hand key point label data in the target training sample data, the shoulder and hand loss value is calculated using the target loss function. Based on the hand loss value and the shoulder-hand loss value, the feature loss value is obtained; Based on the sign language semantic text sequence prediction data and the sign language semantic text sequence label data, the sequence loss value is calculated using the connection time-series classification loss function; The current loss value is calculated based on the feature loss value and the sequence loss value.

3. The vehicle-mounted sign language recognition model training method according to claim 1, characterized in that, The first local feature extraction module includes a first 3D convolutional layer; the first multi-scale feature extraction module includes multiple first multi-scale convolutional units connected in sequence, each first multi-scale convolutional unit including a first dilated convolutional layer, a first convolutional layer, and a second convolutional layer connected in parallel, a first splicing layer connected to the first dilated convolutional layer, the first convolutional layer, and the second convolutional layer respectively, and a first compressed excitation attention layer skipped to the first splicing layer; the first bidirectional feature extraction module includes two first bidirectional long short-term memory layers connected in sequence; the multi-scale feature fusion module includes a multi-branch convolutional layer; the keypoint detection module includes two first fully connected layers; The keypoint detection sub-model extracts features and detects keypoints in the original video stream data to obtain hand keypoint feature data and shoulder and hand keypoint feature data, including: The original video stream data is preprocessed and features are extracted by the first 3D convolutional layer in the first local feature extraction module to obtain the first key point feature image. The first key point feature image is obtained by performing multi-scale feature extraction on the features of the first key point feature image through the first multi-scale convolution unit in multiple first multi-scale feature extraction modules; The second key point feature image is obtained by extracting temporal features from the features of the second key point feature image through the two first bidirectional long short-term memory layers in the first bidirectional feature extraction module; The feature image of the third key point is extracted and fused by the multi-branch convolutional layer in the multi-scale feature fusion module to obtain a feature fusion image; The first fully connected layer in the keypoint detection module is used to perform keypoint detection on the feature fusion image to obtain hand keypoint feature data and shoulder and hand keypoint feature data.

4. The vehicle-mounted sign language recognition model training method according to claim 1 or 3, characterized in that, The key point detection sub-model further includes a random multi-frame feature extraction module; the random multi-frame feature extraction module is connected to the first local feature extraction module and is used to extract the original video stream data in groups to obtain multiple original video frame images.

5. The vehicle-mounted sign language recognition model training method according to claim 1, characterized in that, The sign language recognition sub-model includes a second local feature extraction module, a second multi-scale feature extraction module, a temporal feature extraction module, a second bidirectional feature extraction module, and a sign language action semantic parsing module, connected in sequence. The sign language recognition sub-model performs gesture change trajectory capture and sign language action semantic parsing on the original video stream data and the hand key point feature data and shoulder and hand key point feature data corresponding to multiple frames of the original video frame images to obtain sign language semantic text sequence prediction data, including: The second local feature extraction module captures the hand key point feature data and shoulder and hand key point feature data corresponding to the original video stream data and multiple frames of the original video frame image to obtain the first gesture feature image; The second multi-scale feature extraction module extracts multi-scale features from the gesture feature image to obtain the second gesture feature image. The temporal feature extraction module extracts temporal information from the second gesture feature image to obtain the third gesture feature image; The second bidirectional feature extraction module extracts temporal features from the third gesture feature image to obtain a fused image; The sign language action semantic parsing module performs sign language action semantic parsing on the fused image to obtain sign language semantic text sequence prediction data.

6. The vehicle-mounted sign language recognition model training method according to claim 5, characterized in that, The second local feature extraction module includes a second 3D convolutional layer; the second multi-scale feature extraction module includes multiple sequentially connected second multi-scale convolutional units, each second multi-scale convolutional unit including a second dilated convolutional layer, a third convolutional layer, and a fourth convolutional layer connected in parallel, and a second concatenation layer connected to the second dilated convolutional layer, the third convolutional layer, and the fourth convolutional layer respectively, and a second compressed excitation attention layer skipped to the second concatenation layer; the temporal feature extraction module includes a temporal-spatial convolutional layer; the second bidirectional feature extraction module includes a second bidirectional long short-term memory layer; the sign language action semantic parsing module includes a second fully connected layer; the sign language recognition sub-model performs gesture change trajectory capture and sign language action semantic parsing on the original video stream data and the hand key point feature data and shoulder and hand key point feature data corresponding to multiple frames of the original video frame images to obtain sign language semantic text sequence prediction data, including: The second 3D convolutional layer in the second local feature extraction module captures the hand key point feature data and the shoulder and hand key point feature data corresponding to the original video stream data and multiple frames of the original video frame images to obtain the first gesture feature image; The second gesture feature image is obtained by performing multi-scale feature extraction on the features of the gesture feature image through the second multi-scale convolution unit in the second multi-scale feature extraction module; Temporal information is extracted from the second gesture feature image by the temporal-spatial convolutional layer in the temporal feature extraction module to obtain the third gesture feature image; Temporal features are extracted from the third gesture feature image through the second bidirectional long short-term memory layer in the second bidirectional feature extraction module to obtain a fused image; The fused image is subjected to sign language action semantic parsing through the second fully connected layer in the sign language action semantic parsing module to obtain sign language semantic text sequence prediction data.

7. A method for controlling an intelligent vehicle, characterized in that, include: Acquire video stream data from inside the cockpit of intelligent vehicles; The video stream data is input into the vehicle-mounted sign language recognition model to obtain the sign language semantic text sequence data corresponding to the video stream data; wherein, the vehicle-mounted sign language recognition model is trained using the vehicle-mounted sign language recognition model training method as described in any one of claims 1 to 6; Based on the sign language semantic text sequence data, control commands are generated, and the intelligent vehicle is controlled to execute the control commands.

8. A vehicle-mounted sign language recognition model training device, characterized in that, include: The data acquisition module is used to acquire the training sample data set; The training sample data set includes multiple training sample data sets; each training sample data set includes original video stream data and hand key point label data, shoulder and hand key point label data, and sign language semantic text sequence label data corresponding to sign language actions in the original video stream data; The model training module is used to iteratively train the vehicle-mounted sign language recognition model based on the training sample data set. The vehicle-mounted sign language recognition model includes a keypoint detection sub-model for skip connections and a sign language recognition sub-model. The training operation includes: selecting target training sample data from the training sample data set; inputting the original video stream data from the target training sample data into the vehicle-mounted sign language recognition model, so that the vehicle-mounted sign language recognition model can extract features and detect keypoints from multiple frames of original video frames in the original video stream data using the keypoint detection sub-model to obtain hand keypoints. The feature data and shoulder and hand key point feature data are used by the sign language recognition sub-model to capture gesture change trajectories and perform sign language semantic analysis on the original video stream data and the hand and shoulder key point feature data corresponding to multiple frames of the original video frame images to obtain sign language semantic text sequence prediction data; wherein, the key point detection sub-model includes a first local feature extraction module, a first multi-scale feature extraction module, a first bidirectional feature extraction module, a multi-scale feature fusion module, and a key point detection module connected in sequence; the key point detection sub-model is used to extract features from the original video stream data and The keypoint detection process obtains hand keypoint feature data and shoulder-hand keypoint feature data, including: preprocessing and extracting features from the original video stream data using the first local feature extraction module to obtain a first keypoint feature image; performing multi-scale feature extraction on the features of the first keypoint feature image using the first multi-scale feature extraction module to obtain a second keypoint feature image; performing temporal feature extraction on the features of the second keypoint feature image using the first bidirectional feature extraction module to obtain a third keypoint feature image; extracting and fusing the third keypoint feature image using the multi-scale feature fusion module to obtain a feature fusion image; performing keypoint detection on the feature fusion image using the keypoint detection module to obtain hand keypoint feature data and shoulder-hand keypoint feature data; calculating a current loss value based on the sign language semantic text sequence prediction data, the hand keypoint feature data, and the shoulder-hand keypoint feature data, as well as the sign language semantic text sequence label data, the hand keypoint label data, and the shoulder-hand keypoint label data in the target training sample data; and updating the first weight parameter of the keypoint detection sub-model and the second weight parameter of the sign language recognition sub-model based on the current loss value.

9. An intelligent vehicle control device, characterized in that, include: The acquisition module is used to acquire video stream data from inside the cockpit of intelligent vehicles; The recognition module is used to input the video stream data into the vehicle-mounted sign language recognition model to obtain the sign language semantic text sequence data corresponding to the video stream data; wherein, the vehicle-mounted sign language recognition model is trained using the vehicle-mounted sign language recognition model training method as described in any one of claims 1 to 6. The control module is used to generate control commands based on the sign language semantic text sequence data, and to control the intelligent vehicle to execute the control commands.

10. An electronic device, characterized in that, include: Processor and memory; The memory stores a computer program that can run on the processor, and when the processor executes the computer program, it implements the vehicle sign language recognition model training method as described in any one of claims 1 to 6 or the intelligent vehicle control method as described in claim 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the vehicle sign language recognition model training method as described in any one of claims 1 to 6 or the intelligent vehicle control method as described in claim 7.

Images