A vehicle control method and device based on multi-modal detection and vehicle

Abstract

The application discloses a vehicle control method and device based on multi-modal detection and a vehicle. In the method, in response to detecting a keyword, a first voice signal is acquired, and the first voice signal comprises the keyword; in response to detecting a user, a first portrait signal is acquired; voice features are obtained according to the first voice signal, and a first identity recognition result is obtained; portrait features are obtained according to the first portrait signal, and a second identity recognition result is obtained; after the first identity recognition result indicates successful recognition and the second identity recognition result indicates successful recognition, the voice features and the portrait features are input into a preset multi-modal fusion device to obtain fusion features; and the vehicle is controlled according to a control instruction corresponding to the fusion features. In this way, the vehicle is controlled by comprehensively analyzing the voice signal and the portrait signal, the vehicle can be controlled without a physical key and is not affected by environmental signals, the effectiveness of vehicle control is improved, and the user experience is improved to a certain extent.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and in particular to a vehicle control method, device and vehicle based on multimodal detection. Background Technology

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, vehicle control methods are becoming increasingly diversified. Current vehicle control methods typically include physical control or Bluetooth control. Physical control involves inserting and rotating a physical key to control the vehicle, while Bluetooth control is based on short-range wireless communication technology. When the driver approaches or leaves the vehicle at a fixed distance with the Bluetooth key, the vehicle will automatically unlock or lock.

[0003] However, regarding physical control, if the physical key is lost, the vehicle cannot be controlled, causing significant inconvenience to the user. As for Bluetooth control, Bluetooth signals are easily affected by environmental interference, leading to inaccurate control and a poor user experience. In summary, current vehicle control methods are not very effective, resulting in a poor user experience. Summary of the Invention

[0004] This application provides a vehicle control method, device, and vehicle based on multimodal detection, which can improve the effectiveness of vehicle control and enhance the user experience to a certain extent.

[0005] In a first aspect, this application provides a vehicle control method based on multimodal detection, applied to a vehicle control system, the method comprising:

[0006] In response to the detection of a keyword, a first voice signal is acquired, wherein the first voice signal includes the keyword;

[0007] In response to triggering pedestrian detection, the first human image signal is acquired;

[0008] Based on the first speech signal, speech features and a first identity recognition result are obtained;

[0009] Based on the first facial image signal, facial features and second identity recognition results are obtained;

[0010] After the first identity recognition result and the second identity recognition result are used to indicate successful recognition, the voice features and the facial features are input into a preset multimodal fusion processor to obtain fused features;

[0011] The vehicle is controlled according to the control commands corresponding to the fusion features.

[0012] Optionally, the step of inputting the speech features and the facial features into a preset multimodal fusion processor to obtain fused features includes:

[0013] The speech features and the facial features are added point by point according to the fully connected layer in the preset multimodal fusion machine to obtain the fused features.

[0014] Optionally, controlling the vehicle according to the control command corresponding to the fused feature includes:

[0015] The fused features are classified to obtain classification results;

[0016] The vehicle is controlled according to the control command corresponding to the classification result.

[0017] Optionally, the vehicle control system includes a preset voice detection network, and the step of obtaining voice features and a first identity recognition result based on the first voice signal includes:

[0018] The first speech signal is input into the preset speech detection network to obtain the speech features and the first similarity, wherein the first similarity is used to characterize the similarity between the speech features and the preset voiceprint features;

[0019] The first identity recognition result is determined based on the first similarity.

[0020] Optionally, the preset speech detection network includes an input layer, a hidden layer, and an output layer. The step of inputting the first speech signal into the preset speech detection network to obtain the speech features and the first similarity includes:

[0021] The first speech signal is input into the input layer of the preset speech detection network for preprocessing, and the second speech signal is output.

[0022] The second speech signal is input into the hidden layer for feature learning, and the speech features are output. The hidden layer is used to learn features of the second speech signal based on a preset activation function.

[0023] The speech features are input into the output layer of the preset speech detection network for feature comparison, and the first similarity is output. The output layer of the preset speech detection network is used to calculate the similarity between the speech features and the preset voiceprint features.

[0024] Optionally, the vehicle control system includes a preset video detection network, and the step of obtaining facial features and a second identity recognition result based on the first facial signal includes:

[0025] The first portrait signal is input into the preset video detection network to obtain the portrait features and the second similarity, wherein the second similarity is used to characterize the similarity between the portrait features and the preset portrait features;

[0026] The second identity recognition result is determined based on the second similarity.

[0027] Optionally, the preset video detection network includes an input layer, a convolutional layer, a fully connected layer, and an output layer. The step of inputting the first portrait signal into the preset video detection network for feature extraction to obtain the portrait features includes:

[0028] The first portrait signal is input into the input layer of the preset video detection network for preprocessing, and the second portrait signal is output.

[0029] The second portrait signal is input into the convolutional layer for feature learning and feature extraction, and a third portrait signal is output; the convolutional layer is used to learn features from the second portrait signal based on a preset activation function;

[0030] The third human image signal is input into the fully connected layer for feature integration, and the human image features are output.

[0031] The portrait features are input into the output layer of the preset video detection network for feature comparison, and the second similarity is output. The output layer of the preset video detection network is used to calculate the similarity between the portrait features and the preset portrait features.

[0032] Optionally, the step of acquiring the first voice signal in response to detecting a keyword includes:

[0033] In response to the detection of the keyword, an initial voice signal is acquired;

[0034] The initial speech signal is subjected to mean filtering to obtain the first speech signal.

[0035] Secondly, embodiments of this application also provide a vehicle control device based on multimodal detection, applied to a vehicle control system, the device comprising:

[0036] A speech detection unit is configured to acquire a first speech signal in response to the detection of a keyword, wherein the first speech signal includes the keyword;

[0037] A human face detection unit is used to acquire a first human face signal in response to triggering pedestrian detection.

[0038] The first obtaining unit is configured to obtain speech features and a first identity recognition result based on the first speech signal;

[0039] The second obtaining unit is used to obtain facial features and a second identity recognition result based on the first facial signal;

[0040] The third obtaining unit is used to input the voice features and the facial features into a preset multimodal fusion unit after the first identity recognition result is used to indicate successful recognition and the second identity recognition result is used to indicate successful recognition, so as to obtain fused features;

[0041] The control unit is used to control the vehicle according to the control commands corresponding to the fusion features.

[0042] Optionally, the third obtaining unit is specifically used for:

[0043] The speech features and the facial features are added point by point according to the fully connected layer in the preset multimodal fusion machine to obtain the fused features.

[0044] Optionally, the control unit is specifically used for:

[0045] The fused features are classified to obtain classification results;

[0046] The vehicle is controlled according to the control command corresponding to the classification result.

[0047] Optionally, the vehicle control system includes a preset voice detection network, and the first obtaining unit includes:

[0048] The speech feature module is used to input the first speech signal into the preset speech detection network to obtain the speech features and a first similarity, wherein the first similarity is used to characterize the similarity between the speech features and the preset voiceprint features;

[0049] The first identity determination module is used to determine the first identity recognition result based on the first similarity.

[0050] Optionally, the preset speech detection network includes an input layer, a hidden layer, and an output layer, and the speech feature module is specifically used for:

[0051] The first speech signal is input into the input layer of the preset speech detection network for preprocessing, and the second speech signal is output.

[0052] The second speech signal is input into the hidden layer for feature learning, and the speech features are output. The hidden layer is used to learn features of the second speech signal based on a preset activation function.

[0053] The speech features are input into the output layer of the preset speech detection network for feature comparison, and the first similarity is output. The output layer of the preset speech detection network is used to calculate the similarity between the speech features and the preset voiceprint features.

[0054] Optionally, the vehicle control system includes a preset video detection network, and the second obtaining unit includes:

[0055] A human face feature module is used to input the first human face signal into the preset video detection network to obtain the human face features and a second similarity, wherein the second similarity is used to characterize the similarity between the human face features and the preset human face features;

[0056] The second identity determination module is used to determine the second identity recognition result based on the second similarity.

[0057] Optionally, the preset video detection network includes an input layer, a convolutional layer, a fully connected layer, and an output layer, and the portrait feature module is specifically used for:

[0058] The first portrait signal is input into the input layer of the preset video detection network for preprocessing, and the second portrait signal is output.

[0059] The second portrait signal is input into the convolutional layer for feature learning and feature extraction, and a third portrait signal is output; the convolutional layer is used to learn features from the second portrait signal based on a preset activation function;

[0060] The third human image signal is input into the fully connected layer for feature integration, and the human image features are output.

[0061] The portrait features are input into the output layer of the preset video detection network for feature comparison, and the second similarity is output. The output layer of the preset video detection network is used to calculate the similarity between the portrait features and the preset portrait features.

[0062] Optionally, the speech detection unit is specifically used for:

[0063] In response to the detection of the keyword, an initial voice signal is acquired;

[0064] The initial speech signal is subjected to mean filtering to obtain the first speech signal.

[0065] Thirdly, this application also provides a vehicle that includes the vehicle control device based on multimodal detection provided in the second aspect above.

[0066] Therefore, this application has the following beneficial effects:

[0067] This application provides a vehicle control method, device, and vehicle based on multimodal detection. The method includes: acquiring a first voice signal, the first voice signal including the keyword, in response to detecting a keyword; acquiring a first human image signal in response to triggering pedestrian detection; obtaining voice features and a first identity recognition result based on the first voice signal; obtaining human image features and a second identity recognition result based on the first human image signal; after the first identity recognition result and the second identity recognition result indicate successful recognition, inputting the voice features and human image features into a preset multimodal fusion processor to obtain fused features; and controlling the vehicle according to the control command corresponding to the fused features. Thus, by comprehensively analyzing the voice signal and human image signal to obtain the first and second identity recognition results, and when both recognition results indicate successful recognition, inputting the voice features and human image features into a preset multimodal fusion processor to obtain fused features, and controlling the vehicle according to the control command corresponding to the fused features, the method eliminates the need for a physical key and is unaffected by environmental signals, improving the effectiveness of vehicle control and enhancing the user experience to some extent. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0069] Figure 1 This is a flowchart illustrating a vehicle control method based on multimodal detection in an embodiment of this application.

[0070] Figure 2 This is an interactive schematic diagram of a vehicle control method based on multimodal detection in an embodiment of this application;

[0071] Figure 3 This is a flowchart illustrating an embodiment of a vehicle control method based on multimodal detection in this application.

[0072] Figure 4 This is a schematic diagram of the structure of a vehicle control device 400 based on multimodal detection provided in an embodiment of this application. Detailed Implementation

[0073] The "multiple" mentioned in the embodiments of this application refers to two or more. It should be noted that in the description of the embodiments of this application, terms such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0074] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and specific implementation methods. It should be understood that the specific embodiments described herein are merely for explaining this application and are not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to this application are shown in the accompanying drawings, not the entire structure.

[0075] Currently, vehicle control methods include physical control, Bluetooth control, and fingerprint control. However, regarding physical control, if the physical key is lost, the vehicle cannot be controlled, causing great inconvenience to the user. As for Bluetooth control, it usually relies on measuring the strength of the Bluetooth signal, but due to various factors, this method is not very accurate. For example, the Bluetooth signal may be blocked or reflected by surrounding objects such as walls or people, causing a deviation between the actual signal strength and the measured value. In other words, the Bluetooth signal is easily affected by environmental interference, resulting in inaccurate control and a poor user experience. As for fingerprint control, since it requires the user to actively make physical contact with the vehicle, fingerprints may become unrecognizable due to sweat or skin conditions.

[0076] Based on this, embodiments of this application provide a vehicle control method, device, and vehicle based on multimodal detection. The method includes: in response to detecting a keyword, acquiring a first voice signal, the first voice signal including the keyword; in response to triggering pedestrian detection, acquiring a first human image signal; obtaining voice features and a first identity recognition result based on the first voice signal; obtaining human image features and a second identity recognition result based on the first human image signal; after the first identity recognition result is used to indicate successful recognition and the second identity recognition result is used to indicate successful recognition, inputting the voice features and human image features into a preset multimodal fusion processor to obtain fused features; and controlling the vehicle according to the control command corresponding to the fused features.

[0077] Thus, by comprehensively analyzing voice signals and facial signals, a first identity recognition result and a second identity recognition result are obtained. When both recognition results indicate successful recognition, the voice features and facial features are input into a preset multimodal fusion unit to obtain fused features. The vehicle is then controlled according to the control commands corresponding to the fused features. Compared to traditional physical control, this embodiment does not require a physical key, reducing the risk of lost or stolen keys, improving the vehicle's anti-theft performance, and enhancing vehicle security to a certain extent. Compared to Bluetooth control, this embodiment does not require carrying any equipment and is not affected by environmental signals, reducing misoperation and device compatibility issues. By comprehensively analyzing voice signals and facial signals through a preset multimodal fusion unit, vehicle control can be achieved, improving the effectiveness of vehicle control and enhancing the user experience to a certain extent.

[0078] To facilitate understanding of the specific implementation of the vehicle control method based on multimodal detection provided in the embodiments of this application, the following description will be provided in conjunction with the accompanying drawings.

[0079] It should be noted that the main body implementing this vehicle control method based on multimodal detection can be the vehicle control device based on multimodal detection provided in the embodiments of this application, or it can be carried in an electronic device or a functional module of an electronic device. The electronic device in the embodiments of this application can be any device capable of implementing the vehicle control method based on multimodal detection in the embodiments of this application, such as an Internet of Things (IoT) device.

[0080] Please see Figure 1 , Figure 1 This application provides a flowchart illustrating a vehicle control method based on multimodal detection. This method can be applied to a vehicle control device based on multimodal detection, such as... Figure 4 The vehicle control device 400 based on multimodal detection is shown.

[0081] In this embodiment of the application, the vehicle control system applied to a vehicle may include, for example, the following steps:

[0082] S101: In response to the detection of a keyword, acquire the first voice signal and execute S103.

[0083] It should be noted that the first voice signal includes keywords, which are used to trigger voice detection. When a keyword is detected, the first voice signal including the keyword is acquired. In addition, the keywords are preset, such as "open the door" and "close the door". Of course, in some implementations, the keywords can also be set to "Plan A" or "Plan B", etc., which does not affect the implementation of the embodiments of this application.

[0084] In some implementations, in order to obtain a higher quality first speech signal, the method provided in this application embodiment for obtaining a first speech signal in response to detecting a keyword may include: obtaining an initial speech signal in response to detecting a keyword; and performing mean filtering on the initial speech signal to eliminate noise and smooth the signal to obtain the first speech signal.

[0085] As an example, a filter with a step size of T≥5 is used for mean filtering. The filter step size is greater than 5 to consider the noise cancellation effect. The larger the step size, the better the noise cancellation effect, as shown in formula (1):

[0086]

[0087] It should be noted that in formula (1), A iThe value of j represents the i-th value of the speech signal. The square brackets [] indicate rounding down. The value of j indicates the starting point from the current time point. Usually, the signal is accumulated from the beginning of half the length of the filter window until half the length of the filter window is received in the future. Dividing by T indicates averaging the accumulated sum of the signals, which is then used as the output of the current speech signal.

[0088] S102: In response to triggering pedestrian detection, acquire the first human image signal and execute S104.

[0089] It should be noted that S101 and S102 can be executed only if the vehicle's locking control state S is in automatic mode, i.e., the vehicle control function based on multimodal detection is enabled by default, and the vehicle's control system has pre-stored the driver's preset voiceprint features and preset facial features. Furthermore, S101 and S102 can be executed simultaneously or sequentially; that is, S101 can be executed first, followed by S102, or vice versa, without affecting the implementation of this embodiment. Specifically, detection can be performed using the vehicle's camera to capture three-channel RGB (red, green, blue) color signals, including the environment and the human image. Generally, the pedestrian detection distance is the maximum distance that the camera can distinguish, typically between 10 and 30 meters, and this value can be adjusted according to the specific usage environment.

[0090] S103: Obtain speech features and first identity recognition results based on the first speech signal.

[0091] It should be noted that the vehicle control system includes a preset voice detection network. Therefore, the method provided in this application for obtaining voice features and a first identity recognition result based on a first voice signal includes: inputting the first voice signal into the preset voice detection network to obtain voice features and a first similarity, wherein the first similarity is used to characterize the similarity between the voice features and preset voiceprint features; and determining a first identity recognition result based on the first similarity. If the first similarity is greater than a preset threshold, the first identity recognition result indicates successful recognition, and the vehicle can then issue a prompt indicating successful voice recognition.

[0092] S104: Obtain facial features and second identity recognition results based on the first facial signal.

[0093] It should be noted that the vehicle control system includes a preset video detection network. Therefore, the method provided in this application for obtaining facial features and a second identity recognition result based on a first facial signal may include: inputting the first facial signal into the preset video detection network to obtain facial features and a second similarity, wherein the second similarity is used to characterize the similarity between the facial features and preset facial features, such as pre-recorded facial features of the driver; and determining a second identity recognition result based on the second similarity. If the second similarity is greater than a preset threshold, the second identity recognition result indicates successful recognition, and the vehicle can issue a prompt indicating successful facial recognition. If the second identity recognition result indicates recognition failure, the vehicle can issue a prompt indicating failed facial recognition.

[0094] It should be noted that S103 is executed immediately after S101 is executed, and S104 is executed immediately after S102 is executed. S103 and S104 can be executed simultaneously or sequentially, i.e., S103 is executed first and then S104, or S104 is executed first and then S103. Neither of these will affect the implementation of the embodiments of this application.

[0095] S105: After the first identity recognition result is used to indicate successful recognition and the second identity recognition result is used to indicate successful recognition, the voice features and facial features are input into a preset multimodal fusion device to obtain fused features.

[0096] It should be noted that when both speech recognition and image recognition are successful, the speech features and image features are input into a preset multimodal fusion processor to obtain fused features. In some implementations, step S105 may include: adding the speech features and image features point by point according to the fully connected layer in the preset multimodal fusion processor to obtain fused features, which contain the corresponding speech features and image features.

[0097] S106: Control the vehicle according to the control command corresponding to the fusion feature.

[0098] It should be noted that controlling the vehicle according to the control commands corresponding to the fused features includes: classifying the fused features to obtain classification results; and controlling the vehicle according to the control commands corresponding to the classification results, which may include locking, unlocking, and no operation.

[0099] In some implementations, the classification result can be a one-dimensional vector of length 3, where each element of the vector is in the range of 0-1. When the first element of the vector is the largest, the vehicle is locked; when the second element is the largest, the vehicle is unlocked; and when the third element is the largest, the vehicle is left inactive.

[0100] "No operation" means not performing any operation. This usually occurs when the driver's facial features and voice features do not match. For example, the vehicle includes two pre-registered drivers, driver A and driver B. When the vehicle control system enters information, driver A's voice feature A corresponds to facial feature A, and driver B's voice feature B corresponds to facial feature B. However, in steps S101 to S105, driver A's voice feature A and driver B's facial feature B are obtained respectively. At this time, voice feature A and facial feature B obviously do not match. Therefore, the control command is "no operation," meaning that the vehicle is controlled not to perform any operation.

[0101] Thus, the vehicle control method based on multimodal detection provided in this application uses a preset multimodal fusion unit to achieve comprehensive analysis of voice and human image, which can more comprehensively determine the status of the vehicle user. Compared with single voice and human image recognition technology, it has higher recognition accuracy and robustness, does not require a physical key and is not affected by environmental signals, and can realize remote control of the vehicle in some scenarios, improving the effectiveness of vehicle control and enhancing the user experience.

[0102] Please see Figure 2 , Figure 2 This is an interactive schematic diagram of a vehicle control method based on multimodal detection provided in an embodiment of this application. The vehicle control system includes a voice control system and a video control system. The voice control system includes a preset voice detection network N1, and the video control system includes a preset video detection network N2. The voice detection network N1 extracts voice features A1 from a first voice signal A, and the video detection network N2 extracts facial features V1 from a first facial signal V. A preset multimodal fusion unit N3 in the multimodal fusion detector receives A1 and V1 and obtains fused features S1. The vehicle control system obtains classification results by classifying S1, and controls the vehicle based on the control commands corresponding to the classification results.

[0103] Please see Figure 3 , Figure 3 This is a flowchart illustrating an embodiment of a vehicle control method based on multimodal detection provided in this application. The following is a summary of the process. Figure 2 The specific interaction scenarios shown are for Figure 3 Explanation may include, for example, the following steps:

[0104] S301: In response to the detection of a keyword, acquire the first speech signal A, and then execute S303.

[0105] S302: In response to triggering pedestrian detection, acquire the first human image signal V, and then execute S305.

[0106] S303: Input the first speech signal A into the preset speech detection network N1 to obtain speech features A1 and the first similarity.

[0107] It should be noted that the first similarity is used to characterize the similarity between speech feature A1 and the preset voiceprint feature. The process of obtaining speech feature A1 is shown in formula (2):

[0108] A1=N1(A) Formula (2)

[0109] It should be noted that the preset speech detection network includes an input layer, a hidden layer, and an output layer. Taking into account the memory size, computing speed, and computing accuracy of the vehicle control system, the number of neurons in the input layer, hidden layer, and output layer are set to 512, 1024, and 512, respectively. S303 provided in this embodiment may include:

[0110] S3031: Input the first speech signal A into the input layer of the preset speech detection network N1 for preprocessing, and output the second speech signal.

[0111] The preprocessing step is used to convert the first speech signal A into a format that can be used for training and prediction, and may include, for example, amplification or noise reduction.

[0112] S3032: Input the second speech signal into the hidden layer for feature learning and output speech features A1. The hidden layer is used to learn features of the second speech signal based on a preset activation function; the preset activation function is, for example, the Swish function.

[0113] It should be noted that the Swish function is a self-gated activation function that can automatically adjust the transmission of information, enabling the network to better learn and adapt to different data features. Through the calculation and transformation of the hidden layer, the key features of the second speech signal are further extracted to obtain speech feature A1.

[0114] S3033: Input the speech features into the output layer of the preset speech detection network N1 for feature comparison and output the first similarity.

[0115] It should be noted that the output layer of the preset speech detection network N1 is used to calculate the similarity between speech feature A1 and preset voiceprint features.

[0116] S304: Determine the first identity recognition result based on the first similarity, and then execute S307.

[0117] S305: Input the first human image signal V into the preset video detection network N2 to obtain human image features V1 and the second similarity.

[0118] It should be noted that the second similarity is used to characterize the similarity between portrait feature V1 and the preset portrait feature. The process of obtaining portrait feature V1 is shown in formula (3):

[0119] V1=N2(V) Formula (3)

[0120] It should be noted that the preset video detection network N2 includes an input layer, a convolutional layer, a fully connected layer, and an output layer. S305 provided in this embodiment may include:

[0121] S3051: Input the first portrait signal V into the input layer of the preset video detection network N2 for preprocessing, and output the second portrait signal.

[0122] It should be noted that, through preprocessing, the first portrait signal V can be converted into a 32x32 pixel matrix, which is the second portrait signal. The second portrait signal can be regarded as data in image format.

[0123] S3052: Input the second portrait signal into the convolutional layer for feature learning and feature extraction, and output the third portrait signal; the convolutional layer is used to learn features from the second portrait signal based on a preset activation function.

[0124] In some implementations, the preset detection network may include two convolutional layers. The first convolutional layer contains six 5x5 convolutional kernels with a stride of 1 and no padding. After processing by the first convolutional layer, the size of the second human image signal becomes 28x28, and the number of feature maps increases to six. In addition, the first convolutional layer uses the Swish function as its preset activation function for feature learning. The first convolutional layer is followed by a max pooling layer to downsample the image size to 14x14.

[0125] The second convolutional layer contains 16 5x5 convolutional kernels. After processing by the second convolutional layer, the image size becomes 10x10, and the number of feature maps increases to 16. Similarly, the second convolutional layer also uses the Swish activation function for feature learning. Following the second convolutional layer is a max-pooling layer, which downsamples the image size to 5x5, thus obtaining a 5x5x16 three-dimensional matrix. Furthermore, before entering the fully connected layer, this 5x5x16 three-dimensional matrix is ​​flattened into a one-dimensional vector. Specifically, the three-dimensional matrix is ​​expanded row by row and column by column, and then sorted in order to form a one-dimensional vector of length 400.

[0126] S3053: Input the third human image signal into the fully connected layer for feature integration and output human image feature V1.

[0127] It should be noted that the preset detection network may include two fully connected layers; the first fully connected layer contains 120 neurons, which outputs a one-dimensional vector of length 400 as a one-dimensional vector of length 120 by setting a learnable weight matrix of size (400, 120); the second fully connected layer contains 84 neurons, which outputs a one-dimensional vector of length 120 as a one-dimensional vector of length 84 by setting a learnable weight matrix of size (120, 84).

[0128] S3054: Input the human image feature V1 into the output layer of the preset video detection network N2 for feature comparison and output the second similarity.

[0129] It should be noted that the output layer of the preset video detection network N2 is used to calculate the similarity between human features and preset human features.

[0130] S306: Determine the second identity recognition result based on the second similarity.

[0131] S307: After the first identity recognition result is used to indicate successful recognition and the second identity recognition result is used to indicate successful recognition, the voice feature A1 and the human face feature V1 are input into the preset multimodal fusion unit N3 to obtain the fused feature S1.

[0132] It should be noted that the preset multimodal fusion machine can be regarded as a three-layer perceptron, and the Swish function is used as its activation function for feature learning. The process of obtaining the fused feature S1 is shown in formula (4):

[0133] S1=N3(A1,V1) Formula (4)

[0134] As discussed above, the facial image feature is a one-dimensional vector of length 84. For speech feature fusion, the speech feature is input into a fully connected layer containing 84 neurons to adjust its length to 84, resulting in the adjusted speech feature. Then, the adjusted speech feature and the facial image feature are added point-by-point, that is, the corresponding elements of the speech and facial image features are added together to obtain the fused feature. It should be noted that the point-by-point addition process ensures that the information from both features is effectively integrated into the fused feature, so that each element in the fused feature contains information from both speech and facial image features. This results in a fused feature of length 84, which is then input into a fully connected layer containing 3 neurons for processing, resulting in a fused feature of length 3.

[0135] S308: Control the vehicle according to the control command corresponding to the fusion feature S1.

[0136] Then, the classification result corresponding to the processed fusion feature S1 is determined, and the vehicle is controlled according to the control command corresponding to the classification result.

[0137] See Figure 4 This application also provides a vehicle control device 400 based on multimodal detection, applied to a vehicle control system. The device 400 includes:

[0138] The voice detection unit 401 is configured to acquire a first voice signal in response to the detection of a keyword, wherein the first voice signal includes the keyword;

[0139] The human face detection unit 402 is used to acquire a first human face signal in response to triggering pedestrian detection;

[0140] The first obtaining unit 403 is used to obtain voice features and a first identity recognition result based on the first voice signal;

[0141] The second obtaining unit 404 is used to obtain facial features and a second identity recognition result based on the first facial signal;

[0142] The third obtaining unit 405 is used to input the voice features and the human face features into a preset multimodal fusion processor after the first identity recognition result is used to indicate successful recognition and the second identity recognition result is used to indicate successful recognition, so as to obtain fused features;

[0143] The control unit 406 is used to control the vehicle according to the control command corresponding to the fusion feature.

[0144] Optionally, the third obtaining unit 405 is specifically used for:

[0145] The speech features and the facial features are added point by point according to the fully connected layer in the preset multimodal fusion machine to obtain the fused features.

[0146] Optionally, the control unit 406 is specifically used for:

[0147] The fused features are classified to obtain classification results;

[0148] The vehicle is controlled according to the control command corresponding to the classification result.

[0149] Optionally, the vehicle control system includes a preset voice detection network, and the first obtaining unit 403 includes:

[0150] The speech feature module is used to input the first speech signal into the preset speech detection network to obtain the speech features and a first similarity, wherein the first similarity is used to characterize the similarity between the speech features and the preset voiceprint features;

[0151] The first identity determination module is used to determine the first identity recognition result based on the first similarity.

[0152] Optionally, the preset speech detection network includes an input layer, a hidden layer, and an output layer, and the speech feature module is specifically used for:

[0153] The first speech signal is input into the input layer of the preset speech detection network for preprocessing, and the second speech signal is output.

[0154] The second speech signal is input into the hidden layer for feature learning, and the speech features are output. The hidden layer is used to learn features of the second speech signal based on a preset activation function.

[0155] The speech features are input into the output layer of the preset speech detection network for feature comparison, and the first similarity is output. The output layer of the preset speech detection network is used to calculate the similarity between the speech features and the preset voiceprint features.

[0156] Optionally, the vehicle control system includes a preset video detection network, and the second obtaining unit 404 includes:

[0157] A human face feature module is used to input the first human face signal into the preset video detection network to obtain the human face features and a second similarity, wherein the second similarity is used to characterize the similarity between the human face features and the preset human face features;

[0158] The second identity determination module is used to determine the second identity recognition result based on the second similarity.

[0159] Optionally, the preset video detection network includes an input layer, a convolutional layer, a fully connected layer, and an output layer, and the portrait feature module is specifically used for:

[0160] The first portrait signal is input into the input layer of the preset video detection network for preprocessing, and the second portrait signal is output.

[0161] The second portrait signal is input into the convolutional layer for feature learning and feature extraction, and a third portrait signal is output; the convolutional layer is used to learn features from the second portrait signal based on a preset activation function;

[0162] The third human image signal is input into the fully connected layer for feature integration, and the human image features are output.

[0163] The portrait features are input into the output layer of the preset video detection network for feature comparison, and the second similarity is output. The output layer of the preset video detection network is used to calculate the similarity between the portrait features and the preset portrait features.

[0164] Optionally, the voice detection unit 401 is specifically used for:

[0165] In response to the detection of the keyword, an initial voice signal is acquired;

[0166] The initial speech signal is subjected to mean filtering to obtain the first speech signal.

[0167] It should be noted that the specific implementation method and the technical effects achieved by the device 400 can be found in [reference needed]. Figure 1 or Figure 3 The relevant descriptions in the methods shown.

[0168] Furthermore, embodiments of this application also provide a vehicle, including a vehicle control device based on multimodal detection, wherein the vehicle control device based on multimodal detection may be, for example, the one described above. Figure 4 The vehicle control device 400 based on multimodal detection.

[0169] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0170] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the objectives of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0171] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A vehicle control method based on multimodal detection, characterized in that, A vehicle control system applied to a vehicle, the method comprising: In response to the detection of a keyword, a first voice signal is acquired, wherein the first voice signal includes the keyword; In response to triggering pedestrian detection, the first human image signal is acquired; Based on the first speech signal, speech features and a first identity recognition result are obtained; Based on the first facial image signal, facial features and second identity recognition results are obtained; After the first identity recognition result is used to indicate successful recognition and the second identity recognition result is used to indicate successful recognition, the voice features and the facial features are added point by point according to the fully connected layer in the preset multimodal fusion unit to obtain the fused features; The fused features are classified to obtain classification results; The vehicle is controlled according to the control command corresponding to the classification result.

2. The method according to claim 1, characterized in that, The vehicle control system includes a preset voice detection network, and the step of obtaining voice features and a first identity recognition result based on the first voice signal includes: The first speech signal is input into the preset speech detection network to obtain the speech features and the first similarity, wherein the first similarity is used to characterize the similarity between the speech features and the preset voiceprint features; The first identity recognition result is determined based on the first similarity.

3. The method according to claim 2, characterized in that, The preset speech detection network includes an input layer, a hidden layer, and an output layer. The step of inputting the first speech signal into the preset speech detection network to obtain the speech features and the first similarity score includes: The first speech signal is input into the input layer of the preset speech detection network for preprocessing, and the second speech signal is output. The second speech signal is input into the hidden layer for feature learning, and the speech features are output. The hidden layer is used to learn features of the second speech signal based on a preset activation function. The speech features are input into the output layer of the preset speech detection network for feature comparison, and the first similarity is output. The output layer of the preset speech detection network is used to calculate the similarity between the speech features and the preset voiceprint features.

4. The method according to claim 1, characterized in that, The vehicle control system includes a preset video detection network, and the step of obtaining facial features and a second identity recognition result based on the first facial signal includes: The first portrait signal is input into the preset video detection network to obtain the portrait features and the second similarity, wherein the second similarity is used to characterize the similarity between the portrait features and the preset portrait features; The second identity recognition result is determined based on the second similarity.

5. The method according to claim 4, characterized in that, The preset video detection network includes an input layer, a convolutional layer, a fully connected layer, and an output layer. The step of inputting the first portrait signal into the preset video detection network for feature extraction to obtain the portrait features includes: The first portrait signal is input into the input layer of the preset video detection network for preprocessing, and the second portrait signal is output. The second portrait signal is input into the convolutional layer for feature learning and feature extraction, and a third portrait signal is output; the convolutional layer is used to learn features from the second portrait signal based on a preset activation function; The third human image signal is input into the fully connected layer for feature integration, and the human image features are output. The portrait features are input into the output layer of the preset video detection network for feature comparison, and the second similarity is output. The output layer of the preset video detection network is used to calculate the similarity between the portrait features and the preset portrait features.

6. The method according to claim 1, characterized in that, The step of acquiring a first voice signal in response to detecting a keyword includes: In response to the detection of the keyword, an initial voice signal is acquired; The initial speech signal is subjected to mean filtering to obtain the first speech signal.

7. A vehicle control device based on multimodal detection, characterized in that, A vehicle control system applied to a vehicle, the device comprising: A speech detection unit is configured to acquire a first speech signal in response to the detection of a keyword, wherein the first speech signal includes the keyword; A human face detection unit is used to acquire a first human face signal in response to triggering pedestrian detection. The first obtaining unit is configured to obtain speech features and a first identity recognition result based on the first speech signal; The second obtaining unit is used to obtain facial features and a second identity recognition result based on the first facial signal; The third obtaining unit is used to add the voice features and the human face features point by point according to the fully connected layer in the preset multimodal fusion unit after the first identity recognition result is used to indicate successful recognition and the second identity recognition result is used to indicate successful recognition, thereby obtaining fused features. The control unit is used to classify the fused features to obtain classification results; and to control the vehicle according to the control commands corresponding to the classification results.

8. A vehicle, characterized in that, Includes the vehicle control device based on multimodal detection as described in claim 7.

Images