Vehicle-mounted communication method and device and electronic equipment

Abstract

The embodiment of the invention relates to the technical field of vehicles, and provides a vehicle-mounted communication method and device and electronic equipment. The method comprises the following steps: acquiring a multi-modal signal of a first user, wherein the multi-modal signal comprises a first sound signal and an image signal of the first user; determining text information corresponding to the multi-modal signal based on the multi-modal signal of the first user; determining a second sound signal corresponding to the text information based on the text information; and outputting the second sound signal to the second user. Therefore, according to the technical scheme provided by the embodiment of the invention, the original voice is not output to the second user after noise reduction, but the new voice, namely the second sound signal, is synthesized to the second user, so that the interference of noise can be avoided, and the sound quality of the vehicle-mounted call voice is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle technology, and in particular to a vehicle-mounted communication method, device, and electronic equipment. Background Art

[0002] In-vehicle communications refer to communications conducted within a vehicle, such as voice calls. With the continuous advancement of vehicle technology and the increasing trend toward intelligentization, in-vehicle communications are playing an increasingly important role in enhancing the in-vehicle experience.

[0003] In-vehicle voice calls are often subject to interference from noise, such as from other passengers and wind noise from open windows, which reduces voice clarity. Therefore, existing in-vehicle communication methods typically require processing the speaker's voice, such as removing background noise and enhancing it, before transmitting the processed voice to the receiver. Summary of the Invention

[0004] Current in-vehicle communication methods have poor noise reduction effectiveness in some scenarios, such as those with directional, non-stationary interference and when the interference energy is relatively high. Furthermore, noise reduction can also lead to a loss of sound quality. Consequently, current in-vehicle communication methods offer low voice quality and a poor user experience.

[0005] In order to solve the above technical problems, the embodiments of the present disclosure provide a vehicle-mounted communication method, device and electronic equipment, which can avoid noise interference and ensure the sound quality of vehicle-mounted call voice.

[0006] A first aspect of the present disclosure provides an in-vehicle communication method, comprising: acquiring a multimodal signal from a first user, the multimodal signal including a first sound signal and an image signal of the first user; determining text information corresponding to the multimodal signal based on the multimodal signal of the first user; determining a second sound signal corresponding to the text information based on the text information; and outputting the second sound signal to a second user.

[0007] According to a second aspect of the present disclosure, a vehicle-mounted communication device is provided, which includes: a signal acquisition component, including a microphone and a camera, wherein the microphone is configured to obtain a first sound signal in a multimodal signal of a first user, and the camera is configured to obtain an image signal in the multimodal signal of the first user; one or more processors are configured to determine, based on the multimodal signal of the first user, text information corresponding to the multimodal signal, and, based on the text information, determine a second sound signal corresponding to the text information when executing instructions stored in a memory; and a communication component, including a wireless communication circuit, is configured to output the second sound signal to a second user.

[0008] According to a third aspect of the present disclosure, an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program for executing the in-vehicle communication method provided in the first aspect.

[0009] In the fourth aspect of the present disclosure, an embodiment of the present disclosure provides an electronic device, which includes: a processor; a memory for storing processor executable instructions; wherein the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the vehicle-mounted communication method provided in the first aspect.

[0010] Based on the in-vehicle communication method provided by the present disclosure, text information can be determined based on the first sound signal and image signal in the multimodal signal of the first user. In this way, determining the text information from the two dimensions of voice and sound can ensure the accuracy of the text information and avoid noise interference in the first sound signal. Thereafter, a second sound signal is determined based on the text information and sent to the second user. In this way, the second user receives the second sound signal generated based on the text information, rather than the original voice received by the microphone of the first user end, so the second sound signal does not contain the noise received by the microphone of the first user end. Through such a design, on the basis of allowing the second user to accurately know the content that the first user wants to express, the sound quality of the call voice can also be effectively guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other purposes, features, and advantages of the present disclosure will become more apparent through a more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and are not intended to limit the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 is a structural block diagram of a vehicle communication system provided by an exemplary embodiment of the present disclosure;

[0013] Figure 2 is a structural block diagram of a vehicle communication system provided by another exemplary embodiment of the present disclosure;

[0014] Figure 3 is a flowchart of a travel planning method based on an intelligent driving system provided by an exemplary embodiment of the present disclosure;

[0015] Figure 4 is a flow chart of a vehicle communication method provided by another exemplary embodiment of the present disclosure;

[0016] Figure 5 is a flow chart of a vehicle communication method provided by another exemplary embodiment of the present disclosure;

[0017] Figure 6 is a flow chart of a vehicle communication method provided by another exemplary embodiment of the present disclosure;

[0018] Figure 7 is a flow chart of a vehicle communication method provided by another exemplary embodiment of the present disclosure;

[0019] Figure 8 is a flow chart of a vehicle communication method provided by another exemplary embodiment of the present disclosure;

[0020] Figure 9 is a flow chart of a vehicle communication method provided by another exemplary embodiment of the present disclosure;

[0021] Figure 10 is a structural diagram of a vehicle-mounted communication device provided by an exemplary embodiment of the present disclosure;

[0022] Figure 11 A structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0024] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0025] Application Overview

[0026] In-car voice calls are often subject to interference from other passengers, wind noise from open windows, and other factors, which can reduce voice clarity. In noisy environments, the speaker needs to raise their voice to improve clarity, thereby compensating for the shortcomings of voice enhancement technology, such as inevitable human voice interference and residual ambient noise. However, this approach degrades the speaker's call experience and affects other passengers' ability to communicate effectively.

[0027] Existing technical solutions typically use voice enhancement technology to eliminate noise interference and thus improve the quality of in-car calls. Specifically, the speaker's voice is processed by removing background noise and enhancing the voice, and then the processed voice is sent to the receiving end.

[0028] However, current in-vehicle communication methods struggle to remove directional, non-stationary interference (such as interference from a nearby passenger talking) and when the interference energy is relatively high. Consequently, the speech heard by the receiver is tinged with noise. Furthermore, current speech enhancement technologies also suffer from a loss of sound quality when reducing noise. In summary, existing in-vehicle voice communication methods offer low voice quality, resulting in a poor user experience.

[0029] The embodiment of the present disclosure provides an in-vehicle communication method, which can collect multimodal signals such as the first sound signal and image signal of the first user, and then determine the text information from the two dimensions of voice and sound based on the multimodal signals, thereby ensuring the accuracy of the text information and avoiding noise interference in the first sound signal. Thereafter, the embodiment of the present disclosure determines the second sound signal based on the text information and sends it to the second user. In this way, the second user receives the second sound signal generated based on the text information, rather than the original voice received by the microphone of the first user end, so the second sound signal does not contain the noise received by the microphone of the first user end. Through such a design, on the basis of allowing the second user to accurately know the content that the first user wants to express, it can also effectively guarantee the sound quality of the call voice.

[0030] Exemplary Systems

[0031] Figure 1 FIG. 4 is a structural block diagram of a vehicle-mounted communication system provided by an exemplary embodiment of the present disclosure.

[0032] like Figure 1 As shown, in one embodiment, the vehicle communication system may include a collection system 10, a voice processing system 20, and a voice transmission system 30. Furthermore, the voice processing system 20 performs data transmission with the collection system 10 and the voice transmission system 30 respectively.

[0033] The acquisition system 10 may include a microphone and at least one sensor for acquiring a multimodal signal of the first user, which may specifically include a first sound signal and an image signal of the first user. The microphone is arranged in the vehicle, and may specifically be arranged in the dome light area, or may be arranged above the instrument panel or in the central control display area, etc. In some examples, multiple microphones may also be arranged in locations such as the roof to form a microphone array. The sensor may be an image sensor, such as a camera, an infrared sensor, etc. The sensor is arranged in the vehicle, and may specifically be arranged in locations such as above the instrument panel and the dome light area. The acquisition system 10 can transmit data with the voice processing system 20. For example, the acquisition system 10 sends the collected multimodal signal to the voice processing system 20.

[0034] Based on the multimodal signals collected by the acquisition system 10, the speech processing system 20 can identify the speech content that the first user intends to send to the second user, convert the speech content into text information, and then synthesize a second sound signal based on the content of the text information. The speech processing system 20 can also communicate data with the speech transmission system 30. For example, the speech processing system 20 can send the synthesized second sound signal to the speech transmission system 30.

[0035] In one implementation, the speech processing system 20 may include one or more processors 201. The processor 201 may include a general-purpose processor, such as a central processing unit (CPU) or a graphics processing unit (GPU), or may include an accelerated computing unit designed for deep learning tasks, autonomous driving tasks, etc., such as a neural processing unit (NPU).

[0036] In one implementation, the speech processing system 20 may further include one or more memories 202 , in which program instructions executable by the processor 201 may be stored. The processor 201 may load and execute the program instructions in the memory 202 to implement the functions of the speech processing system 20 .

[0037] In addition, the memory 202 can also be used to cache or store intermediate data or result data generated by the processor 201 during operation, as well as store system files, application files, data files, etc. For example, the memory 202 can store multimodal signals collected by the acquisition system 10, etc.

[0038] Exemplarily, the memory 202 may include, for example, a volatile memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.; and may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, etc.

[0039] In one implementation, the speech processing system 20 may be located inside the vehicle. In another implementation, the speech processing system 20 may be located on a server or in the cloud. In another implementation, the speech processing system 20 may be located partially inside the vehicle and partially on a server or in the cloud.

[0040] Figure 2 FIG. 4 is a structural block diagram of a vehicle-mounted communication system provided by another exemplary embodiment of the present disclosure.

[0041] like Figure 2 As shown, the text recognition module 21 of the voice processing system 20 is arranged inside the vehicle, and is used to determine the corresponding text information based on the multimodal signal collected by the acquisition system 10; the first sending module 22 of the voice processing system 20 is arranged inside the vehicle, and is used to send the text information determined by the text recognition module 21 to the server or the cloud; the first receiving module 23 of the voice processing system 20 is arranged on the server or the cloud, and is used to receive the text information sent by the first sending module 22; the voice synthesis module 24 of the voice processing system 20 is arranged on the server or the cloud, and is used to synthesize the second sound signal based on the text information received by the first receiving module 23.

[0042] In one implementation, the text recognition module 21 may include a multimodal signal interface circuit, and the text recognition module 21 and the acquisition system 10 perform data transmission through the interface circuit, and receive the multimodal signal acquired by the acquisition system 10. For example, the multimodal signal interface circuit may include a Mobile Industry Processor Interface-Camera Serial Interface (MIPI-CSI) and an Integrated Interchip Sound Bus (I 2 S) interface.

[0043] In one implementation, the text recognition module 21 may include one or more processors, which may include a general-purpose processor, such as a central processing unit (CPU), a graphics processing unit (GPU), etc., and may also include an accelerated computing unit designed for deep learning tasks, autonomous driving tasks, etc., such as a neural processing unit (NPU), etc.

[0044] In one implementation, the text recognition module 21 may further include one or more memories. The memories of the text recognition module 21 may store program instructions executable by a processor. The processor of the text recognition module 21 may load and execute the program instructions in the memories to implement the functions of the text recognition module. For example, the memory of the text recognition module 21 may store a neural network model that has text recognition functions.

[0045] In addition, the memory of the text recognition module 21 can also be used to cache or store intermediate data or result data generated by the processor of the text recognition module 21 during operation, as well as to store system files, application files, data files, etc. For example, the memory of the text recognition module 21 can store multimodal signals collected by the collection system 10, text information recognized by the text recognition module 21, etc.

[0046] In one implementation, the first sending module 22 may include a CAN bus interface, and the text information stored in the memory of the text recognition module 21 may be transmitted to the first sending module 22 via the CAN bus interface.

[0047] In one implementation, the first sending module 22 may include a wireless communication circuit, and the text information may be transmitted to the first receiving module 23 via the communication module. For example, the wireless communication circuit may be a Wi-Fi module, a 4G / 5G communication module, or the like.

[0048] In one implementation, the first receiving module 23 may include a communication circuit, and the text information may be transmitted to the first receiving module 23 via the communication circuit. For example, the communication circuit may be a Wi-Fi module, a 4G / 5G communication module, or a network interface card (NIC).

[0049] In one implementation, the first receiving module 23 may include a memory, and the text information sent by the first sending module 22 may be stored in the memory of the first receiving module 23. For example, the memory of the first receiving module 23 may adopt a distributed storage architecture.

[0050] In one implementation, the speech synthesis module 24 may include one or more processors, which may include general-purpose processors, such as a central processing unit (CPU), a graphics processing unit (GPU), etc., and may also include accelerated computing units designed for deep learning tasks, autonomous driving tasks, etc., such as a neural processing unit (NPU), etc.

[0051] In one implementation, the speech synthesis module 24 may further include one or more memories. The memories of the speech synthesis module 24 may store program instructions executable by a processor. The processor of the speech synthesis module 24 may load and execute the program instructions in the memories to implement the functions of the speech synthesis module. For example, the memory of the speech synthesis module 24 may store a neural network model that has speech synthesis functions.

[0052] In addition, the memory of the speech synthesis module 24 can also be used to cache or store intermediate data or result data generated by the processor of the speech synthesis module 24 during operation, as well as to store system files, application files, data files, etc. For example, the memory of the speech synthesis module 24 can store text information sent by the text recognition module 21, the second sound signal synthesized by the speech synthesis module 24, etc.

[0053] Please continue to combine Figure 1 The voice transmission system 30 may include a wireless communication circuit 301 for wirelessly transmitting the second voice signal synthesized by the voice processing system 20 to a receiving device at the second user end, and for receiving a voice signal (hereinafter referred to as the third voice signal) sent by the second user end, thereby enabling a two-way conversation between the first user and the second user. The wireless communication circuit may be a Bluetooth module, a Wi-Fi module, a 4G / 5G communication module, or the like.

[0054] Exemplary Methods

[0055] Figure 3 This is a flow chart of a travel planning method based on an intelligent driving system provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 3 As shown, the method includes the following steps:

[0056] Step 100: Acquire a multimodal signal of a first user, where the multimodal signal includes a first sound signal and an image signal of the first user.

[0057] In step 100, the first user is a driver or passenger in a current vehicle, engaged in a voice call with a second user; the second user is a remote user outside the current vehicle, engaged in a voice call with the first user. The second user may be in another vehicle, communicating with the first user through an in-vehicle communication system installed therein; or the second user may be in another scenario, communicating with the first user through a mobile terminal such as a mobile phone. The in-vehicle communication system may acquire multimodal signals from the first user via the acquisition system 10. In one example, the microphone of the acquisition system 10 may acquire a first sound signal from the first user. In addition to the first user's voice, the first sound signal may also include noise from other people in the vehicle, ambient noise, and other factors. In one example, the camera of the acquisition system 10 may acquire an image signal from the first user. The image signal may be a photo or video, with the first user as the subject. For example, the image signal may be a photo that includes the first user's facial expressions, lip movements, and other information.

[0058] Step 200: Determine text information corresponding to the multimodal signal based on the multimodal signal of the first user.

[0059] In step 200, based on the multimodal signal of the first user, the first sound signal and the image signal are comprehensively analyzed to determine the corresponding text signal. For example, the speech content of the first user's speech in the first sound signal can be identified by using Automatic Speech Recognition (ASR) technology and a Transformer / Conformer model, such as a Whisper model. The action of the first user in the image signal can be identified by using action recognition (Action Recognition) technology and a Convolutional Neural Networks (CNN) model, such as a ResNet model. Then, the speech content of the first user's speech in the first sound signal and the action of the first user in the image signal are comprehensively analyzed to determine the content that the first user wants to send to the second user and convert it into corresponding text information.

[0060] In some examples, the text signal is determined by a Transformer structure model. Before determining the text signal, the Transformer structure model can be trained in advance. Specifically, a conversation video containing character dialogue can be obtained and input into the Transformer structure model as a training set. Using cross entropy as the loss function, the model output is compared with the actual text content corresponding to the conversation video. The model is optimized with the goal of complete text reproduction or text semantic proximity to obtain a trained Transformer structure model. Before inputting the training set into the Transformer structure model, the training set can also be preprocessed, such as adding noise to the conversation video, or removing some images in the conversation video, or removing some speech in the conversation video. This can improve the generalization ability of the Transformer structure model, thereby ensuring the accuracy of the trained Transformer structure model in determining text information.

[0061] Step 300: Determine a second sound signal corresponding to the text information based on the text information.

[0062] In step 300, the text information in text format is converted into a second sound signal in voice format. For example, text-to-speech (TTS) technology can be used to convert the text information into natural and fluent voice using models such as Tacotron and FastSpeech to obtain the second sound signal.

[0063] In one implementation, steps 200 and / or 300 can be performed on the server or in the cloud. For example, in the case where step 200 is performed within the vehicle and step 300 is performed in the cloud, the text recognition module 21 within the vehicle determines text information based on the multimodal signal and transmits the text information to the cloud. Subsequently, the speech synthesis module 24 in the cloud synthesizes a second sound signal based on the text information and transmits the synthesized second sound signal to the second user. This eliminates the need for the in-vehicle communication system to transmit the first sound signal to the second user in real time, thus avoiding the problem of discontinuous voice calls caused by network instability in traditional voice call methods. In other words, even if the first user's network is unstable or the network bandwidth is insufficient, the second user can still receive a coherent second sound signal, thereby improving the second user's call experience. Furthermore, since the speech processing system 20 does not transmit audio data to the cloud, but only sends text information to the cloud, the network bandwidth requirements between the vehicle and the cloud can be reduced.

[0064] Taking the example of steps 200 and 300 both being performed in the cloud, the acquisition system 10 is located within the vehicle, and the voice processing system is located in the cloud. The acquisition system 10 within the vehicle can collect multimodal signals using devices such as microphones and cameras, and then transmit them to the voice processing system 20 in the cloud. The voice processing system 20 determines the text information and the corresponding second sound signal based on the multimodal information. In this example, considering that low-sampling-rate audio can be perceived as noticeably distorted and unnatural by the human ear, traditional voice calls require a higher sampling frequency. Since the primary components of human voices are within 4 kHz, extracting only audio features within 4 kHz can meet basic voice recognition requirements. In other words, the sampling frequency requirements for voice recognition are lower than those for voice calls. Therefore, the acquisition system 10 can use a lower sampling frequency when collecting the first sound signal. Thus, compared to the in-vehicle communication system transmitting high-frequency sound signals to the second user, in this example, the acquisition system 10 within the vehicle transmits the first sound signal with a lower sampling frequency to the voice processing system 20 in the cloud, reducing data transmission volume and conserving network resources.

[0065] Step 400: output a second sound signal to a second user.

[0066] In step 400, the in-vehicle communication system transmits the synthesized second voice signal to the second user via the voice transmission system 30. In this way, the second user receives a clear, noise-free voice signal synthesized based on the text information.

[0067] It can be seen from the above technical solutions that the method provided by the embodiment of the present disclosure can determine the text information based on the first sound signal and the image signal in the multimodal signal of the first user. In this way, determining the text information from the two dimensions of voice and sound can ensure the accuracy of the text information and avoid noise interference in the first sound signal. Thereafter, the second sound signal is determined based on the text information and sent to the second user. In this way, the second user receives the second sound signal generated based on the text information, rather than the original voice received by the microphone on the first user side. Therefore, the second sound signal does not contain the noise received by the microphone on the first user side. Through such a design, on the basis of enabling the second user to accurately know the content that the first user wants to express, the sound quality of the call voice can also be effectively guaranteed.

[0068] Furthermore, since the second user doesn't listen to the original voice received by the first user's microphone, the first user doesn't need to raise their voice in noisy environments to ensure the second user can clearly hear what they are saying. This improves the first user's conversation experience while also preventing others from hearing their conversation or disturbing other passengers, thus enhancing the privacy of the voice call.

[0069] Figure 4 is a flowchart of a vehicle communication method provided by another exemplary embodiment of the present disclosure.

[0070] like Figure 4 As shown in the above Figure 3 Based on the embodiment shown, step 200 may include the following steps:

[0071] Step 210: Determine audio features and image features of the first user based on the first sound signal and the image signal in the multimodal signal, where the image features include facial features and / or hand features of the first user.

[0072] In step 210, the audio features and image features of the first user can be determined based on the first sound signal and image signal in the multimodal signal. Specifically, the first sound signal can be subjected to frame windowing processing to extract audio features. The audio features may include Mel Frequency Cepstral Coefficients (MFCC) features, Mel Filter Bank Energy Features (FBANK) features, Per-Channel Energy Normalization (PCEN) features, etc. of the first sound signal, which are used to reflect the voice content, speaking speed, intonation and other information of the first user.

[0073] Image features corresponding to key points on the first user's body can be extracted from the image signal. Key points are locations that reflect the first user's expressions and movements, and can include facial key points and finger key points, specifically eyes, lips, and finger joints. Image features can include facial features of the first user, such as head movement features such as nodding or shaking the head, facial expression features such as frowning or smiling, eye movement features, lip movement features, etc.; image features can also include hand features of the first user, such as hand gesture features.

[0074] Step 220: Determine a first semantic meaning corresponding to the multimodal signal based on the facial features and / or hand features, and the audio features.

[0075] In step 220, the first semantics corresponding to the multimodal signal can be determined based on facial features and audio features, or based on hand features and audio features, or based on facial features, hand features, and audio features.

[0076] In one example, image features (such as facial features and / or hand features) may be used to assist in determining the first semantics of the first user's speech, thereby improving the accuracy of the text information.

[0077] For example, if the facial features include a head shake, it can be determined that the first user's intention is negative, and the content of the first semantic meaning can be considered to be negative. In this case, if the first user's audio features indicate "yes", it can be assumed that due to factors such as noise interference, the audio features in the first sound signal do not accurately express the first user's true intention. Therefore, the first semantic meaning can be modified based on the head shake in the facial features to indicate "no".

[0078] For example, if the first sound signal is missing some content, but the image signal shows that the first user is still speaking, for example, the first user's lips are still moving, then lip reading recognition can be performed based on the lip movement features in the facial features, and the voice content to be expressed by the first user can be completed based on the recognized lip reading, thereby obtaining the completed and complete first semantics.

[0079] Step 230: Generate text information based on the first semantics.

[0080] In step 230, after obtaining the first semantics, text information can be generated based on the first semantics. The text information can be words, phrases, or sentences expressed in natural language, and is used to express the content that the first user wants to convey to the second user. In one example, the text information can be generated using a natural language generation model.

[0081] As can be seen from the above technical solutions, the method provided by the embodiments of the present disclosure can obtain the audio and image features of the first user based on the first sound signal and image signal, respectively, and then comprehensively analyze the audio and image features of the first user to determine the content that the first user intended to express, thereby obtaining accurate first semantics and text information. Therefore, compared to methods that obtain text information based solely on audio features, the embodiments of the present disclosure can improve the accuracy of text information.

[0082] Figure 5 is a flowchart of a vehicle communication method provided by another exemplary embodiment of the present disclosure.

[0083] like Figure 5 As shown in the above Figure 4 Based on the embodiment shown, step 230 may include the following steps:

[0084] Step 231: Determine first text reference information.

[0085] In step 231, when generating text information based on the first semantics, first text reference information may be pre-determined to ensure the accuracy and naturalness of the text information. The first text reference information is information used to assist in generating the text information. In one example, the first text reference information may include information related to the first user's language expression characteristics, such as the first user's historical conversation records, idioms, and personalized text templates preset by the first user. In another example, the first text reference information may also include information related to the current conversation context, such as contextual information.

[0086] Step 232: Generate text information based on the first semantics and the first text reference information.

[0087] In step 232, after the first text reference information is obtained, the first text reference information may be combined with the first semantics to generate text information corresponding to the first semantics.

[0088] It can be seen from the above technical solution that the method provided by the embodiment of the present disclosure determines the first text reference information and introduces the first text reference information in the process of generating text information. In this way, the generated text information is closer to the language expression habits of the first user and conforms to the current dialogue context, thereby improving the accuracy and naturalness of the text information.

[0089] Figure 6 is a flowchart of a vehicle communication method provided by another exemplary embodiment of the present disclosure.

[0090] like Figure 6 As shown in the above Figure 5 Based on the illustrated embodiment, step 231 may include the following:

[0091] Step 2311: Determine context information in the first text reference information based on the current conversation information between the first user and the second user.

[0092] In step 2311, the first text reference information includes contextual information about the current conversation between the first and second users. Based on the current conversation between the first and second users, the contextual information can be acquired in real time. The in-vehicle communication system can acquire the portion of the current conversation information output by the first user via the acquisition system 10 and the portion of the current conversation information output by the second user via the voice transmission system 30, such as the third sound signal in the aforementioned embodiment.

[0093] The context information may include the topic discussed by the first user and the second user in the current conversation, as well as the details mentioned. For example, if the first user mentions "rain in the forecast" in the current conversation, the first user's intention can be determined based on the previously discussed topic "weekend picnic" in the context information. Based on the first semantic meaning of "rain in the forecast" and "weekend picnic" in the context information, a text message "The weather forecast predicts rain this weekend. Do you want to move the picnic to another day?" is generated that more accurately expresses the first user's intention.

[0094] As can be seen from the above technical solution, the method provided in the embodiments of the present disclosure determines contextual information in the first text reference information based on the current conversation between the first and second users. This contextual information can be used as a reference when generating text information, ensuring that the generated text information matches the current conversation context and improving the coherence of the conversation between the first and second users.

[0095] Step 2312: Determine language habit information in the first text reference information based on the first historical voice information of the first user.

[0096] In step 2312, the first text reference information includes the first user's language habit information. At this point, the first user's language habit information, such as the first user's catchphrases and idiomatic expressions, can be obtained by analyzing the first user's historical voice information. Subsequently, the first user's language habit information can be used as a reference to assist in generating text information. For example, if the first user's idiomatic expression for affirmative intent is "OK," then when generating the text information, based on the first user's language habit information, affirmative content in the text information, such as "OK," can be replaced with "OK."

[0097] As can be seen from the above technical solution, the method provided in the embodiments of the present disclosure determines language habit information in the first text reference information based on the first historical voice information of the first user. This allows the language habit information to be used as a reference when generating text information, making the generated text information more consistent with the first user's language expression habits. Consequently, when the second user receives the second sound signal generated based on the text information, it will feel more familiar and natural.

[0098] In some examples, step 231 may include only one of step 2311 and step 2312, and the specific implementation method and technical effect are not repeated here.

[0099] Figure 7 is a flowchart of a vehicle communication method provided by another exemplary embodiment of the present disclosure.

[0100] like Figure 7 As shown in the above Figure 3 Based on the embodiment shown, step 300 may include the following steps:

[0101] Step 310: Determine sound adjustment information.

[0102] In step 310, when synthesizing the second sound signal based on the text information, sound adjustment information may be pre-determined to improve the fluency and naturalness of the second sound signal and ensure that the second sound signal more accurately expresses the first user's intention. The sound adjustment information is information used to assist in generating the second sound signal and can be used to adjust sound parameters of the second sound signal, such as speech rate and intonation.

[0103] Step 320: Determine a second sound signal based on the text information and the sound adjustment information.

[0104] In step 320, after the sound adjustment information is obtained, the sound adjustment information may be combined with the text information to generate a second sound signal corresponding to the text information.

[0105] It can be seen from the above technical solutions that the method provided by the embodiment of the present disclosure determines the sound adjustment information and introduces the sound adjustment information in the process of generating the second sound signal. In this way, the second sound signal can be made smoother and more natural, which is conducive to improving the experience of the second user.

[0106] Figure 8 is a flowchart of a vehicle communication method provided by another exemplary embodiment of the present disclosure.

[0107] like Figure 8 As shown in the above Figure 7 Based on the illustrated embodiment, step 310 may include the following steps:

[0108] Step 311: Determine timbre information in the sound adjustment information based on the first historical voice information of the first user.

[0109] In step 311, the sound adjustment information includes the first user's timbre information. In this case, the first user's timbre characteristics, such as the average fundamental frequency, fundamental frequency variation range, time-domain characteristics of fundamental frequency variation over time, formant position, and formant width, can be extracted from the first historical voice information of the first user and used as the timbre information in the sound adjustment information. Subsequently, the first user's timbre information can be used as a reference to adjust the timbre characteristics of the synthesized second sound signal based on the timbre information, so that the timbre of the second sound signal closely matches the timbre of the first user.

[0110] In one example, a first user can register their voice before a voice call. Specifically, the in-vehicle communication system receives the first user's first historical voice information, processes it within the vehicle or in the cloud, and obtains the first user's voice characteristics as voice information. This allows the system to synthesize the second sound signal based solely on the first user's voice registration record.

[0111] As can be seen from the above technical solution, the method provided in the embodiments of the present disclosure determines the timbre information in the sound adjustment information based on the first historical voice information of the first user. This timbre information can be used as a reference for generating the second sound signal, making the generated second sound signal more closely resemble the timbre of the first user. This can enhance the second user's sense of realism and immersion during the voice call.

[0112] Step 312: Determine language variant information in the sound adjustment information based on the language variant setting instruction input by the first user, the identity information of the second user, and at least one of the second historical voice information of the second user.

[0113] In step 312, the voice adjustment information includes language variant information, which may include language information and dialect information. The language variant information may be specified by the first user or determined by the in-vehicle communication system based on the identity information of the second user and the second historical voice.

[0114] In one example, the language variant information can be determined by a language variant setting instruction input by the first user. For example, if the first user inputs "Sichuan dialect," the language variant information can be determined to be the Sichuan dialect. Based on this, when synthesizing the second sound signal, the pronunciation characteristics of the second sound signal can be adjusted so that the second sound signal appears in the form of the Sichuan dialect.

[0115] In another example, the language variant information can be determined based on the second user's identity information. The second user's identity information can be input by the first user or obtained from an application such as a contact list. The second user's identity information includes nationality and / or region information. For example, if the second user's identity information is "British," the language variant information can be determined to be English. Based on this, when synthesizing the second sound signal, the text information can first be translated into English, and then the second sound signal can be synthesized based on the converted text information.

[0116] In another example, language variant information can be determined based on the second user's second historical voice information. This second historical voice information can be voice recordings from a previous voice call between the second user and the first user, or it can be voice recordings from the current conversation between the second user and the first user. Specifically, the in-vehicle call system can analyze the second user's second historical voice information to determine the second user's commonly used language and / or dialect, which will be used as the language variant information in the voice adjustment information.

[0117] In one implementation, after determining the language variant information, whether to enable the language variant information can be determined based on the first user's identity information. The first user's identity information includes nationality information and / or region information. For example, if the language variant information is English and the first user's identity information is British, the language variant information may not be enabled. This eliminates the need to perform language and dialect adjustments when synthesizing the second sound signal, thereby improving the efficiency of synthesizing the second sound signal.

[0118] As can be seen from the above technical solution, the method provided in the embodiments of the present disclosure determines the language variant information in the sound adjustment information based on the language variant setting instruction input by the first user, the identity information of the second user, and the second historical voice information of the second user. It then generates a second sound signal based on the language and dialect information indicated by the language variant information. In this way, the first user can use the language and pronunciation they are proficient in without any deliberate adjustments to achieve dialect or cross-language conversation with the second user, thereby enhancing the communication experience between the first and second users.

[0119] Step 313: Determine the emotion information in the sound adjustment information based on the emotion setting instruction input by the first user.

[0120] In step 313, the sound adjustment information includes emotional information. The emotional information may include the direction of the emotional adjustment, specifically, enhancement or attenuation, for enhancing or attenuating the emotion in the first sound signal, respectively. Furthermore, the emotional information may include the type of emotion, such as happy, angry, or sad.

[0121] The emotion information may be determined by an emotion setting instruction input by the first user.

[0122] In one example, the emotion setting instruction includes an emotion adjustment direction. In this case, the emotion setting instruction can be parsed to obtain the emotion adjustment direction, and the emotion type can be determined based on the first sound signal in response to the emotion setting instruction. For example, if the first user inputs the emotion setting instruction "reduce emotion", the emotion type expressed by the first sound signal output by the first user can be determined based on the first sound signal, for example, the emotion type expressed by the first sound signal can be determined to be anger. In this case, the emotional information can be determined to be "reduce anger". Therefore, when generating the second sound signal, the average pitch, pitch fluctuation range, peak volume, syllable interval and other parameters can be adjusted so that the synthesized second sound signal does not reflect anger, thereby avoiding emotional opposition and conflict with the second user.

[0123] In one example, the emotion setting instruction includes an emotion adjustment direction and an emotion type. In this case, the emotion setting instruction can be parsed to obtain the emotion adjustment direction and emotion type. This eliminates the need to analyze the first sound signal to obtain the emotion type, thereby improving the accuracy of emotion type determination and the efficiency of emotion adjustment.

[0124] In another example, the emotion setting instruction may further include an emotion adjustment range. In this case, the emotion setting instruction may be parsed to obtain emotion information including the emotion adjustment range. The emotion adjustment range may be set in numerical form, such as 30%, 50%, 80%, etc., or in a level form, such as mild, moderate, strong, etc.

[0125] As can be seen from the above technical solution, the method provided in the embodiments of the present disclosure determines the emotional information in the sound adjustment information based on the emotion setting instruction input by the first user. In this way, the emotional information can be used as a reference for generating the second sound signal, and one or more emotions can be enhanced or weakened based on the needs of the first user, thereby avoiding emotional confrontation between the first and second users and achieving better communication results.

[0126] Step 314 : Determine the background sound information in the sound adjustment information based on the background sound setting instruction input by the first user and / or the vehicle environment information.

[0127] In step 314, the sound adjustment information includes background sound information, wherein the background sound information may include environmental background sound, pure music background sound, etc. For example, the environmental background sound may include office background sound of human voices, background sound of waves, background sound of wind, etc., and the pure music background sound may include pure music of various styles.

[0128] The background sound information may be specified by the first user or determined by the in-vehicle communication system based on vehicle environment information. In one example, the background sound information may be determined by a background sound setting instruction input by the first user. For example, if the first user inputs the background sound setting instruction "ocean wave background sound," the background sound information may be determined to be ocean wave sound. Based on this, when synthesizing the second sound signal, the sound signal corresponding to the ocean wave background sound may be superimposed on the sound signal synthesized based on the text information, thereby generating the second sound signal containing the background sound.

[0129] It should be noted that, in order to ensure the voice clarity in the second sound signal so that the second user can clearly hear what the first user wants to express, a background volume threshold can be set and the volume of the background sound can be controlled to be lower than the background volume threshold.

[0130] As can be seen from the above technical solution, the method provided by the embodiment of the present disclosure can determine the background sound information in the sound adjustment information based on the background sound setting instruction input by the first user. In this way, when the first user does not want to expose the current environment, he or she can specify the background sound information through the background sound setting instruction, so that the sound signal of the corresponding background sound is superimposed when synthesizing the second sound signal. At this time, the second user receives the second sound signal containing the background sound and cannot determine the current environment of the first user based on the second sound signal. Through such a design, the real environment of the first user can be hidden and the privacy of the first user can be protected.

[0131] In another example, the in-car communication system can collect vehicle environment information through an onboard camera and then determine background sound information based on the vehicle environment information. For example, the onboard camera first captures an image of the vehicle's exterior environment, then processes the image using a pattern recognition algorithm to extract image features. If raindrop features are extracted, the vehicle environment information can be determined to be rainy, and the background sound information can be determined to be the sound of rain corresponding to a rainy environment.

[0132] As can be seen from the above technical solution, the method provided by the embodiments of the present disclosure can determine the background sound information in the sound adjustment information based on the vehicle environment information. In this way, when the second user receives the second sound signal containing the background sound, they can determine the current environment of the first user based on the vehicle environment information, thereby enhancing the second user's sense of immersion during the call with the first user.

[0133] In another example, the background sound setting instruction input by the first user and the vehicle environment information can be combined to obtain the final background sound information. For example, if the first user inputs "pure music background sound A" and the on-board camera captures an image of rain outside the car, the background sound information can be determined to be pure music A and rain sounds. Based on this, the sound signals corresponding to pure music A and rain sounds can be superimposed on the sound signal synthesized based on the text information to obtain a second sound signal containing background sound. Please refer to the aforementioned example for its technical effect, which will not be repeated here.

[0134] In some examples, step 310 may only include one or more of steps 311 to 314, and the specific implementation methods and technical effects are not repeated here.

[0135] Figure 9 is a flowchart of a vehicle communication method provided by another exemplary embodiment of the present disclosure.

[0136] like Figure 9 As shown in the above Figure 3 Based on the embodiment shown, the following steps may be further included before step 300:

[0137] Step 500: Determine second text reference information based on the emotion setting instruction input by the first user.

[0138] In step 500, in order to more accurately enhance the emotion that the first user wishes to convey using the second sound signal, or to conceal the emotion that the first user does not wish to convey using the second sound signal, before generating the second sound signal based on the text message, a second text message may be pre-determined based on the emotion setting instruction input by the first user. The second text message is used to adjust the text message so that the text message enhances the emotion that the first user wishes to convey, or conceals the emotion that the first user does not wish to convey.

[0139] In one example, the second text information may be obtained based on a text library that matches the emotion type. For example, the emotion type may be obtained by parsing the emotion setting instruction or by obtaining the emotion type based on the first sound signal, and then a text library that matches the emotion type may be determined. The text library may include emotion text corresponding to the emotion type, specifically words, phrases, etc. corresponding to the emotion type.

[0140] In one example, if the emotion setting instruction specifies "enhancement" for emotion adjustment, a text library matching the emotion type may be determined, and the emotion text in the text library may be determined as the second text information. For example, if the emotion setting instruction specifies "enhancement of happiness," emotion text in the text library matching the happiness emotion, such as "great," "haha," etc., may be determined as the second text information. The emotion text in the text library may be updated in real time based on usage records or feedback from the first user or other users, using methods such as machine learning.

[0141] In one example, if the emotion adjustment direction in the emotion setting instruction is weakening, a text library matching the emotion type may be determined, and a text segment in the text information that is identical or similar to the emotion text in the text library may be determined as the second text information.

[0142] Step 600: Adjust the text information based on the second text reference information.

[0143] In step 600, after obtaining the second text reference information, the text information can be adjusted based on the second text reference information. Specifically, the text information can be adjusted by adding content to the text information, replacing part of the text information, deleting part of the text information, etc.

[0144] In one example, if the emotion setting instruction specifies that the emotion adjustment direction be enhanced, adjusting the text information may include adding a second text reference or a sentence generated based on the second text reference to an appropriate location in the text information, replacing words or phrases with a lower emotional intensity in the text information with the second text reference, etc. Emotional intensity is a quantitative indicator used to quantify the intensity of the emotion expressed by a word or phrase. For example, if the emotion setting instruction is to "enhance happy emotions," "haha" may be added to an appropriate location in the text information, and words with lower emotional intensity, such as "okay" or "not bad," may be replaced with "great."

[0145] In one example, if the emotion adjustment direction in the emotion setting instruction is to weaken, adjusting the text message may include deleting the second text reference information in the text message, such as deleting the text message "Do you have any sense of time?". Adjusting the text message may also include replacing the second text reference information in the text message with a word or phrase with lower emotional intensity, such as replacing "hate" with "disagree." Adjusting the text message may also include rewriting the second text reference information to a text segment with lower emotional intensity. For example, the text message "You are a whole hour late!" can be rewritten as "You are an hour later than the agreed time."

[0146] As can be seen from the above technical solutions, the method provided by the embodiment of the present disclosure, by determining the second text reference information and adjusting the text information based on the second text reference information, can make the text information better express the emotions that the first user wants to convey, or hide the emotions that the first user does not want to convey. In this way, based on the adjusted text information, a second sound signal is synthesized and sent to the second user. The second user can perceive the emotions that the first user wants to convey more strongly, and cannot perceive the emotions that the first user does not want to convey. Therefore, it can ensure that the call communication is smoother, which is conducive to improving the call experience between the first user and the second user.

[0147] The solutions involved in the present disclosure are not limited to the above-mentioned embodiments.

[0148] It should be understood that the order of execution of the steps in the above embodiments does not necessarily imply a specific order of execution. The order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. For example, the steps described in the above embodiments can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved. The present disclosure does not impose any restrictions on this.

[0149] Exemplary devices

[0150] The above describes the vehicle-mounted communication method provided by the embodiments of the present disclosure. It is understood that, in order to implement the various functions of the vehicle-mounted communication method, the vehicle-mounted communication system may include corresponding hardware and software for implementing the hardware functions.

[0151] Those skilled in the art should readily appreciate that, in conjunction with the steps of an in-vehicle communication method described in each embodiment of the present disclosure, each embodiment of the present disclosure can be implemented in the form of hardware or in a combination of software-driven hardware. Whether a function is executed in hardware or software-driven hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0152] Figure 10 2 is a schematic structural diagram of a vehicle-mounted communication device provided by an exemplary embodiment of the present disclosure.

[0153] like Figure 10 As shown, in one embodiment, the vehicle-mounted communication device 1000 includes: a signal acquisition component 1010 , one or more processors 1020 , and a communication component 1030 .

[0154] The signal acquisition component 1010 includes a microphone and a camera. The microphone is configured to acquire a first sound signal in the multimodal signal of the first user, and the camera is configured to acquire an image signal in the multimodal signal of the first user.

[0155] The processor 1020 is configured to execute instructions stored in the memory to determine text information corresponding to the multimodal signal based on the multimodal signal of the first user, and to determine a second sound signal corresponding to the text information based on the text information.

[0156] The communication component 1030 includes a wireless communication circuit, such as a Bluetooth module, a Wi-Fi module, a 4G / 5G communication module, etc., and is configured to output the second sound signal to the second user.

[0157] In one embodiment, the processor 1020 is configured to execute instructions stored in the memory to perform the following operations: determine the audio features and image features of the first user based on the first sound signal and image signal in the multimodal signal, the image features including the facial features and / or hand features of the first user; determine the first semantics corresponding to the multimodal signal based on the facial features and / or hand features, and the audio features; and generate text information based on the first semantics.

[0158] In one embodiment, the processor 1020 is configured to execute instructions stored in the memory to perform the following operations: determining first text reference information; and generating text information based on the first semantic meaning and the first text reference information.

[0159] In one embodiment, the processor 1020 is configured to execute instructions stored in the memory to perform the following operations: determine context information in the first text reference information based on current conversation information between the first user and the second user; and / or determine language habit information in the first text reference information based on the first historical voice information of the first user.

[0160] In one embodiment, the processor 1020 is configured to execute instructions stored in the memory to perform the following operations: determine sound adjustment information for determining a second sound signal based on the text information and the sound adjustment information.

[0161] In one embodiment, the processor 1020 is configured to execute instructions stored in the memory to perform the following operations: determine the timbre information in the sound adjustment information based on the first historical voice information of the first user; and / or determine the language variant information in the sound adjustment information based on the language variant setting instruction input by the first user, the identity information of the second user, and at least one of the second historical voice information of the second user; and / or determine the emotion information in the sound adjustment information based on the emotion setting instruction input by the first user; and / or determine the background sound information in the sound adjustment information based on the background sound setting instruction input by the first user and / or the vehicle environment information.

[0162] In one embodiment, the processor 1020 is configured to execute instructions stored in the memory to perform the following operations: determining second text reference information based on the emotion setting instruction input by the first user; and adjusting the text information based on the second text reference information.

[0163] Exemplary electronic devices

[0164] Figure 11 A structural diagram of an electronic device provided in an embodiment of the present disclosure includes at least one processor 1110 and a memory 1120.

[0165] The processor 1110 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1100 to perform desired functions.

[0166] The memory 1120 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1110 may execute the one or more computer program instructions to implement the vehicle communication method and / or other desired functions of the various embodiments of the present disclosure described above.

[0167] In one example, the electronic device 1100 may further include an input device 1130 and an output device 1140 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0168] The input device 1130 may also include, for example, a keyboard, a mouse, and the like.

[0169] The output device 1140 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0170] Of course, to simplify, Figure 11 Only some of the components related to the present disclosure in the electronic device 1100 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1100 may further include any other appropriate components.

[0171] Exemplary computer program products and computer-readable storage media

[0172] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the bandwidth control method according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0173] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0174] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the bandwidth control method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0175] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, but is not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical memory device, a magnetic memory device, or any suitable combination thereof.

[0176] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0177] The block diagrams of the devices, apparatuses, equipment, and systems referred to in this disclosure are intended to be illustrative examples only and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems may be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words that mean "including but not limited to," and may be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and may be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and may be used interchangeably therewith.

[0178] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0179] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0180] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A vehicle communication method, comprising: Acquire a multimodal signal of a first user, where the multimodal signal includes a first sound signal and an image signal of the first user; Determining, based on the multimodal signal of the first user, text information corresponding to the multimodal signal; Based on the text information, determining a second sound signal corresponding to the text information; The second sound signal is output to a second user.

2. The vehicle-mounted communication method according to claim 1, wherein: The determining, based on the multimodal signal of the first user, text information corresponding to the multimodal signal includes: determining, based on the first sound signal and the image signal in the multimodal signal, audio features and image features of the first user, the image features including facial features and / or hand features of the first user; Determining a first semantic meaning corresponding to the multimodal signal based on the facial feature and / or the hand feature, and the audio feature; The text information is generated based on the first semantics.

3. The vehicle-mounted communication method according to claim 2, wherein: The generating the text information based on the first semantics includes: determining first text reference information; The text information is generated based on the first semantics and the first text reference information.

4. The vehicle-mounted communication method according to claim 3, wherein: The determining of the first text reference information includes: determining context information in the first text reference information based on current conversation information between the first user and the second user; and / or, Based on the first historical voice information of the first user, language habit information in the first text reference information is determined.

5. The vehicle-mounted communication method according to claim 1, wherein: The determining, based on the text information, a second sound signal corresponding to the text information includes: Determine sound adjustment information; The second sound signal is determined based on the text information and the sound adjustment information.

6. The vehicle-mounted communication method according to claim 5, wherein: The determining of the sound adjustment information includes: determining the timbre information in the sound adjustment information based on the first historical voice information of the first user; and / or, determining the language variant information in the sound adjustment information based on at least one of the language variant setting instruction input by the first user, the identity information of the second user, and the second historical voice information of the second user; and / or, determining the emotion information in the sound adjustment information based on the emotion setting instruction input by the first user; and / or, The background sound information in the sound adjustment information is determined based on the background sound setting instruction input by the first user and / or the vehicle environment information.

7. The vehicle-mounted communication method according to claim 1, wherein: Before determining the second sound signal corresponding to the text information based on the text information, the method further includes: determining second text reference information based on the emotion setting instruction input by the first user; The text information is adjusted based on the second text reference information.

8. A vehicle-mounted communication device, comprising: a signal acquisition component, comprising a microphone and a camera, wherein the microphone is configured to acquire a first sound signal in a multimodal signal of a first user, and the camera is configured to acquire an image signal in the multimodal signal of the first user; One or more processors configured to, when executing instructions stored in the memory, determine, based on the multimodal signal of the first user, text information corresponding to the multimodal signal, and, based on the text information, determine a second sound signal corresponding to the text information; The communication component, including wireless communication circuitry, is configured to output the second sound signal to a second user.

9. An electronic device comprising: One or more processors, and a memory; the memory stores computer instructions; the computer instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7. 10 . A computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the processor is caused to perform the method according to claim 1 .

Images