A vehicle communication method, system, computer device and readable storage medium

Abstract

The present application provides a vehicle-mounted communication method, system, computer device and readable storage medium, wherein the vehicle-mounted processor collects the sound inside the vehicle through a microphone array, and collects the driver's facial image through a vehicle-mounted camera; the vehicle-mounted processor determines the sound source position based on the position and acquisition time of each microphone group; the microphone array is adjusted based on the sound source position and non-sound source position to collect the sound inside the vehicle, and the vehicle-mounted camera is controlled to collect the sound source position image in real time; the cloud server identifies the image data to obtain the key points of the face to determine whether the driver is at the sound source position; if so, the cloud server performs voice recognition on the audio data and corrects it to obtain text data; the user terminal plays the audio data to the user and displays the text data to the user. The above method is used to achieve clear and effective communication with the outside world, while avoiding driving safety hazards and improving the safety of vehicle-mounted communications.

Description

Technical Field

[0001] The present invention relates to the technical field of smart cabins, and in particular to a vehicle-mounted communication method, system, computer equipment and readable storage medium. Background Art

[0002] Nowadays, with the improvement of people's living standards, cars have entered more and more families and gradually become a means of transportation for the general public and families. At the same time, fast-paced life and busy work have gradually occupied people's daily lives. Even when driving to work or traveling, it is inevitable to communicate with the outside world to deal with tasks that need to be handled at work or matters that need to be handled in life. In the prior art, when a driver needs to communicate with the outside world while driving a car, he usually communicates with the outside world through a mobile terminal (such as a mobile phone, tablet computer, etc.) that he carries with him.

[0003] However, the study found that since these mobile terminals are non-fixed devices, if the driver forgets to bring them with him, he will not be able to contact the outside world while driving. Even if the driver carries a mobile device with him, due to the limited sound reception effect of the mobile device, if he wants to achieve clear and effective voice communication with the outside world, he needs to hold the mobile device and keep it placed at the head position to clearly receive the words he says, but this will seriously affect the driver's driving posture, pose a threat to driving safety, and cause driving safety hazards. However, if the mobile device is only placed randomly in the car, it is difficult to clearly and effectively collect the driver's voice in the larger environment space inside the vehicle, and thus it is impossible to achieve clear and effective communication with the outside world. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a vehicle-mounted communication method, system, computer device and readable storage medium to achieve clear and effective communication with the outside world, while avoiding driving safety hazards and improving the safety of vehicle-mounted communications.

[0005] In a first aspect, an embodiment of the present application provides a vehicle-mounted communication method, which is applied to a vehicle-mounted communication system, wherein the system includes a vehicle-mounted processor, a cloud server, and a user terminal, wherein the cloud server can communicate with the vehicle-mounted processor and the user terminal, respectively, and the method includes:

[0006] When the driver enters the target vehicle, the vehicle-mounted processor collects the sound in the target vehicle through the microphone array to obtain the first audio data, and collects the facial image of the driver through the vehicle-mounted camera to obtain the first image data, wherein the microphone array is a plurality of microphone groups arranged at different positions in the target vehicle to capture the voice signals at different positions in the target vehicle;

[0007] The vehicle-mounted processor determines the sound source position in the target vehicle based on the installation position of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data; and determines the non-sound source position in the target vehicle based on the sound source position;

[0008] The on-board processor controls the main lobe of the microphone array to point to the sound source position, and controls the null point of the microphone array to point to the non-sound source position, so as to adjust the microphone array;

[0009] The vehicle-mounted processor collects the sound in the target vehicle through the adjusted microphone array to obtain second audio data, controls the vehicle-mounted camera to collect the image at the sound source position in real time to obtain second image data, and sends the second audio data and the second image data to the cloud server;

[0010] The cloud server performs face recognition on the first image data and the second image data using a face recognition detection model, and obtains the sound source face key points contained in the first image data and the driver face key points contained in the second image data, respectively, wherein the sound source face key points and the driver face key points both correspond to standard 68 face key points;

[0011] The cloud server judges whether the person at the sound source position is the driver based on the key points of each sound source face and the key points of each driver face, so as to obtain a judgment result of whether the person at the sound source position is the driver;

[0012] If the sound source is located at the driver, the cloud server performs speech recognition on the second audio data to obtain initial text data, corrects the initial text data based on the key points of each driver's face to obtain target text data, and sends the second audio data and the target text data to the user terminal;

[0013] If the sound source is not located at the driver, the cloud server sends the second audio data to the user terminal;

[0014] The user terminal plays the second audio data to the user through a speaker, and displays the target text data to the user through a display screen.

[0015] Optionally, the vehicle-mounted processor determines the sound source position in the target vehicle based on the installation position of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data, including:

[0016] For every two microphone groups, determining a data collection time difference between the two microphone groups according to the time when the two microphone groups collect and obtain the first audio data;

[0017] Determine the similarity between each two data collection time differences based on a preset similarity algorithm;

[0018] Randomly determine a number of cluster centers, determine the cluster center with the closest similarity to the time difference between each two data collections, and assign the first audio data corresponding to the time difference between each two data collections to the cluster corresponding to the cluster center, to obtain a number of first audio data groups;

[0019] Determine the quasi-sound source position of each first audio data group based on the data collection time difference of each microphone group in each first audio data group and the setting position of each microphone group in each first audio data group in the target vehicle;

[0020] The center point of the sound source-like position of each first audio data group is determined as the sound source position in the target vehicle.

[0021] Optionally, judging whether the person at the sound source position is the driver based on each sound source face key point and each driver face key point to obtain a judgment result of whether the person at the sound source position is the driver includes:

[0022] Calculating the position offset between the sound source face key point and the driver face key point corresponding to each standard key point in the standard 68 face key points respectively;

[0023] The total offset between each sound source face key point and each driver face key point is calculated based on the position offset corresponding to each standard key point;

[0024] Determining whether the total offset exceeds a preset offset threshold;

[0025] If the total offset exceeds the preset offset threshold, determining the judgment result as the location of the sound source is not the driver;

[0026] If the total offset does not exceed the preset offset threshold, the judgment result is determined as the sound source position is the driver.

[0027] Optionally, the correcting the initial text data based on the key points of each driver's face to obtain target text data includes:

[0028] Determining the lip movements of the driver according to the mouth key points indicating the mouth among the facial key points of each driver;

[0029] Determining actual speech information output through the driver's mouth based on the driver's lip movements;

[0030] The initial text data is corrected based on the actual voice information to obtain the target text data.

[0031] In a second aspect, an embodiment of the present application provides a vehicle-mounted communication system, the system comprising a vehicle-mounted processor, a cloud server, and a user terminal, the cloud server being capable of communicating with the vehicle-mounted processor and the user terminal respectively:

[0032] The vehicle-mounted processor is used for collecting the sound in the target vehicle through the microphone array to obtain first audio data, and collecting the facial image of the driver through the vehicle-mounted camera to obtain first image data when the driver enters the target vehicle, wherein the microphone array is a plurality of microphone groups arranged at different positions in the target vehicle to capture voice signals at different positions in the target vehicle;

[0033] The vehicle-mounted processor is used to determine the sound source position in the target vehicle based on the installation position of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data; and determine the non-sound source position from the target vehicle based on the sound source position;

[0034] The vehicle-mounted processor is used to control the main lobe of the beam of the microphone array to point to the sound source position, and control the null point of the beam of the microphone array to point to the non-sound source position, so as to adjust the microphone array;

[0035] The vehicle-mounted processor is used to collect the sound in the target vehicle through the adjusted microphone array to obtain second audio data, and control the vehicle-mounted camera to collect the image at the sound source position in real time to obtain second image data, and send the second audio data and the second image data to the cloud server;

[0036] The cloud server is used to perform face recognition on the first image data and the second image data using a face recognition detection model, and obtain the sound source face key points contained in the first image data and the driver face key points contained in the second image data, respectively, wherein the sound source face key points and the driver face key points both correspond to standard 68 face key points;

[0037] The cloud server is used to judge whether the person at the sound source position is the driver based on the key points of each sound source face and the key points of each driver face, so as to obtain a judgment result of whether the person at the sound source position is the driver;

[0038] The cloud server is configured to perform speech recognition on the second audio data to obtain initial text data if the sound source is located at the driver, and to correct the initial text data based on key points of each driver's face to obtain target text data, and to send the second audio data and the target text data to the user terminal;

[0039] The cloud server is configured to send the second audio data to the user terminal if the sound source is not located at the driver;

[0040] The user terminal is used to play the second audio data to the user through a speaker, and at the same time display the target text data to the user through a display screen.

[0041] Optionally, the vehicle-mounted processor determines the sound source position in the target vehicle based on the installation position of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data, including:

[0042] For every two microphone groups, determining a data collection time difference between the two microphone groups according to the time when the two microphone groups collect and obtain the first audio data;

[0043] Determine the similarity between each two data collection time differences based on a preset similarity algorithm;

[0044] Randomly determine a number of cluster centers, determine the cluster center with the closest similarity to the time difference between each two data collections, and assign the first audio data corresponding to the time difference between each two data collections to the cluster corresponding to the cluster center, to obtain a number of first audio data groups;

[0045] Determine the quasi-sound source position of each first audio data group based on the data collection time difference of each microphone group in each first audio data group and the setting position of each microphone group in each first audio data group in the target vehicle;

[0046] The center point of the sound source-like position of each first audio data group is determined as the sound source position in the target vehicle.

[0047] Optionally, judging whether the person at the sound source position is the driver based on each sound source face key point and each driver face key point to obtain a judgment result of whether the person at the sound source position is the driver includes:

[0048] Calculating the position offset between the sound source face key point and the driver face key point corresponding to each standard key point in the standard 68 face key points respectively;

[0049] The total offset between each sound source face key point and each driver face key point is calculated based on the position offset corresponding to each standard key point;

[0050] Determining whether the total offset exceeds a preset offset threshold;

[0051] If the total offset exceeds the preset offset threshold, determining the judgment result as the location of the sound source is not the driver;

[0052] If the total offset does not exceed the preset offset threshold, the judgment result is determined as the sound source position is the driver.

[0053] Optionally, the correcting the initial text data based on the key points of each driver's face to obtain target text data includes:

[0054] Determining the lip movements of the driver according to the mouth key points indicating the mouth among the facial key points of each driver;

[0055] Determining actual speech information output through the driver's mouth based on the driver's lip movements;

[0056] The initial text data is corrected based on the actual voice information to obtain the target text data.

[0057] In the third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the vehicle-mounted communication method described in any optional implementation manner of the second aspect above are performed.

[0058] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the vehicle-mounted communication method described in any optional implementation manner in the above-mentioned second aspect are executed.

[0059] The technical solution provided by this application includes but is not limited to the following beneficial effects:

[0060] In the vehicle communication system provided by the present application, when the driver enters the target vehicle, the vehicle processor uses the microphone array to collect the sound in the target vehicle to obtain the first audio data, and uses the vehicle camera to collect the driver's face image to obtain the first image data, so that the driver's information can be collected based on the fixed device to avoid the inability to communicate with the outside world due to the loss of the mobile device;

[0061] Then, the on-board processor determines the sound source position in the target vehicle based on the setting position of each microphone group in the target vehicle and the time when each microphone group collects the first audio data; determines the non-sound source position from the target vehicle based on the sound source position; controls the main lobe of the microphone array beam to point to the sound source position, and controls the beam null point of the microphone array to point to the non-sound source position to adjust the microphone array; collects the sound in the target vehicle through the adjusted microphone array to obtain the second audio data, and controls the on-board camera to collect the image at the sound source position in real time to obtain the second image data, and sends the second audio data and the second image data to the cloud server; through the above steps, the on-board equipment can cooperate with each other to adjust the sound collection mode of the microphone array based on the sound source position in the vehicle, so as to achieve low-noise and low-interference collection of the sound in the vehicle, as well as collection of the driver's facial information.

[0062] Then, the cloud server uses the face recognition detection model to perform face recognition on the first image data and the second image data, and obtains the sound source face key points contained in the first image data and the driver face key points contained in the second image data, respectively, wherein the sound source face key points and the driver face key points both correspond to the standard 68 face key points; based on each sound source face key point and each driver face key point, it is judged whether the sound source position is the driver to obtain a judgment result of whether the sound source position is the driver; if the sound source position is the driver, the cloud server performs speech recognition on the second audio data to obtain initial text data, and based on The initial text data is corrected at the key points of each driver's face to obtain the target text data, and the second audio data and the target text data are sent to the user terminal; if the sound source is not at the driver's location, the cloud server sends the second audio data to the user terminal; through the above steps, a large number of image processing and data calculation tasks are handed over to the cloud server for processing, which can reduce the workload of the on-board processor and release its computing resources; at the same time, the collected audio data can be processed into text, and the text processing can be corrected according to the driver's lip movements, so that the voice information and corresponding text information of the driver in the car can be accurately obtained.

[0063] Finally, the user terminal plays the second audio data to the user through the speaker, and displays the target text data to the user through the display screen, and can broadcast and display the driver's voice information and corresponding text information in the car to external users, so that the external user can communicate with the driver by listening to the voice information while also being able to communicate with the driver through text information, thereby avoiding the situation where the user cannot hear the voice information clearly due to the noisy external environment, making it difficult to achieve effective communication.

[0064] By adopting the above scheme, through the mutual cooperation of the vehicle processor, cloud server and user terminal in the vehicle communication system, the vehicle processor controls the vehicle microphone and the vehicle camera to collect the voice data and image data of the driver to determine the sound source position, and then adjusts the microphone sound collection method based on the sound source position to collect the voice data and image data in the car in real time; then the cloud server analyzes and processes the image data to determine whether the sound source is the driver, and converts the voice data in the car into text data and corrects it; finally, the user terminal displays the voice data and text data to external users; thereby achieving clear and effective communication with the outside world, while avoiding the occurrence of driving safety hazards and improving the safety of vehicle communications.

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 A flow chart of a vehicle communication method provided by the first embodiment of the present invention is shown;

[0068] Figure 2 A flow chart of a method for determining a sound source position provided by Embodiment 1 of the present invention is shown;

[0069] Figure 3 A flow chart of a method for determining a judgment result provided by Embodiment 1 of the present invention is shown;

[0070] Figure 4 A flowchart of a text data correction method provided by the first embodiment of the present invention is shown;

[0071] Figure 5 A schematic diagram showing the structure of a vehicle-mounted communication system provided by a second embodiment of the present invention is shown;

[0072] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention is shown. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0074] Embodiment 1

[0075] To facilitate understanding of this application, Figure 1 The flowchart of a vehicle communication method provided by the first embodiment of the present invention is shown to describe the contents of the first embodiment of the present application in detail.

[0076] See also Figure 1 As shown, Figure 1 A flow chart of a vehicle communication method provided by a first embodiment of the present invention is shown, wherein the method is applied to a vehicle communication system, the system includes a vehicle processor, a cloud server and a user terminal, the cloud server can communicate with the vehicle processor and the user terminal respectively, and the method includes steps S101 to S109:

[0077] S101: When the driver enters the target vehicle, the on-board processor collects the sound inside the target vehicle through the microphone array to obtain first audio data, and collects the driver's facial image through the on-board camera to obtain first image data, wherein the microphone array is a plurality of microphone groups arranged at different positions inside the target vehicle to capture voice signals at different positions inside the target vehicle.

[0078] Specifically, when the driver enters the target vehicle, the sound in the target vehicle is collected by each microphone group to obtain first audio data, and the first audio data is used to confirm the sound source position of the sound in the target vehicle. The driver's face image is collected by the vehicle-mounted camera to obtain first image data, and the first image data is used to confirm the facial features of the driver in the target vehicle. The number of microphone groups included in the microphone array is more than three, and the microphone array is used to capture voice signals at different positions in the target vehicle.

[0079] S102: The vehicle-mounted processor determines the sound source position in the target vehicle based on the installation position of each microphone group in the target vehicle and the time when each microphone group collects the first audio data; and determines the non-sound source position from within the target vehicle based on the sound source position.

[0080] Specifically, since the microphone groups are arranged at different positions in the target vehicle, the time when each microphone group collects the first audio data when collecting the sound emitted by the same sound source will also be different. The sound source position is determined based on the time difference when each two microphone groups collect the first audio data when collecting the sound emitted by the same sound source and the arrangement position of each microphone group in the target vehicle, and the center point of the area in the target vehicle that does not belong to the sound source position is determined as the non-sound source position.

[0081] S103: The onboard processor controls the main lobe of the microphone array to point to the sound source position, and controls the null point of the microphone array to point to the non-sound source position, so as to adjust the microphone array.

[0082] Specifically, the non-sound source positions in the target vehicle other than the sound source position are the interfering sound source positions. In order to shield the interfering sound sources in the target vehicle as much as possible and achieve the purpose of noise reduction, a clearer and more effective collection of the sound at the sound source position is achieved. Based on the main lobe of the beam being used to gain the voice signal collected in the corresponding direction, and the beam null point or zero point being used to attenuate the voice signal collected in the corresponding direction, the main lobe of the beam of the controlled microphone array is pointed to the sound source position, and the beam null point of the microphone array is controlled to point to the non-sound source position, so as to adjust the microphone array's sound collection at the sound source to a state corresponding to the optimal effect.

[0083] S104: The on-board processor collects the sound in the target vehicle through the adjusted microphone array to obtain second audio data, controls the on-board camera to collect the image at the sound source position in real time to obtain second image data, and sends the second audio data and the second image data to the cloud server.

[0084] Specifically, the vehicle-mounted processor collects the sound in the target vehicle through the adjusted microphone array to obtain the second audio data, and obtains the second audio data with less noise than the first audio data after interference filtering. The vehicle-mounted camera is controlled to collect the image at the sound source position in real time to obtain the second image data, and the second image data is used to confirm the identity of the person at the sound source position.

[0085] S105: The cloud server performs face recognition on the first image data and the second image data using a face recognition detection model, and obtains the sound source face key points contained in the first image data and the driver face key points contained in the second image data, respectively, wherein the sound source face key points and the driver face key points both correspond to standard 68 face key points.

[0086] Specifically, the first image data and the second image data are respectively input into the face recognition detection model to obtain the key point information of the sound source face key points and the key point information of the driver face key points corresponding to the standard 68 key points of the face.

[0087] S106: The cloud server judges whether the person at the sound source position is the driver based on the facial key points of each sound source and the facial key points of each driver, so as to obtain a judgment result of whether the person at the sound source position is the driver.

[0088] Specifically, the cloud server compares the key point information of the sound source face key points corresponding to the standard 68 key points of the face and the key point information of the driver's face key points, and determines whether the sound source position is the driver based on the comparison result.

[0089] S107: If the sound source is located at the driver, the cloud server performs speech recognition on the second audio data to obtain initial text data, and corrects the initial text data based on the key points of each driver's face to obtain target text data, and sends the second audio data and the target text data to the user terminal.

[0090] Specifically, if the sound source is the driver, it means that the second audio data collected at the sound source is the sound of the driver, so the second audio data is subjected to speech recognition to obtain the initial text data, and the content of the driver's words is obtained. Since the second audio data may have interference noise and unclearness, the initial text data obtained after speech recognition may be wrong and inconsistent with the actual words of the driver, so the data with errors in the initial text data is corrected according to the position information of the key points indicating the mouth in the key points of each driver's face, and the target text data that completely corresponds to the words of the driver is obtained.

[0091] S108: If the sound source is not located at the driver, the cloud server sends the second audio data to the user terminal.

[0092] Specifically, if the sound source is not at the driver's location, the cloud server only sends the second audio data to the user terminal and does not perform voice recognition on the second audio data, thereby reducing the workload of the server and alleviating its workload.

[0093] S109: The user terminal plays the second audio data to the user through a speaker, and displays the target text data to the user through a display screen.

[0094] Specifically, the user terminal plays the second audio data to the user through a speaker to realize communication interaction between the driver in the car and external users. At the same time, the target text data is displayed to the user through a display screen to assist the user in communicating with the driver, providing communication convenience for users in noisy environments or with hearing impairments.

[0095] In one possible embodiment, see Figure 2 As shown, Figure 2 A flow chart of a method for determining a sound source position provided in a first embodiment of the present invention is shown, wherein the vehicle-mounted processor determines the sound source position in the target vehicle based on the location of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data, including steps S201 to S205:

[0096] S201: For every two microphone groups, determine a data collection time difference between the two microphone groups according to the time when the two microphone groups collect and obtain the first audio data.

[0097] Specifically, for every two microphone groups, the data collection time difference ΔT=T1-T2 of the two microphone groups is determined according to the times (respectively recorded as T1 and T2) when the two microphone groups (respectively recorded as A1 and A2) collect and obtain the first audio data.

[0098] S202: Determine the similarity between every two data collection time differences based on a preset similarity algorithm.

[0099] Specifically, the preset similarity algorithms include Euclidean distance algorithm, cosine distance algorithm, etc.

[0100] S203: Randomly determine a number of cluster centers, determine the cluster center closest to the similarity between each two data collection time differences, and distribute the similarity between each two data collection time differences to the cluster corresponding to its cluster center, and determine the first audio data corresponding to the similarity in each cluster as a first audio data group.

[0101] Specifically, firstly, a number of cluster centers are randomly determined, and for each time difference between two data collections, the closest cluster center is found for the similarity between the two data collection time differences, and the similarity between the two data collection time differences is assigned to the cluster corresponding to the cluster center. Then, the cluster center of each cluster is updated to the average value of all similarities in the cluster, until all similarities are closest to the corresponding cluster center, the clustering process is completed, and a number of clusters are obtained. Finally, the first audio data corresponding to the similarity contained in each cluster is taken as a first audio data group, and a number of first audio data groups are obtained.

[0102] S204: Determine the quasi-sound source position of each first audio data group based on the data collection time difference of each microphone group in each first audio data group and the installation position of each microphone group in each first audio data group in the target vehicle.

[0103] Specifically, for each first audio data group, the data collection time difference corresponding to the first audio data contained in the first audio data group is used to determine the pitch angle and azimuth angle of the sound source to the microphone group that collects each first audio data, and then the equation of a straight line is determined according to the pitch angle and azimuth angle from the sound source to each microphone group, and then the quasi-sound source position of the first audio data group is determined according to the intersection of the straight line equations from the sound source to each microphone group in the same spatial coordinate system.

[0104] S205: Determine the center point of the quasi-sound source position of each first audio data group as the sound source position in the target vehicle.

[0105] Specifically, the center point of the quasi-sound source position of each first audio data group is calculated according to the spatial coordinate information of the quasi-sound source position of each first audio data group, and the center point is used as the sound source position in the target vehicle.

[0106] In one possible embodiment, see Figure 3 As shown, Figure 3 A flow chart of a method for determining a judgment result provided by the first embodiment of the present invention is shown, wherein the method of determining whether the person at the position of the sound source is the driver based on the key points of each sound source face and each driver face to obtain a judgment result of whether the person at the position of the sound source is the driver comprises steps S301 to S305:

[0107] S301: Calculate the position offset between the sound source face key point and the driver face key point corresponding to each standard key point in the standard 68 face key points respectively.

[0108] S302: Calculate the total offset between each sound source face key point and each driver face key point based on the position offset corresponding to each standard key point.

[0109] S303: Determine whether the total offset exceeds a preset offset threshold.

[0110] S304: If the total offset exceeds the preset offset threshold, the judgment result is determined as the sound source is not located at the driver.

[0111] S305: If the total offset does not exceed the preset offset threshold, the judgment result is determined as the location of the sound source is the driver.

[0112] Specifically, for each key point in the standard 68 key points of the face, the point distance between the key point of the sound source face corresponding to the key point and the key point of the driver's face is calculated as the position offset. The sum of the position offsets corresponding to each key point in the standard 68 key points of the face, or the weighted sum, is calculated to obtain the total offset.

[0113] In one possible embodiment, see Figure 4 As shown, Figure 4 A flowchart of a text data correction method provided in the first embodiment of the present invention is shown, wherein the initial text data is corrected based on the key points of each driver's face to obtain target text data, including steps S401 to S403:

[0114] S401: Determine the lip movements of the driver according to the mouth key points indicating the mouth in the facial key points of each driver.

[0115] Specifically, the key points of the mouth indicating the mouth are numbered 49-68, and the coordinates and serial numbers of the key points numbered 49-68 corresponding to the key points of each driver's face are input into the lip shape recognition model to obtain the driver's lip shape movements.

[0116] S402: Determine actual voice information output through the driver's mouth based on the driver's lip movements.

[0117] Specifically, the driver's lip movements are compared with the lip shapes in the lip shape database to match the actual voice information that matches the lip movement. The lip shape database stores various lip movements and the voice information that matches them. For example, when the driver's lip movement is a wide open mouth in a circular shape, the actual voice information matched is "oh".

[0118] S403: Correct the initial text data based on the actual voice information to obtain the target text data.

[0119] Specifically, the actual voice information is used to modify the erroneously recognized data in the initial text data that is different from the actual voice information to obtain the target text data. For example, if the actual voice information is "oh", but the data corresponding to this part in the initial text data is "ah", then the data corresponding to this part in the initial text data is modified from "ah" to "oh" to obtain the target text data.

[0120] Embodiment 2

[0121] See also Figure 5 As shown, Figure 5 A schematic diagram of the structure of a vehicle communication system provided by Embodiment 2 of the present invention is shown, wherein the system includes a vehicle processor 501, a cloud server 502 and a user terminal 503, and the cloud server can communicate with the vehicle processor and the user terminal respectively:

[0122] The vehicle-mounted processor is used for collecting the sound in the target vehicle through the microphone array to obtain first audio data, and collecting the facial image of the driver through the vehicle-mounted camera to obtain first image data when the driver enters the target vehicle, wherein the microphone array is a plurality of microphone groups arranged at different positions in the target vehicle to capture voice signals at different positions in the target vehicle;

[0123] The vehicle-mounted processor is used to determine the sound source position in the target vehicle based on the installation position of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data; and determine the non-sound source position from the target vehicle based on the sound source position;

[0124] The vehicle-mounted processor is used to control the main lobe of the beam of the microphone array to point to the sound source position, and control the null point of the beam of the microphone array to point to the non-sound source position, so as to adjust the microphone array;

[0125] The vehicle-mounted processor is used to collect the sound in the target vehicle through the adjusted microphone array to obtain second audio data, and control the vehicle-mounted camera to collect the image at the sound source position in real time to obtain second image data, and send the second audio data and the second image data to the cloud server;

[0126] The cloud server is used to perform face recognition on the first image data and the second image data using a face recognition detection model, and obtain the sound source face key points contained in the first image data and the driver face key points contained in the second image data, respectively, wherein the sound source face key points and the driver face key points both correspond to standard 68 face key points;

[0127] The cloud server is used to judge whether the person at the sound source position is the driver based on the key points of each sound source face and the key points of each driver face, so as to obtain a judgment result of whether the person at the sound source position is the driver;

[0128] The cloud server is configured to perform speech recognition on the second audio data to obtain initial text data if the sound source is located at the driver, and to correct the initial text data based on key points of each driver's face to obtain target text data, and to send the second audio data and the target text data to the user terminal;

[0129] The cloud server is configured to send the second audio data to the user terminal if the sound source is not located at the driver;

[0130] The user terminal is used to play the second audio data to the user through a speaker, and at the same time display the target text data to the user through a display screen.

[0131] In a feasible implementation manner, the vehicle-mounted processor determines the sound source position in the target vehicle based on the location of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data, including:

[0132] For every two microphone groups, determining a data collection time difference between the two microphone groups according to the time when the two microphone groups collect and obtain the first audio data;

[0133] Determine the similarity between each two data collection time differences based on a preset similarity algorithm;

[0134] Randomly determine a number of cluster centers, determine the cluster center with the closest similarity to the time difference between each two data collections, and assign the first audio data corresponding to the time difference between each two data collections to the cluster corresponding to the cluster center, to obtain a number of first audio data groups;

[0135] Determine the quasi-sound source position of each first audio data group based on the data collection time difference of each microphone group in each first audio data group and the setting position of each microphone group in each first audio data group in the target vehicle;

[0136] The center point of the sound source-like position of each first audio data group is determined as the sound source position in the target vehicle.

[0137] In a feasible implementation manner, judging whether the person at the sound source position is the driver based on the key points of each sound source face and each driver face to obtain a judgment result of whether the person at the sound source position is the driver includes:

[0138] Calculating the position offset between the sound source face key point and the driver face key point corresponding to each standard key point in the standard 68 face key points respectively;

[0139] The total offset between each sound source face key point and each driver face key point is calculated based on the position offset corresponding to each standard key point;

[0140] Determining whether the total offset exceeds a preset offset threshold;

[0141] If the total offset exceeds the preset offset threshold, determining the judgment result as the location of the sound source is not the driver;

[0142] If the total offset does not exceed the preset offset threshold, the judgment result is determined as the sound source position is the driver.

[0143] In a feasible implementation manner, the initial text data is corrected based on the key points of each driver's face to obtain the target text data, including:

[0144] Determining the lip movements of the driver according to the mouth key points indicating the mouth among the facial key points of each driver;

[0145] Determining actual speech information output through the driver's mouth based on the driver's lip movements;

[0146] The initial text data is corrected based on the actual voice information to obtain the target text data.

[0147] Embodiment 3

[0148] Based on the same application concept, see Figure 6 As shown, Figure 6 FIG. 4 shows a schematic diagram of the structure of a computer device provided by Embodiment 3 of the present invention, wherein: Figure 6 As shown, a computer device 600 provided in Embodiment 3 of the present application includes:

[0149] A processor 601, a memory 602 and a bus 603, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the computer device 600 is running, the processor 601 communicates with the memory 602 via the bus 603. When the processor 601 is running, the machine-readable instructions execute the steps of the vehicle-mounted communication method shown in the above-mentioned embodiment 1.

[0150] Embodiment 4

[0151] Based on the same application concept, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the vehicle-mounted communication method described in any one of the above embodiments are executed.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0153] The computer program product for vehicle-mounted communication provided in an embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0154] The vehicle communication system provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the system provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the system embodiment can refer to the corresponding contents in the aforementioned method embodiment. It can be clearly understood by technicians in the relevant field that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0155] In the embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0156] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0158] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a service device, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0159] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0160] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A vehicle communication method, characterized in that: Applied to an in-vehicle communication system, the system includes an in-vehicle processor, a cloud server and a user terminal, the cloud server can communicate with the in-vehicle processor and the user terminal respectively, and the method includes: When the driver enters the target vehicle, the vehicle-mounted processor collects the sound in the target vehicle through the microphone array to obtain the first audio data, and collects the facial image of the driver through the vehicle-mounted camera to obtain the first image data, wherein the microphone array is a plurality of microphone groups arranged at different positions in the target vehicle to capture the voice signals at different positions in the target vehicle; The vehicle-mounted processor determines the sound source position in the target vehicle based on the installation position of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data; and determines the non-sound source position in the target vehicle based on the sound source position; The on-board processor controls the main lobe of the microphone array to point to the sound source position, and controls the null point of the microphone array to point to the non-sound source position, so as to adjust the microphone array; The vehicle-mounted processor collects the sound in the target vehicle through the adjusted microphone array to obtain second audio data, controls the vehicle-mounted camera to collect the image at the sound source position in real time to obtain second image data, and sends the second audio data and the second image data to the cloud server; The cloud server performs face recognition on the first image data and the second image data using a face recognition detection model, and obtains the driver's face key points contained in the first image data and the sound source face key points contained in the second image data, respectively, wherein the sound source face key points and the driver's face key points both correspond to standard 68 face key points; The cloud server judges whether the person at the sound source position is the driver based on the key points of each sound source face and the key points of each driver face, so as to obtain a judgment result of whether the person at the sound source position is the driver; If the sound source is located at the driver, the cloud server performs speech recognition on the second audio data to obtain initial text data, corrects the initial text data based on the key points of each driver's face to obtain target text data, and sends the second audio data and the target text data to the user terminal; If the sound source is not located at the driver, the cloud server sends the second audio data to the user terminal; The user terminal plays the second audio data to the user through a speaker, and displays the target text data to the user through a display screen.

2. The method according to claim 1, characterized in that: The vehicle-mounted processor determines the sound source position in the target vehicle based on the installation position of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data, including: For every two microphone groups, determining a data collection time difference between the two microphone groups according to the time when the two microphone groups collect and obtain the first audio data; Determine the similarity between each two data collection time differences based on a preset similarity algorithm; Randomly determine a number of cluster centers, determine the cluster center with the closest similarity to the time difference between each two data collections, and assign the first audio data corresponding to the time difference between each two data collections to the cluster corresponding to the cluster center, to obtain a number of first audio data groups; Determine the quasi-sound source position of each first audio data group based on the data collection time difference of each microphone group in each first audio data group and the setting position of each microphone group in each first audio data group in the target vehicle; The center point of the sound source-like position of each first audio data group is determined as the sound source position in the target vehicle.

3. The method according to claim 1, characterized in that The step of judging whether the person at the sound source position is the driver based on the key points of each sound source face and the key points of each driver face to obtain a judgment result of whether the person at the sound source position is the driver includes: Calculating the position offset between the sound source face key point and the driver face key point corresponding to each standard key point in the standard 68 face key points respectively; The total offset between each sound source face key point and each driver face key point is calculated based on the position offset corresponding to each standard key point; Determining whether the total offset exceeds a preset offset threshold; If the total offset exceeds the preset offset threshold, determining the judgment result as the location of the sound source is not the driver; If the total offset does not exceed the preset offset threshold, the judgment result is determined as the sound source position is the driver.

4. The method according to claim 1, characterized in that: The step of correcting the initial text data based on the key points of each driver's face to obtain target text data includes: Determining the lip movements of the driver according to the mouth key points indicating the mouth among the facial key points of each driver; Determining actual speech information output through the driver's mouth based on the driver's lip movements; The initial text data is corrected based on the actual voice information to obtain the target text data.

5. A vehicle-mounted communication system, characterized in that: The system includes an on-board processor, a cloud server and a user terminal, wherein the cloud server can communicate with the on-board processor and the user terminal respectively: The vehicle-mounted processor is used for collecting the sound in the target vehicle through the microphone array to obtain first audio data, and collecting the facial image of the driver through the vehicle-mounted camera to obtain first image data when the driver enters the target vehicle, wherein the microphone array is a plurality of microphone groups arranged at different positions in the target vehicle to capture voice signals at different positions in the target vehicle; The vehicle-mounted processor is used to determine the sound source position in the target vehicle based on the installation position of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data; and determine the non-sound source position from the target vehicle based on the sound source position; The vehicle-mounted processor is used to control the main lobe of the beam of the microphone array to point to the sound source position, and control the null point of the beam of the microphone array to point to the non-sound source position, so as to adjust the microphone array; The vehicle-mounted processor is used to collect the sound in the target vehicle through the adjusted microphone array to obtain second audio data, and control the vehicle-mounted camera to collect the image at the sound source position in real time to obtain second image data, and send the second audio data and the second image data to the cloud server; The cloud server is used to perform face recognition on the first image data and the second image data using a face recognition detection model, and obtain the driver's face key points contained in the first image data and the sound source face key points contained in the second image data, respectively, wherein the sound source face key points and the driver's face key points both correspond to standard 68 face key points; The cloud server is used to judge whether the person at the sound source position is the driver based on the key points of each sound source face and the key points of each driver face, so as to obtain a judgment result of whether the person at the sound source position is the driver; The cloud server is configured to perform speech recognition on the second audio data to obtain initial text data if the sound source is located at the driver, and to correct the initial text data based on key points of each driver's face to obtain target text data, and to send the second audio data and the target text data to the user terminal; The cloud server is configured to send the second audio data to the user terminal if the sound source is not located at the driver; The user terminal is used to play the second audio data to the user through a speaker, and at the same time display the target text data to the user through a display screen.

6. The system according to claim 5, characterized in that The vehicle-mounted processor determines the sound source position in the target vehicle based on the installation position of each microphone group in the target vehicle and the time when each microphone group collects and obtains the first audio data, including: For every two microphone groups, determining a data collection time difference between the two microphone groups according to the time when the two microphone groups collect and obtain the first audio data; Determine the similarity between each two data collection time differences based on a preset similarity algorithm; Randomly determine a number of cluster centers, determine the cluster center with the closest similarity to the time difference between each two data collections, and assign the first audio data corresponding to the time difference between each two data collections to the cluster corresponding to the cluster center, to obtain a number of first audio data groups; Determine the quasi-sound source position of each first audio data group based on the data collection time difference of each microphone group in each first audio data group and the setting position of each microphone group in each first audio data group in the target vehicle; The center point of the sound source-like position of each first audio data group is determined as the sound source position in the target vehicle.

7. The system according to claim 5, characterized in that The step of judging whether the person at the sound source position is the driver based on the key points of each sound source face and the key points of each driver face to obtain a judgment result of whether the person at the sound source position is the driver includes: Calculating the position offset between the sound source face key point and the driver face key point corresponding to each standard key point in the standard 68 face key points respectively; The total offset between each sound source face key point and each driver face key point is calculated based on the position offset corresponding to each standard key point; Determining whether the total offset exceeds a preset offset threshold; If the total offset exceeds the preset offset threshold, determining the judgment result as the location of the sound source is not the driver; If the total offset does not exceed the preset offset threshold, the judgment result is determined as the sound source position is the driver.

8. The system according to claim 5, characterized in that The step of correcting the initial text data based on the key points of each driver's face to obtain target text data includes: Determining the lip movements of the driver according to the mouth key points indicating the mouth among the facial key points of each driver; Determining actual speech information output through the driver's mouth based on the driver's lip movements; The initial text data is corrected based on the actual voice information to obtain the target text data.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the vehicle-mounted communication method as described in any one of claims 1 to 4 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the vehicle communication method as claimed in any one of claims 1 to 4 are executed.

Images