Vehicle and method for determining characteristic lip movement patterns

Abstract

The invention relates to a vehicle (1) comprising media playback means (2) for playing songs in the vehicle interior. The vehicle according to the invention is characterized by vocal detection means (3) for detecting and recording the singing along of at least one vehicle occupant (4) during the playback of a song; vocal analysis means (5) for detecting deviations between the singing along (6) of the vehicle occupant (4) and the singing contained in the song; and vocal indication means (7), which are configured to display to the vehicle occupant (4) at least the deviations between the singing along (6) and the singing and / or to record the singing along (6), to compensate for deviations of the singing along (6) by means of automatic pitch correction, and to output the corrected singing along via the media playback means (2).

Description

The invention relates to a vehicle of the type defined in more detail in the preamble of claim 1 and to a method for determining characteristic lip movement patterns.Media contents such as movies, series, music videos, hearing books, podcasts, songs and the like can be played via the infotainment system of a vehicle. The music reproduced in the vehicle can excite the vehicle occupants to join or sum them together. This can be considered both positive and negative by individual vehicle occupants.If a vehicle occupant, for example, slate, i.e., the respective pitches of the notes do not meet or do not sate synchronously with the song text of the ging, this can easily be perceived as annoying. In the worst case, this may result in a scew, which to a particular extent can deflect the vehicle-guiding person from his driving task.However, the particular joint singinging can also have an increasing effect and thus ensure more fun when using the vehicle.With the aid of the so-called audio decomposition, voices of accompanying music or background noises can be separated. For this purpose, reference is made, for example, to Dridger, J., & Muller, M. (2015, April). Extracting singing voice from music recordings by cascadeing audio decomposition techniques. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 126-130). IEEE.Furthermore, methods for computer-assisted assessment of karaoke ging performances are known, with which the computer can assess the ging performance in the manner in which humans would also do. For this purpose, reference is made to Tsai, W. H., & Lee, H. C. (2011). Automatic evaluation of karaoke singing based on pitch, volume, and rhythm features. IEEE transactions on audio, speech, and language processing, 20(4), 1233-1243, and Mayor, O., Bonada, J., & Loscos, A. (2009, February). Performance analysis and scoring of the singing voice. In Proc. 35th AES Intl. Conf., London, UK (pp. 1-7).US 2011 / 0 251 841 A1 also discloses coordinating and mixing gesang, which was recorded by springs geographically remote from one another. The document describes an app executable on a mobile terminal such as a smartphone for providing a karaoke function. In this case, the smartphone can be used to listen to the song, wherein in particular the song text and pitch instructions are displayed on the display of the smartphone. The user of the app can then join with it, wherein the user's s s s sangulation is recorded by means of the smartphone-integrated microphone. Data generated via the smartphone can be transmitted to a central server. Several persons geographically remote from one another can singe the same song. Recordings made by the different persons can be mixed together by the server. The app or the server also offers the functionality of carrying out an automatic pitch correction, so that obliquely sowed passages in the sound are corrected. The ging performance can also be evaluated and the best children can be listed in a ranking list. Information can be exchanged between the respective devices, inter alia, in the so-called MIDI format. A respective app user must thereby want to independently perform the intention of karaoke. The corresponding appendice requires a high cognitive outlay and is therefore completely unsuitable for execution by a vehicle-guiding person.In addition, U.S. Pat. No. 2019) / 0 147 841 A1 discloses means for illustrating a karaoke interface. The document also describes the use of a mobile terminal such as a smartphone for participating in a common karaoke activity. In this case, the individual users can take films with the front camera of their smartphone and video or images of their karaoke presentation and share them among one another. In this case, a camera-based face recognition can be carried out in order to identify characteristic face features and use these as reference points for imprinting a virtual mask. For example, a user's face may be replaced with a famous personality, a mask pot, or a combi figure face. The gerang quality of the individual karaoke performances can also be determined and evaluated. This task can be carried out both by a computer system and by the users of a corresponding app. In this case, the quality of the sounds evaluated by the computer model or the users can also be weighted. Successful users can be listed in a ranking list on the front locations.In addition, DE 10 2015 014 652 B4 discloses a motor vehicle and a method for operating the motor vehicle, in which a text of a piece of music is output. While karaoke is being performed by the vehicle occupants in the motor vehicle, the motor vehicle is operated at least semi-autonomously. In this case, a sound track is recorded in the vehicle interior and it is determined from this how "good" the vehicle occupant is sinking the song.In addition, CN 1 13 270 082 A discloses a vehicle with a karaoke function. In this case, the gesang contained in a song is filtered out of an audio signal by means of methods of voice suppression and the audio signal is output in the vehicle. In addition, the acoustics are recorded in the vehicle interior to record an ambient audio signal. Echoes are filtered out of the ambient audio signal, so that the co-action of a singing vehicle occupant is better perceptible.In addition, DE 10 2018 214 976 A1 discloses a method for controlling a multimedia device and a computer program. In this case, a reaction of a vehicle occupant to the output of a multimedia piece in a vehicle is detected sensorially and the further output of multimedia pieces is planned by means of a machine learning method.In addition, DE 10 2022 107 293 A1 discloses an assistance system and an assistance method for a vehicle. In this case, a reaction of a vehicle occupant to the output of a multimedia piece in a vehicle is detected by sensors and a vehicle function is activated as a function of the reaction.Furthermore, CN 1 08 346 427 A discloses a method for voice recognition and a device and a storage medium that can be used for this purpose. In this case, acoustic captured speech and speech captured by lip reading are matched.The object of the present invention is to specify an improved vehicle which is distinguished by an improved user interaction, in particular during travel.According to the invention, this object is achieved by a vehicle having the features of claim 1. Advantageous embodiments and developments and a method for determining characteristic lip movement patterns are evident from the claims dependent thereon.A vehicle of the generic type, comprising media playback means for outputting pages in the vehicle interior, comprises singing determination means for detecting and detecting the presence of at least one vehicle occupant during the output of a page; sing analysis means for detecting deviations between the presence of the vehicle occupant and the singing contained in the page; and singing indication means which are configured to indicate to the vehicle occupant at least the deviations between the presence of the vehicle occupant and the singing and record the presence of the presence, to compensate deviations of the presence of the presence by means of automatic pitch correction and to output the compensated presence of the presence via the media playback means.According to the invention, the gerangication means are furthermore configured to train a machine learning model with the gerang behavior of a specific vehicle occupant, such that the machine learning model learns for defined song deviations between the gerang and the gerang in the output of the respective song in the vehicle interior before the vehicle occupant drives along a respective song passage, and to determine pitch adaptations usable for automatic pitch correction by means of the trained machine learning model and to apply these.The vehicle according to the invention is thus able to react appropriately on a situation-specific basis with the aid of the singing determination means, singing analysis means and singing indication means to a respective situation in which a vehicle occupant is driving during use of the vehicle. By displaying the deviations between the joint sang and the sang, the respective vehicle occupant is informed by a further source that he is possibly singinging obliquely. If this is already justified by other vehicle occupants, but if the walking vehicle occupant does not believe the other vehicle occupants, a snatch arising can be prevented as a result, since the vehicle evaluates the co-action from a neutral point of view. Moreover, the slateness of a vehicle occupant can also be corrected automatically by applying said automatic pitch correction, also referred to as auto-tuning. This is preferably done in a subtil manner, so that the respective vehicle occupant does not notice this. This also makes it possible to counteract the occurrence of stile, caused by the poor sangability of the singing vehicle occupant. This ultimately improves traffic safety since the vehicle-guiding person is not distracted by the potential stut in the vehicle interior.However, the individual vehicle occupants may also have friends at the Mitsingen. By displaying the deviations between the co-sang and the sang, the respective vehicle occupant can train and thus improve his sangability. By correcting skew-going ride by auto-tuning, poor children are also allowed to have friends at ride. This increases the mood of the vehicle occupants, which ultimately improves the vehicle's experience of use.The interaction between the vehicle occupant or occupants and the vehicle takes place in a subtil manner, so that the respective vehicle occupants, in particular the vehicle-guiding person, are not excessively treated in cognitive terms. Thus, in particular, the singing determination means can proactively detect a co-singing in the vehicle interior and do not have to be activated for this purpose by a vehicle occupant. Accordingly, the deviation between the joint and the sang is also automatically determined and the gerang indexing means is controlled accordingly.In the simplest case, only the vehicle-guiding person stops in the vehicle and drives from a starting location to a destination location. Music is reproduced via the media reproduction means, for example the infotainment system of the vehicle; for example, the radio of the vehicle is used as an audio source, a sound file stored by a storage medium such as a hard disk, an SSD, an SD card or a USB stick is read out or music is obtained via a music streaming service provider. Randomly or intentionally, a song that is particularly preferred by the vehicle-guiding person is played back. The vehicle-guiding person starts to join suddenly, although this was not intended at all. Without the carer noticing this, the co-input is detected, the deviation to the input is determined, the co-input is compensated by the automatic pitch correction, and the compensated co-input is outputted accordingly via the media reproduction means. Thus, the car driver does not notice that he is descending obliquely, which might otherwise frustration them, which in turn might impair their ability to safely guide the vehicle. This can be reliably prevented with the aid of the vehicle according to the invention.There could also be more vehicle occupants in the vehicle, such as two, three, four or even more vehicle occupants. The joint input can also be individually detected and evaluated for the individual vehicle occupants. For this purpose, the vehicle has a corresponding number of sangling determination means, sangling analysis means and sangling indication means.In another situation, the co-input from a particular vehicle occupant could be evaluated and the deviation of the co-input to the co-input could be displayed in the vehicle. For this purpose, any display device in the vehicle can be used, such as the display of the instrument cluster, the head unit, a dedicated passenger display, a display device integrated into a headrest and the like. The follow-up of the ging performance of the vehicle occupant by the respective other vehicle occupants can have an increasing effect and thus ensure an increased driving experience.The media playback means comprise at least acoustic output means such as sound transducers, in particular in the form of loudspeakers, and a corresponding computing unit for controlling the acoustic output means. The computing unit furthermore requires a source of song, for example in the form of said radio, in the form of a sound file read out from a computer-readable storage medium and / or in the form of the Internet stream. The media playback means or the computing unit can also analyze the audio content to be output via the acoustic output means and identify underlying music features. However, these could also be present explicitly in a corresponding sound file, for example determined by the respective distributor of the song. This can be, for example, the song text, the semantic content of the song text, the harmonism, the melody and / or the rhythm of at least one passage of the song or preferably of the complete song.Thus, the harmonies, melody and rhythm represent three basic ordering systems for classifying music. In this case, the harmonism describes the structure of pitches, that is to say the simultaneous sounding of a plurality of individual sounds. The melody describes the temporal arrangement or sequence of tones. The rhythm indicates the dynamics of music and can in turn be divided into clock, metrum and tempo.Specific embodiments of the sang determination means are mentioned below.With the aid of the singing determination means, singing features corresponding to the corresponding music features can be identified, which allows comparison by the sing analysis means.The singing analysis means compares the singing features with the music features and thereby can identify the deviations between the singinging and the singing. In particular, the sang analysis means examine whether the particular singing vehicle occupant meets the appropriate pitch at the appropriate time and / or the correct song text is singered. In the simplest case, the ging analysis means are formed by a program routine executed on a computing unit. This can be the same arithmetic unit which also serves to form the media playback means. In general, both the media playback means, the sang determination means, the sang analysis means and / or the gangling indication means can use capacities provided by one or more different computing units.Passengers identified by the vehicle can likewise be entered in a best list and their respective rearing performances can thus be tracked. A respective best list can be stored and maintained locally in a respective vehicle, so that respective vehicle occupants within the vehicle can enter into a gingwet competition against one another. A corresponding best list can also be passed over the vehicles of a fleet of vehicles, for example managed by a central computing device external to the vehicle, such as a cloud server. Thus, the users of different vehicles can enter into a ginging competition against one another. For this purpose, the respective vehicle users can log on to their vehicle with an individual user identifier, for example in the form of a user name and a secret password, which links the respective sounds to the user identifier and correspondingly stores them at the vehicle-external central computing device.The vehicle may have communication means such as a telecommunication unit. A respective computing unit of the vehicle can establish a wireless communication connection to the vehicle-external central computing device by means of the telecommunication unit, for example by mobile radio and / or Wi-Fi. Thus, information can be exchanged with the Internet by mobile radio or also via a Wi-Fi hotspot and can be transmitted accordingly to the central computing device connected to the Internet.As already described above, the gerangication means are furthermore configured to train a machine learning model with the gerang behavior of a specific vehicle occupant, such that the machine learning model learns for defined song deviations between the gerang and the gerang in the output of the respective song in the vehicle interior before the vehicle occupant draws along a respective song passage, and to determine pitch adaptations which can be used for automatic pitch correction by means of the trained machine learning model and to apply these. This increases the reliability in determining the compensated margin. Thus, although classical auto-tune methods are capable of performing automatic pitch correction in near real time, this is still associated with some latency. Thus, the balanced co-input output in the vehicle interior does not completely coincide in time with the co-input. With the aid of the correspondingly trained model of machine learning, however, this disadvantage can be overcome. Thus, the corresponding model of machine learning gradually learns the sangling behavior of the respective vehicle occupant and can thereby be enabled to predict obliquely singled song passages. Not only is the general slatesing itself determined, but also the extent, i.e. to what extent, tones are not made in terms of their pitch. Accordingly, the machine learning model is able to determine a suggested correction to what extent the pitch correction must be made. This makes it possible to correct the pitch already during the singinging, so that the compensated singing can be output simultaneously with the singing in the vehicle interior. This further improves the user interaction with the vehicle.In particular, temporally convolutional neural networks can be used for this purpose in conjunction with short memories such as GRUs, also referred to as gated recurrent unit, or LSTMs, also referred to as long short-term memory.An advantageous further development of the vehicle according to the invention further provides that the singing determination means comprise at least one microphone detecting the acoustics in the vehicle interior and a computing unit for processing acoustic signals generated by the microphone, wherein the computing unit is configured to recognize singing features in the acoustic signals, in particular in the form of the song text, the semantic content of the song text, the harmonies, the melody and / or the rhythm of at least one passage of the song, and wherein the computing unit is additionally preferably configured to receive the song content from the media reproduction and subtract it from the acoustic signals. This enables acoustic detection of the input in the vehicle interior.Preferably, a plurality of microphones arranged distributed in the vehicle interior are used for capturing the co-input. This enables even more reliable detection of the mitogen. By analyzing a runtime difference with which the input is detected by the respective microphones and / or by taking into account a different amplitude of the respective sound signal, the singing vehicle occupant can also be located in the vehicle interior. This allows a determination as to which vehicle occupant is currently sinking.A song is reproduced in the vehicle interior via the media reproduction means. The song overlays the ride of the vehicle occupant or occupants. This can make it difficult to detect the input in an acoustic manner. In order to counteract this, techniques known from so-called audio decomposition can be used, such as harmonic-percutaneous-residual decomposition, melody-residual decomposition, transient-residual decomposition, robust principle component analysis and / or methods of machine learning, such as discriminators within the scope of a generative adventaric network. This enables the separation of different sound sources from an audio signal. Thus, the co-sang can be differentiated from the sang contained in the song and the sang of different vehicle occupants can also be differentiated from one another. This makes it possible to "hear even more clearly" the vehicle occupant or occupants and thus to extract them more reliably. In particular, the computing unit draws the song content, i.e. those sounds to be output via the loudspeakers of the vehicle, directly from the media playback means, so that the computing unit is reliably able to filter the song content from the acoustic signal recorded by the microphone or microphones.However, in a purely acoustic-based analysis for detecting a co-generation, there is always a residual risk that the co-generation in the vehicle interior will not be correctly identified. According to a further advantageous embodiment of the vehicle according to the invention, the sangling determination means thus comprise at least one camera which captures at least the face of at least one vehicle occupant and a computing unit for processing camera images generated by the camera, wherein the computing unit is configured to recognize lip movements of the vehicle occupant in the camera images, to compare these with lip movement patterns characteristic for the sangling and to recognize the exertion of sangling by the vehicle occupant if the camera images contain at least one such characteristic lip movement pattern. Thus, the singinging of a vehicle occupant can be determined not only acoustically, but also visually.This can be used particularly preferably to automatically initiate the corresponding analysis of the co-input and display of the deviations between the co-input and the input or the ascertainment of a compensated co-input and its output via the media playback means. Thus, preferably no acoustic detection of the vehicle interior takes place, which improves the data protection and thus the privacy of the vehicle occupants. However, as soon as lip movements characteristic of the singinging are identified, the acoustic vehicle interior detection can then be activated and the ging analysis means and gingangnic indexing means can be controlled accordingly. This further reduces the degree of distraction of the vehicle driver, since no manual operating actions need be entered. This further improves the user interaction with the vehicle and also increases the safety in road traffic due to the reduction of the degree of distraction of the vehicle-guiding person.The lip movement patterns characteristic of the singing can be stored in a database stored on a respective computer unit. Characteristic lip movement patterns may have been identified, for example, by a service provider or else by the vehicle manufacturer in preceding measurement campaigns. Lip movements can be identified in the camera images with the aid of machine vision. For this purpose, methods based on artificial intelligence can also be used, for example models of machine learning, in particular in the form of artificial neural networks.The computing unit is preferably further configured to recognize ging features in the detected lip movements by the comparison with the characteristic lip movement patterns, in particular in the form of the song text, the semantic content of the song text, the harmonies, the melody and / or the rhythm of at least one passage of the song. Thus, by means of the visual detection of the vehicle occupants, not only can the sharing per se generally be detected, but the singinging features can additionally be determined. This can support or even replace the acoustic detection of the sang features. The respective ging features can be linked to the characteristic lip movement patterns. For this purpose, corresponding measurement campaigns can have been carried out in order to establish a relationship between the sang features and characteristic lip movement patterns.The computing unit can thus be enabled to read lips, i.e. to identify the words spoken or spoken in each case by purely visually observing the lip movements of a vehicle occupant. These words can be converted into a text string and read into a model for recognizing speech. Thus, not only the song text per se, but also its semantic content can be determined.A further advantageous embodiment of the vehicle according to the invention further provides that the singing determination means are further configured to actuate the media playback means upon detecting the co-singinging by at least one vehicle occupant in order to cause the output of a karaoke version of the song output in the vehicle interior, wherein the karaoke version comprises a sing lane reduced in its volume or is completely free of sing lanes. This makes it easier for the vehicle occupant to be included because the singing in the song is heard less strongly or even completely missing, so that the respective vehicle occupants can better concentrate on their own sing. For this purpose, corresponding karaoke versions can be stored in a computing unit of the vehicle or else can be called up as required via the Internet via a corresponding service provider.A further advantageous embodiment of the vehicle according to the invention further provides that the gerangication means are further configured to actuate the media playback means for reducing the volume of the gerang track contained in the outputted song and / or for increasing the volume of the compensated co-input. The computing unit can also be configured to reduce the volume of the respective sang lane contained in the song instead of a karaoke version of the respective song output in the vehicle interior and to reproduce the compensated co-sang louder determined by the sangling indexing means for this purpose. This makes it easier for the vehicle occupant or occupants to perceive their own co-operation. Since this is the compensated consequence, this increases the well-being of the respective vehicle occupants to a particular extent. Thus, the respective vehicle occupants perceive that they are hitting the sounds of the singing, which ultimately convincing them from their own singing abilities.A further advantageous embodiment of the vehicle according to the invention further provides that the singing analysis means are further configured to, when detecting a temporal deviation between the singing and the co-sing, cause the output of at least one visual and / or at least one haptic indication in the vehicle interior at exactly the time at which the sing requires the use of the vehicle occupant. This can improve the vehicle occupant's ability to singe in the rhythm of the song. It is thus possible to prevent the respective vehicle occupant from lowering a specific song passage too early or too late. For the temporal indication of when which song text must be sent, methods and means are already known from the prior art, such as the temporal change of a color profile of the displayed song text, wherein the boundary of the color profile tracks with exactly the song passage to be sent. According to the invention, however, devices in the vehicle can be used to output the visual and / or haptic indications. For example, a light arranged in the interior of the vehicle can be switched on exactly when the use of the vehicle occupant is required. For this purpose, for example, a so-called ambient light could also be used. Additionally or alternatively, haptic indications can be output, for example, by the actuators of a massage seat or also by the exertion of oscillations on vehicle elements touched by the vehicle occupant, such as the steering wheel or a gear lever.In particular, the output of haptic instructions is particularly subtil, since only the respective vehicle occupant notes the haptic instruction. This makes it easier for him to join in a time-related manner to the singing, without the other vehicle occupants noticing the assistance in this regard.According to a further advantageous embodiment of the vehicle according to the invention, the singing analysis means are furthermore configured to objectively evaluate the deviation between the singinging and the jointing using permanently predefined evaluation metrics and / or to subjectively evaluate it using a machine learning model, wherein the machine learning model has been trained using the subjective evaluations of the jointing of a plurality of test horns in measurement campases for selected carried out. In this case, weighting can be effected between an objective evaluation and a subjective evaluation. For objective assessment of the deviation of the joint to the joint, for example, differences can be detected on a musical instrument digital interface (MIDI) note scale for determining a correct pitch. Short-term log-energy can be used to determine relative volume errors. Deviations in the geezing rhythm can be determined by means of hidden Markov models. A classification of the mood, also referred to as expression classification, can also be carried out, a sequence being assigned the most likely expression sought. On the basis of this additional dimension, the evaluation is possible to also evaluate ging elements deviating from the target at a higher level of abstraction.According to the invention, a method for determining characteristic lip movement patterns comprises the following method steps:transmitting, by a computing unit provided in a vehicle described above, an assignment of lip movements and singing features to a computing unit external to the vehicle, wherein the computing unit receives music features from the media playback means and extracts the singing features to be assigned to the lip movements from the music features;aggregating the assignments obtained from a plurality of different computing units in a correspondence database by the computing device;examining the aggregated assignments for characteristic patterns in order to identify lip movement patterns characteristic for specific singing features, by the computing device; andproviding the lip movement patterns identified in this way for retrieval by the respective computing units.With the aid of the method according to the invention, it is possible to identify new characteristic lip movement patterns and to distribute them for application to the respective computing units of the vehicles. This makes it possible to determine even more reliable sang by visual detection of the vehicle occupants and, in addition, also to determine the respective sang features. To examine the aggregated assignments for characteristic patterns, similar lip movement patterns may be arranged into groups, respectively. Proven grouping algorithms, also referred to as clustering algorithms, can be used for this purpose, such as the k-means algorithm.As already mentioned above, the music features such as song text, semantic content of the song text, harmonies, melody and / or rhythms of at least one song passage can be contained in a corresponding sound file or can be determined by the computing unit by analyzing the sound file. These music features are then assigned to corresponding sangling features, which in turn allows a corresponding assignment to the lip movements.Further advantageous embodiments of the vehicle according to the invention also result from the exemplary embodiment, which is described in more detail below with reference to the figure.FIG. 1 shows a schematic illustration of a vehicle according to the invention.FIG. 1 shows a vehicle 1 according to the invention, which is designed to detect the movement 6 of at least one vehicle occupant 4 to a song output in the vehicle interior and to react to it in a situation-specific manner.For this purpose, the vehicle 1 comprises media playback means 2 for outputting song. The media playback means 2 comprise at least acoustic output means, for example in the form of loudspeakers 13, and a computing unit 9 for controlling the loudspeakers 13, The computing unit 9 can use different sources of songs, such as, for example, a radio, not shown in more detail, a computer-readable storage medium to which the computing unit 9 has access, and / or an Internet stream. A mobile terminal, not shown in more detail, such as a smartphone, can also be coupled to the computing unit 9, for example wirelessly via Bluetooth, NFC or Wi-Fi, or else by cable, for example via USB cable or Ethernet cable, and a sound file stored on a computer-readable storage medium of the mobile terminal can be read out.The vehicle 1 further comprises a singing determination means 3 for detecting and detecting the assistance 6 of the at least one vehicle occupant 4 during the output of the song. Purely acoustic monitoring of the vehicle interior by means of one or more microphones 8 is possible for this purpose, and alternatively or additionally visual detection of the vehicle interior by means of at least one camera 10, by means of which at least the face of the at least one vehicle occupant 4 can be detected. This makes it possible to detect lip movements of the vehicle occupant 4. Sensor data generated by the microphone or microphones 8 or the cameras 10 can likewise be evaluated by the computing unit 9. In corresponding acoustic signals, the joint 6 of the vehicle occupant 4 can be recognized particularly easily. However, there is the risk that the co-generation 6 will be over-sunned, for example by the insides of the vehicle or by other noise emitted in the vehicle interior, and thus cannot be detected correctly. Lip movement patterns recognized in the corresponding camera images can be matched by the arithmetic unit 9 with known lip movement patterns which are produced during the sinking, as a result of which the biting of the vehicle occupant 4 can be identified. The characteristic lip movement patterns can also contain indicia for ging features, such as the song text as well as the harmonies, melody and / or rhythm of the song. Thus, the computing unit 9 can read lips while processing the camera images and thereby determine the spoken song text. By semantic analysis of the lyrics, for example with the aid of natural language recognition models, the semantic content of the lyrics can also be determined.The ging analysis means 5 can be formed by program routines executed on the arithmetic unit 9. Deviations between the joint 6 of the vehicle occupant 4 and the joint contained in the song are detected by means of the ging analysis means 5. Furthermore, the vehicle 1 has gerangication means 7 which are configured to display to the vehicle occupant 4 at least the deviations between the joint 6 and the joint and record the joint 6, to compensate deviations of the joint 6 by automatic pitch correction and to output the compensated joint via the media playback means 2. Corresponding program routines for providing corresponding method steps can likewise be stored on the arithmetic unit 9 and executed there accordingly. Furthermore, at least one display device 14 is part of the gerangication means 7 in order to be able to display the corresponding deviations between the joint 6 and the gerang.With the aid of a method according to the invention, it is also possible to identify new lip movement patterns which are characteristic of the sinking. Thus, the arithmetic unit 9 determines music features corresponding to the respective ging features from the information obtained from the media playback means 2 and assigns these correspondingly identified lip movements. These assignments are transmitted by the computing unit 9 to a computing device 11 external to the vehicle, which computing device maintains a correspondence database 12 for storing these assignments. For this purpose, the vehicle 1 can have a telecommunication unit 15. The vehicle-external central computing device 11 examines the assignments of music or ging features to lip movements for the presence of characteristic patterns. Thus, similar lip movement characteristic patterns can be grouped for similar sang features. Lip movement patterns with a high similarity are assigned to the same music or sangling features and can thus be provided for retrieval by the respective arithmetic units 9 in order to be used there for determining sangling features with analysis of the lip movements.

Claims

Vehicle (1) comprising media playback means (2) for outputting inside-cabin songs, comprising singing determination means (3) for detecting and detecting the presence of at least one vehicle occupant (4) during the output of a song; sing analysis means (5) for detecting deviations between the presence (6) of the vehicle occupant (4) and the presence of the song; and singing indication means (7) which are configured to indicate to the vehicle occupant (4) at least the deviations between the presence (6) and the presence and record the presence (6), to compensate deviations of the presence (6) by means of an automatic pitch correction and to output the compensated presence via the media playback means (2), characterized in that, wherein the gerangication means (7) are further configured to train a machine learning model with the gerang behavior of a specific vehicle occupant (4) such that the machine learning model learns for defined song deviations between the gerang and the gerang (6) upon the output of the respective song in the vehicle interior before the vehicle occupant (4) drives along a respective song passage, and to determine pitch adaptation usable for automatic pitch correction by means of the trained machine learning model and to apply these.Vehicle (1) according to Claim 1, characterized in that the singing determination means (3) have at least one microphone (8) which detects the acoustics in the vehicle interior and a computing unit (9) for processing acoustic signals generated by the microphone (8), wherein the computing unit (9) is configured to identify singing features in the acoustic signals, in particular in the form of the song text, the semantic content of the song text, the harmonies, the melody and / or the rhythm of at least one passage of the song, and wherein the computing unit (9) is additionally preferably configured to receive the song content from the media reproduction means (2) and subtract it from the acoustic signals.Vehicle (1) according to Claim 1 or 2, characterized in that the singing determination means (3) comprise at least one camera (10) which captures at least the face of at least one vehicle occupant (4) and a computing unit (9) for processing camera images generated by the camera (10), wherein the computing unit (9) is configured to detect lip movements of the vehicle occupant (4) in the camera images, to compare these with lip movement patterns which are characteristic of the singinging and to detect the exertion of sing by the vehicle occupant (4) if the camera images contain at least one such characteristic lip movement pattern.Vehicle (1) according to claim 3, characterised in that the computing unit (9) is further configured to recognize, by matching with the characteristic lip movement patterns, ging features in the detected lip movements, in particular in the form of the song text, the semantic content of the song text, the harmonies, the melody and / or the rhythm of at least one passage of the song.Vehicle (1) according to one of Claims 1 to 4, characterized in that the singing determination means (3) are furthermore configured to actuate the media reproduction means (2) on detecting the co-singinging by at least one vehicle occupant (4) in order to cause the output of a karaoke version of the song output in the vehicle interior, wherein the karaoke version comprises a sing lane which is reduced in its volume or is completely free of sing lanes.Vehicle (1) according to one of Claims 1 to 5, characterized in that the gerangication means (7) are furthermore configured to actuate the media reproduction means (2) for reducing the volume of the gerang trail contained in the emitted song and / or for increasing the volume of the compensated co-bass.Vehicle (1) according to one of Claims 1 to 6, characterized in that the singing analysis means (5) are furthermore configured, when a temporal deviation between the singing and the co-sing (6) is detected, to bring about the output of at least one visual and / or at least one haptic indication in the vehicle interior at exactly the point in time at which the sing requires the use of the vehicle occupant (4).Vehicle (1) according to one of Claims 1 to 7, characterized in that the singing analysis means (5) are furthermore configured to evaluate the deviation between the singinging and the co-singing (6) objectively on the basis of firmly predefined evaluation metrics and / or to evaluate it subjectively by means of a machine learning model, wherein the machine learning model has been trained with the subjective evaluations of the co-sing (6) of a multiplicity of test horns in measurement campaigns for selected carried out leders.Method for determining characteristic lip movement patterns, characterized bythe following method steps: - transmitting, by a computing unit (9) provided in a vehicle according to one of Claims 4 or 5 to 8 with reference to Claim 4, an assignment of lip movements and sangling features to a computing device (11) external to the vehicle, wherein the computing unit (9) receives music features from the media reproduction means (2) and extracts the sangling features to be assigned to the lip movements from the music features; - aggregating the assignments relating to a plurality of different computing units (9) in a correspondence database (12) by the computing device (11); - examining the aggregated assignments for characteristic patterns in order to identify lip movement patterns characteristic for specific sangling features, by the computing device (11); and - providing the lip movement patterns identified in this way for retrieval by the respective computing units (9).

Images