Vehicle-mounted multimedia playing control method and device and computer readable storage medium

Abstract

The invention provides a vehicle-mounted multimedia playing control method and device and a computer readable storage medium, and the method comprises the steps: obtaining the passenger state and position information of a target passenger in a vehicle through the original hardware on the vehicle, determining a sound field control strategy corresponding to the passenger state through the passenger state, and carrying out the sound field control through the sound field control strategy. A passenger sleep state and a sound field control strategy are judged by using a passenger state detection module, and dynamic audio partition control is performed on partitions in a vehicle in combination with position information acquired by acoustic positioning. And based on the sound field control strategy and an acoustic transfer function corresponding to the position information of the target passenger, generating a gain coefficient matrix, and adjusting parameters of multimedia playing equipment through the gain coefficient matrix so as to dynamically adjust a target sound field corresponding to the position information of the target passenger. And automation and intelligentization of sound field control are realized. According to the scheme, intelligent sound field control without newly adding hardware is realized through software algorithm upgrading.

Description

Technical Field

[0001] This application relates to the field of automotive technologies, and in particular, to a method, device, and computer-readable storage medium for controlling in-vehicle multimedia playback. Background Art

[0002] In long-distance driving scenarios, the in-vehicle audio system needs to balance the functional requirements of the driver (such as navigation prompts, music playback to stay alert) and the comfort requirements of passengers (such as a rest environment free from interference). Traditional solutions rely on physical acoustic isolation technologies and additional hardware. For example, integrating a directional speaker system within the driver's headrest to attempt to create a local private sound field. However, this technology has significant limitations: high cost and lack of flexibility as it only supports fixed sound field modes and is difficult to adapt to different usage scenarios. Summary of the Invention

[0003] To overcome the problems in the related art, this specification provides a method, device, and computer-readable storage medium for controlling in-vehicle multimedia playback.

[0004] According to the first aspect of the embodiments of this specification, a method for controlling in-vehicle multimedia playback is provided. The method includes:

[0005] Obtaining the passenger state and position information of a target passenger in the vehicle;

[0006] Determining a sound field control strategy corresponding to the passenger state;

[0007] Generating a gain coefficient matrix based on the acoustic transfer function corresponding to the sound field control strategy and the position information of the passenger, for adjusting the parameters of the multimedia playback device through the gain coefficient matrix to dynamically adjust the target sound field corresponding to the position information of the target passenger.

[0008] According to a method for controlling in-vehicle multimedia playback provided by this application,

[0009] The obtaining the passenger state and position information of a target passenger in the vehicle includes:

[0010] Using a biometric sensing device in the vehicle to obtain visual data of the target passenger;

[0011] Performing multi-feature fusion decision on the visual data based on a neural network model of time series to determine the passenger state of the target passenger.

[0012] According to a method for controlling in-vehicle multimedia playback provided by this application, after determining the passenger state of the target passenger, the method further includes:

[0013] Setting a state label for the passenger state;

[0014] Extract the position information of the target passenger and the status label from the visual data, and discard the original visual data.

[0015] According to a vehicle-mounted multimedia playback control method provided by the present application,

[0016] The visual data includes status data of at least one of the head, eyes, and limbs of the target passenger.

[0017] According to a vehicle-mounted multimedia playback control method provided by the present application,

[0018] The obtaining of the passenger status and position information of the target passenger in the vehicle includes:

[0019] Collect the sound in the vehicle through the acoustic sensor in the vehicle, and determine the position information of the passenger based on the acoustic positioning algorithm.

[0020] According to a vehicle-mounted multimedia playback control method provided by the present application,

[0021] Based on a neural network model of deep learning, predict an acoustic transfer function that characterizes the acoustic characteristics when sound waves travel from the multimedia playback device to each seat area in the vehicle.

[0022] According to a vehicle-mounted multimedia playback control method provided by the present application,

[0023] The method further includes:

[0024] When the environment in the vehicle changes, collect noise data through the acoustic sensor on the vehicle;

[0025] Based on the noise data, reversely optimize the parameters of the transfer function model.

[0026] According to a vehicle-mounted multimedia playback control method provided by the present application,

[0027] The reversely optimizing the parameters of the transfer function model includes:

[0028] Adjust the parameters of the transfer function model through the gradient descent algorithm.

[0029] According to a vehicle-mounted multimedia playback control method provided by the present application,

[0030] The passenger status includes a sleeping state and a non-sleeping state.

[0031] Determine the sound field control strategy corresponding to the passenger status, including:

[0032] When the passenger status is the sleeping state, the sound field control strategy is to perform directional noise reduction on the sound field.

[0033] When the passenger state is a non-sleep state, the sound field control strategy is to perform directional enhancement on the sound field.

[0034] According to a vehicle-mounted multimedia playback control method provided by the present application,

[0035] The directional enhancement includes:

[0036] Based on the beamforming algorithm, according to the position information of the target passenger in the vehicle, determine the weight parameters of each speaker in the multimedia playback device to adjust the sound pressure of the media playback device for enhancing the target sound field;

[0037] The position of the speaker corresponds to the position of the target passenger;

[0038] And / or,

[0039] Adjust the frequency response curve according to the head position of the target passenger.

[0040] According to a vehicle-mounted multimedia playback control method provided by the present application,

[0041] The directional noise reduction includes:

[0042] Use the acoustic sensor on the vehicle to collect the environmental noise signal in the area where the target passenger is located;

[0043] Process the collected environmental noise signal through the least mean square algorithm to generate a sound wave signal with a phase opposite to that of the environmental noise signal;

[0044] Play the generated sound wave signal to the target sound field of the target passenger through the multimedia playback device;

[0045] And / or,

[0046] Construct a sound wave dissipation area in the target sound field to reduce the energy of the noise.

[0047] According to a vehicle-mounted multimedia playback control method provided by the present application,

[0048] The method further includes:

[0049] Segment the seat area in the vehicle to obtain multiple areas;

[0050] Determine the sound field control strategy corresponding to the passenger state, including:

[0051] When there are multiple passengers in the area, determine the passenger states of the multiple passengers;

[0052] Determine the sound field control strategy according to the passenger state with the largest proportion as the demand.

[0053] The present application also provides an in-vehicle multimedia playback control device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the in-vehicle multimedia playback control method described in any one of the above is implemented.

[0054] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the in-vehicle multimedia playback control method described in any one of the above is implemented.

[0055] In the embodiments of this specification, compared with the current method of relying on additional hardware to implement fixed sound field control, the in-vehicle multimedia playback control method, device, and computer-readable storage medium utilize the original hardware on the vehicle to obtain the passenger status and position information of the target passenger inside the vehicle. Based on the passenger status, the sound field control strategy corresponding to the passenger status is determined. The passenger sleep status and the sound field control strategy are judged by the occupant status detection module, and combined with the position information obtained by acoustic positioning, dynamic audio zone control is performed on the zones inside the vehicle. Based on the acoustic transfer function corresponding to the sound field control strategy and the position information of the target passenger, a gain coefficient matrix is generated, and the parameters of the multimedia playback device are adjusted through the gain coefficient matrix to dynamically adjust the target sound field corresponding to the position information of the target passenger, realizing the automation and intelligence of sound field control. This solution realizes intelligent sound field control without adding new hardware through software algorithm upgrade.

[0056] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with this specification, and are used together with the specification to explain the principles of this specification.

[0058] Figure 1 is a flowchart of an in-vehicle multimedia playback control method shown according to an exemplary embodiment of this specification;

[0059] Figure 2 is a schematic diagram of an in-vehicle multimedia playback control system shown according to an exemplary embodiment of this specification;

[0060] Figure 3 is another flowchart of an in-vehicle multimedia playback control method shown according to an exemplary embodiment of this specification;

[0061] Figure 4 is a schematic diagram of an in-vehicle multimedia playback control device shown according to an exemplary embodiment of this specification;

[0062] Figure 5 It is a schematic block diagram of an in-vehicle multimedia playback control device shown in this specification according to an exemplary embodiment. Specific embodiments

[0063] Here, in combination with the accompanying drawings, the technical solutions in the embodiments (or "embodiments") of the present application will be clearly and completely described. When the following description involves the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0064] If there are terms related to directional indications or positional relationships in the embodiments of the present application (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, lateral, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationships and motion conditions between components in a certain specific posture (as shown in the accompanying drawings); if this specific posture changes, then the directional indication or positional relationship also changes accordingly. In addition, terms such as "first" and "second" involved in the embodiments of the present application are only for the purpose of convenient description and cannot be understood as indicating or implying relative importance.

[0065] The present application provides an in-vehicle multimedia playback control method, device, and computer-readable storage medium. The following will describe the present application in detail in combination with the accompanying drawings. Without conflict, the features in the following embodiments and embodiments can be combined with each other.

[0066] During vehicle driving, the audio output of the in-vehicle audio (such as music, navigation prompts) may interfere with the rest environment of the co-pilot or rear passengers. Existing solutions (such as headrest speakers) rely on additional hardware, with high costs and insufficient flexibility.

[0067] To solve the above technical problems, this specification provides an in-vehicle multimedia playback control method.

[0068] It aims to dynamically adjust the in-vehicle sound field through software algorithm optimization and multi-modal data fusion, without adding new hardware, using existing in-vehicle devices to achieve intelligent sound field control, reduce costs, and accurately control the audio propagation range according to the needs of passengers to improve the user experience.

[0069] Specifically, using biological perception devices such as in-vehicle cameras, combined with image processing algorithms and face recognition technology, visual detection (sleep state) is achieved; the positions of passengers are determined through an acoustic positioning system; dynamic sound field control technology is used to fuse multi-modal data. Develop a real-time adaptive algorithm for dynamic audio zone regulation based on passenger status and position, create a dedicated sound field, and achieve visual processing with low privacy risks.

[0070] Figure 1 It is a schematic flowchart of a vehicle multimedia playback control method provided by an embodiment of this specification, including the following steps:

[0071] Step S110, obtain the passenger status and position information of the target passenger in the vehicle;

[0072] Step S120, determine the sound field control strategy corresponding to the passenger status;

[0073] Step S130, generate a gain coefficient matrix based on the sound field control strategy and the acoustic transfer function corresponding to the position information of the passenger, and use the gain coefficient matrix to adjust the parameters of the multimedia playback device to dynamically adjust the target sound field corresponding to the position information of the target passenger.

[0074] As an example, referring to Figure 2 , the vehicle multimedia playback control system includes an in-cabin biological monitoring system, an in-vehicle audio system, an acoustic positioning system, and an in-vehicle algorithm logic control system. The vehicle includes an in-cabin camera array and an audio system, and their structures are the same as those of the hardware on the market, so they will not be elaborated here.

[0075] Among them, the in-cabin camera array: The in-vehicle camera array deploys wide-angle cameras, supports infrared night vision (supports night scenes), and is used to collect passenger status data. Audio system: A multi-channel speaker array (such as a 5.1-channel or customized partitioned speaker), supports independent sound field control, and realizes personalized audio output. Edge computing unit: An in-vehicle computing module integrating GPU / TPU (used to process visual data in real time).

[0076] In step S110, obtain the passenger status and position information of the target passenger in the vehicle.

[0077] As an example, the target passengers in the vehicle include the driver and other occupants.

[0078] During the driving of the vehicle, if the co-pilot or the rear passengers are in a resting state, the sound played by the in-vehicle audio may disturb them, and manually adjusting the volume or turning off the audio will interfere with the driver's or other people's auditory experience. Therefore, determine the passenger status of each passenger in the vehicle and perform dynamic sound field control on the passengers in different passenger statuses. At the same time, determine the position information of each passenger, and according to the in-vehicle space structure and the passenger position, according to the sound source and method, through the dynamic audio zoning control algorithm, form surround sound at a certain position, so that the propagation of audible sound also has directivity. Create an independent and optimized acoustic environment for each passenger in the vehicle, that is, the so-called exclusive sound field.

[0079] As an example, divide the seat area in the vehicle to obtain multiple areas.

[0080] Use YOLO or Mask R-CNN to detect the positions of the passengers in the vehicle and segment the seat area to obtain multiple regions. For example, the vehicle interior is divided into regions such as the front left, front right, rear left, and rear right according to the seat positions of the passengers.

[0081] Based on the acoustic characteristics of each region, establish an in-vehicle sound field model so that each region corresponds to an independent sound field, and subsequently, dynamically adjust the in-vehicle sound field according to the positions and states of the occupants to achieve personalized audio playback.

[0082] In some embodiments, the obtaining of the passenger status and position information of the target passenger in the vehicle includes:

[0083] Use the biometric sensing device in the vehicle to obtain the visual data of the target passenger;

[0084] Perform multi-feature fusion decision on the visual data based on a neural network model of time series to determine the passenger status of the target passenger.

[0085] Use biometric sensing devices such as in-vehicle cameras to continuously monitor the status of passengers other than the driver. Through advanced image processing algorithms and face recognition technologies, accurately capture the eye movements of the passengers. When it is found that someone closes their eyes continuously for more than a preset time (e.g., 5 minutes), yawns, has their head droop or sway from side to side, the system can quickly and accurately identify that they are in a sleeping state, providing a reliable basis for subsequent sound field adjustment.

[0086] As an example, the visual data includes the status data of at least one of the head, eyes, and limbs of the target passenger.

[0087] Among them, use the head pose estimation algorithm to detect the six degrees of freedom of the head (three rotational degrees of freedom and three translational degrees of freedom) to determine whether the occupant is in a supine or head-down position, etc. If the occupant maintains a supine or head-down position for a long time and the eye state conforms to the sleep characteristics, it is used as a basis for judging sleep. The head pose estimation algorithm can be but is not limited to the 6DoF pose detection algorithm.

[0088] Use the eye state classification algorithm to judge the eye state in combination with the blink frequency and the duration of eye closure. For example, the PERCLOS algorithm judges whether a person is fatigued or sleeping by calculating the proportion of the time the eyes are closed within a certain period.

[0089] Use the body micro-motion analysis algorithm to detect the movement of the limbs and calculate the duration of limb stillness. If the limbs are still for a long time, combined with other features, it can assist in judging whether the occupant is in a sleeping state. The body micro-motion analysis algorithm can be but is not limited to the optical flow method.

[0090] After obtaining multi-feature data such as head posture, eye state, and body micro-movement through the above algorithm, fusion is performed based on multi-feature fusion decision-making and decision-making based on time-series data to comprehensively judge the probability of the occupant sleeping. Among them, the multi-feature fusion decision-making is based on the LSTM model (Long Short-Term Memory, a type of recurrent neural network model) of time series.

[0091] The probability of the occupant sleeping is used to feedback data on the likelihood of the passenger being in a sleeping state. By comparing the probability of the occupant sleeping with a set threshold, the passenger state of the passenger can be determined.

[0092] In step S120, according to the passenger state, the sound field control strategy for the target sound field corresponding to the target passenger is determined.

[0093] As an example, the passenger state includes a sleeping state and a non-sleeping state.

[0094] When the probability of the occupant sleeping calculated based on the visual data of the target passenger is less than the threshold, the system believes that the likelihood of the passenger being in a sleeping state is relatively low, and the passenger state is determined to be a non-sleeping state. When the passenger state is a sleeping state, the sound field control strategy is to perform directional noise reduction on the sound field. This is because in this case, the passenger may not have fully entered deep sleep. Retaining a certain amount of high-frequency ambient sound can not only reduce interference to them but also allow the passenger to perceive the surrounding environment to a certain extent, avoiding discomfort or potential danger caused by sudden complete noise reduction. Measures for directional noise reduction include but are not limited to reducing the noise reduction intensity and adopting a strategy of only weakening the middle frequency band and retaining high-frequency ambient sound.

[0095] Conversely, when the probability of the occupant sleeping calculated based on the visual data of the target passenger is greater than the threshold, the system believes that the likelihood of the passenger being in a sleeping state is relatively high, and the passenger state is determined to be a sleeping state. When the passenger state is a sleeping state, the sound field control strategy is to perform directional enhancement on the sound field.

[0096] When there are multiple passengers in the vehicle and their states are different, the conflict of audio control is resolved according to the multi-passenger conflict resolution algorithm, and the audio resources are reasonably allocated. Among them, the multi-passenger conflict resolution algorithm dynamically adjusts the audio of different passenger areas according to the priority rules.

[0097] As an example, the priority rules include safety first, majority first, etc.

[0098] As an example, safety first includes always retaining the navigation prompt sound in the driver's area.

[0099] As an example, determining the sound field control strategy corresponding to the passenger state includes:

[0100] When there are multiple passengers in the area, determine the passenger states of the multiple passengers; based on the passenger state with the largest proportion as the demand, determine the sound field control strategy.

[0101] In the same passenger area (such as the rear row), if the majority of the passengers are not in the sleeping state, increase the sound pressure in their area proportionally, and activate noise reduction in the area of the minority passengers. Regional sound pressure weight distribution: If only one of the two passengers in the rear row is awake, the sound pressure in the sub-area where he is located is increased by 20% (for example, the reference sound pressure of 60 dB is increased to 72 dB). The sound pressure in the area of the majority passengers (such as two of the three are awake) is increased by 30%, and active noise reduction is activated in the minority area.

[0102] In other examples, based on the multi-passenger sound field equilibrium algorithm of game theory, on the basis of safety priority, balance the sound pressure demands of different passengers through mathematical modeling to maximize the overall comfort in the vehicle.

[0103] In this embodiment, it is allowed to control the on / off of each sound field separately. When it is detected that a passenger is resting, his exclusive sound field can be automatically turned off without affecting the auditory experience of other passengers, realizing personalized customization and precise regulation of the sound field.

[0104] In step S130, based on the acoustic transfer function corresponding to the sound field control strategy and the position information of the passenger, generate a gain coefficient matrix, which is used to adjust the parameters of the multimedia playback device through the gain coefficient matrix to dynamically adjust the target sound field corresponding to the position information of the target passenger.

[0105] When predicting the acoustic transfer function, use CNN to extract the spatial features of acoustic data, and use LSTM to process the time series features. Through learning a small amount of measured data and simulation data, predict the acoustic transfer function from different coordinates to the speaker. Specifically, obtain the measured data of 5 set reference points, and based on the measured data, combine acoustic theory and mathematical models to simulate acoustic data under different in-vehicle acoustic environments. The in-vehicle acoustic environment includes but is not limited to seat layout, material sound absorption coefficient, window opening and closing state, etc.

[0106] For example, by adjusting the parameters in the acoustic propagation formula to simulate different scenario changes, rather than collecting a large amount of on-site data. This method can greatly reduce the actual measurement workload, and can effectively cover a variety of possible scenarios, improving the generality of the model. For different scenarios, instead of establishing completely independent models, under the same model framework, adapt to different scenario changes by adjusting the model input parameters or training data. For example, add data under different scenarios when training the model, so that the model learns the acoustic feature laws of different scenarios, thereby having the ability to process multiple scenarios.

[0107] In practical applications, fusion data such as the position and status of passengers is obtained through an acoustic positioning system and a camera. By combining the analysis of the interior space structure of the vehicle, the acoustic characteristics of each area are determined. Based on a neural network model of deep learning, an acoustic transfer function is predicted, which characterizes the acoustic characteristics when sound waves travel from the multimedia playback device to each seat area in the vehicle.

[0108] In some other embodiments, when the interior acoustic environment of the vehicle changes, noise data is collected through the acoustic sensors on the vehicle; based on the noise data, the parameters of the transfer function model are optimized in reverse.

[0109] As an example, the parameters for optimizing the transfer function model in reverse include: adjusting the parameters of the transfer function model through a gradient descent algorithm.

[0110] When a change in the interior acoustic environment of the vehicle is detected (such as temperature, window status), calibration is initiated. The current actual sound pressure data is collected through a microphone array, and the parameters of the CNN-LSTM model are adjusted through a gradient descent algorithm to make the predicted transfer function closer to the measured data and minimize the error. The sound field parameters are regenerated using the updated model to optimize the effect.

[0111] In this embodiment, the purpose of the dynamic environment calibration mechanism is to optimize the sound field model in real time to ensure that the acoustic control algorithm adapts to changes in the interior environment of the vehicle (such as window opening and closing, passenger movement). Its core is to collect environmental noise through on-vehicle microphones and adjust the parameters of the acoustic transfer function model in reverse to ensure the accuracy of sound field control.

[0112] Next, the transfer function prediction model is called to generate a gain coefficient matrix G for each speaker in the multimedia playback device. This gain coefficient matrix G determines parameters such as the intensity and phase of the sound output by each speaker and is used for subsequent adjustment of the audio signal to achieve the purpose of creating a personalized sound field for the passenger, controlling the sound propagation direction and intensity. For example, when directly reducing the volume, the sound intensity in each frequency band is uniformly reduced in proportion; when adjusting the frequency response, the degree of enhancement or attenuation of sounds at different frequencies is changed, such as boosting high frequencies and reducing low frequencies.

[0113] In some embodiments, according to the fused multi-modal data, the sound field dynamic control module adopts a zoning hybrid control strategy. For the sleep area where the passenger status in the area is in a sleeping state, directional noise reduction is performed on the sound field. For example, adaptive active noise cancellation (AANC) is used in combination with the acoustic black hole effect to reduce sound interference.

[0114] As an example, an acoustic sensor on the vehicle is used to collect the ambient noise signal in the area where the target passenger is located; the collected ambient noise signal is processed to generate a sound wave signal with a phase opposite to that of the ambient noise signal; and the generated sound wave signal is played to the target sound field of the target passenger through the multimedia playback device.

[0115] After the amplitude and phase of the original audio signal are adjusted using a gain coefficient matrix, the inverted waveform is obtained by taking the inverse (Anti_Signal = -G * Original_Audio). This process focuses on changing the phase of the audio. The generated inverted sound wave is played through a speaker to the sleeping area. When the generated inverted sound wave meets the original noise in space, due to their opposite phases, they will interfere with and cancel each other, thereby reducing the noise intensity in the sleeping area. It should be noted that the system continuously collects noise signals, calculates inverted sound waves, and outputs them, forming a real-time adaptive noise reduction process to cope with the changing noise environment.

[0116] As another example, a sound wave dissipation area is constructed in the target sound field to reduce the energy of the noise.

[0117] In the high-frequency band (frequency band > 1 kHz), by carefully designing the layout of the speakers and using technical means such as acoustic metamaterials, a sound wave dissipation area, i.e., an acoustic black hole area, is constructed in the sleeping area. These acoustic metamaterials have special physical properties that can guide sound waves to converge to a specific area and gradually consume the energy of the sound waves in that area.

[0118] Through the above method, the interference of high-frequency noise on sleeping passengers is effectively reduced.

[0119] In other examples, for ultra-low frequencies that cannot be canceled (such as frequency band < 100 Hz), low-volume white noise is added to avoid the abrupt feeling of silence.

[0120] In some other embodiments, for the non-sleeping area where the passenger state in the area is in a non-sleeping state, the sound field is directionally enhanced. For example, the sound quality is enhanced through focused beamforming, and the frequency response curve is adjusted according to the position of the passenger's head.

[0121] As an example, based on the beamforming algorithm, according to the position information of the target passenger in the vehicle, the weight parameters of each speaker in the multimedia playback device are determined to adjust the sound pressure of the media playback device to enhance the target sound field; the position of the speaker corresponds to the position of the target passenger.

[0122] Suppress multipath interference and adjacent cell leakage through the MVDR (Minimum Variance Distortionless Response) beamforming algorithm, focus the sound on the target area, precisely control the sound wave phase and amplitude of the speaker, and form a directional sound beam (such as only the driver hears the navigation sound). The beamforming algorithm in this specification calculates the weight coefficient of each speaker according to the passenger position information, so that the sounds emitted by each speaker are in-phase superimposed at the target sound field (the position where the passenger is located), thereby enhancing the sound pressure in this area. During the calculation process, the algorithm will consider the complex acoustic environment in the vehicle, such as factors like sound reflection and refraction, to optimize the sound propagation path, minimize interference to the greatest extent, and ensure that the sound can be effectively focused on the passenger position.

[0123] As another example, adjust the frequency response curve according to the head position of the target passenger.

[0124] Since the posture of passengers in the vehicle may change, sounds of different frequencies will be affected differently by the vehicle environment (such as seats, interior trims, etc.) during propagation, and there are also differences in the perception of sound frequencies by passengers at different positions. For example, when a passenger's head is close to a certain speaker, the sounds of some frequencies may be over-amplified, and at this time the system will correspondingly reduce the gain of these frequencies; conversely, for some frequencies with greater attenuation, the gain is appropriately increased.

[0125] Therefore, use an in-vehicle camera or other sensors to real-time track the change of the passenger's head position, and dynamically adjust the frequency response curve of the audio signal according to the obtained passenger head position data. To optimize the frequency distribution of the sound, make the sound heard by the passenger clearer and more natural, and enhance the overall auditory experience.

[0126] Among them, based on the position information of the target passenger, find the beam direction that can maximize the signal-to-noise ratio (SNR, Signal-Noise Ratio), and precisely focus the sound on the position where the passenger is located based on this beam direction, enhance the intensity of the useful signal received by the passenger, and at the same time suppress the noise and interference signals coming from other directions. For example, in the non-sleeping area, let the passenger hear the audio more clearly.

[0127] In other examples, to avoid the discomfort caused by sudden changes in volume during the control process, the system will real-time monitor the amplitude change of the audio signal, and by processing the amplitude of the audio signal, avoid sudden large changes in volume. For example, when it is detected that the amplitude change of the audio signal is too large and may cause a sudden change in volume, appropriately attenuate the part with a larger amplitude and moderately increase the part with a smaller amplitude.

[0128] In this embodiment, maintain the comfort and stability of in-vehicle audio playback and enhance the user experience.

[0129] In the above embodiments, by optimizing the layout and acoustic design of the speaker, it is ensured that the audio signal is mainly concentrated within the audible range of the driver, thereby reducing the impact on other passengers. And through the automated biometric perception and audio adjustment process, the need for manual intervention by the user is reduced, and passengers can obtain a comfortable driving environment without cumbersome operations.

[0130] After adjusting the target sound field, the actual sound pressure distribution is detected by the microphone array. If the noise reduction of the target sound field does not meet the expectation (for example, the error > 3 dB), an operation to recalibrate the transfer function model is triggered. And, the passenger status of the target passenger is requested for review to prevent continuous noise reduction caused by misjudgment.

[0131] In some embodiments, to ensure driving safety and avoid potential safety hazards that may be brought to the driver due to frequent adjustment of equipment, further processing is performed on the visual data.

[0132] As an example, after determining the passenger status of the target passenger, the method further includes:

[0133] Setting a status label for the passenger status; extracting the position information and the status label of the target passenger in the visual data, and discarding the original visual data.

[0134] The visual data is only processed locally. After the detection is completed, the original image is discarded, and only the coordinates and the status label are retained. At the same time, it supports passengers to manually turn off the detection function to ensure that passengers have control over their own privacy.

[0135] It should be noted that the privacy protection mechanism runs through the whole process to ensure that the privacy of passengers is not leaked during the data collection and processing process, and to ensure the security and reliability of the system.

[0136] Through this embodiment, potential safety hazards that may be brought to the driver due to frequent adjustment of equipment are avoided, driving safety is ensured, the driver can be more focused on driving, and the operation efficiency and safety of the vehicle are improved.

[0137] Refer to Figure 3 , Figure 3 which is another process schematic diagram of a vehicle-mounted multimedia playback control method provided in this specification.

[0138] 1) Detect the status of the target sound field of the occupant: including checking whether the target sound field is turned on.

[0139] If the sound field is turned on, continue with the following process; if the sound field is turned off, end the process.

[0140] 2) Detect the occupant detection permission: confirm whether to allow the use of the camera to detect the passenger status (privacy protection).

[0141] If the permission is enabled, start the occupant status detection program; if the permission is not enabled, prompt the driver and end the process.

[0142] 3) Occupant status detection: Continuously monitor the duration of the passenger's eyes closed, head posture, etc., and determine whether the passenger enters the sleep state.

[0143] Continuously monitor the occupant status.

[0144] If it is detected that the occupant status meets the sleep state, proceed to the next step.

[0145] 4) Prompt the driver:

[0146] Through the voice TTS broadcast of the driver-exclusive sound field and the pop-up window on the central control large screen, prompt the driver to "turn off the occupant target sound field".

[0147] 5) Driver confirmation:

[0148] If the driver confirms "turn off" through voice or pop-up window, then execute the next step.

[0149] If the driver does not confirm, continue to monitor the occupant status.

[0150] 6) Turn off the occupant target sound field:

[0151] Turn off the occupant target sound field, stop the sound playback in this area, and start noise reduction.

[0152] 7) Recording and logging:

[0153] Record the log of the entire event, including the time points of occupant status detection, driver confirmation, and sound field closure.

[0154] 8) End of the process:

[0155] After the occupant target sound field is turned off, the process ends and waits for the next detection cycle.

[0156] Based on the above embodiments, referring to Figure 1 , the in-cabin biological monitoring system detects the status of the vehicle occupants through various algorithms and models, and transmits the detection results (such as whether sleeping, sleep probability, etc.) to the in-vehicle logic control system. The in-vehicle logic control system combines the passenger position information obtained by the acoustic positioning system and the occupant status detection results, and uses technologies such as sound field mapping optimization and beamforming optimization to precisely control the sounds in each area of the vehicle through the speakers of the audio system, realizing dynamic audio zone control. All parties cooperate with each other to form a complete independent sound field playback control system.

[0157] The present application provides a method, a device, and a computer-readable storage medium for in-vehicle multimedia playback control. Compared with the current method that relies on additional hardware to implement fixed sound field control, the original hardware on the vehicle is utilized to obtain the passenger state and position information of the target passenger in the vehicle. Based on the passenger state, the sound field control strategy corresponding to the passenger state is determined. The passenger sleep state and the sound field control strategy are judged by using the occupant state detection module, and combined with the position information obtained by acoustic positioning, dynamic audio zone control is performed on the zones in the vehicle. Based on the acoustic transfer function corresponding to the sound field control strategy and the position information of the target passenger, a gain coefficient matrix is generated, and the parameters of the multimedia playback device are adjusted through the gain coefficient matrix to dynamically adjust the target sound field corresponding to the position information of the target passenger, realizing the automation and intelligence of sound field control. This solution realizes intelligent sound field control without adding new hardware through software algorithm upgrade.

[0158] Based on the same inventive concept as the above method, an embodiment of the present application also proposes an in-vehicle multimedia playback control device, as Figure 4 shown.

[0159] The device includes:

[0160] A data acquisition module 602, configured to acquire the passenger state and position information of the target passenger in the vehicle;

[0161] A decision-making module 604, configured to determine the sound field control strategy corresponding to the passenger state;

[0162] A sound field control module 606, configured to generate a gain coefficient matrix based on the acoustic transfer function corresponding to the sound field control strategy and the position information of the target passenger, and configured to adjust the parameters of the multimedia playback device through the gain coefficient matrix to dynamically adjust the target sound field corresponding to the position information of the target passenger.

[0163] For the specific implementation process of the functions and effects of each module / sub-module / unit in the above device, please refer to the implementation process of the corresponding steps in the above method, which can achieve the same technical effects and will not be elaborated here.

[0164] Figure 5 An example of the physical structure diagram of an in-vehicle multimedia playback control device is shown as Figure 5As shown in the figure, the in-vehicle multimedia playback control device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the in-vehicle multimedia playback control method.

[0165] In addition, when the logic instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0166] On the other hand, the present application also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the in-vehicle multimedia playback control method provided by the above-mentioned various methods.

[0167] On yet another hand, the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the in-vehicle multimedia playback control method provided by the above-mentioned various methods.

[0168] It should be noted that the technical solutions or technical features described in the above embodiments may be combined or supplemented with each other without conflict. The scope of protection of the present application is not limited to the precise structures described in the above embodiments and shown in the drawings; all modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A vehicle-mounted multimedia playback control method, characterized in that, The method includes: Obtaining the passenger status and location information of a target passenger in the vehicle; Determining the sound field control strategy corresponding to the passenger status; Generating a gain coefficient matrix based on the acoustic transfer function corresponding to the sound field control strategy and the location information of the target passenger, for adjusting the parameters of the multimedia playback device through the gain coefficient matrix to dynamically adjust the target sound field corresponding to the location information of the target passenger.

2. The in-vehicle multimedia playback control method according to claim 1, wherein: The obtaining of the passenger status and location information of a target passenger in the vehicle includes: Using the biometric sensing device in the vehicle to obtain the visual data of the target passenger; Performing multi-feature fusion decision on the visual data based on a neural network model of time series to determine the passenger status of the target passenger.

3. The vehicle-mounted multimedia playback control method according to claim 2, wherein, After determining the passenger status of the target passenger, the method further includes: Setting a status label for the passenger status; Extracting the location information and the status label of the target passenger in the visual data, and discarding the original visual data.

4. The in-vehicle multimedia playback control method according to claim 2, wherein: The visual data includes the status data of at least one of the head, eyes, and limbs of the target passenger.

5. The in-vehicle multimedia playback control method according to claim 2, wherein: The obtaining of the passenger status and location information of a target passenger in the vehicle includes: Collecting the sounds in the vehicle through the acoustic sensors in the vehicle, and determining the location information of the passenger based on the acoustic positioning algorithm.

6. The in-vehicle multimedia playback control method according to claim 1, wherein: Predicting an acoustic transfer function representing the acoustic characteristics when sound waves travel from the multimedia playback device to each seat area in the vehicle based on a neural network model of deep learning.

7. The in-vehicle multimedia playback control method according to claim 6, wherein: The method further includes: When the in-vehicle acoustic environment of the vehicle changes, collecting noise data through the acoustic sensors on the vehicle; Based on the noise data, reversely optimizing the parameters of the transfer function model.

8. The in-vehicle multimedia playback control method according to claim 7, wherein: The reversely optimizing the parameters of the transfer function model includes: Adjusting the parameters of the transfer function model through the gradient descent algorithm.

9. The in-vehicle multimedia playback control method according to claim 1, wherein: The passenger status includes a sleep state and a non-sleep state, Determining the sound field control strategy corresponding to the passenger status includes: When the passenger status is the sleep state, the sound field control strategy is to perform directional noise reduction on the sound field; When the passenger status is the non-sleep state, the sound field control strategy is to perform directional enhancement on the sound field.

10. The in-vehicle multimedia playback control method according to claim 9, wherein: The directional enhancement includes: Based on the beamforming algorithm, determining the weight parameters of each speaker in the multimedia playback device according to the location information of the target passenger in the vehicle, so as to adjust the sound pressure of the media playback device for enhancing the target sound field. The position of the speaker corresponds to the position of the target passenger; and / or adjust the frequency response curve according to the head position of the target passenger.

11. The vehicle-mounted multimedia playback control method according to claim 9, wherein the directional noise reduction includes: collecting the environmental noise signal in the area where the target passenger is located by using the acoustic sensor on the vehicle; processing the collected environmental noise signal to generate a sound wave signal with a phase opposite to that of the environmental noise signal; playing the generated sound wave signal to the target sound field of the target passenger through the multimedia playback device; and / or constructing a sound wave dissipation area in the target sound field to reduce the energy of the noise.

12. The vehicle-mounted multimedia playback control method according to claim 10, wherein the method further includes: segmenting the seat area in the vehicle to obtain a plurality of areas; determining the sound field control strategy corresponding to the passenger state, including: when there are multiple passengers in the area, determining the passenger states of the multiple passengers; determining the sound field control strategy according to the passenger state with the largest proportion as the requirement.

13. A vehicle-mounted multimedia playback control device, characterized in that, It includes a memory, a processor, and a vehicle-mounted multimedia playback control program stored on the memory and executable on the processor. When the processor executes the vehicle-mounted multimedia playback control program, the steps of the vehicle-mounted multimedia playback control method according to any one of claims 1-12 are implemented.

14. A computer-readable storage medium, characterized in that, The vehicle-mounted multimedia playback control program is stored on the computer-readable storage medium. When the vehicle-mounted multimedia playback control program is executed, the steps of the vehicle-mounted multimedia playback control method according to any one of claims 1-12 are implemented.

Images