Vehicle cabin adjustment method

Abstract

This application discloses a vehicle cabin adjustment method, which relates to the field of vehicle control technology. The method comprises: obtaining an occupant's identity information; determining a first adjustment mode based on the identity information, and performing preliminary adjustments to the vehicle cabin based on the first adjustment mode; obtaining the occupant's emotional information and posture information in the first adjustment mode; determining a second adjustment mode based on the emotional information and posture information, and further adjusting the vehicle cabin based on the second adjustment mode. This application enables rapid adjustment of the cabin environment.

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and in particular to a vehicle cabin adjustment method. Background Art

[0002] The cabin environment generally requires manual adjustment by passengers, but the entire adjustment process is very cumbersome, which will affect the passengers' riding experience.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a vehicle cabin adjustment method, device, vehicle and storage medium, aiming to solve the technical problem that passengers need to manually adjust the cabin environment when riding in the vehicle, which easily affects the passengers' riding experience.

[0005] To achieve the above objectives, the present application proposes a vehicle cabin adjustment method, the vehicle cabin adjustment method comprising:

[0006] Obtaining the occupant's identity information;

[0007] determining a first adjustment mode according to the identity information, and performing preliminary adjustments to the vehicle cabin based on the first adjustment mode;

[0008] In the first adjustment mode, acquiring the occupant's emotional information and posture information;

[0009] A second adjustment mode is determined according to the emotion information and the posture information, and the vehicle cabin is readjusted based on the second adjustment mode.

[0010] In one embodiment, the step of determining the first adjustment mode according to the identity information includes:

[0011] Obtaining memory information corresponding to the identity information, wherein the memory information is a plurality of previous cabin environment information of the occupant at different riding time periods;

[0012] determining target cabin environment information for the occupant at different riding time periods based on the memory information;

[0013] determining a target riding section for the passenger's current riding time;

[0014] The target cabin environment information corresponding to the target riding section is determined as a first adjustment mode.

[0015] In one embodiment, the step of obtaining the occupant's identity information includes:

[0016] Obtaining facial features, clothing features, body shape features, and voice tone features of the occupant;

[0017] Performing multimodal feature fusion on the facial features, the clothing features, the body shape features, and the voice tone features to obtain fused features;

[0018] Inputting the fused features into a preset multimodal model to obtain a probability distribution;

[0019] The identity information of the occupant is determined according to the probability distribution.

[0020] In one embodiment, the step of obtaining the facial features, clothing features, body shape features, and voice tone features of the occupant includes:

[0021] Acquiring a facial image of the occupant, performing a sharpening process on the facial image to obtain a first facial image, determining a facial position of the occupant in the first facial image, and obtaining a second facial image after correcting the angle and lighting of the first facial image based on the facial position of the occupant, and extracting facial features from the second facial image;

[0022] acquiring an overall image of the occupant, performing standardization processing on the overall image to obtain a target overall image, and extracting color information, pattern information, clothing type information, and brand identification information from the target overall image and integrating them into clothing features;

[0023] Acquiring three-dimensional point cloud data of the occupant, determining outline information and key body shape information of the occupant based on the three-dimensional point cloud data, and determining body shape features based on the outline information and the key body shape information;

[0024] Conversation information between the passenger and the driver is obtained, denoised and separated to obtain target conversation information, and intonation information, pitch information, and volume information determined based on the target conversation information are integrated into a voice tone feature.

[0025] In one embodiment, the step of performing multimodal feature fusion on the facial features, the clothing features, the body shape features, and the voice tone features to obtain fused features includes:

[0026] Performing time alignment and spatial alignment on the facial features, clothing features, body shape features, and voice tone features to obtain target facial features, target clothing features, target body shape features, and target voice tone features, wherein the time alignment is used to synchronize audio data with video data, and the spatial alignment is used to align two-dimensional data with three-dimensional data;

[0027] Multimodal feature fusion is performed on the target facial features, the target clothing features, the target body features, and the target voice tone features to obtain fused features, wherein the multimodal feature fusion includes feature-level fusion and decision-level fusion, the feature-level fusion includes one of splicing, weighted averaging, or deep neural network, and the decision-level fusion includes weighted voting.

[0028] In one embodiment, the step of inputting the fusion features into a preset multimodal model to obtain a probability distribution includes:

[0029] collecting an occupant data set, wherein the occupant data set includes occupant data annotated with different identity information;

[0030] Training a multimodal model based on the occupant dataset to obtain a preset multimodal model;

[0031] The fused features are input into a preset multimodal model to obtain a probability distribution.

[0032] In one embodiment, the step of obtaining the occupant's emotional information and posture information includes:

[0033] Acquiring a facial video and a full-body video of the occupant;

[0034] Based on the facial video, determining the occupant's emotional information according to a preset deep learning model;

[0035] Based on the full-body video, the posture information of the occupant is determined according to a preset posture detection algorithm.

[0036] In one embodiment, the step of determining the occupant's emotional information based on the facial video according to a preset deep learning model includes:

[0037] Extracting local features and global features from the facial image of the facial video, wherein the local features are used to represent changes in facial features of the occupant, and the global features are used to represent changes in facial contours of the occupant;

[0038] The local features and the global features are input into the preset deep learning model to obtain the emotional information of the occupant.

[0039] In one embodiment, the step of determining the posture information of the occupant based on the full-body video according to a preset posture detection algorithm includes:

[0040] identifying body key points in a full-body image of the full-body video;

[0041] Analyze the geometric relationship between key points of the body to obtain key point features;

[0042] Determine dynamic features based on body key points in full-body images at different moments;

[0043] The key point features and the dynamic features are input into a preset time series model to obtain the posture information of the occupant.

[0044] In one embodiment, the step of determining the second adjustment mode according to the emotion information and the posture information includes:

[0045] determining a current state of the occupant based on the emotion information and the posture information;

[0046] The cockpit setting information of the current state is searched, and the second adjustment mode is determined based on the cockpit setting information.

[0047] In addition, to achieve the above-mentioned purpose, the present application also proposes a vehicle cabin adjustment device, the vehicle cabin adjustment device comprising:

[0048] An acquisition module, used to obtain the identity information of the occupants;

[0049] a determination module, configured to determine a first adjustment mode according to the identity information, and perform preliminary adjustments to the vehicle cabin based on the first adjustment mode;

[0050] The acquisition module is further configured to acquire the occupant's emotional information and posture information in the first adjustment mode;

[0051] The determination module is further configured to determine a second adjustment mode according to the emotion information and the posture information, and to adjust the vehicle cabin again based on the second adjustment mode.

[0052] In addition, to achieve the above-mentioned purpose, the present application also proposes a vehicle, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the vehicle cabin adjustment method as described above.

[0053] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the vehicle cabin adjustment method described above are implemented.

[0054] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the vehicle cabin adjustment method as described above.

[0055] One or more technical solutions proposed in this application have at least the following technical effects:

[0056] The vehicle cabin adjustment method proposed in this application obtains the identity information of the occupant; determines a first adjustment mode based on the identity information, and performs preliminary adjustments to the vehicle cabin based on the first adjustment mode; obtains the emotional information and posture information of the occupant in the first adjustment mode; determines a second adjustment mode based on the emotional information and posture information, and adjusts the vehicle cabin again based on the second adjustment mode. This solves the problem that the occupant's riding experience is easily affected by the need to manually adjust the cabin environment when riding. Compared with the existing technology, this application can preliminarily determine and adjust the first adjustment mode based on the occupant's identity information, and then dynamically adjust the cabin environment settings by real-time monitoring of the occupant's emotional information and posture information. The occupant does not need to manually adjust the cabin environment. This not only enables rapid adjustment of the cabin environment, but also more accurately captures the changes in the occupant's status to provide real-time dynamic cabin adjustment, thereby improving the occupant's riding experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0059] Figure 1 A flowchart of the first embodiment of the vehicle cabin adjustment method of the present application is provided;

[0060] Figure 2 A flowchart of the second embodiment of the vehicle cabin adjustment method of the present application is provided;

[0061] Figure 3 This is a schematic diagram of the module structure of the vehicle cabin adjustment device according to an embodiment of the present application;

[0062] Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the vehicle cabin adjustment method in the embodiment of the present application.

[0063] The purpose, features and advantages of this application will be further explained with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0064] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0065] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0066] The main solution of the embodiment of the present application is: obtaining the identity information of the occupant; determining a first adjustment mode based on the identity information, and making preliminary adjustments to the vehicle cabin based on the first adjustment mode; in the first adjustment mode, obtaining the emotional information and posture information of the occupant; determining a second adjustment mode based on the emotional information and the posture information, and adjusting the vehicle cabin again based on the second adjustment mode.

[0067] In this embodiment, for ease of description, the following description will be made with vehicle identification as the execution subject.

[0068] Because the existing technology requires passengers to manually adjust the cabin environment when riding in a vehicle, it is easy to affect the passenger's riding experience.

[0069] The present application provides a solution that can preliminarily determine and adjust the first adjustment mode based on the identity information of the occupants, and then dynamically adjust the cabin environment settings by real-time monitoring of the occupants' emotional information and posture information. The occupants do not need to manually adjust the cabin environment. Not only can the cabin environment be quickly adjusted, but the occupants' riding experience can also be improved.

[0070] As can be seen from the above embodiments, the present application obtains the identity information of the passenger; determines a first adjustment mode based on the identity information, and performs preliminary adjustments to the vehicle cabin based on the first adjustment mode; obtains the emotional information and posture information of the passenger in the first adjustment mode; determines a second adjustment mode based on the emotional information and posture information, and adjusts the vehicle cabin again based on the second adjustment mode. This solves the problem that the passenger's riding experience is easily affected by the need to manually adjust the cabin environment when riding. Compared with the existing technology, the present application can preliminarily determine and adjust the first adjustment mode based on the passenger's identity information, and then dynamically adjust the cabin environment settings by real-time monitoring of the passenger's emotional information and posture information. The passenger does not need to manually adjust the cabin environment. This not only enables rapid adjustment of the cabin environment, but also more accurately captures the changes in the members' status to provide real-time dynamic cabin adjustment, thereby improving the passenger's riding experience.

[0071] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or vehicle capable of implementing the above functions. The following uses a vehicle as an example to illustrate this embodiment and the following embodiments.

[0072] Based on this, the embodiment of the present application provides a vehicle cabin adjustment method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the vehicle cabin adjustment method of the present application.

[0073] In this embodiment, the vehicle cabin adjustment method includes steps S10 to S40:

[0074] Step S10, obtaining the passenger's identity information;

[0075] It should be noted that the passenger's identity information refers to the relationship information between the passenger and the driver. The passenger's identity information includes boss, parents, children, lovers, friends, strangers, etc.

[0076] In a specific implementation, the identity information of the occupant can be determined by collecting the occupant's facial image, or by the conversation information between the occupant and the driver, or by the occupant's clothing or body shape.

[0077] Step S20, determining a first adjustment mode according to the identity information, and performing preliminary adjustments to the vehicle cabin based on the first adjustment mode;

[0078] It should be noted that the cabin environment of the occupant when riding in the vehicle in the past can be determined based on the occupant's identity information, so that the first adjustment mode can be determined directly based on the previous cabin environment, for example, based on the adjustment information corresponding to the previous cabin environment.

[0079] It is understandable that in each adjustment mode, multiple devices need to be adjusted, such as seat adjustment, air conditioning temperature adjustment, entertainment equipment adjustment, lighting equipment adjustment, fragrance equipment adjustment, and personalized equipment adjustment, etc.

[0080] In a specific implementation, when the passenger's identity information is the boss, the first adjustment mode can be set to:

[0081] (1) Seat adjustment: It can provide passengers with seat settings that meet their office needs, such as a higher backrest and additional lumbar support, and an adjustable small table for easy work.

[0082] (2) Air conditioning temperature adjustment: Provide a more formal and stable temperature environment to avoid overcooling or overheating, with gentle airflow and avoid direct blowing.

[0083] (3) Entertainment equipment adjustment: A quiet environment or light music may be required, and news, radio or business-related audio content may be provided.

[0084] (4) Lighting equipment adjustment: The lighting is set to bright and soft, suitable for office and reading.

[0085] (5) Fragrance equipment adjustment: Choose elegant fragrances to keep the air fresh and avoid strong odors.

[0086] (6) Personalized device adjustment: Provides meeting reminders, business information push and emergency message notifications. In a specific implementation, when the passenger's identity information is a parent, the first adjustment mode can be set to:

[0087] (1) Seat adjustment: The seat is adjusted to a comfortable rest mode, zero gravity mode, and the massage function, heating or ventilation function is turned on.

[0088] (2) Air conditioning temperature adjustment: The temperature is set to a moderate comfortable temperature, which may be more inclined to warm. The airflow can be slightly increased to provide good ventilation.

[0089] (3) Entertainment equipment adjustment: Play soothing music or audiobooks, and also provide movies and TV programs suitable for the parents' age group.

[0090] (4) Lighting equipment adjustment: Provide warm and soft lighting to create a comfortable environment.

[0091] (5) Fragrance equipment adjustment: Choose comfortable fragrances, such as lavender or woody fragrance, to enhance comfort.

[0092] (6) Personalized device adjustment: providing health tips, medication reminders, and daily activity suggestions.

[0093] In a specific implementation, when the passenger's identity information is a child, the first adjustment mode may be set to:

[0094] (1) Seat adjustment: Provides a safety seat for children, ensuring they are secure and comfortable. At the same time, the height and tilt of the seat can be adapted to the child's body shape.

[0095] (2) Air conditioning temperature adjustment: Make sure the temperature in the car is suitable for children, avoid being too cold or too hot, and the air flow is soft and even, avoiding blowing directly on the child.

[0096] (3) Entertainment equipment adjustment: play children's music, cartoons or educational content, and provide interactive games and entertainment functions.

[0097] (4) Lighting equipment adjustment: The lighting should be bright and soft in color, avoiding direct exposure to the eyes. A night light mode may be provided at night.

[0098] (5) Fragrance equipment adjustment: Choose safe and gentle fragrances, such as citrus, to ensure that you are not allergic.

[0099] (6) Personalized device adjustment: Provide learning reminders, game time management and safety tips.

[0100] In a specific implementation, when the passenger's identity information is a couple, the first adjustment mode can be set to:

[0101] (1) Seat adjustment: Temperature and airflow can be adjusted based on the occupant's personal preferences, possibly favoring a warmer setting for increased comfort.

[0102] (2) Air conditioning temperature adjustment: Temperature and airflow can be adjusted according to the occupant's personal preferences, which may be biased towards a warmer environment to increase comfort.

[0103] (3) Entertainment equipment adjustment: Play romantic music and provide movie or TV series options to increase intimacy.

[0104] (4) Lighting equipment adjustment: The lighting is soft and can be adjusted to warm tones to create a romantic atmosphere.

[0105] (5) Fragrance equipment adjustment: Choose romantic fragrances, such as rose or vanilla, to create an intimate atmosphere.

[0106] (6) Personalized device adjustment: providing appointment reminders, anniversary reminders and romantic suggestions.

[0107] In a specific implementation, when the passenger's identity information is a friend, the first adjustment mode may be set to:

[0108] (1) Seat adjustment: The seats maintain moderate comfort and provide standard spacing and privacy protection.

[0109] (2) Air conditioning temperature adjustment: The temperature and air flow are set to a more comfortable and flexible mode, suitable for interaction and communication.

[0110] (3) Entertainment equipment adjustment: play popular music or provide multi-person interactive games. The video content can be the latest movies or variety shows.

[0111] (4) Lighting equipment adjustment: The lighting is set to be bright and lively, suitable for interaction and chatting.

[0112] (5) Fragrance equipment adjustment: Choose lively and refreshing fragrances, such as mint or lemon, to increase vitality.

[0113] (6) Personalized device adjustment: providing activity suggestions, entertainment recommendations, and social interaction prompts.

[0114] In a specific implementation, when the passenger's identity information is a stranger, the first adjustment mode can be set to:

[0115] (1) Seat adjustment: The seats maintain moderate comfort and provide standard spacing and privacy protection.

[0116] (2) Air conditioning temperature control: Provide a neutral temperature and airflow setting to ensure everyone is comfortable.

[0117] (3) Entertainment equipment adjustment: Provide neutral music or broadcast, and the video content can be universal news or documentaries.

[0118] (4) Lighting equipment adjustment: The lighting should be set to neutral and moderate, providing sufficient brightness but not glaring.

[0119] (5) Adjustment of fragrance equipment: Keep the air fresh, avoid using fragrance, and ensure universality.

[0120] (6) Personalized device adjustment: Provide basic service reminders, such as destination arrival time, itinerary suggestions and safety tips.

[0121] In a feasible embodiment, the step of determining the first adjustment mode based on the identity information includes: obtaining memory information corresponding to the identity information, wherein the memory information is multiple previous cabin environment information of the occupant in different riding time periods; based on the memory information, determining the target cabin environment information of the occupant in different riding time periods; determining the target riding section of the occupant's current riding time; and determining the target cabin environment information corresponding to the target riding section as the first adjustment mode.

[0122] It should be noted that the memory information refers to the cabin environment set by the occupant when riding the vehicle in the past, and the previous cabin environment information refers to the cabin environment information of the occupant in the historical riding time period; since the occupant's riding status will change at different riding times, the setting of the occupant's cabin environment at different riding times will also be different, so the occupant's target cabin environment information at different riding time periods can be determined in advance based on the member's memory information. For example, a day can be divided into multiple riding time periods, and then all the previous environment cabin information of the occupant in different riding time periods when riding the vehicle in the past can be collected, and then the previous cabin environment information with the largest number in this riding time period can be determined as the target ring cabin environment information, or the target cabin environment information in this riding time period can be determined based on the average value of all the previous cabin environment information in this riding time period.

[0123] In this embodiment, the historical information of the occupant can be determined according to the identity information of the occupant, and then the first adjustment mode can be determined based on the historical information. In this way, the cabin environment can be adjusted after the first adjustment mode is quickly determined.

[0124] Step S30, in the first adjustment mode, obtaining the occupant's emotional information and posture information;

[0125] It should be noted that after the vehicle cabin environment is set based on the first adjustment mode, the emotional state and posture information of the occupants can be monitored in real time to dynamically adjust the cabin environment to meet the current needs of the occupants.

[0126] In a specific implementation, the facial expressions of the occupants can be analyzed based on the expression classification model to identify the occupants' current emotional states, such as happiness, anger, sadness, etc.

[0127] In a specific implementation, posture estimation algorithms can also be used to analyze the body postures of passengers, such as normal sitting posture, resting posture, working posture, and entertainment posture, etc.

[0128] In a feasible embodiment, the step of obtaining the emotional information and posture information of the occupant includes: obtaining a facial video and a full-body video of the occupant; based on the facial video, determining the emotional information of the occupant according to a preset deep learning model; based on the full-body video, determining the posture information of the occupant according to a preset posture detection algorithm.

[0129] It should be noted that emotional information can include happiness, anger, sadness, etc., and posture information can include normal sitting posture, resting posture, working posture, and entertainment posture, etc.; the preset deep learning model is a trained deep learning model, and the preset deep learning model can be a VGGFace model, a FaceNet model, etc.

[0130] In a specific implementation, facial videos or full-body videos of passengers can be collected through cameras installed on the vehicle.

[0131] In a feasible embodiment, the step of determining the emotional information of the occupant based on the facial video according to a preset deep learning model includes: extracting local features and global features from the facial image of the facial video, wherein the local features are used to represent changes in the facial areas of the occupant, and the global features are used to represent changes in the facial contours of the occupant; inputting the local features and the global features into the preset deep learning model to obtain the emotional information of the occupant.

[0132] It should be noted that changes in the occupant's facial features can include subtle changes in the eyes, mouth, eyebrows, and other areas; changes in the occupant's facial contours can include changes in the overall facial contour and shape. The occupant's emotional information can include happiness, anger, surprise, sadness, fear, or neutrality.

[0133] In a feasible embodiment, the step of determining the posture information of the occupant based on the full-body video according to a preset posture detection algorithm includes: identifying the body key points in the full-body image of the full-body video; analyzing the geometric relationship between the body key points to obtain key point features; determining dynamic features based on the body key points of the full-body image at different times; and inputting the key point features and the dynamic features into a preset time series model to obtain the posture information of the occupant.

[0134] It should be noted that geometric relationships include angle relationships, distance relationships and position relationships. Geometric relationships describe the geometric characteristics of human body posture; dynamic characteristics are the characteristics of key points changing over time. By comparing the key points of the body in full-body images at different time points and analyzing the changes of key points over time, the dynamic characteristics of the human body, such as movement speed, acceleration and movement trajectory, can be determined.

[0135] It should be noted that the time series model can be a recurrent neural network (RNN), a long short-term memory network (LSTM), or other types of sequence prediction models.

[0136] Step S40 : determining a second adjustment mode according to the emotion information and the posture information, and adjusting the vehicle cabin again based on the second adjustment mode.

[0137] In a specific implementation, the second adjustment mode can be set according to the emotion information:

[0138] (1) The emotional information is pleasant: maintain the current cabin settings, or increase the comfort level appropriately (such as playing lighter music).

[0139] (2) If the emotional information is anger or anxiety: reduce the air-conditioning speed, play soothing music, and adjust the lighting to soft tones.

[0140] (3) The emotional message is sadness: play comforting audio content, provide warm lighting and appropriate fragrance.

[0141] (4) The emotional message is surprise: provide prompts or ask if help is needed to ensure the safety of the occupants.

[0142] In a specific implementation, the second adjustment mode can be set according to the emotion information:

[0143] (1) Posture information is normal sitting posture: maintain basic comfort settings to ensure the occupant's posture is healthy.

[0144] (2) Posture information is work posture: adjust the seat to a more supportive work posture and provide a stable desktop and light.

[0145] (3) Posture information is rest: tilt the seat to a zero-gravity angle, add a seat massage function, and play soft music.

[0146] (4) Posture information for reading: adjust light brightness and direction, provide stable seat support, and reduce external interference.

[0147] (5) Posture information is sleeping: tilt the seat further, dim the lights, maintain a quiet environment, and reduce the air volume.

[0148] In a feasible embodiment, the step of determining the second adjustment mode based on the emotional information and the posture information includes: determining the current state of the occupant based on the emotional information and the posture information; searching for the cabin setting information of the current state, and determining the second adjustment mode based on the cabin setting information.

[0149] In a specific implementation, the cockpit setting information of the occupants in different states can be pre-set, and then the current state of the occupants can be predicted based on the occupants' emotional information and posture information. For example, the occupants' emotional information and posture information can be input into a state prediction model to predict the occupants' current state, and then the cockpit setting information corresponding to the current state can be searched, and finally the cockpit setting information can be determined as the second adjustment mode.

[0150] In this embodiment, the identity information of the occupant is obtained; a first adjustment mode is determined based on the identity information, and the vehicle cabin is preliminarily adjusted based on the first adjustment mode; in the first adjustment mode, the emotional information and posture information of the occupant are obtained; a second adjustment mode is determined based on the emotional information and the posture information, and the vehicle cabin is further adjusted based on the second adjustment mode. This solves the technical problem that the occupant's riding experience is easily affected by the need to manually adjust the cabin environment when riding. Compared with the existing technology, the present application can preliminarily determine and adjust the first adjustment mode based on the occupant's identity information, and then dynamically adjust the cabin environment settings by real-time monitoring of the occupant's emotional information and posture information. The occupant does not need to manually adjust the cabin environment. This not only enables rapid adjustment of the cabin environment, but also more accurately captures the changes in the occupant's status to provide real-time dynamic cabin adjustment, thereby improving the occupant's riding experience.

[0151] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 2 Before step S10, the vehicle cabin adjustment method further includes steps S101 to S104:

[0152] Step S101, obtaining the facial features, clothing features, body shape features and voice tone features of the passenger;

[0153] In a specific implementation, the facial image and overall image of the occupant can be collected by a camera installed on the vehicle, and then the facial features of the occupant are determined based on the facial image, and then the clothing features of the occupant are determined based on the overall image.

[0154] In a specific implementation, body shape features can be determined through point cloud data collected by the depth sensor on the vehicle, and tone features can be determined through conversation data between the driver and passengers.

[0155] In a feasible embodiment, the step of obtaining the facial features, clothing features, body shape features and voice tone features of the occupant includes: obtaining a facial image of the occupant, clarifying the facial image to obtain a first facial image, determining the facial position of the occupant in the first facial image, and obtaining a second facial image after correcting the angle and lighting of the first facial image based on the facial position of the occupant, and extracting facial features from the second facial image; obtaining an overall image of the occupant, standardizing the overall image to obtain a target overall image, and integrating color information, pattern information, clothing type information and brand logo information extracted from the target overall image into clothing features; obtaining three-dimensional point cloud data of the occupant, determining the contour information and key body shape information of the occupant based on the three-dimensional point cloud data, and determining body shape features based on the contour information and the key body shape information; obtaining conversation information between the occupant and the driver, denoising and separating the conversation information to obtain target conversation information, and integrating the tone information, pitch information and volume information determined based on the target conversation information into voice tone features.

[0156] It should be noted that the facial image of the occupant can be captured by a camera installed inside the vehicle, and then the facial image can be preprocessed, for example, the facial image can be denoised and enhanced to clarify the facial image. Specifically, the facial image (i.e., the first facial image) can be clarified using filters and enhancement algorithms, and then face detection is performed on the first facial image, such as using a face detection algorithm (such as OpenCV, MTCNN, etc.) to locate the facial area of ​​the occupant, and then image correction is performed on the facial area in the first facial image, such as correcting the angle and lighting influence of the facial area to obtain a second facial image, and finally a deep learning model (such as CNN, FaceNet, etc.) is used to extract facial features from the second facial image.

[0157] It should be noted that the overall image of the occupant captured by the camera can be preprocessed (i.e., standardized) to obtain the target overall image. For example, the overall image can be cropped and focused. Specifically, the image can be focused on the occupant's clothing. Then, the clothing part of the target overall image can be subjected to color analysis and pattern analysis, such as standardized color and pattern, so as to extract clothing features. Then, computer vision algorithms can be used to extract color information, pattern information, clothing type information (such as formal wear, casual wear, etc.) and brand logos from the clothing image.

[0158] It should be noted that a depth sensor can be used to obtain three-dimensional point cloud data of the occupants, and then point cloud processing can be performed on the three-dimensional point cloud data. For example, the three-dimensional point cloud data can be cleaned and simplified, and then the contour information and key body shape information of the occupants can be extracted from the three-dimensional point cloud data. Finally, the height, body shape, posture, etc. (i.e., body shape characteristics) of the occupants can be analyzed based on the contour information and key body shape information.

[0159] It should be noted that after the conversation information between the passengers and the driver is collected in real time through a microphone array, the conversation information can be denoised and separated to improve the voice quality. Then, the long voice signal can be divided into segments for easy processing. The conversation information can be converted into text using automatic speech recognition (ASR) technology. Then, the sentiment analysis model is used to identify the intonation information, pitch information and volume information in the voice. Finally, the intonation information, pitch information and volume information are integrated into tone features.

[0160] Step S102, performing multimodal feature fusion on the facial features, the clothing features, the body shape features, and the voice tone features to obtain fused features;

[0161] In a feasible implementation manner, the step of performing multimodal feature fusion on the facial features, the clothing features, the body features and the voice tone features to obtain fused features includes: performing time alignment and spatial alignment on the facial features, the clothing features, the body features and the voice tone features to obtain target facial features, target clothing features, target body features and target voice tone features, the time alignment is used to time synchronize audio data with video data, and the spatial alignment is used to align two-dimensional data with three-dimensional data; performing multimodal feature fusion on the target facial features, the target clothing features, the target body features and the target voice tone features to obtain fused features, wherein the multimodal feature fusion includes feature-level fusion and decision-level fusion, the feature-level fusion includes one of splicing, weighted averaging or deep neural network, and the decision-level fusion includes weighted voting.

[0162] It should be noted that feature alignment can align features of different modalities for unified processing. Feature alignment includes time alignment and spatial alignment. Time alignment is used to synchronize audio data and video data to ensure that all features are analyzed at the same time point; spatial alignment is used to align two-dimensional image features (i.e., two-dimensional data) with three-dimensional body shape data (i.e., three-dimensional data) to better understand the overall characteristics of the occupants.

[0163] It should be noted that multimodal feature fusion is the use of multimodal fusion technology to integrate features from different sensors; multimodal feature fusion includes feature-level fusion and decision-level fusion, among which feature-level fusion is to fuse feature vectors of different modalities into a comprehensive feature vector through splicing, weighted averaging or deep neural networks; decision-level fusion is to fuse data of different modalities at the decision layer through weighted voting.

[0164] Step S103: inputting the fusion features into a preset multimodal model to obtain a probability distribution;

[0165] In a feasible embodiment, the step of inputting the fused features into a preset multimodal model to obtain a probability distribution includes: collecting an occupant dataset, wherein the occupant dataset includes occupant data labeled with different identity information; training a multimodal model based on the occupant dataset to obtain a preset multimodal model; and inputting the fused features into the preset multimodal model to obtain a probability distribution.

[0166] In the specific implementation, the deep learning model (i.e., multimodal model) is trained using a multimodal dataset with real labels (i.e., passenger dataset):

[0167] (1) Labeled data: Collect and label passenger data sets of different identities.

[0168] (2) Model architecture: Use a model architecture that can handle multimodal data, such as a Transformer-based multimodal model.

[0169] (3) Training objective: The model learns how to map features of different modalities to the identity categories of the occupants (such as boss, parent, child, etc.).

[0170] (4) Real-time reasoning: In practical applications, a trained model (i.e., a preset multimodal model) is used to reason about multimodal data collected in real time.

[0171] 1) Comprehensive judgment: Identify the occupant's identity information based on a comprehensive analysis of facial features, clothing features, body shape features, and voice tone features.

[0172] 2) Identity classification: Determine the occupant's identity information based on the probability distribution output by the model (for example, the model will output the probability of each identity information, and the identity information with the highest probability is the occupant's identity information).

[0173] Step S104: determining the identity information of the occupant according to the probability distribution.

[0174] It should be noted that the probability distribution output by the model refers to the probability of each type of identity information, and the identity information with the highest probability is the identity information of the occupant.

[0175] In this embodiment, the occupant's facial features, clothing features, body shape features, and voice tone features are obtained; these features are subjected to multimodal feature fusion to generate fused features; these fused features are input into a preset multimodal model to generate a probability distribution; and the occupant's identity information is determined based on the probability distribution. This embodiment identifies the occupant's identity by combining multiple features. This avoids identification failures caused by using a single feature, thereby improving identification accuracy and enabling precise control of the cabin environment.

[0176] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the vehicle cabin adjustment method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0177] This application also provides a vehicle cabin adjustment device, please refer to Figure 3 , the vehicle cabin adjustment device includes:

[0178] An acquisition module 10 is used to obtain the identity information of the occupant;

[0179] a determination module 20, configured to determine a first adjustment mode according to the identity information, and perform preliminary adjustments to the vehicle cabin based on the first adjustment mode;

[0180] The acquisition module 10 is further configured to acquire the occupant's emotional information and posture information in the first adjustment mode;

[0181] The determining module 20 is further configured to determine a second adjustment mode according to the emotion information and the posture information, and to adjust the vehicle cabin again based on the second adjustment mode.

[0182] The vehicle cabin adjustment device provided in this application, utilizing the vehicle cabin adjustment method described in the aforementioned embodiment, can address the technical issue of passengers needing to manually adjust the cabin environment while riding, which can easily affect their riding experience. Compared to the prior art, the vehicle cabin adjustment device provided in this application achieves the same beneficial effects as the vehicle cabin adjustment method described in the aforementioned embodiment. Other technical features of the vehicle cabin adjustment device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0183] The present application provides a vehicle, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the vehicle cabin adjustment method of the above-mentioned embodiment one.

[0184] Reference below Figure 4 , which shows a schematic structural diagram of a vehicle suitable for implementing the embodiments of the present application. The vehicle in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The vehicle shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present application.

[0185] like Figure 4As shown, the vehicle may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for device operation. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speakers, and vibrator; storage device 1003 including, for example, a magnetic tape or hard disk; and communication device 1009. The communication device 1009 can allow the vehicle to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a vehicle with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0186] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0187] The vehicle provided in this application utilizes the vehicle cabin adjustment method of the aforementioned embodiment, resolving the technical issue of passengers having to manually adjust the cabin environment while riding, which can easily affect their riding experience. Compared to the prior art, the vehicle provided in this application achieves the same beneficial effects as the vehicle cabin adjustment method of the aforementioned embodiment, and the other technical features of this vehicle are the same as those disclosed in the aforementioned embodiment, and are not further detailed here.

[0188] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0189] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0190] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the vehicle cabin adjustment method in the above-mentioned embodiment.

[0191] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0192] The computer-readable storage medium may be included in the vehicle, or may exist independently without being installed in the vehicle.

[0193] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the vehicle, the vehicle: obtains the identity information of the occupant; determines a first adjustment mode based on the identity information, and performs preliminary adjustments to the vehicle cabin based on the first adjustment mode; in the first adjustment mode, obtains the emotional information and posture information of the occupant; determines a second adjustment mode based on the emotional information and the posture information, and adjusts the vehicle cabin again based on the second adjustment mode.

[0194] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0195] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0196] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0197] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the vehicle cabin adjustment method described above. This computer-readable storage medium can address the technical issue of passengers needing to manually adjust the cabin environment while riding, which can easily affect their riding experience. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the vehicle cabin adjustment method provided in the aforementioned embodiment and are not further elaborated here.

[0198] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A vehicle cabin adjustment method, characterized in that: The method comprises: Obtaining the occupant's identity information; determining a first adjustment mode according to the identity information, and performing preliminary adjustments to the vehicle cabin based on the first adjustment mode; In the first adjustment mode, acquiring the occupant's emotional information and posture information; determining a second adjustment mode according to the emotion information and the posture information, and further adjusting the vehicle cabin based on the second adjustment mode; The step of obtaining the identity information of the occupant includes: Obtaining facial features, clothing features, body shape features, and voice tone features of the occupant; Performing multimodal feature fusion on the facial features, the clothing features, the body shape features, and the voice tone features to obtain fused features; Inputting the fused features into a preset multimodal model to obtain a probability distribution; determining the identity information of the occupant according to the probability distribution; The step of performing multimodal feature fusion on the facial features, the clothing features, the body shape features, and the voice tone features to obtain fused features includes: Performing time alignment and spatial alignment on the facial features, clothing features, body shape features, and voice tone features to obtain target facial features, target clothing features, target body shape features, and target voice tone features, wherein the time alignment is used to synchronize audio data with video data, and the spatial alignment is used to align two-dimensional data with three-dimensional data; Multimodal feature fusion is performed on the target facial features, the target clothing features, the target body features, and the target voice tone features to obtain fused features, wherein the multimodal feature fusion includes feature-level fusion and decision-level fusion, the feature-level fusion includes one of splicing, weighted averaging, or deep neural network, and the decision-level fusion includes weighted voting.

2. The method according to claim 1, wherein The step of determining the first adjustment mode according to the identity information includes: Obtaining memory information corresponding to the identity information, wherein the memory information is a plurality of previous cabin environment information of the occupant at different riding time periods; determining target cabin environment information for the occupant at different riding time periods based on the memory information; determining a target riding section for the passenger's current riding time; The target cabin environment information corresponding to the target riding section is determined as a first adjustment mode.

3. The method according to claim 1, wherein The step of obtaining the facial features, clothing features, body shape features and voice tone features of the occupant includes: Acquiring a facial image of the occupant, performing a sharpening process on the facial image to obtain a first facial image, determining a facial position of the occupant in the first facial image, and obtaining a second facial image after correcting the angle and lighting of the first facial image based on the facial position of the occupant, and extracting facial features from the second facial image; acquiring an overall image of the occupant, performing standardization processing on the overall image to obtain a target overall image, and extracting color information, pattern information, clothing type information, and brand identification information from the target overall image and integrating them into clothing features; Acquiring three-dimensional point cloud data of the occupant, determining outline information and key body shape information of the occupant based on the three-dimensional point cloud data, and determining body shape features based on the outline information and the key body shape information; Conversation information between the passenger and the driver is obtained, denoised and separated to obtain target conversation information, and intonation information, pitch information, and volume information determined based on the target conversation information are integrated into a voice tone feature.

4. The method according to claim 1, wherein The step of inputting the fusion features into a preset multimodal model to obtain a probability distribution includes: collecting an occupant data set, wherein the occupant data set includes occupant data annotated with different identity information; Training a multimodal model based on the occupant dataset to obtain a preset multimodal model; The fused features are input into a preset multimodal model to obtain a probability distribution.

5. The method according to claim 1, wherein The step of obtaining the occupant's emotional information and posture information includes: Acquiring a facial video and a full-body video of the occupant; Based on the facial video, determining the occupant's emotional information according to a preset deep learning model; Based on the full-body video, the posture information of the occupant is determined according to a preset posture detection algorithm.

6. The method according to claim 5, wherein The step of determining the occupant's emotional information based on the facial video according to a preset deep learning model includes: Extracting local features and global features from the facial image of the facial video, wherein the local features are used to represent changes in facial features of the occupant, and the global features are used to represent changes in facial contours of the occupant; The local features and the global features are input into the preset deep learning model to obtain the emotional information of the occupant.

7. The method according to claim 5, wherein The step of determining the posture information of the occupant based on the full-body video according to a preset posture detection algorithm includes: identifying body key points in a full-body image of the full-body video; Analyze the geometric relationship between key points of the body to obtain key point features; Determine dynamic features based on body key points in full-body images at different moments; The key point features and the dynamic features are input into a preset time series model to obtain the posture information of the occupant.

8. The method according to claim 1, wherein The step of determining the second adjustment mode according to the emotion information and the posture information includes: determining a current state of the occupant based on the emotion information and the posture information; The cockpit setting information of the current state is searched, and the second adjustment mode is determined based on the cockpit setting information.

Images