Response mode determination method, storage medium, electronic equipment and vehicle

Abstract

The invention relates to a response mode determination method, a storage medium, electronic equipment and a vehicle, the method is applied to the vehicle, the vehicle comprises multi-modal data acquisition equipment, and the method comprises the step of determining a response mode of the vehicle in response to user interaction in an intelligent control process based on multi-modal data acquired by the multi-modal data acquisition equipment. The specific requirements of the user and the specific conditions of the vehicle can be recognized through the multi-modal data, so that the vehicle can intelligently select the user interaction response mode conforming to the current scene in the intelligent control process according to the multi-modal data, and efficient and accurate user instruction feedback is achieved; and the driving experience and the interaction experience of the user are improved.

Description

Technical Field

[0001] The present application relates to the field of vehicles, and in particular to a response mode determination method, a storage medium, an electronic device and a vehicle. Background Art

[0002] In the current vehicle usage process, after the user issues a control command through intelligent control methods, the vehicle will respond to the control command. Choosing the appropriate response method is crucial to ensuring driving safety and improving user experience.

[0003] Therefore, how to automatically select an appropriate response method to respond to user control instructions has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0004] An embodiment of the present application provides a method for determining a response mode, and a vehicle can automatically select an appropriate response mode to at least partially solve the above-mentioned technical problem.

[0005] In order to achieve the above-mentioned object, according to a first aspect of the present application, a response mode determination method is provided, which is applied to a vehicle, wherein the vehicle includes a multimodal data acquisition device, and the method includes:

[0006] A response mode of the vehicle in response to user interaction during intelligent control is determined based on the multimodal data collected by the multimodal data collection device.

[0007] According to a second aspect of the present application, a response mode determination device is provided, which is applied to a vehicle, wherein the vehicle includes a multimodal data acquisition device, and the device includes:

[0008] A response module is used to determine a response mode of the vehicle in response to user interaction during an intelligent control process based on the multimodal data collected by the multimodal data collection device.

[0009] According to a third aspect of the present application, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0010] According to a fourth aspect of the present application, there is also provided an electronic device having a computer program stored thereon, which implements the steps of the above method when the computer program is executed by a processor.

[0011] According to a fifth aspect of the present application, a vehicle is also provided, comprising the above-mentioned electronic device.

[0012] According to a sixth aspect of the present application, a computer program product is also provided, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0013] To summarize, in the embodiments of the present application, multimodal data is collected by the vehicle's multimodal data collection device, and the vehicle's response method in response to user interaction during the intelligent control process is determined through the multimodal data. Because the user's specific needs and the vehicle's specific conditions can be identified through multimodal data, the vehicle can intelligently select a user interaction response method that suits the current scenario during the intelligent control process based on the multimodal data, thereby achieving efficient and accurate user command feedback and improving the user's driving experience and interaction experience.

[0014] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0016] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same figure numbers represent the same parts in the following description.

[0017] Figure 1 It is one of the flowcharts of the response mode determination method provided in the exemplary embodiment of the present application;

[0018] Figure 2 This is the second flow chart of the response mode determination method provided in the exemplary embodiment of the present application;

[0019] Figure 3 is one of the flowcharts of the driving mode determination method provided in the exemplary embodiment of the present application;

[0020] Figure 4 This is a second flowchart of a driving mode determination method provided in an exemplary embodiment of the present application;

[0021] Figure 5 This is the third flow chart of the response mode determination method provided in the exemplary embodiment of the present application;

[0022] Figure 6 This is the fourth flow chart of the response mode determination method provided in the exemplary embodiment of the present application;

[0023] Figure 7 This is the fifth flowchart of the response mode determination method provided in the exemplary implementation of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0025] Based on the problems mentioned in the aforementioned background technology, in the related technology, after the user in the vehicle's smart cockpit uses intelligent control to issue a control command, the response method obtained is generally a voice broadcast, and the feedback method is single, and the response method cannot be adaptively adjusted according to the actual scenario (the response method may include the form and volume of the response, etc.).

[0026] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0027] Figure 1 is one of the flow charts of the response mode determination method provided in the exemplary embodiment of the present application, such as Figure 1 As shown, the response mode determination method provided in the embodiment of the present application includes step 110. The method flow steps are only used as a possible implementation of the present application.

[0028] Step 110 , determining a response mode of the vehicle in response to user interaction during an intelligent control process based on the multimodal data collected by the multimodal data collection device.

[0029] Specifically, the response mode determination method provided in the embodiment of the present application is applied to a vehicle, wherein the vehicle includes a multimodal data acquisition device.

[0030] The execution subject of the response mode determination method provided in the embodiment of the present application is a response mode determination device, which can be a hardware device independently set in the vehicle terminal or a software program running in the vehicle terminal.

[0031] The multimodal data acquisition device is a device for acquiring multimodal data, which is used to capture the vehicle's internal cabin data and user data, as well as the vehicle's own data (such as vehicle speed, etc.). The multimodal data acquisition device may include a sound acquisition module, an image acquisition module, a vehicle speed acquisition module, and a light acquisition module.

[0032] Multimodal data includes data of multiple dimensions related to the vehicle and the user, such as images inside the vehicle cabin and user sounds, which together describe the details of the vehicle cabin, the user, and the vehicle itself.

[0033] The intelligent control process refers to the process in which the response mode determination device identifies user needs by analyzing and processing multimodal data and makes corresponding decisions accordingly.

[0034] The response mode refers to the mode in which the response mode determination device selects an appropriate response form (such as voice prompt, graphical interface update, vibration feedback, etc.) according to the analysis results to provide information to the user or confirm the action. The response mode can be composed of one or more response forms. For example, the response mode can include one or more response forms such as voice broadcast, headrest broadcast, screen image animation, and steering wheel vibration. At the same time, if sound response is involved, the volume can also be adjusted adaptively.

[0035] Users can use the intelligent control system in the vehicle cockpit to control the vehicle. The control methods include but are not limited to intelligent voice, gesture control and eye control.

[0036] After receiving the control instruction sent by the user, the vehicle response mode determination device determines the response mode of the vehicle in response to the user interaction during the intelligent control process according to the multimodal data collected by the multimodal data collection device.

[0037] The multimodal data can be periodically collected and stored in the vehicle terminal, and the collection period can be set according to the actual situation. For example, after receiving the control instruction, the multimodal data collected in the previous collection period stored in the current vehicle terminal is obtained, and the response mode is determined according to the multimodal data.

[0038] The multimodal data may also be collected after receiving the control instruction. For example, after receiving the control instruction, the current multimodal data is collected based on the modal data collection device, and the response mode is determined according to the multimodal data.

[0039] It is also possible to analyze the driving scenarios corresponding to the multimodal data. After receiving the control instructions issued by the user, it automatically selects a response method that suits the current scenario based on the driving scenario to provide a comfortable user experience.

[0040] For example, when a user performs intelligent control (such as voice control and gesture control) of the vehicle in the vehicle cabin, the vehicle detection module and the user status recognition module obtain multimodal data such as the vehicle's internal and external environment, the vehicle's driving status, and the user's current status. Based on the multimodal data, the system automatically determines the specific response methods to be adopted when responding to control commands in the current driving scenario of the vehicle.

[0041] For example, multimodal data includes the sound, image and vehicle speed in the cockpit. If the sound in the current vehicle cockpit is very noisy, with music playing and passengers talking to each other, the vehicle is in a lively driving scene, and the response method may be a loud voice broadcast. If the sound in the current vehicle cockpit is only a steady background noise or tire noise, no one is talking, and the passengers are in a closed-eye state, it means that the vehicle is in a quiet driving scene, and the vehicle is driving at high speed, and the driver uses intelligent control to issue instructions, the response method may be steering wheel vibration or a low-volume voice broadcast.

[0042] The response mode determination method provided in the embodiment of the present application collects multimodal data through the vehicle's multimodal data acquisition device, and determines the response mode of the vehicle in response to user interaction during the intelligent control process through the multimodal data. Because the user's specific needs and the vehicle's specific conditions can be identified through the multimodal data, the vehicle can intelligently select the user interaction response mode that suits the current scenario during the intelligent control process based on the multimodal data, thereby achieving efficient and accurate user command feedback and improving the user's driving experience and interaction experience.

[0043] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0044] In some embodiments, step 110 includes:

[0045] Determining the driving scene of the vehicle based on the multimodal data collected by the multimodal data collection device;

[0046] A response mode of the vehicle in response to user interaction during intelligent control is determined according to the driving scenario.

[0047] Specifically, a driving scenario refers to a specific situation during the use of a vehicle, which is composed of multiple factors such as the vehicle's internal cabin, the user, and the vehicle itself.

[0048] Figure 2 This is a second flow chart of the response mode determination method provided in the exemplary embodiment of the present application; Figure 2 As shown, the embodiment of the present application determines the driving scene of the vehicle through multimodal data analysis of relevant information of the internal cockpit, the user and the vehicle itself; and determines the response method of the vehicle to the user interaction during the intelligent control process based on the driving scene.

[0049] Figure 3 is one of the flowcharts of the driving mode determination method provided in the exemplary embodiment of the present application; Figure 4 This is a second flow chart of the driving mode determination method provided in an exemplary embodiment of the present application; Figure 3 and Figure 4 As shown, the driving scenario can be determined by a neural network model, which can be determined based on a pre-stored scenario data set.

[0050] Driving scenarios can be determined by neural network models.

[0051] For example, the multimodal data collected by the multimodal data collection device includes the sound, light, current vehicle speed and user status in the cabin, and the driving scene of the current vehicle is comprehensively judged by combining the sound, light, current vehicle speed and user status in the cabin. The multimodal data is input into the scene prediction model; the scene prediction model outputs the driving scene of the current vehicle based on the multimodal data.

[0052] The driving scenario may also be determined based on a pre-stored scenario dataset.

[0053] For example, the scene data set includes multiple groups of multimodal data and driving scenes corresponding to each group of multimodal data. The current multimodal data is matched with each group of multimodal data in the scene data set, and the driving scene corresponding to the multimodal data matched with the current multimodal data is used as the driving scene corresponding to the current multimodal data. The driving scene corresponding to the current multimodal data is the driving scene in which the current vehicle is located.

[0054] The embodiment of the present application analyzes relevant information of the internal cockpit, the user, and the vehicle itself through multimodal data to obtain an accurate driving scenario of the vehicle, thereby determining an accurate response method for the specific driving scenario.

[0055] In some embodiments, determining the driving scene of the vehicle based on the multimodal data collected by the multimodal data collection device includes:

[0056] The multimodal data is input into a scene prediction model to obtain the driving scene of the vehicle output by the scene prediction model.

[0057] The multimodal data is input into a scene prediction model to obtain the driving scene of the vehicle output by the scene prediction model, including:

[0058] Inputting the multimodal data into a feature extraction network in the scene prediction model for processing to obtain multimodal features corresponding to the multimodal data;

[0059] The multimodal features are fused to obtain fused features;

[0060] Based on the fused features, the driving scene is obtained.

[0061] Specifically, the models involved in this application can be constructed using convolutional neural network models, fully connected neural network models, recurrent neural network models, long short-term memory neural network models, etc. The specific construction method can be determined based on the characteristics of the seller's input data.

[0062] When pre-training a model, it is usually trained with limited labeled data. Since the amount of data may be small, the following methods can be used to optimize model performance: 1) Data augmentation: increase the diversity of data, such as slight transformation of facial features and action features; 2) Transfer learning: use feature extraction networks pre-trained on large datasets (such as VGG and ResNet, etc.) to extract facial and action features; 3) Cross-validation: use k-fold cross-validation to better evaluate the performance of the model.

[0063] For example, the scene prediction model can be constructed using a convolutional neural network model, a fully connected neural network model, a recurrent neural network model, and a long short-term memory neural network model.

[0064] Before the model is formed, a training set can be obtained to train the model. The trained model is then input into the test set, and the results output by the model are labeled to form a training set again.

[0065] For example, the scene prediction model is obtained by training the initial scene prediction model based on the sample multimodal data and the sample driving scene corresponding to the sample multimodal data. A limited number of training sets can be obtained first to train the initial scene prediction model.

[0066] The collected multimodal data is input into the feature extraction network in the scene prediction model to extract the multimodal features of the multimodal data. After feature vectorization and fusion processing, the fused features are obtained. The scene prediction model analyzes and identifies the fused features.

[0067] The scenario prediction model predicts the driving scenario of the vehicle based on the analysis and recognition results, and the driving scenario will serve as the basis for the subsequent response process.

[0068] The models involved in this application can update the training set according to the output results, and optimize the model training according to the updated training set.

[0069] For example, the output driving scenes and multimodal data are labeled and the training set is updated, and the scene prediction model is optimized and trained based on the updated training set.

[0070] When the training set is updated, the scene prediction model needs to be pre-trained and fully tested before it can be released for online use.

[0071] The response mode determination method provided in the embodiment of the present application can accurately identify the driving scenario of the vehicle through multimodal data, and automatically and intelligently select the appropriate response mode according to the driving scenario, thereby improving the user experience.

[0072] In some embodiments, determining, according to the driving scenario, a response mode of the vehicle in response to user interaction during intelligent control includes:

[0073] The driving scenario is input into a first response prediction model to obtain a response mode corresponding to the driving scenario output by the first response prediction model.

[0074] The first response prediction model is trained by using the driving scenario and the response mode corresponding to the driving scenario as new training samples.

[0075] Specifically, the first response prediction model is obtained by training an initial first response prediction model based on the sample driving scenario and the response mode corresponding to the sample driving scenario.

[0076] The driving scenario is input into the first response prediction model, and the first response prediction model predicts a response method suitable for the driving scenario based on the input driving scenario.

[0077] The response mode and driving scenario output by the first response prediction model are labeled and the training set is updated, and the first response prediction model is optimized and trained based on the updated training set.

[0078] When the training set is updated, the first response prediction model needs to be pre-trained and fully tested before being released for online use.

[0079] The response mode determination method provided in the embodiment of the present application determines the response mode of the vehicle in response to user interaction during the intelligent control process through a first response prediction model and a driving scenario, thereby improving the efficiency of determining the response mode.

[0080] In some embodiments, determining, according to the driving scenario, a response mode of the vehicle in response to user interaction during intelligent control includes:

[0081] The driving scenario and historical user preference data are input into a second response prediction model to obtain the response mode output by the second response prediction model.

[0082] Before inputting the driving scenario and the historical user preference data into the second response prediction model, the method further includes:

[0083] Acquiring user characteristics of the user based on the multimodal data acquisition device;

[0084] The user characteristics are matched with the user characteristics in the user preference data set to obtain the historical user preference data of the user.

[0085] After obtaining the response mode output by the second response prediction model, the method further includes:

[0086] receiving new user preference data fed back by the user based on the response mode;

[0087] The historical user preference data in the user preference data set is updated based on the new user preference data.

[0088] Specifically, user preference data refers to data that reflects the user's personalized preferences for different response methods during driving.

[0089] User features refer to data indicators captured and extracted by multimodal data collection equipment and used to identify user identities, such as voiceprints and facial images.

[0090] The user preference data set includes multiple user features and historical user preference data corresponding to each set of user features.

[0091] The second response prediction model is obtained by training the initial second response prediction model based on the sample driving scenario, the sample historical user preference data, and the response methods corresponding to the sample driving scenario and the sample historical user preference data.

[0092] Figure 5 This is the third flow chart of the response mode determination method provided in the exemplary embodiment of the present application. Figure 6 This is a fourth flow chart of the response mode determination method provided in the exemplary embodiment of the present application, such as Figure 5 and Figure 6 As shown, after the vehicle receives the control instruction issued by the user, it responds to the control instruction issued by the user in the intelligent control process by responding. After receiving the response, the user can express his or her own preferences, such as "don't make the volume too loud" and "I like the headrest broadcast", etc. The system binds and stores the user's user preference data with the user's user characteristics, and automatically obtains the user's historical user preference data the next time the user issues a control instruction.

[0093] The driving scene and historical user preference data of users in the current vehicle cabin are input into a second response prediction model; the second response prediction model identifies feature information of the input driving scene and historical user preference data and predicts a response method suitable for the driving scene and the historical user preference data, thereby outputting a response method suitable for the current driving scene and the current user.

[0094] If the user who issues the control instruction determines that the user preference data set includes the user feature after user feature identification, the historical user preference data corresponding to the user feature is obtained and the historical user preference data is input as a parameter into the second response prediction model.

[0095] For example, the driving scene and historical user preference data are input into the feature extraction network in the second response prediction model to extract the driving scene features and historical user preference data features, and then undergo feature vectorization and fusion processing to obtain fused features. The second response prediction model analyzes the fused features to identify the predicted response method.

[0096] The response mode determination method provided in the embodiment of the present application determines the response mode of the vehicle in response to user interaction during the intelligent control process through a second response prediction model, driving scenarios and user preference data, thereby improving the efficiency of determining the response mode.

[0097] In some embodiments, step 110 includes:

[0098] The multimodal data is input into a third response prediction model to obtain the response mode output by the third response prediction model.

[0099] Wherein, inputting the multimodal data into a third response prediction model to obtain the response mode output by the third response prediction model includes:

[0100] Inputting the multimodal data into a feature extraction network in the third response prediction model for processing to obtain multimodal features corresponding to the multimodal data;

[0101] The multimodal features are fused to obtain fused features;

[0102] Based on the fusion features, the response mode is obtained.

[0103] Specifically, the third response prediction model is obtained by training the initial third response prediction model based on the sample multimodal data and the response mode corresponding to the sample multimodal data.

[0104] The collected multimodal data is input into a third response prediction model; the third response prediction model identifies feature information of the input multimodal data and predicts a response method suitable for the multimodal data, thereby outputting a response method suitable for the current multimodal data.

[0105] For example, the collected multimodal data is input into the feature extraction network in the third response prediction model to extract the multimodal features of the multimodal data, and then the fused features are obtained through feature vectorization and fusion processing. The third response prediction model analyzes the fused features to identify the predicted response mode.

[0106] The response mode determination method provided in the embodiment of the present application determines the response mode of the vehicle in response to user interaction during the intelligent control process through a third response prediction model and multimodal data, thereby improving the efficiency of determining the response mode.

[0107] In some embodiments, step 110 includes:

[0108] The response mode of the vehicle in response to user interaction during the intelligent control process is determined based on the driving scene corresponding to the multimodal data collected by the multimodal data collection device in the previous collection cycle.

[0109] Specifically, a storage module may be provided in the vehicle terminal, and the multimodal data in each acquisition cycle and the driving scene corresponding to the multimodal data may be stored through the storage device. The storage device may also only store the multimodal data in the latest acquisition cycle and the driving scene corresponding to the multimodal data, and the latest acquisition cycle is the previous acquisition cycle.

[0110] When the user issues a control instruction, the driving scene corresponding to the multimodal data collected in the previous collection cycle can be quickly obtained from the storage module, thereby shortening the judgment time of the driving scene and the response time to the user control instruction.

[0111] A storage module can be set to store the collected multimodal data and set the corresponding timestamp. The multimodal data acquisition device continuously collects and stores multimodal data at short intervals, and identifies and judges the driving scene by analyzing the multimodal data collected in the last acquisition cycle (for example, the last acquisition cycle is within the last 1 minute). Because the time when the user issues a control instruction is uncertain, multimodal data is continuously collected and the judgment of the driving scene is continuously updated, so that when the user issues a control instruction, the accurate driving scene can be quickly obtained and the response method can be selected, which shortens the response time to the user's intelligent control.

[0112] The response mode determination method provided in the embodiment of the present application improves the response efficiency by determining the response mode of the vehicle in response to user interaction during the intelligent control process based on the driving scenario corresponding to the multimodal data collected by the multimodal data acquisition device in the previous acquisition cycle.

[0113] In some embodiments, the multimodal data acquisition device includes at least two of a sound acquisition module, an image acquisition module, a vehicle speed acquisition module, and a light acquisition module.

[0114] Wherein, determining the driving scene of the vehicle based on the multimodal data collected by the multimodal data collection device includes:

[0115] Determining the data level corresponding to each modality data based on the multimodal data;

[0116] A driving scenario of the vehicle is determined according to the data level.

[0117] Determining the data level corresponding to each modality data based on the multimodal data respectively includes:

[0118] If the multimodal data acquisition device includes the sound acquisition module, determining the volume level to which the sound signal belongs according to the volume of the sound signal in the vehicle acquired by the sound acquisition module;

[0119] If the multimodal data acquisition device includes the image acquisition module, the user information level of the user information in the vehicle is determined according to the image in the vehicle acquired by the image acquisition module; the user information includes the number of users, user status and / or user type. The user information may also include whether there are infants among the passengers.

[0120] If the multimodal data acquisition device includes the vehicle speed acquisition module, determining the speed grade to which the vehicle speed belongs according to the vehicle speed acquired by the vehicle speed acquisition module;

[0121] If the multimodal data acquisition device includes the illumination acquisition module, the illumination level to which the illumination data in the vehicle belongs is determined according to the illumination data in the vehicle acquired by the illumination acquisition module.

[0122] Determining the driving scene of the vehicle according to the data level includes: determining the driving scene of the vehicle according to at least two of the volume level, the user information level, the speed level, and the light level.

[0123] Specifically, the multimodal data acquisition device includes at least two of a sound acquisition module, an image acquisition module, a vehicle speed acquisition module, and a light acquisition module.

[0124] The sound collection module may include a microphone in the vehicle cabin. For example, a microphone used for intelligent voice pickup is used to construct a sound collection module to collect sound signals.

[0125] The image acquisition module is used to collect images in the vehicle, such as images and / or videos, etc. The user's face, body movements, eye opening and closing status, expression, approximate age group, user location and number of people, etc. can be analyzed through images and / or videos to obtain user information levels such as the number of users, user status and / or user type. The image acquisition module may include a driver monitoring system (DMS) camera and a screen-mounted camera. The user types include drivers and passengers; the user type of each user is determined based on the location information of each user in the vehicle. For example, when a user uses intelligent control, the user's orientation can be identified to determine whether the user is a driver or a passenger.

[0126] The light collection module may include a light sensor for collecting light data in the vehicle cabin.

[0127] The vehicle speed collection module may include a vehicle speed sensor for collecting vehicle speed.

[0128] The sound acquisition module, image acquisition module, vehicle speed acquisition module and illumination acquisition module of the multimodal data acquisition device collect multimodal data, and determine the data level corresponding to each modal data.

[0129] For example, a first volume threshold, a second volume threshold, and a third volume threshold may be set, and the volume level of the sound signal may be determined based on the relationship between the volume of the sound signal and the first volume threshold, the second volume threshold, and the third volume threshold.

[0130] An information level table is set, which includes multiple groups of user information and user information levels corresponding to each group of information levels. User information in the vehicle is determined based on the image, and the user information is matched with the user information in the information level table to determine the user information level of the user information.

[0131] A first illumination threshold and a second illumination threshold are set, and the illumination level to which the illumination data belongs is determined based on the relationship between the illumination data and the first illumination threshold and the second illumination threshold. A vehicle speed threshold is set, and the speed level to which the vehicle speed belongs is determined based on the relationship between the vehicle speed and the vehicle speed threshold.

[0132] Taking the data in Table 1 as an example, Table 1 records multimodal data, the definition method of data levels, and the data levels corresponding to each modal data.

[0133] Table 1 Data level corresponding to each modal data

[0134]

[0135]

[0136] The sound acquisition module, image acquisition module, vehicle speed acquisition module and illumination acquisition module of the multimodal data acquisition device are used to combine and judge the driving scene or comprehensively judge the driving scene.

[0137] The volume level, user information level, speed level and light level are combined to judge the driving scene or comprehensively judge the driving scene. The volume level, user information level, speed level and light level can be combined in pairs or in groups of three.

[0138] For example, the volume level and user information level are combined to judge the driving scene; the volume level, user information level, speed level and light level are comprehensively used to judge the driving scene, etc.

[0139] The user information level can also be divided into a first user information level and a second user information level. For example, user information levels 1 to 3 belong to the first user information levels 1 to 3 respectively; user information levels 4 to 7 belong to the second user information levels 1 to 4 respectively.

[0140] The first user information level, the second user information level, the volume level, the speed level, and the light level may be arbitrarily combined to determine the driving scene, such as a combination of two, three, or four; the driving scene may also be determined in combination.

[0141] For example, the volume level and the first user information level may be combined to determine the driving scene; or the volume level, the first user information level, and the second user information level may be combined to determine the driving scene.

[0142] The main driver wakes up the vehicle, that is, the driver issues control commands; the non-main driver wakes up the vehicle, that is, the passenger issues control commands. The driving scenario can be judged by combining the data level and the type of user issuing the control commands.

[0143] Figure 7 FIG. 5 is a flowchart of a method for determining a response mode provided in an exemplary embodiment of the present application. Figure 7 As shown, the appropriate response method can be selected according to the driving scenario. The response methods may include voice broadcast, sound prompt, voice image animation, steering wheel vibration, headrest broadcast and other response forms, and the volume can be automatically adjusted.

[0144] "Voice broadcast" means informing the user that the command has been executed through a response in the form of voice broadcast. "Sound effect" refers to a response in the form of a short prompt sound similar to a short message reminder sound. "Screen image animation" refers to the screen in the cockpit, which can prompt the user through some visual animation responses such as voice image animation and screen special effects animation. "Steering wheel vibration" refers to a response in which the steering wheel vibrates at a certain frequency so that the driver can feel it. "Headrest broadcast" refers to the broadcast of the built-in speaker installed in the headrest of the user's seat, and the broadcast sound can only be heard by the user in that position.

[0145] The response method can be any combination or combination of response forms such as voice broadcast, sound prompt, voice image animation, steering wheel vibration and headrest broadcast.

[0146] According to the driving scenario, which involves a sound response form, the appropriate reminder volume can be adaptively adjusted.

[0147] Taking the data in Table 2 as an example, Table 2 records driving scenarios determined by multiple data levels and the corresponding response method for each driving scenario.

[0148] Table 2 Response methods corresponding to driving scenarios

[0149]

[0150]

[0151] The obtained multimodal data can be stored uniformly, classified according to Table 1, and then a comprehensive analysis of the multimodal data can be performed according to Table 2 to obtain the driving scene and the response method corresponding to the driving scene.

[0152] A set of signal analysis algorithms can be built into the vehicle terminal to analyze and identify the collected multimodal data. Table 1 is a classification of the collected multimodal data in the specific implementation. The collected multimodal data includes sound signals, number recognition, image recognition (user's closed eyes, facial expressions and body movements, etc.), average age group recognition, lighting data and vehicle speed, etc. Each modal data can be classified according to the actual value and actual situation, and a code is given to each classification. In the specific implementation of Table 2, each classification code is combined with each other, combined with the user type judgment (driver and passenger) who issued the user command, and the current driving scene is obtained through comprehensive analysis. The corresponding response form and volume are selected according to the driving scene to obtain the response method, and finally presented to the user side.

[0153] The embodiment of the present application mainly describes a method for driving scenario analysis and adaptive response mode selection. The multimodal data classification in Table 1 and the formulation of the driving scenario and response mode in Table 2 are not fixed. The multimodal data can be thresholded according to the actual scenario and needs, or more required multimodal data can be collected. The driving scenario and response mode can also be adjusted according to the actual demand definition.

[0154] The response mode determination method provided in the embodiment of the present application comprehensively judges the current driving scene through the sound, light, vehicle speed and user information in the vehicle cabin. This information can improve the accuracy of the judgment of the vehicle's driving scene and intelligently select a suitable response mode.

[0155] The following describes a response mode determination device provided in an embodiment of the present application. The response mode determination device described below and the response mode determination method described above can be referenced to each other.

[0156] The embodiment of the present application further provides a response mode determination device, which is applied to a vehicle, wherein the vehicle includes a multimodal data acquisition device, and the device includes:

[0157] A response module is used to determine a response mode of the vehicle in response to user interaction during an intelligent control process based on the multimodal data collected by the multimodal data collection device.

[0158] Among them, the response module can be used to execute step 110 in the embodiment corresponding to the above-mentioned response mode determination method. For the specific implementation methods of these modules and more details, please refer to the corresponding method part, which will not be repeated here.

[0159] It should be noted here that the response mode determination device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned response mode determination method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0160] An embodiment of the present application also provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the above-mentioned mobile charging scheduling method.

[0161] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0162] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0166] The memory may include non-permanent memory in a computer-readable storage medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable storage medium.

[0167] Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated communication signals and carrier waves.

[0168] The embodiment of the present application also provides a vehicle, which includes the above-mentioned electronic device. In this embodiment, the vehicle can be a fuel vehicle, a plug-in hybrid vehicle or a new energy vehicle, etc., and this application does not make specific limitations on this.

[0169] An embodiment of the present application also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0170] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0171] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0172] The embodiments, implementation methods and related technical features of the present application can be combined and replaced with each other without conflict.

[0173] The above are only preferred embodiments of the present application and do not constitute any form of limitation to the present application. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A method for determining a response mode, characterized in that: Applied to a vehicle, the vehicle includes a multimodal data acquisition device, and the method includes: A response mode of the vehicle in response to user interaction during intelligent control is determined based on the multimodal data collected by the multimodal data collection device.

2. The method according to claim 1, characterized in that: Determining a response mode of the vehicle in response to user interaction during intelligent control based on multimodal data collected by a multimodal data collection device includes: Determining the driving scene of the vehicle based on the multimodal data collected by the multimodal data collection device; A response mode of the vehicle in response to user interaction during intelligent control is determined according to the driving scenario.

3. The method according to claim 2, characterized in that Determining the driving scene of the vehicle based on the multimodal data collected by the multimodal data collection device includes: The multimodal data is input into a scene prediction model to obtain the driving scene of the vehicle output by the scene prediction model.

4. The method according to claim 3, characterized in that: Inputting the multimodal data into a scene prediction model to obtain a driving scene of the vehicle output by the scene prediction model includes: Inputting the multimodal data into a feature extraction network in the scene prediction model for processing to obtain multimodal features corresponding to the multimodal data; The multimodal features are fused to obtain fused features; Based on the fused features, the driving scene is obtained.

5. The method according to claim 2, characterized in that: Determining, according to the driving scenario, a response mode of the vehicle in response to user interaction during intelligent control, including: The driving scenario is input into a first response prediction model to obtain a response mode corresponding to the driving scenario output by the first response prediction model.

6. The method according to claim 5, characterized in that After obtaining the response mode corresponding to the driving scenario output by the first response prediction model, the method further includes: The first response prediction model is trained by using the driving scenario and the response mode corresponding to the driving scenario as new training samples.

7. The method according to claim 2, characterized in that Determining, according to the driving scenario, a response mode of the vehicle in response to user interaction during intelligent control, including: The driving scenario and historical user preference data are input into a second response prediction model to obtain the response mode output by the second response prediction model.

8. The method according to claim 7, characterized in that Before inputting the driving scenario and historical user preference data into the second response prediction model, the method further includes: Acquiring user characteristics of the user based on the multimodal data acquisition device; The user characteristics are matched with the user characteristics in the user preference data set to obtain the historical user preference data of the user.

9. The method according to claim 8, characterized in that After obtaining the response mode output by the second response prediction model, the method further includes: receiving new user preference data fed back by the user based on the response mode; The historical user preference data in the user preference data set is updated based on the new user preference data.

10. The method according to claim 1, characterized in that Determining a response mode of the vehicle in response to user interaction during intelligent control based on multimodal data collected by a multimodal data collection device includes: The multimodal data is input into a third response prediction model to obtain the response mode output by the third response prediction model.

11. The method according to claim 10, characterized in that Inputting the multimodal data into a third response prediction model to obtain the response mode output by the third response prediction model includes: Inputting the multimodal data into a feature extraction network in the third response prediction model for processing to obtain multimodal features corresponding to the multimodal data; The multimodal features are fused to obtain fused features; Based on the fusion features, the response mode is obtained.

12. The method according to claim 1, characterized in that Determining a response mode of the vehicle in response to user interaction during intelligent control based on multimodal data collected by a multimodal data collection device includes: The response mode of the vehicle in response to user interaction during the intelligent control process is determined based on the driving scene corresponding to the multimodal data collected by the multimodal data collection device in the previous collection cycle.

13. The method according to any one of claims 2 to 12, characterized in that: The multimodal data acquisition device includes at least two of a sound acquisition module, an image acquisition module, a vehicle speed acquisition module and a light acquisition module.

14. The method according to claim 13, characterized in that Determining the driving scene of the vehicle based on the multimodal data collected by the multimodal data collection device includes: Determining the data level corresponding to each modality data based on the multimodal data; A driving scenario of the vehicle is determined according to the data level.

15. The method according to claim 14, characterized in that Determining the data level corresponding to each modality data based on the multimodal data respectively includes: If the multimodal data acquisition device includes the sound acquisition module, determining the volume level to which the sound signal belongs according to the volume of the sound signal in the vehicle acquired by the sound acquisition module; If the multimodal data acquisition device includes the image acquisition module, determining the user information level of the user information in the vehicle according to the image in the vehicle acquired by the image acquisition module; If the multimodal data acquisition device includes the vehicle speed acquisition module, determining the speed grade to which the vehicle speed belongs according to the vehicle speed acquired by the vehicle speed acquisition module; If the multimodal data acquisition device includes the illumination acquisition module, the illumination level to which the illumination data in the vehicle belongs is determined according to the illumination data in the vehicle acquired by the illumination acquisition module.

16. The method according to claim 15, characterized in that Determining the driving scene of the vehicle according to the data level includes: determining the driving scene of the vehicle according to at least two of the volume level, the user information level, the speed level, and the light level.

17. The method according to claim 15, characterized in that The user information includes the number of users, user status and / or user type.

18. The method according to claim 17, characterized in that The user types include drivers and passengers; the user type of each user is determined based on position information of each user in the vehicle.

19. The method according to claim 1, characterized in that The response mode corresponds to at least one of voice broadcast, headrest broadcast, screen image animation and steering wheel vibration.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 19 are implemented.

21. An electronic device having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 19 are implemented.

22. A vehicle, characterized in that: An electronic device comprising the electronic device described in claim 21.

23. A computer program product, characterized in that The method comprises a computer program or instructions, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 19.

Images