Multi-mode vehicle machine interaction system and method based on artificial intelligence

Abstract

The invention discloses a multi-mode vehicle machine interaction system and method based on artificial intelligence, and relates to the technical field of vehicle machine interaction, and the interaction method comprises the following steps: collecting the driving data of a user in each driving process, and generating an interaction behavior set of a driving record; identifying a user state in any driving record; analyzing an interaction behavior set in each driving record, and identifying and dividing an interaction mode of the user; abnormal conditions of any driving record are identified, and association analysis is carried out on adaptation conditions between different user states and interaction modes; for any user state, establishing an interaction mode matching condition to confirm matching priorities of various interaction modes; and acquiring the driving data of the user at the current moment in real time, generating a real-time driving record, performing quality evaluation, and performing abnormal adjustment on the interaction mode of the user based on the recognition condition of the real-time interaction mode.

Description

Technical Field

[0001] The present invention relates to the field of vehicle-computer interaction technology, and in particular to a multimodal vehicle-computer interaction system and method based on artificial intelligence. Background Art

[0002] Multimodal vehicle-to-machine interaction aims to improve the usability, safety, and user experience of vehicle-to-machine systems, enabling users to interact more naturally and efficiently. Vehicle-to-machine interaction systems are gradually evolving from traditional physical buttons to digital interfaces. With continued development, the vehicle-to-machine interaction systems in more and more vehicles are becoming increasingly intelligent, such as various types of mobile vehicles, yachts and ships, outdoor off-road vehicles, low-altitude aircraft, and so on. Consequently, various intelligent vehicles are gradually encompassing multiple interaction modes, including conferencing, entertainment, leisure, personal health status, health management, and medical notifications.

[0003] Existing systems usually adopt fixed interaction modes and cannot flexibly switch according to factors such as user status and environmental noise, resulting in a poor interactive experience and even safety hazards. For example, during driving, voice recognition has a low recognition rate in noisy environments, and touch interaction may distract users. It is difficult to establish an association model between user status and interaction mode, which makes it difficult for the system to select the appropriate interaction method when facing complex scenarios, thus affecting the user experience. Summary of the Invention

[0004] The purpose of the present invention is to provide a multimodal vehicle-machine interaction system and method based on artificial intelligence to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal vehicle-machine interaction method based on artificial intelligence, the interaction method comprising the following steps:

[0006] Step S100: collecting driving data of the user during each driving process and extracting interactive behaviors to generate an interactive behavior set for a driving record; conducting multi-dimensional analysis on the driving data in any driving record and identifying the user status in any driving record;

[0007] Step S200: Analyze the interactive behavior set in each driving record and identify and classify the user's interaction mode; conduct a multi-dimensional evaluation of the driving data in any driving record and identify abnormal situations in any driving record;

[0008] Step S300: Based on the abnormality recognition results of any driving record, correlation analysis is performed on the adaptability between different user states and interaction modes; for any user state, interaction mode matching conditions are established to confirm the matching priority of various interaction modes;

[0009] Step S400: Real-time collection of the user's current driving data, generation of a real-time driving record and identification of the real-time user status; quality assessment of the real-time driving record, and abnormal adjustment of the user's interaction mode based on the identification of the real-time interaction mode.

[0010] Furthermore, step S100 includes the following steps:

[0011] Step S101: Several monitoring devices are installed inside. Data is collected at unit time points by various monitoring devices to obtain a set of driving data of the driving process and generate a driving record. The data collected by any monitoring device is set as monitoring data of one dimension. The monitoring data of each dimension are aligned according to the monitoring time point to obtain the monitoring data of each dimension at any time point and generate a set of monitoring data of each time point. The detection equipment includes three types of equipment: one is a monitoring device for collecting user interaction behavior, such as a camera, a voice recognition device, and a tactile sensing device; one is a monitoring device for monitoring user status, such as a body posture collection device; and the other is a monitoring device for monitoring driving status, such as driving speed, driving route, etc.

[0012] Step S102: Pre-construct an interactive behavior database, wherein the interactive behavior database includes several interactive behaviors, and each interactive behavior is matched with several feature dimensions and a feature value range corresponding to each feature dimension; arbitrarily select a monitoring data set and an interactive behavior at a time point; if the monitoring data of each dimension in the selected monitoring data set are the same as the feature dimensions of the selected interactive behavior and are within the corresponding feature data range, then the selected interactive behavior is set as the target interactive behavior at the selected time point; if the selected time point corresponds to several interactive behaviors, then the interactive behavior with the largest number of feature dimensions is selected and set as the target interactive behavior; the target interactive behaviors at each time point are summarized to obtain the interactive behavior set of a driving record;

[0013] Step S103: Construct a two-dimensional rectangular coordinate system with the time point as the horizontal coordinate and the monitoring data as the vertical coordinate, present the changes in the monitoring data of each dimension, and obtain the data difference between any two adjacent time points in any dimension. Preset a time window containing several time points, divide the monitoring data of each dimension in the two-dimensional rectangular coordinate system into several time intervals, and obtain the average difference of each dimension in different time intervals;

[0014] Step S104: arbitrarily select a time interval, aggregate the average differences of each dimension in the selected time interval, and obtain a difference set for the selected time interval; pre-build a state database that stores several types of user states, where any user state corresponds to an expected difference range of several feature dimensions; arbitrarily select a user state, and if the average differences of each dimension are within the expected difference range of the feature dimension corresponding to the selected user state, then set the selected user state as the user state in the selected time interval; user interaction behavior is basically instantaneous, so it is obtained from the detection data at each time point, and the user state is a continuous process, so state identification needs to be performed in continuous time intervals;

[0015] Step S105: If there is a same time interval between two time intervals and the user status of the two time intervals is the same, then the two time intervals are combined to obtain various user statuses of the user in the driving record and the time interval corresponding to each user status.

[0016] Furthermore, step S200 includes the following steps:

[0017] Step S201: Randomly select an interaction behavior set from a driving record, sort the interaction behaviors in the interaction behavior set according to the chronological order of the time intervals, and obtain the behavior occurrence sequence of the selected driving record; randomly select an interaction behavior, set the position of the selected interaction behavior in the behavior occurrence sequence to a, and simultaneously obtain the interaction behavior with the position a+1 in the behavior occurrence sequence, and set them as the two target interaction behaviors;

[0018] Step S202: Obtain the order of occurrence of each interaction behavior in each of the remaining driving records, and obtain the position order of the two target interaction behaviors in each driving record. If the positions of the two target interaction behaviors are adjacent to each other and the order is the same as that of the selected driving record, then set the driving record containing the two target interaction behaviors as the target driving record;

[0019] Step S203: Count the number of target driving records as m, and set the total number of all driving records as M total , then the proportion of the target driving record is η=m / M total ; Preset a quantity ratio threshold η th , if η<η th , then reselect the target interaction behavior; if η≥η th , then the two target interaction behaviors are set as the expected behavior combination, and the target interaction behavior with the order a+2 in the behavior occurrence sequence is continued to recalculate the target driving record proportion threshold corresponding to the three target interaction behaviors until η<η is satisfied. thThe generated expected behavior combination is set as the user's expected interaction pattern. A complete user behavior pattern may include one interaction behavior or several interaction behaviors. For example, navigation requires multiple interaction behaviors such as entering the destination and selecting a route, while querying a location only requires entering the destination. Therefore, it is necessary to carefully screen the interaction patterns to better analyze the user's behavior habits.

[0020] Step S204: Pre-construct an interaction pattern set containing several interaction patterns. Any interaction pattern corresponds to an interaction behavior set. If the order of several interaction behaviors in the expected interaction pattern is the same as that of the interaction pattern, an interaction pattern of the user is obtained. The remaining interaction behaviors in the expected interaction pattern are re-pattern-matched to obtain several interaction patterns of the user.

[0021] Step S205: arbitrarily select a dimension. If the selected dimension is not a characteristic dimension of any interactive behavior, set the selected dimension as the target dimension to obtain several target dimensions of any driving record; preset the corresponding expected value range (x1, x2) for any target dimension, and arbitrarily select the monitoring data of the i-th target dimension at the t-th time point from the selected driving record as x1, x2. (i,t) , if x (i,t) ∈(x1,x2), then the deviation amplitude β of the i-th target dimension at the t-th time point is obtained (i,t) =0, otherwise according to the formula:

[0022]

[0023] Among them, Max() is the maximum value function; the deviation amplitude β of the i-th target dimension at the t-th time point is calculated (i,t) ;

[0024] Step S206: Preset an initial evaluation value P0 according to the formula:

[0025]

[0026] Where i is a positive integer and i∈[1,c], c is the number of target dimensions in the selected driving record; the evaluation value P of the selected driving record at time point t is calculated t ; Preset an abnormal assessment threshold P th , if P t <P th , then the tth time point of the selected driving record is set as the abnormal time point.

[0027] Furthermore, step S300 includes the following steps:

[0028] Step S301: randomly selecting a driving record, extracting abnormal time points in the selected driving record, and merging adjacent and continuous abnormal time points to obtain an abnormal time interval;

[0029] Step S302: Obtain several interaction modes of the user in the selected driving record, and extract the time interval corresponding to each interaction mode, compare the time interval of each interaction mode with each abnormal time interval, and arbitrarily select the time interval of an interaction mode. If there is an abnormal time interval that has a common time interval with the time interval of the selected interaction mode, the time length of the common time interval is obtained as ΔL, and the time length of the abnormal time interval is set as ΔT. The abnormal time ratio γ of the selected interaction mode is calculated as ΔL / ΔT; a threshold value of abnormal time ratio γ is preset. th , if γ≥γ th , then extract the user status in the common time interval, and set the extracted user status as an abnormal state of the selected interaction mode;

[0030] Step S303: arbitrarily select the jth user state and the kth interaction mode, and count the number of the jth user state as the abnormal state of the kth interaction mode in all driving records as n (j,k) , set the number of k-th interaction modes to N k , the abnormal occurrence frequency of the kth interaction mode in the jth user state is calculated to be f (j,k) =n (j,k) / N k ; Get the total number of interaction modes contained in all driving records as N ’ , the frequency of the kth interaction mode is calculated to be f k =N k / N ’ ; According to the formula:

[0031]

[0032] Calculate the correlation value G(j,k) between the jth user state and the kth interaction mode. The correlation between user states and interaction modes mainly considers the abnormal frequencies of various interaction modes under a specific user state, as well as the occurrence frequencies of various interaction modes. The abnormal frequencies reflect the repulsion between the two, while the occurrence frequencies reflect the user's interaction preferences. Combining the two can yield a more accurate correlation.

[0033] Step S304: arbitrarily select the j-th user state, extract the correlation values ​​between the j-th user state and various interaction modes, and sort the correlation values ​​from large to small to obtain the matching order of the j-th user state to various interaction modes.

[0034] Furthermore, step S400 includes the following steps:

[0035] Step S401: When the user is in a driving state at the current moment, monitoring data of various dimensions at the current moment is acquired and a real-time driving record is generated; the monitoring data of various dimensions in the real-time driving record is compared with the user status in the user status database to obtain the real-time user status of the user at the current moment;

[0036] Step S402: Obtain the expected value range of each dimension, calculate the real-time deviation amplitude of each dimension at the current moment, and calculate the real-time evaluation value P of the real-time driving record. now , if P now <P th , where P th If the abnormal assessment threshold is exceeded, an abnormal reminder is sent and the user's real-time interaction mode at the current moment is obtained;

[0037] Step S403: Obtain the interaction mode matching order corresponding to the real-time user status, set the interaction mode with the largest correlation value as the expected interaction mode, if the real-time interaction mode is not the expected interaction mode, adjust the user's current interaction mode to the expected interaction mode, if the real-time interaction mode is the expected interaction mode, send an abnormal reminder to the user.

[0038] In order to better implement the above method, a multimodal vehicle-machine interaction system is also proposed. The interaction system includes an interaction information analysis module, an interaction mode analysis module, a state mode matching module and a real-time interaction adjustment module;

[0039] The interactive information analysis module is used to collect driving data and extract interactive behaviors during each driving process of the user, generating a set of interactive behaviors for a driving record; conduct multi-dimensional analysis of driving data in any driving record, and identify the user status in any driving record;

[0040] The interaction pattern analysis module is used to analyze the interactive behavior set in each driving record and identify and classify the user's interaction pattern; conduct a multi-dimensional evaluation of the driving data in any driving record and identify abnormal situations in any driving record;

[0041] The state pattern matching module is used to analyze the correlation between different user states and interaction modes based on the abnormal recognition results of any driving record. For any user state, interaction pattern matching conditions are established to confirm the matching priority of various interaction modes.

[0042] The real-time interaction adjustment module is used to collect the user's current driving data in real time, generate real-time driving records and identify the real-time user status; perform quality assessment on the real-time driving records, and make abnormal adjustments to the user's interaction mode based on the identification of real-time interaction modes.

[0043] Furthermore, the interaction information analysis module includes a historical interaction collection unit and a user status identification unit;

[0044] The historical interaction collection unit is used to collect driving data of each user during their driving process and extract interactive behaviors to generate a set of interactive behaviors for a driving record; the user state identification unit is used to conduct multi-dimensional analysis of the driving data in any driving record and identify the user state in any driving record.

[0045] Furthermore, the interaction pattern analysis module includes a pattern recognition and division unit and an interaction quality assessment unit;

[0046] The pattern recognition and division unit is used to analyze the set of interactive behaviors in each driving record and identify and divide the user's interaction pattern; the interaction quality assessment unit is used to conduct a multi-dimensional assessment of the driving data in any driving record and identify abnormal situations in any driving record.

[0047] Furthermore, the state pattern matching module includes a pattern adaptation association unit and an expectation pattern matching unit;

[0048] The mode adaptation association unit is used to perform association analysis on the adaptation between different user states and interaction modes based on the abnormal recognition results of any driving record; the expected mode matching unit is used to establish interaction mode matching conditions for any user state and confirm the matching priority of various interaction modes.

[0049] Furthermore, the real-time interaction adjustment module includes a real-time status acquisition unit and an interaction anomaly adjustment unit;

[0050] The real-time status acquisition unit is used to collect the user's current driving data in real time, generate real-time driving records and identify the real-time user status; the interaction anomaly adjustment unit is used to evaluate the quality of the real-time driving records and make abnormal adjustments to the user's interaction mode based on the identification of the real-time interaction mode.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. This invention collects and analyzes multi-dimensional data to accurately identify the user's driving status and provides an interaction mode suitable for the user based on the user's driving status. This can help improve the user's interactive experience in vehicle-to-machine interaction and reduce the negative experience caused by incorrect interactive behavior.

[0053] 2. This invention uses historical driving data to detect anomalies, continuously analyzes user behavior habits and scenario characteristics, and establishes a correlation between user status and interaction mode. This helps users match efficient interaction modes in different scenarios, effectively improving user adaptability and user experience in different scenarios.

[0054] 3. The present invention can evaluate the interaction quality in real time and set adjustment mechanisms by detecting status and user behavior in real time, which can greatly improve the response speed and reliability of the system and help users to interact smoothly and safely in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of the steps of a multimodal vehicle-computer interaction method based on artificial intelligence;

[0056] Figure 2 This is a structural diagram of a multimodal vehicle-computer interaction system based on artificial intelligence. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] Example: Figures 1 to 2 As shown, the present invention provides a multimodal vehicle-computer interaction method based on artificial intelligence, and the interaction method includes the following steps:

[0059] Step S100: collecting driving data of the user during each driving process and extracting interactive behaviors to generate an interactive behavior set for a driving record; conducting multi-dimensional analysis on the driving data in any driving record and identifying the user status in any driving record;

[0060] Wherein, step S100 includes the following steps:

[0061] Step S101: Several monitoring devices are installed internally. Data is collected by various monitoring devices at unit time points to obtain a set of driving data during the driving process and generate a driving record. The data collected by any monitoring device is set as monitoring data of one dimension. The monitoring data of each dimension is aligned according to the monitoring time point to obtain the monitoring data of each dimension at any time point and generate a set of monitoring data for each time point.

[0062] Step S102: Pre-construct an interactive behavior database, wherein the interactive behavior database includes several interactive behaviors, and each interactive behavior is matched with several feature dimensions and a feature value range corresponding to each feature dimension; arbitrarily select a monitoring data set and an interactive behavior at a time point; if the monitoring data of each dimension in the selected monitoring data set are the same as the feature dimensions of the selected interactive behavior and are within the corresponding feature data range, then the selected interactive behavior is set as the target interactive behavior at the selected time point; if the selected time point corresponds to several interactive behaviors, then the interactive behavior with the largest number of feature dimensions is selected and set as the target interactive behavior; the target interactive behaviors at each time point are summarized to obtain the interactive behavior set of a driving record;

[0063] Step S103: Construct a two-dimensional rectangular coordinate system with the time point as the horizontal coordinate and the monitoring data as the vertical coordinate, present the changes in the monitoring data of each dimension, and obtain the data difference between any two adjacent time points in any dimension. Preset a time window containing several time points, divide the monitoring data of each dimension in the two-dimensional rectangular coordinate system into several time intervals, and obtain the average difference of each dimension in different time intervals;

[0064] Step S104: Randomly select a time interval and aggregate the average differences of each dimension in the selected time interval to obtain a difference set for the selected time interval; pre-build a state database that stores several types of user states, and any user state corresponds to an expected difference range of several feature dimensions; randomly select a user state, and if the average differences of each dimension are all within the expected difference range of the feature dimension corresponding to the selected user state, then set the selected user state as the user state for the selected time interval;

[0065] Step S105: If there is a same time interval between two time intervals and the user status of the two time intervals is the same, then the two time intervals are combined to obtain various user statuses of the user in the driving record and the time interval corresponding to each user status.

[0066] Step S200: Analyze the interactive behavior set in each driving record and identify and classify the user's interaction mode; conduct a multi-dimensional evaluation of the driving data in any driving record and identify abnormal situations in any driving record;

[0067] Wherein, step S200 includes the following steps:

[0068] Step S201: Randomly select an interaction behavior set from a driving record, sort the interaction behaviors in the interaction behavior set according to the chronological order of the time intervals, and obtain the behavior occurrence sequence of the selected driving record; randomly select an interaction behavior, set the position of the selected interaction behavior in the behavior occurrence sequence to a, and simultaneously obtain the interaction behavior with the position a+1 in the behavior occurrence sequence, and set them as the two target interaction behaviors;

[0069] Step S202: Obtain the order of occurrence of each interaction behavior in each of the remaining driving records, and obtain the position order of the two target interaction behaviors in each driving record. If the positions of the two target interaction behaviors are adjacent to each other and the order is the same as that of the selected driving record, then set the driving record containing the two target interaction behaviors as the target driving record;

[0070] Step S203: Count the number of target driving records as m, and set the total number of all driving records as M total , then the proportion of the target driving record is η=m / M total ; Preset a quantity ratio threshold η th , if η<η th , then reselect the target interaction behavior; if η≥η th , then the two target interaction behaviors are set as the expected behavior combination, and the target interaction behavior with the order a+2 in the behavior occurrence sequence is continued to recalculate the target driving record proportion threshold corresponding to the three target interaction behaviors until η<η is satisfied. th So far, the generated expected behavior combination is set as an expected interaction mode of the user;

[0071] Step S204: Pre-construct an interaction pattern set containing several interaction patterns. Any interaction pattern corresponds to an interaction behavior set. If the order of several interaction behaviors in the expected interaction pattern is the same as that of the interaction pattern, an interaction pattern of the user is obtained. The remaining interaction behaviors in the expected interaction pattern are re-pattern-matched to obtain several interaction patterns of the user.

[0072] Step S205: arbitrarily select a dimension. If the selected dimension is not a characteristic dimension of any interactive behavior, set the selected dimension as the target dimension to obtain several target dimensions of any driving record; preset the corresponding expected value range (x1, x2) for any target dimension, and arbitrarily select the monitoring data of the i-th target dimension at the t-th time point from the selected driving record as x1, x2. (i,t) , if x (i,t) ∈(x1,x2), then the deviation amplitude β of the i-th target dimension at the t-th time point is obtained (i,t) =0, otherwise according to the formula:

[0073]

[0074] Among them, Max() is the maximum value function; the deviation amplitude β of the i-th target dimension at the t-th time point is calculated (i,t) ;

[0075] Step S206: Preset an initial evaluation value P0 according to the formula:

[0076]

[0077] Where i is a positive integer and i∈[1,c], c is the number of target dimensions in the selected driving record; the evaluation value P of the selected driving record at time point t is calculated t ; Preset an abnormal assessment threshold P th , if P t <P th , then the tth time point of the selected driving record is set as the abnormal time point;

[0078] Example 1: Set the selected driving record to contain three target dimensions, and the deviation amplitudes of the three target dimensions at a certain time point are 10%, 20% and 10% respectively; set the initial evaluation value to 100, and calculate the evaluation value at a certain time point to be P = 100 × 90% × 80% × 90% = 64.8.

[0079] Step S300: Based on the abnormality recognition results of any driving record, correlation analysis is performed on the adaptability between different user states and interaction modes; for any user state, interaction mode matching conditions are established to confirm the matching priority of various interaction modes;

[0080] Wherein, step S300 includes the following steps:

[0081] Step S301: randomly selecting a driving record, extracting abnormal time points in the selected driving record, and merging adjacent and continuous abnormal time points to obtain an abnormal time interval;

[0082] Step S302: Obtain several interaction modes of the user in the selected driving record, and extract the time interval corresponding to each interaction mode, compare the time interval of each interaction mode with each abnormal time interval, and arbitrarily select the time interval of an interaction mode. If there is an abnormal time interval that has a common time interval with the time interval of the selected interaction mode, the time length of the common time interval is obtained as ΔL, and the time length of the abnormal time interval is set as ΔT. The abnormal time ratio γ of the selected interaction mode is calculated as ΔL / ΔT; a threshold value of abnormal time ratio γ is preset. th , if γ≥γ th , then extract the user status in the common time interval, and set the extracted user status as an abnormal state of the selected interaction mode;

[0083] Step S303: arbitrarily select the jth user state and the kth interaction mode, and count the number of the jth user state as the abnormal state of the kth interaction mode in all driving records as n (j,k) , set the number of k-th interaction modes to N k , the abnormal occurrence frequency of the kth interaction mode in the jth user state is calculated to be f (j,k) =n (j,k) / N k ; Get the total number of interaction modes contained in all driving records as N ’ , the frequency of the kth interaction mode is calculated to be f k =N k / N ’ ; According to the formula:

[0084]

[0085] Calculate the correlation value G(j,k) between the jth user state and the kth interaction mode;

[0086] Example 2: Select a user state as a healthy state and an interaction mode as voice input. In the healthy state, the number of voice input abnormalities is 1. The total number of voice inputs is set to 10, and the abnormality frequency is 10%. The total number of all interaction modes is obtained as 50, and the frequency of voice input is calculated to be 10 / 50=20%. The correlation value between the healthy state and voice input is calculated to be G=20% / 10%=2.

[0087] Step S304: arbitrarily select the j-th user state, extract the correlation values ​​between the j-th user state and various interaction modes, and sort the correlation values ​​from large to small to obtain the matching order of the j-th user state to various interaction modes.

[0088] Step S400: Real-time collection of the user's current driving data, generating a real-time driving record and identifying the real-time user status; performing a quality assessment on the real-time driving record, and adjusting the user's interaction mode based on the identification of the real-time interaction mode;

[0089] Step S400 includes the following steps:

[0090] Step S401: When the user is in a driving state at the current moment, monitoring data of various dimensions at the current moment is acquired and a real-time driving record is generated; the monitoring data of various dimensions in the real-time driving record is compared with the user status in the user status database to obtain the real-time user status of the user at the current moment;

[0091] Step S402: Obtain the expected value range of each dimension, calculate the real-time deviation amplitude of each dimension at the current moment, and calculate the real-time evaluation value P of the real-time driving record. now , if P now <P th , where P th If the abnormal assessment threshold is exceeded, an abnormal reminder is sent and the user's real-time interaction mode at the current moment is obtained;

[0092] Step S403: Obtain the interaction mode matching order corresponding to the real-time user status, set the interaction mode with the largest correlation value as the expected interaction mode, if the real-time interaction mode is not the expected interaction mode, adjust the user's current interaction mode to the expected interaction mode, if the real-time interaction mode is the expected interaction mode, send an abnormal reminder to the user.

[0093] A multimodal vehicle-machine interaction system, the interaction system includes an interaction information analysis module, an interaction mode analysis module, a state mode matching module and a real-time interaction adjustment module;

[0094] The interactive information analysis module is used to collect driving data and extract interactive behaviors during each driving process of the user, generating a set of interactive behaviors for a driving record; conduct multi-dimensional analysis of driving data in any driving record, and identify the user status in any driving record;

[0095] The interaction pattern analysis module is used to analyze the interactive behavior set in each driving record and identify and classify the user's interaction pattern; conduct a multi-dimensional evaluation of the driving data in any driving record and identify abnormal situations in any driving record;

[0096] The state pattern matching module is used to analyze the correlation between different user states and interaction modes based on the abnormal recognition results of any driving record. For any user state, interaction pattern matching conditions are established to confirm the matching priority of various interaction modes.

[0097] The real-time interaction adjustment module is used to collect the user's current driving data in real time, generate real-time driving records and identify the real-time user status; perform quality assessment on the real-time driving records, and make abnormal adjustments to the user's interaction mode based on the identification of real-time interaction modes.

[0098] Among them, the interaction information analysis module includes a historical interaction collection unit and a user status identification unit;

[0099] The historical interaction collection unit is used to collect driving data of each user during their driving process and extract interactive behaviors to generate a set of interactive behaviors for a driving record; the user state identification unit is used to conduct multi-dimensional analysis of the driving data in any driving record and identify the user state in any driving record.

[0100] Among them, the interaction pattern analysis module includes a pattern recognition and division unit and an interaction quality assessment unit;

[0101] The pattern recognition and division unit is used to analyze the set of interactive behaviors in each driving record and identify and divide the user's interaction pattern; the interaction quality assessment unit is used to conduct a multi-dimensional assessment of the driving data in any driving record and identify abnormal situations in any driving record.

[0102] Among them, the state pattern matching module includes a pattern adaptation association unit and an expected pattern matching unit;

[0103] The mode adaptation association unit is used to perform association analysis on the adaptation between different user states and interaction modes based on the abnormal recognition results of any driving record; the expected mode matching unit is used to establish interaction mode matching conditions for any user state and confirm the matching priority of various interaction modes.

[0104] Among them, the real-time interaction adjustment module includes a real-time status acquisition unit and an interaction abnormality adjustment unit;

[0105] The real-time status acquisition unit is used to collect the user's current driving data in real time, generate real-time driving records and identify the real-time user status; the interaction anomaly adjustment unit is used to evaluate the quality of the real-time driving records and make abnormal adjustments to the user's interaction mode based on the identification of the real-time interaction mode.

[0106] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A multimodal vehicle-machine interaction method based on artificial intelligence, characterized by: The interaction method comprises the following steps: Step S100: collecting driving data of the user during each driving process and extracting interactive behaviors to generate an interactive behavior set for a driving record; conducting multi-dimensional analysis on the driving data in any driving record and identifying the user status in any driving record; Step S200: Analyze the interactive behavior set in each driving record and identify and classify the user's interaction mode; conduct a multi-dimensional evaluation of the driving data in any driving record and identify abnormal situations in any driving record; Step S300: Based on the abnormality recognition results of any driving record, correlation analysis is performed on the adaptability between different user states and interaction modes; for any user state, interaction mode matching conditions are established to confirm the matching priority of various interaction modes; Step S400: Real-time collection of the user's current driving data, generation of a real-time driving record and identification of the real-time user status; quality assessment of the real-time driving record, and abnormal adjustment of the user's interaction mode based on the identification of the real-time interaction mode.

2. The multimodal vehicle-computer interaction method based on artificial intelligence according to claim 1, characterized in that: The step S100 includes the following steps: Step S101: Several monitoring devices are installed internally. Data is collected by various monitoring devices at unit time points to obtain a set of driving data during the driving process and generate a driving record. The data collected by any monitoring device is set as monitoring data of one dimension. The monitoring data of each dimension is aligned according to the monitoring time point to obtain the monitoring data of each dimension at any time point and generate a set of monitoring data for each time point. Step S102: Pre-construct an interactive behavior database, wherein the interactive behavior database includes several interactive behaviors, and each interactive behavior is matched with several feature dimensions and a feature value range corresponding to each feature dimension; arbitrarily select a monitoring data set and an interactive behavior at a time point; if the monitoring data of each dimension in the selected monitoring data set are the same as the feature dimensions of the selected interactive behavior and are within the corresponding feature data range, then the selected interactive behavior is set as the target interactive behavior at the selected time point; if the selected time point corresponds to several interactive behaviors, then the interactive behavior with the largest number of feature dimensions is selected and set as the target interactive behavior; the target interactive behaviors at each time point are summarized to obtain the interactive behavior set of a driving record; Step S103: Construct a two-dimensional rectangular coordinate system with the time point as the horizontal coordinate and the monitoring data as the vertical coordinate, present the changes in the monitoring data of each dimension, and obtain the data difference between any two adjacent time points in any dimension. Preset a time window containing several time points, divide the monitoring data of each dimension in the two-dimensional rectangular coordinate system into several time intervals, and obtain the average difference of each dimension in different time intervals; Step S104: Randomly select a time interval and aggregate the average differences of each dimension in the selected time interval to obtain a difference set for the selected time interval; pre-build a state database that stores several types of user states, and any user state corresponds to an expected difference range of several feature dimensions; randomly select a user state, and if the average differences of each dimension are all within the expected difference range of the feature dimension corresponding to the selected user state, then set the selected user state as the user state for the selected time interval; Step S105: If there is a same time interval between two time intervals and the user status of the two time intervals is the same, then the two time intervals are combined to obtain various user statuses of the user in the driving record and the time interval corresponding to each user status.

3. The multimodal vehicle-computer interaction method based on artificial intelligence according to claim 2, characterized in that: The step S200 includes the following steps: Step S201: Randomly select an interaction behavior set from a driving record, sort the interaction behaviors in the interaction behavior set according to the chronological order of the time intervals, and obtain the behavior occurrence sequence of the selected driving record; randomly select an interaction behavior, set the position of the selected interaction behavior in the behavior occurrence sequence to a, and simultaneously obtain the interaction behavior with the position a+1 in the behavior occurrence sequence, and set them as the two target interaction behaviors; Step S202: Obtain the order of occurrence of each interaction behavior in each of the remaining driving records, and obtain the position order of the two target interaction behaviors in each driving record. If the positions of the two target interaction behaviors are adjacent to each other and the order is the same as that of the selected driving record, then set the driving record containing the two target interaction behaviors as the target driving record; Step S203: Count the number of target driving records as m, and set the total number of all driving records as M total , then the proportion of the target driving record is η=m / M total ; Preset a quantity ratio threshold η th , if η<η th , then reselect the target interaction behavior; if η≥η th , then the two target interaction behaviors are set as the expected behavior combination, and the target interaction behavior with the order a+2 in the behavior occurrence sequence is continued to recalculate the target driving record proportion threshold corresponding to the three target interaction behaviors until η<η is satisfied. th So far, the generated expected behavior combination is set as an expected interaction mode of the user; Step S204: Pre-construct an interaction pattern set containing several interaction patterns. Any interaction pattern corresponds to an interaction behavior set. If the order of several interaction behaviors in the expected interaction pattern is the same as that of the interaction pattern, an interaction pattern of the user is obtained. The remaining interaction behaviors in the expected interaction pattern are re-pattern-matched to obtain several interaction patterns of the user. Step S205: arbitrarily select a dimension. If the selected dimension is not a characteristic dimension of any interactive behavior, set the selected dimension as the target dimension to obtain several target dimensions of any driving record; preset the corresponding expected value range (x1, x2) for any target dimension, and arbitrarily select the monitoring data of the i-th target dimension at the t-th time point from the selected driving record as x1, x2. (i,t) , if x (i,t) ∈(x1,x2), then the deviation amplitude β of the i-th target dimension at the t-th time point is obtained (i,t) =0, otherwise according to the formula: Among them, Max() is the maximum value function; the deviation amplitude β of the i-th target dimension at the t-th time point is calculated (i,t) ; Step S206: Preset an initial evaluation value P0 according to the formula: Where i is a positive integer and i∈[1,c], c is the number of target dimensions in the selected driving record; the evaluation value P of the selected driving record at time point t is calculated t ; Preset an abnormal assessment threshold P th , if P t <P th , then the tth time point of the selected driving record is set as the abnormal time point.

4. The multimodal vehicle-computer interaction method based on artificial intelligence according to claim 3, characterized in that: The step S300 includes the following steps: Step S301: randomly selecting a driving record, extracting abnormal time points in the selected driving record, and merging adjacent and continuous abnormal time points to obtain an abnormal time interval; Step S302: Obtain several interaction modes of the user in the selected driving record, and extract the time interval corresponding to each interaction mode, compare the time interval of each interaction mode with each abnormal time interval, and arbitrarily select the time interval of an interaction mode. If there is an abnormal time interval that has a common time interval with the time interval of the selected interaction mode, the time length of the common time interval is obtained as ΔL, and the time length of the abnormal time interval is set as ΔT. The abnormal time ratio γ of the selected interaction mode is calculated as ΔL / ΔT; a threshold value of abnormal time ratio γ is preset. th , if γ≥γ th , then extract the user status in the common time interval, and set the extracted user status as an abnormal state of the selected interaction mode; Step S303: arbitrarily select the jth user state and the kth interaction mode, and count the number of the jth user state as the abnormal state of the kth interaction mode in all driving records as n (j,k) , set the number of k-th interaction modes to N k , the abnormal occurrence frequency of the kth interaction mode in the jth user state is calculated to be f (j,k) =n (j,k) / N k ; Get the total number of interaction modes contained in all driving records as N ’ , the frequency of the kth interaction mode is calculated to be f k =N k / N ’ ; According to the formula: Calculate the correlation value G(j,k) between the jth user state and the kth interaction mode; Step S304: arbitrarily select the j-th user state, extract the correlation values ​​between the j-th user state and various interaction modes, and sort the correlation values ​​from large to small to obtain the matching order of the j-th user state to various interaction modes.

5. The multimodal vehicle-computer interaction method based on artificial intelligence according to claim 4, characterized in that: The step S400 includes the following steps: Step S401: When the user is in a driving state at the current moment, monitoring data of various dimensions at the current moment is acquired and a real-time driving record is generated; the monitoring data of various dimensions in the real-time driving record is compared with the user status in the user status database to obtain the real-time user status of the user at the current moment; Step S402: Obtain the expected value range of each dimension, calculate the real-time deviation amplitude of each dimension at the current moment, and calculate the real-time evaluation value P of the real-time driving record. now , if P now <P th , where P th If the abnormal assessment threshold is exceeded, an abnormal reminder is sent and the user's real-time interaction mode at the current moment is obtained; Step S403: Obtain the interaction mode matching order corresponding to the real-time user status, set the interaction mode with the largest correlation value as the expected interaction mode, if the real-time interaction mode is not the expected interaction mode, adjust the user's current interaction mode to the expected interaction mode, if the real-time interaction mode is the expected interaction mode, send an abnormal reminder to the user.

6. A multimodal vehicle-computer interaction system, configured to execute the artificial intelligence-based multimodal vehicle-computer interaction method according to any one of claims 1 to 5, characterized in that: The interactive system includes an interactive information analysis module, an interactive mode analysis module, a state mode matching module and a real-time interactive adjustment module; The interactive information analysis module is used to collect driving data of the user during each driving process and extract interactive behaviors to generate an interactive behavior set of a driving record; Conduct multi-dimensional analysis of driving data in any driving record and identify the user status in any driving record; The interaction pattern analysis module is used to analyze the interaction behavior set in each driving record, identify and classify the user's interaction pattern; conduct multi-dimensional evaluation of the driving data in any driving record, and identify abnormal situations in any driving record; The state pattern matching module is used to perform correlation analysis on the adaptability between different user states and interaction modes based on the abnormal recognition results of any driving record; for any user state, interaction pattern matching conditions are established to confirm the matching priority of various interaction modes; The real-time interactive adjustment module is used to collect the user's current driving data in real time, generate real-time driving records and identify the real-time user status; Perform quality assessment on real-time driving records and make abnormal adjustments to user interaction patterns based on the recognition of real-time interaction patterns.

7. The multimodal vehicle-machine interaction system according to claim 6, characterized in that: The interaction information analysis module includes a history interaction collection unit and a user status identification unit; The historical interaction collection unit is used to collect the driving data of the user during each driving process and extract the interactive behavior to generate an interactive behavior set of a driving record; The user status identification unit is used to conduct multi-dimensional analysis on the driving data in any driving record and identify the user status in any driving record.

8. The multimodal vehicle-machine interaction system according to claim 6, characterized in that: The interaction pattern analysis module includes a pattern recognition and division unit and an interaction quality assessment unit; The pattern recognition and division unit is used to analyze the interactive behavior sets in each driving record and identify and divide the user's interaction patterns; the interaction quality assessment unit is used to conduct a multi-dimensional assessment of the driving data in any driving record and identify abnormal situations in any driving record.

9. The multimodal vehicle-machine interaction system according to claim 6, characterized in that: The state pattern matching module includes a pattern adaptation association unit and an expected pattern matching unit; The mode adaptation association unit is used to perform association analysis on the adaptation between different user states and interaction modes based on the abnormal recognition results of any driving record; the expected mode matching unit is used to establish interaction mode matching conditions for any user state and confirm the matching priority of various interaction modes.

10. The multimodal vehicle-machine interaction system according to claim 6, characterized in that: The real-time interaction adjustment module includes a real-time status acquisition unit and an interaction anomaly adjustment unit; The real-time status acquisition unit is used to collect the user's current driving data in real time, generate a real-time driving record and identify the real-time user status; the interaction anomaly adjustment unit is used to evaluate the quality of the real-time driving record and make abnormal adjustments to the user's interaction mode based on the identification of the real-time interaction mode.

Images