Vehicle-mounted internet dynamic interaction adaptation method based on multi-modal fusion

Abstract

The application discloses a vehicle-mounted Internet dynamic interaction adaptation method based on multi-modal fusion and relates to the technical field of human-computer interaction. The application obtains multi-modal sensing data and analyzes the passenger identity and seat position association matrix and the vehicle dynamic state code; generates a scene risk coefficient based on the vehicle dynamic state code; when the scene risk coefficient does not exceed the dynamic safety boundary threshold, obtains the spatial distance from the passenger to the target interface and calculates a spatial cost factor in combination with the scene risk coefficient, then retrieves historical interaction records with the identity identifier and the interaction content to obtain a historical correction weight, and corrects the spatial cost factor to generate an interaction execution factor; and determines whether to execute the instruction according to the comparison result of the interaction execution factor and the execution threshold. The application takes the scene risk coefficient as a safety hard constraint, combines the spatial cost and the historical correction to quantize the interaction execution factor layer by layer, and provides dynamic and personalized interaction executability judgment for passengers in different seats within the safety window.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a method for dynamic interaction adaptation of in-vehicle Internet based on multimodal fusion. Background Technology

[0002] As the intelligent transformation of the automotive industry continues to deepen, automobiles are gradually evolving from traditional means of transportation into mobile intelligent spaces integrating travel, work, and entertainment. The level of human-machine interaction in intelligent cockpits has become a key indicator for measuring the degree of vehicle intelligence. Against this backdrop, multimodal fusion and proactive interaction in in-vehicle interactions have become a development focus. At the same time, the composition of cockpit occupants is becoming increasingly complex, and vehicles need to simultaneously meet the personalized interaction needs of drivers and multiple passengers. How to provide accurate, efficient, and non-interfering interactive experiences for passengers in different positions within a multi-passenger cockpit environment has become an important research direction in the field of intelligent cockpit human-machine interaction.

[0003] To achieve precise interactive control in multi-passenger scenarios, several technological approaches have emerged in this field. Regarding identity recognition, existing technologies identify users by collecting facial or voiceprint information and loading pre-stored preference settings, enabling personalized services with different configurations for different users. In terms of spatial partitioning, seat presence detection and sound source localization divide the cabin into different physical areas and bind commands to these areas, achieving independent control of each area in the audio, display, and touch domains. Identity recognition maps users to preferences, and spatial partitioning binds areas to commands; however, all of these technological approaches are built upon traditional distributed cockpit architectures.

[0004] When faced with the need for multi-dimensional information fusion and decision-making, existing in-vehicle internet interaction technologies lack a systematic mechanism for real-time joint decision-making based on multi-dimensional contextual information, resulting in a lack of cross-domain collaborative capabilities in scenarios with concurrent interactions among multiple passengers. Especially when driving scenarios change dynamically, the feasibility of the same interactive action performed by users with the same identity and location can vary. Existing in-vehicle internet interaction technologies face the technical challenge of failing to provide a unified, safe, and personalized feasibility evaluation scale for concurrent interactions among multiple passengers in dynamic driving environments. Summary of the Invention

[0005] Therefore, it is necessary to provide a method for dynamic interaction adaptation of in-vehicle Internet based on multimodal fusion to address the above-mentioned technical problems.

[0006] The present invention adopts the following technical solution: This invention provides a method for dynamic interaction adaptation of in-vehicle internet based on multimodal fusion, including: Acquire multimodal sensor data and parse out the occupant identity and seat position association matrix, as well as the multi-dimensional vehicle dynamic state code of the current driving scenario; Data from multiple dimensions are extracted from the vehicle dynamic state code to calculate the risk sub-score for each dimension. The maximum and second-largest values ​​of the risk sub-scores are taken and combined with the relative difference between the maximum and second-largest values ​​to obtain the scenario risk coefficient. The system obtains a sequence of scene risk coefficients within a preset time window prior to the current moment, and dynamically generates a safety boundary threshold based on the statistical characteristic values ​​of the scene risk coefficient sequence. When the scene risk coefficient is greater than the preset safety boundary threshold, the interaction execution factor is set to a preset high-level clamping value, and the interaction command is deemed unsuitable for execution. When the scene risk coefficient is less than or equal to the safety boundary threshold, the system determines the seat reference point of the interaction command issuer based on the identity and seat position association matrix and obtains the spatial distance to the center point of the target interaction interface. The system calculates a spatial cost factor based on the scene risk coefficient and the spatial distance. Then, the system retrieves historical interaction records using the issuer's identity identifier and the interaction content type code of the current interaction command to obtain historical correction weights. The historical correction weights characterize the user's historical intervention in the interaction content type. The system uses the historical correction weights to correct the spatial cost factor and generate an interaction execution factor characterizing the executability of the current interaction command. Whether to execute the current interactive instruction is determined based on the comparison result between the interactive execution factor and the preset execution threshold.

[0007] Preferably, the computational space cost factor includes: Using the scenario risk coefficient as the basic weight, the degree of closeness between the spatial distance and the preset ergonomic limit distance as the adjustment coefficient, and the ratio of the difference between the maximum and second largest values ​​in each dimension of risk sub-scores to the gap coefficient as the composite risk adjustment term, the spatial cost factor is obtained by multiplying the basic weight, the adjustment coefficient and the composite risk adjustment term.

[0008] Preferably, the method for calculating the scenario risk coefficient includes: The standard deviation of each risk sub-score is used as the gap coefficient, the maximum value in the risk sub-score is used as the main risk component, and the second largest value in the risk sub-score is used as the correction increment. The contribution of the correction increment to the scenario risk coefficient is inversely correlated with the relative gap obtained by dividing the difference between the maximum value and the second largest value by the gap coefficient. When the fusion result exceeds the preset upper limit, the scenario risk coefficient is clamped to the preset upper limit.

[0009] Preferably, the calculation of risk sub-scores for each dimension includes: The speed risk sub-score is calculated based on the standardized vehicle speed value using a monotonically increasing power function mapping relationship; the handling complexity sub-score is calculated based on the variance of the steering wheel angle change and the frequency of the brake master cylinder pressure change within a set time window; the road type sub-score is calculated based on the ratio of the legal maximum speed limit corresponding to the road type code to the vehicle's own maximum speed limit; and the traffic density sub-score is calculated based on the standardized value of the distance to the vehicle ahead, with a higher traffic density sub-score for a smaller standardized value of the distance to the vehicle ahead.

[0010] Preferably, the calculation of the control complexity sub-score based on the variance of the steering wheel angle change and the frequency of the brake master cylinder pressure change within a set time window includes: After dividing the steering wheel angle variance and braking frequency by their respective upper limits to make them dimensionless, they are fused using a geometric mean relationship. The fused values ​​are then linearly mapped to a standardized interval as a sub-score for handling complexity.

[0011] Preferably, obtaining the historical correction weights includes: A joint index is constructed using the sender's identity identifier and the interaction content type code. The historical interaction record database is searched, and records that completely match the identity identifier, have the same interaction content type, and whose difference between the scenario risk coefficient at the time of interaction and the current scenario risk coefficient is within a preset tolerance range are selected as valid samples from the search results. If the effective sample size is less than the preset minimum sample size threshold, a neutral value is dynamically calculated based on the proportion of interaction records with intervention in all of the user's historical interaction records, and this neutral value is used as the historical correction weight. If the effective sample size is sufficient, a weighted average is performed based on the chronological order of each effective sample and the corresponding user intervention status coefficient to obtain the historical correction weight.

[0012] Preferably, the weighted average based on the chronological order of each valid sample and the corresponding user intervention state coefficient specifically includes: The weight coefficients of each valid sample are determined according to the exponential decay law based on the chronological order of the valid samples. The valid samples that are closer to the current time have larger weight coefficients. The intervention state coefficients of each valid sample are weighted and averaged according to the weight coefficients. The value of the intervention state coefficient when the user does not actively correct or cancel the interaction result is greater than the value when the user actively corrects or cancels the interaction result. The weighted average result is used as the historical correction weight.

[0013] Preferably, generating the interaction execution factor characterizing the executability of the current interaction instruction includes: Using the spatial cost factor as the base value, the degree of positive deviation of the historical correction weight relative to the neutral value as the deviation amount, and the product of the scenario risk coefficient and the deviation amount as the reduction index, the interaction execution factor is obtained by multiplying the base value by the exponential decay factor based on the reduction index. The greater the degree of positive deviation of the historical correction weight relative to the neutral value, the greater the reduction of the interaction execution factor.

[0014] Preferably, determining whether to execute the current interaction instruction based on the comparison result between the interaction execution factor and the preset execution threshold includes: When the interaction execution factor is less than the execution threshold, the instruction is determined to be allowed to be executed. When the interaction execution factor is greater than or equal to the execution threshold, the instruction is determined not to be executed for the time being. When multiple interaction instructions are pending processing at the same time, the instructions with interaction execution factors less than the execution threshold are included in the execution set. The multiple instructions in the execution set are arranged in ascending order according to the interaction execution factor from smallest to largest, and the one with the smallest interaction execution factor is executed first.

[0015] Preferably, the in-vehicle internet dynamic interaction adaptation method based on multimodal fusion further includes: After each trip, the user intervention rate of all executed instructions in that trip is calculated. The user intervention rate is the proportion of instructions that are actively corrected or canceled by the user within a preset intervention window after execution. When the user intervention rate exceeds the upper limit of the preset intervention rate tolerance range, the preset execution threshold is gradually increased. When the user intervention rate is lower than the lower limit of the intervention rate tolerance range, the preset execution threshold is gradually decreased. The corrected execution threshold is stored in the vehicle storage unit for subsequent trips.

[0016] The above-mentioned technical solution adopted in this invention can achieve the following beneficial effects: In the in-vehicle internet dynamic interaction adaptation method based on multimodal fusion provided by this invention, a three-layer progressive joint quantification logic is constructed: The first layer determines the dominant risk pattern by the difference between the maximum and second-largest values ​​in the multi-dimensional risk sub-scores. Single high-risk and compound high-risk have different safety margins, thereby generating a scenario risk coefficient as a prerequisite hard constraint. The second layer, within the safety window, couples this coefficient with the spatial distance from the seat reference point to the target interface to form a spatial cost factor, dynamically amplifying or reducing the operational cost weight of environmental risks, achieving cross-dimensional fusion of risk and spatial location. The third layer, based on the above unified quantification results, uses the user's historical intervention rate as a fine-tuning item for vertical correction, and sets a correction ratio coefficient to prevent historical factors from excessively dominating real-time safety judgments. The three-layer structure tightens progressively and is interdependent, forming a complete judgment chain from safety boundaries to real-time costs and then to personalized adaptation.

[0017] This invention enables the scenario risk coefficient to simultaneously perform the dual functions of safety gating and spatial cost weighting through a progressively layered quantitative logic, thereby achieving dynamic coupling between environmental risk and operational cost. Based on the results of the first two layers of physical quantification, it makes limited fine-tuning by combining historical corrections, avoiding subjective factors from overriding objective safety constraints. Under the premise of ensuring the absolute priority of hard safety constraints, it achieves a unified quantification evaluation of the feasibility of concurrent interactions among multiple passengers. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0019] Figure 1 This is a schematic diagram of the process for the in-vehicle internet dynamic interaction adaptation method based on multimodal fusion provided in this specification; Figure 2 This is the audio zoning diagram of the in-vehicle interactive system provided in this manual; Figure 3 This is a schematic diagram of the interaction factor generation logic provided in this specification. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this invention.

[0021] In terms of identity recognition and personalized configuration, existing technologies identify users by collecting facial or voiceprint information, retrieve pre-stored preference settings from the cloud, and adjust vehicle configurations such as seats, air conditioning, and multimedia accordingly. This achieves the goal of personalized service for "different users, different configurations," allowing each passenger to obtain a cabin environment that suits their habits. Regarding spatial zoning and area binding, existing technologies divide the cabin into different physical areas through seat presence detection and sound source localization, binding interactive commands to the areas where commands are issued. In the audio field, a multi-zone dynamic sound field intervention mechanism is constructed to achieve independent acoustic control of each area. In the display and touch fields, independent adjustment of each display area is achieved through area screen processing, effectively reducing mutual interference between cross-area commands. Figure 2 The image shows the audio partitioning diagram of the in-vehicle interactive system.

[0022] However, these paths operate relatively independently at the system architecture level, and the information transmission and coordination mechanisms between them have not yet been fully established.

[0023] Existing technologies employ a rigid, coarse-grained binary separation logic when dealing with driving safety and interactive experience. Restrictions on interaction based on driving status are typically implemented in the form of hard thresholds or blacklists / whitelists. For example, if the vehicle speed exceeds a certain fixed value, video playback is forcibly disabled or touch operation is locked. This results in the system either excessively restricting the experience under low-risk conditions or still allowing high-cost interactions under medium- to high-risk conditions, creating a rigid opposition between safety margin and experience quality.

[0024] This invention uses driving scenario risk as a hard gating constraint, combines user identity and location characteristics, and comprehensively quantifies interaction execution factors. In this way, it breaks down the information gap between scenario risk, identity recognition and spatial partitioning within a unified mathematical framework, and maximizes the satisfaction of the personalized real-time interaction needs of passengers in different seats within a safe and permissible window.

[0025] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 This is a flowchart illustrating the in-vehicle internet dynamic interaction adaptation method based on multimodal fusion as described in this specification, which specifically includes the following steps: S101: Acquire multimodal sensor data and parse out the occupant identity and seat position association matrix, as well as the multi-dimensional vehicle dynamic state code of the current driving scenario.

[0027] Multi-sensor acquisition is synchronously triggered at a fixed sampling frequency. Physical parameters with different dimensions are transformed into a standardized numerical range through deterministic mapping rules. Finally, the identity-location association matrix and vehicle dynamic state code are parsed from the standardized features.

[0028] The onboard computing platform synchronously triggers the following data channels at a fixed sampling frequency of 30 Hz: video frames containing the occupant's face and upper body posture are acquired through an infrared camera array deployed in the cabin; cabin audio signals are acquired through a distributed microphone array; pressure distribution data for each seat is acquired through a pressure sensor array under the seat cushion; vehicle speed, yaw rate, steering wheel angle, and brake master cylinder pressure signals are acquired through a CAN (Controller Area Network) bus subscription; and road grade, speed limit information, and distance to the vehicle ahead are acquired through navigation and positioning and external perception modules.

[0029] The above physical quantities are transformed to a unified standardized value range of [0,1] through piecewise linear mapping. The three-dimensional coordinates of the face space are linearly mapped to a standardized range using the driver's seat reference point as the origin. The ranges for the X-axis (vehicle longitudinal), Y-axis (vehicle lateral), and Z-axis (vehicle vertical) are set to [-0.6, 1.0], [-0.3, 1.2], and [-0.4, 0.4], respectively. These ranges are estimated based on the cabin dimensions of a typical B-class sedan and can be fine-tuned according to specific vehicle models. The azimuth angle of the sound source is mapped with the vehicle's longitudinal centerline as the zero-degree reference, and a range of -90° to 90° is set for mapping. Vehicle speed is piecewise linearly mapped from 0 to the maximum speed limit (e.g., 150 km / h). The mapping slope is gentle in the low-speed range and steeply increases in the high-speed range. In this embodiment, 60 km / h is used as the dividing point; speeds less than or equal to the dividing point are considered low-speed, and speeds greater than the dividing point are considered high-speed. All other physical quantities are mapped according to their respective GB / T standards. The 10000-2023 "Anthropometric Dimensions of Chinese Adults" and vehicle dynamics standards are pre-calibrated to linearly map the original dimensional values ​​to a standardized numerical range, and clamp values ​​that exceed the reasonable range to the boundary values. For example, the original seat pressure value of 0 to 150 kg is linearly mapped to the range [0,1], and the original yaw rate value of -50° / s to 50° / s is linearly mapped to the range [0,1]. The road type coding rules are as follows: for example, 01 represents expressway, 02 represents national / provincial highway, 03 represents urban arterial road, and 04 represents urban branch road / other road.

[0030] The vehicle speed, yaw rate, road type code, and other values ​​are output in a structured data format as vehicle dynamic status codes and basic environmental information.

[0031] The facial feature point set of each occupant is compared with the user facial feature templates pre-stored in the vehicle's storage unit. The identity identifier corresponding to the template with the highest similarity exceeding a set threshold of 0.85 is selected as the identity recognition result, and each occupant's identity identifier is associated with its spatial three-dimensional coordinates. The spatial coordinates of each occupant are logically compared with the preset spatial range of each seat to determine the specific seat position identifier of each occupant, forming a binary association pair between the seat position identifier and the occupant identity identifier. This association pair is output in the form of an N-row × 2-column matrix (N is the total number of occupants detected in the vehicle). The first column is the seat position identifier, and the second column is the occupant identity identifier. If there is no occupant in a certain seat position, the corresponding row is output as null. If a pressure signal exists in a seat but the visual modality cannot match the existing user template, a temporary visitor identifier is automatically generated and marked as a temporary visitor.

[0032] This step provides standardized input parameters with clear physical meaning and uniform dimensions for subsequent occupant intention recognition or interactive decision calculations based on standardized parameters.

[0033] S102: Extract data from multiple dimensions of the vehicle dynamic state code to calculate the risk sub-score for each dimension. Take the maximum and second-largest values ​​of the risk sub-scores and combine them with the relative difference between the maximum and second-largest values ​​to obtain the scenario risk coefficient.

[0034] The safety constraints imposed by driving scenarios on interactive activities are objectively existing physical constraints. When a vehicle is in a complex maneuvering state, such as high-speed driving or continuous turning, the driver's cognitive resources are heavily consumed. This step aims to quantify driving safety constraints into numerical factors with clear physical meaning, serving as hard safety constraints for subsequent interactive decisions.

[0035] The following dimensions of data are extracted from the standardized values ​​output by S101 to calculate risk sub-scores, with each sub-score falling within the standardized value range: Speed ​​risk sub-score: Calculated based on standardized vehicle speed values. A monotonically increasing power function is used for mapping, with the power function exponent set to 0.6. This ensures that the sub-score increases gradually in the low-range of standardized vehicle speed values ​​and increases accordingly in the high-range. The power function exponent is a pre-calibrated nonlinear factor based on vehicle dynamics characteristics, typically ranging from 0.5 to 0.8 for passenger vehicles, used to characterize the nonlinear relationship between wind resistance and driving force with vehicle speed.

[0036] The handling complexity sub-score is calculated based on the variance of the steering wheel angle change and the frequency of the brake master cylinder pressure change within a set time window T. In this embodiment, the time window T is 15 seconds, corresponding to 10 times the typical driver's cognitive reaction cycle of 1.5 seconds, to fully cover continuous curves or stop-and-go driving conditions. The steering wheel angle variance and braking frequency are divided by their respective upper limits for dimensionless calculation, and then fused using a geometric mean relationship. That is, the basic value of handling complexity is equal to the square root of the product of the dimensionless variance and the dimensionless frequency. The fused value is then linearly mapped to a standardized interval as the handling complexity sub-score. The upper limit of the steering wheel angle variance is 540°, which is the range from the center position of the steering wheel to the full rotation angle, and the upper limit of the braking frequency is 60 times / minute.

[0037] Road type sub-score: Calculated based on the ratio of the legally mandated maximum speed limit corresponding to the road type code to the vehicle's own maximum speed limit, utilizing... The model inversely maps the ratio to the (0,1] interval, then the road type sub-score ,in This represents an exponential function with the natural constant as its base. This represents the ratio of the legally mandated maximum speed limit corresponding to the road type code to the vehicle's own maximum speed limit. In this embodiment, the vehicle's own maximum speed limit is 150 km / h. The legally mandated maximum speed limits for each road type are: 120 km / h for expressways, 80 km / h for national or provincial highways, 60 km / h for urban arterial roads, and 40 km / h for urban secondary roads or other roads. Substituting these ratios, the risk benchmark value corresponding to each road type is obtained.

[0038] Traffic density sub-score: based on standardized values ​​of distance to vehicles ahead The closer the distance to the vehicle ahead, the greater the traffic density is indicated by the traffic density sub-score. The higher the traffic density score, the more attention the driver needs to pay to maintain following distance.

[0039] After obtaining the risk sub-scores for the above multiple dimensions, the maximum and second-largest values ​​among the sub-scores are taken to calculate the scenario risk coefficient. Scenario Risk Coefficient The calculation methods include:

[0040] , (1) in, The coefficient of variation represents the degree of dispersion of risk sub-scores across different dimensions. This represents the standard deviation function, where... This represents the standard deviation of all risk sub-scores. This represents the maximum value among the risk sub-scores. This represents the second largest value in the risk sub-score; when The smaller the value, the closer the risk level of the dimension corresponding to the maximum and second-largest values ​​in the risk sub-scores is. This indicates the relative significance of the difference with respect to the global dispersion. A more significant difference indicates that the overall risk is dominated by the maximum value in the risk sub-score, and the second largest value has a smaller impact as an increment in the calculation of the scenario risk coefficient. Conversely, a smaller difference indicates a larger impact of the second largest value as an increment. Specifically, when... At that time, clamp it to 1.0.

[0041] This step, based on vehicle dynamic state coding and external environment information, generates a scenario risk coefficient for the current driving scenario through quantification.

[0042] S103: Obtain the scene risk coefficient sequence within a preset time window before the current moment, and dynamically generate a safety boundary threshold based on the statistical characteristic value of the scene risk coefficient sequence; when the scene risk coefficient is greater than the preset safety boundary threshold, set the interaction execution factor to a preset high-level clamping value and determine that the interaction command is not suitable for execution; when the scene risk coefficient is less than or equal to the safety boundary threshold, determine the seat reference point of the interaction command issuer based on the identity and seat position association matrix and obtain the spatial distance to the center point of the target interaction interface, calculate the spatial cost factor based on the scene risk coefficient and the spatial distance, and then retrieve historical interaction records using the identity identifier of the issuer of the current interaction command and the interaction content type code to obtain historical correction weights. The historical correction weights characterize the user's historical intervention in the interaction content type, and use the historical correction weights to correct the spatial cost factor to generate an interaction execution factor characterizing the executability of the current interaction command.

[0043] This step proceeds layer by layer from security boundary assessment to real-time interaction cost evaluation, and then to historical experience correction, ultimately outputting an interaction execution factor that characterizes the feasibility of the current interaction action. This step aims to establish a unified quantitative framework with security as a hard constraint, breaking down the information gap between identity recognition and spatial partitioning.

[0044] When the scenario risk is already high, non-emergency interactions should not be executed regardless of the ease of operation, response speed, or historical performance of the interaction itself. A safety boundary threshold is obtained based on the statistical values ​​of the scenario risk coefficient sequence within the time window T preceding the current moment. Here, the time window T is consistent with the setting in S102, being 10 times the typical cognitive reaction cycle of a driver (1.5 seconds). The sum of the mean and standard deviation of the scenario risk coefficient sequence within the time window T is taken as the current safety boundary threshold. When the scenario risk coefficient exceeds the safety boundary threshold, it indicates that the current risk has abnormally increased, and the interaction execution factor is directly output as a preset high-level restraint value of 1.0, representing that execution is not appropriate.

[0045] When occupants touch different interactive interfaces from different seat positions, the degree of forward lean and arm extension varies. The greater the distance, the more attention resources are required, and the greater the interference with driving posture. Based on the occupant's seat position and the target interactive interface type, the coordinates of the seat reference point and the interface center point are obtained from system preset constants. When a function can be operated through multiple interfaces, the interface with the closest spatial distance is selected as the target interface to minimize operational costs. The three-dimensional Euclidean distance from the seat reference point to the center point of the target interface is calculated.

[0046] Because higher environmental risks necessitate stricter constraints on the same spatial operation costs, a spatial cost factor is derived by combining the scenario risk coefficient with the three-dimensional Euclidean distance from the seat reference point to the target interface center point. Spatial Cost Factor The calculation methods include:

[0047] (2) in, Indicates the risk coefficient of the scenario. This represents the maximum value among the risk sub-scores. This represents the second largest value in the risk sub-score. Indicates the difference coefficient; A larger value indicates a more complex risk scenario. The arm length is defined as the ergonomically compliant maximum reach. In this embodiment, the arm length data is based on the 95th percentile for adult males in GB / T 10000-2023 "Anthropometric Dimensions of Chinese Adults". , This represents the three-dimensional Euclidean distance from the seat reference point to the center point of the target interface. This represents the Sigmoid function.

[0048] Furthermore, if an interaction has historically been stable and reliable for the user, the execution criteria can be appropriately relaxed. Therefore, a historical dimension is introduced as a longitudinal correction, allowing the system to retain the user's personalized adaptation capabilities based on real-time physical evaluation.

[0049] A composite index is constructed using the three fields of the current interaction command: the sender's identity identifier and the interaction content type code. This composite index is used to retrieve the historical interaction record database. The historical interaction record database is continuously updated based on all past interactions. Each record contains at least: a timestamp, a user's identity identifier, an interaction content type code, a scenario risk coefficient at the time of interaction, and a binary identifier indicating whether the user actively corrected or canceled the interaction within a set time window T after execution. Here, the time window T is consistent with the setting in S102, which is 10 times the typical cognitive reaction cycle of a driver (1.5 seconds) to cover the entire process of the user's perception, judgment, and correction of the interaction result.

[0050] Valid samples are selected from the search results. The selection criteria are: exact match in identity identifiers, identical interaction content types, and the difference between the scenario risk coefficient at the time of interaction and the current scenario risk coefficient is within a preset tolerance range. If the number of valid samples is less than the preset minimum sample size threshold, it is determined that the current interaction type lacks sufficient historical data, and the historical correction weight is set to a neutral value. In this embodiment, the preset tolerance range is ±0.15, the preset minimum sample size threshold is 5 records, and the neutral value of the historical correction weight is obtained by the proportion of intervention-related interaction records in all of the user's historical interaction records. The historical correction weight is obtained based on valid samples. The calculation methods include:

[0051] (3) in, This represents the total number of valid samples. This represents the intervention status coefficient. A value of 1 indicates that the user did not intervene, and a value of 0 indicates that the user intervened. It refers to an exponential function with the natural constant as its base; where intervention refers to the user actively modifying or canceling the interaction result after the interaction is executed.

[0052] Figure 3 This is a schematic diagram illustrating the logic for generating interaction factors in this step, as follows: Figure 3 The spatial cost factor is adjusted using historical correction weights, resulting in the final interaction execution factor. (Interaction execution factor) The calculation methods include:

[0053] (4) in, This represents the scenario risk coefficient. The higher the scenario risk, the more the calculation of the interaction execution factor is dominated by the real-time scenario, and the reference value of historical experience should be reduced. Indicates historical adjustment weights, This represents the neutral value of the historical adjustment weights. This represents the space cost factor.

[0054] This step generates an interaction execution factor, which represents the degree to which the current interaction action is executable. The lower the value of the interaction execution factor, the more suitable the current interaction action is to be executed in the current context, and the higher the value, the less suitable it is to be executed.

[0055] S104: Determine whether to execute the current interactive instruction based on the comparison result between the interactive execution factor and the preset execution threshold.

[0056] This step transforms interactive execution factors into specific execution decisions, and adaptively adjusts the decision parameters based on the actual execution results.

[0057] Set a single execution threshold. When the interaction execution factor is less than the execution threshold, the instruction is determined to be allowed to execute; when the interaction execution factor is greater than or equal to the execution threshold, the instruction is determined to be temporarily not executed and can be sent to the cache queue to wait for subsequent re-evaluation or discarded at an appropriate time.

[0058] When multiple interactive commands are pending processing simultaneously, commands with an interaction execution factor less than the execution threshold are assigned to the execution set, while the rest are assigned to the non-execution set. Commands in the execution set are arranged in ascending order of their interaction execution factors, with the command with the smallest factor executed first. When both commands from the driver and passengers are present in the execution set, the driver's command is automatically prioritized.

[0059] The initial value of the execution threshold is preset based on the system design goals and continuously optimized during operation through an adaptive correction mechanism. In this embodiment, the initial value ranges from 0.40 to 0.60; 0.6 is used as an example in this embodiment, and the specific value can be determined based on system calibration test data. After each run, the user intervention rate of all executed instructions in that run is calculated, that is, the proportion of instructions actively corrected or canceled by the user within the preset intervention window after execution. When this proportion exceeds the upper limit of the intervention rate tolerance interval, it indicates that the current execution threshold is too low, and the execution threshold is gradually increased; when the proportion is lower than the lower limit of the tolerance interval, it indicates that the execution threshold may be too strict, and the execution threshold is gradually decreased. Each correction amount is multiplied by a preset damping coefficient to ensure smooth step size. The upper and lower limits of the tolerance interval, the step size, and the damping coefficient are all preset based on system calibration test data and continuously optimized and adjusted in subsequent runs based on the actual intervention rate statistics.

[0060] The revised execution threshold is stored in the vehicle's storage unit for subsequent trips.

[0061] If a temporarily suspended instruction sent to the cache queue is resubmitted to step S103 to calculate a new interaction execution factor and make a new decision when the risk coefficient of the current scenario is reduced to the level when the instruction was stored in the cache, the instruction is discarded. If the cached state exceeds 30 seconds or the re-evaluation condition is not met after the vehicle state stabilizes, the instruction is discarded. The preset re-entry judgment criterion is based on the pre-calibrated range of the system risk coefficient and is used to characterize that the scenario risk has substantially fallen to the level at which the interaction instruction can be re-evaluated.

[0062] This step determines the execution strategy for each instruction based on the interactive execution factor and establishes a data-driven deterministic feedback mechanism, enabling the system to continuously adapt to the preferences of different users and the characteristics of different driving scenarios.

[0063] This concludes the example.

[0064] based on Figure 1The in-vehicle internet dynamic interaction adaptation method shown uses a progressively layered quantification logic to enable the scenario risk coefficient to simultaneously assume the dual functions of safety gating and spatial cost weighting, thereby achieving dynamic coupling of environmental risk and operational cost. Based on the physical quantification results of the first two layers, it makes limited fine-tuning based on historical corrections to avoid subjective factors overriding objective safety constraints. Under the premise of ensuring the absolute priority of hard safety constraints, it achieves a unified quantification evaluation of the executability of concurrent interactions among multiple passengers.

[0065] When applying the in-vehicle internet dynamic interaction adaptation method based on multimodal fusion provided in this manual, it is not necessary to consider... Figure 1 The steps shown are executed in sequence. The specific execution order of each step can be determined as needed, and this manual does not impose any restrictions on it.

[0066] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for dynamic interaction adaptation of in-vehicle internet based on multimodal fusion, characterized in that, include: Acquire multimodal sensor data and parse out the occupant identity and seat position association matrix, as well as the multi-dimensional vehicle dynamic state code of the current driving scenario; Data from multiple dimensions are extracted from the vehicle dynamic state code to calculate the risk sub-score for each dimension. The maximum and second-largest values ​​of the risk sub-scores are taken and combined with the relative difference between the maximum and second-largest values ​​to obtain the scenario risk coefficient. Obtain the sequence of scenario risk coefficients within a preset time window prior to the current moment, and dynamically generate a safety boundary threshold based on the statistical characteristic values ​​of the scenario risk coefficient sequence; When the scenario risk coefficient is greater than the preset safety boundary threshold, the interaction execution factor is set to the preset high-level clamping value and the interaction instruction is determined to be unsuitable for execution. When the scenario risk coefficient is less than or equal to the safety boundary threshold, the seat reference point of the interactive command issuer is determined based on the identity and seat position association matrix, and the spatial distance to the center point of the target interactive interface is obtained. The spatial cost factor is calculated based on the scenario risk coefficient and the spatial distance. Then, the historical interaction records are retrieved using the identity identifier of the issuer of the current interactive command and the interaction content type code to obtain the historical correction weight. The historical correction weight represents the user's historical intervention in the interaction content type. The spatial cost factor is corrected using the historical correction weight to generate an interaction execution factor that represents the executability of the current interactive command. Whether to execute the current interactive instruction is determined based on the comparison result between the interactive execution factor and the preset execution threshold.

2. The in-vehicle internet dynamic interaction adaptation method based on multimodal fusion as described in claim 1, characterized in that, The computational space cost factor includes: Using the scenario risk coefficient as the basic weight, the degree of closeness between the spatial distance and the preset ergonomic limit distance as the adjustment coefficient, and the ratio of the difference between the maximum and second largest values ​​in each dimension of risk sub-scores to the gap coefficient as the composite risk adjustment term, the spatial cost factor is obtained by multiplying the basic weight, the adjustment coefficient and the composite risk adjustment term.

3. The in-vehicle internet dynamic interaction adaptation method based on multimodal fusion as described in claim 1, characterized in that, The method for calculating the scenario risk coefficient includes: The standard deviation of each risk sub-score is used as the gap coefficient, the maximum value in the risk sub-score is used as the main risk component, and the second largest value in the risk sub-score is used as the correction increment. The contribution of the correction increment to the scenario risk coefficient is inversely correlated with the relative gap obtained by dividing the difference between the maximum value and the second largest value by the gap coefficient. When the fusion result exceeds the preset upper limit, the scenario risk coefficient is clamped to the preset upper limit.

4. The in-vehicle internet dynamic interaction adaptation method based on multimodal fusion as described in claim 1, characterized in that, The calculation of risk sub-scores for each dimension includes: The speed risk sub-score is calculated based on the standardized vehicle speed value using a monotonically increasing power function mapping relationship; the handling complexity sub-score is calculated based on the variance of the steering wheel angle change and the frequency of the brake master cylinder pressure change within a set time window; the road type sub-score is calculated based on the ratio of the legal maximum speed limit corresponding to the road type code to the vehicle's own maximum speed limit; and the traffic density sub-score is calculated based on the standardized value of the distance to the vehicle ahead, with a higher traffic density sub-score for a smaller standardized value of the distance to the vehicle ahead.

5. The in-vehicle internet dynamic interaction adaptation method based on multimodal fusion as described in claim 4, characterized in that, The sub-score for handling complexity, calculated based on the variance of steering wheel angle changes and the frequency of brake master cylinder pressure changes within a set time window, includes: After dividing the steering wheel angle variance and braking frequency by their respective upper limits to make them dimensionless, they are fused using a geometric mean relationship. The fused values ​​are then linearly mapped to a standardized interval as a sub-score for handling complexity.

6. The in-vehicle internet dynamic interaction adaptation method based on multimodal fusion as described in claim 1, characterized in that, The process of obtaining historical correction weights includes: A joint index is constructed using the sender's identity identifier and the interaction content type code. The historical interaction record database is searched, and records that completely match the identity identifier, have the same interaction content type, and whose difference between the scenario risk coefficient at the time of interaction and the current scenario risk coefficient is within a preset tolerance range are selected as valid samples from the search results. If the effective sample size is less than the preset minimum sample size threshold, a neutral value is dynamically calculated based on the proportion of interaction records with intervention in all of the user's historical interaction records, and this neutral value is used as the historical correction weight. If the effective sample size is sufficient, a weighted average is performed based on the chronological order of each effective sample and the corresponding user intervention status coefficient to obtain the historical correction weight.

7. The in-vehicle internet dynamic interaction adaptation method based on multimodal fusion as described in claim 6, characterized in that, The weighted average based on the chronological order of each valid sample and the corresponding user intervention state coefficient specifically includes: The weight coefficients of each valid sample are determined according to the exponential decay law based on the chronological order of the valid samples. The valid samples that are closer to the current time have larger weight coefficients. The intervention state coefficients of each valid sample are weighted and averaged according to the weight coefficients. The value of the intervention state coefficient when the user does not actively correct or cancel the interaction result is greater than the value when the user actively corrects or cancels the interaction result. The weighted average result is used as the historical correction weight.

8. The in-vehicle internet dynamic interaction adaptation method based on multimodal fusion as described in claim 6, characterized in that, The generation of the interaction execution factor, which characterizes the executability of the current interaction instruction, includes: Using the spatial cost factor as the base value, the degree of positive deviation of the historical correction weight relative to the neutral value as the deviation amount, and the product of the scenario risk coefficient and the deviation amount as the reduction index, the interaction execution factor is obtained by multiplying the base value by the exponential decay factor based on the reduction index. The greater the degree of positive deviation of the historical correction weight relative to the neutral value, the greater the reduction of the interaction execution factor.

9. The in-vehicle internet dynamic interaction adaptation method based on multimodal fusion as described in claim 1, characterized in that, The step of determining whether to execute the current interaction instruction based on the comparison result between the interaction execution factor and the preset execution threshold includes: When the interaction execution factor is less than the execution threshold, the instruction is determined to be allowed to be executed. When the interaction execution factor is greater than or equal to the execution threshold, the instruction is determined not to be executed for the time being. When multiple interaction instructions are pending processing at the same time, the instructions with interaction execution factors less than the execution threshold are included in the execution set. The multiple instructions in the execution set are arranged in ascending order according to the interaction execution factor from smallest to largest, and the one with the smallest interaction execution factor is executed first.

10. The in-vehicle internet dynamic interaction adaptation method based on multimodal fusion as described in claim 1, characterized in that, Also includes: After each trip, the user intervention rate of all executed instructions in that trip is calculated. The user intervention rate is the proportion of instructions that are actively corrected or canceled by the user within a preset intervention window after execution. When the user intervention rate exceeds the upper limit of the preset intervention rate tolerance range, the preset execution threshold is gradually increased. When the user intervention rate is lower than the lower limit of the intervention rate tolerance range, the preset execution threshold is gradually decreased. The corrected execution threshold is stored in the vehicle storage unit for subsequent trips.

Images