Multi-view fused virtual reality interaction method and system

Abstract

The application discloses a multi-view fusion virtual reality interaction method and system, relates to the technical field of virtual reality, and comprises the following steps: connecting a wearable interaction device; in the process of driving a vehicle, action capture is performed to obtain user action information; a tactile detection module is used to collect tactile detection information; historical interaction voice information is extracted; correlation feature mining is performed to obtain correlation feature indexes; a virtual interaction module is built based on the correlation feature indexes, the historical interaction voice information and a virtual test environment; control instructions are set based on the user action information and the tactile detection information, the control instructions are sent to the virtual interaction module, dynamic adjustment and optimization are performed in a virtual scene, and virtual optimization results are output. The application solves the technical problems of poor driving experience and low safety in the prior art, and achieves the technical effects of improving driving comfort and safety.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and more specifically to a virtual reality interaction method and system for multi-view fusion. Background Technology

[0002] With the development of technology and people's increasing demands for driving experience, traditional driving methods can no longer meet people's needs for a more intelligent, safe, and comfortable driving experience. Traditional driving methods require users to focus on the road and vehicle operation, making it impossible to handle multiple tasks simultaneously, such as checking navigation or answering phone calls, resulting in a poor user experience. Furthermore, in emergency situations, traditional driving methods cannot respond to these situations in a timely and accurate manner, posing driving safety issues. Existing technologies suffer from poor driving experience and low safety. Summary of the Invention

[0003] This application provides a multi-view fusion virtual reality interaction method and system, which effectively solves the technical problems of poor driving experience and low safety in the prior art, and achieves the technical effect of improving driving comfort and safety.

[0004] This application provides a multi-view fusion virtual reality interaction method and system, the technical solution of which is as follows:

[0005] In a first aspect, embodiments of this application provide a multi-view fusion virtual reality interaction method, the method comprising:

[0006] Connect to wearable interactive devices, the wearable interactive devices including head-mounted display interactive devices and wrist interactive devices, the wrist interactive devices including wrist interactive devices and ankle interactive devices;

[0007] Based on the wearable interactive device, motion capture is performed during vehicle operation to obtain user motion information, including facial expressions, body movements, and hand movements.

[0008] Tactile detection modules are installed on the car steering wheel, brake pedal, and accelerator pedal. The tactile detection modules are used to collect tactile detection information, including pressure detection information and vibration detection information.

[0009] Extract historical interactive voice information, which is stored in a virtual reality interactive database and includes a timestamp.

[0010] In the virtual reality interaction database, the historical interactive voice information is used to perform correlation feature mining to obtain correlation feature indicators, including in-vehicle temperature and in-vehicle humidity indicators.

[0011] Based on the associated feature indicators, the historical interactive voice information, and the virtual test environment, a virtual interaction module is built, which includes an emergency control unit.

[0012] Based on the user action information and the tactile detection information, control commands are set and sent to the virtual interaction module. The module dynamically adjusts and optimizes the virtual scene and outputs virtual optimization results, which include speed optimization control information and direction optimization control information.

[0013] Secondly, embodiments of this application provide a multi-view fusion virtual reality interaction system, the system comprising:

[0014] A wearable interactive device connection module is used to connect a wearable interactive device, the wearable interactive device including a head-mounted display interactive device and a wrist interactive device, the wrist interactive device including a wrist interactive device and an ankle interactive device;

[0015] The user action information acquisition module is used to capture user action information during vehicle operation based on the wearable interactive device. The user action information includes facial expressions, body movements, and hand movements.

[0016] A tactile detection information acquisition module is used to configure tactile detection modules on car steering wheels, brake pedals, and accelerator pedals, and to acquire tactile detection information, including pressure detection information and vibration detection information.

[0017] A historical interactive voice information extraction module is used to extract historical interactive voice information, which is stored in a virtual reality interactive database and includes a timestamp.

[0018] The associated feature index acquisition module is used to perform associated feature mining on the historical interactive voice information in the virtual reality interaction database to obtain associated feature indexes, including in-vehicle temperature index and in-vehicle humidity index.

[0019] A virtual interaction module building module is used to build a virtual interaction module based on the associated feature indicators, the historical interactive voice information, and the virtual test environment. The virtual interaction module includes an emergency control unit.

[0020] A virtual optimization result output module is used to set control commands based on the user action information and the tactile detection information, send the control commands to the virtual interaction module, perform dynamic adjustment and optimization in the virtual scene, and output virtual optimization results, including speed optimization control information and direction optimization control information.

[0021] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0022] This application first connects to wearable interactive devices, including a head-mounted display interactive device and a wrist interactive device, the wrist interactive device including a wrist interactive device and an ankle interactive device. Then, based on the wearable interactive device, motion capture is performed during vehicle operation to obtain user motion information, including facial expressions, body movements, and hand movements. Tactile detection modules are configured on the car steering wheel, brake pedal, and accelerator pedal. The tactile detection modules are used to collect tactile detection information, including pressure detection information and vibration detection information. Then, historical interactive voice information is extracted and stored in a virtual reality interaction database. This historical interactive voice information includes timestamps. Then, in the virtual reality interaction database, correlation feature mining is performed on the historical interactive voice information to obtain correlation feature indicators, including in-vehicle temperature and humidity indicators. Based on these correlation feature indicators, the historical interactive voice information, and the virtual testing environment, a virtual interaction module is built. This virtual interaction module includes an emergency control unit. Finally, based on the user's action information and the tactile detection information, control commands are set and sent to the virtual interaction module. Dynamic adjustments and optimizations are performed in the virtual scene, and virtual optimization results are output, including speed optimization control information and direction optimization control information. This effectively solves the technical problems of poor driving experience and low safety in existing technologies, achieving the technical effect of improving driving comfort and safety. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram of a multi-view fusion virtual reality interaction method provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the structure of a multi-view fusion virtual reality interaction system provided in an embodiment of this application.

[0026] Figure labeling: 1. Wearable interactive device connection module; 2. User action information acquisition module; 3. Tactile detection information acquisition module; 4. Historical interactive voice information extraction module; 5. Related feature index acquisition module; 6. Virtual interaction module construction module; 7. Virtual optimization result output module. Detailed Implementation

[0027] This application provides a multi-view fusion virtual reality interaction method and system to address the technical problems of poor driving experience and low safety in existing technologies.

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0029] It should be noted that the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product, or device.

[0030] Example 1

[0031] like Figure 1 As shown, this invention provides a multi-view fusion virtual reality interaction method to improve driving comfort and safety. The method includes:

[0032] Connecting wearable interactive devices via Bluetooth or USB to computers or other smart terminals enables human-computer interaction and information exchange. These wearable interactive devices include head-mounted displays and wrist-based interactive devices. The head-mounted displays (e.g., smart glasses or helmets) include components such as a display, camera, and microphone to capture the user's visual and auditory information and display virtual scenes. The wrist-based interactive devices include wrist-based devices (e.g., smartwatches, smart bracelets) and ankle-based devices (e.g., smart ankle bracelets, smart insoles). The wrist-based devices capture and sense the user's hand and wrist movements and postures, including wrist flexion and extension, finger opening and closing, and hand position and posture on the steering wheel. The ankle-based devices capture the user's foot movements and postures, including the force, angle, and duration of pressing the accelerator, brake, and clutch pedals, as well as foot position and posture on the pedals.

[0033] Based on the wearable interactive device, motion capture is performed during vehicle operation to acquire user action information. This information is collected and transmitted through sensors and circuits, including facial expressions, body movements, and hand movements. The user's facial expressions, body movements, and hand movements are used to reflect the user's emotions, fatigue level, and other conditions. For example, capturing facial expressions determines the user's emotional state, such as whether they are happy, nervous, or tired; capturing body movements determines whether the user is tired or distracted; and capturing hand movements determines whether the user is controlling the vehicle's direction.

[0034] Tactile detection modules are installed on the car's steering wheel, brake pedal, and accelerator pedal. These modules include pressure sensors and vibration sensors to collect tactile detection information, which includes pressure and vibration data. Specifically, a pressure sensor on the steering wheel detects the pressure distribution and force applied by the user's hands, thereby determining the user's operating intentions and driving state. Vibration sensors on the brake and accelerator pedals detect the force and frequency of the user's pedal presses, thereby determining the user's driving state and operating habits.

[0035] Historical interactive voice information is extracted. This refers to voice information generated during past interactions, including user voice commands to the system, system responses, and other interaction-related information. Historical interactive voice information is stored as data records in a virtual reality interaction database. Each record contains interactive voice information and a corresponding timestamp. The timestamp is an identifier of the time point in the recorded interactive voice information, used to help determine the order and chronological sequence of the interactive voice information. Records in the virtual reality interaction database are sorted according to the timestamps to find interactive voice information related to a specific historical time. By retrieving historical interactive voice information containing the required timestamps, voice interaction records related to a specific time period or event are obtained.

[0036] In the virtual reality interactive database, association feature mining is performed on the historical interactive voice information to obtain association feature indicators. Association feature mining is a data mining technique used to discover association rules in a dataset. Association rules refer to the relationships between different items in a dataset, reflecting potential patterns and correlations in the data. The obtained association feature indicators include in-vehicle temperature and humidity indicators. First, the extracted voice data is preprocessed, including speech recognition, transcription, and word segmentation. Features related to in-vehicle temperature and humidity are extracted from the preprocessed voice data, including specific words, phrases, and commands, such as "too hot," "cool down," and "adjust humidity to 50%." Then, association rule mining algorithms, such as Apriori and FP-Growth, are used to analyze these features and find the association rules between them. For example, the frequently occurring phrase "too hot" suggests that the user is dissatisfied with the in-vehicle temperature and needs to adjust it. Finally, based on the association rule mining results, corresponding association feature indicators are generated, such as counting the frequency of the phrase "too hot" or the number of times the command "adjust humidity to 50%" is executed. The generated association feature indicators are stored in the virtual reality interactive database.

[0037] Based on the aforementioned correlation feature indicators, historical interactive voice information, and a virtual testing environment, a virtual interaction module is constructed. This virtual interaction module is an interaction model within a virtual reality environment, used to simulate the interaction process between the user and the vehicle. This model achieves intelligent interaction with the user by capturing and analyzing the user's voice commands and other interactive information. The virtual interaction module includes an emergency control unit, which is responsible for handling interactive operations and control systems in emergency situations. When an emergency occurs in the vehicle or the virtual reality environment, the emergency control unit can respond quickly and take corresponding control measures, such as automatic braking and emergency steering, to avoid accidents or mitigate damage. The virtual interaction model and the emergency control unit are integrated into a virtual testing environment that simulates the actual operating environment of the vehicle and the user's interaction process in virtual reality. This environment is used to test and verify the functionality and performance of the virtual interaction module. Testing and optimization are performed in the virtual testing environment to ensure that the virtual interaction module can accurately capture and analyze the user's voice commands and other interactive information, and can perform intelligent control according to the user's needs and instructions. Simultaneously, it is ensured that the emergency control unit can respond quickly and take correct control measures in emergency situations.

[0038] Based on the user's action information and tactile detection information, the system acquires the user's intentions and needs, such as walking speed and turning direction. Based on this information, control commands are set, such as adjusting the vehicle's speed or steering angle. These control commands are sent to the virtual interaction module. The virtual interaction module dynamically adjusts and optimizes the virtual scene according to the commands and the current vehicle state. For example, if the user suddenly accelerates in the virtual scene, the virtual interaction module automatically adjusts the vehicle's speed and direction to match the user's actions and needs. After dynamic adjustment and optimization in the virtual scene, the system outputs virtual optimization results. These results include speed optimization control information and direction optimization control information. Speed ​​optimization control information refers to control commands that optimize and adjust the vehicle's speed based on the user's actions and needs in the virtual scene, including acceleration, deceleration, and constant speed, to match the user's actions and needs. Direction optimization control information refers to control commands that optimize and adjust the vehicle's direction based on the user's turning needs in the virtual scene, including left turn, right turn, and adjusting the steering wheel angle, to help the user better control the vehicle in the virtual scene. This achieves the technical effect of improving driving comfort and safety.

[0039] In a preferred embodiment provided in this application, in the virtual reality interaction database, correlation feature mining is performed on the historical interactive voice information to obtain correlation feature indicators. The method includes:

[0040] Historical interactive voice information is preprocessed, including speech recognition, transcription, and word segmentation, to extract keywords and semantic information, resulting in preprocessed text information. Based on the keywords in the preprocessed text information, keyword localization is performed to obtain keyword-annotated text information. Keywords include important words and phrases related to vehicle control and the environment; for example, if a user says "It's too hot," then "hot" is considered a keyword. After obtaining the keyword-annotated text information, association feature mining is performed on this information in the virtual reality interaction database to obtain association feature indicators, including word frequency, keyword co-occurrence frequency, and correlation between keywords. For example, if the keyword "hot" appears frequently, it indicates that the user is concerned about temperature. Through association feature mining, the interaction patterns and habits of users in the virtual reality environment are identified, such as which commands and words users tend to use while driving. Potential rules and patterns related to vehicle control and navigation are also sought. For example, most users will lower the temperature after saying "It's too hot" in the virtual reality environment; therefore, the vehicle's temperature control strategy is optimized based on this rule. This preferred implementation reduces the interference of noise and redundant information on the system by using keyword positioning for associated feature mining, thereby achieving the technical effect of improving the accuracy and efficiency of associated feature index extraction.

[0041] In another preferred embodiment provided in this application, in the virtual reality interactive database, association feature mining is performed based on the keyword-annotated text information. The method includes:

[0042] In the virtual reality interactive database, targeted correlation feature mining is performed based on the keyword-annotated text information. Specifically, the keyword-annotated text information is correlated with historical vehicle environment information and historical user action information in the virtual reality interactive database to determine a feature correlation mapping set. First, the acquired historical vehicle environment information and historical user action information are preprocessed to transform the raw data into an analyzable form, such as converting user behavior into understandable instructions. Then, features related to the keyword-annotated text information are extracted from the preprocessed data, including vehicle environment features (e.g., vehicle speed, temperature, lighting), user action features (e.g., gestures, body posture, voice), and keyword-annotated text information features (e.g., vocabulary, phrases, semantics). Subsequently, the extracted vehicle environment features, user action features, and keyword-annotated text information features are correlated by constructing a feature correlation mapping set to establish two sets: a vehicle environment-keyword-annotated text information correlation mapping subset and a user action-keyword-annotated text information correlation mapping subset.

[0043] Feature indexing is performed through the aforementioned feature association mapping set. The first association feature index is determined by constructing feature vectors, calculating association degrees, and optimizing association feature indices. This includes quantitative indicators such as frequency, probability, and relevance, or qualitative indicators such as classification labels. These reflect user behavior patterns and preferences in the virtual reality environment, such as user behavior patterns using specific words or phrases in a specific environment. Finally, the first association feature index is added to the association feature indices for subsequent data analysis and decision-making. This preferred implementation increases the dimension of features by constructing two sets: a vehicle environment-keyword-annotated text information association mapping subset and a user action-keyword-annotated text information association mapping subset. This achieves the technical effect of enriching feature dimensions and improving the accuracy of mining association features.

[0044] In another preferred embodiment provided in this application, in the virtual reality interactive database, targeted association feature mining is performed based on the keyword-annotated text information. The method further includes:

[0045] In the virtual reality interactive database, non-directional association feature mining is performed based on the keyword-annotated text information. That is, index fusion analysis (integrating all information) is performed on two subsets in the feature association mapping set through methods such as weighted fusion and decision-level fusion to obtain the index fusion analysis results. For example, if certain specific vehicle environment features have a high similarity to the keyword-annotated text information, then the degree of association between these vehicle environment features and the keyword-annotated text information is high, and a high weight is assigned to this feature association relationship.

[0046] The results of the fusion analysis of the aforementioned indicators are used as constraint information for feature association analysis in the virtual reality interaction database. This involves a deeper feature association analysis of the data in the virtual reality interaction database to determine a feature fusion association mapping set. For example, assuming that certain specific vehicle environment features are found to have a high correlation with keyword-annotated text information in the indicator fusion analysis results, this serves as constraint information. This guides the subsequent feature association analysis to find more user action features related to these vehicle environment features, thereby discovering more complex association patterns. Finally, feature indicators are generated using the feature fusion association mapping set to determine a second association feature indicator (reflecting the user's complex behavioral patterns and deeper preferences in the virtual reality environment). The second association feature indicator is then incorporated into the association feature indicator. This step is the same as the method for determining the first association feature indicator described above, and will not be repeated here. This preferred embodiment, through feature fusion analysis, comprehensively analyzes the user's behavioral patterns and needs in the virtual reality environment from multiple angles and levels. It can obtain richer and more comprehensive association feature information than analyzing only one aspect (such as the vehicle environment or user actions), achieving the technical effect of obtaining more comprehensive and deeper association feature information.

[0047] In another preferred embodiment provided in this application, a virtual interaction module is built based on the associated feature indicators, the historical interactive voice information, and the virtual testing environment. The method includes:

[0048] Based on timestamps, the associated feature indicators and the historical interactive voice information data are aligned or synchronized in the time dimension. After data synchronization, the associated feature indicators and historical interactive voice information are integrated together to form a comprehensive dataset containing multiple types of information, namely virtual test data.

[0049] The virtual test data is preprocessed, including data cleaning, standardization, and normalization, to remove noise, handle missing values ​​and outliers, and adjust the data to the same scale to facilitate subsequent model training. The preprocessed virtual test data is obtained after preprocessing.

[0050] Using the preprocessed virtual test data as training data, a model algorithm (e.g., a Transformer-based model) is selected and parameters (including learning rate, batch size, hidden layer size, encoder / decoder size, etc.) are configured and trained to build the virtual interaction module. During this process, the model gradually adapts to and learns the mapping relationship between input and output through repeated iterations and learning. Ultimately, it can generate or predict the corresponding output based on the associated feature indicators of the input and historical interactive voice information. This preferred implementation obtains a series of time-series virtual test data using timestamps, ensuring the data is synchronous, continuous, and complete, achieving the technical effect of constructing an efficient and reliable virtual interaction module.

[0051] In another preferred embodiment provided in this application, the virtual interaction module includes an emergency control unit, and the method further includes:

[0052] The emergency control unit is equipped with a preset emergency handling program. This program, stored within the emergency control unit, is used to identify and handle preset emergency situations. For example, this program will be activated when the vehicle experiences a serious malfunction or a dangerous situation is detected. When the preset emergency handling program is activated, it automatically takes over vehicle control, that is, it takes over the vehicle's accelerator, brake, steering wheel, and other control functions from the regular driving system to prevent the vehicle from entering a dangerous state.

[0053] If the automatic takeover of vehicle control authority is activated, a simulation test is conducted in the virtual test environment. That is, in the virtual test environment, various emergency situations are simulated, such as vehicle malfunctions and road hazards. Then, the vehicle's reaction and performance, user feedback information, etc. are recorded to obtain the simulation test results.

[0054] Based on the simulation test results and user feedback, the shortcomings of the virtual interaction module were identified and optimized. After multiple iterations, an updated virtual interaction module (a superior virtual interaction module) was obtained. The updated virtual interaction module provides better performance in handling emergencies. By optimizing algorithms and improving data processing methods, it can respond to emergencies more quickly and take appropriate measures to avoid or reduce potential dangers. The updated virtual interaction module has higher accuracy. By improving the precision of data acquisition, processing, and analysis, it can more accurately identify user intentions and needs, thereby providing more personalized services. For example, during braking, if the vehicle's deceleration accuracy is only accurate to the tenths (0.1), then in some cases, if the user wants the vehicle to brake faster, but the vehicle cannot accurately meet the user's expectations, if the system determines that an emergency has occurred (insufficient braking performance or dangerous road conditions, etc.), it may jump to the emergency control unit. However, if the updated virtual interaction module can improve the deceleration accuracy to the thousandths (0.001), then it can more accurately meet the user's expectations, thereby avoiding unnecessary jumps and providing a smoother and safer driving experience. This preferred embodiment enables the emergency control unit to continuously optimize and improve the virtual interaction module while handling emergency situations, thereby achieving the technical effects of improving module performance and enhancing robustness.

[0055] Example 2

[0056] Based on the same inventive concept as the multi-view fusion virtual reality interaction method in the foregoing embodiments, such as Figure 2 As shown, this application provides a multi-view fusion virtual reality interaction system. The system and method embodiments in this application are based on the same inventive concept. The system includes:

[0057] Wearable interactive device connection module 1, the wearable interactive device connection module 1 is used to connect wearable interactive devices, the wearable interactive devices include head-mounted display interactive devices and wrist interactive devices, the wrist interactive devices include wrist interactive devices and ankle interactive devices;

[0058] User action information acquisition module 2 is used to capture user action information during vehicle operation based on the wearable interactive device. The user action information includes facial expressions, body movements, and hand movements.

[0059] The tactile detection information acquisition module 3 is used to configure tactile detection modules on the car steering wheel, brake pedal, and accelerator pedal. The tactile detection modules are used to collect tactile detection information, which includes pressure detection information and vibration detection information.

[0060] Historical interactive voice information extraction module 4 is used to extract historical interactive voice information, which is stored in a virtual reality interactive database and includes a timestamp.

[0061] The associated feature index acquisition module 5 is used to perform associated feature mining on the historical interactive voice information in the virtual reality interaction database to obtain associated feature indexes, including in-vehicle temperature index and in-vehicle humidity index.

[0062] Virtual interaction module building module 6 is used to build a virtual interaction module based on the associated feature indicators, the historical interactive voice information and the virtual test environment. The virtual interaction module includes an emergency control unit.

[0063] The virtual optimization result output module 7 is used to set control commands based on the user action information and the tactile detection information, send the control commands to the virtual interaction module, perform dynamic adjustment and optimization in the virtual scene, and output virtual optimization results. The virtual optimization results include speed optimization control information and direction optimization control information.

[0064] Furthermore, the associated feature index acquisition module 5 is used to perform the following method:

[0065] Historical interactive voice information is preprocessed to obtain preprocessed text information;

[0066] Based on the keywords in the preprocessed text information, keyword positioning is performed to obtain keyword-annotated text information;

[0067] In the virtual reality interactive database, association feature mining is performed based on the keyword-annotated text information to obtain association feature indicators.

[0068] Furthermore, the associated feature index acquisition module 5 is used to perform the following method:

[0069] The keyword-annotated text information is subjected to feature association analysis with the historical vehicle environment information and historical user action information in the virtual reality interaction database to determine the feature association mapping set. The feature association mapping set includes a vehicle environment-keyword-annotated text information association mapping subset and a user action-keyword-annotated text information association mapping subset.

[0070] Feature indexing is performed using the feature association mapping set to determine a first associated feature index, and the first associated feature index is then added to the associated feature index.

[0071] Furthermore, the associated feature index acquisition module 5 is used to perform the following method:

[0072] Perform index fusion analysis on the feature association mapping set to obtain the index fusion analysis results;

[0073] The results of the index fusion analysis are used as constraint information to perform feature association analysis in the virtual reality interaction database to determine the feature fusion association mapping set.

[0074] Feature indexing is performed using the feature fusion association mapping set to determine the second associated feature index, and the second associated feature index is then added to the associated feature index.

[0075] Furthermore, the virtual interaction module building module 6 is used to execute the following methods:

[0076] Based on the timestamp, the associated feature indicators and the historical interactive voice information are integrated to obtain virtual test data;

[0077] The virtual test data is preprocessed to obtain virtual test preprocessed data;

[0078] The virtual interaction module is configured and trained using the virtual test preprocessing data as training data, and the virtual interaction module is built.

[0079] Furthermore, the virtual interaction module building module 6 is used to execute the following methods:

[0080] The emergency control unit is equipped with a preset emergency handling program, which is used to automatically take over vehicle control.

[0081] If the automatic takeover vehicle control authority is activated, a simulation test is performed in the virtual test environment to obtain the simulation test results;

[0082] The virtual interaction module is optimized and iterated based on the simulation test results and user feedback information to obtain an updated virtual interaction module.

[0083] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0084] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0085] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and modifications fall within the scope of this application and its equivalents, this application intends to include such modifications and modifications.

Claims

1. A multi-view fusion virtual reality interaction method, characterized in that, The method includes: Connect to wearable interactive devices, the wearable interactive devices including head-mounted display interactive devices and wrist interactive devices, the wrist interactive devices including wrist interactive devices and ankle interactive devices; Based on the wearable interactive device, motion capture is performed during vehicle operation to obtain user motion information, including facial expressions, body movements, and hand movements; and Tactile detection modules are installed on the car steering wheel, brake pedal, and accelerator pedal. The tactile detection modules are used to collect tactile detection information, including pressure detection information and vibration detection information. Extract historical interactive voice information, which is stored in a virtual reality interactive database and includes a timestamp. In the virtual reality interaction database, the historical interactive voice information is used to perform correlation feature mining to obtain correlation feature indicators, including in-vehicle temperature and in-vehicle humidity indicators. Based on the associated feature indicators, the historical interactive voice information, and the virtual test environment, a virtual interaction module is built, which includes an emergency control unit. Based on the user action information and the tactile detection information, control commands are set and sent to the virtual interaction module. The module dynamically adjusts and optimizes the virtual scene and outputs virtual optimization results, which include speed optimization control information and direction optimization control information.

2. The multi-view fusion virtual reality interaction method as described in claim 1, characterized in that, In the virtual reality interaction database, correlation feature mining is performed on the historical interactive voice information to obtain correlation feature indicators. The method includes: Historical interactive voice information is preprocessed to obtain preprocessed text information; Based on the keywords in the preprocessed text information, keyword positioning is performed to obtain keyword-annotated text information; In the virtual reality interactive database, association feature mining is performed based on the keyword-annotated text information to obtain association feature indicators.

3. The multi-view fusion virtual reality interaction method as described in claim 2, characterized in that, In the virtual reality interactive database, the method for performing association feature mining based on the keyword-annotated text information includes: The keyword-annotated text information is subjected to feature association analysis with the historical vehicle environment information and historical user action information in the virtual reality interaction database to determine the feature association mapping set. The feature association mapping set includes a vehicle environment-keyword-annotated text information association mapping subset and a user action-keyword-annotated text information association mapping subset. Feature indexing is performed using the feature association mapping set to determine a first associated feature index, and the first associated feature index is then added to the associated feature index.

4. The multi-view fusion virtual reality interaction method as described in claim 3, characterized in that, In the virtual reality interactive database, the method further includes targeted association feature mining based on the keyword-annotated text information: Perform index fusion analysis on the feature association mapping set to obtain the index fusion analysis results; The results of the index fusion analysis are used as constraint information to perform feature association analysis in the virtual reality interaction database to determine the feature fusion association mapping set. Feature indexing is performed using the feature fusion association mapping set to determine the second associated feature index, and the second associated feature index is then added to the associated feature index.

5. The multi-view fusion virtual reality interaction method as described in claim 1, characterized in that, Based on the aforementioned correlation feature indicators, the aforementioned historical interactive voice information, and the virtual testing environment, a virtual interaction module is constructed. The method includes: Based on the timestamp, the associated feature indicators and the historical interactive voice information are integrated to obtain virtual test data; The virtual test data is preprocessed to obtain virtual test preprocessed data; The virtual interaction module is configured and trained using the virtual test preprocessing data as training data, and the virtual interaction module is built.

6. The multi-view fusion virtual reality interaction method as described in claim 5, characterized in that, The virtual interaction module includes an emergency control unit, and the method further includes: The emergency control unit is equipped with a preset emergency handling program, which is used to automatically take over vehicle control. If the automatic takeover vehicle control authority is activated, a simulation test is performed in the virtual test environment to obtain the simulation test results; The virtual interaction module is optimized and iterated based on the simulation test results and user feedback information to obtain an updated virtual interaction module.

7. A multi-view fusion virtual reality interactive system, characterized in that, The system includes: A wearable interactive device connection module is used to connect a wearable interactive device, the wearable interactive device including a head-mounted display interactive device and a wrist interactive device, the wrist interactive device including a wrist interactive device and an ankle interactive device; The user action information acquisition module is used to capture user action information during vehicle operation based on the wearable interactive device. The user action information includes facial expressions, body movements, and hand movements. A tactile detection information acquisition module is used to configure tactile detection modules on car steering wheels, brake pedals, and accelerator pedals, and to acquire tactile detection information, including pressure detection information and vibration detection information. A historical interactive voice information extraction module is used to extract historical interactive voice information, which is stored in a virtual reality interactive database and includes a timestamp. The associated feature index acquisition module is used to perform associated feature mining on the historical interactive voice information in the virtual reality interaction database to obtain associated feature indexes, including in-vehicle temperature index and in-vehicle humidity index. A virtual interaction module building module is used to build a virtual interaction module based on the associated feature indicators, the historical interactive voice information, and the virtual test environment. The virtual interaction module includes an emergency control unit. A virtual optimization result output module is used to set control commands based on the user action information and the tactile detection information, send the control commands to the virtual interaction module, perform dynamic adjustment and optimization in the virtual scene, and output virtual optimization results, including speed optimization control information and direction optimization control information.

Images