Driving test simulation control interaction method and related equipment

Abstract

The invention discloses a driving test simulation control interaction method and related equipment. The method comprises the steps that under the condition that driving test simulation is carried out in a closed VR visual environment, the position and posture of the hand of a user are captured in real time, and the user is located in a simulator cabin; mapping the acquired position and posture of the hand to a virtual environment of a VR visual environment, a virtual operation part in the virtual environment being consistent with a physical part for realizing the same function in the simulator cabin in position and proportion; and obtaining the operation of a user on the physical component and / or the virtual operation component so as to perform driving test simulation control in the simulator cabin. The problem that in driving test simulator training, although a VR simulator can really provide higher simulation reality and immersion, students cannot observe the real world when wearing VR glasses can be solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically, the present invention relates to a driving test simulation control interaction method and related equipment. Background Art

[0002] In driving test simulator training, although VR simulators can indeed provide higher simulation realism and immersion, there is also a disadvantage that trainees cannot observe the real world when wearing VR glasses. This will make it difficult for trainees to find the correct position when performing operations such as turning the key to start the ignition, operating the steering wheel, adjusting the light knob, and pulling the handbrake, thus affecting the training effect. Summary of the invention

[0003] A series of simplified concepts are introduced in the Summary of the Invention, which will be further described in detail in the Detailed Description of the Invention. The Summary of the Invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0004] In order to solve the problem that in driving test simulator training, although VR simulators can indeed provide higher simulation reality and immersion, there is also a disadvantage that trainees cannot observe the real world when wearing VR glasses. In a first aspect, the present invention proposes a driving test simulation control interaction method, the method comprising:

[0005] In the case of a driving test simulation in a closed VR visual environment, real-time capture of the position and posture of a user's hands, the user being in a simulator cockpit;

[0006] Mapping the acquired hand position and posture into a virtual environment of a VR visual environment, wherein virtual operating components in the virtual environment are consistent in position and proportion with physical components in the simulator cockpit that implement the same functions;

[0007] The user's operation on the physical component and / or the virtual operating component is obtained to perform driving test simulation control in the simulator cockpit.

[0008] Specifically, it also includes:

[0009] Generating the simulator cockpit three-dimensional model according to the design information or the collected simulator cockpit image data;

[0010] Conducting actual measurements of the positions and dimensions of the physical components of the simulator cockpit;

[0011] The three-dimensional model and the measured position and size of the physical component are integrated into a unified coordinate system, and the three-dimensional model and the measured position and size of the physical component are calibrated.

[0012] Specifically, mapping the acquired hand position and posture into the virtual environment of the VR visual environment includes:

[0013] The acquired positions and postures of the arms and hands are mapped to the virtual environment of the VR visual environment, and animations of the movement of the arm models and hand models are generated and displayed in the virtual environment of the VR visual environment.

[0014] Specifically, obtaining the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit includes:

[0015] When it is recognized that the hand model moves to the vicinity of the virtual operating component, analyzing the user's current operating intention and determining the intended virtual operating component;

[0016] A matching state effect between the hand model and the intended virtual operation component is generated, wherein the matching state effect includes model adsorption or model centering.

[0017] Specifically, obtaining the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit includes:

[0018] Identify the user's hand status;

[0019] In a case where the user intends to stop operating the currently matched component based on the recognized hand state of the user, the matching state effect is removed to allow the hand model to leave the currently matched component.

[0020] Specifically, it also includes:

[0021] When the user's gesture action matches the preset operation instruction, the preset operation instruction is executed.

[0022] Specifically, when it is recognized that the hand model moves to the vicinity of the virtual operating component, analyzing the user's current operating intention and determining the intended virtual operating component includes:

[0023] When it is recognized that the hand model moves to the vicinity of the virtual operating component, the Markov model is used to analyze the user's current operating intention and determine the intended virtual operating component.

[0024] In a second aspect, the present invention further provides a driving test simulation control interactive device, comprising:

[0025] An acquisition unit, for capturing the position and posture of a user's hand in real time when a driving test simulation is performed in a closed VR visual environment, wherein the user is in a simulator cockpit;

[0026] A mapping unit, used to map the acquired hand position and posture into a virtual environment of a VR visual environment, wherein virtual operating components in the virtual environment are consistent in position and proportion with physical components in the simulator cockpit that realize the same function;

[0027] A control unit is used to obtain the user's operation on the physical components and / or the virtual operation components to perform driving test simulation control in the simulator cockpit.

[0028] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the driving test simulation control interaction method of any one of the first aspects described above when executing the computer program stored in the memory.

[0029] In a fourth aspect, the present invention further proposes a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the driving test simulation control interaction method of any one of the above items in the first aspect is implemented.

[0030] In summary, the driving test simulation control interaction method proposed in this application captures the position and posture of the user's hand in real time in a closed VR visual environment while simulating a driving test, and the user is in the simulator cockpit; maps the acquired hand position and posture to the virtual environment of the VR visual environment, wherein the virtual operating components in the virtual environment are consistent with the physical components that implement the same function in the simulator cockpit in terms of position and proportion; obtains the user's operation on the physical components to perform driving test simulation control in the simulator cockpit. Thus, by capturing the user's hand position and posture in real time and mapping them to the virtual environment, the position and posture of the virtual hand are consistent with the movement of the user's actual hand. The position and proportion of the virtual operating components are consistent with those of the physical components, and the components touched by the user in the virtual environment coincide with the positions of the actual physical components, avoiding position deviations in blind operations and significantly improving the accuracy of operations. The user's hand movements are captured using a high-precision hand tracking device, and the virtual hand accurately reproduces the user's hand movements and postures in the VR environment, providing realistic visual feedback. When the user operates the physical components, synchronous tactile feedback is provided, so that the user feels as if he is actually operating real vehicle components, further enhancing the sense of immersion. By combining visual and tactile feedback, the operation experience obtained by users in the virtual environment is closer to the real driving environment, which helps to improve the effect of driving skills training. When users practice various driving operations in the virtual environment, they can get instant feedback and guidance, helping them to master driving skills faster. The binding of virtual hands with real hands and the alignment of virtual parts with physical parts enable users to operate naturally in the VR environment, reducing the adaptation time and learning curve of VR equipment. Users can operate as in a real vehicle, without the need for additional training and adaptation process, reducing the difficulty of learning. Driving test simulation in a virtual environment can avoid dangerous operations on real roads and improve the safety of the training process. Users can repeatedly practice various complex and dangerous driving operations in a virtual environment, reducing the risks in actual driving. The system can record the user's operation data in the virtual environment, which is convenient for coaches and evaluators to evaluate and feedback the user's driving skills. By analyzing the user's operation data, it is possible to find the problems they have in the driving process and provide targeted guidance and improvement suggestions. This method can be applied to different types of driving simulators and different driving scenarios, with strong flexibility and scalability. By adjusting the positions of models and physical components in the virtual environment, it can adapt to different vehicle types and driving environments to meet diverse training needs.

[0031] The driving test simulation control interaction method of the present invention, other advantages, objectives and features of the present invention will be reflected in part through the following description, and in part will be understood by technical personnel in the field through research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present specification. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0033] Figure 1 A schematic diagram of a driving test simulation control interaction method provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the structure of a driving test simulation control interaction device provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the structure of a driving test simulation control interactive electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0037] In order to solve the problem that in driving test simulator training, although VR simulators can indeed provide higher simulation realism and immersion, there is also a disadvantage that students cannot observe the real world when wearing VR glasses. Figure 1 , is a flow chart of a driving test simulation control interaction method provided in an embodiment of the present application, which may specifically include: steps S110 to S130.

[0038] S110, in the case of a driving test simulation in a closed VR visual environment, capturing the position and posture of the user's hands in real time, wherein the user is in a simulator cockpit.

[0039] For example, when conducting a driving test simulation in a closed VR visual environment, a hand tracking device is used to capture the position and posture of the user's hand in real time. The user is in a simulator cockpit that simulates a real driving environment.

[0040] S120, mapping the acquired hand position and posture to a virtual environment of a VR visual environment, wherein virtual operating components in the virtual environment are consistent in position and proportion with physical components in the simulator cockpit that implement the same functions.

[0041] Exemplarily, the captured hand position and posture data are mapped to a virtual hand in a VR visual environment. A virtual hand is created in the virtual environment that is synchronized with the user's hand movements, ensuring that the position and posture of the virtual hand are consistent with the movement trajectory of the user's actual hand. Ensure that the virtual operating components (such as steering wheels, keys, buttons, etc.) in the virtual environment are consistent with the physical components in the simulator cockpit in position and proportion. Adjust the model in the virtual environment so that when the user touches a certain component in the virtual environment, the real hand can also accurately touch the corresponding physical component.

[0042] S130, obtaining the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit.

[0043] Exemplarily, the user's actual operation of the physical components is monitored and recorded, such as turning the steering wheel, pressing a button, pulling the handbrake, etc. These operations are fed back to the virtual environment to ensure that the operations in the virtual environment are consistent with the user's actual operations. Provide instant visual and tactile feedback. Ensure that the user can obtain accurate tactile and visual feedback when operating the virtual hand to interact with the physical components. Exemplarily, when the user touches a certain component in the virtual environment, corresponding tactile feedback (such as vibration) can be given to enhance the sense of immersion.

[0044] In summary, the driving test simulation control interaction method provided by the embodiment of the present application captures the position and posture of the user's hand in real time in the case of a driving test simulation in a closed VR visual environment, and the user is in the simulator cockpit; the acquired hand position and posture are mapped to the virtual environment of the VR visual environment, wherein the virtual operating components in the virtual environment are consistent with the physical components that implement the same function in the simulator cockpit in position and proportion; the user's operation on the physical components is obtained to perform driving test simulation control in the simulator cockpit. Thus, by capturing the user's hand position and posture in real time and mapping them to the virtual environment, the position and posture of the virtual hand are consistent with the movement of the user's actual hand. The position and proportion of the virtual operating components are consistent with those of the physical components, and the components touched by the user in the virtual environment coincide with the actual physical component positions, avoiding position deviations in blind operations and significantly improving the accuracy of operations. The user's hand movements are captured using a high-precision hand tracking device, and the virtual hand accurately reproduces the user's hand movements and postures in the VR environment, providing realistic visual feedback. When the user operates the physical components, synchronous tactile feedback is provided, so that the user feels as if he is actually operating real vehicle components, further enhancing the sense of immersion. By combining visual and tactile feedback, the operation experience obtained by users in the virtual environment is closer to the real driving environment, which helps to improve the effect of driving skills training. When users practice various driving operations in the virtual environment, they can get instant feedback and guidance, helping them to master driving skills faster. The binding of virtual hands with real hands and the alignment of virtual parts with physical parts enable users to operate naturally in the VR environment, reducing the adaptation time and learning curve of VR equipment. Users can operate as in a real vehicle, without the need for additional training and adaptation process, reducing the difficulty of learning. Driving test simulation in a virtual environment can avoid dangerous operations on real roads and improve the safety of the training process. Users can repeatedly practice various complex and dangerous driving operations in a virtual environment, reducing the risks in actual driving. The system can record the user's operation data in the virtual environment, which is convenient for coaches and evaluators to evaluate and feedback the user's driving skills. By analyzing the user's operation data, it is possible to find the problems they have in the driving process and provide targeted guidance and improvement suggestions. This method can be applied to different types of driving simulators and different driving scenarios, with strong flexibility and scalability. By adjusting the positions of models and physical components in the virtual environment, it can adapt to different vehicle types and driving environments to meet diverse training needs.

[0045] According to some embodiments, further comprising:

[0046] Generating the simulator cockpit three-dimensional model according to the design information or the collected simulator cockpit image data;

[0047] Conducting actual measurements of the positions and dimensions of the physical components of the simulator cockpit;

[0048] The three-dimensional model and the measured position and size of the physical component are integrated into a unified coordinate system, and the three-dimensional model and the measured position and size of the physical component are calibrated.

[0049] Exemplarily, a three-dimensional model of the simulator cockpit is generated based on the design information or the collected simulator cockpit image data. This method ensures that the cockpit model in the virtual environment is highly consistent with the actual physical cockpit in structure, providing a basis for subsequent alignment and calibration. The position and size of the physical components of the simulator cockpit (such as steering wheels, buttons, handbrakes, etc.) are accurately measured. Ensure that the acquired data has sufficient accuracy so that the alignment and interaction between the virtual model and the actual physical components are more accurate. Integrate the generated three-dimensional model with the position and size data of the measured physical components into a unified coordinate system. This integration enables the virtual environment and the physical environment to interact in the same reference frame, ensuring accurate alignment of position and scale. Calibrate the three-dimensional model and the measured physical components in position and size to ensure that they can accurately reflect reality in the virtual environment. Through calibration, adjust the model in the virtual environment so that when the user touches a component in the virtual environment, the real hand can also accurately touch the corresponding physical component. By generating an accurate three-dimensional model and calibrating it, the operating components in the virtual environment are highly consistent with the physical components in position and scale, improving the accuracy and consistency of the operation.

[0050] In some examples, mapping the acquired hand position and posture to a virtual environment of a VR visual environment includes:

[0051] The acquired positions and postures of the arms and hands are mapped to the virtual environment of the VR visual environment, and animations of the movement of the arm models and hand models are generated and displayed in the virtual environment of the VR visual environment.

[0052] Exemplarily, high-precision sensors are used to capture the position and posture data of the user's arms and hands in real time. Ensure the accuracy and real-time of the captured data to ensure the accuracy and synchronization of the model in the virtual environment. Map the captured position and posture data of the arms and hands to the virtual environment of the VR visual environment. Ensure that the mapping process maintains the accuracy of the data so that the virtual arm and hand models can accurately reflect the actual actions of the user. Generate the user's arm and hand models in the VR visual environment. These models need to have a high degree of detail and realism to enhance the user's immersion. Based on the acquired position and posture data, generate and display the movement animation of the arm and hand models in the virtual environment. Ensure the smoothness and naturalness of the animation so that the user's actions are truly reflected in the virtual environment. Provide visual and tactile feedback, and when the user operates in the virtual environment, the virtual arm and hand model can synchronously display the user's actual operation. When the user touches the virtual object, give appropriate tactile feedback, such as vibration or pressure, to enhance the immersion and operation experience.

[0053] In some examples, obtaining the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit includes:

[0054] When it is recognized that the hand model moves to the vicinity of the virtual operating component, analyzing the user's current operating intention and determining the intended virtual operating component;

[0055] A matching state effect between the hand model and the intended virtual operation component is generated, wherein the matching state effect includes model adsorption or model centering.

[0056] Exemplarily, when the hand model moves near the virtual operating component, the system will identify the position of the hand model in real time. The movement of the hand model is captured by a high-precision hand tracking device, and the position and posture of the hand model are updated in real time in the virtual environment. The system analyzes the user's current operation intention based on the position and movement trajectory of the hand model. The algorithm determines the virtual component that the user may want to operate and determines the user's intended virtual operating component. After the system determines the user's intended virtual operating component, it generates a matching state effect between the hand model and the virtual operating component. When the hand model is close to the intended virtual operating component, the hand model will automatically adsorb to the virtual operating component, making it easier for the user to operate accurately. The hand model will automatically center to the center position of the virtual operating component to ensure the accuracy and stability of the operation. When generating the matching state effect, the system will provide corresponding visual and tactile feedback to help the user confirm the operation. Visual feedback can be achieved by changing the color of the hand model or highlighting the intended virtual operating component. Tactile feedback can be provided by vibration or pressure sensing devices. When the matching state between the hand model and the virtual operating component reaches the preset condition, the system confirms the user's operation and executes the corresponding command. After the user completes the operation, the system records the operation data and provides further feedback or guidance.

[0057] Exemplarily, when it is recognized that the hand model moves to the vicinity of the virtual operating component, analyzing the user's current operating intention and determining the intended virtual operating component includes:

[0058] When it is recognized that the hand model moves to the vicinity of the virtual operating component, the Markov model is used to analyze the user's current operating intention and determine the intended virtual operating component.

[0059] For example, the Markov decision process can be used to optimize the decision through state, action and reward to predict the user's operation intention. Assuming that the state set of the hand position is S and the action set of the virtual operation component is A, the core formula of the MDP is:

[0060] V(s)=maXa∈AR(s,a)+γ s′∈S P(s′| s,a)V(s′)

[0061] Among them, V(s) is the value function of state s, R(s,a) is the immediate reward for executing action a in state s, γ is the discount factor, and P(s′|s,a) is the probability of transferring from state s to state s′ by executing action a. By iteratively calculating the value function V(s), selecting the action a that maximizes the value function, and predicting the virtual components that the user may want to operate.

[0062] Therefore, since the Markov model is used to make decisions through the state and action transition probability model, even if the user's operation deviates or makes mistakes at certain moments, and combined with the characteristics of the Markov model that can utilize the state transition probability of hand position and action, error correction guidance is combined with advance prediction to further provide more reliable prediction of user operation intentions.

[0063] In some examples, obtaining the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit includes:

[0064] Identify the user's hand status;

[0065] In a case where the user intends to stop operating the currently matched component based on the recognized hand state of the user, the matching state effect is removed to allow the hand model to leave the currently matched component.

[0066] In some examples, this also includes:

[0067] When the user's gesture action matches the preset operation instruction, the preset operation instruction is executed.

[0068] For example, in the VR virtual environment, human-computer interaction requires selection, confirmation, cancellation, door opening and closing, grabbing, rotation and other interactive operations without a game controller. For ease of use, it can be controlled by identifying the gestures and displacements of the palm and fingers. For example: if the index finger is extended alone, a ray is emitted to select models and buttons; the palm pushes or pulls on the door handle to open and close the door; 2 to 3 fingers slide together to turn the page; the palm can be grasped to grab, and the wrist can be turned to rotate certain models, etc.

[0069] See also Figure 2 , an embodiment of the driving test simulation control interaction device in the embodiment of the present application may include:

[0070] An acquisition unit 21 is used to capture the position and posture of the user's hand in real time when performing a driving test simulation in a closed VR visual environment, wherein the user is in a simulator cockpit;

[0071] A mapping unit 22, used for mapping the acquired hand position and posture into a virtual environment of a VR visual environment, wherein the virtual operating components in the virtual environment are consistent in position and proportion with the physical components in the simulator cockpit that realize the same functions;

[0072] The control unit 23 is used to obtain the user's operation on the physical components and / or the virtual operation components to perform driving test simulation control in the simulator cockpit.

[0073] In summary, the driving test simulation control interactive device provided in the embodiment of the present application captures the position and posture of the user's hand in real time in the case of driving test simulation in a closed VR visual environment, and the user is in the simulator cockpit; the acquired hand position and posture are mapped to the virtual environment of the VR visual environment, wherein the virtual operating components in the virtual environment are consistent with the physical components that implement the same function in the simulator cockpit in position and proportion; the user's operation on the physical components is obtained to perform driving test simulation control in the simulator cockpit. Thus, by capturing the user's hand position and posture in real time and mapping them to the virtual environment, the position and posture of the virtual hand are consistent with the movement of the user's actual hand. The position and proportion of the virtual operating components are consistent with those of the physical components, and the components touched by the user in the virtual environment coincide with the position of the actual physical components, avoiding position deviation in blind operation and significantly improving the accuracy of operation. The user's hand movements are captured using a high-precision hand tracking device, and the virtual hand accurately reproduces the user's hand movements and postures in the VR environment, providing realistic visual feedback. When the user operates the physical components, synchronous tactile feedback is provided, so that the user feels as if he is actually operating real vehicle components, further enhancing the sense of immersion. By combining visual and tactile feedback, the operation experience obtained by users in the virtual environment is closer to the real driving environment, which helps to improve the effect of driving skills training. When users practice various driving operations in the virtual environment, they can get instant feedback and guidance, helping them to master driving skills faster. The binding of virtual hands with real hands and the alignment of virtual parts with physical parts enable users to operate naturally in the VR environment, reducing the adaptation time and learning curve of VR equipment. Users can operate as in a real vehicle, without the need for additional training and adaptation process, reducing the difficulty of learning. Driving test simulation in a virtual environment can avoid dangerous operations on real roads and improve the safety of the training process. Users can repeatedly practice various complex and dangerous driving operations in a virtual environment, reducing the risks in actual driving. The system can record the user's operation data in the virtual environment, which is convenient for coaches and evaluators to evaluate and feedback the user's driving skills. By analyzing the user's operation data, it is possible to find the problems they have in the driving process and provide targeted guidance and improvement suggestions. This method can be applied to different types of driving simulators and different driving scenarios, with strong flexibility and scalability. By adjusting the positions of models and physical components in the virtual environment, it can adapt to different vehicle types and driving environments to meet diverse training needs.

[0074] like Figure 3As shown, the embodiment of the present application further provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 320 and executable on the processor. When the processor 320 executes the computer program 311, the steps of any method of the driving test simulation control interaction described above are implemented:

[0075] In the case of a driving test simulation in a closed VR visual environment, real-time capture of the position and posture of a user's hands, the user being in a simulator cockpit;

[0076] Mapping the acquired hand position and posture into a virtual environment of a VR visual environment, wherein virtual operating components in the virtual environment are consistent in position and proportion with physical components in the simulator cockpit that implement the same functions;

[0077] The user's operation on the physical component and / or the virtual operating component is obtained to perform driving test simulation control in the simulator cockpit.

[0078] Specifically, it also includes:

[0079] Generating the simulator cockpit three-dimensional model according to the design information or the collected simulator cockpit image data;

[0080] Conducting actual measurements of the positions and dimensions of the physical components of the simulator cockpit;

[0081] The three-dimensional model and the measured position and size of the physical component are integrated into a unified coordinate system, and the three-dimensional model and the measured position and size of the physical component are calibrated.

[0082] Specifically, mapping the acquired hand position and posture into the virtual environment of the VR visual environment includes:

[0083] The acquired positions and postures of the arms and hands are mapped to the virtual environment of the VR visual environment, and animations of the movement of the arm models and hand models are generated and displayed in the virtual environment of the VR visual environment.

[0084] Specifically, obtaining the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit includes:

[0085] When it is recognized that the hand model moves to the vicinity of the virtual operating component, analyzing the user's current operating intention and determining the intended virtual operating component;

[0086] A matching state effect between the hand model and the intended virtual operation component is generated, wherein the matching state effect includes model adsorption or model centering.

[0087] Specifically, obtaining the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit includes:

[0088] Identify the user's hand status;

[0089] In a case where the user intends to stop operating the currently matched component based on the recognized hand state of the user, the matching state effect is removed to allow the hand model to leave the currently matched component.

[0090] Specifically, it also includes:

[0091] When the user's gesture action matches the preset operation instruction, the preset operation instruction is executed.

[0092] Specifically, when it is recognized that the hand model moves to the vicinity of the virtual operating component, analyzing the user's current operating intention and determining the intended virtual operating component includes:

[0093] When it is recognized that the hand model moves to the vicinity of the virtual operating component, the Markov model is used to analyze the user's current operating intention and determine the intended virtual operating component.

[0094] Since the electronic device introduced in this embodiment is a device used to implement a driving test simulation control interaction device in the embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation mode of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is not introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present application is within the scope of protection of this application.

[0095] In the specific implementation process, when the computer program 311 is executed by the processor, it can achieve Figure 1 Any implementation method in the corresponding embodiment:

[0096] In the case of a driving test simulation in a closed VR visual environment, real-time capture of the position and posture of a user's hands, the user being in a simulator cockpit;

[0097] Mapping the acquired hand position and posture into a virtual environment of a VR visual environment, wherein virtual operating components in the virtual environment are consistent in position and proportion with physical components in the simulator cockpit that implement the same functions;

[0098] The user's operation on the physical component and / or the virtual operating component is obtained to perform driving test simulation control in the simulator cockpit.

[0099] Specifically, it also includes:

[0100] Generating the simulator cockpit three-dimensional model according to the design information or the collected simulator cockpit image data;

[0101] Conducting actual measurements of the positions and dimensions of the physical components of the simulator cockpit;

[0102] The three-dimensional model and the measured position and size of the physical component are integrated into a unified coordinate system, and the three-dimensional model and the measured position and size of the physical component are calibrated.

[0103] Specifically, mapping the acquired hand position and posture into the virtual environment of the VR visual environment includes:

[0104] The acquired positions and postures of the arms and hands are mapped to the virtual environment of the VR visual environment, and animations of the movement of the arm models and hand models are generated and displayed in the virtual environment of the VR visual environment.

[0105] Specifically, obtaining the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit includes:

[0106] When it is recognized that the hand model moves to the vicinity of the virtual operating component, analyzing the user's current operating intention and determining the intended virtual operating component;

[0107] A matching state effect between the hand model and the intended virtual operation component is generated, wherein the matching state effect includes model adsorption or model centering.

[0108] Specifically, obtaining the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit includes:

[0109] Identify the user's hand status;

[0110] In a case where the user intends to stop operating the currently matched component based on the recognized hand state of the user, the matching state effect is removed to allow the hand model to leave the currently matched component.

[0111] Specifically, it also includes:

[0112] When the user's gesture action matches the preset operation instruction, the preset operation instruction is executed.

[0113] Specifically, when it is recognized that the hand model moves to the vicinity of the virtual operating component, analyzing the user's current operating intention and determining the intended virtual operating component includes:

[0114] When it is recognized that the hand model moves to the vicinity of the virtual operating component, the Markov model is used to analyze the user's current operating intention and determine the intended virtual operating component.

[0115] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0117] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0118] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0120] The present application also provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device is caused to execute the following Figure 1 The process of driving test simulation control interaction in the corresponding embodiment.

[0121] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0123] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0124] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0127] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A driving test simulation control interactive method, characterized in that: include: In the case of a driving test simulation in a closed VR visual environment, real-time capture of the position and posture of a user's hands, the user being in a simulator cockpit; Mapping the acquired hand position and posture to a virtual environment of a VR visual environment, wherein the virtual operating components in the virtual environment are consistent in position and proportion with the physical components in the simulator cockpit that implement the same functions, so that when the user touches the virtual components in the virtual environment, the real hand also touches the corresponding physical components; The user's operation on the physical component and / or the virtual operating component is obtained to perform driving test simulation control in the simulator cockpit.

2. The method according to claim 1, characterized in that Also includes: Generating the simulator cockpit three-dimensional model according to the design information or the collected simulator cockpit image data; Conducting actual measurements of the positions and dimensions of the physical components of the simulator cockpit; The three-dimensional model and the measured position and size of the physical component are integrated into a unified coordinate system, and the three-dimensional model and the measured position and size of the physical component are calibrated.

3. The method according to claim 1, characterized in that Mapping the acquired hand position and posture to the virtual environment of the VR visual environment includes: The acquired positions and postures of the arms and hands are mapped to the virtual environment of the VR visual environment, and animations of the movement of the arm models and hand models are generated and displayed in the virtual environment of the VR visual environment.

4. The method according to claim 1, characterized in that The obtaining of the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit includes: When it is recognized that the hand model moves to the vicinity of the virtual operating component, analyzing the user's current operating intention and determining the intended virtual operating component; A matching state effect between the hand model and the intended virtual operation component is generated, wherein the matching state effect includes model adsorption or model centering.

5. The method according to claim 4, characterized in that The obtaining of the user's operation on the physical component and / or the virtual operation component to perform driving test simulation control in the simulator cockpit includes: Identify the user's hand status; In a case where the user intends to stop operating the currently matched component based on the recognized hand state of the user, the matching state effect is removed to allow the hand model to leave the currently matched component.

6. The method according to any one of claims 1 to 5, characterized in that Also includes: When the user's gesture action matches the preset operation instruction, the preset operation instruction is executed.

7. The method according to claim 4, characterized in that The step of analyzing the user's current operation intention and determining the intended virtual operation component when recognizing that the hand model moves to the vicinity of the virtual operation component includes: When it is recognized that the hand model moves to the vicinity of the virtual operating component, the Markov model is used to analyze the user's current operating intention and determine the intended virtual operating component.

8. A driving test simulation control interactive device, characterized in that: include: An acquisition unit, for capturing the position and posture of a user's hand in real time when a driving test simulation is performed in a closed VR visual environment, wherein the user is in a simulator cockpit; A mapping unit, used to map the acquired hand position and posture into a virtual environment of a VR visual environment, wherein the virtual operating components in the virtual environment are consistent in position and proportion with the physical components implementing the same functions in the simulator cockpit, so that when the user touches the virtual components in the virtual environment, the real hand also touches the corresponding physical components; A control unit is used to obtain the user's operation on the physical components and / or the virtual operation components to perform driving test simulation control in the simulator cockpit.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the driving test simulation control interaction method as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the driving test simulation control interaction method as described in any one of claims 1 to 7 is implemented.

Images