Multi-person online interaction driving simulation training method and system

Abstract

The invention provides a multi-person online interaction driving simulation training method and system, and relates to the technical field of virtual reality, and the method comprises the steps: receiving a training start request inputted by a user side after a virtual reality interaction device user completes wearing; according to the number of the driving item, matching a driving virtual scene, carrying out scene initialization on the virtual reality interaction equipment, prompting a user to start driving training, and collecting a video frame sequence of operation actions of the user at the same time; processing the user operation action video frame sequence through a driving action scoring model in combination with login user identity information; after a user sharing instruction is obtained, the driving action scoring result is shared to other online users selecting the driving item numbers, and feedback information of other users is obtained and sent to the user side. The problem that driving simulation training lacks multi-person interaction, personalized evaluation and real-time feedback in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a multi-person online interactive driving simulation training method and system. Background Art

[0002] With the rapid development of science and technology, virtual reality (VR) technology is increasingly being used in education, training, entertainment and other fields. Especially in the field of driving simulation, VR technology can provide a highly immersive training experience, simulate complex traffic environments and driving scenarios, and help drivers learn driving skills in a virtual environment, avoiding the risks of actual operation. In addition, virtual reality combined with artificial intelligence (AI) technology can evaluate and optimize user operations in real time, further improving the effectiveness and safety of training.

[0003] At present, many driving training systems rely on traditional simulators. Although they can provide simulations of some driving scenarios, most systems lack interactivity and personalized evaluation, and cannot provide effective feedback based on the specific needs of users. At the same time, traditional driving simulators usually have problems such as expensive equipment and complex operation. In addition, existing scoring systems often rely on static standards, making it difficult to accurately evaluate users' driving behaviors and operating details, and are mostly used by a single user, making it impossible to achieve collaborative training for multiple people. Therefore, there is an urgent need for a virtual driving simulation training system that can support online interaction among multiple people, provide personalized scoring and real-time feedback. Summary of the invention

[0004] The present application provides a multi-person online interactive driving simulation training method and system, which is used to solve the problem that driving simulation training in the prior art lacks multi-person interaction, personalized evaluation and real-time feedback.

[0005] In view of the above problems, the present application provides a multi-person online interactive driving simulation training method and system.

[0006] In a first aspect, the present application provides a multi-person online interactive driving simulation training method, the method comprising: after a user has finished putting on a virtual reality interactive device, receiving a training start request input by a user terminal, wherein the training start request includes a driving project number; matching a driving virtual scene according to the driving project number, initializing the scene of the virtual reality interactive device and prompting the user to start driving training, while collecting a user operation action video frame sequence; processing the user operation action video frame sequence through a driving action scoring model in combination with the identity information of the logged-in user to obtain a driving action scoring result; after obtaining a user sharing instruction, sharing the driving action scoring result with other online users who have selected the driving project number, obtaining feedback information from other users, and sending it to the user terminal.

[0007] In a second aspect, the present application provides a multi-person online interactive driving simulation training system, the system comprising: a start request receiving module, used to receive a training start request input by a user end after the user has finished putting on the virtual reality interactive device, wherein the training start request includes a driving project number; a video frame sequence acquisition module, used to match the driving virtual scene according to the driving project number, initialize the scene of the virtual reality interactive device and prompt the user to start driving training, and at the same time collect the user operation action video frame sequence; a scoring result acquisition module, used to process the user operation action video frame sequence through a driving action scoring model in combination with the login user identity information to obtain a driving action scoring result; a feedback information acquisition module, used to share the driving action scoring result with other online users who have selected the driving project number after obtaining a user sharing instruction, obtain other user feedback information, and send it to the user end.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] The present application provides a multi-person online interactive driving simulation training method and system, which relates to the field of virtual reality technology. The system receives the user's training start request through a virtual reality interactive device, matches the driving project number and initializes the corresponding virtual scene, prompting the user to start training. At the same time, the system collects the user's operating action video frame sequence, and processes these video frames through a driving action scoring model combined with user identity information to generate a driving action scoring result. Finally, the scoring results will be shared with other online users and fed back to the user end to promote interaction and learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a multi-person online interactive driving simulation training method is provided for this application;

[0011] Figure 2 A schematic diagram of the structure of a multi-person online interactive driving simulation training system is provided for this application.

[0012] Description of the reference numerals: start request receiving module 11, video frame sequence acquisition module 12, scoring result obtaining module 13, feedback information obtaining module 14. DETAILED DESCRIPTION

[0013] The present application provides a multi-person online interactive driving simulation training method and system to solve the problem of lack of multi-person interaction, personalized evaluation and real-time feedback in the driving simulation training in the prior art.

[0014] Embodiment 1, as Figure 1 As shown, the embodiment of the present application provides a multi-person online interactive driving simulation training method, the method comprising:

[0015] Step S100: After the user has put on the virtual reality interactive device, a training start request input by the user is received, wherein the training start request includes a driving project number.

[0016] Specifically, a virtual reality interactive device refers to a hardware device worn or used by a user, usually including a head-mounted display (HMD), a motion capture sensor, a handle or a controller, etc., which can connect the user's vision, hearing and touch to the virtual world, allowing the user to experience the virtual environment in an immersive way. In this application, the virtual reality interactive device is mainly used to display virtual driving scenes, collect user motion data and interact with the virtual environment. A training start request refers to an operation request from a user to start training in a virtual reality driving simulation training system. The training start request contains the user's intention information, such as the selected driving project number and other configuration options. The driving project number refers to a unique identifier that represents a specific driving training project selected by the user. Each driving project number corresponds to a specific virtual scene or task type, such as "city driving training", "highway driving training" or "parking training".

[0017] After the user wears the VR interactive device, the system will ensure the normal operation of the device, such as ensuring the display of the headset, the detection of the sensor and the connection of the controller. The user selects the driving training project he wants to perform through the VR device's controller or gesture input interface. According to the driving project number provided by the user, the system will match and load the corresponding virtual driving scene.

[0018] Through this step, the system can accurately identify the specific training items selected by the user, and load the corresponding virtual scenes based on this, ensuring that the user can carry out customized driving training according to their own needs. In addition, the entire process ensures the personalization of user training. Users can practice driving in different scenes in a virtual environment, and the system matches and initializes the scenes according to the user's selection, thereby improving the pertinence and effectiveness of the training.

[0019] Step S200: According to the driving project number, a driving virtual scene is matched, the scene of the virtual reality interaction device is initialized, and the user is prompted to start driving training, and a video frame sequence of the user's operation action is collected at the same time.

[0020] Specifically, the driving virtual scene refers to a virtual environment loaded by the system according to the driving project number selected by the user. The virtual scene includes various driving elements, such as roads, traffic signs, traffic signals, other vehicles, pedestrians, etc., aiming to provide users with a realistic driving experience. For example, the "city driving" scene contains busy urban streets, traffic lights and other vehicles, while the "highway driving" scene is a wide highway with less traffic. Scene initialization refers to the system loading and configuring the corresponding virtual scene according to the user's driving project number, including the configuration of environmental elements and the preparation of user interaction. The user operation action video frame sequence refers to the user action data captured by the virtual reality device during the user's driving training, which is usually collected in real time through the device's sensors and cameras. These data usually include the user's hand movements, body postures, and head movement trajectories, which are visual records of the user's operation behavior. This video frame sequence is used for subsequent driving action scoring.

[0021] When the user puts on the virtual reality interactive device, the system first receives the driving project number entered by the user, which indicates the specific training project selected by the user. For example, the user selected "city driving training" numbered "1", or selected "highway driving training" numbered "2". The system matches the virtual scene in the background according to the driving project number entered by the user. Each driving project number corresponds to a specific driving training scene. For example, the number "1" will be recognized by the system as a "city driving" scene, and the system will load a virtual environment including urban roads, traffic signals, pedestrians and other elements. The number "2" will load a highway scene, which will include less traffic interference and open roads.

[0022] After the scene matching is completed, the system will initialize the selected virtual scene. This process includes loading elements such as roads, vehicles, traffic signs, lighting effects, etc. into the virtual environment, and ensuring that all elements of the scene work properly and are ready for interaction with the user. For example, in the "City Driving Training" scene, the system simulates traffic lights, stop signs, and other traffic participants so that users can experience complex urban traffic conditions. After the scene initialization is completed, the system will issue a "Start Training" prompt to the user through the virtual reality device. At this time, the system will display a guidance message indicating that the user can start training and all control inputs are ready. For example, in the "City Driving Training" scene, the user will see the green light ahead and can start operating the steering wheel, accelerator or brake to start driving training.

[0023] After the user starts driving training, the system will collect the user's operating action video frame sequence in real time. Through the camera and sensor of the virtual reality device, the system records every action of the user (such as steering, acceleration, braking, etc.) and generates a continuous video frame sequence. These video frame sequences will be used for subsequent driving action scoring analysis to help the system evaluate the user's driving skills.

[0024] Through this step, the appropriate driving virtual scene can be automatically loaded according to the user's selection, and a realistic and highly interactive training environment can be provided for the user. The user can not only conduct personalized driving training in the virtual world, but also provide accurate input data for the subsequent scoring model through the collection of action videos. It not only improves the fun and interactivity of driving training, but also accurately evaluates the user's driving skills, ensuring the efficiency and practicality of the training effect.

[0025] Step S300: Processing the video frame sequence of the user operation action through the driving action scoring model in combination with the login user identity information to obtain a driving action scoring result.

[0026] Specifically, the driving action scoring model is a model based on artificial intelligence and machine learning algorithms, which is designed to analyze and evaluate the user's operating actions in virtual driving training. The model is usually composed of multiple deep learning layers, which can extract features from the user's operating video frame sequence and evaluate the quality and accuracy of their driving actions. The input of the scoring model is the user's operating action data, and the output is the corresponding scoring result. The login user identity information refers to the personal information provided by the user when entering the system, including the user's basic information (such as name, age, driving experience, etc.) and historical training data (such as past driving test results, common operating habits, etc.). The driving action scoring result is the evaluation result obtained after processing the user's operating video frame sequence through the driving action scoring model, usually presented in the form of a numerical score or grade, reflecting the accuracy, fluency, safety and other aspects of the user's driving action.

[0027] First, when a user logs in to the virtual driving simulation training system, the system will extract the login user identity information related to the user from the database based on the account information (such as user name and password) entered by the user. For example, user A is a driver with 3 years of driving experience, while user B is a beginner. The login information can help the system understand the user's background, thereby providing personalized evaluation criteria in the subsequent scoring process. After the user enters the virtual driving training, the system begins to collect the user's operating action video frame sequence through the virtual reality device. These video frame data record every action of the user in the training in real time.

[0028] The captured video frame sequence of the operation action will be passed as input to the driving action scoring model. The model scores the user's operation action by analyzing these features and combining the logged-in user's identity information (such as the user's driving experience or operation habits). After the model is processed, a driving action scoring result will be generated, usually presented in the form of a score or grade. For example, the model will give a score of 100, indicating that the user's operation fully meets the standard; or the score of some parts is low, prompting the user to improve, such as turning too quickly, braking too late, etc.

[0029] Through this step, the system can accurately score based on the user's operating actions, and provide personalized scoring results and feedback based on the user's background information (such as driving experience, historical performance, etc.). This not only makes the scoring process more intelligent and personalized, but also helps users discover their own driving deficiencies and improve them.

[0030] Step S400: after obtaining a user sharing instruction, the driving action scoring result is shared with other online users who select the driving item number, and feedback information from other users is obtained and sent to the user end.

[0031] Specifically, user sharing instructions refer to requests or operation instructions issued by users during driving training, usually after completing training actions, requiring the system to share or transmit certain information to other online users. These instructions can include requests to obtain scoring results, interact with other users, etc. Other online users who select driving project numbers refer to other users who have selected the same driving project number as the current user in the system. In other words, these users are participating in driving training of the same type or theme and are willing to share or obtain each other's scoring feedback. Feedback information refers to the response of other online users to the scoring results based on their own training experience or evaluation criteria after receiving the driving action scoring results. This can be feedback such as suggestions, praise or criticism of user actions. The user end refers to each user's terminal device, such as a computer, smartphone or virtual reality device, through which the user interacts with the system to obtain training information, scoring results, etc.

[0032] After completing driving training, users will issue a command request, asking the system to share the driving action score results obtained with other online users. For example, after completing a section of virtual road driving, a user chooses to issue a sharing request through the system menu, indicating that he wants other users to view his score results. According to the instructions shared by the user, the system pushes the driving action score results of the user to other online users who have selected the same driving project number through the network. For example, the score results of user A are shared with other users B, C, etc. who are doing the same driving project (such as urban driving simulation), and they can also see the score results of user A and respond. When other online users view the score results, they can provide feedback based on their own driving experience and training. This feedback can be text comments, score updates, or even operation suggestions. For example, user B thinks that user A's braking action is very good, while user C recommends that user A improve the acceleration operation.

[0033] The system pushes this feedback information to user A's device through the network, and user A can view comments and suggestions from other users on his device. For example, user A's virtual reality device will display feedback from users B and C, which may be "acceleration is smooth" or "steering is not accurate enough."

[0034] Through this step, the system can realize online interactive driving training feedback, which not only allows users to obtain instant scoring results, but also obtain valuable feedback from other users. Through the diversification of feedback, users can have a more comprehensive understanding of their driving level and areas for improvement, thereby improving their driving skills. This function enhances the interactivity, personalization and sociality of training, and promotes learning and progress.

[0035] Furthermore, the driving action scoring model includes N driving action scoring layers and a driving action scoring fully connected layer, N≥5, and the driving action scoring model is used to process the user operation action video frame sequence in combination with the login user identity information to obtain the driving action scoring result, including: using the N driving action scoring layers, in combination with the login user identity information, to process the user operation action video frame sequence to obtain N driving action scores; and performing mode analysis on the N driving action scores through the driving action scoring fully connected layer to obtain the driving action scoring result.

[0036] Specifically, the driving action scoring layer is a multiple layer in the scoring model, which is usually implemented by a convolutional layer (CNN) in a neural network or other types of network layers. Each driving action scoring layer is responsible for extracting different types of features, such as the user's motion trajectory during operation, the operation duration, the deviation of the operation from the standard action, etc. Through multi-layer analysis, the model can deeply understand every detail and finally obtain an accurate score for driving behavior. The driving action scoring fully connected layer is an important part of the scoring model and is usually located at the end of the model. The fully connected layer is responsible for synthesizing the feature information extracted by the previous layers to obtain the final evaluation result. In this step, the output of the previous scoring layers will be further processed by the fully connected layer, weighted, calculated or aggregated to produce the final scoring result. Mode analysis refers to statistical analysis of multiple scoring results to take the mode (i.e. the value with the highest frequency).

[0037] The user operation is scored through a multi-level driving action scoring model. The model includes at least N driving action scoring layers, where N ≥ 5. These scoring layers are usually convolutional layers of deep neural networks or other neural network layers. After inputting the user's operation video frame sequence, various feature information in the operation is extracted respectively. In the processing of each driving action scoring layer, the system will also use the logged-in user identity information (such as driving experience, historical training data, etc.) as additional input for personalized scoring adjustment.

[0038] Through N driving action scoring layers, the system processes the user's operation action video frame sequence layer by layer, and finally obtains N separate scores. These scores represent the system's independent evaluation results of each user's driving action. For example, the first scoring layer gives the score of the steering operation, the second scoring layer gives the score of the acceleration operation, and so on.

[0039] Next, the system passes these N independent scoring results to the driving action scoring fully connected layer. In the fully connected layer, the system performs a mode analysis on these scores, that is, counting the most frequently occurring scoring results to obtain a final comprehensive score. For example, if some scoring layers rate the user's operation poorly, while other scoring layers rate it well, the final score will use the frequently occurring score value to reduce the impact of accidental errors.

[0040] After mode analysis, the system will get the final driving action score, which can be a specific score (such as 80 points) or a level (such as good). The result will help users understand their performance in training and provide improvement suggestions for subsequent training.

[0041] Through this step, the system can accurately evaluate the user's driving behavior based on the user's specific operations and identity information, combined with a multi-level scoring model. The design of N scoring layers allows the system to conduct a detailed analysis of the user's driving actions from multiple dimensions, and the mode analysis ensures the accuracy and stability of the scoring, reducing accidental errors. The final scoring result not only reflects the user's driving skills, but also provides targeted improvement suggestions, thereby helping users improve their driving skills more effectively.

[0042] Furthermore, the user operation action video frame sequence is processed through the N driving action scoring layers in combination with the login user identity information to obtain N driving action scores, including: extracting a first driving action scoring layer according to the N driving action scoring layers; matching the login user's skeleton structure features based on the user database according to the login user identity information; performing qualified sample analysis according to the login user's skeleton structure features and the driving item number, and configuring a standard operation action video frame sequence; inputting the standard operation action video frame sequence and the user operation action video frame sequence into the first driving action scoring layer to obtain a first driving action score, and adding it to the N driving action scores.

[0043] Specifically, the logged-in user identity information refers to the personal data provided by the user when logging into the virtual driving system, including name, experience level, historical training data, common driving habits, etc. The user database is a database that stores user identity information, historical training results, driving preferences and other data. Through this database, the system can obtain and analyze the user's specific driving characteristics to help the scoring model more accurately evaluate their driving performance. The logged-in user's skeletal structure characteristics are information such as the user's body posture and joint angles captured by sensors or virtual reality devices, and are often obtained through human skeleton recognition technology. This feature is key data in the user's operation process and is used to analyze whether the user maintains the correct posture during driving or whether there are bad operating habits. Qualified sample analysis refers to the system extracting standard operating actions by analyzing a large number of known qualified driving samples.

[0044] First, the user's operation action video frame sequence is analyzed through N driving action scoring layers. In this process, the system extracts the first driving action scoring layer from the multi-layer scoring model, that is, the first layer of the model, which focuses on analyzing the basic characteristics of the user's driving behavior, such as steering wheel turning, accelerator acceleration, braking and other actions.

[0045] Next, the system will analyze the user's skeletal structural features based on the data in the logged-in user's identity information and the user database. This step is achieved by using the user's body posture information captured by sensors or virtual reality devices. For example, the system will extract posture data such as the user's arm angle, leg movement, and body rotation. These data help the system understand the user's behavior in driving, especially the differences from standard operations. Based on the user's skeletal structural characteristics, the system will perform qualified sample analysis in combination with the driving project number. The system will extract a standard operation action video frame sequence from a large amount of qualified sample data. These standard data represent the performance of ideal driving behavior, such as how to turn the steering wheel smoothly, how to brake reasonably, and how to adjust the speed of the vehicle.

[0046] In this step, the system compares the data extracted from the user's actual operation video frame sequence with the pre-configured standard operation action video frame sequence. For example, if the standard operation sequence shows that the steering wheel should be turned 45 degrees, but the user's actual operation is 40 degrees, the system will mark this detail and score it. After completing the comparison between the standard operation and the user's actual operation, the system inputs the standard operation data and the user's operation data into the first driving action scoring layer. In this scoring layer, the system will calculate the accuracy and fluency of the user's operation and generate the first driving action score. This score takes into account the differences between the standard action and the user's operation and assigns a score to it. For example, 100 points means full compliance with the standard, and a lower score indicates a deviation.

[0047] Finally, the first driving action score obtained will be added to the final scoring result as part of the N scores. This process will analyze different dimensions of driving actions according to multiple scoring layers, and finally form a comprehensive evaluation of user operations.

[0048] This step can achieve a more accurate and personalized driving training evaluation. Each user's operation can be carefully analyzed, and the system can customize the score according to the user's personal characteristics. At the same time, by comparing with standard operation samples, it can effectively identify the user's deficiencies in training. Ultimately, the user will obtain accurate evaluation results to help him continuously improve his driving skills during the training process.

[0049] Furthermore, a qualified sample analysis is performed based on the skeleton structure characteristics of the logged-in user and the driving project number, and a standard operation action video frame sequence is configured, including: collecting a qualified sample operation action video frame sequence of the driving project number, wherein the qualified sample operation action video frame sequence has a sample logged-in user skeleton structure characteristic; when the structural similarity between the sample logged-in user skeleton structure characteristic and the logged-in user skeleton structure characteristic is greater than or equal to a similarity threshold, setting the qualified sample operation action video frame sequence as the standard operation action video frame sequence.

[0050] Specifically, the qualified sample operation action video frame sequence refers to the operation action video frame sequence extracted from a large number of known standard driving operation samples. These video frames record ideal driving behaviors, such as smooth steering, timely braking, reasonable acceleration, etc. Each sample video frame sequence contains a series of continuous operation actions as a reference for standard driving behaviors. The skeleton structure features of the logged-in user refer to the motion data of various parts of the user's body (such as arms, legs, head, etc.) captured by sensors or virtual reality devices, usually extracted through skeleton recognition technology. The skeleton structure features can reflect the user's posture and movement accuracy during driving, which is very important for analyzing user operations. The similarity threshold is a preset value used to measure the degree of similarity between two skeleton structure features. Structural similarity is a numerical indicator to measure the similarity of the skeleton structure features of two users or samples.

[0051] By collecting a large amount of standard driving sample data, a qualified sample operation action video frame sequence is created. These video frames contain high-quality, standard-compliant driving operation sequences. Next, the skeleton structure features in each qualified sample operation video are extracted and stored.

[0052] When the logged-in user starts training, the system captures his or her skeletal structure features through the virtual reality device. This process uses skeleton recognition technology, such as a posture estimation algorithm based on deep learning, to convert the user's body movements and posture data into skeletal structure information. Then, the system calculates the similarity between the user's skeletal structure features and the sample's skeletal structure features to determine whether they meet the standards. Commonly used similarity calculation methods include cosine similarity or Euclidean distance, which can quantify the similarity between two skeletal structures.

[0053] During the comparison process, the system checks whether the similarity between the user's skeleton structure features and the sample skeleton structure features is greater than or equal to the preset similarity threshold. For example, if the threshold is set to 0.8, the system will only consider the user's action to be in line with the standard driving behavior when the similarity value is greater than or equal to 0.8. When the similarity is greater than or equal to the threshold, the system will determine the sample operation action video frame sequence as the standard operation action video frame sequence. This means that the video frame sequence can be used as a reference to guide users to compare with the standard behavior in subsequent training.

[0054] Through this step, qualified standard operation video sequences can be automatically matched according to the user's skeleton structure characteristics. This personalized qualified sample analysis method can significantly improve the pertinence and accuracy of training, ensuring that each user can obtain standard operation references that meet their actual conditions during training, thereby improving the learning effect and training efficiency of driving skills.

[0055] Furthermore, when the structural similarity between the skeleton structure features of the sample logged-in user and the skeleton structure features of the logged-in user is greater than or equal to a similarity threshold, the qualified sample operation action video frame sequence is set as the standard operation action video frame sequence, including: collecting the first user skeleton structure record data, the second user skeleton structure record data and the structural similarity identification result, and training a structural comparison network; through the structural comparison network, the skeleton structure features of the sample logged-in user and the skeleton structure features of the logged-in user are processed to obtain the structural similarity.

[0056] Specifically, the first user's skeleton structure record data refers to the first user's body skeleton structure information captured by a virtual reality device or sensor. These data usually include the coordinates, angles and motion trajectories of each joint, which are used to analyze the user's driving posture and movements. The second user's skeleton structure record data is similar to the first user's skeleton structure data. The second user's skeleton structure record data is also collected by a sensor or a virtual reality device, except that it comes from another user and is used to compare with the first user's data. The structural similarity identification result is a numerical or classification result used to quantify the similarity between two sets of skeleton structure features (such as the skeleton features of the first user and the second user). The structural comparison network refers to a deep learning network that is specifically used to process and compare the skeleton structure features of different users. By learning the skeleton data of a large number of users, the network can effectively measure the similarity between two skeleton structures. This network usually contains multiple convolutional layers, fully connected layers, and similarity calculation layers for feature extraction, comparison and analysis.

[0057] First, it is necessary to collect the skeleton structure recording data of the two users, that is, the body posture data of the first user and the second user during driving training. This data is usually captured in real time through high-precision sensors or virtual reality devices (such as wearable devices, motion capture systems or depth cameras). For example, the device tracks the movement of the user's arms, legs, spine and other parts to generate skeleton point cloud data.

[0058] Next, the system uses the collected skeleton data of multiple users to train the structural comparison network. Specifically, first, a large amount of user skeleton structure data is collected, including skeleton features of different users performing various actions in driving training, and the similarity identifier of each pair of data is annotated (for example, high similarity, low similarity). Next, a deep learning network is designed, which contains convolutional layers, pooling layers, and fully connected layers to extract the spatial and temporal patterns of skeleton features. Then, the paired skeleton feature data is input into the network, and the network parameters are optimized using the back propagation algorithm by calculating the loss between the similarity error (such as Euclidean distance or cosine similarity) and the annotated value. Finally, through iterative training, the network can accurately predict the similarity of skeleton feature pairs. After the training is completed, the validation set is used to evaluate the generalization ability of the model and ensure performance on new data. This network learns how to effectively identify the physical features of different users by processing a large amount of skeleton structure data, and is able to compare two skeleton features. The structural comparison network usually contains multiple convolutional layers, pooling layers, and fully connected layers, which can automatically extract the spatial and temporal features of the user skeleton.

[0059] After the network training is completed, the system uses the structure comparison network to compare the skeleton structure features of the sample user (e.g., qualified sample) with the skeleton structure features of the logged-in user. Through this network, the system calculates the structural similarity between the two. For example, if the first user's arm bending angle is similar to that of the second user, the system will output a higher similarity value (e.g., 0.85). If the similarity is lower than the preset similarity threshold (e.g., 0.7), it is considered that the two skeleton features are too different and cannot be used as qualified samples.

[0060] When the structural similarity is greater than or equal to the similarity threshold, the system considers that the skeleton features of the sample user are highly matched with the skeleton features of the logged-in user. At this point, the system will regard the qualified sample operation action video frame sequence that matches the sample skeleton features as a standard operation action video frame sequence for subsequent training and evaluation of the user's driving operation. Finally, after the above steps, the system will output a standardized operation sequence for users to refer to and learn during driving training. This standard operation sequence helps users understand how to perform ideal driving behaviors, thereby improving their driving skills.

[0061] This step effectively compares the skeleton features of users through the structural comparison network and ensures the quality and accuracy of training data by calculating structural similarity. By comparing with standard operation video sequences, the system can recommend standard training content to each user in a personalized way, thereby improving the effectiveness and pertinence of training.

[0062] Furthermore, the method also includes: when the structural similarity between the skeleton structure feature of the sample logged-in user and the skeleton structure feature of the logged-in user is less than a similarity threshold, updating the qualified sample operation action video frame sequence.

[0063] Specifically, the structural comparison network is used to compare the skeleton structure features of the sample logged-in user with the skeleton structure features of the logged-in user, and the similarity between the two is calculated. Through the deep learning network, this similarity reflects the degree of match between the two skeleton structures.

[0064] If the comparison result shows that the similarity is lower than the preset similarity threshold (for example, the similarity value is less than 0.7), the system believes that the current qualified sample operation action video frame sequence is not suitable for the user, and the user's skeleton features are significantly different from the standard samples. To ensure the effectiveness of the training, the system will automatically update the qualified sample operation action video frame sequence. This means that the system will collect new standard operation video frame sequences that are more in line with the skeleton structure characteristics of the logged-in user and replace the original sample video sequences. For example, if the skeleton structure of user A is less similar to the existing samples, the system will collect more samples similar to the skeleton structure of user A to generate an updated standard operation video sequence.

[0065] The updated qualified sample operation action video frame sequence will be used in the subsequent training process to ensure that the user's training is always highly matched with its skeleton structure, improving the training effect and accuracy. In this way, the system can continuously optimize the training process so that users can better imitate standard driving behaviors.

[0066] Through this update mechanism, the system can dynamically adjust the sample library to ensure that each user's training data always matches their actual movements and postures. This not only optimizes the training process, but also improves the personalization and adaptability of training, helping users to quickly improve their driving skills and reduce the problem of poor training results caused by mismatched samples.

[0067] Furthermore, the steps for constructing the first driving action scoring layer are as follows: configuring standard driving action video frame sequence recording data, driving action video frame sequence recording data to be evaluated and driving action scoring identification data; using the standard driving action video frame sequence recording data as input data of the first input channel of the convolutional neural network, using the driving action video frame sequence recording data to be evaluated as input data of the second input channel of the convolutional neural network, and using the driving action scoring identification data as output supervision data of the convolutional neural network to train the first driving action scoring layer.

[0068] Specifically, standard driving action video frame sequence recording data refers to driving operation video data that has been annotated and confirmed to meet the standards. To-be-evaluated driving action video frame sequence recording data refers to driving action video data performed by the current user during training. These data record the actual driving operations performed by the user, including various actions and the process of action execution. Driving action scoring identification data refers to the scoring information annotated on the video frame sequence, which is used to indicate the pros and cons of each driving action. For example, it can be a score (such as 1 to 5 points), or a label of whether the action is qualified (such as "qualified" and "unqualified"). These scoring data are used to guide neural networks to learn. Convolutional neural network (CNN) is a deep learning model that is particularly good at processing image data. It is composed of multiple convolutional layers, pooling layers, etc., and is used to automatically extract features from input data. Convolutional neural networks are usually used for tasks such as image classification and object detection. The input channel refers to the input layer of the convolutional neural network. The convolutional neural network can have multiple input channels, each channel is used to input different types of data. The data of each channel will be processed independently by the convolutional layer, and then learned and feature extracted through the shared network layer.

[0069] First, three types of data need to be prepared: standard driving action video frame sequence recording data, driving action video frame sequence recording data to be evaluated, and driving action score identification data. Input data into the convolutional neural network: Input the above data into different input channels of the convolutional neural network respectively: First input channel: Input the standard driving action video frame sequence recording data into the first channel of the convolutional neural network. These data are used as a reference to guide the network to understand the correct driving action mode. Second input channel: Input the driving action video frame sequence recording data to be evaluated into the second channel of the network. These data come from the user's operation and will be used to compare with the standard data.

[0070] The labeled driving action score identification data is used as the network's supervisory signal (label data) to inform the network of the expected output at each training step. By inputting standard data and data to be evaluated, the convolutional neural network automatically learns how to extract features from the user's action video and compares it with the standard action video. The network adjusts parameters by optimizing the loss function (such as cross entropy loss or mean square error) so that the network can predict the score or label of each action. During the training process, the network continuously adjusts the weights to minimize the error between the prediction and the actual label.

[0071] After training, the network can automatically evaluate the performance of the user's driving actions based on the characteristics of the video to be evaluated. For example, for a user's steering action, the network can give a score based on the input video frame sequence to indicate whether the steering is smooth and meets the standards.

[0072] Through this step, the convolutional neural network can automatically evaluate the user's driving actions and make corrections based on standard data. This not only improves the accuracy of the scoring, but also greatly reduces the need for manual intervention. Users can adjust their driving behavior based on the scoring feedback provided by the system, thereby improving training efficiency and effectiveness.

[0073] In summary, the multi-person online interactive driving simulation training method provided by the embodiment of the present application at least includes the following technical effects:

[0074] 1. By receiving the training start request input by the user, the system can automatically match the virtual driving scene according to the driving project number and initialize the scene to ensure that each user can start training in a suitable virtual environment, thereby enhancing the immersiveness and personalized experience of driving training.

[0075] 2. By introducing a driving action scoring model and processing the operation actions in combination with user identity information, the user's driving behavior can be evaluated more accurately, and scoring feedback that is more in line with the actual level can be provided, helping users to quickly understand the pros and cons of their own driving skills and guide improvements.

[0076] 3. The training structure comparison network can automatically calculate the similarity between skeleton structures and adjust the sample library based on the results, thereby ensuring that the system's evaluation of the user's actions is always based on the most matching sample data, improving the targetedness and personalization of training, and helping users improve efficiently.

[0077] Embodiment 2, based on the same inventive concept as the multi-person online interactive driving simulation training method in the above embodiment, Figure 2 As shown, the present application provides a multi-person online interactive driving simulation training system, the system comprising:

[0078] The start request receiving module 11 is used to receive a training start request input by the user end after the user has finished wearing the virtual reality interactive device, wherein the training start request includes a driving project number; the video frame sequence acquisition module 12 is used to match the driving virtual scene according to the driving project number, initialize the scene of the virtual reality interactive device and prompt the user to start driving training, and at the same time collect the user operation action video frame sequence; the scoring result obtaining module 13 is used to process the user operation action video frame sequence through a driving action scoring model in combination with the login user identity information to obtain a driving action scoring result; the feedback information obtaining module 14 is used to share the driving action scoring result with other online users who select the driving project number after obtaining a user sharing instruction, obtain other user feedback information, and send it to the user end.

[0079] Furthermore, the driving action scoring model includes N driving action scoring layers and a driving action scoring fully connected layer, N≥5, and the scoring result acquisition module 13 includes: a driving action scoring acquisition unit, used to process the user operation action video frame sequence through the N driving action scoring layers in combination with the login user identity information to obtain N driving action scores; a mode analysis unit, used to perform mode analysis on the N driving action scores through the driving action scoring fully connected layer to obtain the driving action scoring result.

[0080] Furthermore, the driving action score obtaining unit in the scoring result obtaining module 13 includes: a scoring layer extraction unit, used to extract a first driving action scoring layer according to the N driving action scoring layers; a structural feature matching unit, used to match the skeleton structure features of the logged-in user based on the user database according to the login user identity information; a sample analysis unit, used to perform qualified sample analysis according to the skeleton structure features of the logged-in user and the driving item number, and configure a standard operation action video frame sequence; an action score adding unit, used to input the standard operation action video frame sequence and the user operation action video frame sequence into the first driving action scoring layer, obtain a first driving action score, and add it to the N driving action scores.

[0081] Furthermore, the sample analysis unit in the scoring result acquisition module 13 is also used to perform the following steps: collecting a qualified sample operation action video frame sequence of the driving item number, wherein the qualified sample operation action video frame sequence has a sample logged-in user skeleton structure feature; when the structural similarity between the sample logged-in user skeleton structure feature and the logged-in user skeleton structure feature is greater than or equal to a similarity threshold, setting the qualified sample operation action video frame sequence to the standard operation action video frame sequence.

[0082] Furthermore, the sample analysis unit in the scoring result acquisition module 13 is also used to perform the following steps: collecting the first user's skeleton structure record data, the second user's skeleton structure record data and the structural similarity identification result, and training the structural comparison network; through the structural comparison network, processing the sample login user's skeleton structure features and the login user's skeleton structure features to obtain the structural similarity.

[0083] Furthermore, the sample analysis unit in the scoring result acquisition module 13 is also used to perform the following steps: when the structural similarity between the skeleton structure feature of the sample logged-in user and the skeleton structure feature of the logged-in user is less than a similarity threshold, updating the qualified sample operation action video frame sequence.

[0084] Furthermore, the scoring layer extraction unit in the scoring result acquisition module 13 is also used to perform the following steps: configuring standard driving action video frame sequence recording data, driving action video frame sequence recording data to be evaluated and driving action scoring identification data; using the standard driving action video frame sequence recording data as input data of the first input channel of the convolutional neural network, using the driving action video frame sequence recording data to be evaluated as input data of the second input channel of the convolutional neural network, and using the driving action scoring identification data as output supervision data of the convolutional neural network to train the first driving action scoring layer.

[0085] Through the above-mentioned detailed description of a multi-person online interactive driving simulation training method in this specification, those skilled in the art can clearly understand a multi-person online interactive driving simulation training system in this embodiment. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0086] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-person online interactive driving simulation training method, characterized in that: A driving simulation training system for multi-person online interaction, wherein the system is connected to a virtual reality interactive device for communication, and comprises: After the user of the virtual reality interactive device has finished wearing the device, a training start request input by the user is received, wherein the training start request includes a driving project number; According to the driving project number, a driving virtual scene is matched, the scene of the virtual reality interactive device is initialized and the user is prompted to start driving training, and a video frame sequence of the user's operation action is collected at the same time; The driving action scoring model is used to process the video frame sequence of the user's operation action in combination with the login user's identity information to obtain a driving action scoring result; After obtaining the user sharing instruction, the driving action scoring result is shared with other online users who select the driving project number, and feedback information from other users is obtained and sent to the user end.

2. The method according to claim 1, characterized in that The driving action scoring model includes N driving action scoring layers and a driving action scoring fully connected layer, N≥5. The driving action scoring model is used to process the user operation action video frame sequence in combination with the login user identity information to obtain a driving action scoring result, including: Through the N driving action scoring layers, combined with the login user identity information, the user operation action video frame sequence is processed to obtain N driving action scores; The driving action score fully connected layer performs mode analysis on the N driving action scores to obtain the driving action score result.

3. The method according to claim 2, characterized in that Through the N driving action scoring layers, combined with the login user identity information, the user operation action video frame sequence is processed to obtain N driving action scores, including: Extracting a first driving action scoring layer according to the N driving action scoring layers; According to the identity information of the logged-in user, based on the user database, the skeleton structure characteristics of the logged-in user are matched; Perform qualified sample analysis based on the skeleton structure features of the logged-in user and the driving project number, and configure a standard operation action video frame sequence; The standard operation action video frame sequence and the user operation action video frame sequence are input into the first driving action scoring layer to obtain a first driving action score, which is added to the N driving action scores.

4. The method according to claim 3, characterized in that According to the skeleton structure characteristics of the logged-in user and the driving project number, a qualified sample analysis is performed, and a standard operation action video frame sequence is configured, including: Collecting a qualified sample operation action video frame sequence of the driving project number, wherein the qualified sample operation action video frame sequence has a skeleton structure feature of a sample logged-in user; When the structural similarity between the skeleton structure feature of the sample logged-in user and the skeleton structure feature of the logged-in user is greater than or equal to a similarity threshold, the qualified sample operation action video frame sequence is set as the standard operation action video frame sequence.

5. The method according to claim 4, characterized in that When the structural similarity between the skeleton structure feature of the sample logged-in user and the skeleton structure feature of the logged-in user is greater than or equal to a similarity threshold, setting the qualified sample operation action video frame sequence as the standard operation action video frame sequence, including: Collecting skeleton structure record data of the first user, skeleton structure record data of the second user and structural similarity identification results, and training a structural comparison network; The skeleton structure features of the sample logged-in user and the skeleton structure features of the logged-in user are processed through the structure comparison network to obtain the structural similarity.

6. The method according to claim 4, characterized in that Also includes: When the structural similarity between the skeleton structure feature of the sample logged-in user and the skeleton structure feature of the logged-in user is less than a similarity threshold, the qualified sample operation action video frame sequence is updated.

7. The method according to claim 3, characterized in that The steps for constructing the first driving action scoring layer are as follows: Configure standard driving action video frame sequence recording data, driving action video frame sequence recording data to be evaluated and driving action scoring identification data; The standard driving action video frame sequence recording data is used as input data of the first input channel of the convolutional neural network, the driving action video frame sequence recording data to be evaluated is used as input data of the second input channel of the convolutional neural network, and the driving action score identification data is used as output supervision data of the convolutional neural network to train the first driving action scoring layer.

8. A multi-person online interactive driving simulation training system, characterized in that: The system comprises: A start request receiving module, used for receiving a training start request input by a user end after the user has finished wearing the virtual reality interactive device, wherein the training start request includes a driving project number; A video frame sequence acquisition module is used to match the driving virtual scene according to the driving project number, initialize the scene of the virtual reality interaction device and prompt the user to start driving training, and collect the video frame sequence of the user's operation action at the same time; A scoring result obtaining module is used to process the video frame sequence of the user's operation action through a driving action scoring model in combination with the identity information of the logged-in user to obtain a driving action scoring result; The feedback information acquisition module is used to share the driving action scoring result with other online users who select the driving project number after obtaining the user sharing instruction, obtain other user feedback information, and send it to the user end.

9. A multi-person online interactive driving simulation training system as claimed in claim 8, characterized in that: The driving action scoring model includes N driving action scoring layers and a driving action scoring fully connected layer, N≥5, and the scoring result acquisition module includes: A driving action score obtaining unit, configured to process the video frame sequence of the user operation action through the N driving action scoring layers in combination with the login user identity information to obtain N driving action scores; A mode analysis unit is used to perform mode analysis on the N driving action scores through the driving action score fully connected layer to obtain the driving action score result.

10. A multi-person online interactive driving simulation training system as claimed in claim 9, characterized in that: The driving action score obtaining unit comprises: A scoring layer extraction unit, configured to extract a first driving action scoring layer according to the N driving action scoring layers; A structural feature matching unit, used to match the skeleton structural features of the logged-in user based on the user database according to the logged-in user identity information; A sample analysis unit, used to perform qualified sample analysis according to the skeleton structure characteristics of the logged-in user and the driving project number, and configure a standard operation action video frame sequence; The action score adding unit is used to input the standard operation action video frame sequence and the user operation action video frame sequence into the first driving action scoring layer to obtain a first driving action score, and add it to the N driving action scores.

Images