A system and method for adjusting a virtual scene for dynamic driving of a vehicle

Abstract

The embodiment of the application avoids the situation that the driving environment is poor and reduces the driver's perception and reaction ability by collecting scene data when the vehicle is driving at the current position, calibrating each element in the scene from multiple directions, inputting the scene data of the current position of the vehicle into a pre-trained deep learning model to obtain a scene prediction result, and constructing a virtual scene corresponding to the scene prediction result of the current position by using virtual reality technology. The corresponding virtual scene is dynamically adjusted at the same time when the scene prediction result of the current position is updated. The deep learning model is obtained by regression training through learning the corresponding relationship between the vehicle position and the historical scene data containing the calibrated elements, thereby solving the limitations of scene prediction and adjustment in the traditional vehicle-mounted virtual reality system and improving the driving safety and driving experience of the vehicle during driving.

Description

Technical Field

[0001] This invention relates to the automotive field, and more particularly to a system and method for adjusting a virtual scene of vehicle dynamic driving. Background Technology

[0002] Currently, as people pursue a higher quality of life, they place particular emphasis on the driving experience. However, the complexity of driving environments is unavoidable. There are frequent situations where obstacles are unknown due to rain or fog, and there are also instances where drivers fail to recognize road signs in time, leading to wrong turns. This presents drivers with increasing challenges. Traditional vehicle driving relies heavily on reversing cameras, which don't provide a comprehensive understanding of the surrounding environment, thus compromising driving safety.

[0003] To improve driver safety and driving experience, virtual reality technology has been introduced into vehicles. However, due to the limitations of traditional in-vehicle virtual reality systems in scene prediction and adjustment, these systems often result in poor driving safety and a subpar driving experience. Summary of the Invention

[0004] In view of this, in order to solve the problem that traditional in-vehicle virtual reality systems have limitations in scene prediction and adjustment, resulting in poor driving safety and driving experience during vehicle operation, this invention provides a system and method for adjusting the virtual scene of dynamic vehicle driving.

[0005] A system for adjusting a virtual scene of vehicle dynamics includes:

[0006] The data acquisition and calibration module is used to collect scene data of the vehicle at the current location and calibrate various elements in the scene from multiple perspectives.

[0007] The training module is used to input scene data of the vehicle's current position into a pre-trained deep learning model to predict scene elements and obtain scene prediction results.

[0008] The virtual scene construction module uses virtual reality technology to construct a virtual scene corresponding to the scene prediction results of the current location;

[0009] The dynamic adjustment module is used to dynamically adjust the corresponding virtual scene when the scene prediction result at the current location is updated.

[0010] The deep learning model is obtained through regression training by learning the correspondence between vehicle positions and historical scene data containing calibration elements.

[0011] In one possible embodiment, the training module includes:

[0012] The model training submodule is used to perform training of deep learning models.

[0013] In one possible embodiment, the model training submodule includes:

[0014] The learning unit is used to learn the historical scene data, extract the image spatial features of the scene data containing calibration elements at each moment within a preset time period, and determine the correspondence between the image spatial features and the vehicle position.

[0015] The classification unit is used to normalize the image spatial features of the same category according to the preset data category to form a feature sequence subset, and the feature sequence subsets of multiple categories constitute a feature sequence set;

[0016] The test training unit is used to take into input the vehicle position at a certain moment and the scene data containing calibration elements, determine the image spatial features corresponding to the scene data based on the correspondence, and find the first feature sequence subset where the image spatial features are located;

[0017] The training target unit is used to find the target image spatial features in the first feature sequence subset at the next moment with the expectation of achieving a preset prediction accuracy, and to determine the vehicle position corresponding to the target image spatial features and the scene data containing the calibration elements. Training is completed when the expectation is achieved.

[0018] In one possible embodiment, the training module further includes:

[0019] A receiving submodule is used to receive the scene data at the current location;

[0020] The feature extraction submodule is used to extract the image spatial features of the scene data and determine a second feature sequence subset of the image spatial features;

[0021] The result prediction submodule is used to determine the time of the current position based on the second feature sequence subset, predict the vehicle position and scene data containing calibration elements at the next time moment, and output the scene prediction result.

[0022] In one possible embodiment, the virtual scene construction module includes:

[0023] The 3D model submodule is used to build a 3D model of the vehicle and calibration elements in the scene prediction results at the current location;

[0024] The element position submodule is used to determine the element position of the calibrated element based on the image spatial features corresponding to the scene prediction result.

[0025] The virtual location submodule uses virtual reality technology to determine the virtual vehicle position of the vehicle 3D model based on the vehicle position of the scene prediction result, and determines the virtual element position of the 3D model of the calibration element based on the element position of the calibration element.

[0026] The virtual scene submodule is used to determine the virtual scene of the current location based on the virtual location of the vehicle and the virtual location of the element.

[0027] In one possible embodiment, the system further includes:

[0028] A display for projecting the virtual scene into the user's field of vision.

[0029] In one possible embodiment, the system further includes: a plurality of sensors and at least one audio device;

[0030] The sensor is used to track the user's head movement and determine the range of the user's head movement.

[0031] The audio device is used to collect sound information within a preset range of the vehicle and output the sound information as audio information of the corresponding virtual scene within the range of the user's head movement.

[0032] Based on the same concept, a method for adjusting a virtual scene of vehicle dynamics includes:

[0033] Collect scene data of vehicles moving at the current location and calibrate various elements in the scene from multiple perspectives;

[0034] The scene data of the vehicle's current location is input into a pre-trained deep learning model to predict scene elements and obtain scene prediction results.

[0035] Virtual reality technology is used to construct a virtual scene corresponding to the scene prediction results of the current location;

[0036] When the scene prediction result for the current location is updated, the corresponding virtual scene is also dynamically adjusted at the same time;

[0037] The deep learning model is obtained through regression training by learning the correspondence between vehicle positions and historical scene data containing calibration elements.

[0038] In one possible embodiment, training the deep learning model includes:

[0039] Learn the historical scene data, extract the image spatial features from the scene data containing calibration elements at each moment within a preset time period, and determine the correspondence between the image spatial features and the vehicle position;

[0040] According to the preset data category, the image spatial features of the same category are normalized to form a feature sequence subset, and the feature sequence subsets of multiple categories constitute a feature sequence set;

[0041] Input the vehicle position at a certain moment and the scene data containing calibration elements, determine the image spatial features corresponding to the scene data based on the correspondence, and find the first feature sequence subset where the image spatial features are located;

[0042] With the goal of achieving a preset prediction accuracy, the target image spatial features for the next moment are searched in the first feature sequence subset to determine the vehicle position corresponding to the target image spatial features and the scene data containing the calibration elements. Training is completed when the desired accuracy is achieved.

[0043] In one possible embodiment, the step of inputting scene data of the vehicle's current position into a pre-trained deep learning model to predict scene elements and obtain scene prediction results includes:

[0044] Receive the scene data at the current location;

[0045] Extract the image spatial features of the scene data and determine a second feature sequence subset of the image spatial features;

[0046] The time of the current position is determined based on the second feature sequence subset, and the vehicle position and scene data containing calibration elements are predicted at the next time step, and the scene prediction result is output.

[0047] This invention collects scene data of a vehicle in motion at its current location and calibrates various elements in the scene from multiple angles. The scene data of the vehicle's current location is then input into a pre-trained deep learning model to predict scene elements and obtain scene prediction results. Virtual reality technology is used to construct a virtual scene corresponding to the current location's scene prediction results, thereby avoiding situations where poor driving conditions reduce the driver's perception and reaction capabilities. When the current location's scene prediction results are updated, the corresponding virtual scene is also dynamically adjusted simultaneously. The deep learning model is obtained through regression training by learning the correspondence between the vehicle's position and historical scene data containing calibrated elements. This solves the limitations of traditional in-vehicle virtual reality systems in scene prediction and adjustment, which leads to poor driving safety and a poor driving experience during vehicle operation. Attached Figure Description

[0048] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0049] Figure 1A schematic diagram of the architecture of a vehicle dynamic driving virtual scene adjustment system provided in an embodiment of the present invention;

[0050] Figure 2 A flowchart illustrating an embodiment of a method for adjusting a virtual scene of vehicle dynamics provided by an embodiment of the present invention;

[0051] Figure 3 A flowchart illustrating an embodiment of another method for adjusting a virtual scene of vehicle dynamic driving provided by an embodiment of the present invention;

[0052] Figure 4 A flowchart illustrating an embodiment of a method for adjusting a virtual scene of vehicle dynamics provided by the present invention;

[0053] Figure 5 A flowchart illustrating an embodiment of a method for adjusting a virtual scene of vehicle dynamics provided by the present invention;

[0054] Figure 6 This is a schematic diagram of the architecture of another vehicle dynamic driving virtual scene adjustment system provided in an embodiment of the present invention. Detailed Implementation

[0055] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0056] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this invention is for describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0057] See Figure 1 This is a schematic diagram of the architecture of a vehicle dynamic driving virtual scene adjustment system provided in an embodiment of the present invention.

[0058] Figure 1 The system architecture shown includes: a data acquisition and annotation module, a deep learning model training module, a virtual scene construction module, and a virtual reality device integration module, which can be connected in pairs through electrical lines.

[0059] The data acquisition and calibration module is used to collect real-world scene data of the vehicle at the current location during its movement and to calibrate the elements in the scene data.

[0060] The real-world scenario data here can include driving road information, such as road conditions, road signs, obstacle information, and shop names. Elements can include roads, buildings, vehicles, pedestrians, etc., without limitation. The vehicles mentioned above can be cars, buses, trucks, vans, motorcycles, trains, etc., without limitation.

[0061] The deep learning model training module is used to train a deep learning model to output prediction results using the vehicle position collected at the current moment and scene data including calibration information. The prediction results include: the vehicle position at the next moment and scene data including calibration information. The deep learning model here can be a neural network model, a convolutional neural network model (CNN), a recurrent neural network model (RNN), an autoencoder, etc. This embodiment uses convolutional neural networks and recurrent neural networks as examples.

[0062] The training of the deep learning model is explained as follows: First, a convolutional neural network (CNN) is used to process the vehicle position and scene data (images) containing calibration elements. Specifically, convolutional and pooling layers are used to extract image spatial features from the scene data, determine the correspondence between image spatial features and vehicle position, and a fully connected layer is used to classify the image spatial features to determine the sequence data. Next, a recurrent neural network (RNN) is used to process the sequence data. The recurrent connections allow the network to remember previous information while processing the sequence data and to interact between the current input and previous sequence data. Then, the CNN is used again for regression training based on the vehicle position and scene data (images) containing calibration elements. Training ends when the desired prediction accuracy is achieved.

[0063] It should be noted that after the training is completed, if new vehicle positions and scene data are input, regression training will no longer be performed, and the scene preset results will be output directly.

[0064] The virtual scene construction module is used to build virtual scenes based on the scene prediction results mentioned above. The virtual scene includes: 3D models of calibration elements and 3D models of the vehicle itself. These 3D models can be created using 3D modeling software. The model format can conform to the user's workflow, habits, and needs, including but not limited to: MAX, Maya, OBJ, FBX, GLB, STL, SKP, and other model formats.

[0065] A virtual reality device integration module is used to connect pre-built virtual scenes and achieve module integration. It may include: a display that projects the virtual scene to the user's eyes, a controllable controller, multiple sensors that track the user's head movements, and at least one audio device (audio output device) that collects scene sounds, converts them into audio, and provides them to the virtual scene. There are no limitations on this.

[0066] The display here can be a projection display, a data transmission display, a head-mounted display, etc. The aforementioned sensors can be not only tracking sensors, but multiple sound sensors can also be added to the audio device to collect sound from the scene.

[0067] Virtual reality devices allow drivers to observe and interact with virtual scenes. Simultaneously, real-time vehicle status information is fed back to a deep learning model, enabling dynamic adjustments and updates to the virtual scene.

[0068] The method for adjusting the virtual scene of vehicle dynamic driving provided by the present invention will be further explained below with reference to the accompanying drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0069] See Figure 2 This is a flowchart illustrating an embodiment of a method for adjusting a virtual scene of dynamic vehicle driving provided by the present invention. As one embodiment, Figure 2 The illustrated process can be applied to adjustment systems for virtual scenarios of vehicle dynamics. For example... Figure 2 As shown, the process may include the following steps:

[0070] Step 201: Collect scene data of the vehicle at the current location while it is moving, and calibrate each element in the scene from multiple angles.

[0071] The scene data here can be current real-world scene data or historical scene data. Scene data can include driving road information, weather information, etc. Driving road information can include: whether the road conditions are smooth, whether there are obstacles on the road, dynamic obstacle distance range prompts, static obstacle distance range prompts, etc. Weather information can include: sunny, light rain, heavy rain, rainstorm, hail, night, etc., without restriction.

[0072] In one embodiment, the scene surrounding the vehicle often includes not only what the driver can see visually and the obstacles and markers within that scene, but also people walking behind the vehicle and other vehicles that are being driven, which could potentially become obstacles approaching the vehicle. Therefore, it is necessary to label all elements within a preset range around the vehicle when labeling elements.

[0073] Furthermore, collecting scene data from multiple angles allows for the identification of more elements than those within the line of sight. The more elements are labeled, the clearer the correspondence between them and the current position, thus laying the foundation for the accuracy and synchronization of subsequent virtual scene construction and dynamic updates.

[0074] The elements to be calibrated in the scene may include: roads, buildings, vehicles, moving objects, trees, etc., without any restrictions.

[0075] Step 202: Input the scene data of the vehicle's current position into the pre-trained deep learning model to predict scene elements and obtain scene prediction results.

[0076] The deep learning model is obtained through regression training by learning the correspondence between vehicle positions and historical scene data containing calibration elements.

[0077] In one embodiment, the deep learning model makes a prediction every time the vehicle's current location changes. Of course, the user can also set the number of predictions at the current location to determine the number of times the deep learning model predicts the scenario; this is not limited here.

[0078] Optionally, for deep learning models, training data may include: vehicle location and historical scene data containing calibration elements. Finding the correspondence between these elements during model training helps in accurately predicting data at subsequent times when new data is input. However, the directly obtainable data can only correspond to vehicle location and historical scene information containing calibration elements; however, this historical scene information can change at any time.

[0079] For example, in historical scene data, at a certain moment, a car is driving on a road. The vehicle's location is on Street 1, where the marked elements include the road itself, three willow trees, a second pharmacy, and recycling bins. If, upon re-entering this location, the recycling bins are no longer present, the likelihood of scene prediction errors increases. If the historical scene data also includes more iconic buildings, it helps in quickly and accurately predicting the scene at the next moment.

[0080] Furthermore, vehicle location and historical scene data containing calibration elements are data over a period of time, i.e., data from consecutive moments. This data is complex, and therefore, historical scene data can be classified during deep learning model training. Specifically, this can be achieved through... Figure 3 The flowchart shown illustrates the training of a deep learning model. Figure 3 The diagram shown is a flowchart of another embodiment of a method for adjusting a virtual scene of vehicle dynamic driving provided by the present invention. The process may include the following steps:

[0081] Step 301: Learn historical scene data, extract image spatial features from scene data containing calibration elements at each time within a preset time period, and determine the correspondence between image spatial features and vehicle positions.

[0082] Step 302: According to the preset data category, the image spatial features of the same category are normalized to form a feature sequence subset, and the feature sequence subsets of multiple categories constitute a feature sequence set.

[0083] The following provides a unified explanation of steps 301 to 302:

[0084] In one embodiment, as illustrated in step 202, learning from historical scene data can determine the correspondence between image spatial features and vehicle positions through model regression training. To improve the speed and accuracy of prediction, image spatial features can be classified.

[0085] The preset data categories can include vehicle location and label element similarity as data classification categories. The label element similarity can be set and is not limited here.

[0086] Optionally, image spatial features of the same category can include image spatial features of this category at various times. Thus, a feature time series set can be formed from front to back according to the timeline. Alternatively, a feature location sequence set can be determined by marking a location that is close to the location within a preset range according to the location line and sorting the locations according to the distance from that location. No restrictions are imposed here.

[0087] Step 303: Input the vehicle position at a certain moment and the scene data containing the calibration elements. Based on the correspondence, determine the image spatial features corresponding to the scene data and find the first feature sequence subset where the image spatial features are located.

[0088] Step 304: With the desired prediction accuracy, predict the image spatial features of the next moment based on the first feature sequence subset, determine the vehicle position and scene data containing the calibration elements at the next moment, and complete the training when the desired accuracy is achieved.

[0089] The following provides a unified explanation of steps 303 to 304:

[0090] In one embodiment, prediction can be made based on the vehicle position at a certain moment in historical scene data and scene data containing calibration elements. It is then determined whether the predicted vehicle position at the next moment in historical scene data and scene data containing calibration elements are the predicted predictions. An expected prediction accuracy is set for the prediction. The aforementioned prediction accuracy is taken as a percentage. The closer it is to 1, the greater the expectation and the easier it is to complete the training.

[0091] In another embodiment, historical scene data can be divided into training and testing sets. This allows for data-driven differentiation between the training and testing of the deep learning model, improving training accuracy. The deep learning model is trained using the training set. Upon reaching convergence, the training results are tested using the testing set. The testing process can refer to steps 303 to 304 above and will not be repeated here. After meeting the testing requirements, scene prediction can be performed. The convergence condition can be a preset number of training iterations. The testing requirement can be prediction accuracy, which is not limited here.

[0092] As for the specific process of inputting scene data of the vehicle's current position into a pre-trained deep learning model to predict scene elements and obtain scene prediction results, it can be found through... Figure 4 The flowchart of the illustrated embodiment is implemented as follows. Figure 4 The diagram shown is a flowchart of another embodiment of a method for adjusting a virtual scene of vehicle dynamic driving provided by the present invention. The process may include the following steps:

[0093] Step 401: Receive scene data at the current location.

[0094] Step 402: Extract the image spatial features of the scene data and determine the second feature sequence subset of the image spatial features.

[0095] Step 403: Determine the time of the current position based on the subset of the second feature sequence, predict the vehicle position and scene data containing calibration elements at the next time step, and output the scene prediction result.

[0096] For explanations of steps 401 to 403, please refer to the explanations of the deep learning model training module and steps 301 to 303 above, which will not be detailed here.

[0097] Step 203: Use virtual reality technology to construct a virtual scene corresponding to the scene prediction results of the current location.

[0098] In practice, virtual reality technology is mostly applied to vehicles for vehicle demonstrations, which are closer to reality. While saving on the cost of new vehicles, it allows users to gain a certain understanding of the vehicle's exterior and interior through virtual images, thus creating an immersive experience.

[0099] In one embodiment, virtual reality technology is effective for displaying static vehicles, but it struggles to seamlessly transition between dynamic scenes while driving. Therefore, scene prediction results from a deep learning model are applied to construct a virtual scene, enabling a continuous and dynamic virtual experience and providing the driver with a better driving experience. The specific method for constructing a virtual scene corresponding to the current location's scene prediction results using virtual reality technology can be described through… Figure 5 The flowchart of the illustrated embodiment is implemented as follows. Figure 5 The diagram shown is a flowchart of another embodiment of a method for adjusting a virtual scene of vehicle dynamic driving provided by the present invention. The process may include the following steps:

[0100] Step 501: Construct a 3D model of the vehicle and calibration elements in the scene prediction results at the current location.

[0101] Step 502: Determine the element position of the calibration element based on the image spatial features corresponding to the scene prediction result.

[0102] Step 503: Using virtual reality technology, determine the virtual position of the vehicle in the 3D model of the vehicle based on the vehicle position of the scene prediction results, and determine the virtual position of the elements in the 3D model of the calibration elements based on the element positions of the calibration elements.

[0103] Step 504: Determine the virtual scene of the current location based on the virtual location of the vehicle and the virtual location of the elements.

[0104] The following provides a unified explanation of steps 501 to 504:

[0105] In one embodiment, to present a virtual scene of a vehicle in motion, a specific real-world scene can be simulated within the virtual scene, thus requiring all elements within the scene to be presented. However, common virtual reality technologies can only simulate a limited number of elements. A 3D modeling approach can be used to construct a 3D model of the vehicle and calibration elements from the scene prediction results. Then, the image spatial features of the scene prediction results can be obtained to determine the location of the calibration elements. Combined with the vehicle's position, the location of the vehicle and calibration elements within the virtual scene can be determined, thereby improving the driver's perception and reaction capabilities.

[0106] Step 204: When the scene prediction result for the current location is updated, the corresponding virtual scene is also dynamically adjusted at the same time.

[0107] In one embodiment, it is not very meaningful to convert a scene that has already occurred to the driver into a virtual scene for timely updates. In this embodiment, based on the scene prediction result output in step 202 above, when the vehicle's current position is updated, the virtual scene for the next moment will be output. By combining the predicted virtual scene with the real scene, a helpful prompt is given to the driver, improving driving safety and ensuring a good driving experience.

[0108] In addition, to increase the feasibility of this embodiment, the predicted virtual scene can be adjusted in a timely manner according to the real scene, and no restrictions are imposed here.

[0109] This invention collects scene data of a vehicle in motion at its current location and calibrates various elements in the scene from multiple angles. The scene data of the vehicle's current location is then input into a pre-trained deep learning model to predict scene elements and obtain scene prediction results. Virtual reality technology is used to construct a virtual scene corresponding to the current location's scene prediction results, thereby avoiding situations where poor driving conditions reduce the driver's perception and reaction capabilities. When the current location's scene prediction results are updated, the corresponding virtual scene is also dynamically adjusted simultaneously. The deep learning model is obtained through regression training by learning the correspondence between the vehicle's position and historical scene data containing calibrated elements. This solves the limitations of traditional in-vehicle virtual reality systems in scene prediction and adjustment, which leads to poor driving safety and a poor driving experience during vehicle operation.

[0110] See Figure 6 This is a schematic diagram of the architecture of another vehicle dynamic driving virtual scene adjustment system provided in an embodiment of the present invention. Figure 6 As shown, it includes:

[0111] The data acquisition and calibration module is used to collect scene data of the vehicle at the current location and calibrate various elements in the scene from multiple perspectives.

[0112] The training module is used to input scene data of the vehicle's current position into a pre-trained deep learning model to predict scene elements and obtain scene prediction results.

[0113] The virtual scene construction module uses virtual reality technology to construct a virtual scene corresponding to the scene prediction results of the current location;

[0114] The dynamic adjustment module is used to dynamically adjust the corresponding virtual scene when the scene prediction result at the current location is updated.

[0115] The deep learning model is obtained through regression training by learning the correspondence between vehicle positions and historical scene data containing calibration elements.

[0116] In one possible embodiment, the training module includes:

[0117] The model training submodule is used to perform training of deep learning models.

[0118] In one possible embodiment, the model training submodule includes:

[0119] The learning unit is used to learn the historical scene data, extract the image spatial features of the scene data containing calibration elements at each moment within a preset time period, and determine the correspondence between the image spatial features and the vehicle position.

[0120] The classification unit is used to normalize the image spatial features of the same category according to the preset data category to form a feature sequence subset, and the feature sequence subsets of multiple categories constitute a feature sequence set;

[0121] The test training unit is used to take into input the vehicle position at a certain moment and the scene data containing calibration elements, determine the image spatial features corresponding to the scene data based on the correspondence, and find the first feature sequence subset where the image spatial features are located;

[0122] The training target unit is used to find the target image spatial features in the first feature sequence subset at the next moment with the expectation of achieving a preset prediction accuracy, and to determine the vehicle position corresponding to the target image spatial features and the scene data containing the calibration elements. Training is completed when the expectation is achieved.

[0123] In one possible embodiment, the training module further includes:

[0124] A receiving submodule is used to receive the scene data at the current location;

[0125] The feature extraction submodule is used to extract the image spatial features of the scene data and determine a second feature sequence subset of the image spatial features;

[0126] The result prediction submodule is used to determine the time of the current position based on the second feature sequence subset, predict the vehicle position and scene data containing calibration elements at the next time moment, and output the scene prediction result.

[0127] In one possible embodiment, the virtual scene construction module includes:

[0128] The 3D model submodule is used to build a 3D model of the vehicle and calibration elements in the scene prediction results at the current location;

[0129] The element position submodule is used to determine the element position of the calibrated element based on the image spatial features corresponding to the scene prediction result.

[0130] The virtual location submodule uses virtual reality technology to determine the virtual vehicle position of the vehicle 3D model based on the vehicle position of the scene prediction result, and determines the virtual element position of the 3D model of the calibration element based on the element position of the calibration element.

[0131] The virtual scene submodule is used to determine the virtual scene of the current location based on the virtual location of the vehicle and the virtual location of the element.

[0132] In one possible embodiment, the system further includes (not shown in the figures):

[0133] A display for projecting the virtual scene into the user's field of vision.

[0134] In one possible embodiment, the system further includes: a plurality of sensors and at least one audio device;

[0135] The sensor is used to track the user's head movement and determine the range of the user's head movement.

[0136] The audio device is used to collect sound information within a preset range of the vehicle and output the sound information as audio information of the corresponding virtual scene within the range of the user's head movement.

[0137] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative in all respects and are not the only ones. All modifications within the scope of this invention or its equivalents are included in this invention.

[0138] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0140] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0141] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A system for adjusting a virtual scene of vehicle dynamics, characterized in that, include: The data acquisition and calibration module is used to collect scene data of the vehicle at the current location and calibrate various elements in the scene from multiple perspectives. The training module is used to input scene data of the vehicle's current position into a pre-trained deep learning model to predict scene elements and obtain scene prediction results. The virtual scene construction module uses virtual reality technology to construct a virtual scene corresponding to the scene prediction results of the current location; The dynamic adjustment module is used to dynamically adjust the corresponding virtual scene when the scene prediction result at the current location is updated. Among them, the deep learning model is obtained by learning the correspondence between vehicle position and historical scene data containing calibration elements, and then undergoing regression training. The training module includes: The model training submodule is used to perform training of deep learning models; The model training submodule includes: The learning unit is used to learn the historical scene data, extract the image spatial features of the scene data containing calibration elements at each moment within a preset time period, and determine the correspondence between the image spatial features and the vehicle position. The classification unit is used to normalize the image spatial features of the same category according to the preset data category to form a feature sequence subset, and the feature sequence subsets of multiple categories constitute a feature sequence set; The test training unit is used to take into input the vehicle position at a certain moment and the scene data containing calibration elements, determine the image spatial features corresponding to the scene data based on the correspondence, and find the first feature sequence subset where the image spatial features are located; The training target unit is used to find the target image spatial features in the first feature sequence subset at the next moment with the expectation of achieving a preset prediction accuracy, and to determine the vehicle position corresponding to the target image spatial features and the scene data containing the calibration elements. The training is completed when the expectation is achieved. The training module also includes: A receiving submodule is used to receive the scene data at the current location; The feature extraction submodule is used to extract the image spatial features of the scene data and determine a second feature sequence subset of the image spatial features; The result prediction submodule is used to determine the time of the current position based on the second feature sequence subset, predict the vehicle position and scene data containing calibration elements at the next time moment, and output the scene prediction result. The virtual scene construction module includes: The 3D model submodule is used to build a 3D model of the vehicle and calibration elements in the scene prediction results at the current location; The element position submodule is used to determine the element position of the calibrated element based on the image spatial features corresponding to the scene prediction result. The virtual location submodule is used to determine the virtual vehicle location of the vehicle 3D model based on the vehicle location predicted by the scene using virtual reality technology, and to determine the virtual element location of the 3D model of the calibration element based on the element location of the calibration element. The virtual scene submodule is used to determine the virtual scene of the current location based on the virtual location of the vehicle and the virtual location of the element.

2. The system according to claim 1, characterized in that, Also includes: A display for projecting the virtual scene into the user's field of vision.

3. The system according to claim 1, characterized in that, Also includes: Multiple sensors, at least one audio device; The sensor is used to track the user's head movement and determine the range of the user's head movement. The audio device is used to collect sound information within a preset range of the vehicle and output the sound information as audio information of the corresponding virtual scene within the range of the user's head movement.

4. A method for adjusting a virtual scene of vehicle dynamics, characterized in that, include: Collect scene data of vehicles moving at the current location and calibrate various elements in the scene from multiple perspectives; The scene data of the vehicle's current location is input into a pre-trained deep learning model to predict scene elements and obtain scene prediction results. Virtual reality technology is used to construct a virtual scene corresponding to the scene prediction results of the current location; When the scene prediction result for the current location is updated, the corresponding virtual scene is also dynamically adjusted at the same time; Among them, the deep learning model is obtained by learning the correspondence between vehicle position and historical scene data containing calibration elements, and then undergoing regression training. The training of the deep learning model includes: Learn the historical scene data, extract the image spatial features of the scene data containing calibration elements at each moment within a preset time period, and determine the correspondence between the image spatial features and the vehicle position; According to the preset data category, the image spatial features of the same category are normalized to form a feature sequence subset, and the feature sequence subsets of multiple categories constitute a feature sequence set; Input the vehicle position at a certain moment and the scene data containing calibration elements, determine the image spatial features corresponding to the scene data based on the correspondence, and find the first feature sequence subset where the image spatial features are located; With the goal of achieving a preset prediction accuracy, the target image spatial features for the next moment are searched in the first feature sequence subset, and the vehicle position corresponding to the target image spatial features and the scene data containing the calibration elements are determined. Training is completed when the goal is achieved. The step of inputting scene data of the vehicle's current position into a pre-trained deep learning model to predict scene elements and obtain scene prediction results includes: Receive the scene data at the current location; Extract the image spatial features of the scene data and determine a second feature sequence subset of the image spatial features; The time of the current position is determined based on the second feature sequence subset, and the vehicle position and scene data containing calibration elements are predicted at the next time step, and the scene prediction result is output. The construction of a virtual scene corresponding to the scene prediction result at the current location using virtual reality technology includes: Construct a 3D model of the vehicle and calibration elements in the scene prediction results for the current location; Based on the image spatial features corresponding to the scene prediction results, the element positions of the calibrated elements are determined; The virtual position of the vehicle in the three-dimensional model is determined by using virtual reality technology based on the vehicle position of the scene prediction result, and the virtual position of the element in the three-dimensional model of the calibration element is determined based on the element position of the calibration element. A virtual scene is determined based on the virtual location of the vehicle and the virtual location of the element to determine the current location.

Images