A method, device, and medium for generating a vehicle virtual model based on a large model

Abstract

The present application discloses a method, device, and medium for generating a vehicle virtual model based on a large model, which relates to the technical field of image recognition. The method includes: collecting surrounding vehicle images and environmental parameters within the observation range; generating a surrounding virtual scene of the vehicle according to the environmental parameters and identifying feature points in the surrounding vehicle images; determining the minimum reconstruction ratio corresponding to the surrounding vehicles according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles; performing three-dimensional reconstruction on the surrounding vehicles to obtain a virtual unit model corresponding to the surrounding vehicles; determining whether the virtual unit model meets the minimum reconstruction ratio, and if not, matching the virtual unit model with the unit models corresponding to each structure included in a preset vehicle type library, so as to assemble a vehicle virtual model corresponding to the surrounding vehicles according to the matched unit models; and importing the vehicle virtual model into the surrounding virtual scene to realize the restoration of the surrounding vehicle conditions of the vehicle.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and particularly to a method, device, and medium for generating a vehicle virtual model based on a large model. Background Art

[0002] In intelligent driving, the surrounding road conditions can be recognized through the virtual model of the vehicle, which can assist users in achieving safe driving. Most of the current virtual models are imported through manual modeling and cannot generate the virtual models of surrounding vehicles according to the real-time road conditions during the driving process of the vehicle, making it difficult to reflect the current actual vehicle conditions and affecting the vehicle perception effect. Moreover, generating virtual models often consumes a large amount of computing resources, bringing an operating pressure to the vehicle central control system. Summary of the Invention

[0003] To solve the above problems, this application proposes a method for generating a vehicle virtual model based on a large model, including:

[0004] Collect the surrounding vehicle images and environmental parameters within the observation range through the acquisition device set on the vehicle; wherein, the surrounding vehicle images are monocular images of consecutive frames;

[0005] Input the surrounding vehicle images and the environmental parameters into a preset large model, and through the large model, generate the surrounding virtual scene of the vehicle according to the environmental parameters and identify the feature points in the surrounding vehicle images;

[0006] Determine the minimum reconstruction ratio corresponding to the surrounding vehicle according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles;

[0007] Based on the feature points in the surrounding vehicle images, perform three-dimensional reconstruction on the surrounding vehicles to obtain the virtual unit models corresponding to the surrounding vehicles;

[0008] Determine whether the virtual unit model meets the minimum reconstruction ratio. If not, match the virtual unit model with the unit models corresponding to each structure included in the preset vehicle type library, and assemble the vehicle virtual model corresponding to the surrounding vehicle according to the matched unit models;

[0009] Establish a mapping relationship between the surrounding virtual scene and the real scene, and import the vehicle virtual model into the surrounding virtual scene according to the mapping relationship to realize the restoration of the surrounding vehicle conditions of the vehicle.

[0010] In an implementation manner of this application, determining the minimum reconstruction ratio corresponding to the surrounding vehicle according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles specifically includes:

[0011] Identify whether there is a preset driving intention of the vehicle in a preset section ahead through the navigation route of the vehicle. If so, switch the vehicle to the 3D driving mode;

[0012] Based on the 3D driving mode, determine the surrounding vehicles that the vehicle needs to observe and whether the surrounding vehicles and the vehicle are in the coaxial direction;

[0013] If so, determine that the minimum reconstruction ratio corresponding to the surrounding vehicle is the standard reconstruction ratio corresponding to the monocular image;

[0014] If not, through the radar device of the vehicle, determine the deviation direction of the surrounding vehicle relative to the vehicle within a continuous number of observation time periods, and the deviation change rate corresponding to the lateral relative distance between the vehicle and the surrounding vehicle;

[0015] Adjust the standard reconstruction ratio according to the relationships between the deviation direction and the deviation change rate and the driving intention respectively, to obtain the minimum reconstruction ratio corresponding to the surrounding vehicle.

[0016] In an implementation manner of the present application, adjusting the standard reconstruction ratio according to the relationships between the deviation direction and the deviation change rate and the driving intention respectively to obtain the minimum reconstruction ratio corresponding to the surrounding vehicle specifically includes:

[0017] Determine whether the deviation change rate is less than 0, and whether the deviation direction is close to the target driving direction corresponding to the driving intention;

[0018] If so, determine the adjustment coefficient of the standard reconstruction ratio according to the change rate interval where the absolute value of the deviation change rate is located;

[0019] Adjust the standard reconstruction ratio through the adjustment coefficient to obtain the minimum reconstruction ratio corresponding to the surrounding vehicle; wherein, the adjustment coefficient and the absolute value of the deviation change rate are in a positive correlation relationship.

[0020] In an implementation manner of the present application, match the virtual unit model with the unit models corresponding to each structure included in a preset vehicle type library, so as to assemble the vehicle virtual model corresponding to the surrounding vehicle according to the matched unit models, specifically including:

[0021] Determine whether there is a direct feature identifier in the vehicle virtual model; wherein, the direct feature identifier includes at least any one or more of the following: vehicle logo, vehicle identification number, vehicle type identifier;

[0022] When the direct feature identifier exists in the vehicle virtual model, determine whether the direct feature identifier can uniquely identify the vehicle model. If so, according to the direct feature identifier, match the corresponding target vehicle model from the preset vehicle model library, and screen out the unit models for assembling the vehicle virtual model from the unit models of each structure corresponding to the target vehicle model;

[0023] When the direct feature identifier does not exist in the vehicle virtual model, or the direct feature identifier cannot uniquely identify the vehicle model, extract at least part of the appearance features of the virtual unit model, and calculate the similarity between the appearance features included in the unit models corresponding to each vehicle model in the vehicle model library and the at least part of the appearance features;

[0024] According to the similarity, determine the target vehicle model associated with the virtual unit model, and use the unit models corresponding to the target vehicle model as the unit models for assembling the vehicle virtual model.

[0025] In an implementation manner of the present application, the method further includes:

[0026] Determine the structural importance degree corresponding to the unit model according to the structure corresponding to the matched unit model;

[0027] According to the actual perspective relationship between the vehicle and the surrounding vehicles corresponding to the vehicle virtual model, and the connection relationship between the unit model and the virtual unit model, determine the connection importance degree corresponding to the unit model;

[0028] According to the structural importance degree and the connection importance degree, calculate the assembly importance degree corresponding to each unit model for assembling the vehicle virtual model, and sequentially select the unit model and the virtual unit model for assembly according to the assembly importance degree until the vehicle virtual model meets the minimum reconstruction ratio.

[0029] In an implementation manner of the present application, after assembling the vehicle virtual model corresponding to the surrounding vehicles, the method further includes:

[0030] Compare the minimum reconstruction ratio corresponding to the vehicle virtual model with the standard reconstruction ratio to determine the magnitude relationship between the minimum reconstruction ratio and the standard reconstruction ratio;

[0031] When the minimum reconstruction ratio is not less than the standard reconstruction ratio, determine the relative movement speed between the surrounding vehicles according to the relative position relationship between the surrounding vehicles and the vehicle within a continuous observation time period;

[0032] According to the change of the relative motion speed, a running curve corresponding to the surrounding vehicle within the continuous observation time period is fitted, and according to the running curve and the deviation direction, the driving parameters of the surrounding vehicle in the next observation time period are predicted; wherein, the driving parameters include a driving direction and a driving speed;

[0033] According to the driving parameters, the driving trajectory of the surrounding vehicle in the next observation time period table is determined, the driving trajectory is rendered, a driving trajectory model of the surrounding vehicle is generated, and the driving trajectory model is superimposed on the vehicle virtual model.

[0034] In an implementation manner of the present application, identifying the feature points in the surrounding vehicle image specifically includes:

[0035] Performing gray processing on the surrounding vehicle image to obtain a corresponding gray image;

[0036] Determining a sliding window for target detection of the surrounding vehicle image, and traversing each pixel point included in the gray image according to the sliding window;

[0037] During the process of traversing the pixel points, taking the pixel point as the central pixel point within the sliding window, and determining the first gray value corresponding to other pixel points around the central pixel point and the second gray value corresponding to the central pixel point;

[0038] For each other pixel point, determining the difference between the first gray value corresponding to the other pixel point and the second gray value, and the number of other pixel points whose difference is greater than a preset threshold;

[0039] In the case that the number of the other pixel points is greater than a preset value, taking the central pixel point as a candidate feature point in the surrounding vehicle image;

[0040] Traversing any one surrounding vehicle image, and determining the geometric distance between a candidate feature point pair in the any one surrounding vehicle image and a candidate feature point pair in another surrounding vehicle image; wherein, the geometric distance between the candidate feature point pair and the candidate feature point is the smallest;

[0041] Calculating the ratio between the geometric distances respectively corresponding to the candidate feature point pairs, and taking the candidate feature point pairs with the ratio less than a preset threshold as the feature points of the surrounding vehicle image.

[0042] In an implementation manner of the present application, based on the feature points in the surrounding vehicle image, performing three-dimensional reconstruction on the surrounding vehicle to obtain a virtual unit model corresponding to the surrounding vehicle, specifically including:

[0043] Compare any two of the surrounding vehicle images to determine the degree of coincidence of feature points in any two of the surrounding vehicle images;

[0044] Use the two surrounding vehicle images with the largest degree of coincidence as the target surrounding vehicle images for three-dimensional reconstruction of the surrounding vehicle;

[0045] Perform three-dimensional reconstruction on the surrounding vehicle according to the three-dimensional coordinates of each feature point in the target surrounding vehicle image to obtain a virtual unit model corresponding to the surrounding vehicle.

[0046] An embodiment of the present application provides a vehicle virtual model generation device based on a large model, and the device includes:

[0047] At least one processor;

[0048] And a memory communicatively connected to the at least one processor;

[0049] Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for generating a vehicle virtual model based on a large model as described in any one of the above.

[0050] An embodiment of the present application provides a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are set as:

[0051] A method for generating a vehicle virtual model based on a large model as described in any one of the above.

[0052] A method for generating a vehicle virtual model based on a large model proposed by the present application can bring the following beneficial effects:

[0053] By using the acquisition device provided on the vehicle to collect surrounding vehicle images and environmental parameters in real time, road changes can be captured immediately, thereby generating a vehicle virtual model consistent with the current actual vehicle condition. The vehicle virtual model reflects the current road condition in real time during vehicle driving, improving the accuracy and real-time performance of vehicle perception. Determine the minimum reconstruction ratio of the surrounding vehicle according to the driving intention of the vehicle and the relative position relationship, and restore some vehicle features according to the minimum reconstruction ratio. While maintaining the necessary detailed features of the vehicle, it can also reduce unnecessary calculation amounts, significantly reduce the consumption of computing resources, and relieve the operating pressure of the vehicle central control system. Description of the Drawings

[0054] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0055] Figure 1 A schematic flowchart of a method for generating a vehicle virtual model based on a large model provided by an embodiment of the present application;

[0056] Figure 2 A schematic structural diagram of a device for generating a vehicle virtual model based on a large model provided by an embodiment of the present application. Detailed implementation manners

[0057] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0058] The technical solutions provided in each embodiment of the present application will be described in detail below with reference to the drawings.

[0059] As Figure 1 shown, a method for generating a vehicle virtual model based on a large model provided by an embodiment of the present application includes:

[0060] S101: Collect surrounding vehicle images and environmental parameters within the observation range through a collection device provided on the vehicle; among them, the surrounding vehicle images are monocular images of consecutive frames.

[0061] High-precision cameras, lidars and other sensors are usually equipped on the vehicle as collection devices for capturing images and environmental parameters around the vehicle in real time. Among them, the environmental parameters include information such as weather, light, and road facilities. Visual sensors such as cameras are usually installed at the front, rear, and sides of the vehicle. Currently, due to cost reasons, most vehicles are equipped with monocular cameras, that is, the camera viewing angle of each camera is single and fixed, and the observation range is limited. For ordinary vehicles, there are limitations in vehicle height and it is impossible to capture the top images of vehicles that are too tall. Therefore, it is difficult for the vehicle plane images collected only by cameras to reflect the actual vehicle conditions around the vehicle.

[0062] Based on this, instead of using the traditional camera observation method to understand the surrounding vehicle conditions in the embodiments of this application, a virtual model of the surrounding vehicles is constructed, and the virtual models of the current vehicle and its surrounding vehicles are restored to the center console. In this way, the driver can timely understand the vehicle conditions around during the vehicle driving process to assist in safe driving. First, the surrounding vehicle images and environmental parameters within the observation range are collected through the collection devices arranged on the vehicle, and then the real-time road conditions of the vehicle are restored through the monocular images and environmental parameters of the surrounding vehicles. The embodiments of this application are particularly applicable to situations with vehicle confluences such as narrow lanes and turns, where the vehicle spacing is too small. At this time, the center console switches to the 3D driving mode, which can help users better understand the vehicle conditions around and improve the driving safety of the vehicle. It should be noted that the surrounding vehicle images are monocular images of consecutive frames. For the surrounding vehicles on the side of the vehicle, since most vehicles are equipped with at least one camera, in the case where the vehicle is equipped with multiple cameras, the surrounding vehicle images collected by the cameras with overlapping multiple perspectives can be stitched, and the surrounding vehicles can be reconstructed through the stitched surrounding vehicle images.

[0063] S102: Input the surrounding vehicle images and environmental parameters into a preset large model. Through the large model, generate the surrounding virtual scene of the vehicle according to the environmental parameters, and identify the feature points in the surrounding vehicle images.

[0064] Preprocess the collected surrounding vehicle images and environmental parameters, and input the preprocessed surrounding vehicle images and environmental parameters into the large model. Through the large model, the surrounding virtual scene of the vehicle can be generated according to the environmental parameters. While generating the virtual scene, the large model will also perform feature point recognition on the input surrounding vehicle images. The feature points are used for reconstructing the virtual models of the surrounding vehicles. Specifically, first encode the environmental parameters through CLIP to map them to the same feature space, and then based on the DALL·E model, generate the surrounding virtual scene of the vehicle according to the encoded environmental parameters. The surrounding virtual scene includes 3D reconstructions of static elements such as roads and traffic signs and environmental factors such as weather and lighting.

[0065] The feature points in the surrounding vehicle images refer to the positions with obvious geometric features, such as the intersections or turning points of key parts such as the vehicle head, windows, headlights, and doors. When detecting such feature points, they can be identified according to the pixel value differences around each pixel point in the image.

[0066] Specifically, first perform grayscale processing on the surrounding vehicle images to obtain the corresponding grayscale images. Then, set a sliding window for object detection of the surrounding vehicle images. Generally, the sliding window can be selected 、 Windows of odd sizes such as this can ensure that the central pixel is more clearly defined. Move the sliding window on the grayscale image according to a preset step size to traverse each pixel point in the grayscale image. During the process of traversing pixel points, it is necessary to ensure that there is a pixel point in the grayscale image that is the central pixel point within the current sliding window each time it slides. For each pixel point within the sliding window, determine the first grayscale value corresponding to the other pixel points around the central pixel point within the sliding window and the second grayscale value corresponding to the central pixel point, and calculate the difference between each pixel point and the central pixel point, that is, calculate the difference between the first grayscale value and the second grayscale value. Set a preset threshold, count the number of other pixel points within the sliding window whose differences are greater than this threshold. If the number of other pixel points within the sliding window is greater than the preset value, then the central pixel point within this sliding window is considered a candidate feature point.

[0067] The above process identifies the feature points of each frame of the surrounding vehicle images. However, since the vehicle posture is in a real-time changing state, there may be a situation of partial feature loss in the obtained consecutive frame images. Therefore, after determining the feature points included in each surrounding vehicle image, in order to improve the recognition accuracy of the feature points and reduce the probability of false matching, it is also necessary to further select candidate feature points by means of multiple surrounding vehicle images (usually two) to determine whether there is an incorrect matching situation for the identified candidate feature points.

[0068] Specifically, traverse any one of the surrounding vehicle images and determine the geometric distance between the candidate feature points in any one of the surrounding vehicle images and the candidate feature point pairs in another surrounding vehicle image. Among them, the candidate feature point pair is composed of two candidate feature points in another surrounding vehicle image with the smallest geometric distance from the candidate feature point. After obtaining the above geometric distance, calculate the ratio between the geometric distances corresponding to the candidate feature point pairs respectively. This ratio refers to the ratio of the smaller distance to the larger distance among the geometric distances. This ratio is used to evaluate the geometric matching degree of the candidate feature points in the two surrounding vehicle images. The smaller the ratio, the more consistent the relative positions of the candidate feature points in the two surrounding vehicle images, and the more reliable the finally selected feature points. Therefore, set a preset threshold as the critical value for evaluating the reliability of the candidate feature point pairs. When the ratio is less than the preset threshold, the candidate feature point pairs have a high matching quality, and these two candidate feature points can be used as the feature points of the surrounding vehicle images. Repeat the above process to screen out a large number of feature points as the reference points for reconstructing the vehicle virtual model.

[0069] S103: Determine the minimum reconstruction ratio corresponding to the surrounding vehicle according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles.

[0070] Since the surrounding road conditions are complex during vehicle driving, the number of vehicle virtual models to be reconstructed is large, and the computing resources required to reconstruct the virtual model of the entire vehicle are large. However, under certain vehicle driving conditions, only some vehicle virtual models can assist the user in driving. Therefore, in order to reduce the computing resources used, it is possible to predict which vehicle conditions are critical and which are secondary for the current vehicle based on the driving intention of the vehicle and the relative position relationship between the vehicle and surrounding vehicles. When reconstructing the vehicle virtual model, priority is given to highly reconstructing the vehicles that have a greater impact on the safety of the vehicle itself. Based on this, it is necessary to determine the minimum reconstruction ratio required according to the actual situation of the vehicle. The minimum reconstruction ratio refers to the ratio of the reconstructed vehicle virtual model to the entire vehicle model. It should be noted that the minimum reconstruction ratio needs to ensure that the reconstructed vehicle virtual model can reflect the actual surrounding road conditions of the vehicle.

[0071] In one embodiment, based on the navigation route of the vehicle, the driving intention of the vehicle in a preset section ahead is identified. For example, if there is an intersection ahead and the navigation route shows a right turn, then at this time the vehicle can be identified as having a turning driving intention. The above driving intention refers to whether the vehicle has the intention of changing lanes, turning, or merging into a narrow lane. Among them, the driving intention of changing lanes can be predicted by a preset support vector machine model with the help of the navigation route and the historical driving behavior of the vehicle.

[0072] After identifying that the vehicle has a driving intention of turning or confluence, the vehicle spacing will show a shrinking trend. To improve the driving safety of the vehicle, at this time, it is necessary to switch the driving mode from the planar mode to the 3D driving mode. By restoring the 3D vehicle virtual models of surrounding vehicles on the center console, the user can more comprehensively master the surrounding vehicle conditions and timely adjust the driving strategy.

[0073] The minimum reconstruction ratio of the vehicle virtual model is determined according to the relative position relationship between the surrounding vehicles and the vehicle. Specifically, first, it is necessary to determine the surrounding vehicles that the vehicle needs to observe and whether the surrounding vehicles and the vehicle are in the coaxial direction. The coaxial direction refers to whether the vehicles are in the same straight driving direction, that is, directly in front or directly behind. If the surrounding vehicles and the vehicle are in the coaxial direction, then only the front and rear of the vehicle in front or behind need to be concerned. At this time, the minimum reconstruction ratio of the surrounding vehicle is the standard reconstruction ratio corresponding to the monocular image. Here, the standard reconstruction ratio is a fixed value, generally taking 1 / 2 or 1 / 4, that is, the finally constructed vehicle virtual model is half or one-fourth of the actual volume of the entire vehicle.

[0074] If the surrounding vehicle is not in the coaxial direction with the vehicle, and at this time the surrounding vehicle is located on the side of the vehicle, it is necessary to use the radar equipment of the vehicle to determine the deviation direction of the surrounding vehicle relative to the surrounding vehicle within a continuous number of observation time periods, and the deviation change rate corresponding to the lateral relative distance between the vehicle and the surrounding vehicle. The deviation change rate is used to characterize the speed at which the lateral relative distance between the surrounding vehicle and the vehicle changes over time, and is calculated based on the ratio between the deviation amount of the lateral relative distance and the duration of the observation time period. After obtaining the deviation direction and the deviation change rate, the standard reconstruction ratio is adjusted according to the relationship between the deviation direction and the deviation change rate and the driving intention respectively, to obtain the minimum reconstruction ratio corresponding to the surrounding vehicle. If it is identified according to the deviation direction and the deviation change rate that there is a relationship of common lane change, common turning, common merging into a narrow lane between the surrounding vehicle and the vehicle, or the surrounding vehicle approaching the vehicle, it indicates that there is a conflict in the driving routes between the surrounding vehicle and the vehicle. At this time, the minimum reconstruction ratio of the surrounding vehicle can be appropriately increased.

[0075] Specifically, it is determined whether the deviation change rate is less than 0, and whether the deviation direction is close to the target driving direction corresponding to the driving intention. When the deviation change rate is less than 0 and the deviation direction is close to the target driving direction corresponding to the vehicle's driving intention, it indicates that the surrounding vehicle is gradually approaching the vehicle. At this time, the collision risk between the vehicles gradually increases, and it is necessary to adjust the minimum reconstruction ratio of the surrounding vehicle.

[0076] The adjustment of the minimum reconstruction ratio is achieved through an adjustment coefficient. According to the change rate interval where the absolute value of the deviation change rate is located, the adjustment coefficient of the standard reconstruction ratio is determined. After obtaining the adjustment coefficient, the standard reconstruction ratio is enlarged according to the following formula, , where, represents the standard reconstruction ratio, represents the minimum reconstruction ratio, and r represents the adjustment coefficient.

[0077] It should be noted that each change rate interval corresponds to a positive correlation adjustment coefficient. For example, the change rate interval can be divided into [0, 0.2), [0.2, 0.5), , and their corresponding adjustment coefficients are 0.3, 0.6, 1.5 in sequence. The adjustment coefficient has a clear maximum limit value, and it is necessary to ensure that the adjusted minimum reconstruction ratio does not exceed the maximum value of 1. The mapping relationship between the above change rate interval and the adjustment coefficient can be determined by the following distribution function, specifically , Control the maximum adjustment amplitude of each interval. For example, for the change rate interval , when the standard reconstruction ratio is 1 / 2, the maximum adjustment amplitude is 1.5, Control the growth rate, The larger the value, the faster the adjustment coefficient in the low change rate range increases. d represents the deviation change rate. The values of and in the above function are determined through experiments or simulations. In this way, after clarifying the interval where the change rate is located, the corresponding adjustment coefficient can be obtained.

[0078] S104: Based on the feature points in the surrounding vehicle images, perform three-dimensional reconstruction on the surrounding vehicles to obtain the virtual unit models corresponding to the surrounding vehicles.

[0079] After clarifying the minimum reconstruction ratio corresponding to each surrounding vehicle, it is necessary to perform three-dimensional reconstruction on the surrounding vehicles based on the feature points in the above-mentioned recognized surrounding vehicle images. At this time, the reconstructed three-dimensional model is not the whole vehicle model of the vehicle, but a virtual unit model reconstructed based on some feature points from the perspective of a monocular camera, only restoring some features of the vehicle.

[0080] In one embodiment, when performing three-dimensional reconstruction, it depends on the correspondence of the same feature points in different surrounding vehicle images. Therefore, in order to improve the model accuracy, any two surrounding vehicle images are compared to determine the coincidence degree of the feature points in any two surrounding vehicle images. The two surrounding vehicle images with a higher coincidence degree indicate that they are more similar and there are a large number of highly coincident feature points. By using any two surrounding vehicle images with the largest coincidence degree as the target surrounding vehicle images for three-dimensional reconstruction of the surrounding vehicles, it can not only ensure a sufficient number of matching points but also improve the calculation accuracy of the three-dimensional coordinates, thus avoiding model distortion caused by incorrect matching. After screening out the target surrounding vehicle images, according to the feature points and the internal parameters of the acquisition device, the relative pose between the target surrounding vehicle images can be calculated. Through the pixel coordinates and camera pose of the feature points, the three-dimensional coordinates of each feature point can be obtained. In this way, three-dimensional reconstruction of the surrounding vehicles can be achieved through the three-dimensional coordinates, and thus the virtual unit models corresponding to the surrounding vehicles can be obtained.

[0081] S105: Determine whether the virtual unit model meets the minimum reconstruction ratio. If not, match the virtual unit model with the unit models corresponding to each structure included in the preset vehicle type library, so as to assemble the vehicle virtual model corresponding to the surrounding vehicle according to the matched unit models.

[0082] The virtual unit model obtained in S104 above only restores some features of the vehicle, and it is necessary to determine whether the virtual unit model meets the minimum reconstruction ratio of its corresponding vehicle. When calculating the reconstruction ratio corresponding to the virtual unit model, it can be determined according to the size comparison between the current virtual unit model and its corresponding minimum bounding box. If the restored virtual unit model does not meet the minimum reconstruction ratio, it means that the current virtual unit model cannot meet the driving requirements of the vehicle, and it is also necessary to match the virtual unit model with the unit models corresponding to each structure included in the preset vehicle model library, and then assemble the vehicle virtual model corresponding to the surrounding vehicles according to the matched unit models. It should be noted that if the virtual unit model does not include key structures of the vehicle, such as wheels, headlights, doors, roof, etc., model assembly is also required based on the virtual unit model at this time.

[0083] In one embodiment, through object detection of the vehicle virtual model, it is determined whether there are direct feature identifiers in the vehicle virtual model; among them, the direct feature identifiers at least include any one or more of the following: vehicle logo, vehicle identification number, vehicle model identifier. In the case where there are direct feature identifiers in the vehicle virtual model, it is determined whether the direct feature identifiers can uniquely identify the vehicle model. If the direct feature identifier or the combined direct feature identifiers can uniquely determine the vehicle model, the direct feature identifier can be used as an index to match the corresponding target vehicle model from the preset vehicle model library, and the unit models for assembling the vehicle virtual model are screened out from the unit models corresponding to each structure of the target vehicle model. In the case where there are no direct feature identifiers in the vehicle virtual model, or the direct feature identifiers cannot uniquely identify the vehicle model, it is necessary to match the vehicle model through the geometric features of the virtual unit model. First, at least some appearance features in the virtual unit model are extracted. The appearance features can be the length-width-height ratio of the vehicle, the shape of the roof, the shape of the headlights, the style of the tires, etc. Then, the similarity between the appearance features included in the unit models corresponding to each vehicle model in the vehicle model library and at least some appearance features is calculated, and according to the similarity, the target vehicle model associated with the virtual unit model is determined, so that the unit models corresponding to the target vehicle model are used as the unit models for assembling the vehicle virtual model.

[0084] The number of unit models obtained through the above process of matching is multiple, but when assembling the vehicle virtual model, it may not be necessary to use all the unit models, but only some unit models can be used to achieve the reconstruction of the surrounding vehicles. Therefore, it is also necessary to screen out the unit models finally used for assembling the vehicle virtual model from the multiple matched unit models.

[0085] Specifically, according to the structure corresponding to the matched unit model, determine the structural importance corresponding to the unit model. The structural importance can be determined through a preset component weight table. The structural importance of each component is fixed, which reflects the importance of different unit models for vehicle reconstruction. For example, parts such as wheels, the front of the vehicle, the rear of the vehicle, and windows that can reflect specific vehicle characteristics have a higher structural importance, while for decorative parts such as vehicle logos and windshield wipers, the importance reflected in restoring the vehicle is relatively low. By determining the structural importance corresponding to the unit model, it is possible to ensure the priority assembly of core components and avoid the inability to support subsequent vehicle condition restoration due to the lack of key structures.

[0086] Furthermore, according to the actual perspective relationship between the vehicle and the surrounding vehicles corresponding to the vehicle virtual model, as well as the connection relationship between the unit model and the virtual unit model, determine the connection importance corresponding to the unit model. The connection importance reflects the spatial dependence relationship of vehicle components, indicating whether the unit model and the virtual unit model can be directly connected. If not, then assembly defects will occur when the selected unit model is assembled. Therefore, it is necessary to preferentially select unit models that can achieve complete matching assembly and connect them to the current virtual unit model.

[0087] Furthermore, determine the functional weights corresponding to the structural importance and the connection importance according to the complexity of the driving scenario. For example, in congested traffic conditions, the interaction between vehicles is large. At this time, the functional weight of the connection importance can be appropriately increased to ensure that the virtual model can accurately reflect the spatial position relationship and interaction between vehicles. According to the above functional weights, as well as the structural importance and the connection importance, it is possible to calculate the assembly importance corresponding to each unit model for assembling the vehicle virtual model. The assembly importance reflects the assembly priority order of different unit models. Select unit models and virtual unit models for assembly in descending order of assembly importance until the assembled vehicle virtual model meets the minimum reconstruction ratio.

[0088] It should be noted that the vehicle virtual model usually mainly reflects the state of the vehicle itself. In order to help the vehicle understand the dynamic information of surrounding vehicles, it is also necessary to superimpose the driving trajectories of surrounding vehicles on the existing vehicle virtual model to provide more comprehensive road condition perception information for the vehicle to make further analysis and decisions.

[0089] First, compare the minimum reconstruction ratio corresponding to the vehicle virtual model with the standard reconstruction ratio to determine the magnitude relationship between the minimum reconstruction ratio and the standard reconstruction ratio. When the minimum reconstruction ratio is not less than the standard reconstruction ratio, it indicates that the currently reconstructed vehicle virtual model already meets the usability level. At this time, based on the relative position relationship between surrounding vehicles and the vehicle within a continuous observation time period, determine the relative movement speed between the surrounding vehicles. According to the change situation of the relative movement speed, use linear fitting or polynomial fitting to fit the operation curve corresponding to the surrounding vehicles within the continuous observation time period. This operation curve can reflect the change trend of the movement state of the surrounding vehicles. It should be noted that due to the frequent change of vehicle condition information, the continuous observation time period needs to be small enough to capture the relatively continuous operation state of the surrounding vehicles. According to the operation curve and the deviation direction, predict the driving parameters of the surrounding vehicles in the next observation time period; among them, the driving parameters include the driving direction and the driving speed. According to the driving parameters, determine the driving trajectory of the surrounding vehicles in the next observation time period table, and then render the driving trajectory, endow the driving trajectory with a certain color or line style, and finally generate the driving trajectory model of the surrounding vehicles. Superimpose the driving trajectory model on the vehicle virtual model, so that on the basis of the vehicle virtual model, the predicted driving trajectory of the surrounding vehicles in the next observation time period can be intuitively displayed, helping the driver perceive the surrounding vehicle conditions and assisting in making corresponding driving decisions.

[0090] S106: Establish a mapping relationship between the surrounding virtual scene and the real scene. According to the mapping relationship, import the vehicle virtual model into the surrounding virtual scene to realize the restoration of the surrounding vehicle conditions of the vehicle.

[0091] After obtaining the vehicle virtual model, it needs to be imported into the surrounding virtual scene generated in real time according to the large model. When importing, first establish a mapping relationship between the surrounding virtual scene and the real scene. The mapping relationship reflects the coordinate projection relationship between each three-dimensional object in the real scene and the surrounding virtual scene. According to the mapping relationship, the generated vehicle virtual model can be imported into the surrounding virtual scene, and finally the restoration of the surrounding vehicles is realized. Through the restored surrounding vehicle conditions, the user can timely understand the distance and route conflicts between the surrounding vehicles and itself during the driving process, and effectively improve the driving safety in the face of special sections such as narrow lane confluences or turns.

[0092] The above is the method embodiment proposed in this application. Based on the same idea, some embodiments of this application also provide the devices and non-volatile computer storage media corresponding to the above methods.

[0093] Figure 2 The structural schematic diagram of a vehicle virtual model generation device based on a large model provided by the embodiment of this application. As Figure 2As shown, it includes:

[0094] At least one processor; and,

[0095] A memory communicatively connected to the at least one processor; wherein,

[0096] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a method for generating a vehicle virtual model based on a large model as described in any one of the above.

[0097] Embodiments of the present application provide a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are set as:

[0098] A method for generating a vehicle virtual model based on a large model as described in any one of the above.

[0099] The various embodiments in the present application are described in a progressive manner. For the same or similar parts among the various embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for the relevant parts, reference can be made to the partial description of the method embodiments.

[0100] The device and medium provided by the embodiments of the present application correspond one-to-one with the method. Therefore, the device and medium also have beneficial technical effects similar to those of the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be elaborated here.

[0101] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0102] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, as well as the combination of flows and / or blocks in the flowchart and / or block diagram. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate a means for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0103] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction means that implements the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0105] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0106] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory is an example of computer-readable media.

[0107] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0108] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising the element.

[0109] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for generating a vehicle virtual model based on a large model, characterized in that: The method comprises: The surrounding vehicle images and environmental parameters within the observation range are collected by a collection device arranged on the vehicle; wherein the surrounding vehicle images are monocular images of continuous frames; Inputting the surrounding vehicle image and the environmental parameters into a preset large model, generating a virtual scene around the vehicle according to the environmental parameters through the large model, and identifying feature points in the surrounding vehicle image; Determining a minimum reconstruction ratio corresponding to the surrounding vehicles according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles; Based on the feature points in the surrounding vehicle images, the surrounding vehicles are three-dimensionally reconstructed to obtain virtual unit models corresponding to the surrounding vehicles; Determine whether the virtual unit model satisfies the minimum reconstruction ratio, and if not, match the virtual unit model with a unit model corresponding to each structure in a preset vehicle model library, so as to assemble a vehicle virtual model corresponding to the surrounding vehicle according to the matched unit model; Establishing a mapping relationship between the surrounding virtual scene and the real scene, and importing the vehicle virtual model into the surrounding virtual scene according to the mapping relationship to restore the surrounding vehicle conditions of the vehicle; Determining a minimum reconstruction ratio corresponding to the surrounding vehicles according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles, specifically includes: Identify, through the navigation route of the vehicle, whether the vehicle has a preset driving intention in a preset road section ahead, and if so, switch the vehicle to a 3D driving mode; Based on the 3D driving mode, determining surrounding vehicles that the vehicle needs to observe and whether the surrounding vehicles and the vehicle are located in a coaxial direction; If yes, determining that the minimum reconstruction ratio corresponding to the surrounding vehicles is the standard reconstruction ratio corresponding to the monocular image; If not, determining, by means of a radar device of the vehicle, the deviation direction of the surrounding vehicles relative to the vehicle and the deviation change rate corresponding to the lateral relative distance between the vehicle and the surrounding vehicles in a number of consecutive observation time periods; According to the relationship between the deviation direction and the deviation change rate and the driving intention, the standard reconstruction ratio is adjusted to obtain a minimum reconstruction ratio corresponding to the surrounding vehicles.

2. The method for generating a vehicle virtual model based on a large model according to claim 1, characterized in that: According to the relationship between the deviation direction and the deviation change rate and the driving intention, the standard reconstruction ratio is adjusted to obtain the minimum reconstruction ratio corresponding to the surrounding vehicles, specifically including: determining whether the deviation change rate is less than 0, and whether the deviation direction is close to the target driving direction corresponding to the driving intention; If so, determining an adjustment coefficient of the standard reconstruction ratio according to a change rate interval in which the absolute value of the deviation change rate lies; The standard reconstruction ratio is adjusted by the adjustment coefficient to obtain the minimum reconstruction ratio corresponding to the surrounding vehicles; wherein the adjustment coefficient and the absolute value of the deviation change rate are positively correlated.

3. The method for generating a vehicle virtual model based on a large model according to claim 1, characterized in that: The virtual unit model is matched with the unit models corresponding to the structures in the preset vehicle model library, so as to assemble the vehicle virtual model corresponding to the surrounding vehicle according to the matched unit model, specifically including: Determine whether there is a direct feature identifier in the vehicle virtual model; wherein the direct feature identifier includes at least one or more of the following: a vehicle logo, a vehicle identification number, and a vehicle model identifier; In the case where the direct feature identifier exists in the vehicle virtual model, determining whether the direct feature identifier can uniquely identify the vehicle model, and if so, matching the corresponding target vehicle model from a preset vehicle model library according to the direct feature identifier, and selecting the unit model for assembling the vehicle virtual model from the unit models of each structure corresponding to the target vehicle model; When the direct feature identifier does not exist in the vehicle virtual model, or the direct feature identifier cannot uniquely identify the vehicle model, extract at least part of the appearance features in the virtual unit model, and calculate the similarity between the appearance features contained in the unit model corresponding to each vehicle model in the vehicle model library and the at least part of the appearance features; According to the similarity, the target vehicle model associated with the virtual unit model is determined, and the unit model corresponding to the target vehicle model is used as the unit model for assembling the vehicle virtual model.

4. The method for generating a vehicle virtual model based on a large model according to claim 3, characterized in that: The method further comprises: According to the structure corresponding to the matched unit model, determining the importance of the structure corresponding to the unit model; Determining the connection importance corresponding to the unit model according to the actual viewing angle relationship between the vehicle and surrounding vehicles corresponding to the vehicle virtual model, and the connection relationship between the unit model and the virtual unit model; According to the structural importance and the connection importance, the assembly importance corresponding to each unit model used to assemble the vehicle virtual model is calculated, and according to the assembly importance, the unit models and the virtual unit models are selected in turn for assembly to obtain the assembled vehicle virtual model, until the vehicle virtual model meets the minimum reconstruction ratio.

5. The method for generating a vehicle virtual model based on a large model according to claim 1, characterized in that: After assembling the vehicle virtual models corresponding to the surrounding vehicles, the method further includes: Comparing the minimum reconstruction ratio corresponding to the vehicle virtual model with the standard reconstruction ratio to determine the size relationship between the minimum reconstruction ratio and the standard reconstruction ratio; When the minimum reconstruction ratio is not less than the standard reconstruction ratio, determining the relative movement speed between the surrounding vehicles according to the relative position relationship between the surrounding vehicles and the vehicle during a continuous observation period; According to the change of the relative motion speed, the operation curve corresponding to the surrounding vehicles in the continuous observation time period is fitted, and according to the operation curve and the deviation direction, the driving parameters of the surrounding vehicles in the next observation time period are predicted; wherein the driving parameters include the driving direction and the driving speed; According to the driving parameters, the driving trajectory of the surrounding vehicles in the next observation time period is determined, the driving trajectory is rendered, a driving trajectory model of the surrounding vehicles is generated, and the driving trajectory model is superimposed on the vehicle virtual model.

6. The method for generating a vehicle virtual model based on a large model according to claim 1, characterized in that: Identifying feature points in the surrounding vehicle image specifically includes: Performing grayscale processing on the surrounding vehicle image to obtain a corresponding grayscale image; Determine a sliding window for performing target detection on the surrounding vehicle image, and traverse each pixel point contained in the grayscale image according to the sliding window; In the process of traversing the pixel point, the pixel point is taken as the central pixel point in the sliding window, and the first grayscale value corresponding to other pixel points around the central pixel point and the second grayscale value corresponding to the central pixel point are determined; For each other pixel point, determine the difference between the first grayscale value and the second grayscale value corresponding to the other pixel point, and the number of other pixel points whose difference is greater than a preset threshold; When the number of the other pixel points is greater than a preset value, the central pixel point is used as a candidate feature point in the surrounding vehicle image; Traversing any surrounding vehicle image, determining the geometric distance between the candidate feature point in any surrounding vehicle image and a candidate feature point pair in another surrounding vehicle image; wherein the geometric distance between the candidate feature point pair and the candidate feature point is the smallest; The ratios of the geometric distances respectively corresponding to the candidate feature point pairs are calculated, and the candidate feature point pairs whose ratios are less than a preset threshold are used as feature points of the surrounding vehicle image.

7. The method for generating a vehicle virtual model based on a large model according to claim 6, characterized in that: Based on the feature points in the surrounding vehicle images, the surrounding vehicles are three-dimensionally reconstructed to obtain virtual unit models corresponding to the surrounding vehicles, specifically including: Comparing the two surrounding vehicle images to determine the overlap of feature points in the two surrounding vehicle images; using the arbitrary two surrounding vehicle images with the largest overlap as target surrounding vehicle images for performing three-dimensional reconstruction of the surrounding vehicles; According to the three-dimensional coordinates of each feature point in the target surrounding vehicle image, the surrounding vehicles are three-dimensionally reconstructed to obtain a virtual unit model corresponding to the surrounding vehicles.

8. A vehicle virtual model generation device based on a large model, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a vehicle virtual model generation method based on a large model as described in any one of claims 1-7.

9. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to: A method for generating a vehicle virtual model based on a large model as described in any one of claims 1 to 7.

Images