Vehicle virtual model generation method and device based on large model and medium

Abstract

The invention discloses a vehicle virtual model generation method and device based on a large model, and a medium, and relates to the technical field of image recognition. The method comprises the following steps: acquiring surrounding vehicle images and environmental parameters in an observation range; generating a peripheral virtual scene of the vehicle according to the environmental parameters, and identifying feature points in the peripheral vehicle image; according to the driving intention of the vehicle and the relative position relation between the vehicle and the surrounding vehicles, the minimum reconstruction proportion corresponding to the surrounding vehicles is determined; performing three-dimensional reconstruction on the surrounding vehicles to obtain virtual unit models corresponding to the surrounding vehicles; determining whether the virtual unit model meets the minimum reconstruction proportion, if not, matching the virtual unit model with unit models corresponding to all structures contained in a preset vehicle model library, and assembling according to the matched unit models to obtain vehicle virtual models corresponding to surrounding vehicles; and importing the vehicle virtual model into the peripheral virtual scene to restore the peripheral vehicle condition of the vehicle.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method, device and medium for generating a vehicle virtual model based on a large model. Background Art

[0002] In intelligent driving, the virtual model of the car can be used to identify the surrounding road conditions to assist users in achieving safe driving. Most of the current virtual models are manually modeled and imported, and it is impossible to generate virtual models of surrounding vehicles based on real-time road conditions during the driving process of the vehicle. It is difficult to reflect the current actual vehicle conditions, which affects the vehicle perception effect. In addition, the generation of virtual models often consumes a lot of computing resources, which brings operating pressure to the vehicle's central control system. Summary of the invention

[0003] In order to solve the above problems, the present application proposes a method for generating a vehicle virtual model based on a large model, comprising: The surrounding vehicle images and environmental parameters within the observation range are collected by a collection device arranged on the vehicle; wherein the surrounding vehicle images are monocular images of continuous frames; Inputting the surrounding vehicle image and the environmental parameters into a preset large model, generating a virtual scene around the vehicle according to the environmental parameters through the large model, and identifying feature points in the surrounding vehicle image; Determining a minimum reconstruction ratio corresponding to the surrounding vehicles according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles; Based on the feature points in the surrounding vehicle images, the surrounding vehicles are three-dimensionally reconstructed to obtain virtual unit models corresponding to the surrounding vehicles; Determine whether the virtual unit model satisfies the minimum reconstruction ratio, and if not, match the virtual unit model with a unit model corresponding to each structure from a preset vehicle model library, so as to assemble a vehicle virtual model corresponding to the surrounding vehicle according to the matched unit model; A mapping relationship between the surrounding virtual scene and the real scene is established, and according to the mapping relationship, the vehicle virtual model is imported into the surrounding virtual scene to restore the surrounding vehicle conditions of the vehicle.

[0004] In one implementation of the present application, determining the minimum reconstruction ratio corresponding to the surrounding vehicles according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles specifically includes: Identify, through the navigation route of the vehicle, whether the vehicle has a preset driving intention in a preset road section ahead, and if so, switch the vehicle to a 3D driving mode; Based on the 3D driving mode, determining surrounding vehicles that the vehicle needs to observe and whether the surrounding vehicles and the vehicle are located in a coaxial direction; If yes, determining that the minimum reconstruction ratio corresponding to the surrounding vehicles is the standard reconstruction ratio corresponding to the monocular image; If not, determining, by means of a radar device of the vehicle, the deviation direction of the surrounding vehicles relative to the vehicle and the deviation change rate corresponding to the lateral relative distance between the vehicle and the surrounding vehicles in a number of consecutive observation time periods; According to the relationship between the deviation direction and the deviation change rate and the driving intention, the standard reconstruction ratio is adjusted to obtain a minimum reconstruction ratio corresponding to the surrounding vehicles.

[0005] In one implementation of the present application, the standard reconstruction ratio is adjusted according to the relationship between the deviation direction and the deviation change rate and the driving intention, so as to obtain the minimum reconstruction ratio corresponding to the surrounding vehicles, specifically including: determining whether the deviation change rate is less than 0, and whether the deviation direction is close to the target driving direction corresponding to the driving intention; If so, determining an adjustment coefficient of the standard reconstruction ratio according to a change rate interval in which the absolute value of the deviation change rate lies; The standard reconstruction ratio is adjusted by the adjustment coefficient to obtain the minimum reconstruction ratio corresponding to the surrounding vehicles; wherein the adjustment coefficient and the absolute value of the deviation change rate are positively correlated.

[0006] In one implementation of the present application, the virtual unit model is matched with a unit model corresponding to each structure in a preset vehicle model library, so as to assemble a vehicle virtual model corresponding to the surrounding vehicle according to the matched unit model, specifically including: Determine whether there is a direct feature identifier in the vehicle virtual model; wherein the direct feature identifier includes at least one or more of the following: a vehicle logo, a vehicle identification number, and a vehicle model identifier; In the case where the direct feature identifier exists in the vehicle virtual model, determining whether the direct feature identifier can uniquely identify the vehicle model, and if so, matching the corresponding target vehicle model from a preset vehicle model library according to the direct feature identifier, and selecting the unit model for assembling the vehicle virtual model from the unit models of each structure corresponding to the target vehicle model; When the direct feature identifier does not exist in the vehicle virtual model, or the direct feature identifier cannot uniquely identify the vehicle model, extract at least part of the appearance features in the virtual unit model, and calculate the similarity between the appearance features contained in the unit model corresponding to each vehicle model in the vehicle model library and the at least part of the appearance features; According to the similarity, the target vehicle model associated with the virtual unit model is determined, and the unit model corresponding to the target vehicle model is used as the unit model for assembling the vehicle virtual model.

[0007] In one implementation of the present application, the method further includes: According to the structure corresponding to the matched unit model, determining the structural importance corresponding to the unit model; Determining the connection importance corresponding to the unit model according to the actual viewing angle relationship between the vehicle and surrounding vehicles corresponding to the vehicle virtual model, and the connection relationship between the unit model and the virtual unit model; According to the structural importance and the connection importance, the assembly importance corresponding to each unit model used to assemble the vehicle virtual model is calculated, and according to the assembly importance, the unit models and the virtual unit models are selected in turn for assembly to obtain the assembled vehicle virtual model, until the vehicle virtual model meets the minimum reconstruction ratio.

[0008] In one implementation of the present application, after assembling the vehicle virtual models corresponding to the surrounding vehicles, the method further includes: Comparing the minimum reconstruction ratio corresponding to the vehicle virtual model with the standard reconstruction ratio to determine the size relationship between the minimum reconstruction ratio and the standard reconstruction ratio; When the minimum reconstruction ratio is not less than the standard reconstruction ratio, determining the relative movement speed between the surrounding vehicles according to the relative position relationship between the surrounding vehicles and the vehicle during a continuous observation period; According to the change of the relative motion speed, the operation curve corresponding to the surrounding vehicles in the continuous observation time period is fitted, and according to the operation curve and the deviation direction, the driving parameters of the surrounding vehicles in the next observation time period are predicted; wherein the driving parameters include the driving direction and the driving speed; According to the driving parameters, the driving trajectory of the surrounding vehicles in the next observation time period table is determined, the driving trajectory is rendered, a driving trajectory model of the surrounding vehicles is generated, and the driving trajectory model is superimposed on the vehicle virtual model.

[0009] In one implementation of the present application, identifying feature points in the surrounding vehicle image specifically includes: Performing grayscale processing on the surrounding vehicle image to obtain a corresponding grayscale image; Determine a sliding window for performing target detection on the surrounding vehicle image, and traverse each pixel point contained in the grayscale image according to the sliding window; In the process of traversing the pixel point, the pixel point is taken as the central pixel point in the sliding window, and the first grayscale value corresponding to other pixel points around the central pixel point and the second grayscale value corresponding to the central pixel point are determined; For each other pixel point, determine the difference between the first grayscale value and the second grayscale value corresponding to the other pixel point, and the number of other pixel points whose difference is greater than a preset threshold; When the number of the other pixel points is greater than a preset value, the central pixel point is used as a candidate feature point in the surrounding vehicle image; Traversing any surrounding vehicle image, determining the geometric distance between the candidate feature point in any surrounding vehicle image and a candidate feature point pair in another surrounding vehicle image; wherein the geometric distance between the candidate feature point pair and the candidate feature point is the smallest; The ratios of the geometric distances respectively corresponding to the candidate feature point pairs are calculated, and the candidate feature point pairs whose ratios are less than a preset threshold are used as feature points of the surrounding vehicle image.

[0010] In one implementation of the present application, based on the feature points in the surrounding vehicle image, the surrounding vehicles are three-dimensionally reconstructed to obtain a virtual unit model corresponding to the surrounding vehicles, which specifically includes: Comparing the two surrounding vehicle images to determine the overlap of feature points in the two surrounding vehicle images; using the arbitrary two surrounding vehicle images with the largest overlap as target surrounding vehicle images for performing three-dimensional reconstruction of the surrounding vehicles; According to the three-dimensional coordinates of each feature point in the target surrounding vehicle image, the surrounding vehicles are three-dimensionally reconstructed to obtain a virtual unit model corresponding to the surrounding vehicles.

[0011] The embodiment of the present application provides a vehicle virtual model generation device based on a large model, the device comprising: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for generating a vehicle virtual model based on a large model as described in any one of the above items.

[0012] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows: A method for generating a vehicle virtual model based on a large model as described in any of the above items.

[0013] The vehicle virtual model generation method based on a large model proposed in this application can bring the following beneficial effects: The acquisition device installed on the vehicle collects images of surrounding vehicles and environmental parameters in real time, and can instantly capture road changes, thereby generating a vehicle virtual model that is consistent with the current actual vehicle condition. The vehicle virtual model reflects the current road conditions in real time during the vehicle's driving process, improving the accuracy and real-time performance of vehicle perception. The minimum reconstruction ratio of surrounding vehicles is determined based on the vehicle's driving intention and relative position relationship, and some vehicle features are restored based on the minimum reconstruction ratio. While maintaining the necessary detailed features of the vehicle, it can also reduce unnecessary calculations, significantly reduce the consumption of computing resources, and reduce the operating pressure of the vehicle's central control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of a process flow of a method for generating a vehicle virtual model based on a large model provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a vehicle virtual model generation device based on a large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0016] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0017] like Figure 1 As shown, a method for generating a vehicle virtual model based on a large model is provided in an embodiment of the present application, comprising: S101: collecting surrounding vehicle images and environmental parameters within an observation range through a collection device provided on the vehicle; wherein the surrounding vehicle images are monocular images of continuous frames.

[0018] Vehicles are usually equipped with high-precision cameras, lidar and other sensors as acquisition devices to capture images and environmental parameters around the vehicle in real time, including weather, lighting, road facilities and other information. Visual sensors such as cameras are usually installed on the front, rear and sides of the vehicle. At present, due to cost reasons, most vehicles are equipped with monocular cameras, that is, each camera has a single fixed viewing angle and a limited observation range. For ordinary vehicles, there is a limit on their height, and it is impossible to capture the top image of a vehicle that is too high. Therefore, the plane image of the vehicle collected by the camera alone is difficult to reflect the actual vehicle conditions around the vehicle.

[0019] Based on this, the embodiment of the present application no longer uses the traditional camera observation method to understand the surrounding vehicle conditions, but restores the virtual models of the current vehicle and its surrounding vehicles to the center console by constructing a virtual model of the surrounding vehicles, so that the driver can timely understand the surrounding vehicle conditions during the vehicle driving process, and assist him in safe driving. First, the surrounding vehicle images and environmental parameters within the observation range are collected by the acquisition device set on the vehicle, and then the real-time road conditions of the vehicle are restored through the monocular images and environmental parameters of the surrounding vehicles. The embodiment of the present application is particularly suitable for situations where there are vehicle confluences such as narrow lanes and turns. At this time, the distance between vehicles is too small, and the center console switches to 3D driving mode, which can help users better understand the surrounding vehicle conditions and improve the driving safety of the vehicle. It should be noted that the surrounding vehicle images are monocular images of continuous frames. For the surrounding vehicles located on the side of the vehicle, since most vehicles are equipped with at least one camera, when the vehicle is equipped with multiple cameras, the surrounding vehicle images collected by multiple cameras with overlapping perspectives can be spliced, and the surrounding vehicles can be reconstructed through the spliced ​​surrounding vehicle images.

[0020] S102: Input the surrounding vehicle images and environmental parameters into a preset large model, generate a virtual scene around the vehicle according to the environmental parameters through the large model, and identify feature points in the surrounding vehicle images.

[0021] The collected surrounding vehicle images and environmental parameters are preprocessed and input into the large model. Through the large model, a virtual scene around the vehicle can be generated according to the environmental parameters. While generating the virtual scene, the large model also recognizes the feature points of the input surrounding vehicle images, and the feature points are used to reconstruct the virtual model of the surrounding vehicles. Specifically, the environmental parameters are first encoded through CLIP to map them to the same feature space, and then based on the DALL·E model, a virtual scene around the vehicle is generated according to the encoded environmental parameters. The surrounding virtual scene includes 3D reconstruction of static elements such as roads and traffic signs and environmental factors such as weather and lighting.

[0022] Feature points in the surrounding vehicle images refer to locations with obvious geometric features, such as intersections or turning points of key parts such as the front of the vehicle, windows, lights, doors, etc. When detecting such feature points, they can be identified based on the difference in pixel values ​​around each pixel in the image.

[0023] Specifically, the surrounding vehicle images are first processed in grayscale to obtain the corresponding grayscale image. Then, a sliding window is set for target detection in the surrounding vehicle images. In general, the sliding window can be selected from , The windows of odd sizes are equal, so that the central pixel can be more clear. According to the preset step size, the sliding window is moved on the grayscale image to traverse each pixel in the grayscale image. In the process of traversing the pixels, it is necessary to ensure that each time a pixel in the grayscale image is traversed, it is necessary to ensure that there is a pixel in the grayscale image that is the central pixel in the current sliding window. For each pixel in the sliding window, the first grayscale value corresponding to the other pixels around the central pixel in the sliding window and the second grayscale value corresponding to the central pixel are determined, and the difference between each pixel and the central pixel is calculated, that is, the difference between the first grayscale value and the second grayscale value is calculated. A preset threshold is set, and the number of other pixels in the sliding window whose difference is greater than the threshold is counted. If the number of other pixels in the sliding window is greater than the preset value, the central pixel in this sliding window is considered to be a candidate feature point.

[0024] The above process identifies the feature points of each frame of the surrounding vehicle image. However, since the vehicle posture is in a real-time changing state, the continuous frame images obtained may have some features missing. Therefore, after determining the feature points contained in each surrounding vehicle image, in order to improve the recognition accuracy of the feature points and reduce the probability of mismatching, it is necessary to use multiple surrounding vehicle images (generally two) to further select candidate feature points to determine whether the identified candidate feature points have mismatches.

[0025] Specifically, traverse any surrounding vehicle image, and determine the geometric distance between the candidate feature point in any surrounding vehicle image and the candidate feature point pair in another surrounding vehicle image. Among them, the candidate feature point pair is composed of two candidate feature points in another surrounding vehicle image with the smallest geometric distance to the candidate feature point. After obtaining the above geometric distance, calculate the ratio between the geometric distances corresponding to the candidate feature point pairs, which refers to the ratio between the smaller distance and the larger distance in the geometric distance. This ratio is used to evaluate the geometric matching degree of the candidate feature points in the two surrounding vehicle images. The smaller the ratio, the more consistent the relative position of the candidate feature points in the two surrounding vehicle images, and the more reliable the feature points finally screened out. Therefore, a preset threshold is set as the critical value for evaluating the reliability of the candidate feature point pair. When the ratio is less than the preset threshold, the candidate feature point pair has a higher matching quality, and the two candidate feature points can be used as feature points of the surrounding vehicle image. Repeat the above process, and a large number of feature points can be screened out as reference points for reconstructing the vehicle virtual model.

[0026] S103: Determine a minimum reconstruction ratio corresponding to the surrounding vehicles according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles.

[0027] Due to the complex road conditions around the vehicle during driving, a large number of vehicle virtual models need to be reconstructed, and the computing resources required to reconstruct the virtual model of the entire vehicle are large. In some vehicle driving conditions, only part of the vehicle virtual model can assist the user in driving. Therefore, in order to reduce the computing resources used, it is possible to predict which vehicle conditions are critical to the current vehicle and which are secondary based on the vehicle's driving intention and the relative position relationship between the vehicle and the surrounding vehicles. When reconstructing the vehicle virtual model, priority is given to vehicles that have a greater impact on the safety of the vehicle and a high proportion of reconstruction. Based on this, it is necessary to determine the minimum reconstruction ratio required according to the actual situation of the vehicle. The minimum reconstruction ratio refers to the ratio of the reconstructed vehicle virtual model to the entire vehicle model. It should be noted that the minimum reconstruction ratio must ensure that the reconstructed vehicle virtual model can reflect the actual road conditions around the vehicle.

[0028] In one embodiment, based on the navigation route of the vehicle, the driving intention of the vehicle in the preset road section ahead is identified. For example, if there is an intersection ahead and the navigation route shows a right turn, then the vehicle can be identified as having a driving intention to turn. The above driving intention refers to whether the vehicle has the intention to change lanes, turn, or merge into a narrow lane, wherein the driving intention of changing lanes can be predicted by a preset support vector machine model with the help of the navigation route and the historical driving behavior of the vehicle.

[0029] After recognizing that the vehicle has the intention to turn or merge, the distance between vehicles will tend to decrease. In order to improve the driving safety of the vehicle, it is necessary to switch the driving mode from the plane mode to the 3D driving mode. By restoring the 3D vehicle virtual model of the surrounding vehicles on the center console, the user can have a more comprehensive understanding of the surrounding vehicle conditions and adjust the driving strategy in time.

[0030] The minimum reconstruction ratio of the vehicle virtual model is determined based on the relative position relationship between the surrounding vehicles. Specifically, first, it is necessary to determine whether the surrounding vehicles that the vehicle needs to observe and the surrounding vehicles are in the same axis direction. The coaxial direction refers to whether the vehicles are in the same straight driving direction, that is, directly in front or directly behind. If the surrounding vehicles are in the same axis direction as the vehicle, then only the front and rear of the front or rear vehicles need to be paid attention to. At this time, the minimum reconstruction ratio of the surrounding vehicles is the standard reconstruction ratio corresponding to the monocular image. The standard reconstruction ratio here is a fixed value, generally 1 / 2 or 1 / 4, that is, the final constructed vehicle virtual model is half or one-quarter of the actual volume of the vehicle.

[0031] If the surrounding vehicles are not in the same direction as the vehicle, and the surrounding vehicles are located at the lateral position of the vehicle, the vehicle's radar equipment is required to determine the deviation direction of the surrounding vehicles relative to the surrounding vehicles in several consecutive observation time periods, as well as the deviation change rate corresponding to the lateral relative distance between the vehicle and the surrounding vehicles. The deviation change rate is used to characterize the speed at which the lateral relative distance between the surrounding vehicles changes with time, and is calculated based on the ratio between the deviation amount of the lateral relative distance and the length of the observation time period. After obtaining the deviation direction and the deviation change rate, the standard reconstruction ratio is adjusted according to the relationship between the deviation direction and the deviation change rate and the driving intention, and the minimum reconstruction ratio corresponding to the surrounding vehicles is obtained. If the surrounding vehicles and the vehicle are identified to have a common lane change, common turn, common merging into a narrow lane, or the surrounding vehicles are approaching the vehicle based on the deviation direction and the deviation change rate, it means that there is a conflict in the driving route between the surrounding vehicles and the vehicle. At this time, the minimum reconstruction ratio of the surrounding vehicles can be appropriately increased.

[0032] Specifically, it is determined whether the deviation change rate is less than 0 and whether the deviation direction is close to the target driving direction corresponding to the driving intention. When the deviation change rate is less than 0 and the deviation direction is close to the target driving direction corresponding to the vehicle's driving intention, it means that the surrounding vehicles are gradually approaching the vehicle. At this time, the collision risk between vehicles is gradually increasing, and the minimum reconstruction ratio of the surrounding vehicles needs to be adjusted.

[0033] The adjustment of the minimum reconstruction ratio is achieved through the adjustment coefficient. The adjustment coefficient of the standard reconstruction ratio is determined according to the change rate interval where the absolute value of the deviation change rate is located. After obtaining the adjustment coefficient, the standard reconstruction ratio is expanded according to the following formula: ,in, represents the standard reconstruction ratio, represents the minimum reconstruction ratio, and r represents the adjustment coefficient.

[0034] It should be noted that each change rate interval corresponds to a positive correlation adjustment coefficient. For example, the change rate interval can be divided into [0,0.2), [0.2,0.5), , and the corresponding adjustment coefficients are 0.3, 0.6, and 1.5 respectively. The adjustment coefficient has a clear maximum limit value, and the minimum reconstruction ratio after adjustment must not exceed the maximum value 1. The mapping relationship between the above change rate interval and the adjustment coefficient can be expressed as the following distribution function, specifically: , Controls the maximum adjustment range for each interval, for example, for the rate of change interval , when the standard reconstruction ratio is 1 / 2, the maximum adjustment range is 1.5, Control the growth rate, The larger the value, the faster the adjustment coefficient of the low change rate interval increases, and d represents the deviation change rate. Determine the above function through experiments or simulations. and In this way, after clarifying the range of the change rate, the corresponding adjustment coefficient can be obtained.

[0035] S104: Based on the feature points in the surrounding vehicle images, three-dimensionally reconstruct the surrounding vehicles to obtain virtual unit models corresponding to the surrounding vehicles.

[0036] After clarifying the minimum reconstruction ratio corresponding to each surrounding vehicle, it is necessary to perform three-dimensional reconstruction of the surrounding vehicles based on the feature points in the above-mentioned identified surrounding vehicle images. At this time, the reconstructed three-dimensional model is not the entire vehicle model of the vehicle, but a virtual unit model reconstructed based on some feature points from the perspective of the monocular camera, which only restores some features of the vehicle.

[0037] In one embodiment, when performing three-dimensional reconstruction, the correspondence between the same feature points in different surrounding vehicle images is relied upon. Therefore, in order to improve the model accuracy, any two surrounding vehicle images are compared to determine the overlap of the feature points in any two surrounding vehicle images. The higher the overlap between the two surrounding vehicle images, the more similar they are, and there are a large number of highly overlapped feature points. By using any two surrounding vehicle images with the largest overlap as the target surrounding vehicle images for three-dimensional reconstruction of the surrounding vehicles, it is possible to ensure a sufficient number of matching points and improve the calculation accuracy of the three-dimensional coordinates, thereby avoiding model distortion caused by incorrect matching. After screening out the target surrounding vehicle images, the relative posture between the target surrounding vehicle images can be calculated based on the feature points and the internal parameters of the acquisition device. The three-dimensional coordinates of each feature point can be obtained through the pixel coordinates of the feature points and the camera posture. In this way, the three-dimensional reconstruction of the surrounding vehicles can be realized through the three-dimensional coordinates, thereby obtaining the virtual unit model corresponding to the surrounding vehicles.

[0038] S105: Determine whether the virtual unit model meets the minimum reconstruction ratio. If not, match the virtual unit model with unit models corresponding to each structure from a preset vehicle model library, so as to assemble a vehicle virtual model corresponding to the surrounding vehicles according to the matched unit model.

[0039] The virtual unit model obtained in the above S104 only restores some features of the vehicle, and it is necessary to determine whether the virtual unit model meets the minimum reconstruction ratio of the corresponding vehicle. When calculating the reconstruction ratio corresponding to the virtual unit model, it can be determined based on the size comparison between the current virtual unit model and its corresponding minimum bounding box. If the restored virtual unit model does not meet the minimum reconstruction ratio, it means that the current virtual unit model cannot meet the driving needs of the vehicle, and the virtual unit model needs to be matched with the unit model corresponding to each structure from the preset vehicle model library, and then the virtual model of the vehicle corresponding to the surrounding vehicles is assembled according to the matched unit model. It should be noted that if the virtual unit model does not contain the key structures of the vehicle, such as wheels, lights, doors, roof, etc., it is also necessary to assemble the model based on the virtual unit model.

[0040] In one embodiment, by performing target detection on the vehicle virtual model, it is determined whether there is a direct feature identifier in the vehicle virtual model; wherein the direct feature identifier includes at least one or more of the following: vehicle logo, vehicle identification number, and vehicle model identifier. In the case where there is a direct feature identifier in the vehicle virtual model, it is determined whether the direct feature identifier can uniquely identify the vehicle model. If the direct feature identifier or the combined direct feature identifier can uniquely identify the vehicle model, the direct feature identifier can be used as an index to match the corresponding target vehicle model from the preset vehicle model library, and the unit model used to assemble the vehicle virtual model is screened out from the unit models of each structure corresponding to the target vehicle model. In the case where there is no direct feature identifier in the vehicle virtual model, or the direct feature identifier cannot uniquely identify the vehicle model, it is necessary to match the vehicle model through the geometric features of the virtual unit model. First, extract at least part of the appearance features in the virtual unit model, and the appearance features can be the length, width and height ratio of the vehicle, the roof shape, the headlight shape, the tire style, etc. Then, calculate the similarity between the appearance features contained in the unit model corresponding to each vehicle model in the vehicle model library and at least part of the appearance features, and determine the target vehicle model associated with the virtual unit model according to the similarity, so that the unit model corresponding to the target vehicle model is used as the unit model for assembling the vehicle virtual model.

[0041] There are multiple unit models matched through the above process. However, when assembling a vehicle virtual model, it may not be necessary to use all the unit models. Instead, only some of the unit models are needed to reconstruct the surrounding vehicles. Therefore, it is necessary to screen out the unit models that are ultimately used to assemble the vehicle virtual model from the multiple matched unit models.

[0042] Specifically, according to the structure corresponding to the matched unit model, the structural importance of the unit model is determined. The structural importance can be determined by a preset component weight table. The structural importance of each component is fixed, which reflects the importance of different unit models for vehicle reconstruction. For example, the wheels, front, rear, and windows, which can reflect the specific characteristics of the vehicle, have a higher structural importance, while the decorative parts such as the logo and wipers are less important when restoring the vehicle. By determining the structural importance corresponding to the unit model, it is possible to ensure that the core components are assembled first to avoid the model being unable to support subsequent vehicle condition restoration due to the lack of key structures.

[0043] Furthermore, the connection importance corresponding to the unit model is determined based on the actual viewing angle relationship between the vehicle and the surrounding vehicles corresponding to the vehicle virtual model, as well as the connection relationship between the unit model and the virtual unit model. The connection importance reflects the spatial dependency of vehicle components, which indicates whether the unit model and the virtual unit model can be directly connected. If not, the selected unit model will produce assembly defects during assembly. Therefore, it is necessary to give priority to selecting a unit model that can achieve complete matching assembly and connect it to the current virtual unit model.

[0044] Furthermore, the functional weights corresponding to the structural importance and connection importance are determined according to the complexity of the driving scene. For example, for congested road conditions, the mutual influence between vehicles is large. At this time, the functional weight of the connection importance can be appropriately increased to ensure that the virtual model can accurately reflect the spatial position relationship and interaction between vehicles. According to the above functional weights and the structural importance and connection importance, the assembly importance corresponding to each unit model used to assemble the vehicle virtual model can be calculated. The assembly importance reflects the assembly priority order of different unit models. In the order of assembly importance from large to small, the unit models and virtual unit models are selected in turn for assembly to obtain the assembled vehicle virtual model until the vehicle virtual model meets the minimum reconstruction ratio.

[0045] It should be noted that the vehicle virtual model usually mainly reflects the status of the vehicle itself. In order to help the vehicle understand the dynamic information of surrounding vehicles, it is also necessary to superimpose the driving trajectories of surrounding vehicles on the existing vehicle virtual model to provide more comprehensive road condition perception information for the vehicle to make further analysis and decisions.

[0046] First, the minimum reconstruction ratio corresponding to the vehicle virtual model is compared with the standard reconstruction ratio to determine the size relationship between the minimum reconstruction ratio and the standard reconstruction ratio. When the minimum reconstruction ratio is not less than the standard reconstruction ratio, it means that the currently reconstructed vehicle virtual model has met the availability. At this time, the relative movement speed between the surrounding vehicles needs to be determined based on the relative position relationship between the surrounding vehicles and the vehicles in the continuous observation time period. According to the change of the relative movement speed, the operation curve corresponding to the surrounding vehicles in the continuous observation time period is fitted by linear fitting or polynomial fitting. This operation curve can reflect the change trend of the movement state of the surrounding vehicles. It should be noted that due to the frequent changes in vehicle condition information, the continuous observation time period needs to be small enough to capture the relatively continuous operation state of the surrounding vehicles. According to the operation curve and the deviation direction, the driving parameters of the surrounding vehicles in the next observation time period are predicted; wherein the driving parameters include the driving direction and the driving speed. According to the driving parameters, the driving trajectory of the surrounding vehicles in the next observation time period table is determined, and then the driving trajectory is rendered, and the driving trajectory is given a certain color or line style, and finally the driving trajectory model of the surrounding vehicles is generated. The driving trajectory model is superimposed on the vehicle virtual model. In this way, based on the vehicle virtual model, the predicted driving trajectory of surrounding vehicles in the next observation time period can be intuitively displayed, helping the driver to perceive the surrounding vehicle conditions and assisting him in making corresponding driving decisions.

[0047] S106: Establishing a mapping relationship between the surrounding virtual scene and the real scene, and importing the vehicle virtual model into the surrounding virtual scene according to the mapping relationship to restore the surrounding vehicle conditions of the vehicle.

[0048] After obtaining the vehicle virtual model, it needs to be imported into the surrounding virtual scene generated in real time based on the large model. When importing, the mapping relationship between the surrounding virtual scene and the real scene must be established first. The mapping relationship reflects the coordinate projection relationship between the real scene and the three-dimensional objects in the surrounding virtual scene. According to the mapping relationship, the generated vehicle virtual model can be imported into the surrounding virtual scene, and finally the surrounding vehicles can be restored. Through the restoration of the surrounding vehicle conditions, users can timely understand the distance and route conflicts between the surrounding vehicles and themselves during driving, which effectively improves driving safety when facing special sections such as narrow lane confluence or turns.

[0049] The above are embodiments of the method proposed in this application. Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0050] Figure 2 A schematic diagram of the structure of a vehicle virtual model generation device based on a large model provided in an embodiment of the present application. Figure 2As shown, including: at least one processor; and, at least one processor is communicatively connected to a memory; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for generating a vehicle virtual model based on a large model as described in any one of the above items.

[0051] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows: A method for generating a vehicle virtual model based on a large model as described in any of the above items.

[0052] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0053] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0054] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0055] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0056] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0057] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0058] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0059] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0060] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0061] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0062] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for generating a vehicle virtual model based on a large model, characterized in that: The method comprises: The surrounding vehicle images and environmental parameters within the observation range are collected by a collection device arranged on the vehicle; wherein the surrounding vehicle images are monocular images of continuous frames; Inputting the surrounding vehicle image and the environmental parameters into a preset large model, generating a virtual scene around the vehicle according to the environmental parameters through the large model, and identifying feature points in the surrounding vehicle image; Determining a minimum reconstruction ratio corresponding to the surrounding vehicles according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles; Based on the feature points in the surrounding vehicle images, the surrounding vehicles are three-dimensionally reconstructed to obtain virtual unit models corresponding to the surrounding vehicles; Determine whether the virtual unit model satisfies the minimum reconstruction ratio, and if not, match the virtual unit model with a unit model corresponding to each structure from a preset vehicle model library, so as to assemble a vehicle virtual model corresponding to the surrounding vehicle according to the matched unit model; A mapping relationship between the surrounding virtual scene and the real scene is established, and according to the mapping relationship, the vehicle virtual model is imported into the surrounding virtual scene to restore the surrounding vehicle conditions of the vehicle.

2. The method for generating a vehicle virtual model based on a large model according to claim 1, characterized in that: Determining a minimum reconstruction ratio corresponding to the surrounding vehicles according to the driving intention of the vehicle and the relative position relationship between the vehicle and the surrounding vehicles, specifically includes: Identify, through the navigation route of the vehicle, whether the vehicle has a preset driving intention in a preset road section ahead, and if so, switch the vehicle to a 3D driving mode; Based on the 3D driving mode, determining surrounding vehicles that the vehicle needs to observe and whether the surrounding vehicles and the vehicle are located in a coaxial direction; If yes, determining that the minimum reconstruction ratio corresponding to the surrounding vehicles is the standard reconstruction ratio corresponding to the monocular image; If not, determining, by means of a radar device of the vehicle, the deviation direction of the surrounding vehicles relative to the vehicle and the deviation change rate corresponding to the lateral relative distance between the vehicle and the surrounding vehicles in a number of consecutive observation time periods; According to the relationship between the deviation direction and the deviation change rate and the driving intention, the standard reconstruction ratio is adjusted to obtain a minimum reconstruction ratio corresponding to the surrounding vehicles.

3. The method for generating a vehicle virtual model based on a large model according to claim 2, characterized in that: According to the relationship between the deviation direction and the deviation change rate and the driving intention, the standard reconstruction ratio is adjusted to obtain the minimum reconstruction ratio corresponding to the surrounding vehicles, specifically including: determining whether the deviation change rate is less than 0, and whether the deviation direction is close to the target driving direction corresponding to the driving intention; If so, determining an adjustment coefficient of the standard reconstruction ratio according to a change rate interval in which the absolute value of the deviation change rate lies; The standard reconstruction ratio is adjusted by the adjustment coefficient to obtain the minimum reconstruction ratio corresponding to the surrounding vehicles; wherein the adjustment coefficient and the absolute value of the deviation change rate are positively correlated.

4. The method for generating a vehicle virtual model based on a large model according to claim 1, characterized in that: The virtual unit model is matched with the unit models corresponding to the structures in the preset vehicle model library, so as to assemble the vehicle virtual model corresponding to the surrounding vehicle according to the matched unit model, specifically including: Determine whether there is a direct feature identifier in the vehicle virtual model; wherein the direct feature identifier includes at least one or more of the following: a vehicle logo, a vehicle identification number, and a vehicle model identifier; In the case where the direct feature identifier exists in the vehicle virtual model, determining whether the direct feature identifier can uniquely identify the vehicle model, and if so, matching the corresponding target vehicle model from a preset vehicle model library according to the direct feature identifier, and selecting the unit model for assembling the vehicle virtual model from the unit models of each structure corresponding to the target vehicle model; When the direct feature identifier does not exist in the vehicle virtual model, or the direct feature identifier cannot uniquely identify the vehicle model, extract at least part of the appearance features in the virtual unit model, and calculate the similarity between the appearance features contained in the unit model corresponding to each vehicle model in the vehicle model library and the at least part of the appearance features; According to the similarity, the target vehicle model associated with the virtual unit model is determined, and the unit model corresponding to the target vehicle model is used as the unit model for assembling the vehicle virtual model.

5. The method for generating a vehicle virtual model based on a large model according to claim 4, characterized in that: The method further comprises: According to the structure corresponding to the matched unit model, determining the importance of the structure corresponding to the unit model; Determining the connection importance corresponding to the unit model according to the actual viewing angle relationship between the vehicle and surrounding vehicles corresponding to the vehicle virtual model, and the connection relationship between the unit model and the virtual unit model; According to the structural importance and the connection importance, the assembly importance corresponding to each unit model used to assemble the vehicle virtual model is calculated, and according to the assembly importance, the unit models and the virtual unit models are selected in turn for assembly to obtain the assembled vehicle virtual model, until the vehicle virtual model meets the minimum reconstruction ratio.

6. The method for generating a vehicle virtual model based on a large model according to claim 2, characterized in that: After assembling the vehicle virtual models corresponding to the surrounding vehicles, the method further includes: Comparing the minimum reconstruction ratio corresponding to the vehicle virtual model with the standard reconstruction ratio to determine the size relationship between the minimum reconstruction ratio and the standard reconstruction ratio; When the minimum reconstruction ratio is not less than the standard reconstruction ratio, determining the relative movement speed between the surrounding vehicles according to the relative position relationship between the surrounding vehicles and the vehicle during a continuous observation period; According to the change of the relative motion speed, the operation curve corresponding to the surrounding vehicles in the continuous observation time period is fitted, and according to the operation curve and the deviation direction, the driving parameters of the surrounding vehicles in the next observation time period are predicted; wherein the driving parameters include the driving direction and the driving speed; According to the driving parameters, the driving trajectory of the surrounding vehicles in the next observation time period is determined, the driving trajectory is rendered, a driving trajectory model of the surrounding vehicles is generated, and the driving trajectory model is superimposed on the vehicle virtual model.

7. The method for generating a vehicle virtual model based on a large model according to claim 1, characterized in that: Identifying feature points in the surrounding vehicle image specifically includes: Performing grayscale processing on the surrounding vehicle image to obtain a corresponding grayscale image; Determine a sliding window for performing target detection on the surrounding vehicle image, and traverse each pixel point contained in the grayscale image according to the sliding window; In the process of traversing the pixel point, the pixel point is taken as the central pixel point in the sliding window, and the first grayscale value corresponding to other pixel points around the central pixel point and the second grayscale value corresponding to the central pixel point are determined; For each other pixel point, determine the difference between the first grayscale value and the second grayscale value corresponding to the other pixel point, and the number of other pixel points whose difference is greater than a preset threshold; When the number of the other pixel points is greater than a preset value, the central pixel point is used as a candidate feature point in the surrounding vehicle image; Traversing any surrounding vehicle image, determining the geometric distance between the candidate feature point in any surrounding vehicle image and a candidate feature point pair in another surrounding vehicle image; wherein the geometric distance between the candidate feature point pair and the candidate feature point is the smallest; The ratios of the geometric distances respectively corresponding to the candidate feature point pairs are calculated, and the candidate feature point pairs whose ratios are less than a preset threshold are used as feature points of the surrounding vehicle image.

8. The method for generating a vehicle virtual model based on a large model according to claim 7, characterized in that: Based on the feature points in the surrounding vehicle images, the surrounding vehicles are three-dimensionally reconstructed to obtain virtual unit models corresponding to the surrounding vehicles, specifically including: Comparing the two surrounding vehicle images to determine the overlap of feature points in the two surrounding vehicle images; using the arbitrary two surrounding vehicle images with the largest overlap as target surrounding vehicle images for performing three-dimensional reconstruction of the surrounding vehicles; According to the three-dimensional coordinates of each feature point in the target surrounding vehicle image, the surrounding vehicles are three-dimensionally reconstructed to obtain a virtual unit model corresponding to the surrounding vehicles.

9. A vehicle virtual model generation device based on a large model, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a vehicle virtual model generation method based on a large model as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to: A method for generating a vehicle virtual model based on a large model as described in any one of claims 1 to 8.

Images